F
Firecrawl•11mo ago
echoes221

Webhook is intermittent

I've been dabbling with the webhook for scrape and I've noticed that its very intermittent/flaky. I don't get all the events through (started/page/completed), and if I do, I usually just get page events through. At the moment I'm only batching one URL through with plans to expand to more, but I really need this functionality to be consistent/reliable. Is there an issue with webhooks currently? This is how I'm calling: 
 const result = await fetch("https://api.firecrawl.dev/v1/batch/scrape", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${context.cloudflare.env.FIRECRAWL_API_KEY}`,
    },
    body: JSON.stringify({
      urls: urlsToScrape,
      formats: ["rawHtml", "links", "markdown"],
      webhook: callbackUrl,
      timeout: 25000,
      waitFor: 3000,
    }),
  })
    .then((response) => response.json())
    .then((response) => {
      console.log(response);
      return response;
    })
    .catch((err) => console.error(err));
 const result = await fetch("https://api.firecrawl.dev/v1/batch/scrape", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${context.cloudflare.env.FIRECRAWL_API_KEY}`,
    },
    body: JSON.stringify({
      urls: urlsToScrape,
      formats: ["rawHtml", "links", "markdown"],
      webhook: callbackUrl,
      timeout: 25000,
      waitFor: 3000,
    }),
  })
    .then((response) => response.json())
    .then((response) => {
      console.log(response);
      return response;
    })
    .catch((err) => console.error(err));
And I have a post endpoint for the webhook url, in local dev I'm using localtunnel URL which works great for stripe/clerk etc.
2 Replies
mogery
mogery•11mo ago
Hi there, we are working on improving the webhooks' reliability. Thanks for reporting, taking a look.
echoes221
echoes221OP•11mo ago
@mogery thank you very much, we're looking to move to paid tiers as want to base some of our BL around this functionality, so any updates on progress/timelines would be much appreciated 🙂

Did you find this page helpful?