Raja - Hi Team,I’m trying to use the batchScr...
Hi Team,
I’m trying to use the batchScrape method through a Node.js script, but I don’t see any jobs appearing in the queue on the UI.
Additionally, when I attempt to start batchScrape using the Playground
, it requires a webhook URL as a mandatory field.
Could someone please help me understand how to handle this or share an example of the correct setup?
Firecrawl Docs
Batch Scrape - Firecrawl Docs
5 Replies
@Raja are you using self hosted instance?
Here's how you can use batch scrape without a webhook URL
Thanks @Gaurav Chadha , Also I want to know about how can we crawl or batchScrape, secure webpages, I am trying to pass token in the headers in ScrapeOptions. If you could help on this, It would be really great.
You can use authenticated scraping - https://docs.firecrawl.dev/developer-guides/advanced-guides/authenticated-scraping
Firecrawl Docs
Authenticated Scraping - Firecrawl Docs
Learn how to scrape content behind authentication using cookies
Hey @Gaurav Chadha, you’ve been really helpful. I’m planning to go ahead with the enterprise plan. It would be really great if someone from Firecrawl could be available to assist us whenever needed, at least until we fully get used to using Firecrawl efficiently.
Hey @Raja happy to help always!
For enterprise quires and setup, you can fill this form - https://fk4bvu0n5qp.typeform.com/to/Ej6oydlg also you can DM your email, I can fast forward it to the team.
Typeform
Typeform
Enterprise Sales Qualification Form
Turn data collection into an experience with Typeform. Create beautiful online forms, surveys, quizzes, and so much more. Try it for FREE.