Need Scraping Benchmarking details
I have a use case that involves scraping ~ 10,000 products at a given time. It would be helpful to understand any available data or insights regarding:
1. Latency per request or per batch. I understand FireCrawl has built-in batching logic, overall latency would help.
2. Accuracy of extracted data (especially for structured fields like price, title, description, etc.)
3. Reliability and consistency at scale...
Extract data node in n8n.
Hi All
I am new to Firecrawl, and I am setting up my first Firecrawl extract data node in n8n. I can get everything to work in the playground. I copy everything to the node settings, including the JSON schema. When I try to run it in n8n, I get this error:
...
firecrawler observer is awesome but one question
Is it possible to run firecrawler observer with a self hosted firecrawler instance?
extract schema
hello,
I was playing around the extract endpoint of firecrawl in node (typescript). when i try to hit a url with the schema mentioned, it returns empty object, but when i remove the schema (even though the prompt mentions the same schema fields), it return them propelry in the result object. can someone tell me why? I am attaching the screenshot of the returned object when teh schema is not used, but returns empty object when schema is mentioned along the prompt.
I am attaching the github gist here which will show both the versions in code
https://gist.github.com/palodaman/f7c049661d88e6b2e1327ebe85511b63...
Firecrawl MCP Issues with SSE Transport Protocol
Hi all, I have been trying to use the Firecrawl MCP server connecting with CrewAI, tested through the inspector and in the code itself. In the inspector case, I am able to see the tools but always times out on requests with error "MCP error -32001: Request timed". With the code, it just times out.
Scrape APIs
Hey all! I am looking to use firecrawl to scrape multiple different public API docs and get back all the endpoints — should I use scrape or crawl and are their any examples of anyone doing anything like this ?
billing successful scrapped urls after timeout crash in n8n workflow
Hello everyone,
I have a workflow in n8n where I used an HTTP request to Firecrawl (scrape) with a default timeout of 30000ms. The node failed after reaching the timeout. I can see in the Firecrawl activity logs that most urls were successfully scrapped. I have removed the timeout constrain to avoid this issue. If I run my workflow again on the same urls batch, will the ones that have been previously scrapped billed again? I was hoping that Firecrawl would identify those scrapped urls and return the data for free, instead of having to download each markdown by hand from the activity log. I haven't found anything about this in the doc or the faq.
Fred....
Help! Half or our scrape is getting failed.
We are scraping thousands of product pages and half of our scrape is getting this error. "Specified URL is failing to load in the browser. Error code: ERR_TUNNEL_CONNECTION_FAILED"
I checked the activity logs and i see its failed. you can check the screenshot.
Please help us. thank you...
Configuring Authorization Header for API Access: Only Authorized Users Should Access the Data
WSL,Firecrawl, Docker)
I’m working on a project where I scrape data from an API with Firecrawl inside a Docker container running on WSL (Windows). I need to restrict access to the API based on the Authorization header:
Only users with a valid Authorization header should be allowed....
Scraping websites with an added click needed
Hi guys i have recently been getting into ai / coding / automation through lots of different sites (Zapier, relevance, ect) now i have been given a task, it seems so damn easy but i have spent 2 days desperately trying to get this to work to no avail.
My task is I’m trying to scrape information from a website every Monday morning at 9am right but theres a catch, i cant just pull the web page i want because it returns an error, i think its because it needs to go to the search page first or something but the way i have to do it is go to url press the search button then scrape the search results, thats all! It seems so simple but yet i have just been struggling so bad.
I went through the fire crawl website and went to scrape then added the agent and put in the prompt “go to url, press search and scrape the search results” and IT WORKED!!! I was like f*** yes it can work but then sadly i discovered that when trying to do this in zapier so it can trigger at 9am every monday > go through a google spreadsheet 1 row at a time and get the urls > post them 1 by one to fire-crawl and then of course scrape the desired search data and add the results to a spreadsheet. It just returns the page that i don’t want the one before the click (using the scrape function i think I’m not to sure as I’m super new to all of this so apologies ) ...
How to setup supabase in self-hosted environment?
Hi, I setup firecrawl in a self-hosted docker environment. First things work awesome, now I would like to get screenshot - from the code I learn, I have to setup supabase. I self host supabase too, from my understanding I need to initialize the supabase (postgreSQL) environment. Is there a call or a schema avalailable?
Getting static response on execution of scrapeUrl
I have integrated firecrawl in nodejs application and I am getting static response on each scrapeUrl call which is
```Introducing Authenticated ScrapingScrape behind auth walls. Join our waitlist today.
:boom:Get 2 months free with yearly plan
The fast, reliableFast, reliable web scraper for LLMs...
API Issue: Asynchronous Endpoint POST /v1/batch/scrape Fails, Synchronous Works
I have done extensive debugging and have isolated a critical issue with your API that is specific to my account.
Summary of Findings:
My API key is valid. The synchronous await app.batch_scrape_urls() method works correctly and processes URLs....
Bug on Version 2.14.0
The python SDK version 2.14.0 has the following code defined in the _async_request function:
ssl_context = ssl.create_default_context(cafile=certifi.where())
async with aiohttp.ClientSession(ssl=ssl_context) as session:...
how can i locally use fireCrawl using my open ai key and do self hosting.
Hello, how can i locally use fireCrawl using my open ai key and do self hosting as well. I want to utilize the scrape and extract feature but dont wanna buy its subscription. i am using FastApi
Intercepting network request data
Hi,
I'm wondering if it's possible to get the data from one of the network requests sent from a page, rather than getting the data off the final page output?
The website I'm trying to scrape sends an API network request for a large amount of data, but only outputs a small subset of it to the actual page. If I could just intercept the API request I could get all of the data. Unfortunately I can't just hit the API directly either....
I am trying to scrape pages from indeed website but it is giving 502 error
I am trying to scrape indeed urls. ,till now it was scraping but now giving 502 error, why ? How to fix it
How do I use Firecrawl to scrap changes made to my airtable ticket?
So, I have a few tables in my Airtable. Each table contains many tickets. I want to write a custom web scrapper to scrap the changes from tickets. The changes can include status changes, comments, updates to the ticket title or description and more. Is there any way I can achieve this using firecrawl? Sorry I am new to web scrapping so I have no idea if this question makes any sense or not. But I would love to have any leads from the brilliant minds here.