Firecrawl

F

Firecrawl

Join the community to ask questions about Firecrawl and get answers from other members.

Join

鉂撯攦community-help

馃挰鈹僩eneral

馃洜锔忊攦self-hosting

Crawling Website Issues with n8n

Hi all! I am having trouble crawling websites (from Google Sheets) looking for specific keywords. Not using a main url. Here is an example: https://www.whitehouse.gov/presidential-actions/executive-orders/ Looking for keywords like:...

Question about PDF Scraping

Hey team 馃憢, I'm working on a data scraping task and had a question about Firecrawl's capabilities for a specific use case. My Goal: To extract interest rates for Certificates of Deposit (CDTs) from a bank's website....
No description

Pricing Model

Sorry guys, I've really tried hard to understand the pricing model, but I'm confused by the overlap. I understood (maybe wrongly) that extract was like an upgraded / higher volume plan than Standard. I've paid for Starter Extract, but still I've exceeded my credits. Do I also need to upgrade my standard API subscription too to get more credits?
No description

One Time Credit Purchase

I'm trying to figure out how to purchase credits. I'm on the Hobby plan and want to buy 2,000 credits for a specific project. However, the purchase page makes it sound like I'm signing up for a recurring monthly purchase. Is there a way to make a one-time purchase of credits to add to my current month's usage?

Firecrawl batch cannot crawl some urls

Hello everyone I am using firecrawl python sdk when I use batch_scrape_urls it can't crawl. Just 1 url is not successfully crawled it will logs error. I wish it will ignore the error url and execute the remaining urls. I have looked for the "ignoreInvalidURLs" attribute but can't find it in python sdk. here is my list of urls: ['https://www.britannica.com/biography/Stephen-Colbert', 'https://www.biography.com/movies-tv/stephen-colbert', 'https://www.cbs.com/shows/the-late-show-with-stephen-colbe...

Extract API - Captcha or Bot Solution

Guys, anyone know if Extract API has captcha/bot detect solution embedded? Because I'm trying to crawler a site and I received status success but the data of crawler was empty.

Can I cancel a subscription plan?

Could someone please tell me if I can cancel a Hobby or Standard plan at any time or once I start I have to pay the whole year monthly?

[self-hosted] Is there a way to get more logs or to change the logging level?

Is there a way to get more information on failed scrape requests when running a self-hosted firecrawl instance? I'm often getting an error which is not very self-explaining (the scraped url is not broken!) This is the full error message: ...

Batch Scrape Webhook

Hey, I am unable to get the webhooks for batch scrape after getting the first one i.e batch_scrape.started. getting this...
No description

Crawl tool is very slow.

I'm using Standard plan. But when crawling a site, it is very slow.
No description

Crawl status tool is returning 500, need help to verify whats wrong with the request.

I am calling firecrawl_check_crawl_status tool in remote mcp with the following argument. {"id": job_id}. Acticity logs shows just one page being crawled although I set depth 2, limit 3. Posted a screenshot of activity logs (showing a different jobid). I have set a depth and limit but the log says one one page is crawl and 5 credit is used. Response: {'result': {'content': [{'type': 'text', 'text': 'Error: Failed to check crawl status. Status code: 500. Error: An unexpected error occurred. Please contact help@firecrawl.com for help. Your exception ID is 59b6e23c4a03481fa61b958c7b08c15c'}], 'isError': True}, 'jsonrpc': '2.0', 'id': 1754498520281}...
No description

getting 404 error in trying to scrape reddit url

any one seeing the same "Error: Request failed with status code 403. Error: This website is no longer supported, please reach out to help@firecrawl.com for more info on how to activate it on your account. "

Keeps getting operation timeout when /scrape

Crashed at this ``` if (result === null) { if (Object.values(meta.results).every(x => x.state === "timeout")) { throw new TimeoutSignal();...

Running docker compose unable to start worker because of max old space

Hey Community! I am playing around running firecrawl with docker-compose on my silicon mac, and after fixing the error with targeting arm; i am seeing an issue that the worker is started with invalid param: 鈥攎ax-old-space=8192 Is there a way to change this param?...

Extraction with search URL plus body

Does Firecrawl currently support the extraction of content that requires attaching a body to the search request? For example, to get product content from a specific retailer's website, chances are that it is location/store-based. E.g., I want to extract all the products from the search term 'toilet paper' from Target, the url will be something like https://www.target.com/s?searchTerm=toilet+paper, but the contents that appear are dependent on the location I'm browsing from or a selected store. And this information is not part of the actual URL pasted above, but inside the body....

Exploring TwitterXApi's Advanced Twitter Search Endpoint: A Guide to Fetching Trump's Tweets on X

API Overview The endpoint is a POST request to https://api.twitterxapi.com/twitter/advanced_search. It requires Bearer token authentication (get yours from the TwitterXApi dashboard). The request body is JSON with search parameters. import requests...

Extract Endpoint Failing Sometimes

Hey, I'm seeing the extract endpoint fail sometimes for the same url with an error saying the url is invalid even though it succeeds for that same url at other times. What's going on and how do I debug this?

Crawl Is Scraping Website Too

I've had a service running for a while, that uses the /crawl endpoint to crawl 15 pages of a website and has run with no issue for some time. Recently I've noticed that it is also scraping each of the 15 pages individually, as well as doing a separate crawl. This is causing us to use double the number of credits per website. Is this a know issue or knock-on from a recent change?...

Need Scraping Benchmarking details

I have a use case that involves scraping ~ 10,000 products at a given time. It would be helpful to understand any available data or insights regarding: 1. Latency per request or per batch. I understand FireCrawl has built-in batching logic, overall latency would help. 2. Accuracy of extracted data (especially for structured fields like price, title, description, etc.) 3. Reliability and consistency at scale...