Firecrawl

F

Firecrawl

Join builders, developers, users and turn any website into LLM-ready data, enabling developers to power their AI applications with clean, structured information crawled from the web.

Join

❓┃community-help

💬┃general

🛠️┃self-hosting

Crawlling job got stuck and never returned

For this JOB_ID: abaf9eff-db2a-4848-ae60-f9d3cdee3697 I tried crawling a website but never received the webhook response. The webhook is configured correctly, but nothing was received. When I checked the activity logs, this job doesn’t appear there. After reviewing my own logs, I manually called the endpoint to check the crawl status, and it’s stuck on “scraping.” It seems to be an issue on Firecrawl’s side, since the job isn’t listed in the activity logs and no webhook was triggered...

Images in PDF

Hello, when we try to scrape a PDF, it gets image links for correct locations but the links are sadly not working which results on the response in the image. What can we do?
No description

Check if a specific subpage exists using Firecrawl?

Hey! I’m trying to use Firecrawl (via n8n) to automatically check whether websites have a specific subpage — for example, domain.com/careers. I don’t care about the content, I just want to know if the page exists or not. I’ve already tried several approaches (using the Firecrawl node and also HTTP requests), but the results weren’t reliable. That’s why I’m asking — does anyone know if this is possible with Firecrawl, and what’s the best way to query it?...

scrape via api

https://api.firecrawl.dev/v2/scrape i am scrapping using the v2 api. everything seems to work well but the data i want to scrape dont show up....
No description

Crawlling pdf page does not retrieve entire content

I recently noticed that your system now supports scraping PDF pages — that’s great! However, I ran into an issue while testing it. I’m trying to crawl the following URL: https://www.druva.com/documents/l4-cyber-investigations.pdf Some content appears to be missing from the first paragraph in the extracted result....

How can I retrieve the crawl result of previous job?

I have finished a long crawling job and i wish to get back the result and the documents of the job. How can i do that? I am currently using Firecrawl API. Are there any other method than downloading the json?
No description

Can I do this?

Can I do either or both of the following: 1) Provide Firecrawl with a list of keywords and/or key phrases and have it do a search, and then crawl the returned websites to scrape for those keywords/phrases? -OR- 2) Provide Firecrawl with a list of urls to crawl and scrape for those keywords/phrases? Thank you....

Need custom solution - who can help for $?

We have no coder available for this task - but we urgently need a custom AI crawler using Firecrawl. We have a list of approx. 30k websites to crawl and we need the results (fixed scheme) back into our database (JSON/ rest api). Anyone who can help and is willing to do this for a reasonable budget?...

🤔 There's a Cookie popup in my scrape

Hello. Question: Is there a Firecrawl config setting I can use to prevent popups from getting into my scrapes? All of a sudden to today I am getting "cookie popup content" when I scrape documentation files. I would like to prevent this if possible....
No description

Unexpected Results from Extract

Hello, I'm trying out the /extract endpoint and the results I keep getting back are not what I expect, could someone help me figure out how to get the results I'm looking for? My request: POST https://api.firecrawl.dev/v2/extract {...

Extract isn't returning data

hey i have code from the playground of Extract that works great but the code it gives me to copy into my app returns success but without any parsed data. I've double checked against docs and it looks like something on my account or a bug but i'm a paying user (for many months) so I don't know what to do

500 errors from Firecrawl Scrape API

I am trying to scrape anywhere between 2 and 25 webpages. I get errors every now and then like this: [scrape_webpage] ERROR Response body: {"success":false,"code":"SCRAPE_SITE_ERROR","error":"Specified URL is failing to load in the browser. Error code: ERR_TUNNEL_CONNECTION_FAILED"} Most pages on the same website go through, but rather consistently I get one with this error. I can run the exact same URL and JSON schema through the Playground API and it works. I have tried using Stealth mode and sometimes this fixes it, but I am wondering if this is some form of bot protection by the website. Any recommendations on how to prevent this? The 500 error does not show on the activity logs on the Firecrawl Admin so it almost feels like the issue is with the Firecrawl API itself....

Only One Result Returned When Scraping – Help Needed pls

Hi, I’m having an issue with the scrap badges. No matter what I do, it always returns only one result instead of multiple results. I want to understand why this happens. ( n8n node javascrpit) Here’s a simplified version of my current code: return [...

Data Governance Queries

I’ve reviewed firecrawl's privacy policy and had a couple of questions regarding their data handling: 1. Does Firecrawl log or store the query content (the actual text or data submitted to your service)? 2. Is any of this query data used for training AI models, analytics beyond service improvement, or shared with third parties for such purposes? This information is important for compliance and internal data governance. I had sent an email to help@firecrawl.com but haven't heard back so I thought maybe the discord community could help me progress. TIA!...

🔥 Challenge: Website requires 2FA to disconnect sessions + 3 session limit - is this solvable?

Hi! I successfully automated login on a website (JusBrasil) using Firecrawl actions and can scrape protected content perfectly. The problem: The website has a "max 3 concurrent browser sessions per account" policy. ...

App keeps crashing, serverless function has timed out

I keep getting this error when trying to navigate through my dashboard. I also wasn't able to properly click on things, when I did nothing worked, so I had to hover over the links, get the URL, and then write it on my browser's URL bar. Not sure if it's just me. I've cleared my application memory and cache etc. but doesn't seem to work either.
Solution:
@JuanC @frishtik can you do a hard-refresh and try again, also try cleaning cache, it should work then. The is now deployed for all instances.
No description

Crawl does not work for a specific website, but scrape does

I’m trying to crawl this URL: https://www.investopedia.com/terms/r/request-for-proposal.asp/ Yes, including the trailing /. The crawl endpoint is returning nothing, while the scrape endpoint retrieves the correct information. Shouldn't both work?...