Error code when trying to create API Key
I am trying to set up an API key with firecrawl, but it gives me an error message stating, " An error occurred in the Server Components render. The specific message is omitted in production builds to avoid leaking sensitive details. A digest property is included on this error instance which may provide additional details about the nature of the error."
How do I fix this and stop this from happening?...
Prevent crawling of .xlsx / non-text/html pages
Is there a way to filter out pages based on content-type header? There is a link in some docs we are crawling, which downloads a .xlsx file.
Firecrawl returns this as raw text, unreadable utf tokens like if you take the .xlsx and change to .txt.
We use formats: ['markdown'], I don't understnad how this is possible
thanks!...
How can i set headers when scraping with Python SDK?
As the title suggests i don't see a headers parameter in the scrape_url function in the python sdk. Can i not set the headers when scraping one page?
/scrape works perfectly if including screenshot for some links but errors if not
I've been trying to scrape some news links and found for example this link "https://www.theverge.com/ai-artificial-intelligence/679768/reddit-sues-anthropic-alleging-its-bots-accessed-reddit-more-than-100000-times-since-last-july" will produce an error for me in the playground if I don't check the box for a screenshot but if I do it works without issue. Any reason this is occuring as I don't need the screenshots but may just have to leave it on if it helps firecrawl.
Can’t get MCP to work with Docker locally. Any tips?
Literally trying this but not working: https://hub.docker.com/r/mcp/firecrawl
FIRE-1 agent is not able to scrape javascript dynamic content. It is erroring out every time.
code:
scrape_result = app.scrape_url(
'https://www.classmateshop.com/',
formats=['markdown', 'html'],...
402 Errors Randomly Started With Plenty Of Credits Left
We have a service that has been running for a couple of weeks with no problems.
Today, we started to get 402 Payment Required response, despite having 50% or more of our monthly credits left.
I've seen that people were experiencing this back in February. Was the issue resolved, and if so, how was it resolved?...
Crawl Status Response type not available for import
Hello everyone, im trying to do an async crawl and then using the next url to get the next chunks of data. I want to use the CrawlStatusResponse class to transform the response in an object, is there any reason why this type is not available for import?
Botpress
Hi, I'm building chatbots using Botpress, I read in your blog that they use Firecrawl, but when I try to add a dynamic website URL to a knowledge base it doesn't scrape the content, so I was wondering if they haven't implemented Firecrawl yet, or I'm doing something wrong, or they implemented it in other part of the platform. I'm very interested in automating webscraping data for these chatbots, so if Botpress doesn't have it yet, maybe Firecrawl is the solution to achieve this.
400 Response from /extract using wildcard
Hey everyone!
I've had success using /extract for individual page URLs, but I am trying to refactor to use the wildcard URL method for efficiency's sake. When making the call, verified with the correct URL and the added "/" , I get a 400 response, specifically " 🔥 Error during Firecrawl fetch: Request failed with status code 400". I will attach the file in question, but the function named "fetchProductsFromDomain" is the one that uses the wildcard. My logs are as such: " 🚀 Fetching products for competitorId: b1e5v3d5yt17PDiKhcPK, baseUrl: https://busy-bounce.com/
Attempting scrape using full domain
Wildcard URL: https://busy-bounce.com/...
Crawl a lead website to create a knowledge base
Hey guys, so I'm new to Firecrawl and I figured I'd find a response quicker if I post it here
For the context:
I'm creating a chatbots and voice ai assistants powered by LLMs for businesses.
...
Agent used up all of my credits...
Since it looks like there is no longer the ability to simply define max pages and page depth, I used the Agent to get content as it seemed like that was the only option besides listing every URL, and it ate up all of my credits in about 30 seconds. Not only that, but it sent me WAY over my limit, so when I tried to re-up, I still couldn't use the service, and to make matters worse, when I realized what was happening, I tried to stop the run, didn't see a button, closed the browser tab, and it s...
EXTRACT API ALWAYS GET SAME EMPTY RESPONSE:
{
success: true,
data: {},
warning: undefined,
error: undefined,...
Request Timeout in Python SDK uses milliseconds but should be seconds?
The scrape_url method in the python SDK expects timeout to be passed in milliseconds and passes it to the requests library without unit conversion.
```python
class FirecrawlApp:
def scrape_url(...):
......
Firecrawl API stopped working when using proxy – anyone else experiencing this?
Hi everyone,
I've been using the Firecrawl API for a while now, and up until today, everything was working fine when using a proxy.
However, today I started getting SSL-related errors like:...
How to set version on the Firecrawl node SDK?
I am trying to Self host Firecrawl.
It works when i make axios commands but doesn't when i instantiate the application and then try to execute any jobs. Seems to be maybe a version mismatch. The FirecrawlAppConfig does have a version parameter, but not initialized in the constructor, I don't know if that's something that can be set, with the app but it can be specified in other cases. Crawling doesn't work when using in SDK but works whne using SDK via local host ...
It works when i make axios commands but doesn't when i instantiate the application and then try to execute any jobs. Seems to be maybe a version mismatch. The FirecrawlAppConfig does have a version parameter, but not initialized in the constructor, I don't know if that's something that can be set, with the app but it can be specified in other cases. Crawling doesn't work when using in SDK but works whne using SDK via local host ...
Metadata structure of using firescrawl on relevance.ai
Hi guys, I am new here and was trying to solve the issue for entire days: data crawled from the url always appeared as below (with a metadata form). And I can't find automatically {{firecrawl.data[*]}}
External Webpage Scraping
Hello All,
I have a question: If I scrape a website and it contains links to other external websites, will those linked pages also be scraped automatically?"...
Using firecrawl crawl inside Make.com
Any guides or documentation for this?
I can't see to get an output I can use - it's just returning a job id - which is something it won't work in my workflow
It works well with scrape...