F
Firecrawl
Join builders, developers, users and turn any website into LLM-ready data, enabling developers to power their AI applications with clean, structured information crawled from the web.
JoinF
Firecrawl
Join builders, developers, users and turn any website into LLM-ready data, enabling developers to power their AI applications with clean, structured information crawled from the web.
JoinHow can I use Firecrawl to crawl all the content from my Twitter search results?
Previously, I used to search for specific keywords on Twitter and read through the posts one by one to understand people’s current opinions and announcements.
Now, how can I use Firecrawl to directly crawl this information and extract frequently mentioned keywords?...
Take a long screenshot
Hi,
I'm having issues taking screenshots of long pages. It cuts off half way, returning the rest of the page in white.
I thought it may be a size issue, so had the idea of capturing multiple screenshots at different spots and stitch them together. But I haven't been able to get this to work (it just takes multiple screenshots of the initial viewport)...
Scraping not returning with FIRE-1
Hello,
I have a scraping task (all in python). My approach is to use the default scraper, review schema results, and if they aren't great, I try again using FIRE-1.
I'm using async in python to wrap a timeout around this, and I set a 120 second timeout, and the FIRE-1 results still are not returning. However, in the playground, I do see it work. ...
Unable to scrape and extract data. Starts to Parallelise then nothing happens
URL is this: https://am-arrowmax.com/collections/*
Prompt:
The Schema was auto generated...
Follow every product link to ensure all variants, including sold-out ones, are included. Capture the title, all variants (size, color, etc.), images per product and variant, price, and product/variant URLs. Ensure a clear mapping of product to variants to images to price to URLs.
Follow every product link to ensure all variants, including sold-out ones, are included. Capture the title, all variants (size, color, etc.), images per product and variant, price, and product/variant URLs. Ensure a clear mapping of product to variants to images to price to URLs.
Country not supported in proxies
Is there a way to scrape a site whose country is not supported in the proxy lists ?
For example i am from greece and some specific sites i want to scrape have access only in my country!...
Help with scraping a paginated website using Firecrawl Actions
Hey all! I'm new to Firecrawl. I have a website I'm looking to scrape that is paginated (think several "next" and "previous" buttons that javascript render different content without changing the URL).
How would I do it with actions? I'm familiar with the "wait" and "click", but how would I actually aggregate the scraped content for each paginated page? the "scrape" isn't working, but I'm probably using it wrong:
"actions": [...
Simple hello world on Search returning error, missing "integration" parameter in payload
I'm getting the following error in the most simple code for search
the code is:
```python ...
firecrawl.v2.utils.error_handler.BadRequestError: Bad Request: Failed to search. Invalid request body - [{'code': 'custom', 'message': "Invalid enum value. Expected 'dify' | 'zapier' | 'pipedream' | 'raycast' | 'langchain' | 'crewai' | 'llamaindex' | 'n8n' | 'camelai' | 'make' | 'flowise' | 'metagpt' | 'relevanceai'", 'path': ['integration']}]
firecrawl.v2.utils.error_handler.BadRequestError: Bad Request: Failed to search. Invalid request body - [{'code': 'custom', 'message': "Invalid enum value. Expected 'dify' | 'zapier' | 'pipedream' | 'raycast' | 'langchain' | 'crewai' | 'llamaindex' | 'n8n' | 'camelai' | 'make' | 'flowise' | 'metagpt' | 'relevanceai'", 'path': ['integration']}]
Extract playground returns expected results on web. API extract returns empty.
I used the web-based extract playground to figure out prompt + URL and it is returning the expected results.
However, when I copy the generated code from the playground to a python file, each API call returns an empty set.
```from firecrawl import Firecrawl...
Help http request firecrawl v2 in N8N
Could someone help me with cURL for HTTP requests within n8n for "map," "extract," "search," and "batch scrape"?
I only managed to get "scrape" and "crawl" to work....
firecrawlapp or firecrawl class help
why does it still say that Firecrawl class doesnt exist and makes me use FirecrawlApp?
I am on the 4.3.1 ver and docs say that there is no FirecrawlApp
ImportError: cannot import name 'Firecrawl' from 'firecrawl'...
Detecting Redirects
I'm using SCRAPE endpoint, is it possible to determine if you've been redirected from the original URL you supplied? Would love to extract the final URL I land at in this case
Got pydantic error when import firecrawl python v2
Hi, I've just imported Firecrawl / AsyncFirecrawl (from firecrawl import Firecrawl, AsyncFirecrawl) in a FastAPI application and got an error: pydantic.errors.PydanticUserError: Decorators defined with incorrect fields: firecrawl.v2.types.SearchRequest:4709679808.validate_parsers (use check_fields=False if you're inheriting from the model and intended this)
Version:
- Pydantic: 2.11.7
- firecrawl-py: 4.1.0...
ID change between extract and get extract status
Hi, I have a weird think happening in N8N.
When a trigger an extract I recieve an ID. Normal
When I use the get extract status with this ID : status is processing
but when I check on the playground I see that the extract is done but the ID is not the same than the one I got. When I copy the ID from the playground to my N8N node manually in the get extract status : i works and I recieve the data.
Anyone have a clue on what happens ?...
Excluding hidden elements from HTML or Markdown
Hey all... I'm crawling a page that has
display: none elements, and they are being included in both the HTML and the Markdown. Is there any way to exclude these?
Thanks!...Firecrawl seems to ignore URL hash fragments for pagination.
- Site: https://europa.provincia.bz.it/it/bandi-e-avvisi (pagination via
#start=N)
- Expectation (Playwright): total=15, page 0 -> 10 items, page 1 -> 5 items (selector div.result.lv_faq).
- Firecrawl scrape_url results: page 1 returns same content as page 0 or misses p.result_status entirely.
- Tried: formats=["html"], formats=["rawHtml"], wait_for=12000–20000, max_age=0 (fresh fetch), correct base URL from container to host.
...Building Chrome Extension
Hi, can someone from the team or anyone (who has tried this) answer if we can build a Chrome Extension if so any video, guide or documentation, please.
Search w/ JSON
hey everyone!
I am trying to test out Firecrawl versus our app's current Perplexity integration, and trying to do a bit of a head to head comparison between sonar-pro w/ structured JSON, and Firecrawl's /search + json schema extraction.
First, I am struggling to get search to return ANY json from the search sdk/endpoint. Is this working as of V2? I'm just trying to get any search or scrape result to return json according to my simple testing schema, and neither are working for search OR scrape, I'm using the v2 SDK....
Trouble extracting URLs from a JavaScript-based paginated search
Hi everyone,
I'm struggling with scraping a list of URLs from a search results page. The site uses JavaScript to load 20 results at a time, and to get the next 20, I have to click a "Next" button. There’s no change in the URL or any query parameters that I can use to trigger pagination directly.
I was thinking about using actions in Firecrawl to simulate clicking through the pages and extracting the links, but I’m not sure if that’s the right approach or the most efficient one....
Have a problem working with Firecrawl (im a beginner)
In all of the tutorials on Youtube for Firecrawl, the first step of extracting is to ''generate parameters''. Problem is that I cant seem to find this generate parameters. I know its a really noob question probably but any help is appreciated 🙂
