ideal input for `search` param in /map endpoint
hi @rafaelmiller @Caleb i was testing out the /map endpoint to fetch a given specific subset of links for a company's domain.
so i wanted to understand what would an ideal query to the
search param should look like ?
for eg. if i want a company's global offices information i would ideally go to the page where locations , offices information is there, soo in the search query i passed location keyword as the param value.
I got the website where the locations are present however i got additional > 50 websites which were not required at all.
so i am hoping to narrow down the filtered list .. ...Why am i not able to use map_url?
```
from firecrawl import FirecrawlApp
app = FirecrawlApp(api_key="fc-xxxx")
...
Is it possible to get a JSON response from CRAWL via API??
Is it possible to get a JSON response from CRAWL via API?? I need JSON in addition to markdown via API call
taking a screenshot of a single element.
Hi i am trying to find a way to take a screenshot of a single element
For example, I want to take a screenshot of a housing data in Federal reserve site
Link: https://fred.stlouisfed.org/series/UTSTHPI
I am assuming that if I set pageOptions as...
Unstable behavior of Crawl Jobs for V1
I'm facing this issue of incomplete crawling where the actual count of pages for the website is different from what the crawl job is able to scrape.
The /map endpoint is giving correct count of URLs in the sitemap, which is 104. [ website - freenome.com ]
but the crawling job is able to scrape sometimes 55, 28 or 29 pages only....
Empty 200-code responses on crawl, but completed on page scrape?
For some sites I get:
- Many empty 200s (md = "")
- but if I just use the scrape endpoint only on them I get a good md back?
I assumed these would be consistent with each other; any idea why?...
How to see why a crawl path was chosen?
Sometimes /crawl doesn't include the base url I set in the set of source urls back.
Ideally the crawler would start at the home page + then some series of urls afterward.
How best to see why it skipped the base url / force it to start there?...
help with scraping inside data
while i used firecrawl to scrape data from a job site it only scraped data from the initial page. but the actual data is present inside the job title link i wanted to extract that data too how can i achievev it? ...here is a sample screenshot of the page
Can I use FireSearch from python sdk?
Reading the doc (https://docs.firecrawl.dev/articles/search-announcement), can I use FireSearch from python sdk? I don't see any info on the doc.
Unable to crawl
I'm unable to crawl this website: https://www.coditas.com
This is my req body:
{
"url": "https://www.coditas.com",
"limit": 5,...
Supabase error
I am gettinng Supabase error after some time. Using docker based self hosted. When i prune and create containers again it works but after sometime it again give same error.
Scraping reddit
any work arounds for this?
[
{
"url": "https://www.reddit.com/r/Vaping/comments/15crn7u/what_are_your_opinions_on_the_smok_nord_4_kit/",...
Social Media Crawling Update
When will social media crawling be available? Specifically for LinkedIn?
Issue with Crawl process for V0
Has anyone been experiencing issues with crawling process for V0 as well?
Seems like the Crawl process is returning None for all the URLs in the data.
Some URLs which were getting crawled easily earlier are now giving issues.
Example: grail.com, immunovia.com...
Need some clarity on the credits usage for /map endpoint
Can anyone provide visibility on the credits usage which would be incurred when using /map endpoint to extract URLs list from the sitemap of a website.
@Adobe.Flash @mogery for visibility....
Best way to get started.
Morning All.
What is the best way to get started without using Python? We normally use n8n.
My use case is to scrape an ecommerce site, and return product descriptions etc. The site is coded terribly and we cannot locate a sitemap. I'm going to attempt to get a list of URLs to thenpoint Firecrawl at.
...
Issue with Crawling Multiple Pages - Only One Page is Crawled
I am experiencing an issue where I am unable to crawl multiple pages; only a single page is being crawled.
What is happening: I am trying to crawl multiple pages, but the crawler is only able to crawl a single page.
Environment: I am self-hosting using Docker.
Setup: I followed the official SELF_HOST.md guide for the setup....
Issue with Null Data Returned After Multiple Attempts (more than 10 Attempts with Python SDK / cURL)
I am writing to report an issue where I consistently receive null data after multiple attempts using both the Python SDK and cURL.
Despite trying various combinations of pageOptions and crawlerOptions, including/excluding different URLs, implementing retries with exponential backoff, and customizing my code to wait and monitor jobs, I am still encountering this problem.
No matter what I attempt, when the limit exceeds approximately 500, the result is always null (it only works for +- 100 urls)....
Firecrawler IP for a whitelist
I'm trying to scrape a website that blocks scraping, but we control.
Is there a firecrawler IP address that we can use to whitelist firecrawler?
Or do we need to self-host...