scrape's markdown contents different on API vs Playground
When I run a scrape on a URL via the API, I get a significantly larger response size than I do when it is run on the playground. The markdown content can be up to 10 times larger.
My API function is just using the standard scrape request config:
const response = await fetch(FIRECRAWL_API_URL, {
method: 'POST',
headers: {
'Authorization':
Bearer ${process.env.FIRECRAWL_API_KEY},
'Content-Type': 'application/json'
},
body: JSON.stringify({
url: formattedUrl,
formats: ['markdown'],
onlyMainContent: true,
timeout: 30000
})
})
Are there any additional configs I should be using to get my results more inline with the Playground?
4 Replies
Do you always get this difference? If you run it on the playground, let's say, 10 times, do you get more content every single time? Trying to understand if this is something related to the Playground vs. API or not.
After running a bunch of tests on this in bulk, I realize that the issue is not API vs. Playground, the issue is that the same URL with the same config can product wildly inconsistent results each time it is run. Both API and Playground runs are prone to this inconsistency of results
Did you test with adding a WaitFor value to ensure the page has enough time to load correctly? The playground may also have some default parameters theyre not exposing like removeBase64Images: true , which could be causing the discrepancy
Yup yup yup, that appears to give me consistency. Thanks for the suggestion @AlexM