Cannout get around of cloudfare with the nodejs API call
This is my code
const params = {
pageOptions: {
includeHtml: true,
onlyMainContent: false,
waitFor: 8000
}
};
const scrapedData = await firecrawlApp.scrapeUrl(link, params);
if (!scrapedData.success) {
throw new Error('Firecrawl crawling failed');
}
console.log(scrapedData)
17 Replies
I noticed that the website play ground has no issues though. What's the reason for the difference?
The curl request will just give a unexpected error
updated to the newest js-sdk to try and still get the cloudfare error
@thomas cloudflare is blocking this one. Can you look into it?
the issue is only with the js-sdk . Not sure why the website playground is good but the js-sdk is bad
weird
that looks like a different url from the one you tried on the playground
ccing @rafaelmiller here for visibility in the js sdk in case that has smth to do with it
I tried two URLs under the nivod.cc domain with js-sdk. Both don't work
see here: https://discord.com/channels/1226707384710332458/1281654421322862754/1281654421322862754
it is also having the cloudfare issue.
hey @Emma ! I tested the js-sdk with both urls and didn't get the cloudflare pages you're seeing. Are you self-hosting firecrawl?
No, we are using the js-sdk. Which website did you test?
Did you test with https://nivod.cc/ ? It will trigger the cloudfare 100% of time
this is the screenshot I got
https://alttmdsdujxrfnakrkyi.supabase.co/storage/v1/object/public/media/screenshot-9946803.png
I got this screenshot with 30000ms waitfor time
https://alttmdsdujxrfnakrkyi.supabase.co/storage/v1/object/public/media/screenshot-7159385.png
this is the scrapedData
this shouldn't be happening. @thomas have you had a chance to look into this?
it is still happening
The difference is that previous it get resolved 0% times . Now, it get resolved 50% . But there are still chances that the cloudfare protection is still resolved and the result is returned back. See the Slack channel for examples
https://alttmdsdujxrfnakrkyi.supabase.co/storage/v1/object/public/media/screenshot-4813021.png
This is what happening this morning. The fix is not stable
