Find Playground Settings and Rotating Proxies?
Is there a way to find the Playground settings?
Sometimes when I'm using Playground, it will scrape URLs successfully, but when I use it in my app (using Cloud version with credits), the same URL will not scrape.
For example: https://www.mountsinai.org/health-library/selfcare-instructions/common-cold-how-to-treat-at-home
When using the code from Playground, it doesn't work in python. I've already tried adding "'proxy': 'stealth'," but that doesn't help either.
I'm assuming it's a proxy issue, but both should have rotating proxies?
Shouldn't the cloud version API be using the same as Playground?
10 Replies
Hey! What error are you getting when scraping via the API?
Hi Gergo! Here's the 403 response I get:
{'markdown': '# Access Denied\n\n\nYou don't have permission to access "http://www.mountsinai.org/health-library/selfcare-instructions/common-cold-how-to-treat-at-home" on this server.\n\nReference #18.a924c317.1742466464.2069b229\n\nhttps://errors.edgesuite.net/18.a924c317.1742466464.2069b229', 'metadata': {'title': 'Access Denied', 'scrapeId': 'bb8f7758-7f8d-4e0d-b484-6ce169afdf7f', 'sourceURL': 'https://www.mountsinai.org/health-library/selfcare-instructions/common-cold-how-to-treat-at-home', 'url': 'https://www.mountsinai.org/health-library/selfcare-instructions/common-cold-how-to-treat-at-home', 'statusCode': 403, 'error': 'Forbidden'}, 'links': [], 'url': 'https://www.mountsinai.org/health-library/selfcare-instructions/common-cold-how-to-treat-at-home', 'scraped_at': '2025-03-20T10:27:47.213018'}
but in Playground, they work fine. The 403 errors usually come from larger sites.
These are separate from the "Cloudflare security bot blocks", which also sometimes work in Playground.
But never with API.
Are you using the self-hosted API? Or the cloud version?
Cloud Version
I didn't want to deal with proxies on my end, so that's why I'm using the cloud version
Hi there! We are currently experiencing some issues with Cloudflare, will look into this further. CCing @thomas on this
Hey, looking into it
Thanks guys!
Hey, I just tried and wasnt able to replicate the issue with this site. Can you try again and let me know if you still get it? If so can you provide the full request you are sending?
You were successful with the API?
Did you use any specific settings/parameters with the API?
Like 'stealth'?
Or user agents or anything.
Just to reiterate @thomas , the Playground is always successful.
The API is not.
Couple more examples where Playground works perfectly, but API doesn't:
1. https://www.npr.org/sections/thesalt/2015/05/21/405190434/chew-on-this-the-science-of-great-nyc-bagels-its-not-the-water
- firecrawl response: Internal Server Error: Failed to scrape URL. (Internal server error) - Specified URL is failing to load in the browser. Error code: ERR_HTTP2_PROTOCOL_ERROR - No additional error details provided
2. https://www.rd.com/article/why-bagels-better-in-new-york/
- firecrawl response - Internal Server Error: Failed to scrape URL. (Internal server error) - timeout - No additional error details provided.
Hey, no we use the same parameters and everything on the api and playground. Can you give me some of the full api requests you are sending that are causing the errors?