Proxy URLs
I am trying to scrape a website and save some data from it. I am using gotScraping to fetch the url but when I try to use proxyConfiguration in gotScraping options object I am getting this error
The "Proxy external access" feature is not enabled for your account. Please upgrade your plan or contact [email protected]
I have tried some other combinations also but I got 403 forbidden error
In my console tab I can see 5 data center IPs under my account and I am copying that url only but still I got error
Has anyone have any experience how to scrape data with proxies ?
Also If I deploy my actor on the store and Proxy is needed for my script to run then will the charge for those scrapings will be from my account or the user using my actor
6 Replies
continuing-cyan•2y ago
How are you passing proxy to gotScraping?
I guess you should use newUrl() method :
https://crawlee.dev/api/core/class/ProxyConfiguration#newUrl
In case of publishing to the store, user will pay for proxy. Not you.
rival-blackOP•2y ago
I have cloned the sample project and trying to run apify run
below is my code in main.js
import { Actor } from 'apify';
import { PuppeteerCrawler } from 'crawlee';
import { router } from './routes.js';
await Actor.init();
const startUrls = ['https://apify.com'];
const proxyConfiguration = await Actor.createProxyConfiguration();
const crawler = new PuppeteerCrawler({
proxyConfiguration,
requestHandler: router,
});
await crawler.run(startUrls);
await Actor.exit();
After commenting the proxyConfiguration code I was able to run the code successfully but I think it won't be safe for scraping
graceful-blue•2y ago
@shivendra5452 free plan has external proxy access disabled, meaning you could use it only on Apify platform.
rival-blackOP•2y ago
OK once I deploy my actor on apify I can test it with proxy
Also will proxy be auto enable for user who use my actor or do I have to make a setting for this
graceful-blue•2y ago
you could set proxy as required in Input schema, or hardcode it in the code
then it won't be possible to run it without proxy
rival-blackOP•2y ago
how can I set proxy as required is it through the isSecret parameter in input_schema property
https://apify.github.io/input-schema-editor-react/
I am creating input_schema using this link
Apify input schema editor
Apify input schema editor