Cloudflare Workers proxy for Web Scraping?
For Web Scraping can i use cloudflare workers as a reverse proxy? Does it really help to prevention of ip block of target server? (want to use it for https://pypi.org/project/instagrapi)
8 Replies
Workers aren't meant for that. They are made the opposite if anything, in the CF-Worker header it identifies the website/zone that send the request.
Also Workers at the moment don't support Python really. You can do some stuff with transpiling to JS or maybe even compiling to WASM but it's not really supported/not going to find any docs on it/etc
i just want to use workers as a proxy as i get an option while creating a new project in wrangler
You can use Workers to Reverse Proxy to stuff you own yea. It can be really helpful with rewriting/dealing with weird stuff
but main question is Does it really help to prevention of ip block of target server? (want to use it for https://pypi.org/project/instagrapi)
*target server = main server
I'm not sure ToS wise but like I said they're just not made for that. Every fetch request you do in a Worker identifies the website it came from in the CF-Worker header, so people could easily see which user is doing it. The IPs they use if the target is a CF is the same
2a06:98c0:3600::103, otherwise they use a smallish set set of Proxy IPs
Some websites like Discord block all workers from fetching their CDN by blocking any request with the CF-Worker header (and you can't remove that)I am not worried about my server's ip because I don't do any illegal stuff just try to prevent my server's ip from being blocked from Instagram servers that's why (for image downloader app)
last question atleast my server ip will be isolated or not if i use it according to you???
are you there??
I mean your worker will use the same IP as other people that use workers so depends what you mean by isolated
If the target you're fetching is CF, your IP will come through as
2a06:98c0:3600::103. If it's a non-CF site, it'd be one of the shared normal Proxy IPs