Python Apify Configuration
I have a Python script for scraping zoopla , i need help in configuration settings of apify . My actor is running really slow , I am using requests module to get url response. I wanna know how to configure concurrency, proxies and request retries.
And also wanna know if I should use requests module only or does apify has some module for python to get URL response
3 Replies
correct-apricot•3y ago
Hello, we currently do not provide scraping library for Python so you will need to use the available ones. I'm sure that e.g. Scrapy manages concurrency. (we plan to create easy Scrapy integration though with tutorials)
As for proxies, look at the Python examples here https://docs.apify.com/proxy/datacenter-proxy/examples#use-ip-rotation
Apify
Examples · Apify Documentation
Learn how to connect to Apify's datacenter proxies from your application with Node.js (axios and got-scraping), Python 2 and 3 and PHP using code examples.
afraid-scarletOP•3y ago
Can you come one on one in DM, want to share my code, and also I wanna know how can i integrate output results to google cloud or bigquery
correct-apricot•3y ago
I won't really help you with Python scraping library since we are not yet into it that much ( we are just starting with broader Python support) but feel free to DM me