CA
absent-sapphire
Actor is Not changing the IPs after setup Sessions
I'm reaching out to the community for some assistance with an issue I'm encountering while using Playwright and Apify. I've built a scraper that incorporates proxy rotation with sessions, but I'm facing a problem where the proxy isn't changing when the session changes. Additionally, I'd like to ensure the same IP address is used for both the proxy and the captcha solver.
Problem Description:
Playwright script utilizes Apify's proxy configuration with session management.
Proxy rotation is set to occur every 10 downloads.
Session IDs are dynamically generated as my_session{index}.
Despite session switching messages, the proxy server remains the same.
Switching to proxy session: my_session1 with {'server': 'http://10.0.94.249:8011', 'username': 'groups-auto,session-my_session1', '}
Switching to proxy session: my_session2 with {'server': 'http://10.0.94.249:8011', 'username': 'groups-auto,session-my_session2', '}
Switching to proxy session: my_session3 with {'server': 'http://10.0.94.249:8011', 'username': 'groups-auto,session-my_session3', }
Code Snippet;
Create proxy configuration
proxy_configuration = await Actor.create_proxy_configuration(
groups=['auto']
)
...
Proxy and session management logic
total_proxies = 30
download_count = 0
proxy_session_index = 0
results = []
for index, manual in enumerate(manuals[:max_pdfs], start=1):
# ... (download logic)
if download_count % 6 == 0:
proxy_session_index += 1
if proxy_session_index > total_proxies:
proxy_session_index = 1
Actor.log.info("Used all proxies. Sleeping before restarting.")
await asyncio.sleep(10)
session_id = f'my_session{proxy_session_index}'
proxy_url = await proxy_configuration.new_url(session_id)
proxy_settings = get_playwright_proxy_settings(proxy_url)
Actor.log.info(f"Switching to proxy session: {session_id} with server {proxy_settings['server']}")
2 Replies
@Abdul just advanced to level 1! Thanks for your contributions! 🎉
View post on community site
This post has been pushed to the community knowledgebase. Any replies in this thread will be synced to the community site.
Apify Community