Crawlee & Apify•4mo ago

Apify Proxy not working with https urls

I want to test the apify proxy and how it works to integrate it with my py code. Running a very simple check I found it's not working with https urls. here's a snippet:

import asyncio, httpx
from apify import Actor
import dotenv

async def main():
    async with Actor:
        proxy_configuration = await Actor.create_proxy_configuration(
            password=dotenv.get_key('.env', 'APIFY_PROXY_PASSWORD'),
        )
        proxy_url = await proxy_configuration.new_url()
        proxies = {
            'http': proxy_url,
            'https': proxy_url,
        }
        async with httpx.AsyncClient(proxy=proxy_url) as client:
            for _ in range(3):
                response = await client.get('https://httpbin.org/ip')
                if response.status_code == 200:
                    print(response.json())
                elif response:
                    print(response.text)

if __name__ == '__main__':
    asyncio.run(main())

import asyncio, httpx
from apify import Actor
import dotenv

async def main():
    async with Actor:
        proxy_configuration = await Actor.create_proxy_configuration(
            password=dotenv.get_key('.env', 'APIFY_PROXY_PASSWORD'),
        )
        proxy_url = await proxy_configuration.new_url()
        proxies = {
            'http': proxy_url,
            'https': proxy_url,
        }
        async with httpx.AsyncClient(proxy=proxy_url) as client:
            for _ in range(3):
                response = await client.get('https://httpbin.org/ip')
                if response.status_code == 200:
                    print(response.json())
                elif response:
                    print(response.text)

if __name__ == '__main__':
    asyncio.run(main())

giveing me a proxy error:

          raise mapped_exc(message) from exc
      httpx.ReadTimeout
[apify] INFO  Exiting Actor ({"exit_code": 91})

          raise mapped_exc(message) from exc
      httpx.ReadTimeout
[apify] INFO  Exiting Actor ({"exit_code": 91})

If i just only change the protocol to http://httpbin.org/ip it works. Apify proxy should support https as stated on the site. Thanks in advance.

4 Replies

Hall•4mo ago

Someone will reply to you shortly. In the meantime, this might help: -# This post was marked as solved by 🟢mido🟢. View answer.

deep-jade•4mo ago

@🟢mido🟢 hey, I'm not able to reproduce the issue. Does this happen repeatedly? It seems to be just a timeout error that could be solved by creating and using another proxy url or increasing the timeout (https://www.python-httpx.org/exceptions/, https://www.python-httpx.org/advanced/timeouts/)

ambitious-aquaOP•4mo ago

@lemurio Hi, Actually yes it happens with any https url. I tried increasing timeout=60 but still getting same error. And http urls even work with timeout=5

deep-jade•4mo ago

I'll write you a DM edit: the issue was the user's connection

Gaming

Programming

Apify Proxy not working with https urls

Did you find this page helpful?