genetic-orange

Using apify proxy

My actor was created using python so I would like to know if how I added the proxy code to my actor is correct. If it is not, then how should I be adding it. Thanks in advance.

import os
import re

from apify_client import ApifyClient
from bs4 import BeautifulSoup
import requests
import urllib.request as request
import ssl

client = ApifyClient(os.environ['APIFY_TOKEN'], api_url=os.environ['APIFY_API_BASE_URL'])

password = "<MY-PROXY-PASSWORD>"
proxy_url = f"http://auto:{password}@proxy.apify.com:8000"
proxy_handler = request.ProxyHandler({
    "http": proxy_url,
    "https": proxy_url,
})

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
httpHandler = request.HTTPSHandler(context=ctx)

opener = request.build_opener(httpHandler,proxy_handler)
print(opener.open("http://proxy.apify.com/?format=json").read())

default_kv_store_client = client.key_value_store(os.environ['APIFY_DEFAULT_KEY_VALUE_STORE_ID'])
actor_input = default_kv_store_client.get_record(os.environ['APIFY_INPUT_KEY'])['value']
input = actor_input['input']

import os
import re

from apify_client import ApifyClient
from bs4 import BeautifulSoup
import requests
import urllib.request as request
import ssl

client = ApifyClient(os.environ['APIFY_TOKEN'], api_url=os.environ['APIFY_API_BASE_URL'])

password = "<MY-PROXY-PASSWORD>"
proxy_url = f"http://auto:{password}@proxy.apify.com:8000"
proxy_handler = request.ProxyHandler({
    "http": proxy_url,
    "https": proxy_url,
})

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
httpHandler = request.HTTPSHandler(context=ctx)

opener = request.build_opener(httpHandler,proxy_handler)
print(opener.open("http://proxy.apify.com/?format=json").read())

default_kv_store_client = client.key_value_store(os.environ['APIFY_DEFAULT_KEY_VALUE_STORE_ID'])
actor_input = default_kv_store_client.get_record(os.environ['APIFY_INPUT_KEY'])['value']
input = actor_input['input']

16 Replies

HonzaS•3y ago

firstly you really should not share your proxy password publicly so please reset it on https://console.apify.com/proxy I am not python expert, so I can just say that this f"http://auto:{password}@proxy.apify.com:8000" seens correct

genetic-orangeOP•3y ago

sorry about that, forgot to take it out. i have removed it from the post and reset it. also, thank you for your input. it feels correct to me as well but i just want to make sure.

rare-sapphire•3y ago

This looks quite correct

Alexey Udovydchenko•3y ago

Sharing my notes below: general approach is to find a way to set proxy at global level or as a wrapper. For Python its possible to define proxy at OS-level as follows: import os os.environ['HTTP_PROXY'] = os.environ['http_proxy'] = 'http://http-connect-proxy:3128/' os.environ['HTTPS_PROXY'] = os.environ['https_proxy'] = 'http://http-connect-proxy:3128/' os.environ['NO_PROXY'] = os.environ['no_proxy'] = '127.0.0.1,localhost,.local' And it will override even Apify client obviously, so before pushing data to Apify cloud proxy needs to be reset. If we need more than that for Python there is https://github.com/darkk/redsocks or https://github.com/Anorov/PySocks as well.

genetic-orangeOP•3y ago

thank you all for the input!

national-gold•3y ago

Hi, I am writing an actor in python, the problem is how can I make a user to must use apify proxy rotation via input, I am unable to find that in docs. I will highly appreciate any help.

rare-sapphire•3y ago

Are you just trying to make it a required input field, or trying to actually implement it?

national-gold•3y ago

Yes I want to make it required input field such that the user uses his own proxy rotation using apify platform.

rare-sapphire•3y ago

Here's a sample INPUT_SCHEMA.json:

{
    "title": "Input for your actor",
    "type": "object",
    "schemaVersion": 1,
    "properties": {
        "proxy": {
            "title": "Proxy configuration",
            "type": "object",
            "description": "Select proxies to be used by your crawler",
            "prefill": {
                "useApifyProxy": true
            },
            "editor": "proxy"
        }
    },
    "required": ["proxy"]
}

{
    "title": "Input for your actor",
    "type": "object",
    "schemaVersion": 1,
    "properties": {
        "proxy": {
            "title": "Proxy configuration",
            "type": "object",
            "description": "Select proxies to be used by your crawler",
            "prefill": {
                "useApifyProxy": true
            },
            "editor": "proxy"
        }
    },
    "required": ["proxy"]
}

national-gold•3y ago

Oh, Thanks alot, how can I make it to use residential proxies only?

MEE6•3y ago

@Jawad Ahmad Khan just advanced to level 1! Thanks for your contributions! 🎉

rare-sapphire•3y ago

They choose which proxies they'd like to use Some users don't have access to residentials

rare-sapphire•3y ago

I'd also recommend referring to this code snippet: https://developers.apify.com/academy/apify-platform/deploying-your-code/inputs-outputs#accepting-input-without-the-apify-sdk

Apify

Inputs & outputs · Apify Developers

Learn to accept input into your actor, do something with it, then return output. Actors can be written in any language, so this concept is language agnostic.

rare-sapphire•3y ago

You can copy the get_input() function

national-gold•3y ago

Thank you very much, I really appreciate that 🙂 However this is what I am getting when using proxy. <Response [403]> The "Proxy external access" feature is not enabled for your account. Please upgrade your plan or contact [email protected] However the actor I am writing needs to use proxy if a user wants to use it, how can I make sure that before any user wants to use my actor has to use proxy, for example if a user comes for testing out the actor and without using the proxy this is what he will get, which is a bad experience. I can not use my own proxy in the actor, as first the pricing issue second I will be exposing my proxy user and password. Can anyone please recommend anything regarding that. Thanks My free plan shows I have access to 20 datacenter proxies, still when I try to use them during actor development phase I get: <Response [403]> The "Proxy external access" feature is not enabled for your account. Please upgrade your plan or contact [email protected]

rare-sapphire•3y ago

@Jawad Ahmad Khan Did you get this resolved?

Gaming

Programming

Using apify proxy

Did you find this page helpful?