Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Join

Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Join

crawlee-js

apify-platform

crawlee-python

💻hire-freelancers

🚀actor-promotion

💫feature-request

💻devs-and-apify

🗣general-chat

🎁giveaways

programming-memes

🌐apify-announcements

🕷crawlee-announcements

👥community

exotic-emerald

4/17/2024

Actors didn't appear on main profile

My actors don't appear on my profile after creating an organization. If I visit the actor with a route like this : https://apify.com/aweworkz/html-web-media-scraper Works fine but if I visit the profile which https://apify.com/aweworkz ...

optimistic-gold

4/17/2024

Can't log in into my apify console with google oauth after paying.

Hi, actually I just became a paying customer, yet after a few days my account seems “locked out” when I try to log in with my Google account via oauth it won't work. The login form only says: Wrong email or password. but how can that be with oauth....

rare-sapphire

4/17/2024

Puppeteer Crawler cannot open the page

Hi, I have a puppeteer scrapper, which worked just fine until this Monday. Nothing is changed, but scrapper stopped working. The HTML markup of the page is not changed, a[data-testid="search-listing-title"] this element is still there. Apify run logs says it is failing to find this HTML element: TimeoutError: waiting for selector a[data-testid="search-listing-title"] failed: timeout 30000ms exceeded...

unwilling-turquoise

4/16/2024

Is there a way to have multi-language readme.md (actor information) files

is it possible to support a readme.md (actor information) in english language, and another one in another language? Maybe by using something like this in the repo? https://github.com/jonatasemidio/multilanguage-readme-pattern...

fascinating-indigo

4/16/2024

Creator plan

do i need a creator plan in order to use my own actors

genetic-orange

4/15/2024

Facebook scraper Real posts and followers number.

Hi everyone, I am a beginner and wanted to make a scraper for my own use. How are facebook scraper getting the real number of posts and followers for an account? Are they using facebook api?

foreign-sapphire

4/15/2024

Getting no proxies on input - expected or a bug?

I'm trying to integrate my Scrapy actor with Playwright, so I attempted to figure out what is the actual format of the proxy input from Apify, so that I could somehow pass it over to Playwright. I printed out what my spider gets and this is what it prints: ``` APIFY_PROXY_SETTINGS: {'apifyProxyGroups': [], 'useApifyProxy': True}...

foreign-sapphire

4/14/2024

Monetization terms

Hey, is it allowed to integrate a captcha solving API into a scraper and monetize it on your platform?

apparent-cyan

4/11/2024

Need Help with Parameters for Content Crawler

Trying to crawl a website using the include urls glob but it is returning nothing using this actor: https://console.apify.com/organization/TrErqhYLFgyc7Gs32/actors/aYG0l9s7dbB7j3gbS/console Can DM a dev specifics if they are willing to help....

fascinating-indigo

4/10/2024

How to set up Apify Proxy correctly with native Puppeteer?

Hi guys, I want to use Apify Proxy with native Puppeteer JS. Is this the correct setup? Thanks in advance! INPUT SCHEMA: "proxyConfiguration": {...

stormy-gold

4/9/2024

Starter Plan Run Times

Hello, I am working on a project that scrapes about 20 different locations using Apify Google Maps Scraper. It takes about 2 - 3 minutes to finish and was wondering if upgrading to the Starter Plan would actually reduce run times since our data size is so small. Thanks.

dependent-tan

4/9/2024

Is it cheaper to scrape in bulk or individually?

In my Python script, where possible should I scrape a set of URLs in the same run? I am wondering if doing each url in separate runs costs more due to the starting and restarting of the Apify client vs doing them in one run.

extended-salmon

4/9/2024

Metadata/description for the columns extracted in the scraped data

But what do the values in the column popularTimesHistogram/Fr/3/hour mean? Moreover is there a metadata file that goes along with my exported data or a support page that tells me what each column in my data means? Thanks...

wise-white

4/5/2024

Error handling/Best Practices Python SDK

Hello, I am using pre-built actors in my application. I use them like this to create the dataset: ```python client = ApifyClientAsync(token=settings.APIFY_API_TOKEN) run = await client.actor(actor.value).start(run_input=run_input)...

genetic-orange

4/5/2024

analytics shows only new trials/paid users

I noticed that in anlytics tab I can see only new paying/trial users, not all for current month. Is this intended behavoiur? If so, how can I check total state?

optimistic-gold

4/5/2024

RequestQueue read/write is timing out constantly

We are receiving the following errors across 1000's of executions ```WARN ApifyClient: API request failed 4 times. Max attempts: 9. Cause:ApifyApiError: Unexpected error: "<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n" clientMethod: KeyValueStoreClient.get...

harsh-harlequin

4/3/2024

Twitter Scrapper

Hi everyone, I need to scrape tweet engagement stats for all tweets containing specific hashtag. I found several Actors, all deliver the result just not sure why in this case the output is capped to 10 tweets? It says I need to get paid plan but not sure how and where. Thanks Actor: apidojo/tweet-scraper...

fascinating-indigo

4/3/2024

hi guys

Hello there, I'm reaching out for assistance with data scraping. Once completed, is there a method to structure the data according to my requirements? Additionally, is there a way to eliminate replicated data and retain only one instance? Your expertise in this matter would be greatly appreciated. Thank you....

xenial-black

4/3/2024

Linking an Actor to a repo from Azure DevOps Git

I am having a lot of trouble trying to link an actor to a private repo from Azure DevOps. I created the public SSH key from the deploy keys link in the actor. However, I am not able to make it happen. To provide context, the actor was developed by a third party who has the repo on his own Github repo. We created an Azure Devops repo and he pushed the code there. I created a copy of the orginal actor and now want to link it to the repo residing in Azure Devops. ...

adverse-sapphire

4/3/2024

Apify HElp

Today is April 3rd, and I haven’t received my March income yet.

Previous Next

Gaming

Programming

Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Crawlee & Apify

This is the official developer community of Apify and Crawlee.