Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Join

Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Join

crawlee-js

apify-platform

crawlee-python

💻hire-freelancers

🚀actor-promotion

💫feature-request

💻devs-and-apify

🗣general-chat

🎁giveaways

programming-memes

🌐apify-announcements

🕷crawlee-announcements

👥community

HonzaS

5/3/2023

Google Sheets Import & Export Actor

Hi, I have tried oauth process with this actor https://apify.com/lukaskrivka/google-sheets. I open liveview then I click on the authorize button as is instructed here https://help.apify.com/en/articles/2424053-google-integration but then I get this App is blocked mesage. Anybody know what to do about this? Is it problem with the app or with some privacy setting on my google account? I know this definitely worked some time ago....

!!!Joefree!!! 👑

5/3/2023

(maybe) Bug

I've created an actor via github, and accidentally set an input enum with empty array: ```json ... "kind": {"title": "Type", "type":"string", "description": "", "enum": []},...

harsh-harlequin

5/2/2023

Change Price after publish

I have published the actor and now want to change the price. When I click on Request changes it opens Intercom bot and after selecting option it does nothing

equal-aqua

5/2/2023

TypeError: __init__() got an unexpected keyword argument 'follow_redirects'

I am using Apify Client to scrape information in a cron job written in airflow dag. When i try to execute the task file manually, it works fine but when i do so from the airflow UI, it throws this error. I tried to upgrade the version of httpx to 0.24 but apify client support 0.23 ```python APPIFY_CLIENT = ApifyClient(config('APIFY_CLIENT_TOKEN')) File "/usr/share/pyshared/venv/airflow/lib/python3.8/site-packages/apify_client/client.py", line 126, in init self.http_client = _HTTPClient(...

constant-blue

5/2/2023

How can I set actor timeout?

I want to set actor timeout to 180seconds, I can set it through web options (see screenshot), but how can I do this via api?...

helpful-purple

5/2/2023

I am trying to use this crawler

https://console.apify.com/actors/QzSjA4f2XE37xGJF9/console; on this website https://www.autotrader.ca/cars/acura/bc/vancouver/ But keep getting an error. I basically paste the link in and hit start but I get an error. What am I missing?...

round-purple

5/1/2023

Bug when using Apify.utils.requestAsBrowser: process seems to crash on timeout from the apify proxy

My client recently contacted me that their old apify actor does not work anymore. Upon changing the code to use the new node docker image and updating apify version a little bit everything seemed to resolve for me locally. But when on apify platform we started testing the actor with apify proxy turned on, from time to time we got the error attached in the screenshot. As I deduced, the issue seems to be coming from Apify.utils.requestAsBrowser -> http2-wrapper (source: https://github.com/szmarczak/http2-wrapper/issues/66). But I have got no idea on how to circumvent the error as we can not (?) simply use the inner libraries 😉 Relevant code snippet: ```js response = await Apify.utils.requestAsBrowser({...

rival-black

4/28/2023

Playwright URL Parameter

Hi, I reckon this is a simple, possibly stupid, question but either I don't know what term to use or it wasn't mentioned here. Is there an easy way to add an URL Parameter to my Playwright Scraper? I used the template for playwright scraper, which works well, but let's say I need to scrape a page that looks like this: ...

ambitious-aqua

4/28/2023

Crawler with playwright doesn't stop

I developing a playwright scraper to do some basic stuffs, after it finish with the urls, it doesn't stop, like my terminal get stuck until I press CTRL + C. Any flag I should enable?...

like-gold

4/28/2023

Upwork scraper is not scraping based on category

Hello, I'm trying to use this webscraper (https://apify.com/trudax/upwork-scraper) and search under a specific category, but it's not scraping it. It keeps scraping the main job search page. I've tried using the GUI options and providing a direct link to the category I want scraped, but no luck. Any idea?

like-gold

4/28/2023

Google maps scraping based on search words and multiple city's.

Hello, I'm new to Apify and am looking into the google maps scraper. I do see the option to dearch for different keywords but location wise i can only do one location at a time. In my case i need to scrap maps for different keywords in different locations. So for example my keywords would be: - Night shop - Gas station...

harsh-harlequin

4/28/2023

Can we use Wise Account for payout?

For actor payout can we use WISE account?

fair-rose

4/28/2023

Actor Privacy Policy?

Hi, I'm a prospective webscrape dev. I'm wondering about the privacy policy regarding the code I submit as an Apify actor, since the Apify privacy policy I saw on the webpage seemed to be directed more towards users than developers.

quickest-silver

4/27/2023

Search unavailable on twitter?

I have been trying to use the Twitter scraper to extract Tweets containing certain terms from certain users, between two dates. I can see warnings about twitter search not working, but the scraper is returning results just without the key word distinction. Is this fixable using the 'login cookies' option? I have limited technical skills but really need this data for an upcoming research project. ...

HonzaS

4/27/2023

crawler stops when there are still pending requests

Hi, I have run cheerio crawler run that has finished but the queue is still showin 7 requests pending, is this normal?

fair-rose

4/27/2023

How to have multiple crawler on the same repo ?

I currently have 5 crawlers on my repo and I would like to create 5 apify actors. Is there a way to make to use the same github repo for all of them but specifiy which file to run ? I found out how to setup continuous integration with apify and build the crawlers in their own separate actors on apify. The issue is that, I can't specify which file to run. Also, is it possible to build all the files on github, then only push a single javascript file to apify ? ...

correct-apricot

4/26/2023

Twitter scraping: retrieving all replies to a Tweet

Hi all! I'm using Apify for the first time, specifically the Twitter Scraper. I want to retrieve tweets by profile name, and get all the replies to the retrieved tweets. I feel like this is likely possible but I can't seem to get the right config to make it happen. Can anyone help please?...

optimistic-gold

4/26/2023

I can't create new task, Build with number "0.0.1" was not found.

rival-black

4/26/2023

Zillow Terms of Use Violation?

This question is not a rhetorical but is instead quite sincere. Please help me understand Apify's interpretation of Zillow's Terms of Use. Zillow's API Terms of Use and FAQ pages (ridiculously) states that their data is not allowed to be stored but only is allowed to be displayed dynamically to the end user. However, Apify's Zillow Real Estate Scraper does just that by its exporting feature into a variety of file formats. Reference: https://www.zillow.com/howto/api/APITerms.htm

flat-fuchsia

4/25/2023

Accessing Apify's Free Trial Period for Actors: Issue with Upgrading to Higher Plan

Hello, I was recently contacted by one of my clients who reported a problem with accessing the free trial period for Apify's Actors. My client is unable to try out the actor without being forced to upgrade to a higher plan, even if they have not used the actor before. I have also tested this with my own personal account and found the same result. Is it normal for users to not be able to access the free trial period without upgrading to a higher plan, even if they have not yet used the actor?...

Previous Next

Gaming

Programming

Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Crawlee & Apify

This is the official developer community of Apify and Crawlee.