Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Join

Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Join

crawlee-js

apify-platform

crawlee-python

💻hire-freelancers

🚀actor-promotion

💫feature-request

💻devs-and-apify

🗣general-chat

🎁giveaways

programming-memes

🌐apify-announcements

🕷crawlee-announcements

👥community

frail-apricot

11/20/2023

working with requestQueue/ catching apify Errors when using Apify API Client for Javascript

Hey everyone. I'm trying to run multiple actor calls simultaneously, and that works fine, unless I go over the allotted memory (32GB). I would like to use a requestQueue, which I assume would run calls in order whenever there is available memory? But I have a hard time understanding how to add actors to the queue with the API. e.g. client.requestQueue("...").addRequest() accepts some unclear parameters that don't seem to contain an actor or the data to pass to the actor. This very much might be because I'm a javascript noob that has a bit of a hard time understanding the documentation XD If I completely misunderstood the idea of a requestQueue, I would like to create my own. For that, I would like to be able to try/catch a normal client.actor(...).call() but specify the error, except I can't seem to find a way to specify the specific error you would get when requesting the use of too much memory (e.g. some error object to import from apify and use). ...

useful-bronze

11/17/2023

Need help ! Question about the apify store !

Hey guys, I have a quick question for all the IT and Web devs around here. Do you know which WP plugin is used in the Apify store ? (circled in the 1st screenshot) ...

sunny-green

11/17/2023

Puppeteer multiple file downloading allow

Hello, scrapers!!! I am just using puppeteer for scraping and also for pdf downloading. Sometimes if the file downloading delays, it shows me 'Multiple File Downloading' notification. Of course I could mannually allow the multiple file downloading, but I want to set this setting when the puppeteer test browser opens. Does anybody have solution to solve this problem?...

other-emerald

11/16/2023

Can website content crawler crawl PDFs?

I'm trying to use the actor "website content crawler" and would like to scrape documents on a website. For example: https://www.ema.europa.eu/en/medicines/human/EPAR/vargatef This URL has many PDFs How can I get my scraper to scrape the documents?...

complex-teal

11/16/2023

Usage isn't adding up

I'm currently on the free plan testing out my first Actor. The usage incurred by the actor suggests that I've exceeded $5, but the counter on the bottom left shows a different sum. Which one should I refer to?

frozen-sapphire

11/15/2023

apify platform bug. Urgently needs to be fixed

Hello. I got an issue that I asked to fix a few weeks ago in bug report channel that seem to have been deleted. I tried to run a scrapy script from local using apify run and it worked fine. I pushed it and when I run it from the platform I get this error. It's kinda an "urgent" issue because I was planning to deploy my Actor and try to monetize it. Is it possible to fix this issue ? Thanks

deep-jade

11/15/2023

I am getting Killed

2023-11-15T12:55:38.594Z ACTOR: Pulling Docker image of build pC2UCjbFN1XUrUq53 from repository. 2023-11-15T12:56:01.109Z ACTOR: Creating Docker container. 2023-11-15T12:56:01.685Z ACTOR: Starting Docker container. 2023-11-15T12:56:04.422Z INFO System info {"apifyVersion":"3.1.12","apifyClientVersion":"2.8.2","crawleeVersion":"3.5.8","osType":"Linux","nodeVersion":"v16.20.2"} 2023-11-15T12:56:04.549Z Killed...

foreign-sapphire

11/15/2023

Remote debugging/changing

Is there a way I can show the remote run happening and allow my customers to interact with the browser. So they can nudge or do the right click if the bot is struck. I want it to happen in following manner. A user gets a URl and they open it and the remote desktop view opens which is then showing the browser current state and user can interact with it if required.

deep-jade

11/15/2023

Duplicate results from multiple runs

Hey, When I run an actor, I can resurrect it, and I assume there will be no duplicate results. But when I want to change some settings, it's a new run, it probably re-runs urls I already ran in the previous run....

foreign-sapphire

11/14/2023

Requests/Results limit

Help me, please. I don't understand why if I set up 5000 requests I get a maximum of 906, I'm using an actor Facebook Groups Scraper...

rare-sapphire

11/14/2023

Apify error - message

I have been getting this error message for days. Can anyone help me?

!!!Joefree!!! 👑

11/14/2023

Console UI problem

I am experimenting with this scenario: I build an Actor, and create 3 fork/build. each have different INPUT_SCHEMA, README.md, etc. When I change version via Version dropdown, I can see INFORMATION page changed. But, the INPUT page does not change, without have to refresh the page, even so, it seem the input conficlting with each other (diffrent fork)....

genetic-orange

11/13/2023

Twitter scraping always give 0 results

I'm not able to make the twitter scraping work, always give 0 as results any idea why? I tried everything Thanks...

ugly-tan

11/13/2023

Tiktok Comments Scraper by Clockworks Inconsistent Results

Hi guys, i tried to scrape comments from all videos using Tiktok Comment Scraper actor by Clockworks. I scraped it for three days straight (10,11,12 November), but the results are inconsistent. There are some comments that exists in the 12 November dataset but not in the 11 November, and 10 November data is incomplete. I also used Tiktok Scraper for posts, and the total number of comment is consistent, 1347 for those three days. Does anyone know why this happen? Thanks!

provincial-silver

11/12/2023

Twitter List Scraper removed?

Hey all, Why was twitter list scraper removed? Was being used actively by me without a problem. Can it be restored? If not whats an alternative? If you are removing without alternative why arent you making a warning for that? I was building a product relying on this Actor. Need your help with this. Please get back asap!...

equal-aqua

11/11/2023

You do not have permission to run this public Actor

I've hit my Free 5$ a month and then i Upgraded to the special basic plan with 500 Credits and ~1$ a month. Now i cant run any actors anymore and on every actor it says "You do not have permission to run this public Actor"...

like-gold

11/10/2023

Need Help Using TikTok Scraper

Hello apify community, first time poster here. I am looking to use the tiktok scraper to go through a list of tiktok accounts on a sheet and add video links if they used a specific audio. I am running music promotion campaigns and have to track the video links of a creator when they use a specific audio. The script will need to run twice daily checking the list of tiktok pages if they used a specific audio and paste the link into a google sheet. ...

typical-coral

11/10/2023

pagination

Hi all . is there any instagram post scraper that would allow me to scrape for example the first 30 posts . Get the result . And start scraping the next 30 posts . And so on until I run out of posts or until I interrupt the cycle. Thanks for reply )

stormy-gold

11/10/2023

100 Page Limit on Content Crawler

Running a content crawler from Stack-Ai and repeatedly scraping only 100 pages. I am just testing out the platform so I am using a trial. Is that the problem? Just tracking down what the problem is across the two apps.

wise-white

11/9/2023

Is there an endpoint to get account available memory?

I do not want to run scrapers/tasks without enough memory. Is there any endpoint to get that information? I'd like to use it in Make.com

Previous Next

Gaming

Programming

Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Crawlee & Apify

This is the official developer community of Apify and Crawlee.