Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Join

Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Join

crawlee-js

apify-platform

crawlee-python

💻hire-freelancers

🚀actor-promotion

💫feature-request

💻devs-and-apify

🗣general-chat

🎁giveaways

programming-memes

🌐apify-announcements

🕷crawlee-announcements

👥community

jolly-crimson

4/16/2023

Instagram Reels Scraper doesn't work

Hey there. I'm trying to scrape instagram reels from a single Reel URL. It doesn't work. Is there a way to scrape a single Reels or a list of Reels? Thanks for your help!

correct-apricot

4/15/2023

Need help optimizing TikTok scraper

Hello, I am rather new to scraping and dont have much of a technical background. I am a social science researcher using ClockWorks TikTok Scraper (https://apify.com/clockworks/tiktok-scraper) to scrape the top 1000 posts from various hashtags. However, I am running into an issue where I am hitting the Apify usage limit of $49 very quickly, where each run of 1000 posts is costing nearly $30 in usage. I am using default settings within the actor. Is there a way I can reduce this cost? Thanks,...

genetic-orange

4/10/2023

Facebook Group Scraper

Hi, I ran an actor on a test account and apparently the scraping was succesful but I did not get any results. Is it because I'm not a paid version? When I pay will i get the actual list of group members with email address?

!!!Joefree!!! 👑

4/7/2023

Dataset Info

Hi ! I want to show/log dataset statistic right after adding values using code below ``` ## data => 1000-ish items # 100 records each to default dataset...

!!!Joefree!!! 👑

4/5/2023

Pyhon SDK on Aborting

I am trying to catch ABORTING event with the following code, without success. Could someone teach me how to do it properly ? Thanks 😆
``` import asyncio from apify import Actor from apify.consts import ActorEventTypes...

correct-apricot

4/5/2023

ChromeDriver Not Updated?

I'm getting an error message that the wrong ChromeDriver version is installed. Is there a solution for this? The run ID is 1kRoKIMf7ld1UVQgf and the message from the log is :

2023-04-05T16:16:49.544Z selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:51597
2023-04-05T16:16:49.545Z from session not created: This version of ChromeDriver only supports Chrome version 112
2023-04-05T16:16:49.546Z Current browser version is 111.0.5563.64

2023-04-05T16:16:49.544Z selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:51597
2023-04-05T16:16:49.545Z from session not created: This version of ChromeDriver only supports Chrome version 112
2023-04-05T16:16:49.546Z Current browser version is 111.0.5563.64

...

other-emerald

4/1/2023

Haven't received payout from Feb

Hi, I just got the payout invoice for March but I haven't got the payment for Feb. Can you help? My org id is vqHHgLqhhsXJTtKkF. Feel free to email me.

like-gold

3/30/2023

Facebook Events Scraper

Hi guys, I am new to Apify and try to get back all future events from some locations My format I tried is https://www.facebook.com/locationname/events But there is no result, just a lot of errors in the log Any idea or suggestion? Thanks a lot...

correct-apricot

3/30/2023

Checking boxes with Selenium

Does anyone know what I'm doing wrong? I've tried finding the element and clicking it but get an error that the element isn't interactable. I have also tried executing a script that would check the boxes and the actor will succeed in running but after looking at the screenshot no checkboxes were checked. Any help in the right direction is appreciated. Here's the code that gets the error of element not interactable: ```checkboxes = driver.find_elements(By.XPATH, "//input[@type='checkbox']") for checkbox in checkboxes:...

variable-lime

3/29/2023

Getting original Facebook pictures

Hi, we are a nonprofit organization. Our Facebook page was taken over, and unfortunately Facebook isn't helpful at all. Our page is over ten years old, and we want to archive all of the posts, comments, and more importantly, the pictures. I started a trial of Apify and everything seems promising, however, all of the images that are returned are cropped into squares and are low resolution. Has anyone else seen this? Is this a bug, or is it as expected? This unfortunately won't work for us if it i...

environmental-rose

3/29/2023

google map scrapper - polygon issues

Hi trying to use the google map scrapper (actor) with the option to define an area based on a polygon. loks like the json I entered is working however I get results outside of the polygon when I run it. any ideas why?

extended-salmon

3/28/2023

Finishing requestHandler() request early

Im using Puppeteer requestHandler() and is there a way for me to end the request early instead of waiting for the whole "script" to finish so I could move on to the next URL in the queue?

provincial-silver

3/28/2023

Public Actor Limits

How many public actors can I create on one profile? I have many actors created overtime, i am thinking of making them public to earn side income. Many of them just gather public data only. ...

ambitious-aqua

3/27/2023

Difference between the scraped amount in browser vs on Python

I am using the twitter scraper in Python and finding that on the browser console I am getting all 300 tweets that I request. I have a counter in my Python script that increments with each item in the client.dataset(run['defaultDatasetId']).iterate_items(). This ends up being around 90, so it seems I am only getting 1/3rd of the tweets I scrape. Anyone know why or recommend what to do?

like-gold

3/25/2023

Twitter scraping by both keyword and profile

It is too computationally intense/slow for me to make the api call for one of the filters and do post processing with the second filter. I am wondering if you can make an api call to scrape filtering by both keyword and profile. Is this possible or can I only do one or the other? Thanks!

extended-salmon

3/25/2023

Scraping instagram profiles

I just tested out the apify/instagram-scraper. Specifically, the profile data retrieval part of it. But unfortunately it doesn't have some data I need. I can get the data I need by going to: https://www.instagram.com/{username}/?__a=1&__d=dis, but this supposedly requires me to be logged in, so I turned to apify to see if I could scrape the same data I need. Attached is an image that has the data I'm looking for, highlighted- Is it possible to scrape this data through apify, ideally through a pre-built actor in the store already?...

like-gold

3/24/2023

Facebook scraper (posts AND comments)

Hi all! I have a question related to Apify I don't know if this is the right place to ask it but here it goes[8:45 PM]I am trying to scrape Facebook data (both the post AND the comments on that post)[8:45 PM]Is there an actor that could extract both these elements for me?

unwilling-turquoise

3/23/2023

tweet scrape

https://console.apify.com/actors/u6ppkMWAx2E2MpEuF/runs/uYQWmfrwJiXAfucpb#output need help completing this tweet scraping. It's stops at 848 tweets while I need it to scrape 5k+ tweets. plz help.

dependent-tan

3/23/2023

Google blocks some Apify proxies searches

My client raised an issue for empty results and I find out Google block some proxies: https://api.apify.com/v2/datasets/njakYUtJmDX6EvPyp/items?clean=true&format=json What should I do in this case?...

like-gold

3/22/2023

Picture Identification CAPTCHA

I am currently working on a page that blocks scrapers with a CAPTCHA. The captcha is an image identification test, i.e. find all images of a car, person, sidewalk etc.. Is it possible to overcome such a CAPTCHA with Apify?...

Previous Next

Gaming

Programming

Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Crawlee & Apify

This is the official developer community of Apify and Crawlee.