Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Join

Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Join

crawlee-js

apify-platform

crawlee-python

💻hire-freelancers

🚀actor-promotion

💫feature-request

💻devs-and-apify

🗣general-chat

🎁giveaways

programming-memes

🌐apify-announcements

🕷crawlee-announcements

👥community

dependent-tan

3/22/2023

Picture Identification CAPTCHA

I am currently working on a page that blocks scrapers with a CAPTCHA. The captcha is an image identification test, i.e. find all images of a car, person, sidewalk etc.. Is it possible to overcome such a CAPTCHA with Apify?...

extended-yellow

3/20/2023

Persistent Fingerprints

Hello folks, is it possible to write the fingerprint to a file? (with Puppeteer) I need a persistent fingerprint and not one who's changing upon every restart... Is there a way to do it?...

wise-white

3/18/2023

Pagination works locally in Crawlee but the same actor on Apify the pagination does not work correct

I have implemented pagination that can start from eg. page 2 and end at including page 5 to scrape all the data from each page. It works correctly on my local machine and I have pushed the newest working code (newest commit id) to GitHub and then to Apify via Webhook, however, when I run the actor on Apify.com it starts at the first page instead of page 2 and does not finish at including page 5. Any suggestions on what might be wrong?

dependent-tan

3/18/2023

Facebook comments scrape with sponsored posts

Hi, can anyone help me with the Facebook comments scraper? For some reason it fails to extract the comments of sponsored posts. There are many other services, for example exportcomments.com that do this without problems

rising-crimson

3/16/2023

What is the best way to scrape as many specific locations as possible?

I am building an app where I need to download the information of as many locations as possible. I know you can use google maps but I am running into a problem where the program scrapes 50 locations when I know there are much more. Is it better to go county by county? Or is there a better way to scrape more locations with better parameters? An example would be downloading the address of every Italian restaurant in NJ, USA. #apify-platform #👋-welcome #🗨crawlee-chat

unwilling-turquoise

3/14/2023

My Actor Run 7 Results Every Time Instead of One

Hi All, I'm new to Apify. I'm using playwright scraper - by the UI, trying to scrap some data from a dashboard. after running the task I see 7 results instead of one and also I see that the runtime take about 7 minutes!! its a very short task, so I fear there's something wrong in my configuration causing it to loop again and again....

dependent-tan

3/13/2023

Save screenshots with standard Apify scrapers

Can a standard Apify scraper (ex: apify/google-search-scraper) to save screenshots of pages? How to add this into the input? Thanks!...

national-gold

3/13/2023

Where does Apify save screenshots?

I found this code in a Puppeteer example and was wondering where Apify saves the screenshots.

console.log('Saving screenshot...');
    const screenshotBuffer = await page.screenshot({ fullPage: true });
    await Apify.setValue('screenshot.png', screenshotBuffer, { contentType: 'image/png' });

console.log('Saving screenshot...');
    const screenshotBuffer = await page.screenshot({ fullPage: true });
    await Apify.setValue('screenshot.png', screenshotBuffer, { contentType: 'image/png' });

...

fair-rose

3/13/2023

Actor readme not working on public page

Hello, i just updated my actor but i see that my public readme is not shown. When i access to the actor using the console i can see the full readme file. url : https://apify.com/bebity/linkedin-jobs-scraper...

wise-white

3/12/2023

input schema does not update on apify, but works locally

How can I force the INPUT.schema file to update the input schema on Apify.com? These input fileds should exist on the apify.com store page.

adverse-sapphire

3/11/2023

How to pass custom header for webhook integration

I am trying to add a webhook integration on my Actor. But, the api is expecting an Authorization header. How do I pass a custom header? Thanks in advance....

conscious-sapphire

3/10/2023

Google is blocking the Google Sheets Actor

stormy-gold

3/10/2023

Changes in apify/instagram-post-scraper?

Hi, I using apify/instagram-post-scraper Has anyone use it recently. I ran the actor, and I'm getting information with undefined data as shown in "captura1.png". If I try to display all the data, instead of getting fields related to posts, I'm getting profile data like in "captura2.png". It seems that post-scraper, comment-scraper, hashtag-scraper are the same profile-scraper...

dependent-tan

3/10/2023

Error: net::ERR_TUNNEL_CONNECTION_FAILED"

Hello, Getting "2023-03-10T08:06:34.890Z Error: net::ERR_TUNNEL_CONNECTION_FAILED at" when running a custom actor...

equal-aqua

3/9/2023

Rebooting/Upgrading old scraping agent for e-commerce catalog data

Hi folks, I am a long-term customer of Apify, we developed some custom agents for ecommerce data catalog scraping in 2019-2020-2021. I have a new series of scrape jobs and potentially a long-term production pipeline I need for ecommerce catalog data and social media data, and need to assess whether the former actors can be used or most likely upgraded.

stormy-gold

3/9/2023

Hello!

Im trying to scrape product data using Web Scraper actor( https://www.opticsplanet.com/s/026509005698), but i cant open product page. If i`m using $('div.specials-contents').find('span.grid__text').click() log writes target closed. How can i add product page to scrape data from it?

deep-jade

3/8/2023

Scraping specific data from Amazon

Hello! I'm trying to gather some specific info from a few book bestseller lists (Example: https://www.amazon.com/Best-Sellers-Kindle-Store-Action-Adventure-Fantasy/zgbs/digital-text/16232448011/ref=zg_bs_nav_digital-text_4_158576011) Besides the title, price, etc., is there a way to also have Apify pull things like the author name, publication date, and other metadata that is easily available on the individual book pages while using the bestseller tool? I'm dreading having to gather all of that information by hand on a daily basis while we track some of these categories. If anyone has a suggestion of how to do this, please let me know....

optimistic-gold

3/8/2023

google play scraper

Trying to get some review data from google play using the "Google Play Scraper" actor but when I run it I'm only getting one result back, extract of the main page. I've checked the box "include reviews" but maybe I need to include something more specific on the url?

plain-purple

3/7/2023

Help limit the number of Columns Extracted on "Instagram Profile Scraper Task"

Great day everyone. I'm not a programmer or developer and just using Apify to scrape Instagram Profiles. Whenever, I use Instagram Profile Scraper, it returns with 2000 columns of data which I don't really need. I only need it to scrape and only return 3-4 Columns (username, externalUrl, fullName). Can anyone please teach me how to set it up? It would be such a great help and I would greatly appreciate any assistance on this. Tried to chat with their support but been waiting now for hours to get...

rival-black

3/6/2023

classcentral.com

Hello, is there anyone who can help me to scrape 1-depth level of this website? https://classcentral.com Thanks in advance...

Previous Next

Gaming

Programming

Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Crawlee & Apify

This is the official developer community of Apify and Crawlee.