Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Join

Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Join

crawlee-js

apify-platform

crawlee-python

💻hire-freelancers

🚀actor-promotion

💫feature-request

💻devs-and-apify

🗣general-chat

🎁giveaways

programming-memes

🌐apify-announcements

🕷crawlee-announcements

👥community

manual-pink

8/8/2023

Concurrent Runs

On the pricing page https://apify.com/pricing it has a row for concurrent runs but I cant find any documentation on how to do this. I am trying to scrape Zillow to create a table/matrix of 9x14=126 cells where each cell is the result (a number) from he API response with parameters on Geography, Time, Status, Size. Each API call currently takes 5 seconds so doing them synchronously takes 126 api calls x 5 sec=630 seconds or 10 MINUTES to complete! If I could send 32 concurrently on my $49 paid plan it should be way faster 126/32=3.94 x 5 seconds=20 seconds to complete vs 630 seconds/10min...

extended-salmon

8/8/2023

dedupe queue

Hey All, When running a Task from the console, is there any way to dedupe the queue? I imagine that would be kind of a default, and an easy toggle in the config screen. I am havnig the same URL crawled twice per run.

extended-salmon

8/8/2023

exclude start URL

Hey All,
There was nothing here when I searched this phrase. I have a bunch of sites where the Glob format of the links to be scraped is the Start URL plus something like /// - Is there some easy way that I just don't see to exclude the StartURLs from being scraped? They produce junk data, as you can imagine. Thanks!...

xenophobic-harlequin

8/8/2023

Tiktok hashtag api

I am using tiktok api I am geeting data but videos urls not working

dependent-tan

8/8/2023

Apify Help

I need help with the google maps scraping , specifically if I can have it not search or include listings that already have a website linked.

extended-salmon

8/7/2023

Where are the just announced Python Starter Templates?

https://discord.com/channels/801163717915574323/1114213126712344596/1134093628957012088 I need a functional Python template, meaning has the fields for Start URLs, Glob, etc. so that I don't need to mess with the JSON, which seems to elude me....

rare-sapphire

8/6/2023

Facebook Post Scraper help

I have purchased Facebook Post Scraper and ran a few times for different pages and the results were not sufficient. Thay always stopped and said "success" after ~160 posts when I asked for 3000 posts. No response on the issue tab when I created on the Actor. I'm in urgent to collect the data so anyone can help please? I'm assuming the problem coming from the proxy (Actor use US proxy when I need to scrape data from Vietnamese pages but don't know how to change it)! Log of one of the runs: https://api.apify.com/v2/logs/oRD1Sb1HjqeGsQLoo...

unwilling-turquoise

8/6/2023

Google Maps scraper - filtering by reviewsCount?

I would like to scrape certain types of places from Google Maps, but I would like to limit my scraping to the most popular ones. My logic is that the places that have the highest number of reviews are the most popular. Example: Category: Restaurant City: Seattle reviewsCount > 100...

plain-purple

8/6/2023

sending a json response and saving it

in my code i want to send a json response to a file and save it how do i install the dependencies needed (mongoose, next, etc.) and without any errors when sending the json and use the fetchAPI or whatever method that can work w/ apify, and save the data do a mongodb database heres the code:...

plain-purple

8/4/2023

saving data in apify actor

ive tried saving the data to a rawdata.json file from the data i scrape from my actors, however i dont get a json output even thought the scraping works how would i save the data to the apify console that i can then use mongodb to take that data and put it in my database -...

fair-rose

8/3/2023

Web Scraper

I am using web scraper (most liked scraper) It has a lot of settings. Is it possible to create a task and still be able to change the url of the website so that I can still make API calls in order to scrape various websites without needing to specify other settings every time (and to be able to change them in Apify Console).

other-emerald

8/1/2023

Facebook Group Scrapper - search query

Can't find any documentation about what Facebook Group Scrapper's search query should look like. I assumed that it should work like a usual Facebook search query (with OR operator) but Scrapper didn't find any post. However, my search query works fine in Facebook group. Can somebody clarify which operators I can use in Facebook Group Scrapper's search query? Thank you!...

helpful-purple

7/31/2023

Python template issues

Hey i made apify actor project from template (nothing is changed) and when i run apify run i get this error ``` C:\Users\style\justjoin-it>apify run Run: C:\Users\style\justjoin-it.venv\Scripts\python.exe -m src...

like-gold

7/30/2023

Actor with this name was not found

I am trying to use an actor with API, I've downloaded the python libray, and copied the code directly from apify to run it, I've updated my api key correctly, but I am getting this error. run = client.actor("o3cej87hBc3Wf7DN6/wHMoznVs94gOcxcZl").call(run_input=run_input) From this line, I see the actor is a combination from my userid/actor id...

absent-sapphire

7/28/2023

I've set webhook from Apify to Make.com but the scraped fields are not passing into Make.com

I have set a webhook from Apify to Make.com with the attached parameters, I have no options in Make.com to setup which Apify fields are incoming. Any ideas? I'm just trying to pull my runs from Apify into Make, to Google Sheets.

adverse-sapphire

7/28/2023

Monorepo - Playwright Chrome Dockerfile

Do you have Playwright Chrome dockerfile for monorepo ?

wise-white

7/28/2023

Noob Puppeteer Question

I'm trying to follow Apify's Puppeteer tutorial (https://docs.apify.com/academy/puppeteer-playwright/page/interacting-with-a-page) and getting nowhere. Their example will launch Chromium and execute through the 4th line of the code below. Then nothing. It looks like Google may have changed the page so that the button text that the example is looking for no longer exists. However, even if I replace the selector text to something that is indeed on the page that Chromium loads, noting happens. As in:

await page.click('button:has-text("I agree")');

await page.click('button:has-text("I agree")');

...

vicious-gold

7/28/2023

APIify java client?

Is there a APIify java client or a swagger / openapi spec to generate a client? Great service BTW!!!

like-gold

7/27/2023

3rd party actor support

Newbie question: What's the best practice of getting support on a 3rd party actor? Have added a record to the issue section of the actor, to no avail, hence my question. thank you!

correct-apricot

7/27/2023

Instagram posts by location

Hey ! I'm currently using the apify/instagram-scrapper actor and it's working great. I can gather posts as i need by hashtags or user, but it doesn't seems to work on location. I always get only one location with a few top posts and a few recent posts. How can i do to make it return results like it does for a hashtag or a user 🧐 ? ...

Previous Next

Gaming

Programming

Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Crawlee & Apify

This is the official developer community of Apify and Crawlee.