Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Join

Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Join

crawlee-js

apify-platform

crawlee-python

💻hire-freelancers

🚀actor-promotion

💫feature-request

💻devs-and-apify

🗣general-chat

🎁giveaways

programming-memes

🌐apify-announcements

🕷crawlee-announcements

👥community

fair-rose

3/2/2024

Google Maps Scraper

Hi, I'm using the Google Maps Scraper from Compass, but it doesn't seem to give me all results. Don't know if I'm doing anything wrong. I'm trying to get a list of all locations in google maps with the category "Kindergarten" in Oslo, Norway for example. After a run, I get 367 results even though I know there are about 650-700 kindergartens in Oslo. To test, I find the google maps URL of a kindergarten in Oslo which is not amongst the 367 results, to see what category it has. The category is kin...

harsh-harlequin

3/1/2024

Apify > Integrately > Sender.net

Hello Apify Team, Firstly, thank you for your time and effort. I would like to integrate Apify with an ESP. I searched the web and found this link https://integrately.com/integrations/apify/sender Sender.net is perfectly fine for my needs (if you have another better suggestion I am open), so I thought I would try to integrate Apify > using integrately > with Sender.net. Naturally, I wish to use Apify to scrape certain web results and acquire email leads which I hope to automatically import into Sender.net so that they may be used in a email campaign. This seems possible. ...

optimistic-gold

3/1/2024

Booking public actor : Not scrap everything

Good morning, I would like to scrape the following url: https://www.booking.com/searchresults.fr.html?ss=Corse%2C+France&ssne=&ssne_untouched=&label=gen173nr-1FCAEoggI46AdIM1gEaE2IAQGYAQ24ARfIAQzYAQHoAQH4AQuIAgGoAgO4ArGq964GwAIB0gIkNThm%20NGFmMjctODQ4Yy00MDdlLWI1YTAtNDM3OTRkMGU0Mjdi2AIG4AIB&sid=c67f4e516010b0465ed7b2fbb3fb5ea1&aid=304142&lang=fr&sb=1&src_elem=%20sb&src%20=index&dest_id=1383&dest_type=region&group_adults=2&no_rooms=1&group_children=0 ...

fair-rose

3/1/2024

Paid subscription?

Hi, I'm considering paying for the starter plan. Just wondering if it's significantly faster than the free plan?

eastern-cyan

2/29/2024

Batch PDF Text extraction

Hello, I'm new to apify and tested your Website Content Crawler which worked great. I downloaded several PDFs in that process which are now stored in a database file on apify. I can manually extract the text using the PDF Text Extractor for each pdf with the key store link. ...

vicious-gold

2/27/2024

Connecting actors

I want to connect 2 actors and pull data from the first as an input in the second. I am scraping indeed then I want to run a google search scrape with the company name. How can I do this?

absent-sapphire

2/27/2024

How to make code private

Hi I pushed an actor online but it made the code public like you see here in this image below. How can I hide this? https://apify.com/mtrunkat/delete-untitled-actors/source-code...

xenial-black

2/26/2024

Hello Friends,

I am a new user and loving what apify can do. Have run some actors already and results are ok. Had a question: a) If I want to write a custom scrapper or host an existing code here then does apify limit scrapping to only public websites or can it scrap those with username and password (which we will provide of course in our scrapper)? b) How do I better control the proxy settings as I want to get country specific IP and see results on the website for those....

solid-orange

2/23/2024

Missed result for own actors, when it's started from API

Hello everyone! I'm developing actor, and i stacked with one small problem. I push actor to Apify, and started it manually from UI (origin: DEVELOPMENT), this start was good, and i getting all expecting results (reviews). But, when i started my actor from API (origin: API) it's missed results. Example: DEVELOPMENT start - 11 urls crawled and 100 reviews extracted...

absent-sapphire

2/22/2024

chatbot down

Hello Apify team, I can't open a new chat, please have a look to the screenshot. Thanks...

like-gold

2/22/2024

Payment Failure

Hello, I tried to pay for subscription by received this message: "The supplied card data failed tokenization. Please ensure you have sufficient funds on your card". Card has no problem. Any helps on this issue? Thanks!...

graceful-beige

2/22/2024

latest version of node-ts does not work and latest commit on Github is ignored

I have been having issues for quite a while now with fixing node-ts that suddenly don't understand TypeScript files and hence won't compile them to javascript that node can run. I have managed to fix it locally using tsx instead of node-ts but since tsx is not supported on apify.com the run fails immediately. In addition when I look in the latest build log I can't recognize that the latest commit on my main branch from github is used. I have cleaned up every build and version so that only the latest build is present. The crawlee and apify docs have not given me some direction to investigate this. This has been a major headache for a while now so I hope I can get some guidance 🙏 ...

eastern-cyan

2/21/2024

Run instance of apify client not returning, cant access data scraped from my Apify agent

Hey guys, I'm getting an issue where I the run instance of the client.actor call for the python SDK is not returning ? it just says the call is 'running' so I can't access the items for the agent : def apify_reddit_agent(json_input) : ```python def apify_reddit_agent(json_input) :
info_array = [] #Changed the API key here to the samuel account instead....

exotic-emerald

2/20/2024

Grant other devs/testers run only access to my actor

Let's say, I am an actor developer, how do I grant other devs/testers access to my actor so that they can only run(not view/edit) an actor without having to subscribe to the actor?

rival-black

2/20/2024

webhook custom variable

i want to update the status of each actors in my multiple users app. how do i attach a custom variable i.e. my app's user id in the webhook?

optimistic-gold

2/17/2024

I can't use my private GitLab repo link

I have a private GitLab repo link simillar to the following: [email protected]:username/folder-1/folder-2/my-scraper.git The page showing that "Git URL is not valid". How can we solve that?...

ugly-tan

2/16/2024

Adapting apify for Crawling

Hi I need some assistance adapting some of the actors for my Crawler Details: my company has a Crawler Java application that crawls 70-100 sites for review data and currently we need to outsource this application functionality to a 3rd party. This engine handles http, api and standard html download , I want actors that can handle these 3 usecase scenarios. My application handles orchestration, download and parsing , we can still handle the parsing phase of the collected chunks and extract the target data internally but we need the orchestration and download phases handled externally. Is this something you can help with ? If so, I have more questions and I need to speak to some one from Apify implementation before going any further...

adverse-sapphire

2/16/2024

Using Apify API to get actor inputs, and `exampleRunInput`

I wanted to be able to use the Apify Actor API to get what inputs a particular actor requires, so I can dynamically fill it and create an actor task, and trigger it. However, I am unable to get that from any API mentioned on the dev docs. Am I missing something? Further, I was looking for something similar, and found exampleRunInput which seemed useful, but I am getting what looks like sample data in there (My current actor version is 0.0.4, if that matters), and not data that my input_schema specifies. Is the usage of exampleRunInput the way I am envisioning correct?...

other-emerald

2/14/2024

ali Express Scrapper

Hi, I need help with increasing the default limit of the number of photos retrieved by the paid actor AliExpress Scraper after running the data extraction process. Currently, I'm only receiving a limited amount, typically six photos, and I want to know how to expand this limit.

fair-rose

2/13/2024

Requests queues and preserving write usage

Hello, i'm creating a supermarket data scraper. The supermarket i'm scraping has a sitemap where are the urls for every product are listed. Currently i'm loading those in like this:

const { urls } = await Sitemap.load('https://.../entities/products/detail.xml');

const { urls } = await Sitemap.load('https://.../entities/products/detail.xml');

...

Previous Next

Gaming

Programming

Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Crawlee & Apify

This is the official developer community of Apify and Crawlee.