Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Join

Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Join

crawlee-js

apify-platform

crawlee-python

💻hire-freelancers

🚀actor-promotion

💫feature-request

💻devs-and-apify

🗣general-chat

🎁giveaways

programming-memes

🌐apify-announcements

🕷crawlee-announcements

👥community

sensitive-blue

12/21/2023

woocommerce

How do I connect apify to woocommerce products ? #apify-platform

rival-black

12/20/2023

Where To Increase Actor Run Timeout

Hello. A run of mine timed out, and the message in the logs says: "ACTOR: The Actor run has reached the timeout of 604800 seconds, aborting it. You can increase the timeout in Settings > Run options." But the only Settings I see is the one in the sidebar that leads to my account, and that doesn't seem to have Run options. Where should I be changing this value?...

deep-jade

12/20/2023

Webhook config per schedule

Hi all, I've found this doc about setting ad-hoc webhooks per actor run but there isn't any info about the schedule case. My use case is I have multiple environments (prod, staging, etc...) and I need an ability to configure webhook url in schedules so different schedule can send results to different webhook endpoint. I know I can add multiple webhooks to an actor but it means for every run, it'll send results to all endpoints which isn't ideal. https://docs.apify.com/platform/integrations/webhooks/ad-hoc-webhooks...

exotic-emerald

12/20/2023

Status / error code upon usage limit, using API?

What is the status/error code (and message if it exists), that occurs when the monthly (money) usage limit- is hit? (when using the api)...

other-emerald

12/19/2023

Apify Scraper Usage

Hello , In my website I want to show the Facebook, TikTok and linkedin posts of the given URL. Is it possible to get the results quickly by using the paid plans? Currently I am using trail plans.

passive-yellow

12/19/2023

YouTube Channel Email scrapper

Hi! I am interested in YouTube Scrapper tool. What I am looking for is to input a keyword or give filters like subscribers, views etc ...

metropolitan-bronze

12/18/2023

De-duplicate dataset results

I have an actor that returns a simple list of IDs. It's possible that during a run, concurrent processes can overlap and produce duplicate results. Is there any accepted way of avoiding this? At the most basic level I'd hoped that I could do something simple like using the returned ID as the key in the dataset (i.e. a duplicate result would write the same entry so a duplicate would not be created), but this doesn't seem to work, presumably because each result is actually a separate JSON file in the dataset. I've also thought about opening the dataset and getting the full list of IDs, then only pushing IDs not present - this could work but adds overhead and also seems to introduce the possibility of race conditions....

noble-gold

12/17/2023

Test Residential Proxy with CURL

Hi! I'm trying out Apify's residential proxy under the free plan. I'm currently building a LinkedIn Scraper for my customers, so I wanted to use Apify's proxy to reduce the risk of being banned. I wanted to have a quick try to see if I can directly use Apify's residential proxy in my code (running on an EC2 instance) to access LinkedIn. Here's the command line that I ran:...

generous-apricot

12/14/2023

A tool that will let me find businesses in Google Reviews that have specific criteria/categories?

Specifically I'm looking for companies that have an average minimum of 45 words per 4 and 5 star review. These businesses would be in a specific list of business categories, like barber, tattoo shop, etc. And in specific locations. The scraper doesn't need to do all this, as I could use the data to filter in a spreadsheet. Possible? My end goal is to end up with a list of all businesses in a specific category in a town that meet my criteria. I'll need to do this for other categories and other towns. ...

modern-teal

12/14/2023

Broken links

FYI, these links on https://docs.apify.com/platform/storage/dataset are broken (see image).

wise-white

12/14/2023

Process tracking

Hello everyone! I'm working on a project where I run tasks on Apify, and I'm wondering if there's a way to obtain estimates on how long a run is expected to continue before completion. I'd like to display this information in my frontend application to keep users informed about task progress. Does anyone have experience with this or know if it's possible? Any insights or suggestions would be greatly appreciated! Thanks in advance!

rival-black

12/14/2023

I Need to send direct messages in twitter without using their API

Tell me a way

afraid-scarlet

12/13/2023

Help to Narrow Apify Actor Search Results

Hi there, new to Apify and wondering about how to narrow results of Actor searches in Google Map Scraper. For example, searching Coffee Shops in an area and wanting to avoid getting results on shops named "Barstucks" or searching for Bakeries and avoid getting results on shops named "Mr. Fields", etc. I've tried Boolean type terms such as "-Barstucks" and "Not Barstucks" on the same search line and on added search lines but I still get those results. Any and all help appreciated!

!!!Joefree!!! 👑

12/13/2023

Run ID Dataset ID

Does a run ID or Dataset ID have a pattern to match (eg: regexp) so I know its a Run ID ? Thanks....

exotic-emerald

12/11/2023

Google Maps Scraper Orchestrator - linking location inputs to results

Sorry if the title isn't incredible. I had no idea how to describe what I'm about to say. I'm using the Google Maps Scraper Orchestrator actor (https://console.apify.com/actors/Uk8ZlE4NVYccUvpHw) Which obviously means I'm also using the Google Maps Scraper (https://console.apify.com/actors/nwua9Gu5YrADL7ZDj)...

foreign-sapphire

12/11/2023

Apify Tiktok comment Scraper

Hello, I paid for the Tiktok comment scaper to scrape the comments from videos. Now the scraper has scraped only 4700 comments out of 14,000 comments and now there's written „succeeded“ But does not go on for the other 10000 comments. Can you help me with this?

modern-teal

12/9/2023

Auto builds don't work at all, getting HTTP 500 errors from Apify both in UI and webhooks

When I click on the radio button Automatic builds, I get an error. Different browser doesn't help. The browser console says: ``` [31mERROR[39m Failed to handle request 'POST - /github-app/setup-webhook/...'[90m {"request":{"method":"post","url":"https://console-backend.apify.com/github-app/setup-webhook/...","headers":{"x-idempotency-key":"...","Authorization":"Bearer ..."},"params":{},"data":{"version":"0.0","enabled":true}},"response":{"status":"error","statusCode":500,"isClientSafe":true,"errorCode":"internal-server-error","errorMessage":"Cannot destructure property 'token' of '(intermediate value)' as it is null.","path":"/github-app/setup-webhook/..."}}[39m [object Object]...

optimistic-gold

12/7/2023

is the source code available for apify actors? most of the links are 404s

In the actors page, you can choose Information > Source Code > View on Github, but most of these links seem to 404. For example apify/google-search-scraper goes to https://github.com/apify-projects/store-google-search/tree/master/actors/google-search-main or apify/website-content-crawler goes to https://github.com/apify/store-website-content-crawler...

quickest-silver

12/7/2023

Instagram profile scraping via API

I have the following scenario: I want to enter Instagram username handle on my website and populate the whole profile with results from apify actor. How do I do it via API? I can not figure out how to send input data JSON from my input to the actor's input, run it and get results. I'd really appreciate some help....

extended-salmon

12/6/2023

Is it safe to push .env file containing aws API keys to Apify?

Previous Next

Gaming

Programming

Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Crawlee & Apify

This is the official developer community of Apify and Crawlee.