Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Join

Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Join

crawlee-js

apify-platform

crawlee-python

💻hire-freelancers

🚀actor-promotion

💫feature-request

💻devs-and-apify

🗣general-chat

🎁giveaways

programming-memes

🌐apify-announcements

🕷crawlee-announcements

👥community

rising-crimson

2/12/2024

Multiple dataset output schemas to show scrape data in Apify console UI

"storages": { "companyInformationDataset": "./dataset_company_information_schema.json", "reviewsDataset": "./dataset_reviews_schema.json" } ...

conscious-sapphire

2/12/2024

Consultant services - Cloudflare Blocked

Does anyone know if Apify provide development consultant services? We have a custom Indeed scraper, it's been running great for years, but now they have some advanced cloudflare protection in place. Our dev team have rewritten the Actor to use all the latest packages with best practices and recommendations as per the docs. It runs successfully locally (using proxies) but when deployed to the platform we cannot bypass the protection....

absent-sapphire

2/11/2024

Typescript export/import causing build errors

I built my first actor today using the Typescript/Crawlee/Cheerio template. I’m a front end developer that uses Typescript every day, so this felt comfortable at first. I quickly ran into build problems though. I split my code into multiple files that exported functions and some types, however the build failed as it did not like my imports into main.ts. No matter how I tried to import my helper functions I would get linting and build errors. I ended up copy pasting my code into main.ts, which di...

foreign-sapphire

2/9/2024

Sheets Execution time issue

Does anyone have a fix to this? Is it better to add a delay or maybe even a repeater?

absent-sapphire

2/9/2024

actor api/xml/google sheets related (looking for advice)

hi, im a noobie to apify and have limited coding knowledge looking to have some custom actors made in general if i can figure out how to use this platform... for now im trying to make a bunch of POST/GET requests (not sure) to USPS Verify api which gives xml output. ...

deep-jade

2/8/2024

youtube scraper issue

I need "likes and Comments" while scraping with Fast Youtube Channel Scraper but its not scraping those fields. Any way I can do this?? #apify-platform

causal-orange

2/8/2024

Integrate google maps email extractor actor with Instagram Profile Scraper.

Hello 🙂 I'm trying to automate data scraping with integrations between different actors. For now I have managed to combine "Google Maps Scraper" with "📩📍 Google Maps Email Extractor"....

conventional-tan

2/8/2024

What happen when I make request?

Hello, let's consider scenario: - I start run - I make request r1 - I make request r2 ...

fair-rose

2/7/2024

Only want to return one result for these scrapers

I'm currently enriching data I got from Storeleads on about 10k eCommerce companies I'm doing outreach to. The data comes with store name, Facebook URL, & Instagram URL, so I'm using the "Facebook Ads Scraper" and "Instagram Profile Scraper" to enrich these companies. I want to find the last post & latest ad (startDate/endDate fields & ad description for FB; latestPost & likes/cmments for IG). The problem is I get thousands of columns with a bunch of other metadata I don't need. I also can't omit the "results" column because I'll lose all results instead of the first one....

deep-jade

2/6/2024

need link insertion on apify blog

I need link insertion in some of the blogs of apify can you please help me #apify

exotic-emerald

2/5/2024

Retrieving origin from webhook

Hi, I am trying to understand if the actor run was triggered via api or through the scheduler. After the run finishes, we have a webhook to send us a notification. I retrieve the origin from the webhook, and it keeps returning "WEBHOOK" vs what the original run was (api or scheduler). How can I retrieve the real origin from the webhook?...

deep-jade

2/5/2024

Playwright expect in apify actor?

Is it possible to use playwright expect in an apify actor? Currently if I try to do something like: import { expect } from 'playwright'; I am told playwright does not provide an export called 'import'....

eastern-cyan

2/5/2024

Reddit Scraper Help

Hi everyone - ive tried to use the reddit scraper to scrape posts, from the past 1 year, from a subreddit channel. The reddit scraper only returns c900 posts back ( only a few days worth!) ...could someone pls help to let me know what i need to do to for the scraper to go back further in time ? i recently signed up and am on the free trial

adverse-sapphire

2/5/2024

Compliance with Instagram TOS

I am planning on getting data from the instagram scraper for academic research. I was not sure if using Apify is against instagram TOS even though it is legal. I don't want to pursue something that a journal might not consider due to failure to comply with data collection TOS so I wanted to ask about this before I proceed. Thanks for any help.

stormy-gold

2/5/2024

How to edit Actor Information?!

Hello, guys! I recently created my first Actor using Apify IDE. Is there a way to hide my source code in actors information tab (see image) ? Thanks!...

!!!Joefree!!! 👑

2/4/2024

Boolean Input Scheme

My actor Input Scheme have hundred of optional BOOLEAN parameters (literally). When user not selecting any of the options, the platform set the default values to False (all hundreds parameters). This could create confusion ? How to prevent this from happening ? How to make platform just set True only for users selected options ? Thanks.

multiple-amethyst

2/4/2024

Hi, I need to increase the limit whom can I contact please on Apify, chat does not work for rme.I n

Hi, I need to increase the limit whom can I contact please on Apify, chat does not work for me.

sharp-indigo

2/3/2024

for update of Instagram reels data issue

Hey dear manager I use apify actors (Instagram reels scraper and Instagram profile scraper) and adds 40 accounts and fetch reels data and profile data and use schedule that data daily fetch automatically .I connect apify with airtable database.when schedule run in apify then store in a airtable ,now i export the apify run data to excel .and check and compare with previous Instagram reels data .second time scraper not give same reel URL data update or new data on same reels ...

metropolitan-bronze

2/3/2024

Error running the runs simultaneously

I keep getting this because it's trying to run 100+ runs at a time. How do I stagger the runs so that they only start when the previous one finishes? Any ideas would be appreciated!...

vicious-gold

2/3/2024

About organizations and private repositories

Hi. I am a member of an organization. I can switch from my personal account to the organization account and install actors there, when they are ready for production. The source code comes from my private Git repository. I have two questions, regarding security: 1. Can the organization owner/admin also switch into my account and see my actors, runs, storage and personal API token just like I can do with that account?...

Previous Next

Gaming

Programming

Crawlee & Apify

This is the official developer community of Apify and Crawlee.

Crawlee & Apify

This is the official developer community of Apify and Crawlee.