crawlee-js
apify-platform
crawlee-python
💻hire-freelancers
🚀actor-promotion
💫feature-request
💻devs-and-apify
🗣general-chat
🎁giveaways
programming-memes
🌐apify-announcements
🕷crawlee-announcements
👥community
Get tweets by date
Hi. Is it possible to get most recent tweets of a twitter user? Preferably since a datetime (e.g. from yesterday)?
HELP! I need to crawl website with sidebar live comments
At first I thought that the site uses something like an api, but upon closer inspection it uses websocket (socket.io). How should I approach this problem?
Crawlee or APIfy tooling?
I'd like to get started with APIfy but down the line I may end up using my own infra. How hard is it to switch from APIfy's tooling (cli etc) to Crawlee?
I understand APIfy is built on top of crawlee, so I imagine it's possible but how much friction/lock-in is there...
google drive integration question
Hi, I need to upload csv file to the google drive. I see there is some new integration tab on apify now, so I have some questions. Sadly there is a lot of settings but no documentation.
Is it possible to upload via this integration file from the key value store?
I have managed to convert csv file to json then push it to the dataset and then push it via the integration to the drive but there are some issues.
1. I do not know how to preserve the filename.
2. converting csv to json change the data a little....
Webhooks - website content crawler
Hey, I'm using website content crawler actor to crawl website. I want to show crawl status update on the client's side like what pages it's found, I've looked into integration/webhooks for this but I can only get ACTOR.RUN.CREATED and ACTOR.RUN.SUCCEEDED', is there any other way I can get the status updates? thanks
ERR_CERT_AUTHORITY_INVALID Issue with Apify Proxy
Hello, I'm using an Apify Proxy in combination with Playwright and ran into an ERR_CERT_AUTHORITY_INVALID error when trying to visit certain sites.
Playwright code (in node):
```javascript
const proxyRouter = ProxyRouter({...
Error when building due to better-sqllite3
My program runs locally, but when I run apify push I get the following error. Any help would be greatly appreciated!
apify/actor-node-puppeteer-chrome:18 builder problem
Hi. I little noobish question. How to add WORKDIR in dockerfile to make it work? I want to use volumes in docker-compose, but can't achive that. Here is my dockerfile:
```docker
Specify the base Docker image. You can read more about
the available images at https://crawlee.dev/docs/guides/docker-images...
Google Maps Email Extractor Actor get frozen
Hi everyone
I'm currently using the actor 📩📍 Google Maps Email Extractor
And the detail scraper that is launch stay blocked at 19 results... the logs froze also
I've already tried to reboot and restart the actor from the begenning but the actor froze again (on the 35th result this time, not the same)
Any ideas ? Thx...
Easy Twitter Search Scraper duplicates output
Hi, I am new to Apify and tried Twitter Search Scrapper. When I analyze the output, I found multiple duplicates for one keyword in the output. Is there something wrong?
Is there a way to add a header the ad-hoc webhook?
One might suspect something like this should be possible:
`await Actor.addWebhook({
eventTypes: ['ACTOR.RUN.SUCCEEDED'],
requestUrl: process.env.RUN_SUCCEEDED_WEBHOOK_URL,...
Get the dataset from the apify-js API client based on the actorRunId?
Is it possible? I can only find actorClient.lastRun() This seems an extremely obvious use case. I did find it here https://docs.apify.com/api/v2#/reference/actor-runs/run-collection/get-dataset but that seems to be for URL's only.
I can't deploy code as written in apify docs
I want to learn how to deploy local code to apify platform, so I followed tutorial from here https://docs.apify.com/academy/deploying-your-code. Here my published actor
https://apify.com/encouraged_keyboard/firstpush...
Why only 200 results from Google Search scrapers...?
I tried 2 different actors for scraping Google search results (Google Search Results Scraper and Fast Google Search Scraper). Both of them stopped at 200 results, declaring "success", even though I specified a much higher limit. What am I missing?
Is there a way to filter/sort questions by number of followers in the Quora Scrapper?
Is there a way to filter/sort questions by number of followers in the Quora Scrapper?
How to set the memory using apify-client?
How can I set the memory of an actor when using the apify-client js client? I didn't see any documentation on this. Is it possible?
How can I get the full-size image from the Facebook Event Scraper.
All my scrapes are returning a thumbnail size image link when I need the full size link instead.
Apify Facebook Event Scraper returns this link (tiny sized):
"imageUrl": "https://scontent-lhr8-1.xx.fbcdn.net/v/t39.30808-6/377862336_697145519119374_1696272395377429810_n.jpg?stp=c129.0.921.614a_dst-jpg_s168x128&_nc_cat=111&ccb=1-7&_nc_sid=934829&_nc_ohc=7921ykIKPkwAX-X55Cp&_nc_ht=scontent-lhr8-1.xx&oh=00_AfCjszdcobeKN-vGQKMxXgs7nJmRRUYiBLPb24gfD_6FgQ&oe=6506FA2A",
...
How to parse posts in group with commenters and reactors?
I need to analyze one group, all posts from a single member, and identify the individuals who commented on and liked their posts. Group Parser give me just top comments and only number of likes.
Webhook security
Hello, I am interested in setting a webhook to receive events when a particular actor run has succeeded. However, I want to be able to secure my endpoint to ensure that only Apify can invoke it.
I found an earlier post from June suggesting that this isn't possible but the Apify documentation suggests otherwise:
https://discord.com/channels/801163717915574323/1115873908046966864
...
Booking Reviews Scrapper
The booking reviews scrapper is not working even with the default example .
When looking at logs it seems like the system is trying to reach the url but it fails "WARN CheerioCrawler: Reclaiming failed request back to the list or queue. Received HTTP 400"
And I assume it's due to "%F" generated in the url "https://www.booking.com/reviewlist.html?cc1=cz&pagename=cz%2Fold-town-square-apartments&type=total&offset=0&rows=10&sort=f_relevance"
Like I said the input url used is the default example one " https://www.booking.com/hotel/us/chicago-t.html?aid=304142&label=gen173nr-1FCAEoggI46AdIM1gEaDqIAQGYATG4AQfIAQzYAQHoAQH4AQKIAgGoAgO4AuLFmqIGwAIB0gIkN2YzZmI0YzktMTY1ZS00OThkLTgzY2ItOTMxODA5OTI5NzNj2AIF4AIB&all_sr_blocks=5924324_246077187_2_0_0;checkin=2023-09-01;checkout=2023-09-15;dest_id=20033173;dest_type=city;dist=0;group_adults=2;group_children=0;hapos=6;highlighted_blocks=5924324_246077187_2_0_0;hpos=6;matching_block_id=5924324_246077187_2_0_0;no_rooms=1;req_adults=2;req_children=0;room1=A%2CA;sb_price_type=total;sr_order=popularity;sr_pri_blocks=5924324_246077187_2_0_0__341393;srepoch=1682350871;srpvid=88466e4a189c0182;type=total;ucfs=1&#hotelTmpl"...