Concurrent Runs
On the pricing page https://apify.com/pricing it has a row for concurrent runs but I cant find any documentation on how to do this.
I am trying to scrape Zillow to create a table/matrix of 9x14=126 cells where each cell is the result (a number) from he API response with parameters on Geography, Time, Status, Size. Each API call currently takes 5 seconds so doing them synchronously takes 126 api calls x 5 sec=630 seconds or 10 MINUTES to complete!
If I could send 32 concurrently on my $49 paid plan it should be way faster 126/32=3.94 x 5 seconds=20 seconds to complete vs 630 seconds/10min
I guess one way is to somehow be able to send 32 separate requests simultaneously and do this 4x as described above.
Another approach would be to somehow send the API request to Apify in a single POST but have the search terms for each 126 cells as an array but not sure if this is possible or how to do it. Also not clear this would be faster
Here is a sample of what I want my final output to look like just trying to speed this up:
Apify
Plans and pricing · Apify
Extract value from the web with Apify. Flexible platform — flexible pricing. Free plan available, no credit card required.

3 Replies
flat-fuchsia•2y ago
Hi! You need to consider doing it as single run with multiple startUrls because concurrent runs require a lot of RAM, not viable for Zillow since scraper is browser based and each run will consume 4Gb of RAM
absent-sapphireOP•2y ago
Thanks. Any suggested docs/demos of how to do that for a newbie?
flat-fuchsia•2y ago
@timj Hi! See https://apify.com/maxcopell/zillow-api-scraper#input - startUrls, zpids and zipcodes are arrays, therefore instead of 126 runs which is impossible anyways you need to reduce runs to smaller array(s), i.e. single run with 123 startUrls or 2 runs with 63 etc