CA
rare-sapphire
Page Navigation is Slow when Running in Apify
I noticed my crawler ran much slower when running in Apify than in my laptop.
In my local (Mac M1 Pro 16G, headless=false): page navigation takes ~ 3-8 seconds;
in Apify (2048MB, headless=true): page navigation takes ~ 10~20 seconds.
is the difference purely caused by hardware? or is there anything I can improve? thanks!
6 Replies
You can try to increase memory, it will increase cpu power as well. That way you will find out if it is hardware dependent.
it could also be because of slow proxy connection, you can increase concurrency to speed up whole scraping process
rare-sapphireOP•3y ago
thanks
rival-black•3y ago
You kinda answered yourself by comparing your laptop specs with the actor. You need to at least match the memory.
rare-sapphireOP•3y ago
I was thinking the page navigation was mostly affected by bandwidth (which Apify should be better than my home network), but it turned all all resources including memory/cpu mattered.
btw, is there any difference Apify manages free v.s. paid plans? say for example, if I run my actor with 2G memory, will it be more ensured to have that 2G computing power in paid than in free plan?
@xli just advanced to level 1! Thanks for your contributions! 🎉
rival-black•3y ago
No, paid plan just allows you to even use more GB, otherwise all specs are the same
But you get the memory/CPU you ask for, it is using Docker to assign these resources