9 Replies
@JAYINGYOUNG just advanced to level 1! Thanks for your contributions! 🎉
quickest-silver•2y ago
I guess, You can. There is just no tutorial for it i docs.
fair-rose•4mo ago
can anyone help me? i created an azure function app locally and used crawlee, It works. But if i deploy it to the azure it doesn't work. I'm getting timeout issue. Normally it doesn't take too much time locally. As i understand, crawlee doesn't start.
quickest-silver•4mo ago
Similar situation, i'm trying to run playwright/crawlee in azure container app.
extended-salmon•4mo ago
I am having the same problem.
@mjh any success ?
So, I managed to make it work on Azure (App Service + Container) using playwright only with the flag args: ['--disable-dev-shm-usage']. However Crawlee keeps being stuck in a loop even with this option:
launchContext: {
launchOptions: {
args: ['--disable-dev-shm-usage'], // ⬅ Ensures it works in Docker
},
},
maybe Crawlee is overriding some options by default ?
ERROR PlaywrightCrawler: Request failed and reached maximum retries. requestHandler timed out after 110 seconds (7P99bQWrhxCt2Fu).
Had this error all day long while trying to run my script with Crawlee.
flat-fuchsia•3mo ago
check your cpu and ram usage in your container. crawlee currently doesnt play nicely with cgroups which are what containers use to enforce resource limits.
flat-fuchsia•3mo ago
im actually working on a pr for this problem as we speak lol
https://github.com/apify/crawlee/pull/2834
GitHub
feat: improved cross platform metric collection by NathanSavageKaim...
This PR improves CPU and RAM metric collection across multiple environments. The CPU metrics are now fully cGroup aware report properly in containerised environments with cpu quota limits. The memo...
extended-salmon•3mo ago
i ended up running directly npx playwright install --width-deps on my app service (not on docker)
i guess its not the best option but it works
flat-fuchsia•3mo ago
this pr has now been merged, try crawlee 3.12.3-beta.28 and enable
systemInfoV2. Crawlee should now accurately read your container stats