RequestQueue.open issue in dockerized app
When i'm trying to run my app with crawlee inside docker container (from apify/actor-node) "RequestQueue.open()" code just infinite executed without any errors and hinders further program execution. During usual execution at my PC it works fine.
13 Replies
extended-salmon•2y ago
Could you provide some repro? Stack trace? Something? I'd say we never experienced anything like that. Also - which crawlee version are you using?
flat-fuchsiaOP•2y ago
You could see simplified code here https://github.com/RDzhalmukhamedov/docker-crawlee-test. I use actual version of crawlee: 3.3.2.
extended-salmon•2y ago
I had a very briefly look, so might have misread it, but are you creating crawler instance first and then assigning the queue to it?
flat-fuchsiaOP•2y ago
Yes, does it matter? When I do not declare the queue separately, but immediately pass the request with link to crawler.run(), the program does not work in the same way. Just freezes on the crawler.run() method
extended-salmon•2y ago
The thing is - if you don't explicitly pass the queue in crawler constructor - it silently creates the queue. It simply is not how it's designed, so I am not really sure what's happening in this case. This does not explain why it hangs on crawler.run() though
flat-fuchsiaOP•2y ago
Ok, got it. I could update project with immediate passing case
@mrkrouly just advanced to level 1! Thanks for your contributions! 🎉
extended-salmon•2y ago
yeah, could you please give it a try, and if it's still the same (but I guess it will still since you mentioned you already tried it this way) - I will pass it to the team together with repro
flat-fuchsiaOP•2y ago
Tried one more time. Still the same behavior. Updated project with immediate passing and little readme
rival-black•2y ago
We need to know more about your platform since it is obviously connected to that, not the bug in the code
flat-fuchsiaOP•2y ago
I tried to run it locally (Win 10 Enterprise Version 22H2, Build 19045.2965; Docker version 23.0.5, build bc4487a; Docker Compose version v2.17.3) and also tried to run at Fly.io (https://fly.io/). Sadly, I had same behavior.
complex-teal•2y ago
Hey @mrkrouly I've just tested out the repo locally and haven't been able to reproduce the issue!
Can you try building your image without having a storage folder?
That said it should work with it..
Found the issue
Now to figure out how to fix it...
Fix figured out! Will PR :D
thanks for reporting
Reminder
as a temporary fix, make sure you have no
storage folder when invoking docker build . (or docker buildx <...>. that will also fix the issueflat-fuchsiaOP•2y ago
Thanks for help!