CA
ambitious-aqua
Concept of a 'queue'?
I think i am confused on the request queue.
Here's my scenario: I will be having many requests for the same actor, where I will pass a URL. I don't know the frequency of these requests.. I may have several in a few minutes, or several in an hour.. and they may take 2-10 minutes to run.
Should I be creating a new run each time I want a request? I know I can START a run with many URL's.. But I may only have one to add at a time.
I'm concerned I will start 10x parallel runs, and crash my memory.
What's the best practice? Can I just keep a single run going, and pass all URL's to this run?
1 Reply
rare-sapphire•16mo ago
You can indeed keep a single run going and pass all URLs to this run. This approach is efficient because it allows you to manage all requests within a single run instance, avoiding potential memory issues from starting multiple runs in parallel.
Another way would be to start new run once You have new URL to process / a batch of URLs to process.
To avoid crash due to parallel runs You can Continuously monitor system resource usage and optimize your implementation accordingly. Adjust parameters such as concurrency limits and queue management strategies based on performance metrics and feedback.
Shortly: if You always have lots of upcoming URLs => go with one "always running" run.
If the number of URLs is not that big / not fast coming - start new run once You have new URLs to process.
Overall, the best practice depends on the specific requirements and constraints of your application.
Also, note that The queue can only contain unique URLs. More precisely, it can only contain Request instances with distinct uniqueKey properties. By default, uniqueKey is generated from the URL, but it can also be overridden. To add a single URL multiple times to the queue, corresponding Request objects will need to have different uniqueKey properties.