requestQueue write costs
Hi there,
was there a change how much it costs to write to the requestQueue?
Now running cost of the actor is determined predominantly by writing to the request queue and not how much CU it takes.
6 Replies
other-emeraldOP•14mo ago
I am using
crawler.addRequests shouldn't this be like one batch write? Or it just iterates over the requests and inserts one after the another under the hood?
@Saurav Jain can somebody reply also here please?optimistic-gold•13mo ago
I have the same question!
It's crazy how much you need to pay for writting into a q. I can introduce my memory Q and store result there, but this is crazy.
@Saurav Jain
rising-crimson•13mo ago
@Alexey Udovydchenko can you please help here?
probable-pink•13mo ago
@Saurav Jain Its billing question, from dev point of view does not matter
generous-apricot•13mo ago
In general usage, RQ is a costly operation.
Besides, what You see in "Usage" table under "Request queue writes", includes all operation with your RQ. "write" => is not only adding new req. to the queue. Could be operation with some already existing request (mark it as "processed"/ "failed", update retryCount value, etc..)
It's all "write to RQ".
So it's mainly about how efficient the logic in your actor is. Maybe there is a space for some improvement?
Also, Maybe try to use RQ v2
https://crawlee.dev/api/next/core/class/RequestQueueV2
It's still experimental, but worth checking. Could be not just faster, but a bit cheaper ?
optimistic-gold•13mo ago
We will try the new version.
Im trying to come up with an idea how to make it cheaper:
Batching made it cheaper a bit, not using the router for all cases but await function in one route.