Spike in usage again
Hi I'm currently and surprisingly having a spike in usage again when this is all monitored like more than a week of not seeing a spike. Up to now i'm still trying to cancell all the queues that related to the background actions that are spiking the usage.
The only change I made is having a once a day schedule for the run of that action though the code in that action, have pagination that may trigger the same action just for the purpose of retrying the action when it's about to time out.
Can you somehow check out the background actions right now? I'm still cancelling it I don't why it's keep coming back. It should have been stopped after I cancelled all of those actions.
THANK YOU.
12 Replies
Hi Benz,
You can view your active background actions in
Queues -> Running
Is there logic within your action that would re-queue the action on failure?Yes i'm aware
On failure, no
But when it's trying to reach the timeout limit, it will
The logic is when the timeout is near, it will run again where it left of. Like what page and record it is in before the timeout
More like continuing the pagination
Could you share the app url and environment so that we can investigate?
https://pwr-storage-solutions-2.gadget.app/
It's in production
Here is some error in the queues that might help:
job-WoGz_BD-veHjTgXa6I1NZ
=========================================
==========================================
job-qBBCroobZpRdnxmGxy2y_
==========================================
From what I can see so far it looks like the custom queue system in
acumaticaFilesV2 lines 134 - 156 being used around Gadgets enqueue is likely excessively enqueuing. As well in the logs it looks like are several 429's from shopify as a result.
Given theres 2 api calls right after the elapsed time and the timeout is 60 seconds, once we get to line 117 we iterate through a list of stockItems, then enqueue for each SKU, and line 156 has a sleep for 1000ms just before the custom re-enqueue. So your custom queue is re-enqueueing quite often.
Is there a reason for this approach over action timeouts?Hmm basically we have 2 api endpoints.
1. Endpoint to get all the active products - INVENTORY_WITH_EXTERNAL
2. Endpoint for getting the files for those products - STOCK_ITEMS_WITH_ATTACHMENTS
So we are looping the endpoint number 1 to get the files from endpoint number 2. We need to process this per product as we need to upload all the files in the metafield in one shopify api call and it makes easier to understand the flow.
What I don't understand is after over a week of monitoring the spike, this occurs. But this time in a different action though related from the previous spike. THe first spike is the action being looped in the acumaticaFilesV2 which was updateProductFilesMetafield.
Also checking the action and its parameters that causing this, it seems that it was stuck in a specific page which is page 29.
After cancelling all the running and waiting queues, the action with same parameter of page keeps on coming back in running
So I had to to deploy in code a
return; before the action start just to stop the queue coming back
Is there a way we can void/reverse this spike usage billing? It went up to 3k request time. Just noticed it when the billing went up to $3000.
Monitored this for weeks and no spike on that specific action then this occurs suddenly when the only I add in the code was scheduling which is just set at once a day.The main issue looks to be the extra earlyTimeout loop. I would recommend removing it.
As well it looks like doing the call to shopify & acumatica concurrently would make sense here.The Client could also be lumped in with them.
Also line 52, findFirst would reduce some request time since the findMany is only getting one record.
You should be able to rely on the action timeout rather than the earlyTimeout logic.
Would fetching the data, loop through it, then enqueue the actions not work?
Ill reach out internally and I will get back to you on this
How would I rely on the action timeout?
By fetching data, do you mean having the initial fetch of products stored in gadget db? Cause technically im doing fetching data directly from acumatica then looping it to enqueue the action of processing the product’s files.
Thank you. Please, the request time is unfairly high when I did nothing else on that action but to set only the scheduler. The action somewhat stuck in a certain state. I had it cancelling in the running queues but keep coming back. I had to stop it by deploying some code before the action starts. It really did not make sense.
So, I did a sync again. I commented the scheduler code and manually trigger the action. It went smoothly. No spiking at all.
I believe the current code is fine. I will try again adding the scheduler and see what will happen
Maybe you guys can be considerate on this one. There's no single call on the shopify APIs as well.
Hey Benz - I'm Franco and work with our partners. I actually have a call set with your team for next week so will speak with them to find a resolution then. Thanks!
Thanks @[Gadget] francovarriano