Request time extremely high
My request time exploded again and I see it's the readFromShopify background action. I just pushed some stuff today that will hopefully improve it.
Also why is it so high and then goes down? We didn't change anything on the background action in that time period. It's like 3-4 times as high in the first few days as now....
Also I (and others) had issues with high request times after trivial updates in the past already. Anything you can do to prevent that / get informed about unusual high spikes? Cause I didn't check for a week and suddenly have to pay 500 USD which sucks
App: keepcart-server
Thanks
6 Replies
Hi Ricardo, we will investigate
Hi Ricardo, im looking at the updateCodeStatus.ts looks to be the culprit.
I would recommend using the
enqueue background action as its identical to the logic there with the concurrency, retries and waiting periods but better optimized for rate limits.
It looks like the action hit a rate limit, then the reconciliation sync as well hit the rate limit, then the script did its daily sync which attempted all the failed attempts + new and 429'd again. To add to that it looks like the monthly sync occurred mid way as well for all shops to sync.
This should help with avoiding 429's and managing large influxes of api calls
https://docs.gadget.dev/guides/actions/background#customizing-background-actionsThanks, we had the background actions before but they consumed so much request time that I had to rewrite it. I need to check it more but had major issues with request time in general after updates. Are there any things planned to prevent stuff like that in the future?
Background actions should definitely not cause you more request time. I think that we should get on a call to discuss this.
Furthermore, the issues with your code are not something that we will work to prevent. When it comes to billing caps, we do have plans to add caps that can be set by the user.
Just jumping in to make sure I understand:
The issue here is that our background actions werent used to enqueue individial requests with retries, but somehow aggregated the writes to shopify effectively meaning the entire time for Shopify to stop the rate limit got billed?
And that if each write to shopify had been enqueued seperately as recommended, this would not have happened?
I wanted to rephrase what I said above because I came into this conversation without background of what was being talked about.
I was making a single comment that background actions should never cause you more request time and they should instead allow you to run things asynchronously. My thought is that we should get on a call to discuss restructuring your self written queuing in to background actions and help explain more why you're billing went so high.
Side question. Have you made any changes to how you call
updateCodeStatus in the last 12 days? Has it always been a scheduled action?
Did you have a global action named readFromShopify at any point?I actually see this problem as being entirely of Gadget's fault and creation cause on writes, rate limit management is difficult given that its also being consumed on reads via Gadget's built-in mechanisms.
I think we need to expose a new concurrency setting/value where we take over and max the rate limit on writes and back off as needed. (e.g.
concurrency: 'shopify')
this way you can simply fire into a background action with that concurrency set and not worry about maxing rate limits and idling.
Shopify is not consistent at sending 429s when overwhelmed and so asking users like yourself to try and manage that is real tricky. I would like to take this off your hands. Do you have some time to chat about this idea a bit to see if it would have solved your problem? There are some nuanced I'd like feedback on.