Queued jobs not being processed quickly
As of around 1 hour ago, one of the crucial background jobs on my app has stopped being processed as quickly as is required, resulting in a rapidly growing back log in the queue and user complaints.
The background job processes orders and from what I can see there are not an unusually large number of orders being created. Instead, I can see that the queued jobs just aren't being processed as often as they should be. There are a lot of 'platform' jobs running and I don't know if their priority is stopping the job that I need to run from being run? I can also see that there is a spike today in my data storage but again don't know if this is relevant.
I didn't change anything with the app before this started. Are there any changes on the Gadget end? Appreciate any help as it's causing a critical issue with functionality in my app.
App url: click-and-drop.gadget.app
9 Replies
Hello,
We'll look into this ASAP. In the meantime, please note this message: https://discord.com/channels/836317518595096598/958764380038782976/1439985658218283109
Hello,
So it looks like you might just need to set a priority on the jobs that you want to run. If you set the priority to
high, they will be run first.Hi Antoine, thanks for this. I'll set to high now and make a deployment. Any reason this would have happened suddenly this afternoon?
Trying to find the docs for you right now
I'm not sure why it would happen all of a sudden. Do you just have more things being enqueued?
Do the 'Platform' ones take priority over 'default' or 'high'? Would it have been them that blocked it?
Have you set queue names for those jobs?
I looked at that but I don't think so
Yeah, I've got a queue name
The backlog does seem to be decreasing a bit now, without changing the priority. But the job was running intermittently, sometimes processing a bunch in a row, causing the queue to decrease a bit, but then it would stop running and only run once every few minutes or so, leading to an increase in the backlog.
Mind keeping an eye on it and telling us if anything weird happens again?
Just to update you on this from my end, since the change to
priority: high new jobs are being executed shortly after they are enqueued which is good.
The built up backlog from earlier of jobs with priority: default is decreasing very slowly. Each job usually takes < 1 second to process, but it's taken around 5 hours for 1000 to be completed. Presumably if I just leave it the remaining 1000 jobs in the backlog will all be completed, but definitely curious as to why this happened starting around 1pm UK time today. I've checked and there was no spike in orders, it just seems that the priority: default jobs started to be executed less frequently (and that continues to be the case now). Keen to understand what happened to be sure that it won't happen again over the busy period coming up.