Cron task routinely looping over a growing list indefinitely
I would like to be able to run a cron task every so often that iterates over a list of items and performs tasks for each one in batches indefinitely. Essentially a "give me the next 100", do some work, repopulate them to the back of the list, and wait x minutes to run again.
For context, this is for a Discord bot, and I want to be able to run routine cleanup tasks efficiently at scale. Currently doing "all" wouldn't be a huge hit since the number of guilds is so low, however when it starts hitting hundreds, thousands, tens of thousands, etc. that's no longer an option. I know this is excessive thinking, but I need to take into consideration a plan for long term operations way down the road.
My initial thought was using Durable Objects and D1, sorting by last update time and updating accordingly, however I'm wondering if queues may be better suited? My only concern is costs for such tasks spiking operational costs well beyond my planned expectations. I'd probably make batch groups to run multiple together to limit transactions but there's still rate limits I have to keep in mind as well as execution time.
For context, this is for a Discord bot, and I want to be able to run routine cleanup tasks efficiently at scale. Currently doing "all" wouldn't be a huge hit since the number of guilds is so low, however when it starts hitting hundreds, thousands, tens of thousands, etc. that's no longer an option. I know this is excessive thinking, but I need to take into consideration a plan for long term operations way down the road.
My initial thought was using Durable Objects and D1, sorting by last update time and updating accordingly, however I'm wondering if queues may be better suited? My only concern is costs for such tasks spiking operational costs well beyond my planned expectations. I'd probably make batch groups to run multiple together to limit transactions but there's still rate limits I have to keep in mind as well as execution time.
