Which Cloudflare product(s) does this pertain to? Workers/Other What version of Wrangler are you using? 2.9.0 What operating system are you using? Mac Describe the Bug Same zone worker <-> wo...
I have 400 users details like email, name, contact etc inside an array. I want to insert/update all these 400 users profiles information inside 3rd party service called klaviyo for email marketing. In adding 1 user information into this 3rd party API, it takes 2 fetch requests (max 2 for update). So for 400 users its taking 800 fetch requests at a time for inserting. I am getting error inside logs that "subrequest limit exceeds" means so many subrequest a time... What is solution of this, can anyone please help? NOTE: I am using premium version not free I have also attached a screenshot of my code with result array of 400 users and then adding them one by one into klaviyo
Either limit the number of customers you send per scheduled run (e.g. in your DB, have a column that is "sent" or whatever) and then retrieve X number per scheduler run and only pull ones where they aren't sent yet to the API. Or use https://developers.cloudflare.com/queues/, and push each customer as a message into a queue, and have a separate worker process the queue
If you change to the new billing model (Standard) you can do 1,000 fetch requests. You could also send all of them to queues instead so you can support retrying the requests over time if needed.
I tried a way called batch processing.... I first add 20 users then added delay function (Promise) wait of 60 seconds or setInterval/setTimeout and then add another 20 users... but it didn't worked, dont know why? It always adding that first 20 users only after every 60 seconds...not going to next 20 users. Added 2 screenshots of delay promise function and setTimeout, but both not worked. always adding/updating same chunk/batch. Whats reason?
Is there any way for batch processing, its fine. I want to implement that after every minute I add 50 users to 3rd party API likewise, I will be able to add all 400 users in 8 minutes. it will be fine... this schedules every day by cron trigger..is there any way I can do this. because by setTimeout or adding promise and fulfill it on 1 minute. both this approaches not working....
It’s not a timing thing, it’s a Workers runtime limit. On your current usage plan, you’re limited to 50 subrequests per request (when your scheduled task is run, it is considered one request). You can change your usage plan to standard, or unbound, and that will gice you 1,000 subrequests (e.g. HTTP calls) from your worker.
Only thing I can guess at is maybe the subrequest limit is different for cron triggers, or perhaps it takes a little bit for your account to be fully migrated to standard. I would consider queues anyway though, as even if the limit is 1,000, you’re not far from that limit already.
@ScaredOfMyRawr can you please explain some more about the queue approach, I am not familiar with cloudflare queue, was just reading the docs and got to know that, we have to make 1 producer worker for adding messages into queue and another consumer worker for fetching the message.... So I have to iterate over result array (400 customers) and add one by one into message queue, all this in producer queue...and in consumer queue, that logic and adding customer into klaviyo. right? Can you please tell more about this approach
You got it spot on You already have your producer worker (your scheduled cron trigger) where you publish from, and then you can I think also just put a “queue” method on your worker and it will be your consumer too
What's the best way to set up service bindings with two workers, one written in Rust, the other in JS? Two separate project spaces referenced together via
It’s pretty much a promotion of mailchannels yes but you don’t need an account on mailchannels you just setup the dns records needed and send emails for free
There’s some limits around how fast you can start sending out emails like you can’t randomly have a giant burst of thousands of emails can’t find the exact message rn from search but generally yeah you shouldn’t really have any issues using it fully free