I have 400 users details like email, name, contact etc inside an array. I want to insert/update all

I have 400 users details like email, name, contact etc inside an array. I want to insert/update all these 400 users profiles information inside 3rd party service called klaviyo for email marketing. In adding 1 user information into this 3rd party API, it takes 2 fetch requests (max 2 for update). So for 400 users its taking 800 fetch requests at a time for inserting. I am getting error inside logs that "subrequest limit exceeds" means so many subrequest a time...
What is solution of this, can anyone please help?
NOTE: I am using premium version not free
I have also attached a screenshot of my code with result array of 400 users and then adding them one by one into klaviyo

Bbharatsuthar07 I have 400 users details like email, name, contact etc inside an array. I want t...

Ashley•11/14/23, 4:52 PM

Either limit the number of customers you send per scheduled run (e.g. in your DB, have a column that is "sent" or whatever) and then retrieve X number per scheduler run and only pull ones where they aren't sent yet to the API. Or use https://developers.cloudflare.com/queues/, and push each customer as a message into a queue, and have a separate worker process the queue

Ashley•11/14/23, 4:52 PM

Well, those are two options anyway

Ashley•11/14/23, 4:53 PM

https://developers.cloudflare.com/workers/platform/limits/#how-many-subrequests-can-i-make also

Limits · Cloudflare Workers docs

Cloudflare Workers plan and platform limits.

Ashley•11/14/23, 4:53 PM

I also don't know what it is for the new pricing model, as they only talk about bound vs unbound there

Ashley•11/14/23, 4:54 PM

As there is a big difference (50 for bound, 1000 for unbound) but not sure what it will be for standard (the new pricing model that is now available)

Bbharatsuthar07 I have 400 users details like email, name, contact etc inside an array. I want t...

Unsmart•11/14/23, 4:55 PM

If you change to the new billing model (Standard) you can do 1,000 fetch requests. You could also send all of them to queues instead so you can support retrying the requests over time if needed.

Ashley•11/14/23, 4:56 PM

That answers my Q on the standard model, thanks

bharatsuthar07OP•11/14/23, 5:02 PM

I tried a way called batch processing.... I first add 20 users then added delay function (Promise) wait of 60 seconds or setInterval/setTimeout and then add another 20 users...
but it didn't worked, dont know why? It always adding that first 20 users only after every 60 seconds...not going to next 20 users.
Added 2 screenshots of delay promise function and setTimeout, but both not worked. always adding/updating same chunk/batch. Whats reason?

bharatsuthar07OP•11/14/23, 5:03 PM

Will try your above approaches also, that queue and limits thanks...but above why not worked? @ScaredOfMyRawr @Unsmart | Tech debt or anyone?

bharatsuthar07OP•11/14/23, 5:14 PM

Is there any way for batch processing, its fine. I want to implement that after every minute I add 50 users to 3rd party API likewise, I will be able to add all 400 users in 8 minutes. it will be fine... this schedules every day by cron trigger..is there any way I can do this. because by setTimeout or adding promise and fulfill it on 1 minute. both this approaches not working....

Bbharatsuthar07 Is there any way for batch processing, its fine. I want to implement that after ...

Ashley•11/14/23, 5:31 PM

It’s not a timing thing, it’s a Workers runtime limit. On your current usage plan, you’re limited to 50 subrequests per request (when your scheduled task is run, it is considered one request). You can change your usage plan to standard, or unbound, and that will gice you 1,000 subrequests (e.g. HTTP calls) from your worker.

bharatsuthar07OP•11/14/23, 5:32 PM

I changed to standard, still same

Ashley•11/14/23, 5:32 PM

Did you deploy?

bharatsuthar07OP•11/14/23, 5:32 PM

Yes deployed

Ashley•11/14/23, 5:35 PM

Only thing I can guess at is maybe the subrequest limit is different for cron triggers, or perhaps it takes a little bit for your account to be fully migrated to standard. I would consider queues anyway though, as even if the limit is 1,000, you’re not far from that limit already.

bharatsuthar07OP•11/14/23, 5:35 PM

yes correct

bharatsuthar07OP•11/14/23, 5:57 PM

@ScaredOfMyRawr can you please explain some more about the queue approach, I am not familiar with cloudflare queue, was just reading the docs and got to know that, we have to make 1 producer worker for adding messages into queue and another consumer worker for fetching the message....
So I have to iterate over result array (400 customers) and add one by one into message queue, all this in producer queue...and in consumer queue, that logic and adding customer into klaviyo. right?
Can you please tell more about this approach

Ashley•11/14/23, 6:04 PM

You got it spot on

You already have your producer worker (your scheduled cron trigger) where you publish from, and then you can I think also just put a “queue” method on your worker and it will be your consumer too

Ashley•11/14/23, 6:05 PM

Just put the customer ID or something into the message you put in the queue (or put in some JSON)

bharatsuthar07OP•11/14/23, 6:08 PM

Okay got it, thanks. Will try

BjornTheProgrammer•11/14/23, 6:21 PM

Hello is it possible to add externals to the esbuild used by wrangler?

BjornTheProgrammer•11/14/23, 6:23 PM

As far as I can tell in the documentation, there is not.
https://developers.cloudflare.com/workers/wrangler/configuration/#bundling

Configuration - Wrangler · Cloudflare Workers docs

Use a

wrangler.toml

wrangler.toml

configuration file to customize the development and deployment setup for your Worker project and other Developer Platform …

kian•11/14/23, 6:56 PM

Nope

kian•11/14/23, 6:56 PM

The only externals Wrangler currently has is cloudflare:*cloudflare:* and node:*node:*

hansy•11/14/23, 10:13 PM

What's the best way to set up service bindings with two workers, one written in Rust, the other in JS? Two separate project spaces referenced together via

wrangler.toml

wrangler.toml

kian•11/14/23, 10:16 PM

Yep

Raiyan•11/15/23, 12:25 AM

btw, can anyone tell me that this post about? https://blog.cloudflare.com/sending-email-from-workers-with-mailchannels/

The Cloudflare Blog

Send email using Workers with MailChannels

MailChannels has created an email sending service specifically for Cloudflare Workers that removes all the friction associated with sending emails

Raiyan•11/15/23, 12:26 AM

I am not quite understanding, is it just a promotion for mailchannels and in order to use it, I have to be a customer of mailchannels? let me know

Unsmart•11/15/23, 12:29 AM

It’s pretty much a promotion of mailchannels yes but you don’t need an account on mailchannels you just setup the dns records needed and send emails for free

Raiyan•11/15/23, 12:31 AM

I really don't get it, let me clarify my misconception, how they can allow sending free email? is there any limit?

Unsmart•11/15/23, 12:35 AM

There’s some limits around how fast you can start sending out emails like you can’t randomly have a giant burst of thousands of emails can’t find the exact message rn from search but generally yeah you shouldn’t really have any issues using it fully free

Raiyan•11/15/23, 12:37 AM

that sounds too good to be true

Raiyan•11/15/23, 12:37 AM

I have a provider that doesn't have api support and only exposes SMTP, so I was looking for some solution, but I think this would be enough

RRaiyan I really don't get it, let me clarify my misconception, how they can allow sendi...

Unsmart•11/15/23, 12:40 AM

Found the message: email-routing

Unsmart•11/15/23, 12:42 AM

Also they may have some features that are paid for workers email sending in the future: email-routing

RRaiyan I have a provider that doesn't have api support and only exposes SMTP, so I was ...

James•11/15/23, 12:57 AM

I’d strongly encourage you to consider another platform like SES, SendGrid, Postmark, etc.

I would not trust or recommend MailChannels, personally. More details: workers-and-pages-discussions

Raiyan•11/15/23, 12:58 AM

It is just my client's company, they run on premise smtp server

JJames I’d strongly encourage you to consider another platform like SES, SendGrid, Post...

Raiyan•11/15/23, 1:35 AM

I saw the talk and it was horrifying, I hope the issue is resolved now!

RRaiyan I saw the talk and it was horrifying, I hope the issue is resolved now!

James•11/15/23, 1:37 AM

Generally yes, with domain lockdown. But their response and handling of the situation was, and still to this day is, terrible, and not a company I can recommend using for any use case unfortunately.

Raiyan•11/15/23, 1:38 AM

The interesting part was their market share, I never heard about them before!

Stone•11/15/23, 10:40 AM

Has anyone used lerna? I was wondering what was the best way to share code between workers. For example custom auth handling functions ?

kian•11/15/23, 10:47 AM

The Workers 'native' way would be having a Worker dedicated to auth that the rest of your Workers can call via https://developers.cloudflare.com/workers/configuration/bindings/about-service-bindings/, but that does incur another request charge.

Stone•11/15/23, 11:04 AM

Interesting.. but could it be done differently? Using lerna and have a common code module? I'm a bit old school

kian•11/15/23, 11:05 AM

Sure, but that's not Workers specific.

kian•11/15/23, 11:06 AM

Wrangler just uses esbuild and bundles your Worker - use NPM packages, import from a file path, etc

Stone•11/15/23, 11:07 AM

https://developers.cloudflare.com/workers/tutorials/manage-projects-with-lerna/
...work with multiple Wrangler projects and share dependencies between them.

Managing multiple Workers projects with Lerna · Cloudflare Workers ...

All of the tutorials assume you have already completed the Get started guide, which gets you set up with a Cloudflare Workers account, C3 , and …

kian•11/15/23, 12:14 PM

For the async part, https://developers.cloudflare.com/workers/runtime-apis/handlers/fetch/#contextwaituntil.

I have 400 users details like email, name, contact etc inside an array. I want to insert/update all

Similar Threads

Similar Threads

Similar Threads