Would the 100 batch size be increased later?

BBurrito Would the 100 batch size be increased later?

After we ship partial batch acknowledgment (explicit ack/retry behavior) - otherwise any consumer failure on a large batch means the entire batch is retried. Likely to be closer to end of this quarter (subject to change)

BBurrito Would the 100 batch size be increased later?

Matt Silverlock•1/18/23, 5:24 AM

(What are you trying to do && what is an “increase” to you?)

BurritoOP•1/18/23, 5:33 AM

I'm building a storage like service with pretty low write volume, for reads I want it to be extremely fast and it's acceptable to show stale state for some seconds, while for writes as long as it shows up instantly on the colo it writes from (so that writer won't assume their write failed) then it can take time propagate to other colos.

Currently I'm thinking:
- R2 to store the data.
- Worker checks cache API for a match and return instantly; if there isn't a match, pull from R2 and store it with cache API as immutable cache that lasts forever.
- When a write happens, uses CF API to purge that cache for all colo.

Problem is that CF API has a rate limit of 1k per minute (or lower?), so I'm thinking batching them with queues. 100 per batch is probably enough for the write volume I need to handle, but just want to see if it would become a problem later down the line.

BBurrito I'm building a storage like service with pretty low write volume, for reads I wa...

Matt Silverlock•1/18/23, 6:54 AM

Sounds like a closer fit to KV? Writes are eventually consistent globally, but cached in the local colo for readers. Reads are fast (and much faster than R2).

MMatt Silverlock Sounds like a closer fit to KV? Writes are eventually consistent globally, but c...

BurritoOP•1/18/23, 6:59 AM

KV doesn't quite work, it's pull based so once the cache in a colo expires it needs to go fetch again; there's also no way to lower the cache below 60 seconds, and lowering that would also mean there would be more fetches.

BBurrito KV doesn't quite work, it's pull based so once the cache in a colo expires it ne...

Matt Silverlock•1/18/23, 7:00 AM

Right, I understand how KV works, but “take time to propagate to other colos” left your requirements unclear.

BurritoOP•1/18/23, 7:01 AM

Good point, yeah a few seconds for writes to show up is fine, but 60 seconds of KV is too long.

kian•1/18/23, 7:13 AM

No idea about timeframes, likely a long way away - but r2 might be of interest down the line with KV

BurritoOP•1/18/23, 7:14 AM

Yep keeping an eye on that too, if that becomes a thing than it would solve my problem nicely without having to hack together all these things.

kian•1/18/23, 7:15 AM

How big is the data you’re storing/caching?

BurritoOP•1/18/23, 7:16 AM

Vast majority should be a few MB, largest can still fit in KV.

kian•1/18/23, 7:17 AM

Shame - if it was small enough I’d say go full mad scientist and use a Worker itself as a data store. Would expect the usual provisioner lag to be good for your eventual consistency needs & be fairly quick reads since it exists in every colo without needing to be pulled from a central store like KV

kian•1/18/23, 7:18 AM

Buuut that only works with sub-1MB data - and would also hit script per account limits - and other bad things

kian•1/18/23, 7:18 AM

kian•1/18/23, 7:20 AM

The wording here is weird, since the Cache API “cache key” is just the URL, so what defines custom or not?

kian•1/18/23, 7:20 AM

But I’d recommend making sure the Purge Cache API fits your needs and actually purges the objects you put into the Cache API from a Worker

BurritoOP•1/18/23, 7:21 AM

Yeah I'll check that for sure.

kian•1/18/23, 7:22 AM

You can use up to 30 cache-tags in one API call and make up to 30,000 purge API calls in a 24-hour period.

kian•1/18/23, 7:22 AM

Naturally, these are your bounds too

BurritoOP•1/18/23, 7:23 AM

Ah, wasn't aware that's a thing.

BurritoOP•1/18/23, 7:23 AM

My problem really just boils down to needing a push mechanism to tell each colo "new data, go fetch again"

kian•1/18/23, 7:23 AM

I guess you could have your own invalidation logic in a Worker, checking the age of a cached response & delete/repopulate yourself

kian•1/18/23, 7:24 AM

So it’s not purged at a specific time, but just on as-requested basis

BurritoOP•1/18/23, 7:24 AM

Yep that's the fallback, basically doing my own "stale while revalidate"

BurritoOP•1/18/23, 7:26 AM

That's unfortunately pull based, so to ensure writes show up quickly the TTL needs to be low, which means revalidation happens a lot and increases cost.

kian•1/18/23, 7:26 AM

Or, back to mad scientist, an all colos health check that runs every (X) to hit the Worker and purge cache in that colo

Except that can’t really be triggered in demand, only on an interval - unless you create the health check on demand.

And requires Enterprise

kian•1/18/23, 7:27 AM

I’ll stop with the insane workarounds, as fun as they are

Rrelero90 Running into some issues with data types and undefined values coming out of queu...

Unsmart•1/18/23, 4:23 PM

What does your code look like for sending a message to the queue that would be more important here

Unsmart•1/18/23, 4:24 PM

I assume info.lastRowId is from D1? I know the types on D1 changed recently (at least I am pretty sure it did) so that is probably your issue.

Rrelero90 Your assumption is correct. Any reason why it would pull a value locally but not...

Max (@rozenmd)•1/18/23, 4:26 PM

miniflare's version of d1 is outdated

Max (@rozenmd)•1/18/23, 4:26 PM

adding RETURNING idRETURNING id or similar to your INSERTINSERT will give you the inserted row's id

AAlex We are interested in using Cloudflare Queues or Cloudflare Workers for long-runn...

Unsmart•1/19/23, 3:42 PM

You can use DOs to connect to websockets already

AAlex Thank you for your fast response. Is it true that Durable Objects (DO) supports ...

Unsmart•1/19/23, 3:49 PM

Not sure about HTTP streams but I know at one point people where connecting to discord gateway via DOs. Dont really have much for code example other than whats on the docs: https://developers.cloudflare.com/workers/learning/using-websockets/#writing-a-websocket-client

AAlex Do you have a general recommendation on how to serverless consume Twitter stream...

Hello, I’m Allie!•1/19/23, 6:17 PM

CF Workers can run indefinitely, but you should expect them to run no more than 30 seconds at a time(after which they may be evicted). For long-running tasks like that, I would recommend a serverless container platform like https://fly.io

AAlex BTW: I found this info to be interesting: "Discord migrated to Cloudflare’s glo...

Max (@rozenmd)•1/20/23, 9:19 AM

we have a case study about them too: https://www.cloudflare.com/case-studies/discord/

kian•1/20/23, 5:31 PM

That has been fixed in Miniflare but there hasn't been a new release yet

kian•1/20/23, 5:31 PM

https://github.com/cloudflare/miniflare/commit/917ec0e50d3bec206f41db0db204375d3d644db8

GitHub

Respect the maxBatchSize setting of queue consumers (#450) · cloudf...

Previously we'd send along arbitrarily large batches to the consumer.

kian•1/20/23, 5:32 PM

Just a bug in Miniflare

Would the 100 batch size be increased later?

Similar Threads