I have a iot project that DO's are a good fit for and im curious how others have handled historical

I have a iot project that DO's are a good fit for and im curious how others have handled historical data?
once a new msg comes in where do you store the old state if you wanted to provide a historical view of what happened?

HHowlOftheSun I have a iot project that DO's are a good fit for and im curious how others have...

1984 Ford Laser•1/22/25, 6:28 AM

Depends on how many msgs you expect to be storing. Could just them by timestamp or something, whether that be in the DO's storage directly or in a separate service like D1

HowlOftheSunOP•1/22/25, 7:01 AM

DO's storage seems right for the device state part, I think at some point I'd want to do historical queries across multiple devices. I know I could make it work with DOs but it feels heavy handed and d1 would work better. Is there a clean way to move the data out of the DO and into d1?

HHowlOftheSun DO's storage seems right for the device state part, I think at some point I'd wa...

1984 Ford Laser•1/22/25, 7:19 AM

Nah, you'd need to build the middleware yourself. One of the benefits of DOs is how expandable they are, just means you need to build it all yourself, to your exact specs

1984 Ford Laser•1/22/25, 7:21 AM

I haven't tried it before, but someone in the past has suggested this as a starting point: https://github.com/outerbase/starbasedb/tree/main/src/export

HHowlOftheSun DO's storage seems right for the device state part, I think at some point I'd wa...

lambrospetrou•1/22/25, 9:36 AM

There are two common patterns I have seen in practice about this and it depends on how much data you have, and how much accuracy you want in your historical/aggregated queries.

If you need 100% accuracy, and actually querying 100% of your data, then there is no alternative than to actually query all your DOs. So, in this case you need to see how much data you have, and maybe reduce the amount of DOs you use (e.g. instead of 1 DO per device, have 1 DO per 100 devices) so that your queries against all of them match your latency requirements.
Each of your DOs publishes aggregated information per N seconds/minutes to a separate group of DOs/D1 which are much less and easy to query all of them, or even better use Workers Analytics Engine (WAE). Each of your DOs sends some events to WAE and then you query WAE across data from all our DOs. Keep in mind, WAE does sampling when you have hundreds/thousands of writes per second for a specific index key you use (https://developers.cloudflare.com/analytics/analytics-engine/sampling/ ).

Rahul gangotri•1/22/25, 7:39 PM

If i make a websocket as a service saas like pusher and host on cloudflare worker and durable objects

To sell to millions of users, do i need 1 worker? Or each user require separate worker?

Rahul gangotri•1/22/25, 7:40 PM

Just curious

RRahul gangotri If i make a websocket as a service saas like pusher and host on cloudflare worke...

geg•1/22/25, 7:44 PM

im doing this with 1 worker

Ggeg im doing this with 1 worker

Rahul gangotri•1/22/25, 7:45 PM

How do you handle the storage limit?

Also Do you store on sql based or use normal storage?

RRahul gangotri How do you handle the storage limit? Also Do you store on sql based or use nor...

geg•1/22/25, 7:47 PM

Oh I thought you meant just pushing messages to many users, no storing

geg•1/22/25, 7:48 PM

For that I would store messages in DO and keep them in cache in colocations

Rahul gangotri•1/22/25, 7:50 PM

I dont want to store messages, rather i want to store user connections, channel yhey are subscribed to and all

Messages will be send as we get with no storage

geg•1/22/25, 7:52 PM

I currently have a setup where a DO handles as many channels possible and only 1 connection per colocation that pushes the messages to Cache api, then all the listeners are just reading from the cache on an interval

geg•1/22/25, 7:53 PM

actually they first read from a global array and 1 listener per worker isolate fills that array from the cache api

geg•1/22/25, 7:53 PM

so like literally 1 connection per colocation to the DO

Rahul gangotri•1/22/25, 7:53 PM

Colocation means?

geg•1/22/25, 7:53 PM

a datacenter

Rahul gangotri•1/22/25, 7:54 PM

Are you using worker for platform or normal worker paid plan?

geg•1/22/25, 7:54 PM

im not offering it as service specifically

geg•1/22/25, 7:54 PM

normal workers

Rahul gangotri•1/22/25, 7:54 PM

That’s what

Rahul gangotri•1/22/25, 7:54 PM

I need to offer as service like ably and pusher

geg•1/22/25, 7:55 PM

i think what i just told you is pretty good step towards it

geg•1/22/25, 7:55 PM

it costs nothing

Rahul gangotri•1/22/25, 7:55 PM

So connection will also scale but hibernated

And channel management needs to be done

So i am just curious that how will it work

geg•1/22/25, 7:55 PM

maybe cloudflare will come knocking with too many users

Ggeg maybe cloudflare will come knocking with too many users

Rahul gangotri•1/22/25, 7:56 PM

Right, i dont have money to buy enterprise plan tho

Rahul gangotri•1/22/25, 8:05 PM

Maybe anyone from cloudflare can help?

RRahul gangotri If i make a websocket as a service saas like pusher and host on cloudflare worke...

kenton•1/22/25, 8:37 PM

If your users upload code to your service and you run the code for them, you need to use Workers for Platforms, otherwise this is insecure. Otherwise, it's up to you.

RRahul gangotri Right, i dont have money to buy enterprise plan tho 😜

kenton•1/22/25, 8:38 PM

for what it's worth, this is a rumor / conspiracy theory. We do not force people to upgrade to enterprise plans just because they "got big" or whatever.

Kkenton for what it's worth, this is a rumor / conspiracy theory. We do not force people...

geg•1/22/25, 8:39 PM

May I ask what you think of my setup I described above? Is it okay to do that to save costs?

Kkenton If your users upload code to your service and you run the code for them, you nee...

Rahul gangotri•1/22/25, 8:40 PM

I dont want user to upload their code rather manage channel subscriptions and all myself, just 1 worker be enough? With multiple DO per app for my user?

Rahul gangotri•1/22/25, 8:40 PM

Similar to ably or pusher

Ggeg May I ask what you think of my setup I described above? Is it okay to do that to...

kenton•1/22/25, 8:40 PM

it sounds fine to me, if your DO is not getting overloaded

RRahul gangotri I dont want user to upload their code rather manage channel subscriptions and al...

kenton•1/22/25, 8:41 PM

If it it works, it works. I don't know enough details about your app to say if you will have scalability issues or something.

Kkenton it sounds fine to me, if your DO is not getting overloaded

geg•1/22/25, 8:41 PM

Cool!! Yep it definitely doesn't get overloaded this way!

Kkenton If it it works, it works. I don't know enough details about your app to say if y...

Rahul gangotri•1/22/25, 8:42 PM

I am telling its a pusher like service saas

Rahul gangotri•1/22/25, 8:42 PM

You know pusher right?

kenton•1/22/25, 8:42 PM

I don't

Ggeg Cool!! Yep it definitely doesn't get overloaded this way!

geg•1/22/25, 8:44 PM

It would be super epic if this was a native feature. But this works for now

Kkenton I don't

Rahul gangotri•1/22/25, 8:44 PM

Ok no problem, so there is no limit on websocket connection right?

RRahul gangotri Ok no problem, so there is no limit on websocket connection right?

kenton•1/22/25, 8:45 PM

there's just the memory limit and the fact that the DO is single-threaded. Each new connection presumably takes some memory and some CPU time to handle messages, but there's no explciit limit on number of connections.

Rahul gangotri•1/22/25, 8:46 PM

Btw here

https://pusher.com/

Pusher | Leader In Realtime Technologies

Simple, scalable and reliable. Hosted realtime APIs loved by developers and trusted by giants. Build live dashboards, notifications, geotracking, chat and more.

Kkenton there's just the memory limit and the fact that the DO is single-threaded. Each ...

Rahul gangotri•1/22/25, 8:46 PM

And whats the memory limit?

geg•1/22/25, 8:47 PM

We can read the docs for this!

Rahul gangotri•1/22/25, 8:48 PM

No need of alarms or ping pong when hibernation is used

geg•1/22/25, 8:49 PM

I encourage everyone to read docs! We can get more intricate responses from staff this way!

I have a iot project that DO's are a good fit for and im curious how others have handled historical

Similar Threads

Similar Threads

Similar Threads