Using Upstash

Hey guys, I was wondering if i could get some insight into the best way to design the following system. I have a rough idea on how to design it but not 100% sure. So I have IoT devices in the field, and only activate when a certain sensor is enabled (which should send a notification to the user with the data). Once activated they will start sending data to my api every Xms for Y seconds. (just say data every 500ms, for 10 seconds) This data is sent over HTTP to my nextjs api with data. Sometimes it might spam the api endpoint if the frequency is quite low. To account for this I dont want to send a user a notification on their phone every time this happens. Currently what i am doing, is inserting all this data into a MessageBus table. which is exactly what it says, a message bus. I've realised this is probably in efficient, and going to move to redis streams on upstash. Now this is where im a little unsure on what to do next. So when data comes in from a device, push it to the redis stream. Cool that works, but then how should i know when to send a notification to the user? I would need some form of server that reads the redis stream, and calls the appropriate actions, but this isnt possible to do in nextjs...
32 Replies
Neto
Netoā€¢2y ago
As it's http based Use some form of rate limiting Loss of data? Possibly But won't spam that much
WOLFLEADER
WOLFLEADERā€¢2y ago
yea ofc not really sure how to handle the stream though because its all serverless
Neto
Netoā€¢2y ago
If you really want You can use planet scale http driver And just send directly to the database And check on next with a cron
WOLFLEADER
WOLFLEADERā€¢2y ago
i need it real time tho i want to be abel to send a notification as soon as data starts coming in (obviously only 1 notif) im not really sure if i need redis at this point
Neto
Netoā€¢2y ago
You can check upstash Kafka then It's based on streams and almost realtime
WOLFLEADER
WOLFLEADERā€¢2y ago
http request with data comes in put data in database send qstash message to request notification if its new data would that be better instead of kafka or redis? same issue tho - still need something that subs to the kafka stream
Neto
Netoā€¢2y ago
If it's based on realtime Serverless is hard
WOLFLEADER
WOLFLEADERā€¢2y ago
yea... hmmm i might need a server then hey maybe qstash publish a message to endpoint?
Neto
Netoā€¢2y ago
Idk if it's possible You can push messages directly to redis And cron it to analyze and flush But it's back to issue 1
WOLFLEADER
WOLFLEADERā€¢2y ago
im not 100% familiar with qstash, but isnt the point of qstash to forward messages?
WOLFLEADER
WOLFLEADERā€¢2y ago
so like when i get data on my api publish it on qstash. oh but ye same issue
Neto
Netoā€¢2y ago
Yeah
WOLFLEADER
WOLFLEADERā€¢2y ago
I guess i could just not use redis, kafka or qstash it would look like IoT --> send data to API --> insert data into planetscale db --> call a function to send a notification to user (this then checks the db when the last message was, it if was more than 5mins, send a notification) i believe that would still be real time
Neto
Netoā€¢2y ago
Sounds nice and easy to maintain tbh
WOLFLEADER
WOLFLEADERā€¢2y ago
yea means i dont have to worry about redis etc.
Neto
Netoā€¢2y ago
Yes And planetscale does have the perf for that
WOLFLEADER
WOLFLEADERā€¢2y ago
and in my notification function i can rate limit notifications by doing a simple check.
Neto
Netoā€¢2y ago
Yep yep
WOLFLEADER
WOLFLEADERā€¢2y ago
now i just need to figure out where i should upload my files too
Neto
Netoā€¢2y ago
S3 Don't handle file uploads on lambdas
WOLFLEADER
WOLFLEADERā€¢2y ago
the IoT sends the image to my nextjs api. then i can just pipe the image to S3?
Neto
Netoā€¢2y ago
Get a signed url, and send directly from the device
WOLFLEADER
WOLFLEADERā€¢2y ago
šŸ‘ ok
Neto
Netoā€¢2y ago
And send just the link to store somewhere
WOLFLEADER
WOLFLEADERā€¢2y ago
any providers like S3? so send the url from the device to my api, which adds it to the table?
Neto
Netoā€¢2y ago
You have other options from providers like azure and such But S3 is easier overall
WOLFLEADER
WOLFLEADERā€¢2y ago
fair
Neto
Netoā€¢2y ago
Less of the worse Create url Send to S3 from device Store url on database
WOLFLEADER
WOLFLEADERā€¢2y ago
šŸ‘ assuming its an easy api to use, just would need an api key or something?
Neto
Netoā€¢2y ago
TomDoesTech
YouTube
Use Presigned PUT URLs to Easily Upload Files to AWS S3
Theo's Tweet: https://twitter.com/t3dotgg/status/1564019039877271552 Repository: https://github.com/TomDoesTech/pre-signed-put-url 0:00 Intro 0:43 Data flow 1:45 Bootstrap application 2:33 S3 bucket setup 5:07 Get PUT URL handler 10:36 Form 15:24 Does it work? šŸŒŽ Follow me here: Discord: https://discord.gg/4ae2Esm6P7 Twitter: https://twitter.c...
WOLFLEADER
WOLFLEADERā€¢2y ago
cheers
eyeino
eyeinoā€¢2y ago
big fan of cloudflare r2, and really cloudflare in general
Want results from more Discord servers?
Add your server