C#•2y ago

API structure

Hi! I have a route that return stations, the station data updates every 5 minutes, the stations data is big and I want to fetch it only when it is updated. How can I check that data is updated? What's better to create two separate routes, GET /stations = List<Station> GET /stations/last-update = DateTime or to have a single route /stations?=lastUpdate = DateTime /stations = List<Station>. Or is there a better way to do it?

deprecatio•3/21/24, 12:43 PM

im a bit lost, the /stations api is what you are writing, right?
so something is querying that api? is it another service/ui?
but you need to return always the full data? or you want to return just the changes?

Ddeprecatio im a bit lost, the /stations api is what you are writing, right? so something is...

AlexOP•3/21/24, 12:57 PM

I have a react client that displays stations on the map, it sends request to my api to get list of stations, (there is timer on the client to send request every two minutes or when the client opens the page). In the backend calls other api (3rd party api) for the station data, transforms it and stores it in redis. The backend also has a timer to make these calls. So how can I see on the client when the data on backend is updated and receive new data for all stations (clear previous one and save backend response). The route always return full data

deprecatio•3/21/24, 1:08 PM

is the react client in a digital signage?
you would have to open a websocket or a sse and send data from backend to web app

Ddeprecatio is the react client in a digital signage? you would have to open a websocket or ...

AlexOP•3/21/24, 1:17 PM

I didn't understand you first sentence, but I maybe can use sse or websocket to check when the data is updated. Will there be impact on performance of server or client if I switch from regular request to websocket for "last update" request?

mldisibio•3/21/24, 3:34 PM

Couple of thoughts:

the backend that transforms the data and stored it in redis can do the compute to determine if the data changed. If it has changed, you can create a very light pub/sub interaction between Redis and your client and push a notification to your client that the data for a particular station has changed. You could abstract this pattern and do it with Signal R or some other push framework.
If you are using http calls, look into ETagETag variations that can be communicated via headers in the http messages, both request and response. Essentially eTags communicate if data has changed and if you treat your station data like a file or binary blob, you could send only the parts that have changed
Your idea of a second api that just tells you if the station data has changed is pretty simple and would work as well

Mmldisibio Couple of thoughts: 1. the backend that transforms the data and stored it in re...

AlexOP•3/21/24, 5:22 PM

That's are good ideas, but I think what would be if for example, 100 users will get message from SignalR that they need to send request to update data, and as result they send 100 request to the server at once. Won't it make a huge load on the server?

AAlex That's are good ideas, but I think what would be if for example, 100 users will ...

mldisibio•3/21/24, 5:30 PM

That's really an infrastructure question. What is your actual client base and how robust is your server, is it in the cloud and can it scale, etc. In today's world, 100 requests is child's play, and if the data is already cached in Redis, there is no compute, just network traffic. If it were really an issue, you could push the client requests onto a queue and respond to them one at time, but I would load test first and see if that is really a problem.

There are reverse proxies and load balancers out there to solve that problem, so look into those before reinventing that wheel.

Also, if you are considering a push architecture like SignalR, then heck, the client doesn't even need to send a request...just push the updated data to each client.

mldisibio•3/21/24, 5:33 PM

This is starting to sound like mobile phone map apps, and for sure that architecture is quite established and easy to find examples of out there

Mmldisibio That's really an infrastructure question. What is your actual client base and ho...

AlexOP•3/21/24, 5:34 PM

I'm curious how it's usually done, for example in google traffic maps and etc. To do it the normal way

mldisibio•3/21/24, 5:35 PM

Exactly. I'm not an expert in the mobile/push/web-socket architectures, but once you can name it, it should be easy to find discussions and examples.

AlexOP•3/21/24, 5:36 PM

okay, thank you

AAlex I'm curious how it's usually done, for example in google traffic maps and etc. T...

deprecatio•3/22/24, 8:42 AM

as an example
https://learn.microsoft.com/en-us/samples/dotnet/samples/orleans-gps-device-tracker-sample/