How to properly test my system ?
Hey
I’m working on a microservices setup and wanted to pick your brains a bit. Here’s what I’ve got so far:
- Service A exposes PATCH/POST /shipments endpoints
- Service B gets calls from Service A and handles PATCH /inventory and /tracking exclusively
- Kafka Topic X connects Service B to Service C
- Service C processes the messages and then forwards them to an external system
- And there’s HornetQ in between Service C and the external system, which can only handle X messages/sec (let’s say X = 10, based on what the external system allows. It does not depend on us)
(see the graph for an architecture diagram. The purple parts are the parts belonging to the system. The circles are external systems)
So I want to do some solid performance testing—peak load, stress, all of that. But I’m trying to make sure I’m asking the right questions upfront. What should I be looking for when it comes to system performance? What’s the best way to push the whole thing to its limits and really explore where it breaks? I’m planning to put together a test plan that helps identify the performance of the system, where the bottlenecks are, etc. but I’m not 100% sure what to focus on or where to start.
(The tools I need to use are k6 and prometheus for metrics)
If anyone’s worked with similar architectures—especially where there’s an external system limiting throughput—any tips or lessons learned would be super helpful.
Thanks in advance!
4 Replies
Any help 🙏 ?
well, you are the one who knows your system, you should be telling what you care about
if the macroscopic operations are the inventory, tracking, and shipment endpoints then i guess those will be the important parts that need to be measured
so you test it like for an hour with a growing load to have at least an idea of what it can handle, then obv. you look at machines and services logs to understand when something starts failing, address the bottleneck, repeat while it makes sense
what to measure exactly then depends on what is limiting the system, but if you have no idea of how the various parts perform then you can only start with a generic test
i think you know you will probably need to have an indicator how how much the queries take/how much db are impacted for a single api call, and how the system behaves >90% load
Maybe i was not clear enough, I am looking for people who done similar things, some models to learn form. Any help ? 🙏
is there something specific you're worried about the external system limits?