Yes, the goal of the sharding abstraction is to grow above the 10GB limit. There are multiple approa
Yes, the goal of the sharding abstraction is to grow above the 10GB limit.
There are multiple approaches to handle ordering ontop of that. You could even have multiple levels of sharding, e.g. a tree of DO shards. The hash function needs to spread the load as much as possible to the shards.
The rest can be done ontop of that.
For example, a super simple approach is to bucket your events depending on your volume. Like buckets of 5 seconds. And then all events mapping to a 5s bucket are using the same partition key, so you have locality for those 5s to batch process all events of those seconds together. It depends on volume though of your application and specific needs.
There are multiple approaches to handle ordering ontop of that. You could even have multiple levels of sharding, e.g. a tree of DO shards. The hash function needs to spread the load as much as possible to the shards.
The rest can be done ontop of that.
For example, a super simple approach is to bucket your events depending on your volume. Like buckets of 5 seconds. And then all events mapping to a 5s bucket are using the same partition key, so you have locality for those 5s to batch process all events of those seconds together. It depends on volume though of your application and specific needs.
