Neon for multi-tenant SaaS clickstream
For a while now i've been looking at alternatives for storing clickstream data and preforming analytics queries on it. We want to simplify our stack and go all-in on postgres and Neon seems to be a good fit for 99% of our use cases. The biggest issue is our clickstream data and analytics.
Our current system cost too much and doesn't scale. We are storing 12 billion clickstream events in a self-hosted Mongodb database (not sharded). That gives us about 6 months of data retention which is not enough. We want to increase the number of events and their payload size and it's just not possible with our current storage. We have about 5-600 customers and 80% of the data/load is generated by a handful of them so we really need better tenant isolation that is hard to do on mongo.
We are debating between making postgress the clickstream store or using something like clickhouse and deal with the complexity of eventual consistency. Like basically everyone, our ideal would be to have the same database for our OLTP and OLAP workloads but i'm not sure we are there yet. I see that Neon support
I would like it to be able to handle at least 10k events/s per tenant and have a 2 years data retention while supporting the rest of the application. I know it's a bit vague and that we need to benchmark it but is Neon be able to handle such load ? Would it just be better to bite the bullet and go multi database with clickhouse or neon should be able to get us there ?
Our current system cost too much and doesn't scale. We are storing 12 billion clickstream events in a self-hosted Mongodb database (not sharded). That gives us about 6 months of data retention which is not enough. We want to increase the number of events and their payload size and it's just not possible with our current storage. We have about 5-600 customers and 80% of the data/load is generated by a handful of them so we really need better tenant isolation that is hard to do on mongo.
We are debating between making postgress the clickstream store or using something like clickhouse and deal with the complexity of eventual consistency. Like basically everyone, our ideal would be to have the same database for our OLTP and OLAP workloads but i'm not sure we are there yet. I see that Neon support
pg_mooncaketimescaledbI would like it to be able to handle at least 10k events/s per tenant and have a 2 years data retention while supporting the rest of the application. I know it's a bit vague and that we need to benchmark it but is Neon be able to handle such load ? Would it just be better to bite the bullet and go multi database with clickhouse or neon should be able to get us there ?
