Neon for multi-tenant SaaS clickstream
For a while now i've been looking at alternatives for storing clickstream data and preforming analytics queries on it. We want to simplify our stack and go all-in on postgres and Neon seems to be a good fit for 99% of our use cases. The biggest issue is our clickstream data and analytics.
Our current system cost too much and doesn't scale. We are storing 12 billion clickstream events in a self-hosted Mongodb database (not sharded). That gives us about 6 months of data retention which is not enough. We want to increase the number of events and their payload size and it's just not possible with our current storage. We have about 5-600 customers and 80% of the data/load is generated by a handful of them so we really need better tenant isolation that is hard to do on mongo.
We are debating between making postgress the clickstream store or using something like clickhouse and deal with the complexity of eventual consistency. Like basically everyone, our ideal would be to have the same database for our OLTP and OLAP workloads but i'm not sure we are there yet. I see that Neon support
pg_mooncake (maybe too new and experimental for now) and timescaledb which would probably be enough for most of our users the issue is really the big whales generating millions of events daily and in spikes.
I would like it to be able to handle at least 10k events/s per tenant and have a 2 years data retention while supporting the rest of the application. I know it's a bit vague and that we need to benchmark it but is Neon be able to handle such load ? Would it just be better to bite the bullet and go multi database with clickhouse or neon should be able to get us there ?3 Replies
sunny-green•5mo ago
@WoofWoofDude I've pinged the team with your questions. You can also reach out here for advice from our solutions team. https://neon.tech/contact-sales
Neon
Contact Sales — Neon
Interested in learning more about our plans and pricing? Contact our sales team.
metropolitan-bronzeOP•5mo ago
@Daniel thanks
There's one thing i didn't know at the time is that gcp is not yet supported so that's a bummer for us, but not necessarily a deal breaker.
sunny-green•5mo ago
There's been discussion of GCP support, but no commitment yet that I know of. We're currently on Azure and AWS, as I'm sure you know already.