Linking Cloudflare R2 as a source - Docs...
My goal is to store events for something like PostHog to injest via R2. Seems like csv and Parquet files format? https://posthog.com/docs/cdp/sources/r2
Would be great to collab on this!
Linking Cloudflare R2 as a source - Docs - PostHog
The data warehouse can link to data in Cloudflare R2. To start, you'll need to: Create a bucket in R2 Set up an access key and secret Add data to the…
5 Replies
Unknown User•4mo ago
Message Not Public
Sign In & Join Server To View
Hey that’s pretty cool thanks for sharing. @disturbing
What events are you trying to store? Are you sending the raw logs from Cloudflare Pipelines to PostHog or are you going to transform the data first?
So the problem statement is:
- It costs $.00005 cents per event to use PostHog events storage + tracking and $.000015 (plus CF Storage cost) to store in a Warehouse + index with Posthog.
These prices go down at scale (100M+).
If Pipelines can keep the savings worth it, then we can take all events from Frontend + Backend services for analytics and store them there.
For exceptions / errors, unfortunately they do not have a warehouse plugin for this and to get it connected + enriched with user data, it may make sense to track errors directly to their SDK.
My question would be how to transform the data and store DIRECTLY in R2 for Posthog to index. @crouton @kagitac
I dont want to have to doule pay buy using webhooks.
Unknown User•4mo ago
Message Not Public
Sign In & Join Server To View
Gotcha - thanks for the info.
I am using Cloudflare workers to forward the events. I can write them to R2 but from the looks of it, it would only be taking in product analytics but none of the others stuff.
So all in all it’s possible it sounds like but writing the events in the particular format (cvs or parquet) in an efficient manner is likely the thing to figure out.
So it sounds like pipelines cannot writer to cvs or parquet yet and we need to wait for transformations to get such a usable output for it.
Using the cloud version since the OSS does lack some features