how reliable is the data in the analytics engine? I want to track my active users with it. However

how reliable is the data in the analytics engine?

I want to track my active users with it.

However, I need to know exactly who was active during a certain period of time (lets say a period of 10 seconds)

is this a good choice, I cannot miss users that were active during a period of time.

Jjimh We're probably going to need more information in order to help you. But have you...

anhhoang•9/26/24, 1:13 AM

Please help me

BBen how reliable is the data in the analytics engine? I want to track my active use...

Isaac McFadyen•9/26/24, 3:24 AM

Probably not. Analytics engine performs sampling which means that some data will be discarded and aggregated over a certain period of time (i.e. over 5 minutes) and you don't control when that happens. https://developers.cloudflare.com/analytics/analytics-engine/sampling/

Isaac McFadyen•9/26/24, 3:24 AM

You should probably use a relational DB or another time-series DB that doesn't sample.

BBen how reliable is the data in the analytics engine? I want to track my active use...

rohin•9/27/24, 12:11 AM

It really depends on the index you choose. If you're tracking a user_id or some other unique identifier for a user, and you set that as the index (https://developers.cloudflare.com/analytics/analytics-engine/sampling/#how-to-select-an-index), then sampling should not affect you. if you write lots of data to a single index value, then you may see that single index value get sampled, but the_sample_interval should be fairly accurate. we still read and process every event, ABR just helps us optimize how we store and serve queries from Clickhouse

Cloudflare Docs

Sampling with WAE | Cloudflare Analytics docs

How data written to Workers Analytics Engine is automatically sampled at scale

Victor•9/27/24, 6:06 PM

Since September 27, 2024, I've been getting from inside queue consumer. Only 1 index, 2 doubles, 5 blobs per write, 1 write per message, consumer batch size of 40 message

Victor•9/27/24, 6:07 PM

The docs mention 25 per fetch worker invocation, was that recently turned of for queue consumer invocations?

Jjimh How many writes do you do per worker invocation?

Victor•9/27/24, 6:41 PM

1 per queue message, so 40 per consumer worker invocation

Tim de Pater•10/1/24, 7:39 PM

Hello! I'm trying out analytics engine in my worker. But whatever I do, I always run into the error.
It's a Typescript worker, and I'm simply calling with

in my

wrangler.toml

wrangler.toml

Wrangler correctly identifies as a analytics engine dataset when generating the types;

Analytics Engine is enabled in CloudFlare. What am I missing?

kingmesal•10/11/24, 12:18 AM

Is there a feature roadmap available?

At this point I have had to disable my usage of AE.

Some things I've asked about in the past

Table Join
I need to be able to set the time on write, because I need to bring historic data into the tables
More than 20 blobs or doubles
Wrangler local dev mode
Much longer retention

I'm currently using D1 to support my analytics, but it isn't optimal. I've been trying to use AE for almost 2 years now... Basically when it launched

Kkingmesal Is there a feature roadmap available? At this point I have had to disable my u...

Advany•10/11/24, 7:07 PM

You might want to look at clickhouse or tinybird for that

AAdvany You might want to look at clickhouse or tinybird for that

kingmesal•10/11/24, 9:53 PM

I checked out clickhouses service a couple weeks ago ... i'll take a look at tinybird in the meantime

Puliczek•10/18/24, 6:58 AM

How can I get data from Web Analytics? I want to copy all data to my own dashboard for example visits and page views from previous 24h. I tried to use /analytics_engine/sql enpoint with "SELECT * FROM my_dataset" but cant understand data. Any tutorial where should I look?

Puliczek•10/18/24, 9:16 AM

ok i figure out how to do that, had to switch from sql to graphql.

Puliczek•10/18/24, 9:28 AM

Does anyone know if i can filter by domainname? not by siteTag_in ID?

Walshy•10/18/24, 9:28 AM

I'd recommend #analytics Analytics Engine is its own specific product

kaelys.js•10/20/24, 1:17 AM

Hey, is there a rough timeline for when billing will start for analytics engine? (: if this can be answered ofc.

nathnialjohn•10/24/24, 12:34 PM

hello

John Spurlock•10/28/24, 9:05 PM

the altinity grafana clickhouse plugin just merged a fix for passing custom headers (like the auth needed for ae) in alert definitions - should see this in the next version

https://github.com/Altinity/clickhouse-grafana/pull/613

nick•11/7/24, 5:53 PM

hey all!
We're building a web3 product which utilizes Cloudflare Analytics Engine and it fits perfectly for our use case except the three month data retention limitation
Who can I talk with about product offering for this? happy to pay for service instead of migrating to smth else

yeehow•11/23/24, 8:41 PM

Saving 4 data points on a cron that runs every minute.
However, when querying I am only seeing 2 data points recorded each time. Seems to be random which ones.

Am I running into some limit?

alan•11/23/24, 10:33 PM

is there a way to delete a datapoint?

alan•11/23/24, 10:34 PM

okay from what i gather in the messages in this channel, no

Jjimh At those volumes you shouldn't hit any limit. Is this cron the only thing that w...

yeehow•11/25/24, 8:39 PM

DMed you a simple repro

Vikash•11/26/24, 11:32 AM

Any plan for custom retention/ttl or an option to delete the data?

For example, I need to delete logs when a customer delete their account as per GDPR.

So needs a way to run or set to auto delete some time later

Aster•11/28/24, 1:36 PM

Hello, what software do you use to visualise your data? Not for production use, but when testing locally quick stuff?
I was thinking of using TablePlus, but I don't think SQL via HTTP is supported. Any alternative software to recommend?

Aster•11/28/24, 3:26 PM

Also, I don't think the macros in Grafana are working anymore... When selecting the column timestamp type to TimeStamp it just errors...
Working with and just works

Rahul gangotri•11/30/24, 7:58 PM

Can we do json function like extraction and all? Using sql?

Hybris•12/5/24, 8:48 AM

hello

i'm a bit confused
if I insert 10 billions datapoints
i will just pay for the insertion price an nothing else ?

like what about the storage it take ?
10 billions insertion should cost you at list 1GB-month of storage under the hood no ? i don't have to pay for that ?

Aster•12/6/24, 10:40 PM

Hello

Maybe a noobie question, but how to display the req/sec on grafana using Analytics engine?

I managed to get this, plotting a graph:

But it doesn't seem correct

(way too low compared to logs)

Walshy•12/6/24, 10:47 PM

Aster•12/6/24, 10:57 PM

Works like a charm, thanks!

bentlegen•12/12/24, 7:38 PM

bentlegen•12/12/24, 7:39 PM

Interested in longer retention periods?

Yes.

lucidplot•1/20/25, 4:10 PM

if I want to collect analytics on the proportion of web page views that lead to a conversion event, what should my parameters to be? so that I could query a conversion rate by page URL for a given timeframe. I guess I would need to add a point for each view and also for each conversion?

lucidplot•1/20/25, 4:39 PM

is it as simple as

and would I want to add an index too?

Walshy•1/20/25, 4:56 PM

Yep that looks good.

For index, I'd read: https://developers.cloudflare.com/analytics/analytics-engine/sampling/#how-to-select-an-index

lucidplot•1/20/25, 4:59 PM

thank you Walshy

lucidplot•1/20/25, 5:58 PM

for anyone else, this was the part that explained samping for me https://developers.cloudflare.com/analytics/analytics-engine/sql-api/#sampling

For example, if your worker serves multiple customers, you might consider making customer ID the index field. This would mean that if one customer starts making a high rate of requests then events from that customer could be sampled while other customers data remains unsampled.

nejc•1/28/25, 6:28 AM

Hi all, is there any plan to give support for Analytics engine inside Workers nativly? Passing token in just feels wrong, where you have all that nice bindings available

James•1/28/25, 3:18 PM

It has been requested many times, so hopefully we’ll see it in future!

bentlegen•1/28/25, 6:49 PM

Hey Cloudflare folks, is there any news (or will there be any news) on future retention changes?

rohin•1/29/25, 1:16 AM

Hey Ben, nothing new re: retention yet. I can restart some of those conversations. We've heard 1 year retention frequently. Is that what you are looking for, too?

Aster•1/31/25, 10:32 AM

Hey

Are there plans to support a batch writeDataPoint? It would help with not reaching the current 25 data points limit per worker invocation

Aster•1/31/25, 1:47 PM

------
Also, to plot our workers latency on Grafana we use the following:

Do you know how I could get the average of a percentile over a time range? For reporting over long time period it's pretty convenient to spot latency regression, but I cannot manage to get it to work

dandoen•2/7/25, 11:57 AM

hi, what's the recommended way to separate multi-tenant analytics data with analytics-engine? performant & secure. can't find anything in the docs about this.

Ddandoen hi, what's the recommended way to separate multi-tenant analytics data with anal...

BenOP•2/7/25, 11:08 PM

https://developers.cloudflare.com/analytics/analytics-engine/sql-api/#sampling

I believe they say to put tenant ids on the index, this way they are sampled seperately

Cloudflare Docs

Workers Analytics Engine SQL API · Cloudflare Analytics docs

The SQL API for Workers Analytics Engine