hi, what's the recommended way to separate multi-tenant analytics data with analytics-engine? perfor

Ddandoen hi, what's the recommended way to separate multi-tenant analytics data with anal...

B

Ben•2/7/25, 11:08 PM

https://developers.cloudflare.com/analytics/analytics-engine/sql-api/#sampling

I believe they say to put tenant ids on the index, this way they are sampled seperately

Cloudflare Docs

Workers Analytics Engine SQL API · Cloudflare Analytics docs

The SQL API for Workers Analytics Engine

S

sathoro•2/8/25, 9:22 AM

would love to use this but only with indefinite storage

E

else•2/8/25, 11:40 PM

is

writeDataPoint

writeDataPoint

writeDataPoint

writeDataPoint supposed to work locally with wrangler dev?

Eelse is `writeDataPoint` supposed to work locally with wrangler dev?

H

Hello, I’m Allie!•2/9/25, 10:38 AM

Nope

E

else•2/9/25, 11:49 AM

any way to make that work?

E

else•2/9/25, 11:49 AM

like, is there an api?

E

else•2/9/25, 11:51 AM

any tricks on improving local dev?

R

rohin•2/11/25, 4:02 PM

T

tornado•2/11/25, 9:21 PM

Hello,

Trying to wrap my head around workers analytics engine. Does it make sense to store a requestID in the blobs or will there be consequences?

My use case is to allow my users to investigate failed requests, there are traces associated with the request in R2. I'm thinking instead of storing every request in my db, I can query analytics engine to get failed requests for a specific customer and then use the requestID to get my traces from R2. Would this work? Any caveats?

Rrohin Empty message

D

DaniFoldi•2/11/25, 10:18 PM

For other, can I add "ability to query from within a worker without the REST API, so local testing can be done more locally? Meaning that local writeDataPoints aren't just blackholed, and queries return production data, also requiring an API token" :MeowHeartCloudflare:

J

judge•2/17/25, 3:24 AM

Has anyone already tried to dump analytics engine data? Would love to keep data past a few months for archival purposes.

J

judge•2/17/25, 3:25 AM

(I imagine I can just do

select *

select *

select *

select *, but wanted to see if anyone has already tried and knows some tips/quirks)

MMax (@rozenmd)an option to dump older data to r2 long-term storage could be cool too

J

judge•2/17/25, 3:26 AM

J

judge•2/17/25, 3:26 AM

it would be nice to have it automatically archive to r2, , as mentioned back here - even if you couldn't restore the data seamlessly.

L

LabbedRat•2/26/25, 5:44 PM

Is there a way to transform a blob into a double or an int when querying with worker analytics engine? Similarly to how

toDateTime()

toDateTime()

toDateTime()

toDateTime() works?

K

kaelys.js•3/1/25, 10:03 PM

Not sure if this has been asked but where is the data for Analytics Engine stored, which geographical locations?

Kkagitac Deletion is OLAP systems are very expensive. What is your use case for deletion?

W

Walshy•3/2/25, 8:31 PM

You can't delete, deletes are expensive

W

Walshy•3/2/25, 8:31 PM

You can use a diff dataset, change the indexId or have a version in your data which you check

W

Walshy•3/2/25, 8:32 PM

or just filter out whatever and let it fall out of retention

J

Jakob Norlin•3/4/25, 2:13 PM

We are seeing data points being sampled incorrectly.

These are 7 different data points we write to analytics. Notice the first data point has

"blob1": "pass"

"blob1": "pass"

"blob1": "pass"

"blob1": "pass" while the others have

"blob1": "fail"

"blob1": "fail"

"blob1": "fail"

"blob1": "fail".

const points = [
  { "indexes": ["abc123"], "blobs": ["pass"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] }
];

for (const point of points) {
  env.ANALYTICS.writeDataPoint(point);
}

const points = [
  { "indexes": ["abc123"], "blobs": ["pass"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] }
];

for (const point of points) {
  env.ANALYTICS.writeDataPoint(point);
}

const points = [
  { "indexes": ["abc123"], "blobs": ["pass"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] }
];

for (const point of points) {
  env.ANALYTICS.writeDataPoint(point);
}

const points = [
  { "indexes": ["abc123"], "blobs": ["pass"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] },
  { "indexes": ["abc123"], "blobs": ["fail"] }
];

for (const point of points) {
  env.ANALYTICS.writeDataPoint(point);
}

SELECT _sample_interval, timestamp, blob1
FROM ANALYTICS
WHERE index1 = 'abc123'
FORMAT JsonEachRow

SELECT _sample_interval, timestamp, blob1
FROM ANALYTICS
WHERE index1 = 'abc123'
FORMAT JsonEachRow

SELECT _sample_interval, timestamp, blob1
FROM ANALYTICS
WHERE index1 = 'abc123'
FORMAT JsonEachRow

SELECT _sample_interval, timestamp, blob1
FROM ANALYTICS
WHERE index1 = 'abc123'
FORMAT JsonEachRow

Running the above query gives the below results. Notice that the samples add up to the count 7, but none of rows have

"blob1": "pass"

"blob1": "pass"

"blob1": "pass"

"blob1": "pass".

{ "_sample_interval":2, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }
{ "_sample_interval":2, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }
{ "_sample_interval":2, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }
{ "_sample_interval":1, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }

{ "_sample_interval":2, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }
{ "_sample_interval":2, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }
{ "_sample_interval":2, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }
{ "_sample_interval":1, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }

{ "_sample_interval":2, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }
{ "_sample_interval":2, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }
{ "_sample_interval":2, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }
{ "_sample_interval":1, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }

{ "_sample_interval":2, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }
{ "_sample_interval":2, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }
{ "_sample_interval":2, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }
{ "_sample_interval":1, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }

How come? Is there a way we can query this differently to get the correct data? Or do you just have to accept that there's some data loss when working with Analytics Engine?

J

Jakob Norlin•3/4/25, 2:40 PM

Yeah having sampling kicking in so aggressively at this tiny scale makes it feel like it will be really inaccurate with larger volumes. Is there a way to force analytics engine to only sample across certain fields? I mean for example this would have been more accurate and more compressed

{ "_sample_interval":6, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }
{ "_sample_interval":1, "timestamp":"2025-03-04 13:49:05", "blob1":"pass" }

{ "_sample_interval":6, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }
{ "_sample_interval":1, "timestamp":"2025-03-04 13:49:05", "blob1":"pass" }

{ "_sample_interval":6, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }
{ "_sample_interval":1, "timestamp":"2025-03-04 13:49:05", "blob1":"pass" }

{ "_sample_interval":6, "timestamp":"2025-03-04 13:49:05", "blob1":"fail" }
{ "_sample_interval":1, "timestamp":"2025-03-04 13:49:05", "blob1":"pass" }

A

Aster•3/4/25, 2:58 PM

Don't you want to add the blob value to the index then?
And have

abc123-${passOrFailValue}

abc123-${passOrFailValue}

abc123-${passOrFailValue}

abc123-${passOrFailValue} as index?

LLabbedRat Is there a way to transform a blob into a double or an int when querying with wo...

J

John Spurlock•3/4/25, 3:11 PM

the only string -> number coercion AE supports seems to be

toUInt32

toUInt32

toUInt32

toUInt32 https://developers.cloudflare.com/analytics/analytics-engine/sql-reference/#touint32

JJohn Spurlock the only string -> number coercion AE supports seems to be `toUInt32` https://de...

L

LabbedRat•3/4/25, 10:33 PM

Thanks a ton! I did do a bit of experiments on my own and can confirm this.

Given the link to clickhouse, I was hoping more conversion functions were available. Fingers crossed for the future!

LLabbedRat Thanks a ton! I did do a bit of experiments on my own and can confirm this. Giv...

J

John Spurlock•3/4/25, 10:56 PM

absolutely, same

L

LabbedRat•3/6/25, 11:53 PM

Hello! Thanks again for answering my last question.

According to the limits, each index is limited to 96 bytes. How do we know how many bytes a string will use for the index?

Could we do the following in JS to estimate the number of bytes:

new TextEncoder().encode(text).length

new TextEncoder().encode(text).length

new TextEncoder().encode(text).length

new TextEncoder().encode(text).length ?

T

Teejers•3/12/25, 5:54 PM

Hey there - how do I enable analytics engine within my account? I have a paid workers account & not seeing any directions in the docs.

T

Teejers•3/12/25, 6:05 PM

within the cloudflare dashboard it says it's enabled, but then in the workers overview it seems like it isn't.

TTeejers within the cloudflare dashboard it says it's enabled, but then in the workers ov...

H

Hard@Work•3/12/25, 6:09 PM

IIRC that's just because you don't have any namespaces yet. Have you tried deploying a Worker with an Analytics Engine binding yet?

HHard@Work IIRC that's just because you don't have any namespaces yet. Have you tried deplo...

T

Teejers•3/12/25, 6:11 PM

yeah, and it shows up in the workers binding section

TTeejers yeah, and it shows up in the workers binding section

H

Hard@Work•3/12/25, 6:12 PM

Have you written anything to it?

HHard@Work Have you written anything to it?

T

Teejers•3/12/25, 6:13 PM

attempting to haha, let me do some digging

TTeejers attempting to haha, let me do some digging

H

Hard@Work•3/12/25, 6:14 PM

Also note too that you can't query Analytics Engine from the Dashboard. All you will see is the number of writes you had over a time period for a given namespace

HHard@Work Also note too that you can't query Analytics Engine from the Dashboard. All you ...

T

Teejers•3/12/25, 6:15 PM

Gotcha - was worried I needed to do something else to enable it

TTeejers Gotcha - was worried I needed to do something else to enable it

H

Hard@Work•3/12/25, 6:15 PM

You shouldn't need to

HHard@Work You shouldn't need to

T

Teejers•3/12/25, 6:20 PM

And to confirm, I can write data using the binding, but will need to fetch the API in order to read it?

TTeejers And to confirm, I can write data using the binding, but will need to fetch the A...

H

Hard@Work•3/12/25, 6:20 PM

That's correct

TTeejers And to confirm, I can write data using the binding, but will need to fetch the A...

V

Visal .In•3/13/25, 12:29 PM

You can also use https://studio.outerbase.com/ (Disclaimer: I am their maintainer)

K

Kenton•3/18/25, 6:32 PM

...is Analytics Engine really just a dead-cheap way to host ClickHouse? We're starting a new project at work.. curious if this is the way to go. Our alternative is the hyper-expensive CH haha

Original message was deleted

K

Kenton•3/18/25, 6:49 PM

haha fair enough

I just like the entrypoint of a decent free tier.. unlike Cloud CH lol.

We may just end up using self-hosted CH in the beginning as well, it seems like some things are definitely missing from WAE + 3 mo data retention is hmm

I

Isaac McFadyen•3/18/25, 6:50 PM

^ also might not make sense if you want full control, WAE locks some things down for operational simplicity

I

Isaac McFadyen•3/18/25, 6:51 PM

For example, no deletes.

K

Kenton•3/18/25, 6:53 PM

Makes sense, thanks for the info!

K

kaelys.js•3/19/25, 2:43 AM

@Leo are you saying that CF analytics data is stored in Portland and Amsterdam? Been curious about this as there’s no info I can find about where/how data is stored.

W

wolf•3/19/25, 6:30 AM

Hi there, so all our #baselime queries and alerts have disappeared without any notice, what’s going on? The product pages are still up, but the functionality has disappeared.

J

Jonathan•3/20/25, 8:39 PM

I'm interested in storing user access events and interactions with the product I'm building. Accuracy is more important than access speed (i.e. it's ok if there is a delay seeing events and counts). However, based on the docs for analytics engine, if I'm understanding correctly, all data is sampled and there is no guarantee precise values? Examples of the two use cases I'm interested in are:

Jon smith watched a video on the site for 10 minutes of total play time. On average Jon smith watches about 15 minutes for all videos jon watches.
Events about jon smith, e.g. logged in, changed password, bought ticket, logged back in 3 days later

Based on my reading it seems like for case 1, I would receive back a fairly close approximation but no gurantee it is exact. For case 2, I could end up missing events depending on how much data was either stored or read back. Is that accurate?

JJonathan I'm interested in storing user access events and interactions with the product I...

V

Victor•3/22/25, 10:20 PM

https://developers.cloudflare.com/analytics/analytics-engine/sampling/
For stuff like #1, sampling is fine and there's even an example for averaging through sampling.
As for #2, depending on your scale and index column, some events will fall through the cracks

LLabbedRat Hello! Thanks again for answering my last question. According to the limits, ea...

J

John Spurlock•3/22/25, 11:33 PM

That's a good question, I'm sure you could insert some example values to find out. On the js-side, your code will give you the utf-8 encoded byte length, but clickhouse does not have the concept of encodings [1].

On the ch/ae side, the

length

length

length

length function is supposed to return the octet length, but the ae implementation appears to return the grapheme length instead (and doesn't implement

lengthUTF8

lengthUTF8

lengthUTF8

lengthUTF8) so I'm not sure you can rely on that.

[1] https://clickhouse.com/docs/sql-reference/data-types/string