hiya, what's the `env.ANALYTICS_DATASET` type? The docs don't specify this, trying to get to a compi

M

mindfulplaysOP•8/7/23, 11:42 PM

Spelunking the TS code, I think it is

AnalyticsEngineDataset

AnalyticsEngineDataset

AnalyticsEngineDataset

AnalyticsEngineDataset but want to be sure...

Mmindfulplays hiya, what's the `env.ANALYTICS_DATASET` type? The docs don't specify this, tryi...

J

John Spurlock•8/8/23, 12:27 AM

here's what I use: https://github.com/skymethod/denoflare/blob/8679e7e191a979ecbfcc843eda4a2e36e94a4d96/common/cloudflare_workers_types.d.ts#L1212

B

brian•8/8/23, 11:56 PM

is it possible to access web traffic data (the server-side product for proxied sites) via an API? it would be nice if a worker could collect the data every 30 days before it gets dumped

K

kian•8/8/23, 11:57 PM

Everything you see in the dashboard is just an API call - usually to the GraphQL API.

K

kian•8/8/23, 11:58 PM

https://developers.cloudflare.com/analytics/graphql-api/

A

AsyncBanana•8/18/23, 3:13 PM

When I try to run my Cloudflare Pages site locally with Analytics Engine bindings, I keep getting an error saying they do not exist. Here is my wrangler.toml:

name = "[REDACTED]"
main = "dist/main.js"
compatibility_date = "7/25/2023"

workers_dev = false

analytics_engine_datasets = [
    { binding = "VIEW_ANALYTICS", dataset = "[REDACTED]_DEV_VIEW_ANALYTICS" },
    { binding = "COMMENT_ANALYTICS", dataset = "[REDACTED]_DEV_COMMENT_ANALYTICS" },
]

name = "[REDACTED]"
main = "dist/main.js"
compatibility_date = "7/25/2023"

workers_dev = false

analytics_engine_datasets = [
    { binding = "VIEW_ANALYTICS", dataset = "[REDACTED]_DEV_VIEW_ANALYTICS" },
    { binding = "COMMENT_ANALYTICS", dataset = "[REDACTED]_DEV_COMMENT_ANALYTICS" },
]

name = "[REDACTED]"
main = "dist/main.js"
compatibility_date = "7/25/2023"

workers_dev = false

analytics_engine_datasets = [
    { binding = "VIEW_ANALYTICS", dataset = "[REDACTED]_DEV_VIEW_ANALYTICS" },
    { binding = "COMMENT_ANALYTICS", dataset = "[REDACTED]_DEV_COMMENT_ANALYTICS" },
]

name = "[REDACTED]"
main = "dist/main.js"
compatibility_date = "7/25/2023"

workers_dev = false

analytics_engine_datasets = [
    { binding = "VIEW_ANALYTICS", dataset = "[REDACTED]_DEV_VIEW_ANALYTICS" },
    { binding = "COMMENT_ANALYTICS", dataset = "[REDACTED]_DEV_COMMENT_ANALYTICS" },
]

The bindings show up when wrangler states "your worker has access to the following bindings," but I can't access them in my runtime code. However, secret bindings are still working in my code. If it is any help, my website is built with Astro using the cloudflare adapter's runtime access methods

Mmiki / 46bit AE isn't supported locally yet, sorry

A

AsyncBanana•8/19/23, 12:20 AM

Thanks! Is there any way I can test cloudflare pages websites remotely like with the

--remote

--remote

--remote

--remote flag for workers?

A

AsyncBanana•8/19/23, 12:25 AM

also, for some reason I cannot seem to set analytics engine bindings within specific environments. Is this a general issue, or am I just not configuring it correctly?

U

Unsmart•8/19/23, 12:27 AM

Cloudflare pages doesn’t use a wrangler.toml and you have to pass options in the command so thats probably why you don’t see anything working

A

Aries•8/19/23, 9:58 PM

Are there any resources about this more? I haven't seen much about it aside from the short documentation bits.

Kkagitac Anything in particular you would like to know more about?

A

Aries•8/20/23, 3:06 AM

I'd like to see overall examples in general.

A

Aries•8/20/23, 3:07 AM

I've been building an ad network of sorts, and I intend on using the Analytics Engine for calculating impressions and other metrics, and with the few examples and references I've seen it's possible, but I'd still like some more insight into it.

M

Mackenly•8/21/23, 2:37 AM

From reading the docs this service seems most useful for looking at trends and data points at a big picture rather than being optimized for accuracy. That makes sense for lots of data points and general analytics, but what's the suggestion for things that require higher levels of accuracy that may not tolerate sampling? For example, what's the suggested way to store view counts? Thinking a durable object may be the best bet but that doesn't give an easy way to do time series data.

Alternatively could use D1 to just store it with SQL, but if there's a best practice or way to use the platform that integrates with analytics engine I'd prefer to go that route.

Thanks for your thoughts!

VVictor `wrangler.toml` ```toml ... analytics_engine_datasets = [ { binding = "TRACK...

A

Aries•8/22/23, 2:02 AM

Did you resolve this?

A

Aries•8/22/23, 2:02 AM

I can't figure out how to use multiple datasets, and it's not mentioned anywhere.

AAries Did you resolve this?

V

Victor•8/22/23, 2:04 AM

The issue was the running local. I had pushed the code up as I was testing it and he tried it on live and that's where the 2 points came.

AAries I can't figure out how to use multiple datasets, and it's not mentioned anywhere...

V

Victor•8/22/23, 2:04 AM

You use them the same way you use a single dataset. Just keep each one to its own binding

A

Aries•8/22/23, 2:04 AM

https://files.aries.fyi/2023/08/21/Code_83154a37af784293.png

VVictor You use them the same way you use a single dataset. Just keep each one to its ow...

A

Aries•8/22/23, 2:04 AM

when accessing dataImpressions, it is undefined.

V

Victor•8/22/23, 2:04 AM

are you running it locally in wrangler?

A

Aries•8/22/23, 2:05 AM

Yes, I deployed it and was getting undefined, so I reverted the deployment.

AAries Yes, I deployed it and was getting undefined, so I reverted the deployment.

V

Victor•8/22/23, 2:08 AM

analytics_engine_datasets = [
    { binding = "TRACKING_CLOUDFLARE_REQUEST", dataset = "TRACKING_CLOUDFLARE_REQUEST" },
    { binding = "TRACKING_CLOUDFLARE_KV", dataset = "TRACKING_CLOUDFLARE_KV" },
    { binding = "TRACKING_CLOUDFLARE_D1", dataset = "TRACKING_CLOUDFLARE_D1" },
    { binding = "TRACKING_OPENAI_LM", dataset = "TRACKING_OPENAI_LM_DEV" },
    { binding = "TRACKING_USER_AI_GENERAL", dataset = "TRACKING_USER_AI_GENERAL_DEV" },
]

analytics_engine_datasets = [
    { binding = "TRACKING_CLOUDFLARE_REQUEST", dataset = "TRACKING_CLOUDFLARE_REQUEST" },
    { binding = "TRACKING_CLOUDFLARE_KV", dataset = "TRACKING_CLOUDFLARE_KV" },
    { binding = "TRACKING_CLOUDFLARE_D1", dataset = "TRACKING_CLOUDFLARE_D1" },
    { binding = "TRACKING_OPENAI_LM", dataset = "TRACKING_OPENAI_LM_DEV" },
    { binding = "TRACKING_USER_AI_GENERAL", dataset = "TRACKING_USER_AI_GENERAL_DEV" },
]

analytics_engine_datasets = [
    { binding = "TRACKING_CLOUDFLARE_REQUEST", dataset = "TRACKING_CLOUDFLARE_REQUEST" },
    { binding = "TRACKING_CLOUDFLARE_KV", dataset = "TRACKING_CLOUDFLARE_KV" },
    { binding = "TRACKING_CLOUDFLARE_D1", dataset = "TRACKING_CLOUDFLARE_D1" },
    { binding = "TRACKING_OPENAI_LM", dataset = "TRACKING_OPENAI_LM_DEV" },
    { binding = "TRACKING_USER_AI_GENERAL", dataset = "TRACKING_USER_AI_GENERAL_DEV" },
]

analytics_engine_datasets = [
    { binding = "TRACKING_CLOUDFLARE_REQUEST", dataset = "TRACKING_CLOUDFLARE_REQUEST" },
    { binding = "TRACKING_CLOUDFLARE_KV", dataset = "TRACKING_CLOUDFLARE_KV" },
    { binding = "TRACKING_CLOUDFLARE_D1", dataset = "TRACKING_CLOUDFLARE_D1" },
    { binding = "TRACKING_OPENAI_LM", dataset = "TRACKING_OPENAI_LM_DEV" },
    { binding = "TRACKING_USER_AI_GENERAL", dataset = "TRACKING_USER_AI_GENERAL_DEV" },
]

This is what I have live right now and it works

export type Bindings = Record<string, any> & {
    ...
    TRACKING_CLOUDFLARE_REQUEST?: AnalyticsEngineDataset;
    TRACKING_CLOUDFLARE_KV?: AnalyticsEngineDataset;
    TRACKING_CLOUDFLARE_D1?: AnalyticsEngineDataset;
    TRACKING_OPENAI_LM?: AnalyticsEngineDataset;
    TRACKING_USER_AI_GENERAL?: AnalyticsEngineDataset;
    ...
};

export type Bindings = Record<string, any> & {
    ...
    TRACKING_CLOUDFLARE_REQUEST?: AnalyticsEngineDataset;
    TRACKING_CLOUDFLARE_KV?: AnalyticsEngineDataset;
    TRACKING_CLOUDFLARE_D1?: AnalyticsEngineDataset;
    TRACKING_OPENAI_LM?: AnalyticsEngineDataset;
    TRACKING_USER_AI_GENERAL?: AnalyticsEngineDataset;
    ...
};

export type Bindings = Record<string, any> & {
    ...
    TRACKING_CLOUDFLARE_REQUEST?: AnalyticsEngineDataset;
    TRACKING_CLOUDFLARE_KV?: AnalyticsEngineDataset;
    TRACKING_CLOUDFLARE_D1?: AnalyticsEngineDataset;
    TRACKING_OPENAI_LM?: AnalyticsEngineDataset;
    TRACKING_USER_AI_GENERAL?: AnalyticsEngineDataset;
    ...
};

export type Bindings = Record<string, any> & {
    ...
    TRACKING_CLOUDFLARE_REQUEST?: AnalyticsEngineDataset;
    TRACKING_CLOUDFLARE_KV?: AnalyticsEngineDataset;
    TRACKING_CLOUDFLARE_D1?: AnalyticsEngineDataset;
    TRACKING_OPENAI_LM?: AnalyticsEngineDataset;
    TRACKING_USER_AI_GENERAL?: AnalyticsEngineDataset;
    ...
};

And calling it

env.TRACKING_OPENAI_LM?.writeDataPoint({
    indexes: [...],
    blobs: [...],
    doubles: [...],
});

env.TRACKING_OPENAI_LM?.writeDataPoint({
    indexes: [...],
    blobs: [...],
    doubles: [...],
});

env.TRACKING_OPENAI_LM?.writeDataPoint({
    indexes: [...],
    blobs: [...],
    doubles: [...],
});

env.TRACKING_OPENAI_LM?.writeDataPoint({
    indexes: [...],
    blobs: [...],
    doubles: [...],
});

A

Aries•8/22/23, 2:09 AM

Optional access operator means it would fail silenty though?

A

Aries•8/22/23, 2:10 AM

https://files.aries.fyi/2023/08/21/Code_ec26a151f9c753c8.png

V

Victor•8/22/23, 2:11 AM

Yea, but that only happens if running locally in wrangler for me

A

Aries•8/22/23, 2:11 AM

analytics is exposed but not

dataImpressions

dataImpressions

dataImpressions

dataImpressions

A

Aries•8/22/23, 2:12 AM

Very weird.

AAries analytics is exposed but not ``dataImpressions``

V

Victor•8/22/23, 2:14 AM

Going to

https://dash.cloudflare.com/<account id>/workers/analytics-engine

https://dash.cloudflare.com/<account id>/workers/analytics-engine

https://dash.cloudflare.com/<account id>/workers/analytics-engine

https://dash.cloudflare.com/<account id>/workers/analytics-engine, do you see stats?

A

Aries•8/22/23, 2:14 AM

Only the old dataset.
https://files.aries.fyi/2023/08/21/firefox_1e1881fe7a266c61.png

A

Aries•8/22/23, 2:16 AM

When uploading, it does not acknowledge it at all.
https://files.aries.fyi/2023/08/21/Code_9957cb0c3fb79cea.png

VVictor Going to `https://dash.cloudflare.com/<account id>/workers/analytics-engine`, do...

A

Aries•8/22/23, 2:17 AM

Weird..

A

Aries•8/22/23, 2:17 AM

And when deployed:
https://files.aries.fyi/2023/08/21/firefox_e676d010ecf6d878.png

VVictor ```toml analytics_engine_datasets = [ { binding = "TRACKING_CLOUDFLARE_REQUE...

A

Aries•8/22/23, 2:34 AM

Copying your syntax exactly doesn't register them, but doing it this way does: https://files.aries.fyi/2023/08/21/Code_a16a743b078b69f7.png

A

Aries•8/22/23, 2:35 AM

dataset not dataset_name as well, my bad.

AAries Copying your syntax exactly doesn't register them, but doing it this way does: h...

V

Victor•8/22/23, 2:36 AM

My syntax is setup for multiple environments, that probably why it's not picking up

A

Aries•8/22/23, 2:49 AM

Is

25 writes (writeDataPoint invocations) per client HTTP request

25 writes (writeDataPoint invocations) per client HTTP request

25 writes (writeDataPoint invocations) per client HTTP request

25 writes (writeDataPoint invocations) per client HTTP request a hard limit?

C

chientrm.com•8/22/23, 3:49 AM

is analytics engine available on free plan?

AAries Is ``25 writes (writeDataPoint invocations) per client HTTP request`` a hard lim...

V

Victor•8/22/23, 5:16 AM

I don't know of any limits

E

Elliot Hesp•8/22/23, 3:20 PM

If I'm wrapping analytics engine in an ORM, what flavour of SQL is it using?

EElliot Hesp If I'm wrapping analytics engine in an ORM, what flavour of SQL is it using?

J

John Spurlock•8/22/23, 3:21 PM

it's a subset of clickhouse: https://clickhouse.com/docs/en/sql-reference

A

Aries•8/24/23, 6:24 AM

In the SQL, how would I say give me the top 100, then group the rest as "Other"?

A

Aries•8/24/23, 6:25 AM

https://files.aries.fyi/2023/08/24/firefox_469e36f8aac75ae2.png

AAries In the SQL, how would I say give me the top 100, then group the rest as "Other"?

D

DaniFoldi•8/24/23, 7:04 AM

In SQL, you can use a statement like below

SELECT id, ...,
CASE
  WHEN id <= 100 THEN id
  ELSE 'Other'
END AS groupedId
FROM myTable

SELECT id, ...,
CASE
  WHEN id <= 100 THEN id
  ELSE 'Other'
END AS groupedId
FROM myTable

SELECT id, ...,
CASE
  WHEN id <= 100 THEN id
  ELSE 'Other'
END AS groupedId
FROM myTable

SELECT id, ...,
CASE
  WHEN id <= 100 THEN id
  ELSE 'Other'
END AS groupedId
FROM myTable

However, I'm not sure if clickhouse/AE supports that as I haven't played around with it too much yet, so if it doesn't work you're likely best selecting the top 100, and summing the rest "manually" in subqueries

I

itsmatteomanf•8/31/23, 12:46 PM

Is there a way to get more than the default 25 ingestions per invocation? I have a queue that processes 100 items which need to push things a few things to AE per item.

I

itsmatteomanf•8/31/23, 1:58 PM

Oh, I read now... maybe I should ping @46bit

hiya, what's the `env.ANALYTICS_DATASET` type? The docs don't specify this, trying to get to a compi

Similar Threads

Similar Threads

Similar Threads