I can't figure out how to use multiple datasets, and it's not mentioned anywhere.

V

Victor•8/22/23, 2:04 AM

The issue was the running local. I had pushed the code up as I was testing it and he tried it on live and that's where the 2 points came.

AAries I can't figure out how to use multiple datasets, and it's not mentioned anywhere...

V

Victor•8/22/23, 2:04 AM

You use them the same way you use a single dataset. Just keep each one to its own binding

A

AriesOP•8/22/23, 2:04 AM

https://files.aries.fyi/2023/08/21/Code_83154a37af784293.png

VVictor You use them the same way you use a single dataset. Just keep each one to its ow...

A

AriesOP•8/22/23, 2:04 AM

when accessing dataImpressions, it is undefined.

V

Victor•8/22/23, 2:04 AM

are you running it locally in wrangler?

A

AriesOP•8/22/23, 2:05 AM

Yes, I deployed it and was getting undefined, so I reverted the deployment.

AAries Yes, I deployed it and was getting undefined, so I reverted the deployment.

V

Victor•8/22/23, 2:08 AM

analytics_engine_datasets = [
    { binding = "TRACKING_CLOUDFLARE_REQUEST", dataset = "TRACKING_CLOUDFLARE_REQUEST" },
    { binding = "TRACKING_CLOUDFLARE_KV", dataset = "TRACKING_CLOUDFLARE_KV" },
    { binding = "TRACKING_CLOUDFLARE_D1", dataset = "TRACKING_CLOUDFLARE_D1" },
    { binding = "TRACKING_OPENAI_LM", dataset = "TRACKING_OPENAI_LM_DEV" },
    { binding = "TRACKING_USER_AI_GENERAL", dataset = "TRACKING_USER_AI_GENERAL_DEV" },
]

analytics_engine_datasets = [
    { binding = "TRACKING_CLOUDFLARE_REQUEST", dataset = "TRACKING_CLOUDFLARE_REQUEST" },
    { binding = "TRACKING_CLOUDFLARE_KV", dataset = "TRACKING_CLOUDFLARE_KV" },
    { binding = "TRACKING_CLOUDFLARE_D1", dataset = "TRACKING_CLOUDFLARE_D1" },
    { binding = "TRACKING_OPENAI_LM", dataset = "TRACKING_OPENAI_LM_DEV" },
    { binding = "TRACKING_USER_AI_GENERAL", dataset = "TRACKING_USER_AI_GENERAL_DEV" },
]

analytics_engine_datasets = [
    { binding = "TRACKING_CLOUDFLARE_REQUEST", dataset = "TRACKING_CLOUDFLARE_REQUEST" },
    { binding = "TRACKING_CLOUDFLARE_KV", dataset = "TRACKING_CLOUDFLARE_KV" },
    { binding = "TRACKING_CLOUDFLARE_D1", dataset = "TRACKING_CLOUDFLARE_D1" },
    { binding = "TRACKING_OPENAI_LM", dataset = "TRACKING_OPENAI_LM_DEV" },
    { binding = "TRACKING_USER_AI_GENERAL", dataset = "TRACKING_USER_AI_GENERAL_DEV" },
]

analytics_engine_datasets = [
    { binding = "TRACKING_CLOUDFLARE_REQUEST", dataset = "TRACKING_CLOUDFLARE_REQUEST" },
    { binding = "TRACKING_CLOUDFLARE_KV", dataset = "TRACKING_CLOUDFLARE_KV" },
    { binding = "TRACKING_CLOUDFLARE_D1", dataset = "TRACKING_CLOUDFLARE_D1" },
    { binding = "TRACKING_OPENAI_LM", dataset = "TRACKING_OPENAI_LM_DEV" },
    { binding = "TRACKING_USER_AI_GENERAL", dataset = "TRACKING_USER_AI_GENERAL_DEV" },
]

This is what I have live right now and it works

export type Bindings = Record<string, any> & {
    ...
    TRACKING_CLOUDFLARE_REQUEST?: AnalyticsEngineDataset;
    TRACKING_CLOUDFLARE_KV?: AnalyticsEngineDataset;
    TRACKING_CLOUDFLARE_D1?: AnalyticsEngineDataset;
    TRACKING_OPENAI_LM?: AnalyticsEngineDataset;
    TRACKING_USER_AI_GENERAL?: AnalyticsEngineDataset;
    ...
};

export type Bindings = Record<string, any> & {
    ...
    TRACKING_CLOUDFLARE_REQUEST?: AnalyticsEngineDataset;
    TRACKING_CLOUDFLARE_KV?: AnalyticsEngineDataset;
    TRACKING_CLOUDFLARE_D1?: AnalyticsEngineDataset;
    TRACKING_OPENAI_LM?: AnalyticsEngineDataset;
    TRACKING_USER_AI_GENERAL?: AnalyticsEngineDataset;
    ...
};

export type Bindings = Record<string, any> & {
    ...
    TRACKING_CLOUDFLARE_REQUEST?: AnalyticsEngineDataset;
    TRACKING_CLOUDFLARE_KV?: AnalyticsEngineDataset;
    TRACKING_CLOUDFLARE_D1?: AnalyticsEngineDataset;
    TRACKING_OPENAI_LM?: AnalyticsEngineDataset;
    TRACKING_USER_AI_GENERAL?: AnalyticsEngineDataset;
    ...
};

export type Bindings = Record<string, any> & {
    ...
    TRACKING_CLOUDFLARE_REQUEST?: AnalyticsEngineDataset;
    TRACKING_CLOUDFLARE_KV?: AnalyticsEngineDataset;
    TRACKING_CLOUDFLARE_D1?: AnalyticsEngineDataset;
    TRACKING_OPENAI_LM?: AnalyticsEngineDataset;
    TRACKING_USER_AI_GENERAL?: AnalyticsEngineDataset;
    ...
};

And calling it

env.TRACKING_OPENAI_LM?.writeDataPoint({
    indexes: [...],
    blobs: [...],
    doubles: [...],
});

env.TRACKING_OPENAI_LM?.writeDataPoint({
    indexes: [...],
    blobs: [...],
    doubles: [...],
});

env.TRACKING_OPENAI_LM?.writeDataPoint({
    indexes: [...],
    blobs: [...],
    doubles: [...],
});

env.TRACKING_OPENAI_LM?.writeDataPoint({
    indexes: [...],
    blobs: [...],
    doubles: [...],
});

A

AriesOP•8/22/23, 2:09 AM

Optional access operator means it would fail silenty though?

A

AriesOP•8/22/23, 2:10 AM

https://files.aries.fyi/2023/08/21/Code_ec26a151f9c753c8.png

V

Victor•8/22/23, 2:11 AM

Yea, but that only happens if running locally in wrangler for me

A

AriesOP•8/22/23, 2:11 AM

analytics is exposed but not

dataImpressions

dataImpressions

dataImpressions

dataImpressions

A

AriesOP•8/22/23, 2:12 AM

Very weird.

AAries analytics is exposed but not ``dataImpressions``

V

Victor•8/22/23, 2:14 AM

Going to

https://dash.cloudflare.com/<account id>/workers/analytics-engine

https://dash.cloudflare.com/<account id>/workers/analytics-engine

https://dash.cloudflare.com/<account id>/workers/analytics-engine

https://dash.cloudflare.com/<account id>/workers/analytics-engine, do you see stats?

A

AriesOP•8/22/23, 2:14 AM

Only the old dataset.
https://files.aries.fyi/2023/08/21/firefox_1e1881fe7a266c61.png

A

AriesOP•8/22/23, 2:16 AM

When uploading, it does not acknowledge it at all.
https://files.aries.fyi/2023/08/21/Code_9957cb0c3fb79cea.png

VVictor Going to `https://dash.cloudflare.com/<account id>/workers/analytics-engine`, do...

A

AriesOP•8/22/23, 2:17 AM

Weird..

A

AriesOP•8/22/23, 2:17 AM

And when deployed:
https://files.aries.fyi/2023/08/21/firefox_e676d010ecf6d878.png

VVictor ```toml analytics_engine_datasets = [ { binding = "TRACKING_CLOUDFLARE_REQUE...

A

AriesOP•8/22/23, 2:34 AM

Copying your syntax exactly doesn't register them, but doing it this way does: https://files.aries.fyi/2023/08/21/Code_a16a743b078b69f7.png

A

AriesOP•8/22/23, 2:35 AM

dataset not dataset_name as well, my bad.

AAries Copying your syntax exactly doesn't register them, but doing it this way does: h...

V

Victor•8/22/23, 2:36 AM

My syntax is setup for multiple environments, that probably why it's not picking up

A

AriesOP•8/22/23, 2:49 AM

Is

25 writes (writeDataPoint invocations) per client HTTP request

25 writes (writeDataPoint invocations) per client HTTP request

a hard limit?

C

chientrm.com•8/22/23, 3:49 AM

is analytics engine available on free plan?

AAries Is ``25 writes (writeDataPoint invocations) per client HTTP request`` a hard lim...

V

Victor•8/22/23, 5:16 AM

I don't know of any limits

E

Elliot Hesp•8/22/23, 3:20 PM

If I'm wrapping analytics engine in an ORM, what flavour of SQL is it using?

EElliot Hesp If I'm wrapping analytics engine in an ORM, what flavour of SQL is it using?

J

John Spurlock•8/22/23, 3:21 PM

it's a subset of clickhouse: https://clickhouse.com/docs/en/sql-reference

A

AriesOP•8/24/23, 6:24 AM

In the SQL, how would I say give me the top 100, then group the rest as "Other"?

A

AriesOP•8/24/23, 6:25 AM

https://files.aries.fyi/2023/08/24/firefox_469e36f8aac75ae2.png

AAries In the SQL, how would I say give me the top 100, then group the rest as "Other"?

D

DaniFoldi•8/24/23, 7:04 AM

In SQL, you can use a statement like below

SELECT id, ...,
CASE
  WHEN id <= 100 THEN id
  ELSE 'Other'
END AS groupedId
FROM myTable

SELECT id, ...,
CASE
  WHEN id <= 100 THEN id
  ELSE 'Other'
END AS groupedId
FROM myTable

SELECT id, ...,
CASE
  WHEN id <= 100 THEN id
  ELSE 'Other'
END AS groupedId
FROM myTable

SELECT id, ...,
CASE
  WHEN id <= 100 THEN id
  ELSE 'Other'
END AS groupedId
FROM myTable

However, I'm not sure if clickhouse/AE supports that as I haven't played around with it too much yet, so if it doesn't work you're likely best selecting the top 100, and summing the rest "manually" in subqueries

I

itsmatteomanf•8/31/23, 12:46 PM

Is there a way to get more than the default 25 ingestions per invocation? I have a queue that processes 100 items which need to push things a few things to AE per item.

I

itsmatteomanf•8/31/23, 1:58 PM

Oh, I read now... maybe I should ping @46bit

J

John Spurlock•8/31/23, 9:28 PM

having used AE from basically day 1, you definitely have to deal with pretty much every

5xx

5xx

error in the book coming back, and retry-after-delay. Most of these errors are either one-offs or transient

P

pigri•9/1/23, 7:22 AM

Hmm... anybody else who also has experience with analytics_engine is not working? My grafana dashboards are down and I can see this error:

Sorry, we were unable to evaluate your query

Sorry, we were unable to evaluate your query

Jjimh Hi @.pigri - can you post an example of a query that returns this error?

P

pigri•9/1/23, 8:29 AM

Sure,
SELECT
timestamp AS t,
*
FROM
usageDataOutput
WHERE
t >= NOW() - INTERVAL '1' DAY
ORDER BY t

P

pigri•9/1/23, 8:29 AM

2 days ago this query is worked

P

pigri•9/1/23, 9:26 AM

Thank you! We try to figure out who can fix it.

Jjimh I searched through errors on our end based on a hunch and I'm pretty sure the is...

K

Kasumi•9/1/23, 9:53 AM

Sorry for disturbing but does it mean that cf cuts the responses off as soon as the response hits 100mb or is it just analytics-engine ?

Jjimh The quickest way to reduce response size for this query will be to remove the `*...

P

pigri•9/1/23, 10:52 AM

Thank you we fixed it!

U

Unsmart•9/1/23, 3:58 PM

Any new plugin we should be using since I guess the altinity plugin thats recommended is using legacy stuff and will eventually break unless they are working on updating it maybe

I can't figure out how to use multiple datasets, and it's not mentioned anywhere.

Similar Threads

Similar Threads

Similar Threads