Loving analytics engine 🙂 When can we expect pricing? Really want to scale it up more to billions..

AAdvany Loving analytics engine 🙂 When can we expect pricing? Really want to scale it u...

M

We intend to announce public pricing by October. We want to give folks confidence to build on Analytics Engine, and recognize that pricing — or sticker shock — is a big point of friction.

A

AdvanyOP•7/20/23, 3:38 PM

2-3 months. Will there be a private annoucement before october lol

U

Unsmart•7/20/23, 3:39 PM

I mean cloudflare needs to know it first thats the private announcement

A

asp•7/21/23, 9:56 AM

Are there any performance implications of storing and filtering by a custom timestamp stored in a

double

double

double

double column compared to the

timestamp

timestamp

timestamp

timestamp column? I'm writing ingesting Logpush batches into Analytics Engine, but they are delayed by up to a couple of minutes, so I need to use the timestamp specified in each log entry.

A

asp•7/21/23, 11:13 AM

Alternatively, if you need to be very accurate, maybe you can filter using both columns? You could increase the range on the timestamp one slightly, if needed.

That sounds like the best solution for now. I know for sure the real timestamp won't be larger than the datapoint's, and, unless there's an issue with Logpush, the datapoint will be written within a couple of minutes of the event.

Jjimh It will be quite inefficient to filter on that double colum, whereas the indexin...

A

asp•7/21/23, 11:14 AM

Are there any plans to allow specifying a custom timestamp?

Jjimh It will be quite inefficient to filter on that double colum, whereas the indexin...

A

asp•7/21/23, 1:55 PM

This method seems to work well. Thanks again!

A

asp•7/21/23, 1:56 PM

Here's the query if anyone else is interested in doing this:

-- The original timestamp is stored in double12 as UNIX nano
SELECT
    (intDiv(intDiv(double12, 1000000000), $interval) * $interval) * 1000 as ts,
    sum(_sample_interval) AS events

FROM honeypot_spectrum_events

WHERE
  -- Use the indexed timestamp to retrieve rows written between $from and $to + 15 minutes
  timestamp >= toDateTime($from) AND timestamp < toDateTime($to + 900) AND
  -- Use the original timestamp to retrieve events that occurred between $from and $to
  double12 >= ($from * 1000000000) AND double12 < ($to * 1000000000)

GROUP BY ts ORDER BY ts

-- The original timestamp is stored in double12 as UNIX nano
SELECT
    (intDiv(intDiv(double12, 1000000000), $interval) * $interval) * 1000 as ts,
    sum(_sample_interval) AS events

FROM honeypot_spectrum_events

WHERE
  -- Use the indexed timestamp to retrieve rows written between $from and $to + 15 minutes
  timestamp >= toDateTime($from) AND timestamp < toDateTime($to + 900) AND
  -- Use the original timestamp to retrieve events that occurred between $from and $to
  double12 >= ($from * 1000000000) AND double12 < ($to * 1000000000)

GROUP BY ts ORDER BY ts

-- The original timestamp is stored in double12 as UNIX nano
SELECT
    (intDiv(intDiv(double12, 1000000000), $interval) * $interval) * 1000 as ts,
    sum(_sample_interval) AS events

FROM honeypot_spectrum_events

WHERE
  -- Use the indexed timestamp to retrieve rows written between $from and $to + 15 minutes
  timestamp >= toDateTime($from) AND timestamp < toDateTime($to + 900) AND
  -- Use the original timestamp to retrieve events that occurred between $from and $to
  double12 >= ($from * 1000000000) AND double12 < ($to * 1000000000)

GROUP BY ts ORDER BY ts

-- The original timestamp is stored in double12 as UNIX nano
SELECT
    (intDiv(intDiv(double12, 1000000000), $interval) * $interval) * 1000 as ts,
    sum(_sample_interval) AS events

FROM honeypot_spectrum_events

WHERE
  -- Use the indexed timestamp to retrieve rows written between $from and $to + 15 minutes
  timestamp >= toDateTime($from) AND timestamp < toDateTime($to + 900) AND
  -- Use the original timestamp to retrieve events that occurred between $from and $to
  double12 >= ($from * 1000000000) AND double12 < ($to * 1000000000)

GROUP BY ts ORDER BY ts

A

asp•7/21/23, 4:35 PM

Any Grafana experts know how I would limit the graph to only display top 10 ports in order of total events during the selected period?

-- The original timestamp is stored in double12 as UNIX nano
SELECT
    (intDiv(intDiv(double12, 1000000000), $interval) * $interval) * 1000 as ts,
    double9 as originPort,
    sum(_sample_interval) AS events

FROM honeypot_spectrum_events

WHERE
  -- Use the indexed timestamp to retrieve rows written between $from and $to + 1 day
  timestamp >= toDateTime($from) AND timestamp <= toDateTime($to + 86400) AND
  -- Use the original timestamp to retrieve events that occurred between $from and $to
  double12 >= ($from * 1000000000) AND double12 <= ($to * 1000000000)

GROUP BY ts, originPort ORDER BY ts

-- The original timestamp is stored in double12 as UNIX nano
SELECT
    (intDiv(intDiv(double12, 1000000000), $interval) * $interval) * 1000 as ts,
    double9 as originPort,
    sum(_sample_interval) AS events

FROM honeypot_spectrum_events

WHERE
  -- Use the indexed timestamp to retrieve rows written between $from and $to + 1 day
  timestamp >= toDateTime($from) AND timestamp <= toDateTime($to + 86400) AND
  -- Use the original timestamp to retrieve events that occurred between $from and $to
  double12 >= ($from * 1000000000) AND double12 <= ($to * 1000000000)

GROUP BY ts, originPort ORDER BY ts

-- The original timestamp is stored in double12 as UNIX nano
SELECT
    (intDiv(intDiv(double12, 1000000000), $interval) * $interval) * 1000 as ts,
    double9 as originPort,
    sum(_sample_interval) AS events

FROM honeypot_spectrum_events

WHERE
  -- Use the indexed timestamp to retrieve rows written between $from and $to + 1 day
  timestamp >= toDateTime($from) AND timestamp <= toDateTime($to + 86400) AND
  -- Use the original timestamp to retrieve events that occurred between $from and $to
  double12 >= ($from * 1000000000) AND double12 <= ($to * 1000000000)

GROUP BY ts, originPort ORDER BY ts

-- The original timestamp is stored in double12 as UNIX nano
SELECT
    (intDiv(intDiv(double12, 1000000000), $interval) * $interval) * 1000 as ts,
    double9 as originPort,
    sum(_sample_interval) AS events

FROM honeypot_spectrum_events

WHERE
  -- Use the indexed timestamp to retrieve rows written between $from and $to + 1 day
  timestamp >= toDateTime($from) AND timestamp <= toDateTime($to + 86400) AND
  -- Use the original timestamp to retrieve events that occurred between $from and $to
  double12 >= ($from * 1000000000) AND double12 <= ($to * 1000000000)

GROUP BY ts, originPort ORDER BY ts

Jjimh I have not found a good way to do this. I guess we'd need support for `WHERE ......

A

asp•7/21/23, 4:48 PM

good way

Is there a bad way to do it?

A

asp•7/22/23, 8:51 AM

Is there a way to convert a double to a blob? I want to create a pie chart of top ports, which are stored as doubles because the blobs are filled up with other stuff, but Grafana doesn't like using numbers as labels.

A

asp•7/22/23, 9:01 AM

If I select a

blob

blob

blob

blob it works as expected (image #1). But a

double

double

double

double breaks the labelling (image #2).

Mmiki / 46bit You can use the `format` function https://developers.cloudflare.com/analytics/an...

A

asp•7/22/23, 2:50 PM

That worked, thanks

J

John Spurlock•7/25/23, 9:41 PM

has anyone gotten grafana alerting working through the Altinity plugin for ClickHouse datasource? The token saved as the custom header in the datasource is seemingly not used by the grafana alert engine (!), so all queries fail auth

Kkrsr is the altinity plugin the preferred one to use? I'm just now setting up grafana...

J

John Spurlock•7/26/23, 5:13 PM

https://developers.cloudflare.com/analytics/analytics-engine/grafana/

We recommend the use of the Altinity plugin for Clickhouse for querying Workers Analytics Engine from Grafana.

J

John Spurlock•7/26/23, 5:15 PM

anyway guess the alerting issue might be a bug in the plugin, they are looking into it: https://github.com/Altinity/clickhouse-grafana/issues/483

T

thomas_sc•7/28/23, 12:47 PM

As of yesterday, I can no longer read from Analytics API (api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/analytics_engine/sql) from a Pages app on production domain. The API returns error 500. When I deploy the app on .pages.dev domain it works fine with exactly the same settings (token, url, accountID). It also works without errors on my local server.
Does anyone have an idea what this could be?

E

Erisa•7/28/23, 1:50 PM

A note was added to https://www.cloudflarestatus.com/incidents/03vm5mx76pc6

Cloudflare API in Pages Functions may also have been affected returning intermittent 500 errors.

A fix has been implemented and we are monitoring the results.

Thanks for the report @thomas_sc

EErisa A note was added to <https://www.cloudflarestatus.com/incidents/03vm5mx76pc6> > ...

T

thomas_sc•7/28/23, 2:40 PM

whow, thank you for quick response!

Jjimh @TomAnalyticsEngine has a decent way of doing this. You make a grafana "query" v...

J

John Spurlock•7/28/23, 2:49 PM

y'all have

HAVING

HAVING

HAVING

HAVING on the roadmap?

J

John Spurlock•7/30/23, 7:58 PM

AE team: small request to have your custom parser handle and strip embedded comments in queries. Grafana/plugin tacks on

 FORMAT JSON /* alerts query */;

FORMAT JSON /* alerts query */;

 FORMAT JSON /* alerts query */;

FORMAT JSON /* alerts query */; to each query in the alerts engine, and right now AE can't handle it. Thanks!

Mmiki / 46bit thanks for the heads up, I'll look into it

J

John Spurlock•8/1/23, 2:49 PM

thanks! In the meantime, I added a new https://denoflare.dev top-level subcommand:

denoflare ae-proxy

denoflare ae-proxy

denoflare ae-proxy

denoflare ae-proxy that can be run on a local machine and be a more compatible endpoint for the current grafana plugin situation. Install denoflare from latest commit, or to run without any installation:

deno run -A https://raw.githubusercontent.com/skymethod/denoflare/6a5c6813715cd66016d584e98a0d3ad7d335236d/cli/cli.ts ae-proxy --port 8123 --basic myuser:mypass --account-id CFACCOUNTID --api-token CFAPITOKEN

deno run -A https://raw.githubusercontent.com/skymethod/denoflare/6a5c6813715cd66016d584e98a0d3ad7d335236d/cli/cli.ts ae-proxy --port 8123 --basic myuser:mypass --account-id CFACCOUNTID --api-token CFAPITOKEN

deno run -A https://raw.githubusercontent.com/skymethod/denoflare/6a5c6813715cd66016d584e98a0d3ad7d335236d/cli/cli.ts ae-proxy --port 8123 --basic myuser:mypass --account-id CFACCOUNTID --api-token CFAPITOKEN

deno run -A https://raw.githubusercontent.com/skymethod/denoflare/6a5c6813715cd66016d584e98a0d3ad7d335236d/cli/cli.ts ae-proxy --port 8123 --basic myuser:mypass --account-id CFACCOUNTID --api-token CFAPITOKEN

It can be protected with basic auth, since basic auth is sent by grafana for both queries and alerts. Also strips inline comments before forwarding to AE and performs automatic retry on the 502s that AE sometimes returns.

M

mindfulplays•8/7/23, 11:41 PM

hiya, what's the

env.ANALYTICS_DATASET

env.ANALYTICS_DATASET

env.ANALYTICS_DATASET

env.ANALYTICS_DATASET type? The docs don't specify this, trying to get to a compile-able TS code for my corresponding

wrangler.toml

wrangler.toml

changes:
export interface Env {
__STATIC_CONTENT: KVNamespace;
XYZ_PACKAGES: R2Bucket;
XYZ_ANALYTICS: ??; // <<- What goes here?
DEBUG: boolean;
}

M

mindfulplays•8/7/23, 11:42 PM

Spelunking the TS code, I think it is

AnalyticsEngineDataset

AnalyticsEngineDataset

AnalyticsEngineDataset

AnalyticsEngineDataset but want to be sure...

Mmindfulplays hiya, what's the `env.ANALYTICS_DATASET` type? The docs don't specify this, tryi...

J

John Spurlock•8/8/23, 12:27 AM

here's what I use: https://github.com/skymethod/denoflare/blob/8679e7e191a979ecbfcc843eda4a2e36e94a4d96/common/cloudflare_workers_types.d.ts#L1212

B

brian•8/8/23, 11:56 PM

is it possible to access web traffic data (the server-side product for proxied sites) via an API? it would be nice if a worker could collect the data every 30 days before it gets dumped

K

kian•8/8/23, 11:57 PM

Everything you see in the dashboard is just an API call - usually to the GraphQL API.

K

kian•8/8/23, 11:58 PM

https://developers.cloudflare.com/analytics/graphql-api/

A

AsyncBanana•8/18/23, 3:13 PM

When I try to run my Cloudflare Pages site locally with Analytics Engine bindings, I keep getting an error saying they do not exist. Here is my wrangler.toml:

name = "[REDACTED]"
main = "dist/main.js"
compatibility_date = "7/25/2023"

workers_dev = false

analytics_engine_datasets = [
    { binding = "VIEW_ANALYTICS", dataset = "[REDACTED]_DEV_VIEW_ANALYTICS" },
    { binding = "COMMENT_ANALYTICS", dataset = "[REDACTED]_DEV_COMMENT_ANALYTICS" },
]

name = "[REDACTED]"
main = "dist/main.js"
compatibility_date = "7/25/2023"

workers_dev = false

analytics_engine_datasets = [
    { binding = "VIEW_ANALYTICS", dataset = "[REDACTED]_DEV_VIEW_ANALYTICS" },
    { binding = "COMMENT_ANALYTICS", dataset = "[REDACTED]_DEV_COMMENT_ANALYTICS" },
]

name = "[REDACTED]"
main = "dist/main.js"
compatibility_date = "7/25/2023"

workers_dev = false

analytics_engine_datasets = [
    { binding = "VIEW_ANALYTICS", dataset = "[REDACTED]_DEV_VIEW_ANALYTICS" },
    { binding = "COMMENT_ANALYTICS", dataset = "[REDACTED]_DEV_COMMENT_ANALYTICS" },
]

name = "[REDACTED]"
main = "dist/main.js"
compatibility_date = "7/25/2023"

workers_dev = false

analytics_engine_datasets = [
    { binding = "VIEW_ANALYTICS", dataset = "[REDACTED]_DEV_VIEW_ANALYTICS" },
    { binding = "COMMENT_ANALYTICS", dataset = "[REDACTED]_DEV_COMMENT_ANALYTICS" },
]

The bindings show up when wrangler states "your worker has access to the following bindings," but I can't access them in my runtime code. However, secret bindings are still working in my code. If it is any help, my website is built with Astro using the cloudflare adapter's runtime access methods

Mmiki / 46bit AE isn't supported locally yet, sorry

A

AsyncBanana•8/19/23, 12:20 AM

Thanks! Is there any way I can test cloudflare pages websites remotely like with the

--remote

--remote

--remote

--remote flag for workers?

A

AsyncBanana•8/19/23, 12:25 AM

also, for some reason I cannot seem to set analytics engine bindings within specific environments. Is this a general issue, or am I just not configuring it correctly?

U

Unsmart•8/19/23, 12:27 AM

Cloudflare pages doesn’t use a wrangler.toml and you have to pass options in the command so thats probably why you don’t see anything working

Loving analytics engine 🙂 When can we expect pricing? Really want to scale it up more to billions..

Similar Threads

Similar Threads

Similar Threads