I want to ingest events in a browser to analytics engine. Is there a better way than calling a worke

AAdvany I want to ingest events in a browser to analytics engine. Is there a better way ...

J

John Spurlock•6/30/23, 10:04 PM

depending on how long the browser session is, you many want to open a websocket and reuse it

JJohn Spurlock depending on how long the browser session is, you many want to open a websocket ...

A

AdvanyOP•6/30/23, 10:35 PM

That would based on unbound and wall clock time right?

Would that be cheaper then using just a unbound worker for each event?

AAdvany That would based on unbound and wall clock time right? Would that be cheaper th...

J

John Spurlock•6/30/23, 10:42 PM

dunno, that just for efficiently sending them asap, if you are concerned about costs, batching em up would probably be the best option

A

AdvanyOP•7/5/23, 1:00 PM

I can't seem to access the analytics engine inside hono. Does anyone know how to do that?

I tried c.env.DOMAIN_ANALYTICS.writeDataPoint(dataPoint);

I have this in wrangler

[[analytics_engine_datasets]]
binding = "DOMAIN_ANALYTICS"
dataset = "DOMAIN_ANALYTICS"

A

AdvanyOP•7/20/23, 11:46 AM

Loving analytics engine

When can we expect pricing? Really want to scale it up more to billions...

AAdvany Loving analytics engine 🙂 When can we expect pricing? Really want to scale it u...

M

Matt Silverlock•7/20/23, 3:06 PM

We intend to announce public pricing by October. We want to give folks confidence to build on Analytics Engine, and recognize that pricing — or sticker shock — is a big point of friction.

A

AdvanyOP•7/20/23, 3:38 PM

2-3 months. Will there be a private annoucement before october lol

U

Unsmart•7/20/23, 3:39 PM

I mean cloudflare needs to know it first thats the private announcement

A

asp•7/21/23, 9:56 AM

Are there any performance implications of storing and filtering by a custom timestamp stored in a

double

double

double

double column compared to the

timestamp

timestamp

timestamp

timestamp column? I'm writing ingesting Logpush batches into Analytics Engine, but they are delayed by up to a couple of minutes, so I need to use the timestamp specified in each log entry.

A

asp•7/21/23, 11:13 AM

Alternatively, if you need to be very accurate, maybe you can filter using both columns? You could increase the range on the timestamp one slightly, if needed.

That sounds like the best solution for now. I know for sure the real timestamp won't be larger than the datapoint's, and, unless there's an issue with Logpush, the datapoint will be written within a couple of minutes of the event.

Jjimh It will be quite inefficient to filter on that double colum, whereas the indexin...

A

asp•7/21/23, 11:14 AM

Are there any plans to allow specifying a custom timestamp?

Jjimh It will be quite inefficient to filter on that double colum, whereas the indexin...

A

asp•7/21/23, 1:55 PM

This method seems to work well. Thanks again!

A

asp•7/21/23, 1:56 PM

Here's the query if anyone else is interested in doing this:

-- The original timestamp is stored in double12 as UNIX nano
SELECT
    (intDiv(intDiv(double12, 1000000000), $interval) * $interval) * 1000 as ts,
    sum(_sample_interval) AS events

FROM honeypot_spectrum_events

WHERE
  -- Use the indexed timestamp to retrieve rows written between $from and $to + 15 minutes
  timestamp >= toDateTime($from) AND timestamp < toDateTime($to + 900) AND
  -- Use the original timestamp to retrieve events that occurred between $from and $to
  double12 >= ($from * 1000000000) AND double12 < ($to * 1000000000)

GROUP BY ts ORDER BY ts

-- The original timestamp is stored in double12 as UNIX nano
SELECT
    (intDiv(intDiv(double12, 1000000000), $interval) * $interval) * 1000 as ts,
    sum(_sample_interval) AS events

FROM honeypot_spectrum_events

WHERE
  -- Use the indexed timestamp to retrieve rows written between $from and $to + 15 minutes
  timestamp >= toDateTime($from) AND timestamp < toDateTime($to + 900) AND
  -- Use the original timestamp to retrieve events that occurred between $from and $to
  double12 >= ($from * 1000000000) AND double12 < ($to * 1000000000)

GROUP BY ts ORDER BY ts

-- The original timestamp is stored in double12 as UNIX nano
SELECT
    (intDiv(intDiv(double12, 1000000000), $interval) * $interval) * 1000 as ts,
    sum(_sample_interval) AS events

FROM honeypot_spectrum_events

WHERE
  -- Use the indexed timestamp to retrieve rows written between $from and $to + 15 minutes
  timestamp >= toDateTime($from) AND timestamp < toDateTime($to + 900) AND
  -- Use the original timestamp to retrieve events that occurred between $from and $to
  double12 >= ($from * 1000000000) AND double12 < ($to * 1000000000)

GROUP BY ts ORDER BY ts

-- The original timestamp is stored in double12 as UNIX nano
SELECT
    (intDiv(intDiv(double12, 1000000000), $interval) * $interval) * 1000 as ts,
    sum(_sample_interval) AS events

FROM honeypot_spectrum_events

WHERE
  -- Use the indexed timestamp to retrieve rows written between $from and $to + 15 minutes
  timestamp >= toDateTime($from) AND timestamp < toDateTime($to + 900) AND
  -- Use the original timestamp to retrieve events that occurred between $from and $to
  double12 >= ($from * 1000000000) AND double12 < ($to * 1000000000)

GROUP BY ts ORDER BY ts

A

asp•7/21/23, 4:35 PM

Any Grafana experts know how I would limit the graph to only display top 10 ports in order of total events during the selected period?

-- The original timestamp is stored in double12 as UNIX nano
SELECT
    (intDiv(intDiv(double12, 1000000000), $interval) * $interval) * 1000 as ts,
    double9 as originPort,
    sum(_sample_interval) AS events

FROM honeypot_spectrum_events

WHERE
  -- Use the indexed timestamp to retrieve rows written between $from and $to + 1 day
  timestamp >= toDateTime($from) AND timestamp <= toDateTime($to + 86400) AND
  -- Use the original timestamp to retrieve events that occurred between $from and $to
  double12 >= ($from * 1000000000) AND double12 <= ($to * 1000000000)

GROUP BY ts, originPort ORDER BY ts

-- The original timestamp is stored in double12 as UNIX nano
SELECT
    (intDiv(intDiv(double12, 1000000000), $interval) * $interval) * 1000 as ts,
    double9 as originPort,
    sum(_sample_interval) AS events

FROM honeypot_spectrum_events

WHERE
  -- Use the indexed timestamp to retrieve rows written between $from and $to + 1 day
  timestamp >= toDateTime($from) AND timestamp <= toDateTime($to + 86400) AND
  -- Use the original timestamp to retrieve events that occurred between $from and $to
  double12 >= ($from * 1000000000) AND double12 <= ($to * 1000000000)

GROUP BY ts, originPort ORDER BY ts

-- The original timestamp is stored in double12 as UNIX nano
SELECT
    (intDiv(intDiv(double12, 1000000000), $interval) * $interval) * 1000 as ts,
    double9 as originPort,
    sum(_sample_interval) AS events

FROM honeypot_spectrum_events

WHERE
  -- Use the indexed timestamp to retrieve rows written between $from and $to + 1 day
  timestamp >= toDateTime($from) AND timestamp <= toDateTime($to + 86400) AND
  -- Use the original timestamp to retrieve events that occurred between $from and $to
  double12 >= ($from * 1000000000) AND double12 <= ($to * 1000000000)

GROUP BY ts, originPort ORDER BY ts

-- The original timestamp is stored in double12 as UNIX nano
SELECT
    (intDiv(intDiv(double12, 1000000000), $interval) * $interval) * 1000 as ts,
    double9 as originPort,
    sum(_sample_interval) AS events

FROM honeypot_spectrum_events

WHERE
  -- Use the indexed timestamp to retrieve rows written between $from and $to + 1 day
  timestamp >= toDateTime($from) AND timestamp <= toDateTime($to + 86400) AND
  -- Use the original timestamp to retrieve events that occurred between $from and $to
  double12 >= ($from * 1000000000) AND double12 <= ($to * 1000000000)

GROUP BY ts, originPort ORDER BY ts

Jjimh I have not found a good way to do this. I guess we'd need support for `WHERE ......

A

asp•7/21/23, 4:48 PM

good way

Is there a bad way to do it?

A

asp•7/22/23, 8:51 AM

Is there a way to convert a double to a blob? I want to create a pie chart of top ports, which are stored as doubles because the blobs are filled up with other stuff, but Grafana doesn't like using numbers as labels.

A

asp•7/22/23, 9:01 AM

If I select a

blob

blob

blob

blob it works as expected (image #1). But a

double

double

double

double breaks the labelling (image #2).

Mmiki / 46bit You can use the `format` function https://developers.cloudflare.com/analytics/an...

A

asp•7/22/23, 2:50 PM

That worked, thanks

J

John Spurlock•7/25/23, 9:41 PM

has anyone gotten grafana alerting working through the Altinity plugin for ClickHouse datasource? The token saved as the custom header in the datasource is seemingly not used by the grafana alert engine (!), so all queries fail auth

Kkrsr is the altinity plugin the preferred one to use? I'm just now setting up grafana...

J

John Spurlock•7/26/23, 5:13 PM

https://developers.cloudflare.com/analytics/analytics-engine/grafana/

We recommend the use of the Altinity plugin for Clickhouse for querying Workers Analytics Engine from Grafana.

J

John Spurlock•7/26/23, 5:15 PM

anyway guess the alerting issue might be a bug in the plugin, they are looking into it: https://github.com/Altinity/clickhouse-grafana/issues/483

T

thomas_sc•7/28/23, 12:47 PM

As of yesterday, I can no longer read from Analytics API (api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/analytics_engine/sql) from a Pages app on production domain. The API returns error 500. When I deploy the app on .pages.dev domain it works fine with exactly the same settings (token, url, accountID). It also works without errors on my local server.
Does anyone have an idea what this could be?

I want to ingest events in a browser to analytics engine. Is there a better way than calling a worke

Similar Threads

Similar Threads

Similar Threads