Why do redirect rules apply to tcp/ssh tunnels? 😅

Ddave Why do redirect rules apply to tcp/ssh tunnels? 😅

C

Cyb3r-Jak3•7/5/23, 9:20 PM

If I remember correctly it’s because tunnels aren’t really TCP unless you use private networking. They are just TCP over websocket

D

daveOP•7/5/23, 9:22 PM

Is there a generic way/expression to avoid apply a ruleset to tunnels?

Original message was deleted

D

daveOP•7/5/23, 9:24 PM

yeah, agreed

C

Cyb3r-Jak3•7/5/23, 9:24 PM

Maybe go client and websocket if you can check if a request is a websocket

D

daveOP•7/5/23, 9:24 PM

hmm, or maybe just all websockets?

M

McSneaky•7/5/23, 9:39 PM

Is it possible to have callback / event from Cloudflare when certain asset is exired in CDN or deleted from cache?
I want to make websocket connection with durable object that's sending live events to client when asset expires so client can know to reload it

MMcSneaky Is it possible to have callback / event from Cloudflare when certain asset is ex...

D

daveOP•7/5/23, 9:40 PM

afaik no but I would love that.

M

McSneaky•7/5/23, 9:43 PM

Daymn

Then I'll just have to assume it's going to be deleted about 10s after I send delete request

M

McSneaky•7/5/23, 9:45 PM

Aah, I'll just disable cache for given asset

Then there's no need to wait

M

McSneaky•7/5/23, 9:46 PM

Works for demo.. Won't work for production

M

McSneaky•7/5/23, 9:47 PM

...
As workaround. There's log stream. Are "asset x has been deleted" in logs? Or only requests are?

S

Smarter•7/5/23, 9:47 PM

Was hoping someone could clear something up for me quick.

S

Smarter•7/5/23, 9:48 PM

I have

cloudflared

cloudflared

cloudflared

cloudflared running inside a docker container, and currently my tunnel is connected and working with 1 instance of that container, the connector shows in the Dashboard.. If I scale that container out, what will happen to the requests coming to the Tunnel, how will they be routed to the multiple connected connectors...

MMcSneaky ... As workaround. There's log stream. Are "asset x has been deleted" in logs? O...

D

daveOP•7/5/23, 9:55 PM

you'll see if a cache miss happened in the log stream

MMcSneaky ... As workaround. There's log stream. Are "asset x has been deleted" in logs? O...

D

daveOP•7/5/23, 9:56 PM

copy pasted from a private channel (my own message/answer to be clear lol).

re: Cache API analytics: You can see Cache API events in logpush HTTP. e.g.:

{
  "ClientRequestSource": "edgeWorkerCacheAPI",
  "ClientIP": "2a06:98c0:3600::103",
  "ClientRequestHost": "ai.moda.invalid",
  "ClientRequestMethod": "PUT",
  "ClientRequestURI": "/get_products/1e347d6d6f008724dcc72f5b6b576c1ef6a6bd6c25443a54ee39041df34ed70eae435cc5f261655414bfa08ef50f4296dd3b13c82546136ba81e73ef4c275f70",
  "EdgeEndTimestamp": "2023-06-27T17:18:21Z",
  "EdgeResponseBytes": 355,
  "EdgeResponseStatus": 204,
  "EdgeStartTimestamp": "2023-06-27T17:18:21Z",
  "RayID": "7ddf5946a1ea006d",
  "WorkerCPUTime": 0,
  "WorkerStatus": "unknown",
  "WorkerSubrequest": true,
  "WorkerSubrequestCount": 0,
  "WorkerWallTimeUs": 0,
  "ZoneName": "ai.moda"
}

{
  "ClientRequestSource": "edgeWorkerCacheAPI",
  "ClientIP": "2a06:98c0:3600::103",
  "ClientRequestHost": "ai.moda.invalid",
  "ClientRequestMethod": "PUT",
  "ClientRequestURI": "/get_products/1e347d6d6f008724dcc72f5b6b576c1ef6a6bd6c25443a54ee39041df34ed70eae435cc5f261655414bfa08ef50f4296dd3b13c82546136ba81e73ef4c275f70",
  "EdgeEndTimestamp": "2023-06-27T17:18:21Z",
  "EdgeResponseBytes": 355,
  "EdgeResponseStatus": 204,
  "EdgeStartTimestamp": "2023-06-27T17:18:21Z",
  "RayID": "7ddf5946a1ea006d",
  "WorkerCPUTime": 0,
  "WorkerStatus": "unknown",
  "WorkerSubrequest": true,
  "WorkerSubrequestCount": 0,
  "WorkerWallTimeUs": 0,
  "ZoneName": "ai.moda"
}

{
  "ClientRequestSource": "edgeWorkerCacheAPI",
  "ClientIP": "2a06:98c0:3600::103",
  "ClientRequestHost": "ai.moda.invalid",
  "ClientRequestMethod": "PUT",
  "ClientRequestURI": "/get_products/1e347d6d6f008724dcc72f5b6b576c1ef6a6bd6c25443a54ee39041df34ed70eae435cc5f261655414bfa08ef50f4296dd3b13c82546136ba81e73ef4c275f70",
  "EdgeEndTimestamp": "2023-06-27T17:18:21Z",
  "EdgeResponseBytes": 355,
  "EdgeResponseStatus": 204,
  "EdgeStartTimestamp": "2023-06-27T17:18:21Z",
  "RayID": "7ddf5946a1ea006d",
  "WorkerCPUTime": 0,
  "WorkerStatus": "unknown",
  "WorkerSubrequest": true,
  "WorkerSubrequestCount": 0,
  "WorkerWallTimeUs": 0,
  "ZoneName": "ai.moda"
}

{
  "ClientRequestSource": "edgeWorkerCacheAPI",
  "ClientIP": "2a06:98c0:3600::103",
  "ClientRequestHost": "ai.moda.invalid",
  "ClientRequestMethod": "PUT",
  "ClientRequestURI": "/get_products/1e347d6d6f008724dcc72f5b6b576c1ef6a6bd6c25443a54ee39041df34ed70eae435cc5f261655414bfa08ef50f4296dd3b13c82546136ba81e73ef4c275f70",
  "EdgeEndTimestamp": "2023-06-27T17:18:21Z",
  "EdgeResponseBytes": 355,
  "EdgeResponseStatus": 204,
  "EdgeStartTimestamp": "2023-06-27T17:18:21Z",
  "RayID": "7ddf5946a1ea006d",
  "WorkerCPUTime": 0,
  "WorkerStatus": "unknown",
  "WorkerSubrequest": true,
  "WorkerSubrequestCount": 0,
  "WorkerWallTimeUs": 0,
  "ZoneName": "ai.moda"
}

https://developers.cloudflare.com/logs/reference/clientrequestsource/

In a reallllly roundabout way you could probably make passthrough Worker that acts as an S3 endpoint, and then use the Workers Analytics Engine to log

ClientRequestSource

ClientRequestSource

ClientRequestSource

ClientRequestSource.

Ddave you'll see if a cache miss happened in the log stream

M

McSneaky•7/5/23, 10:02 PM

Yeah, was thinking about that too, but that assumes I'm doing requests

Ddave copy pasted from a private channel (my own message/answer to be clear lol). re:...

M

McSneaky•7/5/23, 10:03 PM

Tried that one too, but it's like they say in docs. Deleting cached asset may take up to 20sec. And if I refresh instantly after cache clear request has recieved, assets will still be cached
If I wait 10s, then it's 50/50. Sometimes still have to wait 20s for all assets to actually be cleared from cache

MMcSneaky Tried that one too, but it's like they say in docs. Deleting cached asset may ta...

D

daveOP•7/5/23, 10:04 PM

https://gist.github.com/Manouchehri/8db5ebcecbbc6b286ef226cb57122b1d

Gist

delete_from_cache_globally.ts

GitHub Gist: instantly share code, notes, and snippets.

M

McSneaky•7/5/23, 10:05 PM

But cache clearing isn't instant

So refresh happens before actual cache clearing

U

Unsmart•7/5/23, 10:08 PM

cache clearing will never be instant it has to clear from hundreds of data centers

M

McSneaky•7/5/23, 10:09 PM

Yeah, I'm OK with that. If I'd get callback, when was it actually cleared from everywhere. So I'm not waiting blindly

M

McSneaky•7/5/23, 10:09 PM

Or per datacenter or however

U

Unsmart•7/5/23, 10:10 PM

Why does it matter?

U

Unsmart•7/5/23, 10:10 PM

removing the cache is just we will eventually try to clear it if we can

U

Unsmart•7/5/23, 10:11 PM

If cache is messing things up you might wanna try setting a lower cache value

M

McSneaky•7/5/23, 10:12 PM

Dealing with rather complex real-time datafeed, which needs to re-pull new JS files from CDN
Getting milions of requests per second to those JS files at peak times, so no-cache is not an option in production and it has to be cached 100% for global coverage
But at the same time, pulling down updated JS file more than 3-5sec late, is quite bad for business

M

McSneaky•7/5/23, 10:13 PM

Working around some legacy business decisions too.. Can't just directly feed data to client sadly. Had to do pile of hacks to get that somewhat working, but coz of legacy reasons and size constraints have to pull in new JS files and force refresh parent iframe

MMcSneaky Working around some legacy business decisions too.. Can't just directly feed dat...

D

daveOP•7/5/23, 10:29 PM

Do you control the client software at all?

Ddave Do you control the client software at all?

M

McSneaky•7/5/23, 10:30 PM

Jup, but it's huge thing, getting changes thro there is rather long and tedious process

M

McSneaky•7/5/23, 10:32 PM

But I think for proof-of-concept demo, I'll go cacheless and if things go well, then before production can hopefully figure out and do some changes to client software that would allow direct updating from WS connection

M

McSneaky•7/5/23, 10:38 PM

It's basically single .js file Vue application, that needs to take data from parent page, that gets it's data from live feed. Tricky part is, that so far all the data has been baked into that JS file at delivery time, so it's huge change architecturally to change it now

M

McSneaky•7/5/23, 10:40 PM

^ Tool for ad delivery networks and agencies

D

daveOP•7/5/23, 10:40 PM

Why is serving a 5-10 second out of date file so bad?

E

Erisa•7/5/23, 10:47 PM

Ordinarily you would see the .js file keyed by a hash in the filename (e.g. script.abcdef.js) and then the html page specifies which one to load, so when you purge cache it updates the html page to point to the new js but the old js stays around for those on the old html

Ddave Why is serving a 5-10 second out of date file so bad?

M

McSneaky•7/5/23, 10:47 PM

Update needs to happen within some seconds max. File itself could be older actually. But since right now only way to get data in is update file, that needs to be updated
From business side, it's bad if your betting data is 10sec late and decisions are made based on that

E

Erisa•7/5/23, 10:48 PM

For live updates you should 100% be using a websocket

EErisa Ordinarily you would see the .js file keyed by a hash in the filename (e.g. scri...

M

McSneaky•7/5/23, 10:48 PM

Aah, that's actually interesting idea. Currently hashes are rather random. But could make them kinda guessable. So script within HTML can try to "guess" next hash and ping that every x time

MMcSneaky Update needs to happen within some seconds max. File itself could be older actua...

D

daveOP•7/5/23, 10:48 PM

first question, are you using Workers already?

Ddave first question, are you using Workers already?

M

McSneaky•7/5/23, 10:49 PM

Jup, but not modifying files. Thought about that, but modifying them on that scale at delivery time would be too costly sadly

MMcSneaky Aah, that's actually interesting idea. Currently hashes are rather random. But c...

E

Erisa•7/5/23, 10:49 PM

Not quite what I had in mind but for your use case that.. might work? Maybe?

EErisa For live updates you should 100% be using a websocket

M

McSneaky•7/5/23, 10:49 PM

Yeah.. Dealing with legacy stuff.. Trying to figure out faster workarounds

EErisa Not quite what I had in mind but for your use case that.. might work? Maybe?

M

McSneaky•7/5/23, 10:50 PM

So instead of random hashes I'd have like

myfile.v1.js

myfile.v1.js

myfile.v1.js

myfile.v1.js,

myfile.v2.js

myfile.v2.js

myfile.v2.js

myfile.v2.js,

myfile.v3.js

myfile.v3.js

myfile.v3.js

myfile.v3.js etc

MMcSneaky So instead of random hashes I'd have like `myfile.v1.js`, `myfile.v2.js`, `myfil...

D

daveOP•7/5/23, 10:50 PM

have

myfile.latest.js

myfile.latest.js

myfile.latest.js

myfile.latest.js that does a 303 redirect to whatever the current version is

Ddave have `myfile.latest.js` that does a 303 redirect to whatever the current version...

M

McSneaky•7/5/23, 10:51 PM

Aah! That's real good idea!

MMcSneaky Aah! That's real good idea!

D

daveOP•7/5/23, 10:51 PM

it's not an original one, I can't take credit

MMcSneaky So instead of random hashes I'd have like `myfile.v1.js`, `myfile.v2.js`, `myfil...

E

Erisa•7/5/23, 10:51 PM

I see, yeah what dave said

E

Erisa•7/5/23, 10:51 PM

I've seen that done with some success

M

McSneaky•7/5/23, 10:52 PM

Noice. Thnx for discussions!
I can do redirects thro API all good

And that will work in production long enuf until I (or someone else) figure out how to update client side legacy code to support WS directly

E

Erisa•7/5/23, 10:53 PM

websockets are OP for live updates, its how your discord client is showing all our messages in realtime

D

daveOP•7/5/23, 10:54 PM

This is a guess:

Put these headers on the redirected

myfile.latest.js

myfile.latest.js

myfile.latest.js

myfile.latest.js:

cache-control: private, no-transform, no-cache, no-store, max-age=0, s-maxage=0

cache-control: private, no-transform, no-cache, no-store, max-age=0, s-maxage=0

cache-control: private, no-transform, no-cache, no-store, max-age=0, s-maxage=0

cache-control: private, no-transform, no-cache, no-store, max-age=0, s-maxage=0

And these headers on

myfile.v*.js

myfile.v*.js

myfile.v*.js

myfile.v*.js:

cache-control: private, immutable, no-transform, max-age=31536000, s-maxage=31536000

cache-control: private, immutable, no-transform, max-age=31536000, s-maxage=31536000

cache-control: private, immutable, no-transform, max-age=31536000, s-maxage=31536000

cache-control: private, immutable, no-transform, max-age=31536000, s-maxage=31536000

You should change

private

private

private

private to

public

public

if these files aren't custom per-user.

Why do redirect rules apply to tcp/ssh tunnels? 😅

Similar Threads

Similar Threads

Similar Threads