Cinteresting, you may want to try disabling http3 and/or http2 to origin
.On the dashboard, if you make a real time logs, log, does it ever timeout or if you keep the window open it'll always show logs for however long it's open for?
Wit should timeout after 6 hours
Wbut the dash starts to break down after a thousand or so entries, keeping it open for 6 hours i fully expect the tab to crash way before then
.Gotcha, thanks. Thought there would be some kinda "if it goes over this it'll probably crash" lol
.Trying to add Sentry but getting this...?
.Nvm got it
.Can someone help me install sentry? I made a new project but when I try and add it on the dashboard it says I have no options
BI also had similar a couple of days ago, and it's back again. 100% of POST/PUT/PATCH requests against the
workers.dev domain are failing with 520. Requests against custom domains seem to function normally.BThis is actually becoming a huge days-long downtime issue (on our paid workers plan)... is anyone from the Cloudflare team aware of it because I can't see it on the status dashboard?
CI can't recreate. I have https://hello-world-billowing-brook-b77a.cyberjake.workers.dev/ and I can hit it with any method.
BHmm... seems like the others reckoned it might something to do with smart placement. I see you have smart placement on too, but it is placed locally. Mine is placed in
Try
remote-SEA.Try
POST https://api-dev.hhc.workers.dev. That would normally give you 404 from the worker except it isn't even reaching the worker (no logs), instead 520.
Cusing a specific body or something?
gives me
curl --request POST --url https://api-dev.hhc.workers.dev/ --header 'Content-Type: application/json' --header 'User-Agent: insomnia/2023.5.8' --data '{}' -vvvgives me
< HTTP/2 404
< date: Tue, 27 Aug 2024 21:44:09 GMT
< content-type: text/plain;charset=UTF-8
< content-length: 23
< cache-control: private, no-store
< cf-placement: remote-SEA
< report-to: {"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=tZaSH8jYa0BFaB%2Bmmq%2BSBFg2i9tN%2BNx3GXyaAe2cI65i0LiPX4YodMyfQKb1otEwjeh46Av5w4WE93VbpQzhmPIZlGlTEn3V7KzQiTS4XbhHlhDe8Pj5B7Kj3UChSwgMSifL4n778Xxe%2BQ%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 8b9f3dbf9ba841a6-EWR
< alt-svc: h3=":443"; ma=86400
{"message":"Not Found"}
CWhat exact headers do you get on the 520?
BHuh yeah just got the same as you with curl but always 520 with Postman (and browsers), let me try find what the determining factor is...
No body/payload. See attached.
No body/payload. See attached.
Chmm you get back a response w/ 11 bytes, what's the response?
BResponse is
Ah seemingly it's the presence of a
Bad Gateway.Ah seemingly it's the presence of a
Content-Length: 0 header:BRegardless of the method. POST/PUT/PATCH method just causes the browser/postman to attach that header.
Ccurl -i https://api-dev.hhc.workers.dev --header 'Content-Length: 0'Maybe it's because you're hitting AKL? Do you always get 520's with that or just sometimes? The fact that it has cf-placement seems like it's getting to SEA and wouldn't be colo-specific tho
HTTP/2 404
date: Tue, 27 Aug 2024 21:52:24 GMT
content-type: text/plain;charset=UTF-8
content-length: 23
cache-control: private, no-store
cf-placement: remote-SEA
report-to: {"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=mgSlH9e9mLRJNTkvw8TtayWcmmpz1mkZO1vbMMv9xFVajVdHaQDm4JvRQ6kRImJgwsvUaRpAyVjMOhuTsmfSYO5H5v8AVBe9RKV9YznggMdXwMYicrltGl8OLUz8Trn7vYFROWgikON5LA%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 8b9f49d5aa9f0cc0-EWR
alt-svc: h3=":443"; ma=86400
Cabove, were you saying it only occurs with smart placement on?
BYeah just confirmed, with smart placement turned off it functions normally (404). Obviously serving from AKL in my case.
BIt's always 520 for me, but always via AKL colo in my case. It's a staging env so don't have other locations hitting. Did you get 520 from your colo?
Cdo you have any other envs to test with? No way that it could be related to any sort of corporate mitm/firewall/etc?
BOk ok, so I'm using Cloudflare WARP. With WARP off my colo is SYD, and it works fine. With WARP on, 520.
Chmm warp does weird stuff sometimes too, do you have Zero trust proxy w/ the gateway tcp/udp proxy on or just normal warp?
Jdoes a sleep() function add to cpu time?
BJust normal WARP (personal install, default settings).
Interesting why WARP only trips over here with the
Anyway it seems more related to WARP, so sorry for over-hyping that "workers downtime issue".
Interesting why WARP only trips over here with the
workers.dev domain and not a custom domain route?Anyway it seems more related to WARP, so sorry for over-hyping that "workers downtime issue".
Cjs sleep is just done using Promises/setTimeout which isn't cpu time
CI tried reproing with WARP but I can't, wonder if it's related to akl
Wlet me hit it from akl
BYeah I was just about to say, not sure it's completely a WARP issue. The 520 responses take ~500ms which is enough time go go all the way around the world (definitely beyond my machine and local colo which is normally 10-20ms away, SYD is 30-40ms).
And it's a combination of ALL these factors:
And it's a combination of ALL these factors:
- Smart placement ON (or served by SEA, not sure how to isolate but placement OFF + SYD works fine)
- workers.dev domain (custom domains are fine)
- Content-Length: 0 header present
- WARP ON or AKL colo (haven't been able to isolate those two just yet)
W
Cthx, he was saying it only happens when request header
Content-Length: 0 as wellWoops
Wyep hit 520
Wlet me take a little peek
Chmm yea weird but doesn't seem to happen from anywhere else. Curious it has the cf-placement header on the 520 as well
Woh this is weird...
Wwe're failing in smart placement
Wlet me ping the Onboarding team
WSuch a weird situation haha, for now you can disable smart placement @Brett Willis
Onboarding will figure out exactly how this is going wrong and fix
Onboarding will figure out exactly how this is going wrong and fix
BThanks for the support, let me know of any updates if you can 
Wwill do!
Mis there an easy way to see which of my worker's routes are getting requested and how much?
Mor do I have to get down and dirty with workers analytics engine and log the requests myself 
Wthere is not, AE would indeed be the best path
Wor you could look at zone analytics but there's no Workers path, you'd need to do your own filtering there
Mah actually the zone analytics pretty much does the trick for me with some filtering - thank you!
Ewhat happens if you have a cron job running every minute and another running every five minutes? do they merge into one? is it two separate calls?
