Switch to `standard` placement – I also had a huge uptick the last couple days in 520s on our API sp

CCole Switch to `standard` placement – I also had a huge uptick the last couple days i...

M

Moe•8/27/24, 6:26 PM

Yea i did and that fixed it, also images from our public r2 bucket randomly dont work for some users as well

MMoe Yea i did and that fixed it, also images from our public r2 bucket randomly dont...

C

ColeOP•8/27/24, 6:26 PM

is there a similar placement for R2? (I haven't used R2 at all)

D

dave•8/27/24, 6:27 PM

Any good minifiers and/or prettifers that work on HTML in a Worker?

CCole is there a similar placement for R2? (I haven't used R2 at all)

M

Moe•8/27/24, 6:27 PM

No dont think so

MMoe No dont think so

C

ColeOP•8/27/24, 6:28 PM

interesting, you may want to try disabling http3 and/or http2 to origin

.

.gwapes•8/27/24, 7:34 PM

On the dashboard, if you make a real time logs, log, does it ever timeout or if you keep the window open it'll always show logs for however long it's open for?

W

Walshy•8/27/24, 7:35 PM

it should timeout after 6 hours

W

Walshy•8/27/24, 7:36 PM

but the dash starts to break down after a thousand or so entries, keeping it open for 6 hours i fully expect the tab to crash way before then

.

.gwapes•8/27/24, 7:40 PM

Gotcha, thanks. Thought there would be some kinda "if it goes over this it'll probably crash" lol

.

.gwapes•8/27/24, 7:43 PM

Trying to add Sentry but getting this...?

.

.gwapes•8/27/24, 7:45 PM

Nvm got it

.

.gwapes•8/27/24, 8:11 PM

Can someone help me install sentry? I made a new project but when I try and add it on the dashboard it says I have no options

MMoe yo for some reason setting workers smart placement on wrecks any post request fo...

B

Brett Willis•8/27/24, 8:24 PM

I also had similar a couple of days ago, and it's back again. 100% of POST/PUT/PATCH requests against the

workers.dev

workers.dev

domain are failing with 520. Requests against custom domains seem to function normally.

B

Brett Willis•8/27/24, 8:36 PM

This is actually becoming a huge days-long downtime issue (on our paid workers plan)... is anyone from the Cloudflare team aware of it because I can't see it on the status dashboard?

BBrett Willis I also had similar a couple of days ago, and it's back again. 100% of POST/PUT/P...

C

Cyb3r-Jak3•8/27/24, 9:25 PM

I can't recreate. I have https://hello-world-billowing-brook-b77a.cyberjake.workers.dev/ and I can hit it with any method.

CCyb3r-Jak3 I can't recreate. I have https://hello-world-billowing-brook-b77a.cyberjake.work...

B

Brett Willis•8/27/24, 9:41 PM

Hmm... seems like the others reckoned it might something to do with smart placement. I see you have smart placement on too, but it is placed locally. Mine is placed in

remote-SEA

remote-SEA

.
Try

POST https://api-dev.hhc.workers.dev

POST https://api-dev.hhc.workers.dev

. That would normally give you 404 from the worker except it isn't even reaching the worker (no logs), instead 520.

BBrett Willis Hmm... seems like the others reckoned it might something to do with smart placem...

C

Chaika•8/27/24, 9:44 PM

using a specific body or something?

curl --request POST   --url https://api-dev.hhc.workers.dev/   --header 'Content-Type: application/json'   --header 'User-Agent: insomnia/2023.5.8'   --data '{}'  -vvv

curl --request POST   --url https://api-dev.hhc.workers.dev/   --header 'Content-Type: application/json'   --header 'User-Agent: insomnia/2023.5.8'   --data '{}'  -vvv

curl --request POST   --url https://api-dev.hhc.workers.dev/   --header 'Content-Type: application/json'   --header 'User-Agent: insomnia/2023.5.8'   --data '{}'  -vvv

curl --request POST   --url https://api-dev.hhc.workers.dev/   --header 'Content-Type: application/json'   --header 'User-Agent: insomnia/2023.5.8'   --data '{}'  -vvv

gives me

< HTTP/2 404
< date: Tue, 27 Aug 2024 21:44:09 GMT
< content-type: text/plain;charset=UTF-8
< content-length: 23
< cache-control: private, no-store
< cf-placement: remote-SEA
< report-to: {"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=tZaSH8jYa0BFaB%2Bmmq%2BSBFg2i9tN%2BNx3GXyaAe2cI65i0LiPX4YodMyfQKb1otEwjeh46Av5w4WE93VbpQzhmPIZlGlTEn3V7KzQiTS4XbhHlhDe8Pj5B7Kj3UChSwgMSifL4n778Xxe%2BQ%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 8b9f3dbf9ba841a6-EWR
< alt-svc: h3=":443"; ma=86400
{"message":"Not Found"}

C

Chaika•8/27/24, 9:45 PM

What exact headers do you get on the 520?

CChaika using a specific body or something? `curl --request POST --url https://api-dev...

B

Brett Willis•8/27/24, 9:47 PM

Huh yeah just got the same as you with curl but always 520 with Postman (and browsers), let me try find what the determining factor is...
No body/payload. See attached.

BBrett Willis Huh yeah just got the same as you with curl but always 520 with Postman (and bro...

C

Chaika•8/27/24, 9:50 PM

hmm you get back a response w/ 11 bytes, what's the response?

CChaika hmm you get back a response w/ 11 bytes, what's the response?

B

Brett Willis•8/27/24, 9:51 PM

Response is

Bad Gateway

Bad Gateway

.
Ah seemingly it's the presence of a

Content-Length: 0

Content-Length: 0

header:

% curl -i https://api-dev.hhc.workers.dev --header 'Content-Length: 0' 
HTTP/2 520 
date: Tue, 27 Aug 2024 21:50:25 GMT
content-type: text/plain;charset=UTF-8
content-length: 11
cf-placement: remote-SEA
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=j51afelgSe9%2FfmZXkPa9RvAJi7U2pWUpArXDewyLBkKu0sqdUNZzQfrHbo7KzZ3v1ckABzd14UCS30SF5UIObVLlOk7ATBmn5KNLT9yqPHZzJZQbAqQ7hUnKWrNZY6BtuL0xRSeyWcBQ9uXS4kqsul7hQ77DDQ%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 8b9f46efedd31c4d-AKL
alt-svc: h3=":443"; ma=86400

Bad Gateway

% curl -i https://api-dev.hhc.workers.dev --header 'Content-Length: 0' 
HTTP/2 520 
date: Tue, 27 Aug 2024 21:50:25 GMT
content-type: text/plain;charset=UTF-8
content-length: 11
cf-placement: remote-SEA
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=j51afelgSe9%2FfmZXkPa9RvAJi7U2pWUpArXDewyLBkKu0sqdUNZzQfrHbo7KzZ3v1ckABzd14UCS30SF5UIObVLlOk7ATBmn5KNLT9yqPHZzJZQbAqQ7hUnKWrNZY6BtuL0xRSeyWcBQ9uXS4kqsul7hQ77DDQ%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 8b9f46efedd31c4d-AKL
alt-svc: h3=":443"; ma=86400

Bad Gateway

% curl -i https://api-dev.hhc.workers.dev --header 'Content-Length: 0' 
HTTP/2 520 
date: Tue, 27 Aug 2024 21:50:25 GMT
content-type: text/plain;charset=UTF-8
content-length: 11
cf-placement: remote-SEA
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=j51afelgSe9%2FfmZXkPa9RvAJi7U2pWUpArXDewyLBkKu0sqdUNZzQfrHbo7KzZ3v1ckABzd14UCS30SF5UIObVLlOk7ATBmn5KNLT9yqPHZzJZQbAqQ7hUnKWrNZY6BtuL0xRSeyWcBQ9uXS4kqsul7hQ77DDQ%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 8b9f46efedd31c4d-AKL
alt-svc: h3=":443"; ma=86400

Bad Gateway

% curl -i https://api-dev.hhc.workers.dev --header 'Content-Length: 0' 
HTTP/2 520 
date: Tue, 27 Aug 2024 21:50:25 GMT
content-type: text/plain;charset=UTF-8
content-length: 11
cf-placement: remote-SEA
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=j51afelgSe9%2FfmZXkPa9RvAJi7U2pWUpArXDewyLBkKu0sqdUNZzQfrHbo7KzZ3v1ckABzd14UCS30SF5UIObVLlOk7ATBmn5KNLT9yqPHZzJZQbAqQ7hUnKWrNZY6BtuL0xRSeyWcBQ9uXS4kqsul7hQ77DDQ%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 8b9f46efedd31c4d-AKL
alt-svc: h3=":443"; ma=86400

Bad Gateway

B

Brett Willis•8/27/24, 9:52 PM

Regardless of the method. POST/PUT/PATCH method just causes the browser/postman to attach that header.

C

Chaika•8/27/24, 9:53 PM

curl -i https://api-dev.hhc.workers.dev --header 'Content-Length: 0'
HTTP/2 404
date: Tue, 27 Aug 2024 21:52:24 GMT
content-type: text/plain;charset=UTF-8
content-length: 23
cache-control: private, no-store
cf-placement: remote-SEA
report-to: {"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=mgSlH9e9mLRJNTkvw8TtayWcmmpz1mkZO1vbMMv9xFVajVdHaQDm4JvRQ6kRImJgwsvUaRpAyVjMOhuTsmfSYO5H5v8AVBe9RKV9YznggMdXwMYicrltGl8OLUz8Trn7vYFROWgikON5LA%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 8b9f49d5aa9f0cc0-EWR
alt-svc: h3=":443"; ma=86400

Maybe it's because you're hitting AKL? Do you always get 520's with that or just sometimes? The fact that it has cf-placement seems like it's getting to SEA and wouldn't be colo-specific tho

C

Chaika•8/27/24, 9:53 PM

above, were you saying it only occurs with smart placement on?

B

Brett Willis•8/27/24, 9:55 PM

Yeah just confirmed, with smart placement turned off it functions normally (404). Obviously serving from AKL in my case.

B

Brett Willis•8/27/24, 9:58 PM

It's always 520 for me, but always via AKL colo in my case. It's a staging env so don't have other locations hitting. Did you get 520 from your colo?

BBrett Willis It's always 520 for me, but always via AKL colo in my case. It's a staging env s...

C

Chaika•8/27/24, 9:59 PM

no workers-and-pages-discussions hmm

C

Chaika•8/27/24, 10:01 PM

do you have any other envs to test with? No way that it could be related to any sort of corporate mitm/firewall/etc?

B

Brett Willis•8/27/24, 10:04 PM

Ok ok, so I'm using Cloudflare WARP. With WARP off my colo is SYD, and it works fine. With WARP on, 520.

C

Chaika•8/27/24, 10:08 PM

hmm warp does weird stuff sometimes too, do you have Zero trust proxy w/ the gateway tcp/udp proxy on or just normal warp?

J

JoeDoe•8/27/24, 10:09 PM

does a sleep() function add to cpu time?

CChaika hmm warp does weird stuff sometimes too, do you have Zero trust proxy w/ the gat...

B

Brett Willis•8/27/24, 10:12 PM

Just normal WARP (personal install, default settings).
Interesting why WARP only trips over here with the

workers.dev

workers.dev

domain and not a custom domain route?
Anyway it seems more related to WARP, so sorry for over-hyping that "workers downtime issue".

JJoeDoe does a sleep() function add to cpu time?

C

Chaika•8/27/24, 10:39 PM

js sleep is just done using Promises/setTimeout which isn't cpu time

BBrett Willis Just normal WARP (personal install, default settings). Interesting why WARP only...

C

Chaika•8/27/24, 10:39 PM

I tried reproing with WARP but I can't, wonder if it's related to akl

W

Walshy•8/27/24, 10:44 PM

let me hit it from akl

B

Brett Willis•8/27/24, 10:46 PM

Yeah I was just about to say, not sure it's completely a WARP issue. The 520 responses take ~500ms which is enough time go go all the way around the world (definitely beyond my machine and local colo which is normally 10-20ms away, SYD is 30-40ms).
And it's a combination of ALL these factors:

Smart placement ON (or served by SEA, not sure how to isolate but placement OFF + SYD works fine)
workers.dev domain (custom domains are fine)
Content-Length: 0 header present
WARP ON or AKL colo (haven't been able to isolate those two just yet)

W

Walshy•8/27/24, 10:46 PM

< HTTP/2 404
< date: Tue, 27 Aug 2024 22:46:02 GMT
< content-type: text/plain;charset=UTF-8
< content-length: 23
< cache-control: private, no-store
< cf-placement: remote-SEA
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=cYbKoFuCk5Dnjg57XLc%2Bd%2F%2FZtpZOn%2FwmSg%2BvVCeXENxdBtfzpywCkmgGzjg57C3Pcf7L61v4FVSlBRx61wVe48xXFDBKN0tMegx9GZMrvGuRIkG0keTz2lT14pqi4Gbtbfa6MnqLFt3UBg%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 8b9f9867883150bf-AKL

< HTTP/2 404
< date: Tue, 27 Aug 2024 22:46:02 GMT
< content-type: text/plain;charset=UTF-8
< content-length: 23
< cache-control: private, no-store
< cf-placement: remote-SEA
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=cYbKoFuCk5Dnjg57XLc%2Bd%2F%2FZtpZOn%2FwmSg%2BvVCeXENxdBtfzpywCkmgGzjg57C3Pcf7L61v4FVSlBRx61wVe48xXFDBKN0tMegx9GZMrvGuRIkG0keTz2lT14pqi4Gbtbfa6MnqLFt3UBg%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 8b9f9867883150bf-AKL

< HTTP/2 404
< date: Tue, 27 Aug 2024 22:46:02 GMT
< content-type: text/plain;charset=UTF-8
< content-length: 23
< cache-control: private, no-store
< cf-placement: remote-SEA
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=cYbKoFuCk5Dnjg57XLc%2Bd%2F%2FZtpZOn%2FwmSg%2BvVCeXENxdBtfzpywCkmgGzjg57C3Pcf7L61v4FVSlBRx61wVe48xXFDBKN0tMegx9GZMrvGuRIkG0keTz2lT14pqi4Gbtbfa6MnqLFt3UBg%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 8b9f9867883150bf-AKL

< HTTP/2 404
< date: Tue, 27 Aug 2024 22:46:02 GMT
< content-type: text/plain;charset=UTF-8
< content-length: 23
< cache-control: private, no-store
< cf-placement: remote-SEA
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=cYbKoFuCk5Dnjg57XLc%2Bd%2F%2FZtpZOn%2FwmSg%2BvVCeXENxdBtfzpywCkmgGzjg57C3Pcf7L61v4FVSlBRx61wVe48xXFDBKN0tMegx9GZMrvGuRIkG0keTz2lT14pqi4Gbtbfa6MnqLFt3UBg%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 8b9f9867883150bf-AKL

WWalshy let me hit it from akl

C

Chaika•8/27/24, 10:46 PM

thx, he was saying it only happens when request header

Content-Length

Content-Length

:

as well

W

Walshy•8/27/24, 10:47 PM

oops

W

Walshy•8/27/24, 10:47 PM

< HTTP/2 520
< date: Tue, 27 Aug 2024 22:46:58 GMT
< content-type: text/plain;charset=UTF-8
< content-length: 11
< cf-placement: remote-SEA
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=TZj%2FYtiLyiuphFwv0vlGdJHaMT35Mdjt9I0668lEGpt27Kyp8QdBuiVZH1ntXG8OpNS8vGfGoB6xIoUTdeH%2FxIRct9fZsusTSeqtG9F2diC5g7DK6u2u4yBaPqOOAdcXJddYfYzCkjg37A%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 8b9f99c64a9f7257-AKL

< HTTP/2 520
< date: Tue, 27 Aug 2024 22:46:58 GMT
< content-type: text/plain;charset=UTF-8
< content-length: 11
< cf-placement: remote-SEA
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=TZj%2FYtiLyiuphFwv0vlGdJHaMT35Mdjt9I0668lEGpt27Kyp8QdBuiVZH1ntXG8OpNS8vGfGoB6xIoUTdeH%2FxIRct9fZsusTSeqtG9F2diC5g7DK6u2u4yBaPqOOAdcXJddYfYzCkjg37A%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 8b9f99c64a9f7257-AKL

< HTTP/2 520
< date: Tue, 27 Aug 2024 22:46:58 GMT
< content-type: text/plain;charset=UTF-8
< content-length: 11
< cf-placement: remote-SEA
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=TZj%2FYtiLyiuphFwv0vlGdJHaMT35Mdjt9I0668lEGpt27Kyp8QdBuiVZH1ntXG8OpNS8vGfGoB6xIoUTdeH%2FxIRct9fZsusTSeqtG9F2diC5g7DK6u2u4yBaPqOOAdcXJddYfYzCkjg37A%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 8b9f99c64a9f7257-AKL

< HTTP/2 520
< date: Tue, 27 Aug 2024 22:46:58 GMT
< content-type: text/plain;charset=UTF-8
< content-length: 11
< cf-placement: remote-SEA
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=TZj%2FYtiLyiuphFwv0vlGdJHaMT35Mdjt9I0668lEGpt27Kyp8QdBuiVZH1ntXG8OpNS8vGfGoB6xIoUTdeH%2FxIRct9fZsusTSeqtG9F2diC5g7DK6u2u4yBaPqOOAdcXJddYfYzCkjg37A%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 8b9f99c64a9f7257-AKL

yep hit 520

W

Walshy•8/27/24, 10:47 PM

let me take a little peek

C

Chaika•8/27/24, 10:47 PM

hmm yea weird but doesn't seem to happen from anywhere else. Curious it has the cf-placement header on the 520 as well

W

Walshy•8/27/24, 10:48 PM

oh this is weird...

W

Walshy•8/27/24, 10:49 PM

we're failing in smart placement

W

Walshy•8/27/24, 10:49 PM

let me ping the Onboarding team

W

Walshy•8/27/24, 11:15 PM

Such a weird situation haha, for now you can disable smart placement @Brett Willis

Onboarding will figure out exactly how this is going wrong and fix

WWalshy Such a weird situation haha, for now you can disable smart placement @Brett Will...

B

Brett Willis•8/27/24, 11:17 PM

Thanks for the support, let me know of any updates if you can

W

Walshy•8/27/24, 11:17 PM

will do!

M

moonchav•8/27/24, 11:25 PM

is there an easy way to see which of my worker's routes are getting requested and how much?

M

moonchav•8/27/24, 11:27 PM

or do I have to get down and dirty with workers analytics engine and log the requests myself

Switch to `standard` placement – I also had a huge uptick the last couple days in 520s on our API sp

Similar Threads

Similar Threads

Similar Threads