fetch behaviour doesn't match curl or fetch in node/browser

Sending the following signed request fails with 403 from a Worker but succeeds in the browser and node repl. (Note the example is time sensitive and will soon time out for all platforms.)
export default {
async fetch(request, env, ctx) {
return await fetch("https://flowkey.oss-cn-shanghai.aliyuncs.com/1051-j-cole-mini_2017-03-24_10-03-72.jpg", { headers: Object.fromEntries([["authorization","OSS4-HMAC-SHA256 Credential=LTAI5tHz4CPSsjRiV9qzEfZb/20241129/cn-shanghai/oss/aliyun_v4_request,Signature=4beb9ea827ac8143e2e4e2d56b8dfdc8c2610381bc8744da053aee0c3ed92a64"],["x-oss-content-sha256","UNSIGNED-PAYLOAD"],["x-oss-date","20241129T011747Z"]]), method: "HEAD"});
}
}
export default {
async fetch(request, env, ctx) {
return await fetch("https://flowkey.oss-cn-shanghai.aliyuncs.com/1051-j-cole-mini_2017-03-24_10-03-72.jpg", { headers: Object.fromEntries([["authorization","OSS4-HMAC-SHA256 Credential=LTAI5tHz4CPSsjRiV9qzEfZb/20241129/cn-shanghai/oss/aliyun_v4_request,Signature=4beb9ea827ac8143e2e4e2d56b8dfdc8c2610381bc8744da053aee0c3ed92a64"],["x-oss-content-sha256","UNSIGNED-PAYLOAD"],["x-oss-date","20241129T011747Z"]]), method: "HEAD"});
}
}
Why? How do I fix it? The server has a very permissive CORS policy (enabling all domains). Is there a way to debug which headers Cloudflare actually sends in the fetch request? The network panel in the Workers code console shows that the correct headers should be sent, but then again there is a big notice saying "Provisional headers are shown.", so it's hard to say for sure.
22 Replies
Walshy
Walshy2w ago
403 likely means it just being blocked as being from a bot or because they're maybe blocking Workers What's the response body? May give more indication
ephemer
ephemerOP2w ago
403 in this case signifies an incorrect signature – at least, I've seen a lot of 403s developing this feature and it's always been an incorrect signature. I can't seem to access any response body, which again makes me think it's something CORS related (on the Workers end) but it's hard to tell I've deployed the above Worker to test it and I get the following CURL response from it:
< HTTP/2 403
< date: Fri, 29 Nov 2024 01:22:23 GMT
< content-type: application/xml
< content-length: 0
< cf-ray: 8e9ec9506d461cbf-FRA
< cf-cache-status: BYPASS
< x-oss-ec: 0002-00000201
< x-oss-request-id: 6749174F4EA6A23136824C91
< x-oss-server-time: 1
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=z%2F9SzQEHONaRWakHvD0yqK%2Ff2i0mppH42pm0h5E89%2FGiB3mz%2FLqCihjJLZU5A3BEQKNplWM7yS68F%2F0HmMx%2BV%2BMy%2B64639P%2FCNyTiKKD6qwtL%2BJrR%2BmID1CSpzp4w6hr7NxH3bBw1UOQP9gWWXo9AzOCyQ6aS9c%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< alt-svc: h3=":443"; ma=86400
< server-timing: cfL4;desc="?proto=TCP&rtt=23175&min_rtt=18126&rtt_var=7631&sent=7&recv=11&lost=0&retrans=0&sent_bytes=2915&recv_bytes=584&delivery_rate=184699&cwnd=254&unsent_bytes=0&cid=d8139cabe0a025a4&ts=445&x=0"
<
< HTTP/2 403
< date: Fri, 29 Nov 2024 01:22:23 GMT
< content-type: application/xml
< content-length: 0
< cf-ray: 8e9ec9506d461cbf-FRA
< cf-cache-status: BYPASS
< x-oss-ec: 0002-00000201
< x-oss-request-id: 6749174F4EA6A23136824C91
< x-oss-server-time: 1
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=z%2F9SzQEHONaRWakHvD0yqK%2Ff2i0mppH42pm0h5E89%2FGiB3mz%2FLqCihjJLZU5A3BEQKNplWM7yS68F%2F0HmMx%2BV%2BMy%2B64639P%2FCNyTiKKD6qwtL%2BJrR%2BmID1CSpzp4w6hr7NxH3bBw1UOQP9gWWXo9AzOCyQ6aS9c%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< alt-svc: h3=":443"; ma=86400
< server-timing: cfL4;desc="?proto=TCP&rtt=23175&min_rtt=18126&rtt_var=7631&sent=7&recv=11&lost=0&retrans=0&sent_bytes=2915&recv_bytes=584&delivery_rate=184699&cwnd=254&unsent_bytes=0&cid=d8139cabe0a025a4&ts=445&x=0"
<
so there is no body in this case (which is different to the normal 403 case from this service – there I'd get some encoded XML with an error message, which is hinted at by the application/xml mime type) It's possible fetch is truncating the body because it's a HEAD request I tested this again with a GET request. There the signature succeeds in Workers the same as it does on other platforms. This means the server is not blocking Workers generally. It also means I can't test the theory that the error is being truncated only in HEAD requests. Maybe GET request errors are not truncated, but they just work. Is it possible Workers does not pass all headers for HEAD requests?
Walshy
Walshy2w ago
Nah we don't discriminate on method It'll just always pass all headers
ephemer
ephemerOP2w ago
I wonder how it is possible that only HEAD requests don't work in Workers but if I copy/paste the same fetch statement (including method) to another platform it just works
Walshy
Walshy2w ago
Interesting it works for GET and not HEAD
ephemer
ephemerOP2w ago
Can you give me any tips on how to debug this further? I've already been at it for about 4 hours and I'm out of ideas. I can't seem to get any more information out of the Workers dev environment – the Network tab doesn't show me any details for the failing requests. Logging the result there doesn't seem to help either – I get the same nondescript information I pasted above.
Walshy
Walshy2w ago
error is
Walshy
Walshy2w ago
though that may be due to it beign expired at this point
ephemer
ephemerOP2w ago
You probably get that locally now because the signature has expired I'll give you a new one that will work for another 10-15mins fetch("https://flowkey.oss-cn-shanghai.aliyuncs.com/1051-j-cole-mini_2017-03-24_10-03-72.jpg" , { headers: Object.fromEntries([["authorization","OSS4-HMAC-SHA256 Credential=LTAI5tHz4CPSsjRiV9qzEfZb/20241129/cn-shanghai/oss/aliyun_v4_request,Signature=574119a4c1a33ffc373d2d6fa4901f4c923430fa6406704e51ff18f28ba9771d"],["x-oss-content-sha256","UNSIGNED-PAYLOAD"],["x-oss-date","20241129T014427Z"]]), method: "HEAD" }) that works in the node repl for example
Walshy
Walshy2w ago
200 locally, 403 deployed looking at deployed logs Hmm so no error being returned from the API this time since this works locally and that also runs the runtime, I think this may be something within the cf network but i'm not sure what :thonk:
ephemer
ephemerOP2w ago
Everything but HEAD works. I tried a PUT request and was able to upload something to the bucket and then GET it again: all via the Worker. Can you check again that HEAD doesn't have something strange going on regarding the headers it sends?
Walshy
Walshy2w ago
i can confirm the 403 is coming from the origin, we arent changing 200 -> 403 so that's at least a good sign local dev works and that also runs the workers runtime
ephemer
ephemerOP2w ago
I'm also pretty sure it's coming from the origin due to the content-type which is set to application/xml – that only happens in the case of an API error that's interesting. is it a "close enough" runtime or is it really identical?
Walshy
Walshy2w ago
it is literally the runtime, we have an internal layer that just does the cf platform part but all the actual behaviour of apis is all public, open source and used in local/remote https://github.com/cloudflare/workerd for ref
ephemer
ephemerOP2w ago
cool ok So should I tag this issue differently or are you able to mention somebody who could help debug at the network level? I tried something else: If I deliberately mess with the string used to sign the request I get a proper error message back:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
...
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
...
It's weird that the 403 is happening at all, but it's even weirder that it seems unsure itself why. Maybe something in the network is confused because a GET request is triggering a HEAD request? Not sure why that would trigger 403 though Ok something really weird is happening. If the requested file is a .jpg (existent or not) I get the 403, but if I deliberately put a non-existent filename with a different extension I get 404 as expected. If I put an existent .txt file I get 200, including via CF Is cloudflare trying to optimize the image file somehow? Same issue with .png files, and .mp3 🤔 It's not the mime type. setting a .txt file to image/jpeg returns a 200 with the "correct" Content-Type, not a 403. So I guess it's the file extension Still through CF: Uploading a new ".jpg" file with text contents "works" (I can GET it) but HEAD fails with 403. The same weirdness applies for .mp4 and .js file extensions but not for .py. I have the feeling this is something to do with some optimization cloudflare is trying to apply to files with certain extensions. It's quite telling that .js is problematic but .py is not for example. That said: .exe and .zip (I'm running out of extensions to try) suffer from the same issue, which seems more surprising to me.
Walshy
Walshy2w ago
Again though, it's the origin returning 403, optimisation would come after the origin returned (though, you aren't doing any optimisation anyway)
ephemer
ephemerOP2w ago
Right, the origin is returning 403. But only in the CF Worker. So CF must be altering the request to the origin somehow for HEAD requests with certain file extensions 🤔 No file extension is also fine for example. So from the extensions I've tried, only .py , .txt, and no extension at all have worked. You can try it yourself easily: curl -I https://test.flowkey.workers.dev/test curl -I https://test.flowkey.workers.dev/test.mp2 curl -I https://test.flowkey.workers.dev/test.mp3 curl -I https://test.flowkey.workers.dev/test.mp4 curl -I https://test.flowkey.workers.dev/test.mp5 all of those should return with 404, but the "common" filetypes (mp3, mp4) return with 403 instead. Again, this does not happen with the same requests being made from another platform (presumably also not from a local cloudflare dev environment) I have to go to bed. Here's hoping for a miracle or at least some minor enlightenment by the time I get up 👼
Walshy
Walshy2w ago
so this is the error: https://api.aliyun.com/troubleshoot?q=0002-00000201
The request you initiated uses a V4 signature, but the signature provided in the request does not match the signature calculated by OSS.
which is interesting
OpenAPI自助诊断-阿里云OpenAPI开发者门户
OpenAPI自助诊断工具是阿里云OpenAPI开发者门户提供的OpenAPI报错排查工具,根据返回的错误信息、requestid智能匹配诊断方案,助力开发者自闭环阿里云OpenAPI报错问题。
ephemer
ephemerOP2w ago
Can you share how you found that out? That might help debug Normally they tell you what the signature should have been based on (headers etc). If something is altering the original request (the signature is already correct for the unaltered request), that message should indicate what has changed
Walshy
Walshy2w ago
the error code header is set still there's just no error code body for this error
< x-oss-ec: 0002-00000201
ephemer
ephemerOP7d ago
This error message means the Authorization header is being sent (otherwise how would OSS know that it's a v4 signature). But maybe the other required headers (specifically x-oss-date and x-oss-content-sha256) are being swallowed somehow. I am going to try to set up a local server that prints the incoming requests and point the worker to it via a tunnel. An update: I set up netcat locally with a tunnel and sent fetch requests to it from a Worker in the same way I was sending them to OSS. All headers arrived as they should, regardless of the url path. Even if I took the exact request coming in from the worker, with all CF headers, and sent it to OSS, it worked. So I still don’t know what the issue is. That said, I also found a built-in OSS feature that does what I need (the custom functionality I was building), so I’ve canned the project I was working on here entirely. I don’t consider this issue closed (it’s real), but for my part, it has been abandoned. Thanks for your support!
Want results from more Discord servers?
Add your server