perhaps its out of band, and re-created as a header on the incoming request?

AAvi perhaps its out of band, and re-created as a header on the incoming request?

Chaika•4/28/25, 8:21 PM

yea, I think it's entirely out of your direct control/just gets passed through some other property to the Durable Object in the call. The Durable Object makes a request out, which goes through the CDN stuff, and the request out just contains the geo data stuff lingering. It's up to the customer's config on how it's used/if its put into a header/etc

AviOP•4/28/25, 8:21 PM

right

Chaika•4/28/25, 8:21 PM

could try using RPC instead of fetch to further ensure it's not any piece of the request

AviOP•4/28/25, 8:21 PM

my mental model is that cloudflare has its own metadata attached to requests flowing within the CF network

AviOP•4/28/25, 8:21 PM

and somewhere there is geo data

and if the receiving side has the geoIP enrichment feature enabled, they attach that data as headers to the incoming request just before delivering it to the worker

Chaika•4/28/25, 8:22 PM

I've noticed this same behavior of cross-zone/acct fetches using worker's ip in cf-connecting-ip but not replacing the geodata for years, just never fully tested with Durable Objects and everything

AviOP•4/28/25, 8:41 PM

interesting

AviOP•4/28/25, 8:41 PM

do you happen to know where worker cron jobs run?

AviOP•4/28/25, 8:41 PM

like how does cf decide which colo to run those

AviOP•4/28/25, 8:44 PM

hm, it seems like its based on lowest available load, which means it could really be anywhere

AAvi hm, it seems like its based on lowest available load, which means it could reall...

Chaika•4/28/25, 8:51 PM

yea pretty much anywhere, I have a bunch of cron monitoring, in the last 24 hours

AviOP•4/28/25, 8:51 PM

oh wow!

AviOP•4/28/25, 8:52 PM

that's so cool, those are the colos your cron workers ran from in the past 24h?

AAvi that's so cool, those are the colos your cron workers ran from in the past 24h?

Chaika•4/28/25, 8:53 PM

indeed. For a specific account/cron picks a location and usually sticks with it for a few hours, barring any incidents

AviOP•4/28/25, 8:53 PM

very cool

AviOP•4/28/25, 8:53 PM

well... none of those countries seem to be on the openai banned list

AviOP•4/28/25, 8:54 PM

the countries that are blocked by OpenAI but have colos seem to be:

Russia
Saudi Arabia
Hong Kong
China (mainland)

AviOP•4/28/25, 8:55 PM

we're debating solving this problem of the request geolocation by shifting the work to run out-of-band from the inital request, via a cron job

Chaika•4/28/25, 8:56 PM

HKG is going to be the biggest one there and one that supports Durable Objects
CF has Russia stuff but not too much I don't think. China for Cloudflare is all locked behind Enterprise

AviOP•4/28/25, 8:56 PM

well it's just a scheduled worker so i dont think it'd need to support DO

AviOP•4/28/25, 8:56 PM

good to know china is behind enterprise

AAvi well it's just a scheduled worker so i dont think it'd need to support DO

Chaika•4/28/25, 8:57 PM

true but supporting DOs is an indictator of sign/nearby locations (in HKG's case case, CF has ~9 or so colocations making up the entire PoP)

AviOP•4/28/25, 8:58 PM

but why would that impact the likliehood of a cron job getting scheduled to run there?

Chaika•4/28/25, 8:58 PM

only a few locations run cron jobs

CChaika only a few locations run cron jobs

AviOP•4/28/25, 8:59 PM

i see, and it sounds like, so far, it's a safe bet that those WONT be in russia/china/saudi arabia/HK?

AviOP•4/28/25, 8:59 PM

riskiest one being HK, it sounds like

AAvi i see, and it sounds like, so far, it's a safe bet that those WONT be in russia/...

Chaika•4/28/25, 9:01 PM

Pretty much. I've been collecting cron data into this system from 2025-03-17 01:44:20, and I've only seen

    ┌─colo─┬─count─┐
 1. │ EWR  │ 26965 │
 2. │ LHR  │ 38506 │
 3. │ ZRH  │ 38292 │
 4. │ IAH  │ 25177 │
 5. │ KIX  │ 39401 │
 6. │ MEL  │ 35336 │
 7. │ SCL  │ 22145 │
 8. │ WAW  │ 33659 │
 9. │ SLC  │ 28135 │
10. │ TPE  │ 33274 │
11. │ NRT  │ 34217 │
12. │ SIN  │ 41770 │
13. │ MAD  │ 40630 │
14. │ DFW  │ 25319 │
15. │ DTW  │ 30731 │
    └──────┴───────┘

    ┌─colo─┬─count─┐
 1. │ EWR  │ 26965 │
 2. │ LHR  │ 38506 │
 3. │ ZRH  │ 38292 │
 4. │ IAH  │ 25177 │
 5. │ KIX  │ 39401 │
 6. │ MEL  │ 35336 │
 7. │ SCL  │ 22145 │
 8. │ WAW  │ 33659 │
 9. │ SLC  │ 28135 │
10. │ TPE  │ 33274 │
11. │ NRT  │ 34217 │
12. │ SIN  │ 41770 │
13. │ MAD  │ 40630 │
14. │ DFW  │ 25319 │
15. │ DTW  │ 30731 │
    └──────┴───────┘

AviOP•4/28/25, 9:01 PM

amazing

AviOP•4/28/25, 9:01 PM

that's unbelievably helpful

AAvi amazing

Chaika•4/28/25, 9:03 PM

The thing that's a bit iffy is that it's never guaranteed, Cloudflare should start scheduling them wherever

Chaika•4/28/25, 9:03 PM

btw is Taiwan/TPE banned? It's part of those

Chaika•4/28/25, 9:04 PM

green cron execution is a shorter list

   ┌─colo─┬─count─┐
1. │ LHR  │  9680 │
2. │ MAD  │  8311 │
3. │ SLC  │  7795 │
4. │ SIN  │  7770 │
5. │ DTW  │  7602 │
6. │ IAH  │  7601 │
7. │ MEL  │  6692 │
8. │ TPE  │  6268 │
   └──────┴───────┘

   ┌─colo─┬─count─┐
1. │ LHR  │  9680 │
2. │ MAD  │  8311 │
3. │ SLC  │  7795 │
4. │ SIN  │  7770 │
5. │ DTW  │  7602 │
6. │ IAH  │  7601 │
7. │ MEL  │  6692 │
8. │ TPE  │  6268 │
   └──────┴───────┘

AviOP•4/28/25, 9:05 PM

Taiwan is not blocked by OpenAI

AviOP•4/28/25, 9:05 PM

green cron execution is a great trick too

AviOP•4/28/25, 9:05 PM

probably a good bet those countries aren't likely to be green any time soon...

AAvi probably a good bet those countries aren't likely to be green any time soon...

Chaika•4/28/25, 9:07 PM

Durable Object Alarms with eu/fedramp juridictions are another option as well, and perhaps less breaking potential and more precision

AviOP•4/28/25, 9:08 PM

those can only ever have one running at once, right?

Chaika•4/28/25, 9:08 PM

per durable object instance

AviOP•4/28/25, 9:08 PM

that seems pretty complicated/messy for what we need, which is basically running a job queue

AviOP•4/28/25, 9:08 PM

i was using cf workflows for this but it was phenomenally broken

AviOP•4/28/25, 9:09 PM

i cannot believe they call that GA

AviOP•4/28/25, 9:09 PM

you still can't even delete a workflow lol

AAvi that seems pretty complicated/messy for what we need, which is basically running...

Chaika•4/28/25, 9:09 PM

True, although I think the simplest and most reliable way for your use case would be calling it from a vps you have in a specific location or straight up proxying the request from those regions

AviOP•4/28/25, 9:10 PM

we've been discussing this approach... using a VPS would work but it'd mean setting up all new infra

AviOP•4/28/25, 9:10 PM

much simpler to just stick with CF

AviOP•4/28/25, 9:10 PM

to be clear we currently run this cron job on cf

AviOP•4/28/25, 9:10 PM

works nicely, processed many 100s of thousands of tasks

AviOP•4/28/25, 9:10 PM

we just aren't (yet) calling OpenAI from there

AviOP•4/28/25, 9:11 PM

but yes, a VPS with a specific static location would also work. i just don't want to have to manage another infra provider

Chaika•4/28/25, 9:11 PM

you're not the first one to run into the hkg problem, people have mentioned it in regards to ai gateway before

┌─colo─┬─count─┐ 1. │ EWR │ 26965 │ 2. │ LHR │ 38506 │ 3. │ ZRH │ 38292 │ 4. │ IAH │ 25177 │ 5. │ KIX │ 39401 │ 6. │ MEL │ 35336 │ 7. │ SCL │ 22145 │ 8. │ WAW │ 33659 │ 9. │ SLC │ 28135 │ 10. │ TPE │ 33274 │ 11. │ NRT │ 34217 │ 12. │ SIN │ 41770 │ 13. │ MAD │ 40630 │ 14. │ DFW │ 25319 │ 15. │ DTW │ 30731 │ └──────┴───────┘

┌─colo─┬─count─┐ 1. │ LHR │ 9680 │ 2. │ MAD │ 8311 │ 3. │ SLC │ 7795 │ 4. │ SIN │ 7770 │ 5. │ DTW │ 7602 │ 6. │ IAH │ 7601 │ 7. │ MEL │ 6692 │ 8. │ TPE │ 6268 │ └──────┴───────┘

perhaps its out of band, and re-created as a header on the incoming request?

Similar Threads

Similar Threads

Similar Threads