Hmm, curious. One of my DOs moved from LAX to SJC today Is that supposed to happen?
2021-08-04 18:34:58.913 PDT DO resumed in LAX via worker in GRU2021-08-05 01:46:26.744 PDT DO resumed in LAX via worker in FRA2021-08-05 03:22:13.610 PDT DO resumed in SJC via worker in FRA2021-08-05 16:55:44.912 PDT DO resumed in SJC via worker in LAX
2021-08-04 18:34:58.913 PDT DO resumed in LAX via worker in GRU2021-08-05 01:46:26.744 PDT DO resumed in LAX via worker in FRA2021-08-05 03:22:13.610 PDT DO resumed in SJC via worker in FRA2021-08-05 16:55:44.912 PDT DO resumed in SJC via worker in LAX
I believe that DOs persist in one colo, until they are shut down. Once they are triggered again, they are reinitialized in the nearest DO-ready colo to the incoming request.
The way @kentonv_cf described it is that it lives in one colo, but is replicated to a few near colos for backup. It would not "jump" from the EU to the US.
According to your log above, it seems that because there was an extended period of time with no requests, it was terminated(FRA). Once someone else triggered it(Me, probably), it was reestablished in the nearest Colo(SJC) to my request.
So, somewhere within those 13 hours, the SJC Instance was terminated. Then, after 13 hours had passed, someone sent a request, initializing a new instance at LAX
DOs that do not have any external state (used the storage API) can be garbage collected and restarted at some other point.
But if I am not mistaken we did have an issue in LAX recently, so it may have been a fail-over indeed. And SJC being the copy with the data now. And you are correct in that a DO will not leave the cluster it is currently part of, so it won't move from the US to the EU etc.
I've tried this in multiple ways now but can't get it to work. In short, I have a worker which manages incoming websocket connections for GraphQL subscriptions. Once a subscription comes in for a specific chatroom, a connection needs to be opened (or forwarded) to that durable object of that specific chatroom.
The fetch method of stubs don't seem to return a webSocket, created and passed down as response on the durable object.
I can't seem to pass a websocket to the durable object
So when you say it can't auto-migrate, does that mean that if a DO is initialize spawned in the US, but later terminated(Not Wiped), it can never be recreated in Europe?
I'm not so sure about that, I think the tricky thing is is that the connection comes starts in the Worker. The Worker creates and start's listening to WebSocket events with the client. Once a websocket event comes in that subscribes to a specific chatroom channel, I need to somehow communicate from the Worker to the Durable Object in real-time.
I can't start a client / server from the Worker since there is no way I can pass the client in a Fetch request to the Durable Object.
I can't start a client / server from the Durable Object, since the response of the durable object with fetch, doesn't include the webSocket.