RPC Error handling and retries

The new RPC stuff is really great. One question I have is does the RPC calls handle retries? In my current RPC library I retry on these errors which are not uncommon:

const retryOn = new RegExp([
  'Network connection lost',
  'Cannot resolve Durable Object due to transient issue on remote node',
  'Durable Object reset because its code was updated',
  'The Durable Object\'s code has been updated',
].join('|'));

const retryOn = new RegExp([
  'Network connection lost',
  'Cannot resolve Durable Object due to transient issue on remote node',
  'Durable Object reset because its code was updated',
  'The Durable Object\'s code has been updated',
].join('|'));

const retryOn = new RegExp([
  'Network connection lost',
  'Cannot resolve Durable Object due to transient issue on remote node',
  'Durable Object reset because its code was updated',
  'The Durable Object\'s code has been updated',
].join('|'));

const retryOn = new RegExp([
  'Network connection lost',
  'Cannot resolve Durable Object due to transient issue on remote node',
  'Durable Object reset because its code was updated',
  'The Durable Object\'s code has been updated',
].join('|'));

(from https://github.com/dabblewriter/durable-apis/blob/main/src/durable-apis.ts#L12-L17)

Usually 1 retry is all that is needed, though I do use exponential backoff to continue trying for up to several seconds. This is built-in to my RPC lib. It would be great if it were built-in to your implementation. I don't want to wrap every RPC call with a try/catch to retry on regular network/lifecycle errors.

Thanks!

DaniFoldi•4/10/24, 8:39 PM

cc @kenton I couldn't find most of the error messages from DOs in workerd, and same for jsrpc, sadly also not documented. Is there perhaps a list of these retryable error messages you could give us? :MeowHeartCloudflare:

kenton•4/10/24, 8:45 PM

At present there is no automatic retry. Note that error.isDurableObjectReseterror.isDurableObjectReset should be true for the latter two errors on your list. Meanwhile, error.remoteerror.remote should be true if the error was thrown by application code on the remote end (i.e. not by system code in between). We have some plans to add more information along these lines to errors to help with this.

kenton•4/10/24, 8:46 PM

automatic retry wouldn't be safe in general unless either (a) we know that the message was never delivered to application code on the remote end, or (b) somehow the application tells us that the request is idempotent. For (a) we try our best to retry automatically (though we're not as good at this as we should be today). We don't have a mechanism for (b) at present.

Kkenton At present there is no automatic retry. Note that `error.isDurableObjectReset` s...

DaniFoldi•4/10/24, 8:50 PM

Oh interesting, I haven't seen these properties in any docs, I assume these aren't new?

kenton•4/10/24, 8:54 PM

oh sorry, it's actually durableObjectResetdurableObjectReset, not isDurableObjectResetisDurableObjectReset

kenton•4/10/24, 8:54 PM

but indeed it seems missing from docs, that's weird

Jacob WrightOP•4/10/24, 10:14 PM

Thank you @kenton.

Jacob WrightOP•4/10/24, 10:16 PM

So error.remoteerror.remote will be true if the error was from my code, otherwise the error was from something on the platform?

JJacob Wright So `error.remote` will be true if the error was from my code, otherwise the erro...

kenton•4/10/24, 10:23 PM

Right. .remote.remote is documented here FWIW: https://developers.cloudflare.com/durable-objects/reference/error-handling/

Cloudflare Docs

Error handling · Cloudflare Durable Objects docs

Review how to handle errors (exceptions) generated by both your own Durable Object code as well as exceptions thrown by Durable Objects’ …

RPC Error handling and retries

Similar Threads

Similar Threads

Similar Threads