"RuntimeError: memory access out of bounds" in Rust queue consumer
I'm seeing a recurring issue where, after multiple hours of being up, a Worker consuming from a Cloudflare Queue will start erroring with
RuntimeError: memory access out of bounds
RuntimeError: memory access out of bounds
.
Once the Worker is in this state, all of invocations fail with this same error until we deploy the Worker again.
RuntimeError: memory access out of bounds at cloud_worker.wasm.__externref_table_alloc (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[1325]:0x193b7d) at cloud_worker.wasm.queue externref shim (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[11398]:0x2fe4b9) at Object.queue (index.js:238:20)
RuntimeError: memory access out of bounds at cloud_worker.wasm.__externref_table_alloc (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[1325]:0x193b7d) at cloud_worker.wasm.queue externref shim (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[11398]:0x2fe4b9) at Object.queue (index.js:238:20)
Prior to getting into this state, I have observed two other categories of errors:
1. An error indicating that the "Workers runtime canceled this request"
The Workers runtime canceled this request because it detected that your Worker's code had hung and would never generate a response. Refer to: https://developers.cloudflare.com/workers/observability/errors/
The Workers runtime canceled this request because it detected that your Worker's code had hung and would never generate a response. Refer to: https://developers.cloudflare.com/workers/observability/errors/
2. A
RangeError: Invalid array buffer length
RangeError: Invalid array buffer length
error:
[js] JsValue(RangeError: Invalid array buffer length RangeError: Invalid array buffer length at Uint8Array.subarray (<anonymous>) at getStringFromWasm0 (index.js:126:58) at index.js:1258:62 at handleError (index.js:107:14) at imports.wbg.__wbg_put_09c2936b2dcd60d5 (index.js:1252:12) at cloud_worker.wasm.__wbg_put_09c2936b2dcd60d5 externref shim (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[11939]:0x304324) at cloud_worker.wasm.cloudflare_bindings::kv::KvStore::put_text::h2e1ec1929da0ae7c (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[6229]:0x29c4d5) at cloud_worker.wasm.<cloudflare_platform::kv::CloudflareKvStore as platform_api::kv::KvStore<cloudflare_platform::CloudflarePlatform>>::put_text::{{closure}}::h5d32f28a9f3bc505 (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[706]:0x134179) at cloud_worker.wasm.<core::pin::Pin<P> as core::future::future::Future>::poll::he128884e4522b225 (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[13915]:0x312fa3) at cloud_worker.wasm.platform_api::kv::KvStore::put_json::{{closure}}::h6521ad6bca6f8e75 (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[1584]:0x1af558))
[js] JsValue(RangeError: Invalid array buffer length RangeError: Invalid array buffer length at Uint8Array.subarray (<anonymous>) at getStringFromWasm0 (index.js:126:58) at index.js:1258:62 at handleError (index.js:107:14) at imports.wbg.__wbg_put_09c2936b2dcd60d5 (index.js:1252:12) at cloud_worker.wasm.__wbg_put_09c2936b2dcd60d5 externref shim (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[11939]:0x304324) at cloud_worker.wasm.cloudflare_bindings::kv::KvStore::put_text::h2e1ec1929da0ae7c (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[6229]:0x29c4d5) at cloud_worker.wasm.<cloudflare_platform::kv::CloudflareKvStore as platform_api::kv::KvStore<cloudflare_platform::CloudflarePlatform>>::put_text::{{closure}}::h5d32f28a9f3bc505 (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[706]:0x134179) at cloud_worker.wasm.<core::pin::Pin<P> as core::future::future::Future>::poll::he128884e4522b225 (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[13915]:0x312fa3) at cloud_worker.wasm.platform_api::kv::KvStore::put_json::{{closure}}::h6521ad6bca6f8e75 (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[1584]:0x1af558))
The stack trace for this one will vary, which I suspect is indicative of a broader memory issue within the Wasm instance/V8 isolate.