CD
Cloudflare Developers•3w ago
maxdeviant

"RuntimeError: memory access out of bounds" in Rust queue consumer

I'm seeing a recurring issue where, after multiple hours of being up, a Worker consuming from a Cloudflare Queue will start erroring with RuntimeError: memory access out of bounds. Once the Worker is in this state, all of invocations fail with this same error until we deploy the Worker again. 
RuntimeError: memory access out of bounds
    at cloud_worker.wasm.__externref_table_alloc (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[1325]:0x193b7d)
    at cloud_worker.wasm.queue externref shim (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[11398]:0x2fe4b9)
    at Object.queue (index.js:238:20)
RuntimeError: memory access out of bounds
    at cloud_worker.wasm.__externref_table_alloc (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[1325]:0x193b7d)
    at cloud_worker.wasm.queue externref shim (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[11398]:0x2fe4b9)
    at Object.queue (index.js:238:20)
Prior to getting into this state, I have observed two other categories of errors: 1. An error indicating that the "Workers runtime canceled this request" 
The Workers runtime canceled this request because it detected that your Worker's code had hung and would never generate a response. Refer to: https://developers.cloudflare.com/workers/observability/errors/
The Workers runtime canceled this request because it detected that your Worker's code had hung and would never generate a response. Refer to: https://developers.cloudflare.com/workers/observability/errors/
2. A RangeError: Invalid array buffer length error: 
[js] JsValue(RangeError: Invalid array buffer length RangeError: Invalid array buffer length
    at Uint8Array.subarray (<anonymous>)
    at getStringFromWasm0 (index.js:126:58)
    at index.js:1258:62 at handleError (index.js:107:14)
    at imports.wbg.__wbg_put_09c2936b2dcd60d5 (index.js:1252:12)
    at cloud_worker.wasm.__wbg_put_09c2936b2dcd60d5 externref shim (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[11939]:0x304324)
    at cloud_worker.wasm.cloudflare_bindings::kv::KvStore::put_text::h2e1ec1929da0ae7c (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[6229]:0x29c4d5)
    at cloud_worker.wasm.<cloudflare_platform::kv::CloudflareKvStore as platform_api::kv::KvStore<cloudflare_platform::CloudflarePlatform>>::put_text::{{closure}}::h5d32f28a9f3bc505 (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[706]:0x134179)
    at cloud_worker.wasm.<core::pin::Pin<P> as core::future::future::Future>::poll::he128884e4522b225 (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[13915]:0x312fa3)
    at cloud_worker.wasm.platform_api::kv::KvStore::put_json::{{closure}}::h6521ad6bca6f8e75 (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[1584]:0x1af558))
[js] JsValue(RangeError: Invalid array buffer length RangeError: Invalid array buffer length
    at Uint8Array.subarray (<anonymous>)
    at getStringFromWasm0 (index.js:126:58)
    at index.js:1258:62 at handleError (index.js:107:14)
    at imports.wbg.__wbg_put_09c2936b2dcd60d5 (index.js:1252:12)
    at cloud_worker.wasm.__wbg_put_09c2936b2dcd60d5 externref shim (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[11939]:0x304324)
    at cloud_worker.wasm.cloudflare_bindings::kv::KvStore::put_text::h2e1ec1929da0ae7c (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[6229]:0x29c4d5)
    at cloud_worker.wasm.<cloudflare_platform::kv::CloudflareKvStore as platform_api::kv::KvStore<cloudflare_platform::CloudflarePlatform>>::put_text::{{closure}}::h5d32f28a9f3bc505 (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[706]:0x134179)
    at cloud_worker.wasm.<core::pin::Pin<P> as core::future::future::Future>::poll::he128884e4522b225 (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[13915]:0x312fa3)
    at cloud_worker.wasm.platform_api::kv::KvStore::put_json::{{closure}}::h6521ad6bca6f8e75 (wasm://wasm/cloud_worker.wasm-01465b16:wasm-function[1584]:0x1af558))
The stack trace for this one will vary, which I suspect is indicative of a broader memory issue within the Wasm instance/V8 isolate.
8 Replies
maxdeviant
maxdeviantOP•3w ago
A couple of other things to note: 1. We are not using the worker crate. We have our own library that we've written atop wasm_bindgen bindings to the Cloudflare Workers APIs. 2. We have a number of different queue consumers, but this issue is limited to one specific consumer. 3. We are using console_error_panic_hook as our panic hook.
maxdeviant
maxdeviantOP•3w ago
While investigating this, I found this GitHub Issue that seems to describe the same scenario: https://github.com/cloudflare/workerd/issues/3264
GitHub
🐛 Bug Report — Runtime APIs: large wasm memory allocations bri...
Details Reproduction code with steps in README. We&#39;re using automerge library. When we make a large number of allocations a: Range Error: Invalid typed array length: undefined Error is thrown i...
maxdeviant
maxdeviantOP•3w ago
And while I would like to ultimately get to the root cause of why the Worker ends up in this state in the first place, right now I would be happy with a solution that allows for getting the Worker back into a healthy state. I've already successfully been able to have the Worker detect when it is in this unhealthy state, but I don't know what the best way is to get it back into a working state without requiring a redeploy. Is there a way that I can signal to the runtime that it should terminate the isolate and start a new one? I've tried using a panic! to do this, but that doesn't seem to help (as the panic will still be on the Wasm side of things). Would throwing an error on the JS side of things be enough?
maxdeviant
maxdeviantOP•2d ago
I haven't figured out a solution here yet. I did, however, increase the max_concurrency of the queue consumer from 5 to 10, as well as limit the max_batch_size to 1: 
max_concurrency = 10
max_batch_size = 1
max_concurrency = 10
max_batch_size = 1
After doing this, the failure mode changed, with the queue consumer getting into the invalid state much more quickly after a redeploy (the valleys on this graph are when it is in the bad state and not processing any messages):
No description
maxdeviant
maxdeviantOP•2d ago
This seems to support my working theory that this is due to some sort of memory leak, and having more consumers exacerbates the problem.
maxdeviant
maxdeviantOP•2d ago
Right now my line of inquiry is following these docs to do some memory profiling of the queue consumer locally to see if I can spot where a leak might be. Unfortunately don't have a concrete reproduction of the problem yet.
Cloudflare Docs
Profiling Memory
Understanding Worker memory usage can help you optimize performance, avoid Out of Memory (OOM) errors when hitting Worker memory limits, and fix memory leaks.
gbedford
gbedford•2d ago
Make sure your compat date is set to a recent date for finalization registry support and you are running the latest workers-rs and wasm bindgen. You likely have a memory leak due to a binding that cannot be disposed. This can be easy to spot in the chrome inspector memory profiles. All Wasm objects are usually wrapped in finalization registry handling. another thing to check here is if there is a panic happening resulting in undisposed memory we are working on adding unwind support to wasm bindgen to ensure recoverable panics but if it is that, you can plug into the new reinitialization support we added to wasm bindgen which we use by default for workers-rs now (https://github.com/wasm-bindgen/wasm-bindgen/pull/4644) this effectively resets the wasm back to as if it just started, but also requires a proxy on the entry points to ensure proper handling which we carefully do in workers-rs in summary: - Ensure finalization registry is working and testing for leaks with chrome profiles. Miniflare can make this easier. - Perhaps add a Rust panic handler with some logging to see if you are getting panics resulting in undisposed state, and if so implement panic reinitialization. There's also some other sneakier approaches to informing runtime termination here I'm happy to share too if it is this. But it's likely going to be one of the two above.
maxdeviant
maxdeviantOP•2d ago
Thank you! I'll pursue some of these avenues tomorrow and see what turns up

Did you find this page helpful?