Anyone knows how to compress a gzip text on Workers? to/and/or Durable Objects? ... I know that i ca

Anyone knows how to compress a gzip text on Workers? to/and/or Durable Objects? ... I know that i can create a new response and put a gzip content, then call the .text() e get the content decoded . But i dont know to to send a plain text and get a gzip using only the workers.

The ideia is store more than the current 32kib of data... I have a 100kb text that can be easy be compressed to 16kib;

WWallacy Anyone knows how to compress a gzip text on Workers? to/and/or Durable Objects? ...

John Spurlock•8/20/21, 10:57 PM

I don't think it's exposed as a runtime call. It would be cool if CF supported CompressionStreamCompressionStream https://caniuse.com/mdn-api_compressionstream

John Spurlock•8/20/21, 10:58 PM

But you'd quickly burn through your CPU limits

JJohn Spurlock I don't think it's exposed as a runtime call. It would be cool if CF supported ...

WallacyOP•8/20/21, 11:00 PM

That or the storage.put using some gzip (o other compression) argument.

Aasp You can do this with a parameter or a header, or you could set it as the hostnam...

mikeysee•8/21/21, 5:00 AM

awesome thanks

mikeysee•8/21/21, 6:58 AM

How do people unit test their durable objects? I have been trying to use some of the helper classes from miniflare (https://github.com/mrbbot/miniflare/blob/2cca9aea04/src/kv/do.ts) but I get the following errors in jest:

Test suite failed to run

    Cannot find module '../generated/offsets' from '../../node_modules/tz-offset/src/tz-offset.js'

    Require stack:
      C:/dev/me/tinkering-with-durable-objects/node_modules/tz-offset/src/tz-offset.js
      C:/dev/me/tinkering-with-durable-objects/node_modules/node-cron/src/scheduled-task.js
      C:/dev/me/tinkering-with-durable-objects/node_modules/node-cron/src/node-cron.js
      C:/dev/me/tinkering-with-durable-objects/node_modules/miniflare/dist/index.js
      src/projections/ProjectionDurableObject.test.ts

Test suite failed to run

    Cannot find module '../generated/offsets' from '../../node_modules/tz-offset/src/tz-offset.js'

    Require stack:
      C:/dev/me/tinkering-with-durable-objects/node_modules/tz-offset/src/tz-offset.js
      C:/dev/me/tinkering-with-durable-objects/node_modules/node-cron/src/scheduled-task.js
      C:/dev/me/tinkering-with-durable-objects/node_modules/node-cron/src/node-cron.js
      C:/dev/me/tinkering-with-durable-objects/node_modules/miniflare/dist/index.js
      src/projections/ProjectionDurableObject.test.ts

GitHub

miniflare/do.ts at 2cca9aea0412f874f68dfbd768eb132b276d8575 · mrbbo...

Fully-local simulator for Cloudflare Workers. Contribute to mrbbot/miniflare development by creating an account on GitHub.

Mmikeysee How do people unit test their durable objects? I have been trying to use some of...

MrBBot•8/21/21, 8:37 AM

are you setting the moduleFileExtensionsmoduleFileExtensions Jest option? if you are, make sure you've got "json""json" included in that array

mikeysee•8/21/21, 8:37 AM

oh lol apparently I was and indeed just adding "json" makes it work

mikeysee•8/21/21, 8:37 AM

thanks!

mikeysee•8/21/21, 8:38 AM

BTW @mrbbot while I have you, is there a reason DurableObjectIdDurableObjectId isnt exported?

MrBBot•8/21/21, 8:40 AM

probably just forgot

, you should be able to do import { DurableObjectId } from "miniflare/dist/modules/do"import { DurableObjectId } from "miniflare/dist/modules/do", the next version will make it much easier to use individual components like this

mikeysee•8/21/21, 11:57 AM

RraRaRa Hmm there was a nice limitations page that pretty much answered that question fo...

John Spurlock•8/21/21, 9:40 PM

you can do

setTimeout

setTimeout

and they pretty much always fire, but not if the object needs to be destroyed (colo goes down, or service upgrade) or runs out of runtime. For the latter, I believe the cloudflare team said that they are working on an update to ensure they will fire more consistently in the future (search the scrollback), but no public docs for that yet.

RraRaRa Ok great! This is really powerful way to make "serverless" multiplayer games. Ju...

John Spurlock•8/21/21, 9:50 PM

Like all serverless platforms, you have to architect around possible downtime for any reason like network outage, or rolling service upgrades. For something like a timer, you might want to use

setTimeout

setTimeout

for the 99% case, but also store the timer in state and have a cron that runs every minute as a fallback, the cron is guaranteed to run even if the DO is recreated

RraRaRa Yeah gotcha! If DO is re-created for some reason, I assume it's like any new DO,...

John Spurlock•8/21/21, 9:53 PM

the in-memory state will be cleared - yes like websockets, but you can access the same persistent storage using

state.storage

state.storage

John Spurlock•8/21/21, 9:53 PM

which won't help you for websockets : )

John Spurlock•8/21/21, 9:54 PM

Just curious if this is production ready

Depends on your production needs. For a game, maybe. Officially, it's not even GA yet, so not production-ready from an official point of view. Not even billed yet!

Hello, I’m Allie!•8/21/21, 9:59 PM

Hey @john.spurlock, don't know if this is a projected usecase, or if there is even a way to do this with DO, but would it be possible for a DO to receive a stream, and then pass it on to many clients? So, for example, a music streaming platform, that received one input, and streamed it to all of the connected clients?

HHello, I’m Allie!Hey @john.spurlock, don't know if this is a projected usecase, or if there is ev...

John Spurlock•8/21/21, 10:01 PM

I believe @kentonv_cf said streaming requests and responses work inside DOs (like between the entry-point worker and the DO), but I've not tried it myself. Would be a cool proof of concept, would need to be unbound though, or you'd run out of cpu quickly.

JJohn Spurlock I believe @kentonv_cf said streaming requests and responses work inside DOs (lik...

Hello, I’m Allie!•8/21/21, 10:02 PM

Or a flat-rated handoff API, where you'd pass the connection into it, and then it would handle splitting and transmitting it, without it counting as CPU time.

RraRaRa Yeah, I guess it's good enough for a few protoypes 🙂 I've wanted to port my old...

John Spurlock•8/21/21, 10:03 PM

I think this will be a huge game changer for writing online services and games.

I agree, as long as you're fine with tying your architecture to cloudflare

HHello, I’m Allie!Or a flat-rated handoff API, where you'd pass the connection into it, and then i...

John Spurlock•8/21/21, 10:04 PM

You could possibly send the chunks over ws too, instead of trying to stream everything through - anyway cool idea

John Spurlock•8/22/21, 3:48 AM

You can certainly use DOs without ever using

state.storage

state.storage

, it is there if you need it, but you can have your object live only in memory if you want.

SSudonim https://developers.cloudflare.com/workers/learning/using-durable-objects#global-...

Hello, I’m Allie!•8/22/21, 4:26 AM

Note though that there is no guarentee that the DO Instance will persist for that long. In fact, I believe that DOs are automatically purged 30 seconds after the last request.

ItsWendell•8/22/21, 9:54 AM

Are there any reports on avg latencies between different edges / colos from DO to DO? And latencies between same colo DO to same colo DO?

HHello, I’m Allie!Note though that there is no guarentee that the DO Instance will persist for tha...

asp•8/22/21, 10:34 AM

It was exactly 120 seconds after last request last time I checked. That was a little over a week ago.

MrBBot•8/22/21, 11:50 AM

Something brand new

ItsWendell•8/22/21, 3:40 PM

Yeah both have benefits / trade-offs. KV store is better if you have super high reads, but the lobby won't be as accurate since it might take longer to update globally and it isn't transactional. A lobby DO would allow you for a transactional storage API but might have limits on throughput, since your DO might run out of memory if you get too many incoming requests, and your lobby would be 'stuck' on one edge having high latency for some, while others have low latency depending on where the lobby is.

MMallissin Instead of streaming the audio directly, you can also use MPEG-Dash or HLS to ou...

Hello, I’m Allie!•8/22/21, 7:38 PM

That's a good idea. How would it handle getting prematurely shut down, though? Would there be a way to store a reference to a connection in Storage?

RraRaRa I'm wondering what the best way would be to create a lobby that shows list of al...

Erwin•8/23/21, 5:09 AM

My first instinct would be for the rooms to be their own DO and the Lobby to be another. And the room DOs can update the Lobby DO as needed. Then depending on how many people would be in a lobby I would connect them straight to the DO, or go via something like KV

Erwin•8/23/21, 10:31 AM

It is scaling (per DO) issues. They are very horizontally scalable. At 10/20 rooms you’ll be fine!

Anyone knows how to compress a gzip text on Workers? to/and/or Durable Objects? ... I know that i ca

Similar Threads

Similar Threads

Similar Threads