📚 Miniflare 3 API docs: https://github.com/cloudflare/miniflare/blob/tre/packages/miniflare/README

MMrBBot 📚 Miniflare 3 API docs: https://github.com/cloudflare/miniflare/blob/tre/packa...

T

TravisFrank•6/29/23, 6:12 PM

Thanks @mrbbot!

T

TravisFrank•6/29/23, 10:11 PM

Any chance

sourcemap

sourcemap

sourcemap

sourcemap support is on the docket for the API? That's pretty much the only bit of moving to v3 that's got me bummed, the logging leaves a lot to be desired

TTravisFrank Any chance `sourcemap` support is on the docket for the API? That's pretty much...

M

MrBBotOP•6/29/23, 10:40 PM

Source mapping should be supported by default, but due to a bug in

workerd

workerd

, you'll need to wrap your worker with something that returns un-source-mapped stack traces as JSON using the

MF-Experimental-Error-Stack

MF-Experimental-Error-Stack

MF-Experimental-Error-Stack

MF-Experimental-Error-Stack header: https://github.com/cloudflare/workers-sdk/blob/main/packages/wrangler/templates/middleware/middleware-miniflare3-json-error.ts. Wrangler will do this automatically.

As for logging, completely agree. I'd like to try bring Node's log formatting to

workerd

workerd

at some point.

T

TravisFrank•6/29/23, 10:47 PM

Sweet, thanks for the tip

T

TravisFrank•6/29/23, 10:50 PM

Overall the upgrade feels good, it's like a warm safety blanket knowing the runtime is the same. I also actually find myself enjoying doing away with the

wrangler.toml

wrangler.toml

parsing and passing bindings in the config- it's less "magic", but for me it seems easier to debug

TTravisFrank Overall the upgrade feels good, it's like a warm safety blanket knowing the runt...

M

MrBBotOP•6/29/23, 10:53 PM

Nice, glad you're liking it! We're planning to standardise/simplify the

unstable_dev()

unstable_dev()

unstable_dev()

unstable_dev() API (https://developers.cloudflare.com/workers/wrangler/api/) at some point soon, which will allow you to use your

wrangler.toml

wrangler.toml

file, but customise as required in JavaScript too.

T

TravisFrank•6/29/23, 10:55 PM

Yes, love that!

T

TravisFrank•6/29/23, 10:56 PM

Btw, in case I'm missing something- there's no way to run multiple workers with

wrangler dev

wrangler dev

, right? If there is I can save myself the pain of manually putting together the

esbuild

esbuild

esbuild

esbuild watch -> MF reload script.

M

MrBBotOP•6/29/23, 10:58 PM

Yes there is, right now if you run a

wrangler dev

wrangler dev

session in each worker's directory, they should auto-magically connect. We're considering eventually moving to a Docker Compose style solution, where running

wrangler dev

wrangler dev

automatically starts up all required services though.

T

TravisFrank•6/29/23, 11:09 PM

Oh wow

T

TravisFrank•6/29/23, 11:25 PM

T

TravisFrank•6/29/23, 11:27 PM

It doesn't look like service bindings are working for me. I also don't get any sort of "Your worker has access to the following bindings: - Services:"

TTravisFrank It doesn't look like service bindings are working for me. I also don't get any ...

M

MrBBotOP•6/29/23, 11:31 PM

Hmmm ok, could you try restart your computer and then try again? This is kinda why I'd like to see us move to a Docker Compose like thing, sometimes this magic doesn't work.

T

TravisFrank•6/29/23, 11:41 PM

Damn, still no luck :/

T

TravisFrank•6/29/23, 11:42 PM

Where does the service registry live on my machine?

T

TravisFrank•6/30/23, 8:51 AM

Aha! Debugged! User error, bad

toml

toml

toml

toml (I had just added the

placement

placement

placement

placement table and had

services

services

below it, so

services

services

was interpreted as a key on

placement

placement

placement

placement)

MMrBBot Source mapping should be supported by default, but due to a bug in `workerd`, yo...

T

TravisFrank•6/30/23, 10:08 AM

I'm still a bit confused by the logging - none of my

console.logs

console.logs

console.logs

console.logs appear when running via the API, but they are appearing via

wrangler dev

wrangler dev

. I have

log: new Log(LogLevel.VERBOSE),

log: new Log(LogLevel.VERBOSE),

log: new Log(LogLevel.VERBOSE),

log: new Log(LogLevel.VERBOSE), in my options and

verbose: true,

verbose: true,

verbose: true,

verbose: true,, is there something else I'm missing?

TTravisFrank I'm still a bit confused by the logging - none of my `console.logs` appear when ...

M

MrBBotOP•6/30/23, 10:09 AM

Those API options should enable

workerd

workerd

's logging. Wrangler actually uses the DevTools inspector protocol for logging instead.

T

TravisFrank•6/30/23, 10:11 AM

Aha, gotchya

M

matt•6/30/23, 11:43 AM

oh cool, there's a miniflare channel

I just worked on a prototype for developing multiple workers but had to use express. Tried to use the

workers

workers

workers

workers array but the contents of

/scripts/

/scripts/

/scripts/

/scripts/ can change during dev, and this way the latest script is used for each request. is this an ok approach?

app.use('/:name/*', async (req, res) => {
  const mf = new Miniflare({
    scriptPath: `./scripts/${req.params.name}/index.js`,
    bindings: process.env,
  })
  const url = `http://localhost:8787${req.originalUrl}`
  const queryRes = await mf.dispatchFetch(url, {
    method: req.method,
    headers: req.headers,
    body: req.body,
  })
  res.send(await queryRes.text())
  await mf.dispose()
})

app.use('/:name/*', async (req, res) => {
  const mf = new Miniflare({
    scriptPath: `./scripts/${req.params.name}/index.js`,
    bindings: process.env,
  })
  const url = `http://localhost:8787${req.originalUrl}`
  const queryRes = await mf.dispatchFetch(url, {
    method: req.method,
    headers: req.headers,
    body: req.body,
  })
  res.send(await queryRes.text())
  await mf.dispose()
})

app.use('/:name/*', async (req, res) => {
  const mf = new Miniflare({
    scriptPath: `./scripts/${req.params.name}/index.js`,
    bindings: process.env,
  })
  const url = `http://localhost:8787${req.originalUrl}`
  const queryRes = await mf.dispatchFetch(url, {
    method: req.method,
    headers: req.headers,
    body: req.body,
  })
  res.send(await queryRes.text())
  await mf.dispose()
})

app.use('/:name/*', async (req, res) => {
  const mf = new Miniflare({
    scriptPath: `./scripts/${req.params.name}/index.js`,
    bindings: process.env,
  })
  const url = `http://localhost:8787${req.originalUrl}`
  const queryRes = await mf.dispatchFetch(url, {
    method: req.method,
    headers: req.headers,
    body: req.body,
  })
  res.send(await queryRes.text())
  await mf.dispose()
})

Mmatt oh cool, there's a miniflare channel 🙂 I just worked on a prototype for develop...

M

MrBBotOP•6/30/23, 12:15 PM

Hey!

This should work, but ideally you'd want to reuse the

Miniflare

Miniflare

Miniflare

Miniflare instance across requests. Do you know the possible values for

req.params.name

req.params.name

req.params.name

req.params.name ahead of time? Could you cache

mf

mf

mf

mf values for each

name

name

in a

Map<string, Miniflare>

Map<string, Miniflare>

Map<string, Miniflare>

Map<string, Miniflare>? You'll probably also want to copy the headers across from

queryRes

queryRes

queryRes

queryRes to

res

res

, and might consider streaming the body instead of buffering it. See the

writeResponse

writeResponse

writeResponse

writeResponse function in Miniflare's own code for this: https://github.com/cloudflare/miniflare/blob/6984a7f7a35a5b91dea3bb62d657818ca3cb52a2/packages/miniflare/src/index.ts#L307-L354.

M

matt•6/30/23, 12:26 PM

this looks brilliant, thank you! The contents of

./scripts

./scripts

./scripts

./scripts will change quite a lot during development, e.g.
-

./scripts/<worker_name>

./scripts/<worker_name>

./scripts/<worker_name>

./scripts/<worker_name> folders created/deleted
- contents of

./scripts/<worker_name>/index.js

./scripts/<worker_name>/index.js

./scripts/<worker_name>/index.js

./scripts/<worker_name>/index.js updated
id be happy to cache the

mf

mf

mf

mf instances, but is there a way to invalidate the cache in either of these scenarios?

Mmatt this looks brilliant, thank you! The contents of `./scripts` will change quite a...

M

MrBBotOP•6/30/23, 12:27 PM

const promise = cache.get(key)?.dispose(); cache.delete(key); await promise;

const promise = cache.get(key)?.dispose(); cache.delete(key); await promise;

const promise = cache.get(key)?.dispose(); cache.delete(key); await promise;

const promise = cache.get(key)?.dispose(); cache.delete(key); await promise; should invalidate it

M

matt•6/30/23, 12:30 PM

that will definitely work if I can figure out how to wire up the fs events, is there anything exposed by miniflare that can monitor the worker script folders/files? I would imagine perhaps the livereload uses something in this vein

Mmatt that will definitely work if I can figure out how to wire up the fs events, is t...

M

MrBBotOP•6/30/23, 12:31 PM

Unfortunately not, live reload relies on you calling

setOptions()

setOptions()

setOptions()

setOptions() yourself with updates

M

matt•6/30/23, 12:32 PM

got it, thanks for your help! this has been a fun prototype to put together

Original message was deleted

M

MrBBotOP•6/30/23, 12:45 PM

Replied over there

T

TravisFrank•6/30/23, 2:11 PM

Is there anyway to view worker logs while running with

unstable_dev

unstable_dev

? I'm not seeing anything show up in DevTools

T

TravisFrank•6/30/23, 4:27 PM

Ah, nvm, I see in source that's it's not yet implented

MMrBBot Hey! 👋 This should work, but ideally you'd want to reuse the `Miniflare` instan...

M

matt•7/1/23, 6:12 AM

was just reading up on traditional streams vs web streams in node. I may be wrong but I wonder if you can replace the last few lines of this function with:

await response.body?.pipeTo(Writable.toWeb(initialStream))

await response.body?.pipeTo(Writable.toWeb(initialStream))

await response.body?.pipeTo(Writable.toWeb(initialStream))

await response.body?.pipeTo(Writable.toWeb(initialStream))

M

matt•7/1/23, 8:50 AM

Am I doing this wrong? I picked it up rom bindings.spec.ts

mounts: {
  books: {
    // here I define name, scriptPath, modules, compatDate, compatFlags, bindings
  } 
}

mounts: {
  books: {
    // here I define name, scriptPath, modules, compatDate, compatFlags, bindings
  } 
}

mounts: {
  books: {
    // here I define name, scriptPath, modules, compatDate, compatFlags, bindings
  } 
}

mounts: {
  books: {
    // here I define name, scriptPath, modules, compatDate, compatFlags, bindings
  } 
}

Worker "core:user:example"'s binding "BOOKS" refers to a service "core:user:books", but no such service is defined

Worker "core:user:example"'s binding "BOOKS" refers to a service "core:user:books", but no such service is defined

Worker "core:user:example"'s binding "BOOKS" refers to a service "core:user:books", but no such service is defined

Worker "core:user:example"'s binding "BOOKS" refers to a service "core:user:books", but no such service is defined

M

matt•7/1/23, 8:53 AM

for now I am getting away with ... its working but I worry about the overhead

serviceBindings.books = req => {
  const smf = new Miniflare(<same options as mount config>)
  return smf.dispatchFetch(req)  
}

serviceBindings.books = req => {
  const smf = new Miniflare(<same options as mount config>)
  return smf.dispatchFetch(req)  
}

serviceBindings.books = req => {
  const smf = new Miniflare(<same options as mount config>)
  return smf.dispatchFetch(req)  
}

serviceBindings.books = req => {
  const smf = new Miniflare(<same options as mount config>)
  return smf.dispatchFetch(req)  
}

Mmatt was just reading up on traditional streams vs web streams in node. I may be wron...

M

MrBBotOP•7/1/23, 8:53 AM

Yep, that should work in modern Node versions. Miniflare’s minimum supported version is 16.13.0, which doesn’t provide those web stream adapters.

M

matt•7/1/23, 8:54 AM

that's a very good point, I always forget about the older versions of node

Mmatt Am I doing this wrong? I picked it up rom bindings.spec.ts ``` mounts: { books...

M

MrBBotOP•7/1/23, 8:55 AM

Miniflare 3 doesn’t use mounts anymore, checkout the migration guide for a multi-worker example: https://miniflare.dev/get-started/migrating#removed-options

⬆️ Migrating from Version 2 · Miniflare

Fun, full-featured, fully-local simulator for Cloudflare Workers

M

matt•7/1/23, 8:56 AM

this is brilliant, thank you

T

TravisFrank•7/3/23, 10:14 AM

@mrbbot Any idea what's going on here? Will keep digging

Error: write EPIPE
    at afterWriteDispatched (node:internal/stream_base_commons:160:15)
    at writeGeneric (node:internal/stream_base_commons:151:3)
    at Socket._writeGeneric (node:net:930:11)
    at Socket._write (node:net:942:8)
    at writeOrBuffer (node:internal/streams/writable:392:12)
    at _write (node:internal/streams/writable:333:10)
    at Socket.Writable.write (node:internal/streams/writable:337:10)
    at Runtime.updateConfig (/nexus/node_modules/miniflare/src/runtime/index.ts:132:26)
    at async Miniflare.#assembleAndUpdateConfig (/nexus/node_modules/miniflare/src/index.ts:790:23)
    at async Miniflare.#init (/nexus/node_modules/miniflare/src/index.ts:472:5) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}

Error: write EPIPE
    at afterWriteDispatched (node:internal/stream_base_commons:160:15)
    at writeGeneric (node:internal/stream_base_commons:151:3)
    at Socket._writeGeneric (node:net:930:11)
    at Socket._write (node:net:942:8)
    at writeOrBuffer (node:internal/streams/writable:392:12)
    at _write (node:internal/streams/writable:333:10)
    at Socket.Writable.write (node:internal/streams/writable:337:10)
    at Runtime.updateConfig (/nexus/node_modules/miniflare/src/runtime/index.ts:132:26)
    at async Miniflare.#assembleAndUpdateConfig (/nexus/node_modules/miniflare/src/index.ts:790:23)
    at async Miniflare.#init (/nexus/node_modules/miniflare/src/index.ts:472:5) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}

Error: write EPIPE
    at afterWriteDispatched (node:internal/stream_base_commons:160:15)
    at writeGeneric (node:internal/stream_base_commons:151:3)
    at Socket._writeGeneric (node:net:930:11)
    at Socket._write (node:net:942:8)
    at writeOrBuffer (node:internal/streams/writable:392:12)
    at _write (node:internal/streams/writable:333:10)
    at Socket.Writable.write (node:internal/streams/writable:337:10)
    at Runtime.updateConfig (/nexus/node_modules/miniflare/src/runtime/index.ts:132:26)
    at async Miniflare.#assembleAndUpdateConfig (/nexus/node_modules/miniflare/src/index.ts:790:23)
    at async Miniflare.#init (/nexus/node_modules/miniflare/src/index.ts:472:5) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}

Error: write EPIPE
    at afterWriteDispatched (node:internal/stream_base_commons:160:15)
    at writeGeneric (node:internal/stream_base_commons:151:3)
    at Socket._writeGeneric (node:net:930:11)
    at Socket._write (node:net:942:8)
    at writeOrBuffer (node:internal/streams/writable:392:12)
    at _write (node:internal/streams/writable:333:10)
    at Socket.Writable.write (node:internal/streams/writable:337:10)
    at Runtime.updateConfig (/nexus/node_modules/miniflare/src/runtime/index.ts:132:26)
    at async Miniflare.#assembleAndUpdateConfig (/nexus/node_modules/miniflare/src/index.ts:790:23)
    at async Miniflare.#init (/nexus/node_modules/miniflare/src/index.ts:472:5) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}

M

matt•7/4/23, 6:38 AM

hi again

I'm getting this error

workerd/jsg/util.c++:276: error: e = kj/compat/tls.c++:215: failed: TLS peer's certificate is not trusted; reason = unable to get local issuer certificate

workerd/jsg/util.c++:276: error: e = kj/compat/tls.c++:215: failed: TLS peer's certificate is not trusted; reason = unable to get local issuer certificate

workerd/jsg/util.c++:276: error: e = kj/compat/tls.c++:215: failed: TLS peer's certificate is not trusted; reason = unable to get local issuer certificate

workerd/jsg/util.c++:276: error: e = kj/compat/tls.c++:215: failed: TLS peer's certificate is not trusted; reason = unable to get local issuer certificate

it only appears when I use service bindings like this:

serviceBindings[bindingName] = req => {
  const smf = new Miniflare(options)
  return smf.dispatchFetch(req)
}

serviceBindings[bindingName] = req => {
  const smf = new Miniflare(options)
  return smf.dispatchFetch(req)
}

serviceBindings[bindingName] = req => {
  const smf = new Miniflare(options)
  return smf.dispatchFetch(req)
}

serviceBindings[bindingName] = req => {
  const smf = new Miniflare(options)
  return smf.dispatchFetch(req)
}

T

Tin Cap•7/7/23, 10:30 PM

I'm trying to understand the future of Miniflare. It sounds like the v3 integrated into Wrangler is going to have substantially less functionality than v2. Is that correct?

WWallslide I've noticed that if I have a DO alarm setup, and I make a change to the DO's co...

M

MrBBotOP•7/10/23, 10:52 AM

Hey!

Yep, this is correct. Alarms are currently on persisted between script reloads. If you need to test your alarm code, you'll want to shorten your timeouts.

TTin Cap I'm trying to understand the future of Miniflare. It sounds like the v3 integrat...

M

MrBBotOP•7/10/23, 11:00 AM

Miniflare is still it's own product, with an API you can use directly (https://github.com/cloudflare/miniflare/blob/tre/packages/miniflare/README.md), but Wrangler is the primary consumer of that API. The big change with Miniflare v3 is that we're using the same

workerd

workerd

runtime as deployed on Cloudflare's network to run your code, instead of reimplementing workers runtime APIs in Node.js. This significantly increases the fidelity of local mode. However, because we're now running your code in a separate process, some of Miniflare v2's features that relied on passing JavaScript references from inside the workers sandbox to other Node.js code are now much more difficult to implement. Namely, the Jest/Vitest testing environments and the

Miniflare#get*()

Miniflare#get*()

Miniflare#get*()

Miniflare#get*() methods. We're actively working on bringing the testing environments back, and are planning to add back deprecated support for

getBindings()

getBindings()

getBindings()

getBindings()/

getKVNamespace()

getKVNamespace()

getKVNamespace()

getKVNamespace()/

getR2Bucket()

getR2Bucket()

getR2Bucket()

getR2Bucket()/

getCaches()

getCaches()

getCaches()

getCaches()/

getDurableObjectNamespace()

getDurableObjectNamespace()

getDurableObjectNamespace()

getDurableObjectNamespace() whilst we figure out a better way to support framework dev servers. https://github.com/cloudflare/miniflare/discussions/588 has some more discussion around this.

RRaindrops When using the Miniflare for testing I get this error on version 3.20230628.0 I...

M

MrBBotOP•7/10/23, 11:04 AM

Miniflare no longer autoloads configuration from

wrangler.toml

wrangler.toml

. That responsibility has been delgated to Wrangler's

unstable_dev()

unstable_dev()

unstable_dev()

unstable_dev() API (https://developers.cloudflare.com/workers/wrangler/api/#unstable_dev). We're working on stabilising this at the moment, and the API is likely to change significantly in the future. For now, I'd recommend translating your

wrangler.toml

wrangler.toml

to Miniflare API options (https://github.com/cloudflare/miniflare/blob/tre/packages/miniflare/README.md). To fix the specific error you're seeing, you'll likely need to add

modules: true

modules: true

modules: true

modules: true, but you may also need to configure

kvNamespaces

kvNamespaces

kvNamespaces

kvNamespaces/

r2Buckets

r2Buckets

r2Buckets

r2Buckets/etc if you're using those.

RRaindrops Thank you so much! It kinda worked, cause now I am getting a new error

M

MrBBotOP•7/10/23, 3:20 PM

Hmmm, are there any other logs above this?

K

katopz•7/11/23, 8:41 AM

weird

npx miniflare

npx miniflare

npx miniflare

npx miniflare fail on my mac ( via

devDependencies

devDependencies

devDependencies

devDependencies also not working)

36 verbose stack Error: could not determine executable to run
36 verbose stack     at getBinFromManifest (/Users/katopz/.nvm/versions/node/v18.16.0/lib/node_modules/npm/node_modules/libnpmexec/lib/get-bin-from-manifest.js:17:23)
36 verbose stack     at exec (/Users/katopz/.nvm/versions/node/v18.16.0/lib/node_modules/npm/node_modules/libnpmexec/lib/index.js:185:15)
36 verbose stack     at async module.exports (/Users/katopz/.nvm/versions/node/v18.16.0/lib/node_modules/npm/lib/cli.js:134:5)
37 verbose pkgid miniflare@3.20230628.0

36 verbose stack Error: could not determine executable to run
36 verbose stack     at getBinFromManifest (/Users/katopz/.nvm/versions/node/v18.16.0/lib/node_modules/npm/node_modules/libnpmexec/lib/get-bin-from-manifest.js:17:23)
36 verbose stack     at exec (/Users/katopz/.nvm/versions/node/v18.16.0/lib/node_modules/npm/node_modules/libnpmexec/lib/index.js:185:15)
36 verbose stack     at async module.exports (/Users/katopz/.nvm/versions/node/v18.16.0/lib/node_modules/npm/lib/cli.js:134:5)
37 verbose pkgid miniflare@3.20230628.0

36 verbose stack Error: could not determine executable to run
36 verbose stack     at getBinFromManifest (/Users/katopz/.nvm/versions/node/v18.16.0/lib/node_modules/npm/node_modules/libnpmexec/lib/get-bin-from-manifest.js:17:23)
36 verbose stack     at exec (/Users/katopz/.nvm/versions/node/v18.16.0/lib/node_modules/npm/node_modules/libnpmexec/lib/index.js:185:15)
36 verbose stack     at async module.exports (/Users/katopz/.nvm/versions/node/v18.16.0/lib/node_modules/npm/lib/cli.js:134:5)
37 verbose pkgid miniflare@3.20230628.0

36 verbose stack Error: could not determine executable to run
36 verbose stack     at getBinFromManifest (/Users/katopz/.nvm/versions/node/v18.16.0/lib/node_modules/npm/node_modules/libnpmexec/lib/get-bin-from-manifest.js:17:23)
36 verbose stack     at exec (/Users/katopz/.nvm/versions/node/v18.16.0/lib/node_modules/npm/node_modules/libnpmexec/lib/index.js:185:15)
36 verbose stack     at async module.exports (/Users/katopz/.nvm/versions/node/v18.16.0/lib/node_modules/npm/lib/cli.js:134:5)
37 verbose pkgid miniflare@3.20230628.0

and

yarn

yarn

also fail, not thing work on my machine. maybe only me

K

katopz•7/11/23, 9:47 AM

oh not only me and this pretty confusing https://community.cloudflare.com/t/help-with-miniflare-cli-on-mac/515217

📚 Miniflare 3 API docs: https://github.com/cloudflare/miniflare/blob/tre/packages/miniflare/README

Similar Threads

Similar Threads

Similar Threads