Cloudflare Developers•6d ago

Hey 👋

Hey 👋
I'm unable to create streams with dashes in the name (like the example in https://developers.cloudflare.com/pipelines/streams/manage-streams/):

➜  wrangler pipelines streams create my-stream --schema-file ./schema.json --http-enabled

 ⛅️ wrangler 4.40.2
───────────────────
▲ [WARNING] 🚧 `wrangler pipelines streams create` is an open-beta command. Please report any issues to https://github.com/cloudflare/workers-sdk/issues/new/choose


🌀 Creating stream 'my-stream'...

✘ [ERROR] A request to the Cloudflare API (/accounts/.../pipelines/v1/streams) failed.
  [{"type":"invalid_string","path":"name","message":"Invalid"}] [code: 2]

  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

➜  wrangler pipelines streams create my-stream --schema-file ./schema.json --http-enabled

 ⛅️ wrangler 4.40.2
───────────────────
▲ [WARNING] 🚧 `wrangler pipelines streams create` is an open-beta command. Please report any issues to https://github.com/cloudflare/workers-sdk/issues/new/choose


🌀 Creating stream 'my-stream'...

✘ [ERROR] A request to the Cloudflare API (/accounts/.../pipelines/v1/streams) failed.
  [{"type":"invalid_string","path":"name","message":"Invalid"}] [code: 2]

  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

28 Replies

bertOP•6d ago

I believe it's the same issue reported here: https://discord.com/channels/595317990191398933/1359448593110663248/1421027714546143293 It may be something related to my account - I can't seem to be able to create streams at all:

➜  wrangler pipelines streams create my_stream --schema-file ./schema.json --http-enabled

 ⛅️ wrangler 4.40.2
───────────────────
▲ [WARNING] 🚧 `wrangler pipelines streams create` is an open-beta command. Please report any issues to https://github.com/cloudflare/workers-sdk/issues/new/choose


🌀 Creating stream 'my_stream'...

➜  wrangler pipelines streams create my_stream --schema-file ./schema.json --http-enabled

 ⛅️ wrangler 4.40.2
───────────────────
▲ [WARNING] 🚧 `wrangler pipelines streams create` is an open-beta command. Please report any issues to https://github.com/cloudflare/workers-sdk/issues/new/choose


🌀 Creating stream 'my_stream'...

Nothing happens after this :this:

Marc•6d ago

Yeah currently pipelines has issues with dashes in the name - the team is working on better validation here The second issue is odd let me check behind the scenes real quick

bertOP•6d ago

Thanks Marc. Here's my schema: https://gist.github.com/albt-revolucao/70f9b9ec9b4ee62c61867c888c76a2f0

Gist

gist:70f9b9ec9b4ee62c61867c888c76a2f0

GitHub Gist: instantly share code, notes, and snippets.

Marc•6d ago

Thanks for that. I was able to reproduce ... checking if there's something obvious on our side going on

bertOP•6d ago

I think I figured it out - it's the brands field inside context.userAgentData. If you remove it you'll be able to create the stream.

Marc•6d ago

yup - that's it though this appears to be a bug. Trying a quick workaround for you...

bertOP•6d ago

All good! I don't need that field; was just trying to match Segment's event format. I appreciate your help!

Marc•6d ago

roger - still a weird bug we'll need to fix. This list is being stubborn 🙂

bertOP•6d ago

Marc, sorry for the double ping - this isn't urgent (and can wait until Monday): I think I set everything up correctly, but I'm not seeing events flowing to R2 (bucket is empty other than the metadata file, and trying to select everything using the SQL API returns "Query executed successfully with no results"). In the dashboard I see 340kb of data in but no data delivered, event delivered, or events dropped (except for a few that I intentionally sent with fake data). What's interesting is that I used the cURL provided by the Wrangler CLI to send a few events:

➜  pipe curl -X POST https://[...].ingest.cloudflare.com \
     -H "Authorization: Bearer [...]" \
     -H "Content-Type: application/json" \
     -d '[{
  "timestamp": 1758980652684,
  "type": "sample_type",
  "userId": "sample_userId",
  ...

➜  pipe curl -X POST https://[...].ingest.cloudflare.com \
     -H "Authorization: Bearer [...]" \
     -H "Content-Type: application/json" \
     -d '[{
  "timestamp": 1758980652684,
  "type": "sample_type",
  "userId": "sample_userId",
  ...

And those aren't showing up as well. I'll check later today and will let you know if anything changed.

Marc•6d ago

No worries bert - thanks for putting it through its paces. First thing, re the original issue, it looks like we expect an arbitrary name field inside of items - so this worked:

          "fields": [
            {
              "name": "brands",
              "type": "list",
              "required": false,
              "items": {"name":"things", "type": "json" }
            },

          "fields": [
            {
              "name": "brands",
              "type": "list",
              "required": false,
              "items": {"name":"things", "type": "json" }
            },

for the second thing - can you share the command you used to create the sink?

bertOP•6d ago

Sure thing -

wrangler pipelines sinks create [...]_production_pipelines_sinks_analytics --type r2-data-catalog --bucket [...]-production-r2-analytics --namespace segment --table events --catalog-token [...]

wrangler pipelines sinks create [...]_production_pipelines_sinks_analytics --type r2-data-catalog --bucket [...]-production-r2-analytics --namespace segment --table events --catalog-token [...]

Marc•6d ago

aha There's two settings that control the time it takes for data to show up:

      --roll-size              Roll file size in MB  [number]
      --roll-interval          Roll file interval in seconds  [number] [default: 300]

      --roll-size              Roll file size in MB  [number]
      --roll-interval          Roll file interval in seconds  [number] [default: 300]

so by default, it's set to 5 minutes (300 seconds) - you can set this to be much lower if you'd like. The only issue is we don't support editing a sink yet so you'ld have to delete/recreate

bertOP•6d ago

Right! I don't think this is quite it, though. Check this out:

bertOP•6d ago

bertOP•6d ago

I sent those events ~20-30 minutes ago.

Marc•6d ago

oh 🙂

bertOP•6d ago

I'm pretty sure something's wrong with my schema, but it's interesting that I only see 8 events dropped in the dashboard. (My events aren't 25kb each)

Marc•6d ago

hmm looking 👀

bertOP•6d ago

No worries - happy to chat Monday. This isn't urgent at all.

Marc•6d ago

Ok! the last thing I'll ask you for is the schema - you said you essentially copied and pasted the output from the wrangler command to post the messages? It looks like there was some deserialization errors ... which has me wondering if the wrangler example data it outputs is not correct

bertOP•6d ago

Yep! cURL + schema: https://gist.github.com/albt-revolucao/1f4580d94c70b8aefe6906e6a5c0a6e6

Gist

curl

GitHub Gist: instantly share code, notes, and snippets.

Marc•6d ago

you're awesome thank you yeah this one will need me to do a bit more debugging so it might be a bit - maybe Monday. What's in the gist is valid - I'm assuming whatever was outputted from Wrangler was not Looks like the team identified the issue and has a fix on the way soon. Interestingly enough, the bug was found thanks to specific shape of your schema so thank you trying/finding this (also sorry you found this bug)

bertOP•6d ago

Sweet! No worries. I ended up changing context to json since Segment doesn't specify any format in their documentation and everything's working on my end.

bertOP•6d ago

If I may, one last bug report: for some reason when I cmd+click on "Pipelines", "Streams", or "Sinks" in the sidebar I see this

bertOP•6d ago

If I click to open in the same tab it works.

bertOP•6d ago

This is what I see in the console.

Marc•6d ago

Ooo good find. Will get that one fixed

Micah Wylde•3d ago

Hi Bert, wanted to let you know that we've deployed a fix for that issue. Nested structs should now work properly.

Gaming

Programming

Hey 👋

Did you find this page helpful?