Anyone know why AI responses keep switching between structed json and broken json strings? I am usin

SSODaLC Anyone know why AI responses keep switching between structed json and broken jso...

V

Viktor•5/30/25, 8:38 PM

I noticed a lot of these errors when not using complete inputs; e.g. just sending a system prompt or not having a user prompt or tool response as last input.

VViktor I noticed a lot of these errors when not using complete inputs; e.g. just sendin...

S

SODaLCOP•5/31/25, 2:05 PM

Thats good to know for the future. I just checked to make sure; i did have a system prompt and user prompt in my input in that order.

S

Shados•6/1/25, 3:51 AM

Out of curiosity, what do folks who use the Cloudflare Worker AI stuff feel about its pricing model? I did some comparisons since we have Azure OpenAI and Cloudflare available at work, and unlike other CF offerings (eg: Cloudflare Workers themselves), it looked surprisingly expensive. The mid range llama models are priced about the same as Azure OpenAI Nano/Mini per million token, give or take (its not an apple to apple comparison), while I expected them to be much cheaper. The 10k neurons a day not withstanding (which, to be fair, does make a decent difference)

Is there something I'm overlooking? I'm just comparing input/output tokens here, so its a pretty naive comparisons, since again its tough to compare "equivalent" models between Llama models and the GPTs.

SShados 👋 Out of curiosity, what do folks who use the Cloudflare Worker AI stuff feel a...

Z

zensin•6/1/25, 7:40 AM

I'm only using them for dev testing because of price and the models being offered are old/outdated. For me to consider using it, I would need up to date models and pricing similar to openrouter.

Z

zensin•6/1/25, 7:48 AM

I expect there to maybe be a slight premium for being close to all the users and not having cold-start issues but the cases where that makes a big difference are few

Z

zensin•6/1/25, 7:51 AM

the AI gateway on the other hand is great, and has a lot more potential to realize.

Y

yinxingmaiming6409•6/1/25, 2:03 PM

Is there a way to obtain the image embedding on CF?

I

implicit-invocation•6/3/25, 3:27 AM

when to expect the new open source models like

gemini-3n

gemini-3n

gemini-3n

gemini-3n and

chatterbox-tts

chatterbox-tts

chatterbox-tts

chatterbox-tts? is there any place to request and vote?

O

octopus8717•6/3/25, 8:20 PM

Hello, I'm using worker with default LLM Chat App Template. Is it possible to connect remote MCP to it?

G

gabrielstuff•6/5/25, 6:09 PM

Hello, is there any roadmap for custom model support ? Like on the fly model download from hugging face ?

G

gabrielstuff•6/5/25, 6:19 PM

The only useful info I’ve come across so far is from here:
https://community.cloudflare.com/t/newer-ai-models-bring-your-own-model/597274
https://github.com/cloudflare/cloudflare-docs/issues/11039
https://developers.cloudflare.com/changelog/

At the moment, “bring your own model” is something very few platforms do well. Among the better ones, I’d say Scaleway Managed Inference and Deepinfra are leading the pack.

Being able to plug in your own model, especially for tasks like completion or classification, is a huge deal. It opens the door to running smaller, highly optimized models, including those trained for other languages or specific domains. And for fine-tuned models, this could really set Cloudflare apart if done right !

GitHub

Ai model deployment, can we deploy our models to cloudflare? · Iss...

Which Cloudflare product(s) does this pertain to? Workers Subject Matter AI workers Content Location https://github.com/cloudflare/cloudflare-docs/blob/production/content/workers-ai/models/embeddin...

Cloudflare Docs

Changelogs

Ggabrielstuff Hello, is there any roadmap for custom model support ? Like on the fly model dow...

I

Isaac McFadyen•6/5/25, 7:55 PM

?workers-ai-models

S

SuperHelpflare•6/5/25, 7:55 PM

Workers AI currently only supports popular open-source models provided by the Cloudflare team, as well as your own LoRAs that can be applied on top of the Cloudflare-provided models. You cannot currently upload your own models or use a model from HuggingFace. See the documentation for the list of Cloudflare-provided models: https://developers.cloudflare.com/workers-ai/models/

S

SuperHelpflare•6/6/25, 8:02 AM

Please do not ping community members for non-moderation reasons. Doing so will not solve your issue faster and will make people less likely to want to help you.

Z

Zig•6/6/25, 6:16 PM

any plans for adding flux kontext?

ZZig any plans for adding flux kontext?

I

Isaac McFadyen•6/6/25, 7:08 PM

At the moment Flux Kontext is not an open-weight model, so I expect they wouldn't add it until if/when it becomes open-weight the same as Flux is.

Z

Zig•6/6/25, 7:09 PM

Thanks sorry I thought it was

ZZig Thanks sorry I thought it was

I

Isaac McFadyen•6/6/25, 7:10 PM

I believe BFL has announced that they plan to make it open-weight (or, well, a lighter version of it the same as they've done with Flux Dev/Schnell vs Pro) but haven't actually done that yet.

IIsaac McFadyen ?workers-ai-models

G

gabrielstuff•6/7/25, 11:14 AM

Yes

A

Advany•6/7/25, 11:31 AM

For agents sdk, using the starter kit, when doing npm update. It started to fail. I had to add

import { defineWorkersConfig } from "@cloudflare/vitest-pool-workers/config";

export default defineWorkersConfig({
environments: {
ssr: {
keepProcessEnv: true,
},
},
test: {
poolOptions: {
workers: {
wrangler: { configPath: "./wrangler.jsonc" },
},
},
},
resolve: {
alias: {
ajv: 'ajv/dist/ajv.min.js',
},
},
optimizeDeps: {
include: ['ajv'],
},
});

To vitest to make it work again!

AAdvany For agents sdk, using the starter kit, when doing npm update. It started to fail...

A

Advany•6/7/25, 11:32 AM

so the way to replicate it is create agents sdk using the starter, then npm update. It will show something like

 FAIL  tests/index.test.ts [ tests/index.test.ts ]
SyntaxError: Unexpected token ':'
 ❯ Users/advany/Documents/GitHub/generator-agent/node_modules/ajv/lib/definition_schema.js?mf_vitest_no_cjs_esm_shim:3:18
 ❯ Users/advany/Documents/GitHub/generator-agent/node_modules/ajv/lib/keyword.js?mf_vitest_no_cjs_esm_shim:5:24
 ❯ Users/advany/Documents/GitHub/generator-agent/node_modules/ajv/lib/ajv.js?mf_vitest_no_cjs_esm_shim:29:21

 FAIL  tests/index.test.ts [ tests/index.test.ts ]
SyntaxError: Unexpected token ':'
 ❯ Users/advany/Documents/GitHub/generator-agent/node_modules/ajv/lib/definition_schema.js?mf_vitest_no_cjs_esm_shim:3:18
 ❯ Users/advany/Documents/GitHub/generator-agent/node_modules/ajv/lib/keyword.js?mf_vitest_no_cjs_esm_shim:5:24
 ❯ Users/advany/Documents/GitHub/generator-agent/node_modules/ajv/lib/ajv.js?mf_vitest_no_cjs_esm_shim:29:21

 FAIL  tests/index.test.ts [ tests/index.test.ts ]
SyntaxError: Unexpected token ':'
 ❯ Users/advany/Documents/GitHub/generator-agent/node_modules/ajv/lib/definition_schema.js?mf_vitest_no_cjs_esm_shim:3:18
 ❯ Users/advany/Documents/GitHub/generator-agent/node_modules/ajv/lib/keyword.js?mf_vitest_no_cjs_esm_shim:5:24
 ❯ Users/advany/Documents/GitHub/generator-agent/node_modules/ajv/lib/ajv.js?mf_vitest_no_cjs_esm_shim:29:21

 FAIL  tests/index.test.ts [ tests/index.test.ts ]
SyntaxError: Unexpected token ':'
 ❯ Users/advany/Documents/GitHub/generator-agent/node_modules/ajv/lib/definition_schema.js?mf_vitest_no_cjs_esm_shim:3:18
 ❯ Users/advany/Documents/GitHub/generator-agent/node_modules/ajv/lib/keyword.js?mf_vitest_no_cjs_esm_shim:5:24
 ❯ Users/advany/Documents/GitHub/generator-agent/node_modules/ajv/lib/ajv.js?mf_vitest_no_cjs_esm_shim:29:21

I fixed it but adding

 resolve: {
    alias: {
      ajv: 'ajv/dist/ajv.min.js',
    },
  },
  optimizeDeps: {
    include: ['ajv'],
  },

 resolve: {
    alias: {
      ajv: 'ajv/dist/ajv.min.js',
    },
  },
  optimizeDeps: {
    include: ['ajv'],
  },

 resolve: {
    alias: {
      ajv: 'ajv/dist/ajv.min.js',
    },
  },
  optimizeDeps: {
    include: ['ajv'],
  },

 resolve: {
    alias: {
      ajv: 'ajv/dist/ajv.min.js',
    },
  },
  optimizeDeps: {
    include: ['ajv'],
  },

LLiam HUGE loss of about 99% of AI requests failing currently...[Last 3 hours]

G

gabrielstuff•6/7/25, 1:58 PM

I can’t manage to run ai workers properly on my side

Ggabrielstuff Yes

I

Isaac McFadyen•6/7/25, 3:02 PM

See the bot message right below mine.

S

Stupefystar•6/8/25, 12:04 PM

CURL to

/ai/run/@cf/meta/llama-3.2-11b-vision-instruct

/ai/run/@cf/meta/llama-3.2-11b-vision-instruct

with LORA >

"errors": [{
        "message": "AiError: AiError: Lora not compatible with model. (c8afa8e1-48d6-42d4-871d-b04bce9b4c67)",
        "code": 3030
    }]

"errors": [{
        "message": "AiError: AiError: Lora not compatible with model. (c8afa8e1-48d6-42d4-871d-b04bce9b4c67)",
        "code": 3030
    }]

"errors": [{
        "message": "AiError: AiError: Lora not compatible with model. (c8afa8e1-48d6-42d4-871d-b04bce9b4c67)",
        "code": 3030
    }]

"errors": [{
        "message": "AiError: AiError: Lora not compatible with model. (c8afa8e1-48d6-42d4-871d-b04bce9b4c67)",
        "code": 3030
    }]

S

Stupefystar•6/8/25, 12:04 PM

S

Stupefystar•6/8/25, 12:05 PM

Creating new finetune "my-finetune-name" for model "@cf/meta/llama-3.2-11b-vision-instruct".
...

Assets uploaded, finetune "my-finetune-name" is ready to use.

IIsaac McFadyen See the bot message right below mine.

G

gabrielstuff•6/9/25, 7:20 AM

Thanks Isaac, per my question, I was asking about a roadmap. Is that something planned, considered or not. The bot successfully answers on the fact that it is not supported.

Thank you !

1

1984 Ford Laser•6/10/25, 10:52 PM

Wondering if there's plans to allow custom/uploaded models in future? Would love some more image detection and action recognition models for some video pipelines Im working on

K

Kraus•6/11/25, 3:53 PM

I keep getting network connection lost errors every few worker ai calls. What would be the reason for this?

S

s•6/12/25, 5:09 AM

Has LORAs support been updated for the models in this announcement? https://blog.cloudflare.com/workers-ai-improvements/

The docs are a bit confusing on this: Llama 3.1 and Mistral Small 3.1 are listed as supported, but the model finder and API docs don't reflect LORA support, and the blog post lists them as soon

The Cloudflare Blog

Workers AI gets a speed boost, batch workload support, more LoRAs, ...

We just made Workers AI inference faster with speculative decoding & prefix caching. Use our new batch inference for handling large request volumes seamlessly. Build tailored AI apps with more LoRA options. Lastly, new models and a refreshed dashboard round out this Developer Week update for Workers AI.

B

bomboclat•6/12/25, 8:10 AM

Yes, it sort of contradicts https://developers.cloudflare.com/workers-ai/features/fine-tunes/loras/ also wondering what soon means
edit: PR submitted

Cloudflare Docs

Using LoRA adapters

Upload and use LoRA adapters to get fine-tuned inference on Workers AI.

C

cipherfunk•6/14/25, 1:05 AM

When will containers be released? Still June? Will that allow us to run custom ai models? Specifically video gen?

B

bomboclat•6/17/25, 5:19 AM

I'm also looking out for it too, the press release mentioned end of this month

A

Adam•6/17/25, 2:58 PM

Hola!

I can't get the caching to work. Any hints here?
The doc page says skipCache:false should do it, but I'm seeing slow response times even with it set to 24hours. Does skipCache:false really work?

https://developers.cloudflare.com/ai-gateway/providers/workersai/

const aiResponse = await env.AI.run(
  '@cf/meta/llama-3.1-8b-instruct',
  {
    messages: [
      {
        role: 'system',
        content: 'You are a helpful assistant that transforms text into different tones and styles.'
      },
      {
        role: 'user',
        content: fullPrompt
      }
    ],
    max_tokens: 2048
  },
  {
    gateway: {
      skipCache: false,   // Enable caching (default behavior)
      cacheTtl: 86400     // Cache for 24 hours (24 * 60 * 60 seconds)
    }
  }
);

const aiResponse = await env.AI.run(
  '@cf/meta/llama-3.1-8b-instruct',
  {
    messages: [
      {
        role: 'system',
        content: 'You are a helpful assistant that transforms text into different tones and styles.'
      },
      {
        role: 'user',
        content: fullPrompt
      }
    ],
    max_tokens: 2048
  },
  {
    gateway: {
      skipCache: false,   // Enable caching (default behavior)
      cacheTtl: 86400     // Cache for 24 hours (24 * 60 * 60 seconds)
    }
  }
);

const aiResponse = await env.AI.run(
  '@cf/meta/llama-3.1-8b-instruct',
  {
    messages: [
      {
        role: 'system',
        content: 'You are a helpful assistant that transforms text into different tones and styles.'
      },
      {
        role: 'user',
        content: fullPrompt
      }
    ],
    max_tokens: 2048
  },
  {
    gateway: {
      skipCache: false,   // Enable caching (default behavior)
      cacheTtl: 86400     // Cache for 24 hours (24 * 60 * 60 seconds)
    }
  }
);

const aiResponse = await env.AI.run(
  '@cf/meta/llama-3.1-8b-instruct',
  {
    messages: [
      {
        role: 'system',
        content: 'You are a helpful assistant that transforms text into different tones and styles.'
      },
      {
        role: 'user',
        content: fullPrompt
      }
    ],
    max_tokens: 2048
  },
  {
    gateway: {
      skipCache: false,   // Enable caching (default behavior)
      cacheTtl: 86400     // Cache for 24 hours (24 * 60 * 60 seconds)
    }
  }
);

Cloudflare Docs

Workers AI

Use AI Gateway for analytics, caching, and security on requests to Workers AI. Workers AI integrates seamlessly with AI Gateway, allowing you to execute AI inference via API requests or through an environment binding for Workers scripts. The binding simplifies the process by routing requests through your AI Gateway with minimal setup.

AAdam Hola! I can't get the caching to work. Any hints here? The doc page says ski...

A

Adam•6/17/25, 4:37 PM

Ok, I get it. The caching only works for gateways. ChatGPT explained it to me.

Some kind of warning or error would have been nice though.

K

Kate6•6/17/25, 10:26 PM

Is there an example somewhere showing the format for the tools array that needs to be passed into runWithTools when using the embedded function calling? I'm trying to call something that doesn't provide an "openAPI" descriptor...

K

Kate6•6/17/25, 10:31 PM

Keep getting "✘ [ERROR] ERROR: Tool undefined not found, maybe AI hallucinated" in my Wrangler output

X

xingguif1111•6/18/25, 3:58 PM

Can cloudflare ai provide some advanced models instead of some outdated models

Xxingguif1111 Can cloudflare ai provide some advanced models instead of some outdated models

D

deleted_user_29bc0398aM3•6/20/25, 1:50 AM

agree

F

fandy_1•6/20/25, 5:49 AM

Hi team, I'm using traditional function call method, but it seems doesn't work for me . No function/tool could be picked.
Sample :
Request payload:
{"model":"@hf/nousresearch/hermes-2-pro-mistral-7b","messages":[{"role":"user","content":"what is the weather in london"}],"stream":false,"tools":[{"name":"getWeather","description":"Return the weather for a latitude and longitude","parameters":{"type":"object","properties":{"latitude":{"type":"string","description":"The latitude for the given location"},"longitude":{"type":"string","description":"The longitude for the given location"}},"required":["latitude","longitude"]}}]}
Response payload:
{"id":"id-1750398103502","object":"chat.completion","created":1750398103,"model":"@hf/nousresearch/hermes-2-pro-mistral-7b","choices":[{"index":0,"message":{"role":"assistant","content":null},"logprobs":null,"finish_reason":"stop"}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

Z

zido•6/21/25, 3:22 AM

Can we add allenai/Molmo-7B-D-0924

Anyone know why AI responses keep switching between structed json and broken json strings? I am usin

Similar Threads