What you get from the API is guaranteed to be valid JSON, but if you ask a LLM to reply with JSON (i

What you get from the API is guaranteed to be valid JSON, but if you ask a LLM to reply with JSON (inside the

response

response

response

response property), it may not work. LLMs are trained to generate text not JSON.

fry69OP•10/25/24, 11:14 AM

(some very recent advanced LLMs have learned a few JSON tricks, but their responses suffer in quality if you use those)

Ffry69 What you get from the API is guaranteed to be valid JSON, but if you ask a LLM t...

ExploringBildr•10/25/24, 11:15 AM

Thanks. Then, I need help to make my API call return a json. Now, it's returning a string?

EExploringBildr Thanks. Then, I need help to make my API call return a json. Now, it's returning...

fry69OP•10/25/24, 11:16 AM

A string is valid JSON, you have to parse it.

ExploringBildr•10/25/24, 11:16 AM

I tried to parse, but because of all the " and , it's not working for me.

ExploringBildr•10/25/24, 11:18 AM

I need to have a way to make the API call return a json without the unneccesary escape characters

EExploringBildr I need to have a way to make the API call return a json without the unneccesary ...

fry69OP•10/25/24, 11:18 AM

I do not see those escape chars when I use the API, see e.g. -> workers-ai

fry69OP•10/25/24, 11:19 AM

jq

jq

is happy so should any JSON parser

Original message was deleted

ExploringBildr•10/25/24, 5:22 PM

Workers AI REST API

Original message was deleted

ExploringBildr•10/25/24, 5:32 PM

curl -X POST \
https://api.cloudflare.com/client/v4/accounts/[....]/run/@cf/mistral/mistral-7b-instruct-v0.1 \
-H "Authorization: Bearer [....]" \
-d '{"messages":[{"role":"system","content":"You are a friendly assistant that helps write stories"},{"role":"user","content":"Write a short story about a llama that goes on a journey to find an orange cloud "}]}'

ExploringBildr•10/26/24, 4:29 AM

It's solved now

blindChicken•10/26/24, 8:26 PM

I'm trying to use Cloudflare worker-AI with langchain to embed some text. Specifically I'm trying to use the

OpenAIEmbeddings

OpenAIEmbeddings

library to send text to my AI-worker. What I noticed is that the

embed_documents

embed_documents

method doesn't send the text to the endpoint provided but rather sends an array of tokens. On the other hand the CF embedding models/methods expects strings as inputs. Anyone know how to resolve this? Is there specific Langchain library for Cloudflare, or is there a way to input tokens into the CF models?

celeste_y•10/28/24, 2:44 AM

In my application, I need to create embeddings for articles and queries in both Chinese and English. However, the embedding models supported by Worker-AI are English-only models. I am thinking about calling out to an external inference API which supports multi-lingual embedding, then insert into/ query the embedding with Cloudfare Vectorize. Before doing that, I want to check and see if there is a more cloudfare native way to support multi-lingual embedding as I would like to keep my whole workflow on Cloudfare if possible. Thank you!

surgery18•10/28/24, 1:28 PM

Any chance of getting suno/bark and other AI TTS models added to available models we can use in workers AI?

pcnoic•10/28/24, 4:41 PM

Hey folks, we're having a weird issue that may or may not be a direct result of the way we're trying to leverage the AI binding in our worker.

We're using the application context to inject the various instances of Cloudflare bindings (we've done this for KV, R2 and multiple other services and now trying for AI).

import { useContext } from '@fc/core/context';

/**
 * Injects the Ai instance into the application context.
 */
export const withAI = (ai: Ai) => {
  const context = useContext();
  if (!context) {
    throw new Error('Context not found: ensure the AI binding is attached.');
  }
  context.dependencies.set('ai', ai);
};

/**
 * Retrieves the Ai instance from the context.
 * Throws an error if Ai is not found.
 */
export const useAI = (): Ai => {
  const context = useContext();
  if (!context) {
    throw new Error('Context not found: useAI must be called within a Cloudflare Worker.');
  }
  const ai = context.dependencies.get('ai') as Ai;
  if (!ai) {
    throw new Error('AI instance not found in context. Ensure `withAI` is used.');
  }
  return ai;
};

import { useContext } from '@fc/core/context';

/**
 * Injects the Ai instance into the application context.
 */
export const withAI = (ai: Ai) => {
  const context = useContext();
  if (!context) {
    throw new Error('Context not found: ensure the AI binding is attached.');
  }
  context.dependencies.set('ai', ai);
};

/**
 * Retrieves the Ai instance from the context.
 * Throws an error if Ai is not found.
 */
export const useAI = (): Ai => {
  const context = useContext();
  if (!context) {
    throw new Error('Context not found: useAI must be called within a Cloudflare Worker.');
  }
  const ai = context.dependencies.get('ai') as Ai;
  if (!ai) {
    throw new Error('AI instance not found in context. Ensure `withAI` is used.');
  }
  return ai;
};

We setup a

withAI

withAI

and

useAI

useAI

method that allows us to inject the binding instance into our workers from a core package.

We would then do something along the lines of:

 await withContext(requestId, async () => {
    const env = envSchema.parse(_env);

    // AI
    withAI(_env.AI);

 await withContext(requestId, async () => {
    const env = envSchema.parse(_env);

    // AI
    withAI(_env.AI);

Where the

Env

Env

interface was generated by

wrangler types

wrangler types

.

Our

_env.AI

_env.AI

in this case is of

 abstract class Ai

 abstract class Ai

as defined in the workers-type pacakge.

But, when we try to do something similar to in a service:

const cloudflareJobOutput = (await ai.run('@cf/openai/whisper', modelInput)) as WhisperResponse;

const cloudflareJobOutput = (await ai.run('@cf/openai/whisper', modelInput)) as WhisperResponse;

we catch an error which is hard to debug

 undefined: undefined

 undefined: undefined

Ppcnoic Hey folks, we're having a weird issue that may or may not be a direct result of ...

pcnoic•10/28/24, 4:42 PM

Any ideas or help would be appreciated.

javm95•10/29/24, 8:21 AM

Does Cloudflare support any models with response_format parameter? I need a model that can respond with a JSON schema.

Ppcnoic Hey folks, we're having a weird issue that may or may not be a direct result of ...

Vero•10/29/24, 12:05 PM

Hey. Do you have a minimal repro?

Martijn•10/30/24, 12:40 PM

Hello guys,

I'm trying to use image classification/object classification however
AiError: 3006: Request is too large
I can't find any documentation on the file sizes and how to increase this. Any help?

MMartijn Hello guys, I'm trying to use image classification/object classification howeve...

fry69OP•10/30/24, 1:18 PM

No documentation I can find and only this community entry -> https://community.cloudflare.com/t/inferenceupstreamerror-for-large-audio-files-2-mb/624759/6?u=fry69 pointing at a ~2MB limit (at least for whisper), which is tiny. I wonder if there is another way to reference objects to include for AI inference.

Otherwise, I'd deal with this limit and try to transform images to a smaller fixed size before sending them (I think APIs like e.g. Gemini do this internally to a 256x256 size anyway)

Rrichw Hi all. I've got my Worker running AI models, which is great. I can see on https...

Kristian•10/30/24, 1:55 PM

Hey there! I have seen the same issue, I'll ask around internally and see what I can find out for you. Thanks for surfacing this

Ffry69 No documentation I can find and only this community entry -> https://community.c...

Kristian•10/30/24, 1:57 PM

Will work on improving documentation here. Thanks!

KKristian Will work on improving documentation here. Thanks!

Martijn•10/30/24, 2:04 PM

Do you know what the limit is? Specifically for the image AIs

MMartijn Do you know what the limit is? Specifically for the image AIs

Kristian•10/30/24, 2:04 PM

Not sure! Working on figuring that out today for ya

Martijn•10/30/24, 2:05 PM

Awesome thanks! Also let me know if there is a way to increase this limit

pigri•10/30/24, 3:29 PM

If anyone else needs an AI mock service, we've created one! You can check it out on GitHub: https://github.com/openshieldai/mockai or access it directly here: https://mockai.openshield.ai. Currently, OpenAI is supported, but we will add more providers.

Ppigri If anyone else needs an AI mock service, we've created one! You can check it out...

fry69OP•10/30/24, 3:30 PM

GitHub link is 404

Ffry69 GitHub link is 404

pigri•10/30/24, 3:38 PM

Thank you! Public now

Ppigri Thank you! Public now

fry69OP•10/30/24, 3:43 PM

Looks great at a quick glance. I'd also announce this in -> #what-i-built

MMartijn Do you know what the limit is? Specifically for the image AIs

Raylight•10/30/24, 4:09 PM

The limit is "fuzzy". Around 2.5-3MB depending on the content. Seems to apply to all models. (

JSON.stringify(inputs).length

JSON.stringify(inputs).length

needs to be less than 9.5 MB, to be precise. I.e. the input to AI.run is stringified at one point internally and must not exceed ~10 million bytes or you'll get "request to large". cc @Kristian)

Rrichw Hi all. I've got my Worker running AI models, which is great. I can see on https...

kian•10/30/24, 4:16 PM

Your error mentions

8b

8b

, but there is no

8b

8b

Llama 3.2 model

kian•10/30/24, 4:17 PM

Sounds like you typo'd the model name, as you mentioned it's

@cf/meta/llama-3.2-3b-instruct

@cf/meta/llama-3.2-3b-instruct

RRaylight The limit is "fuzzy". Around 2.5-3MB depending on the content. Seems to apply to...

Martijn•10/30/24, 4:36 PM

The image is around 3,5mb so you recommend compressing them?

MMartijn The image is around 3,5mb so you recommend compressing them?

Raylight•10/30/24, 5:11 PM

Yeah, you could try that. The limit for compressed/random data is roughly 2.6MB. In my experience, sending smaller images (file size) has resulted in a significant improvement in latency.

RRaylight Yeah, you could try that. The limit for compressed/random data is roughly 2.6MB....

fry69OP•10/30/24, 5:14 PM

These are still gigantic image sizes for LLMs, it should be possible to downscale them to ~100-300kb size and get similar results.

fry69OP•10/30/24, 5:15 PM

Most likely the AI API will scale those images down anyway before processing.

Martijn•10/30/24, 5:37 PM

What format do you recommend to compress them to jpg? With what setting

MMartijn What format do you recommend to compress them to jpg? With what setting

Isaac McFadyen•10/30/24, 5:38 PM

JPEG would work well. Keep in mind that LLMs actually use very small sizes internally anyway - a size of 640 pixels is good enough, no need for 4k images.

Martijn•10/30/24, 5:38 PM

With the object classification from fb i feel like if you resize the box dimensions dont work anymore

Raylight•10/30/24, 5:39 PM

Either way will work. Compressing or downscaling. The main thing is to get the file size down. When I tested I could shave of half a second of latency simply by sending the (1024x1024) image as a ~200kb jpg instead of a 2MB png.

MMartijn With the object classification from fb i feel like if you resize the box dimensi...

Isaac McFadyen•10/30/24, 5:39 PM

The box dimensions should be based on the pixels you give, so it should be consistent?

Martijn•10/30/24, 5:39 PM

Yeah but if I want to then visualize the boxes on the original image it wouldnt match anymore right

RRaylight Either way will work. Compressing or downscaling. The main thing is to get the f...

Isaac McFadyen•10/30/24, 5:39 PM

I'd recommend resizing over compressing. A compressed 4k image is still going to be resized internally by the model anyway, and by aggressively compressing big images you're going to lose some of the details.

MMartijn Yeah but if I want to then visualize the boxes on the original image it wouldnt ...

Isaac McFadyen•10/30/24, 5:40 PM

Just make sure to calculate the resize ratio.

Martijn•10/30/24, 5:40 PM

I guess yeah

Isaac McFadyen•10/30/24, 5:40 PM

If you have an image that is 3840 and resize to 640 then just multiple the returned coordinates by 6 (3840 divided by 640)

Isaac McFadyen•10/30/24, 5:41 PM

Just an FYI that I'm not picking 640 pixels arbitrarily - that's the actual size used by lots of models (ex. YOLO object detection)

Martijn•10/30/24, 5:42 PM

Kind of difficult since the images I want to classify are user input and could differ but I’m sure ill figure it out

MMartijn Kind of difficult since the images I want to classify are user input and could d...

Isaac McFadyen•10/30/24, 5:43 PM

Hmmm yeah. So depending on what your input pipeline looks like, I'd take the longest dimension (height or width) and then divide that by 640, and then resize to 640, send the image, and multiple all results by that ratio

import { useContext } from '@fc/core/context'; /** * Injects the Ai instance into the application context. */ export const withAI = (ai: Ai) => { const context = useContext(); if (!context) { throw new Error('Context not found: ensure the AI binding is attached.'); } context.dependencies.set('ai', ai); }; /** * Retrieves the Ai instance from the context. * Throws an error if Ai is not found. */ export const useAI = (): Ai => { const context = useContext(); if (!context) { throw new Error('Context not found: useAI must be called within a Cloudflare Worker.'); } const ai = context.dependencies.get('ai') as Ai; if (!ai) { throw new Error('AI instance not found in context. Ensure `withAI` is used.'); } return ai; };

What you get from the API is guaranteed to be valid JSON, but if you ask a LLM to reply with JSON (i

Similar Threads