Hi, I'm having an issue where CF AI has completely stopped working from my local environment (yester

Aac Hi, I'm having an issue where CF AI has completely stopped working from my local...

A

acOP•10/16/24, 7:37 PM

To add to this, it is explicility llama-3-8b-instruct. Changing it to a different model (3.1-8b) fixed it. This seems to only happen from my local environment -- production calls to 3-8b seem to still be working fine.

R

Raylight•10/17/24, 7:34 AM

Any chance of higher free allocations (i.e. tokens/steps/units per day) on the Workers Paid plan? Like.. something that's more in line with the spirit of the rest of the Workers platform where the limits are (more than) generous enough to build a simple prototype or follow a tutorial.

Looking at the table (https://developers.cloudflare.com/workers-ai/platform/pricing/#free-allocation), taking image generation as example, you'd easily hit 250 steps just to fix a simple bug or help someone out.

[[FDG] H4kuna Hi, guys, I have a problem and I have a lot of doubts about how I can solve it, ...

K

Keebs•10/17/24, 4:09 PM

identify the main title, offer alternative options to drill down to the user, probably best solution

K

Keebs•10/17/24, 4:09 PM

its like "apple iphone" :D

A

ajgeiss0702•10/19/24, 7:47 PM

any idea when the dashboard will be updated to show the new unit of billing?
Also is there any way to get that now?

S

shoo•10/20/24, 8:42 PM

guys - are beta models still free? (i remember they used to be)
I couldn't find that in documentation anymore

0

0xPh4s3•10/21/24, 11:19 AM

Hey guys, I would like to ask whether the LLM in cf worker AI is full precision or quantized, such as LLaMA3.1:70b

F

fry69•10/21/24, 11:25 AM

?crossposting

F

Flare•10/21/24, 11:25 AM

Please do not post your question in multiple channels/post it multiple times per the rules at ⁠#

welcome-and-rules. It creates confusion for people trying to help you and doesn't get your issue or question solved any faster.

Sshoo guys - are beta models still free? (i remember they used to be) I couldn't find ...

E

Erisa•10/21/24, 11:41 AM

yes

0

0xPh4s3•10/21/24, 11:44 AM

sry

0

0xPh4s3•10/21/24, 11:45 AM

:MeowHeartCloudflare:

C

crazyjack12•10/21/24, 8:26 PM

Any updates on the use any model beta

�

😈 Donkey 💫•10/22/24, 11:39 AM

i believe @cf/meta/llama-3.1-8b-instruct-fast in problem, in several days, it usually returns empty answer

P

Phanindra Reddy•10/22/24, 4:14 PM

hey guys i am to use llama 3.2 but it is getting an error in workers, how can i use it

B

Ben-xD•10/22/24, 8:11 PM

is there a way to check the context length of models hosted on cf?

E

ExploringBildr•10/23/24, 9:58 AM

Hello, I'm trying to set up Worker AI. I got this {"response":"Content-Type invalid"}
Any thoughts?

EExploringBildr Hello, I'm trying to set up Worker AI. I got this {"response":"Content-Type inva...

F

fry69•10/23/24, 10:11 AM

Check if your AI request works with

curl

curl

curl

curl and when it does implement it inside the worker. Example request that works for me ->

curl -X POST \
  https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/ai/run/@cf/meta/llama-3-8b-instruct \
  -H "Authorization: Bearer ${CLOUDFLARE_API_TOKEN}" \
  -d '{
  "messages": [
    {
      "role": "system",
      "content": "You are a friendly assistant that helps write stories"
    },
    {
      "role": "user",
      "content": "Write a short story about a llama that goes on a journey to find an orange cloud "
    }
  ]
}' \
  | jq

curl -X POST \
  https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/ai/run/@cf/meta/llama-3-8b-instruct \
  -H "Authorization: Bearer ${CLOUDFLARE_API_TOKEN}" \
  -d '{
  "messages": [
    {
      "role": "system",
      "content": "You are a friendly assistant that helps write stories"
    },
    {
      "role": "user",
      "content": "Write a short story about a llama that goes on a journey to find an orange cloud "
    }
  ]
}' \
  | jq

curl -X POST \
  https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/ai/run/@cf/meta/llama-3-8b-instruct \
  -H "Authorization: Bearer ${CLOUDFLARE_API_TOKEN}" \
  -d '{
  "messages": [
    {
      "role": "system",
      "content": "You are a friendly assistant that helps write stories"
    },
    {
      "role": "user",
      "content": "Write a short story about a llama that goes on a journey to find an orange cloud "
    }
  ]
}' \
  | jq

curl -X POST \
  https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/ai/run/@cf/meta/llama-3-8b-instruct \
  -H "Authorization: Bearer ${CLOUDFLARE_API_TOKEN}" \
  -d '{
  "messages": [
    {
      "role": "system",
      "content": "You are a friendly assistant that helps write stories"
    },
    {
      "role": "user",
      "content": "Write a short story about a llama that goes on a journey to find an orange cloud "
    }
  ]
}' \
  | jq

E

ExploringBildr•10/23/24, 10:30 AM

OK. I got it working

E

ExploringBildr•10/23/24, 10:30 AM

Where do I add stream: true, max_tokens: 2048,

EExploringBildr Where do I add stream: true, max_tokens: 2048,

F

fry69•10/23/24, 10:35 AM

On the same level as

messages

messages

, in doubt ask any chatbot website to format this correctly for you.

I

InventorAI•10/23/24, 2:05 PM

Hi I am using https://developers.cloudflare.com/workers-ai/models/llama-2-7b-chat-int8/

can anyone please suggest me how can I calculate Input and Output token for price calculation

Cloudflare Docs

llama-2-7b-chat-int8 | Cloudflare Workers AI docs

Quantized (int8) generative text model with 7 billion parameters from Meta

IInventorAI Hi I am using https://developers.cloudflare.com/workers-ai/models/llama-2-7b-cha...

F

fry69•10/23/24, 2:07 PM

See here -> https://developers.cloudflare.com/workers-ai/platform/pricing/#pricing-structure

Cloudflare Docs

Pricing | Cloudflare Workers AI docs

Workers AI is included in both the Free and Paid Workers plans and is priced based on model task, model size, and units.

Ffry69 See here -> https://developers.cloudflare.com/workers-ai/platform/pricing/#prici...

I

InventorAI•10/23/24, 2:10 PM

Please suggest me programeticaly how can calculate the price when any API call perform

IInventorAI Please suggest me programeticaly how can calculate the price when any API call p...

F

fry69•10/23/24, 2:13 PM

That would require tokenization which is different for every LLM. If you only need one model it might work, but if you plan to switch between models and even providers, you might need to reach out for something like -> https://www.helicone.ai/

Ffry69 That would require tokenization which is different for every LLM. If you only ne...

I

InventorAI•10/24/24, 7:18 AM

Hi right now I am working on https://developers.cloudflare.com/workers-ai/models/llama-2-7b-chat-int8/
This is from cloudflare, so I want to calculation of token for this CF model

Cloudflare Docs

llama-2-7b-chat-int8 | Cloudflare Workers AI docs

Quantized (int8) generative text model with 7 billion parameters from Meta

IInventorAI Hi right now I am working on https://developers.cloudflare.com/workers-ai/models...

F

fry69•10/24/24, 7:25 AM

Well, google for a LLama 2 tokenizer then, e.g. -> https://github.com/belladoreai/llama-tokenizer-js (or use the official Python Sentencepiece tokenizer like this -> https://github.com/meta-llama/llama/blob/main/llama/tokenizer.py)

GitHub

GitHub - belladoreai/llama-tokenizer-js: JS tokenizer for LLaMA 1 a...

JS tokenizer for LLaMA 1 and 2. Contribute to belladoreai/llama-tokenizer-js development by creating an account on GitHub.

Ffry69 Well, google for a LLama 2 tokenizer then, e.g. -> https://github.com/belladorea...

I

InventorAI•10/24/24, 7:30 AM

Is Cloudflare not providing anything related to this?

IInventorAI Is Cloudflare not providing anything related to this?

F

fry69•10/24/24, 7:39 AM

Nope. Cloudflare AI does not even return the token amount in the response or anywhere else as far as I can see.

�

😈 Donkey 💫•10/24/24, 11:12 AM

flux is weird, usually happens

K

kcbrewron•10/24/24, 2:41 PM

Good morning, periodically when I run batches of updates to content using vectorize,I receive an error AiError: 3001: Unknown interal error. How can I get more insights on what's going on?

Could this be too many requests on the AI requests?
For embedding

F

fry69•10/24/24, 2:42 PM

?crossposting

F

Flare•10/24/24, 2:42 PM

Please do not post your question in multiple channels/post it multiple times per the rules at ⁠#

welcome-and-rules. It creates confusion for people trying to help you and doesn't get your issue or question solved any faster.

E

ExploringBildr•10/25/24, 11:09 AM

Hello, the response I'm getting isn't in an accurate json format. Anyone knows how to fix it?

EExploringBildr Hello, the response I'm getting isn't in an accurate json format. Anyone knows h...

F

fry69•10/25/24, 11:12 AM

What you get from the API is guaranteed to be valid JSON, but if you ask a LLM to reply with JSON (inside the

response

response

property), it may not work. LLMs are trained to generate text not JSON.

F

fry69•10/25/24, 11:14 AM

(some very recent advanced LLMs have learned a few JSON tricks, but their responses suffer in quality if you use those)

Ffry69 What you get from the API is guaranteed to be valid JSON, but if you ask a LLM t...

E

ExploringBildr•10/25/24, 11:15 AM

Thanks. Then, I need help to make my API call return a json. Now, it's returning a string?

EExploringBildr Thanks. Then, I need help to make my API call return a json. Now, it's returning...

F

fry69•10/25/24, 11:16 AM

A string is valid JSON, you have to parse it.

E

ExploringBildr•10/25/24, 11:16 AM

I tried to parse, but because of all the " and , it's not working for me.

E

ExploringBildr•10/25/24, 11:18 AM

I need to have a way to make the API call return a json without the unneccesary escape characters

EExploringBildr I need to have a way to make the API call return a json without the unneccesary ...

F

fry69•10/25/24, 11:18 AM

I do not see those escape chars when I use the API, see e.g. -> workers-ai

F

fry69•10/25/24, 11:19 AM

If

jq

jq

jq

jq is happy so should any JSON parser

Original message was deleted

E

ExploringBildr•10/25/24, 5:22 PM

Workers AI REST API

Original message was deleted

E

ExploringBildr•10/25/24, 5:32 PM

curl -X POST \
https://api.cloudflare.com/client/v4/accounts/[....]/run/@cf/mistral/mistral-7b-instruct-v0.1 \
-H "Authorization: Bearer [....]" \
-d '{"messages":[{"role":"system","content":"You are a friendly assistant that helps write stories"},{"role":"user","content":"Write a short story about a llama that goes on a journey to find an orange cloud "}]}'

E

ExploringBildr•10/26/24, 4:29 AM

It's solved now

B

blindChicken•10/26/24, 8:26 PM

I'm trying to use Cloudflare worker-AI with langchain to embed some text. Specifically I'm trying to use the

OpenAIEmbeddings

OpenAIEmbeddings

OpenAIEmbeddings

OpenAIEmbeddings library to send text to my AI-worker. What I noticed is that the

embed_documents

embed_documents

embed_documents

embed_documents method doesn't send the text to the endpoint provided but rather sends an array of tokens. On the other hand the CF embedding models/methods expects strings as inputs. Anyone know how to resolve this? Is there specific Langchain library for Cloudflare, or is there a way to input tokens into the CF models?

C

celeste_y•10/28/24, 2:44 AM

In my application, I need to create embeddings for articles and queries in both Chinese and English. However, the embedding models supported by Worker-AI are English-only models. I am thinking about calling out to an external inference API which supports multi-lingual embedding, then insert into/ query the embedding with Cloudfare Vectorize. Before doing that, I want to check and see if there is a more cloudfare native way to support multi-lingual embedding as I would like to keep my whole workflow on Cloudfare if possible. Thank you!

S

surgery18•10/28/24, 1:28 PM

Any chance of getting suno/bark and other AI TTS models added to available models we can use in workers AI?

P

pcnoic•10/28/24, 4:41 PM

Hey folks, we're having a weird issue that may or may not be a direct result of the way we're trying to leverage the AI binding in our worker.

We're using the application context to inject the various instances of Cloudflare bindings (we've done this for KV, R2 and multiple other services and now trying for AI).

import { useContext } from '@fc/core/context';

/**
 * Injects the Ai instance into the application context.
 */
export const withAI = (ai: Ai) => {
  const context = useContext();
  if (!context) {
    throw new Error('Context not found: ensure the AI binding is attached.');
  }
  context.dependencies.set('ai', ai);
};

/**
 * Retrieves the Ai instance from the context.
 * Throws an error if Ai is not found.
 */
export const useAI = (): Ai => {
  const context = useContext();
  if (!context) {
    throw new Error('Context not found: useAI must be called within a Cloudflare Worker.');
  }
  const ai = context.dependencies.get('ai') as Ai;
  if (!ai) {
    throw new Error('AI instance not found in context. Ensure `withAI` is used.');
  }
  return ai;
};

import { useContext } from '@fc/core/context';

/**
 * Injects the Ai instance into the application context.
 */
export const withAI = (ai: Ai) => {
  const context = useContext();
  if (!context) {
    throw new Error('Context not found: ensure the AI binding is attached.');
  }
  context.dependencies.set('ai', ai);
};

/**
 * Retrieves the Ai instance from the context.
 * Throws an error if Ai is not found.
 */
export const useAI = (): Ai => {
  const context = useContext();
  if (!context) {
    throw new Error('Context not found: useAI must be called within a Cloudflare Worker.');
  }
  const ai = context.dependencies.get('ai') as Ai;
  if (!ai) {
    throw new Error('AI instance not found in context. Ensure `withAI` is used.');
  }
  return ai;
};

import { useContext } from '@fc/core/context';

/**
 * Injects the Ai instance into the application context.
 */
export const withAI = (ai: Ai) => {
  const context = useContext();
  if (!context) {
    throw new Error('Context not found: ensure the AI binding is attached.');
  }
  context.dependencies.set('ai', ai);
};

/**
 * Retrieves the Ai instance from the context.
 * Throws an error if Ai is not found.
 */
export const useAI = (): Ai => {
  const context = useContext();
  if (!context) {
    throw new Error('Context not found: useAI must be called within a Cloudflare Worker.');
  }
  const ai = context.dependencies.get('ai') as Ai;
  if (!ai) {
    throw new Error('AI instance not found in context. Ensure `withAI` is used.');
  }
  return ai;
};

import { useContext } from '@fc/core/context';

/**
 * Injects the Ai instance into the application context.
 */
export const withAI = (ai: Ai) => {
  const context = useContext();
  if (!context) {
    throw new Error('Context not found: ensure the AI binding is attached.');
  }
  context.dependencies.set('ai', ai);
};

/**
 * Retrieves the Ai instance from the context.
 * Throws an error if Ai is not found.
 */
export const useAI = (): Ai => {
  const context = useContext();
  if (!context) {
    throw new Error('Context not found: useAI must be called within a Cloudflare Worker.');
  }
  const ai = context.dependencies.get('ai') as Ai;
  if (!ai) {
    throw new Error('AI instance not found in context. Ensure `withAI` is used.');
  }
  return ai;
};

We setup a

withAI

withAI

withAI

withAI and

useAI

useAI

useAI

useAI method that allows us to inject the binding instance into our workers from a core package.

We would then do something along the lines of:

 await withContext(requestId, async () => {
    const env = envSchema.parse(_env);

    // AI
    withAI(_env.AI);

 await withContext(requestId, async () => {
    const env = envSchema.parse(_env);

    // AI
    withAI(_env.AI);

 await withContext(requestId, async () => {
    const env = envSchema.parse(_env);

    // AI
    withAI(_env.AI);

 await withContext(requestId, async () => {
    const env = envSchema.parse(_env);

    // AI
    withAI(_env.AI);

Where the

Env

Env

Env

Env interface was generated by

wrangler types

wrangler types

wrangler types

wrangler types.

Our

_env.AI

_env.AI

_env.AI

_env.AI in this case is of

 abstract class Ai

abstract class Ai

 abstract class Ai

abstract class Ai as defined in the workers-type pacakge.

But, when we try to do something similar to in a service:

const cloudflareJobOutput = (await ai.run('@cf/openai/whisper', modelInput)) as WhisperResponse;

const cloudflareJobOutput = (await ai.run('@cf/openai/whisper', modelInput)) as WhisperResponse;

const cloudflareJobOutput = (await ai.run('@cf/openai/whisper', modelInput)) as WhisperResponse;

const cloudflareJobOutput = (await ai.run('@cf/openai/whisper', modelInput)) as WhisperResponse;

we catch an error which is hard to debug

 undefined: undefined

 undefined: undefined

 undefined: undefined

 undefined: undefined

Hi, I'm having an issue where CF AI has completely stopped working from my local environment (yester

Similar Threads

Similar Threads

Similar Threads