Hi, I have a problem with Cloudflare

kNite · 2024-05-29T00:19:24.686Z

Hi, I have a problem with Cloudflare Workers + Hono + Cloudflare AI, I have a route defined (i.e. /v1/chat/completions) and when the user triggers that route I try to run env.AI.run([model], [etc]), but I get this error: `TypeError: Cannot read properties of undefined (reading 'AI')`. This is my Env interface: ```typescript interface Env { AI: any; } const env = (globalThis as any).env as Env ``` Also tried removing the `const env = [...]` and running AI inside a function with `function exampleFunction(env: Env)` but it also didn't work.

J

James•5/29/24, 12:27 AM

env

env

isn't available on

globalThis

globalThis

. It's in your

fetch

fetch

handler, or in the case of Hono, available on

context.env

context.env

.

K

kNiteOP•5/29/24, 12:27 AM

So I could call context.env.AI?

J

James•5/29/24, 12:28 AM

Yep. Something like this (pseudo):

type Bindings = {
  AI: Ai;
}

const app = new Hono<{ Bindings: Bindings }>()

app.get('/foo', async (context) => {
  const data = await context.env.AI.run(...);

  // do something
})

export default app

type Bindings = {
  AI: Ai;
}

const app = new Hono<{ Bindings: Bindings }>()

app.get('/foo', async (context) => {
  const data = await context.env.AI.run(...);

  // do something
})

export default app

type Bindings = {
  AI: Ai;
}

const app = new Hono<{ Bindings: Bindings }>()

app.get('/foo', async (context) => {
  const data = await context.env.AI.run(...);

  // do something
})

export default app

type Bindings = {
  AI: Ai;
}

const app = new Hono<{ Bindings: Bindings }>()

app.get('/foo', async (context) => {
  const data = await context.env.AI.run(...);

  // do something
})

export default app

K

kNiteOP•5/29/24, 12:28 AM

Thank you very much

K

kNiteOP•5/29/24, 12:28 AM

gonna try rn

J

James•5/29/24, 12:29 AM

https://hono.dev/getting-started/cloudflare-workers is a great docs reference if you hit any issues

K

kNiteOP•5/29/24, 12:32 AM

okay now the function is working, but Typescript says that '@cf/tinyllama/tinyllama-1.1b-chat-v1.0' (example model) isn't any of the posible models (

""@cf/unum/uform-gen2-qwen-500m" | "@cf/llava-hf/llava-1.5-7b-hf""

""@cf/unum/uform-gen2-qwen-500m" | "@cf/llava-hf/llava-1.5-7b-hf""

)

KkNite okay now the function is working, but Typescript says that '@cf/tinyllama/tinyll...

I

Isaac McFadyen•5/29/24, 12:32 AM

Make sure your version of

@cloudflare/workers-types

@cloudflare/workers-types

is up to date in your package.json

K

kNiteOP•5/29/24, 12:33 AM

just updated

K

kNiteOP•5/29/24, 12:35 AM

yeah i'm on the latest version

K

kNiteOP•5/29/24, 12:41 AM

okay this is weird

K

kNiteOP•5/29/24, 12:41 AM

if I deploy the code it just works????

K

kNiteOP•5/29/24, 12:41 AM

but TypeScript says it shouldn't

J

James•5/29/24, 12:41 AM

the types for Workers AI can be a little weird. Can you share what you're passing to

AI.run()

AI.run()

?

JJames the types for Workers AI can be a little weird. Can you share what you're passin...

K

kNiteOP•5/29/24, 12:42 AM

sure, one moment

K

kNiteOP•5/29/24, 12:44 AM

const answer = await c.env.AI.run('@cf/tinyllama/tinyllama-1.1b-chat-v1.0', {
      messages: [
        {
          role: "system",
          content: `[very large system prompt]`
        },
        {
          role: "user",
          content: `How many pounds of food does a person eat in a year?`
        }
      ],
      stream: false
})

const answer = await c.env.AI.run('@cf/tinyllama/tinyllama-1.1b-chat-v1.0', {
      messages: [
        {
          role: "system",
          content: `[very large system prompt]`
        },
        {
          role: "user",
          content: `How many pounds of food does a person eat in a year?`
        }
      ],
      stream: false
})

const answer = await c.env.AI.run('@cf/tinyllama/tinyllama-1.1b-chat-v1.0', {
      messages: [
        {
          role: "system",
          content: `[very large system prompt]`
        },
        {
          role: "user",
          content: `How many pounds of food does a person eat in a year?`
        }
      ],
      stream: false
})

const answer = await c.env.AI.run('@cf/tinyllama/tinyllama-1.1b-chat-v1.0', {
      messages: [
        {
          role: "system",
          content: `[very large system prompt]`
        },
        {
          role: "user",
          content: `How many pounds of food does a person eat in a year?`
        }
      ],
      stream: false
})

this is literally everything

K

kNiteOP•5/29/24, 12:45 AM

and after that I have a

return c.json({ response: answer })

return c.json({ response: answer })

K

kNiteOP•5/29/24, 12:45 AM

I don't even look for messages from the user because I need it to work first lol

KkNite ```typescript const answer = await c.env.AI.run('@cf/tinyllama/tinyllama-1.1b-ch...

K

kNiteOP•5/29/24, 12:46 AM

edit: stream is a boolean constant that I set to false, I edited it for clarity

J

James•5/29/24, 12:48 AM

something isn't matching up to hit this function overload: https://github.com/cloudflare/workerd/blob/b3c613b9a5e3265a7e468e7c62d1a14df1e057df/types/defines/ai.d.ts#L197

J

James•5/29/24, 12:49 AM

I can't see exactly what in your snippet, but something isn't quite accurate. At runtime, it's likely just being ignored, but the types get more angry

K

kNiteOP•5/29/24, 12:50 AM

I got into a wrangler tail console, and this is what I got from the worker:

@ 28/5/2024, 21:48:32
  (log) Model: @cf/tinyllama/tinyllama-1.1b-chat-v1.0 ---> debug console.log
  (error) AiError: 5006: must have required property 'prompt', must be boolean, must be array, must be boolean, must match exactly one schema in oneOf

@ 28/5/2024, 21:48:32
  (log) Model: @cf/tinyllama/tinyllama-1.1b-chat-v1.0 ---> debug console.log
  (error) AiError: 5006: must have required property 'prompt', must be boolean, must be array, must be boolean, must match exactly one schema in oneOf

@ 28/5/2024, 21:48:32
  (log) Model: @cf/tinyllama/tinyllama-1.1b-chat-v1.0 ---> debug console.log
  (error) AiError: 5006: must have required property 'prompt', must be boolean, must be array, must be boolean, must match exactly one schema in oneOf

@ 28/5/2024, 21:48:32
  (log) Model: @cf/tinyllama/tinyllama-1.1b-chat-v1.0 ---> debug console.log
  (error) AiError: 5006: must have required property 'prompt', must be boolean, must be array, must be boolean, must match exactly one schema in oneOf

K

kNiteOP•5/29/24, 12:51 AM

maybe messages isn't supported anymore?

K

kNiteOP•5/29/24, 12:51 AM

and now I have to use prompt

K

kNiteOP•5/29/24, 12:51 AM

lol

K

kNiteOP•5/29/24, 12:51 AM

I don't think so

J

James•5/29/24, 12:52 AM

I'd recommend running with

wrangler dev

wrangler dev

locally for testing - that should be much quicker for testing

I

Isaac McFadyen•5/29/24, 12:53 AM

So

prompt

prompt

is for when you're doing raw prompt input yourself - for example, if you're manually formatting ChatML.

I

Isaac McFadyen•5/29/24, 12:53 AM

You should be able to use

messages

messages

for that model though, odd that the types aren't allowing you to.

JJames I'd recommend running with `wrangler dev` locally for testing - that should be m...

K

kNiteOP•5/29/24, 12:53 AM

I tried to use it, but then I remembered that I'm using postman for testing, so I couldn't use 127.0.0.1.

KkNite I tried to use it, but then I remembered that I'm using postman for testing, so ...

K

kNiteOP•5/29/24, 12:54 AM

the web version of postman, to clarify

IIsaac McFadyen You should be able to use `messages` for that model though, odd that the types a...

K

kNiteOP•5/29/24, 12:55 AM

Time to move to JavaScript lol

J

James•5/29/24, 12:58 AM

cc @michelle @Logan Grasby some good feedback in here about the types being really difficult to use/debug with them relying on function overloads

IIsaac McFadyen You should be able to use `messages` for that model though, odd that the types a...

R

Raylight•5/29/24, 7:44 AM

Those errors can become really weird. I got hit with

AiError: 5006: must have required property 'prompt', must be >= 1, must be >= 1, must match exactly one schema in oneOf

AiError: 5006: must have required property 'prompt', must be >= 1, must be >= 1, must match exactly one schema in oneOf

AiError: 5006: must have required property 'prompt', must be >= 1, must be >= 1, must match exactly one schema in oneOf

AiError: 5006: must have required property 'prompt', must be >= 1, must be >= 1, must match exactly one schema in oneOf when adding "seed: 0". (Trying to tell me that seed must be >= 1.)

Hi, I have a problem with Cloudflare

Similar Threads

Similar Threads

Similar Threads