Beta models are free rn afaik

J

James•5/13/24, 1:42 PM

It's quite disappointing to see the AI team's approach to backwards compatibility thus far, going against all of the great paradigms the Workers runtime has enforced for years.

Some examples:

https://github.com/cloudflare/workerd/pull/2044
- it took me quite some time to argue the point about this being a breaking change, despite pushback from multiple team members and saying the risk was "non-existent". Turns out, after this was merged, it broke things and had to be partially reverted
https://github.com/cloudflare/workerd/pull/2103#discussion_r1598483980
- another recent example that was just merged. I truly hope this doesn't break anyone's applications when it rolls out

As a long-time Workers user, I'm a massive fan of their compat guarantees as outlined at https://blog.cloudflare.com/backwards-compatibility-in-cloudflare-workers, and it seems the AI team isn't taking this very seriously for a GA product. There's precedent for other bindings implementing breaking changes behind compat flags, too: https://github.com/cloudflare/workerd/pull/1297

A change to the Workers Runtime must never break an application that is live in production.

.

.neurorotic.•5/13/24, 4:21 PM

The docs seem out of date, there is no "Account details section contains your Account ID" section on workers and pages ->overview https://developers.cloudflare.com/fundamentals/setup/find-account-and-zone-ids/#find-account-id-workers-and-pages
And the only way I found to find it is by creating a domain (even though I did not intend to register any w/ CF) but then finally noticed the account_id is after the dash.cloudflare.com/{ACCOUNT_ID}/ on the site

This could be much clearer for first-time users.

Rrob Beta models are free rn afaik

R

rachelswany•5/13/24, 4:47 PM

Is there a known limit?

M

Mrinank•5/13/24, 6:05 PM

const stream = await retrivalChain.stream({
input: 'what is hello world',
});

return new Response(stream, {
    headers: {
    'content-type': 'text/event-stream',
    'Access-Control-Allow-Origin': '*',
    },
});

const stream = await retrivalChain.stream({
input: 'what is hello world',
});

return new Response(stream, {
    headers: {
    'content-type': 'text/event-stream',
    'Access-Control-Allow-Origin': '*',
    },
});

const stream = await retrivalChain.stream({
input: 'what is hello world',
});

return new Response(stream, {
    headers: {
    'content-type': 'text/event-stream',
    'Access-Control-Allow-Origin': '*',
    },
});

const stream = await retrivalChain.stream({
input: 'what is hello world',
});

return new Response(stream, {
    headers: {
    'content-type': 'text/event-stream',
    'Access-Control-Allow-Origin': '*',
    },
});

is this the correct way to send streams from cloudflare worker?
getting typescript error,

Argument of type 'IterableReadableStream<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to parameter of type 'BodyInit | null | undefined'.
  Type 'IterableReadableStream<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to type 'ReadableStream<Uint8Array>'.
    The types returned by 'getReader()' are incompatible between these types.
      Type 'ReadableStreamDefaultReader<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to type 'ReadableStreamDefaultReader<Uint8Array>'.
        Type '{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }' is missing the following properties from type 'Uint8Array': BYTES_PER_ELEMENT, buffer, byteLength, byteOffset, and 27 more.ts(2345)
const stream: IterableReadableStream<{
    context: Document<Record<string, any>>[];
    answer: string;
} & {
    [key: string]: unknown;
}>

Argument of type 'IterableReadableStream<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to parameter of type 'BodyInit | null | undefined'.
  Type 'IterableReadableStream<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to type 'ReadableStream<Uint8Array>'.
    The types returned by 'getReader()' are incompatible between these types.
      Type 'ReadableStreamDefaultReader<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to type 'ReadableStreamDefaultReader<Uint8Array>'.
        Type '{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }' is missing the following properties from type 'Uint8Array': BYTES_PER_ELEMENT, buffer, byteLength, byteOffset, and 27 more.ts(2345)
const stream: IterableReadableStream<{
    context: Document<Record<string, any>>[];
    answer: string;
} & {
    [key: string]: unknown;
}>

Argument of type 'IterableReadableStream<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to parameter of type 'BodyInit | null | undefined'.
  Type 'IterableReadableStream<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to type 'ReadableStream<Uint8Array>'.
    The types returned by 'getReader()' are incompatible between these types.
      Type 'ReadableStreamDefaultReader<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to type 'ReadableStreamDefaultReader<Uint8Array>'.
        Type '{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }' is missing the following properties from type 'Uint8Array': BYTES_PER_ELEMENT, buffer, byteLength, byteOffset, and 27 more.ts(2345)
const stream: IterableReadableStream<{
    context: Document<Record<string, any>>[];
    answer: string;
} & {
    [key: string]: unknown;
}>

Argument of type 'IterableReadableStream<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to parameter of type 'BodyInit | null | undefined'.
  Type 'IterableReadableStream<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to type 'ReadableStream<Uint8Array>'.
    The types returned by 'getReader()' are incompatible between these types.
      Type 'ReadableStreamDefaultReader<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to type 'ReadableStreamDefaultReader<Uint8Array>'.
        Type '{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }' is missing the following properties from type 'Uint8Array': BYTES_PER_ELEMENT, buffer, byteLength, byteOffset, and 27 more.ts(2345)
const stream: IterableReadableStream<{
    context: Document<Record<string, any>>[];
    answer: string;
} & {
    [key: string]: unknown;
}>

MMrinank ``` const stream = await retrivalChain.stream({ input: 'what is hello world', })...

M

Mrinank•5/13/24, 10:48 PM

i am using import { createRetrievalChain } from 'langchain/chains/retrieval';

S

scotto•5/14/24, 8:55 PM

whister should work with webm format? in the openai docs says it's supported but testing agains the API from cloudflare i see an error, Invalid or incomplete input for the model: model returned: Invalid audio input: F

C

codebam•5/14/24, 10:40 PM

is there a way to wrap

env.AI.run()

env.AI.run()

env.AI.run()

env.AI.run() in

ctx.passThroughOnException

ctx.passThroughOnException

ctx.passThroughOnException

ctx.passThroughOnException while still saving the result?

C

codebam•5/14/24, 10:44 PM

I don't actually understand how

passThroughOnException

passThroughOnException

passThroughOnException

passThroughOnException works. I just want my worker to respond properly when the models time out

J

James•5/14/24, 11:17 PM

you'd probably want to just:

try {
    await env.AI.run(...)
} catch (e) {
    return new Response('handle errors here')
}

try {
    await env.AI.run(...)
} catch (e) {
    return new Response('handle errors here')
}

try {
    await env.AI.run(...)
} catch (e) {
    return new Response('handle errors here')
}

try {
    await env.AI.run(...)
} catch (e) {
    return new Response('handle errors here')
}

and then consider retrying with errors if it makes sense.

J

James•5/14/24, 11:18 PM

passThroughOnException

passThroughOnException

passThroughOnException

passThroughOnException is more intended for if you have an origin behind the worker

J

James•5/14/24, 11:18 PM

so that rather than showing an error page, it'll just send the request as-is to your origin. This is often referred to as a "fail-open" design

JJames you'd probably want to just: ```js try { await env.AI.run(...) } catch (e) {...

C

codebam•5/14/24, 11:34 PM

oh okay. I'll try this, thank you

S

Shanmukeshwar•5/15/24, 6:54 AM

Workers AI are very slow avg: 10sec

S

Shanmukeshwar•5/15/24, 6:54 AM

Any tweaks we can do here?

S

Shanmukeshwar•5/15/24, 6:56 AM

Hey @James
Workers AI are very slow avg: 10sec
Any tweaks we can do here?

Thanks

SShanmukeshwar Hey @James Workers AI are very slow avg: 10sec Any tweaks we can do here? Than...

C

codebam•5/15/24, 9:28 AM

if it's a text generation model you can lower the

max_tokens

max_tokens

C

codebam•5/15/24, 9:28 AM

you'll get shorter replies though

Ccodebam if it's a text generation model you can lower the `max_tokens`

S

Shanmukeshwar•5/15/24, 9:28 AM

I managed the UI to stream the response

S

Shanmukeshwar•5/15/24, 9:29 AM

Thanks for the reply

R

raj.js•5/15/24, 5:22 PM

How much is prompt limit for Llama 3 model? Is there any model on cloudfalre which has atleast 8k token limit

Rraj.js How much is prompt limit for Llama 3 model? Is there any model on cloudfalre whi...

C

codebam•5/15/24, 7:59 PM

4096 characters. no

I

Isaac McFadyen•5/16/24, 2:37 AM

^ just to clarify this: that's tokens, not characters.

I

Isaac McFadyen•5/16/24, 2:38 AM

A token is almost always more than 1 character (to see this for yourself, you can put your text in here, although note that this is not the same tokenizer that the Workers AI models use: https://platform.openai.com/tokenizer)

I

Isaac McFadyen•5/16/24, 2:38 AM

But yes, 4096 tokens maximum (for Mistral v0.2).

A

AndyJessop•5/16/24, 2:05 PM

Does anyone know where I can find a price for Llama3-8b-instruct? There's no pricing for it on the models page, and it's not in the dropdown on the calculator.

https://developers.cloudflare.com/workers-ai/platform/pricing/#text-generation
https://ai.cloudflare.com/#pricing-calculator

AAndyJessop Does anyone know where I can find a price for Llama3-8b-instruct? There's no pri...

I

Isaac McFadyen•5/16/24, 2:21 PM

https://developers.cloudflare.com/workers-ai/platform/pricing/ It's still beta so unbilled currently.

I

Isaac McFadyen•5/16/24, 2:21 PM

>You can still enjoy unlimited usage on the beta models in the catalog until they graduate out of beta.

IIsaac McFadyen >You can still enjoy unlimited usage on the beta models in the catalog until the...

A

AndyJessop•5/16/24, 2:26 PM

That's great news, I thought I was going to bre in for a huge bill...

M

Mrinank•5/16/24, 2:28 PM

does langsmith works with workers AI?

Sscotto whister should work with webm format? in the openai docs says it's supported but...

S

scotto•5/16/24, 10:41 PM

news?

I

Isaac McFadyen•5/16/24, 11:34 PM

Note that depending on your source of WebM files (if it's the browser microphone) you can take a look at that link I sent to force the browser to output a more standardized format

IIsaac McFadyen But yes, 4096 tokens maximum (for Mistral v0.2).

R

Raylight•5/17/24, 2:36 PM

The limit for Llama 3 (on Cloudflare) seems to be ~2800 tokens though. The model stops mid-sentence when prompt/messages + generated text goes above that limit.

RRaylight The limit for Llama 3 (on Cloudflare) seems to be ~2800 tokens though. The model...

I

Isaac McFadyen•5/17/24, 2:40 PM

Interesting - doesn't seem to be listed on the model page but the limit for LLaMA 2 is 2048 tokens so perhaps they share similar limits? https://developers.cloudflare.com/workers-ai/models/llama-2-7b-chat-int8/#properties

I

Isaac McFadyen•5/17/24, 2:40 PM

The 4096 limit I found was for Mistral v0.2: https://developers.cloudflare.com/workers-ai/models/mistral-7b-instruct-v0.2/

I

Isaac McFadyen•5/17/24, 2:40 PM

So I expect each model is different.

I

Isaac McFadyen•5/17/24, 2:41 PM

Edited my original message to clarify, thanks for bringing it up

IIsaac McFadyen Interesting - doesn't seem to be listed on the model page but the limit for LLaM...

R

Raylight•5/17/24, 3:08 PM

No idea. I'm reconstructing the raw prompt and counting with the total with https://github.com/belladoreai/llama3-tokenizer-js. The limit is 2827 for some reason.

IIsaac McFadyen Note that depending on your source of WebM files (if it's the browser microphone...

S

scotto•5/17/24, 7:30 PM

sure that works on mobile?

Sscotto sure that works on mobile?

I

Isaac McFadyen•5/17/24, 7:40 PM

I'm not sure, no. You'd have to test.

Rraj.js How much is prompt limit for Llama 3 model? Is there any model on cloudfalre whi...

R

Raylight•5/17/24, 8:57 PM

Looks like

@cf/qwen/qwen1.5-7b-chat-awq

@cf/qwen/qwen1.5-7b-chat-awq

has a context length way above 8k. Managed to chop up and squeeze in ~20k tokens, ~100k characters, and ask a question about a detail buried in the beginning of the text.

S

SciCat•5/18/24, 8:23 PM

Hello I was wondering, do anyone know if it is possible to pass image as input ?

S

Sergey•5/20/24, 6:04 AM

Hey! Is there any rough ETA when text2img models will get out of beta?

SSciCat Hello I was wondering, do anyone know if it is possible to pass image as input ?

A

AngusMa•5/20/24, 2:06 PM

@cf/llava-hf/llava-1.5-7b-hf

@cf/llava-hf/llava-1.5-7b-hf

and

@cf/unum/uform-gen2-qwen-500m

@cf/unum/uform-gen2-qwen-500m

accept an image as input.
https://developers.cloudflare.com/workers-ai/models/#image-to-text

Cloudflare Docs

Models · Cloudflare Workers AI docs

Browse our entire catalog of models.

A

AngusMa•5/20/24, 2:11 PM

The model

bge-large-en-v1.5

bge-large-en-v1.5

appears on the model list, but not on the dashboard.
Was it forgotten?

AAngusMa The model `bge-large-en-v1.5` appears on the model list, but not on the dashboar...

R

Raylight•5/20/24, 2:31 PM

@cf/openchat/openchat-3.5-0106

@cf/openchat/openchat-3.5-0106

,

@cf/qwen/qwen1.5-14b-chat-awq

@cf/qwen/qwen1.5-14b-chat-awq

,

@hf/google/gemma-7b-it

@hf/google/gemma-7b-it

are also gone. Seems like the dashboard has a limit of 50 models.

Mmichelle taking a look - definitely not the intention.

C

Chaika•5/20/24, 6:10 PM

you've probably already figured it out, but fwiw the per_page query param on the search models endpoint has been broken for a while, I forgot to mention it and have just been working around it lol, it's always 50 no matter what you specify (in this case, the dash is trying 1000, but getting 50 back, and clicking through the list pages in the dash isn't paginating via the api)

M

Mrinank•5/20/24, 7:32 PM

can we use pinecone or upstash vector db in cf workers??

MMrinank can we use pinecone or upstash vector db in cf workers??

M

Mrinank•5/20/24, 7:35 PM

Getting an error while connecting : const pinecone = new Pinecone({ apiKey: '' });

[ERROR] Error compiling schema, function code: const schema2 = scope.schema[2];const schema1 = scope.schema[1]......

X [ERROR] Error in fetch handler: EvalError: Code generation from strings disallowed for this context

[ERROR] Error compiling schema, function code: const schema2 = scope.schema[2];const schema1 = scope.schema[1]......

X [ERROR] Error in fetch handler: EvalError: Code generation from strings disallowed for this context

[ERROR] Error compiling schema, function code: const schema2 = scope.schema[2];const schema1 = scope.schema[1]......

X [ERROR] Error in fetch handler: EvalError: Code generation from strings disallowed for this context

[ERROR] Error compiling schema, function code: const schema2 = scope.schema[2];const schema1 = scope.schema[1]......

X [ERROR] Error in fetch handler: EvalError: Code generation from strings disallowed for this context

Similar Threads

Similar Threads

Similar Threads