Should workers be compatible with transformers.js yet? I've tried setting it up, hoping to cache eve

SSam White Should workers be compatible with transformers.js yet? I've tried setting it up,...

I

Isaac McFadyen•5/13/24, 4:25 AM

If you're trying to load WASM at runtime from JSDelivr that won't work in Workers.

I

Isaac McFadyen•5/13/24, 4:25 AM

Workers intentionally don't support dynamically loading WASM for security reasons, it must be uploaded at deploy time.

I

Isaac McFadyen•5/13/24, 4:25 AM

See https://developers.cloudflare.com/workers/runtime-apis/web-standards/#javascript-standards (sorry, wrong link, updated)

IIsaac McFadyen Workers intentionally don't support dynamically loading WASM for security reason...

S

Sam WhiteOP•5/13/24, 4:32 AM

I see. Damn. Thank you for the reply!

R

rachelswany•5/13/24, 7:02 AM

hey guys... iam loving the CF hosted Workers AI inference and all the models! I have a few questions:
1/ what is the pricing for beta mode models?
2/ what does it mean by it is in "beta"? CF wont pull the plug on the model I assume?
3/ are beta models suitable for staging type situations? ie not exactly prod, but close

A

Andus•5/13/24, 8:30 AM

Hi. When I'll end my limit from Workers Free will cloudflare start the paid pricing automatically or do I need to enable it myself? If it starts automatically can I turn it off somehow?

R

rob•5/13/24, 12:13 PM

Beta models are free rn afaik

J

James•5/13/24, 1:42 PM

It's quite disappointing to see the AI team's approach to backwards compatibility thus far, going against all of the great paradigms the Workers runtime has enforced for years.

Some examples:

https://github.com/cloudflare/workerd/pull/2044
- it took me quite some time to argue the point about this being a breaking change, despite pushback from multiple team members and saying the risk was "non-existent". Turns out, after this was merged, it broke things and had to be partially reverted
https://github.com/cloudflare/workerd/pull/2103#discussion_r1598483980
- another recent example that was just merged. I truly hope this doesn't break anyone's applications when it rolls out

As a long-time Workers user, I'm a massive fan of their compat guarantees as outlined at https://blog.cloudflare.com/backwards-compatibility-in-cloudflare-workers, and it seems the AI team isn't taking this very seriously for a GA product. There's precedent for other bindings implementing breaking changes behind compat flags, too: https://github.com/cloudflare/workerd/pull/1297

A change to the Workers Runtime must never break an application that is live in production.

.

.neurorotic.•5/13/24, 4:21 PM

The docs seem out of date, there is no "Account details section contains your Account ID" section on workers and pages ->overview https://developers.cloudflare.com/fundamentals/setup/find-account-and-zone-ids/#find-account-id-workers-and-pages
And the only way I found to find it is by creating a domain (even though I did not intend to register any w/ CF) but then finally noticed the account_id is after the dash.cloudflare.com/{ACCOUNT_ID}/ on the site

This could be much clearer for first-time users.

Rrob Beta models are free rn afaik

R

rachelswany•5/13/24, 4:47 PM

Is there a known limit?

M

Mrinank•5/13/24, 6:05 PM

const stream = await retrivalChain.stream({
input: 'what is hello world',
});

return new Response(stream, {
    headers: {
    'content-type': 'text/event-stream',
    'Access-Control-Allow-Origin': '*',
    },
});

const stream = await retrivalChain.stream({
input: 'what is hello world',
});

return new Response(stream, {
    headers: {
    'content-type': 'text/event-stream',
    'Access-Control-Allow-Origin': '*',
    },
});

const stream = await retrivalChain.stream({
input: 'what is hello world',
});

return new Response(stream, {
    headers: {
    'content-type': 'text/event-stream',
    'Access-Control-Allow-Origin': '*',
    },
});

const stream = await retrivalChain.stream({
input: 'what is hello world',
});

return new Response(stream, {
    headers: {
    'content-type': 'text/event-stream',
    'Access-Control-Allow-Origin': '*',
    },
});

is this the correct way to send streams from cloudflare worker?
getting typescript error,

Argument of type 'IterableReadableStream<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to parameter of type 'BodyInit | null | undefined'.
  Type 'IterableReadableStream<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to type 'ReadableStream<Uint8Array>'.
    The types returned by 'getReader()' are incompatible between these types.
      Type 'ReadableStreamDefaultReader<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to type 'ReadableStreamDefaultReader<Uint8Array>'.
        Type '{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }' is missing the following properties from type 'Uint8Array': BYTES_PER_ELEMENT, buffer, byteLength, byteOffset, and 27 more.ts(2345)
const stream: IterableReadableStream<{
    context: Document<Record<string, any>>[];
    answer: string;
} & {
    [key: string]: unknown;
}>

Argument of type 'IterableReadableStream<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to parameter of type 'BodyInit | null | undefined'.
  Type 'IterableReadableStream<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to type 'ReadableStream<Uint8Array>'.
    The types returned by 'getReader()' are incompatible between these types.
      Type 'ReadableStreamDefaultReader<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to type 'ReadableStreamDefaultReader<Uint8Array>'.
        Type '{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }' is missing the following properties from type 'Uint8Array': BYTES_PER_ELEMENT, buffer, byteLength, byteOffset, and 27 more.ts(2345)
const stream: IterableReadableStream<{
    context: Document<Record<string, any>>[];
    answer: string;
} & {
    [key: string]: unknown;
}>

Argument of type 'IterableReadableStream<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to parameter of type 'BodyInit | null | undefined'.
  Type 'IterableReadableStream<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to type 'ReadableStream<Uint8Array>'.
    The types returned by 'getReader()' are incompatible between these types.
      Type 'ReadableStreamDefaultReader<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to type 'ReadableStreamDefaultReader<Uint8Array>'.
        Type '{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }' is missing the following properties from type 'Uint8Array': BYTES_PER_ELEMENT, buffer, byteLength, byteOffset, and 27 more.ts(2345)
const stream: IterableReadableStream<{
    context: Document<Record<string, any>>[];
    answer: string;
} & {
    [key: string]: unknown;
}>

Argument of type 'IterableReadableStream<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to parameter of type 'BodyInit | null | undefined'.
  Type 'IterableReadableStream<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to type 'ReadableStream<Uint8Array>'.
    The types returned by 'getReader()' are incompatible between these types.
      Type 'ReadableStreamDefaultReader<{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }>' is not assignable to type 'ReadableStreamDefaultReader<Uint8Array>'.
        Type '{ context: Document<Record<string, any>>[]; answer: string; } & { [key: string]: unknown; }' is missing the following properties from type 'Uint8Array': BYTES_PER_ELEMENT, buffer, byteLength, byteOffset, and 27 more.ts(2345)
const stream: IterableReadableStream<{
    context: Document<Record<string, any>>[];
    answer: string;
} & {
    [key: string]: unknown;
}>

MMrinank ``` const stream = await retrivalChain.stream({ input: 'what is hello world', })...

M

Mrinank•5/13/24, 10:48 PM

i am using import { createRetrievalChain } from 'langchain/chains/retrieval';

S

scotto•5/14/24, 8:55 PM

whister should work with webm format? in the openai docs says it's supported but testing agains the API from cloudflare i see an error, Invalid or incomplete input for the model: model returned: Invalid audio input: F

C

codebam•5/14/24, 10:40 PM

is there a way to wrap

env.AI.run()

env.AI.run()

env.AI.run()

env.AI.run() in

ctx.passThroughOnException

ctx.passThroughOnException

ctx.passThroughOnException

ctx.passThroughOnException while still saving the result?

C

codebam•5/14/24, 10:44 PM

I don't actually understand how

passThroughOnException

passThroughOnException

passThroughOnException

passThroughOnException works. I just want my worker to respond properly when the models time out

J

James•5/14/24, 11:17 PM

you'd probably want to just:

try {
    await env.AI.run(...)
} catch (e) {
    return new Response('handle errors here')
}

try {
    await env.AI.run(...)
} catch (e) {
    return new Response('handle errors here')
}

try {
    await env.AI.run(...)
} catch (e) {
    return new Response('handle errors here')
}

try {
    await env.AI.run(...)
} catch (e) {
    return new Response('handle errors here')
}

and then consider retrying with errors if it makes sense.

J

James•5/14/24, 11:18 PM

passThroughOnException

passThroughOnException

passThroughOnException

passThroughOnException is more intended for if you have an origin behind the worker

J

James•5/14/24, 11:18 PM

so that rather than showing an error page, it'll just send the request as-is to your origin. This is often referred to as a "fail-open" design

JJames you'd probably want to just: ```js try { await env.AI.run(...) } catch (e) {...

C

codebam•5/14/24, 11:34 PM

oh okay. I'll try this, thank you

S

Shanmukeshwar•5/15/24, 6:54 AM

Workers AI are very slow avg: 10sec

S

Shanmukeshwar•5/15/24, 6:54 AM

Any tweaks we can do here?

S

Shanmukeshwar•5/15/24, 6:56 AM

Hey @James
Workers AI are very slow avg: 10sec
Any tweaks we can do here?

Thanks

SShanmukeshwar Hey @James Workers AI are very slow avg: 10sec Any tweaks we can do here? Than...

C

codebam•5/15/24, 9:28 AM

if it's a text generation model you can lower the

max_tokens

max_tokens

max_tokens

max_tokens

C

codebam•5/15/24, 9:28 AM

you'll get shorter replies though

Ccodebam if it's a text generation model you can lower the `max_tokens`

S

Shanmukeshwar•5/15/24, 9:28 AM

I managed the UI to stream the response

S

Shanmukeshwar•5/15/24, 9:29 AM

Thanks for the reply

R

raj.js•5/15/24, 5:22 PM

How much is prompt limit for Llama 3 model? Is there any model on cloudfalre which has atleast 8k token limit

Rraj.js How much is prompt limit for Llama 3 model? Is there any model on cloudfalre whi...

C

codebam•5/15/24, 7:59 PM

4096 characters. no

I

Isaac McFadyen•5/16/24, 2:37 AM

^ just to clarify this: that's tokens, not characters.

I

Isaac McFadyen•5/16/24, 2:38 AM

A token is almost always more than 1 character (to see this for yourself, you can put your text in here, although note that this is not the same tokenizer that the Workers AI models use: https://platform.openai.com/tokenizer)

I

Isaac McFadyen•5/16/24, 2:38 AM

But yes, 4096 tokens maximum (for Mistral v0.2).

A

AndyJessop•5/16/24, 2:05 PM

Does anyone know where I can find a price for Llama3-8b-instruct? There's no pricing for it on the models page, and it's not in the dropdown on the calculator.

https://developers.cloudflare.com/workers-ai/platform/pricing/#text-generation
https://ai.cloudflare.com/#pricing-calculator

AAndyJessop Does anyone know where I can find a price for Llama3-8b-instruct? There's no pri...

I

Isaac McFadyen•5/16/24, 2:21 PM

https://developers.cloudflare.com/workers-ai/platform/pricing/ It's still beta so unbilled currently.

I

Isaac McFadyen•5/16/24, 2:21 PM

>You can still enjoy unlimited usage on the beta models in the catalog until they graduate out of beta.

IIsaac McFadyen >You can still enjoy unlimited usage on the beta models in the catalog until the...

A

AndyJessop•5/16/24, 2:26 PM

That's great news, I thought I was going to bre in for a huge bill...

M

Mrinank•5/16/24, 2:28 PM

does langsmith works with workers AI?

Sscotto whister should work with webm format? in the openai docs says it's supported but...

S

scotto•5/16/24, 10:41 PM

news?

I

Isaac McFadyen•5/16/24, 11:34 PM

Note that depending on your source of WebM files (if it's the browser microphone) you can take a look at that link I sent to force the browser to output a more standardized format

IIsaac McFadyen But yes, 4096 tokens maximum (for Mistral v0.2).

R

Raylight•5/17/24, 2:36 PM

The limit for Llama 3 (on Cloudflare) seems to be ~2800 tokens though. The model stops mid-sentence when prompt/messages + generated text goes above that limit.

RRaylight The limit for Llama 3 (on Cloudflare) seems to be ~2800 tokens though. The model...

I

Isaac McFadyen•5/17/24, 2:40 PM

Interesting - doesn't seem to be listed on the model page but the limit for LLaMA 2 is 2048 tokens so perhaps they share similar limits? https://developers.cloudflare.com/workers-ai/models/llama-2-7b-chat-int8/#properties

I

Isaac McFadyen•5/17/24, 2:40 PM

The 4096 limit I found was for Mistral v0.2: https://developers.cloudflare.com/workers-ai/models/mistral-7b-instruct-v0.2/

I

Isaac McFadyen•5/17/24, 2:40 PM

So I expect each model is different.

I

Isaac McFadyen•5/17/24, 2:41 PM

Edited my original message to clarify, thanks for bringing it up

IIsaac McFadyen Interesting - doesn't seem to be listed on the model page but the limit for LLaM...

R

Raylight•5/17/24, 3:08 PM

No idea. I'm reconstructing the raw prompt and counting with the total with https://github.com/belladoreai/llama3-tokenizer-js. The limit is 2827 for some reason.

IIsaac McFadyen Note that depending on your source of WebM files (if it's the browser microphone...

S

scotto•5/17/24, 7:30 PM

sure that works on mobile?

Sscotto sure that works on mobile?

I

Isaac McFadyen•5/17/24, 7:40 PM

I'm not sure, no. You'd have to test.

Rraj.js How much is prompt limit for Llama 3 model? Is there any model on cloudfalre whi...

R

Raylight•5/17/24, 8:57 PM

Looks like

@cf/qwen/qwen1.5-7b-chat-awq

@cf/qwen/qwen1.5-7b-chat-awq

@cf/qwen/qwen1.5-7b-chat-awq

@cf/qwen/qwen1.5-7b-chat-awq has a context length way above 8k. Managed to chop up and squeeze in ~20k tokens, ~100k characters, and ask a question about a detail buried in the beginning of the text.

S

SciCat•5/18/24, 8:23 PM

Hello I was wondering, do anyone know if it is possible to pass image as input ?

Should workers be compatible with transformers.js yet? I've tried setting it up, hoping to cache eve

Similar Threads

Similar Threads

Similar Threads