Hello there, Please I'll like to know if it's possible to use a HuggingFace model or even an OpenAI

JJames Hello there, Please I'll like to know if it's possible to use a HuggingFace mod...

I

Isaac McFadyen•4/28/24, 4:57 PM

?workers-ai-models See below.

F

Flare•4/28/24, 4:57 PM

Workers AI currently only supports popular open-source models provided by the Cloudflare team, as well as your own LoRAs that can be applied on top of the Cloudflare-provided models. You cannot currently upload your own models or use a model from HuggingFace. See the documentation for the list of Cloudflare-provided models: https://developers.cloudflare.com/workers-ai/models/

I

Isaac McFadyen•4/28/24, 4:58 PM

You can use OpenAI models via #ai-gateway-beta but it will still be provided by OpenAI, just proxied through AI Gateway.

J

JamesOP•4/28/24, 5:00 PM

Alright, thank you!

K

kingmesal•4/29/24, 7:58 PM

Is there a max time length or file size that whisper can handle?
I've got a 5.8MB mp3 file and I'm getting

"InferenceUpstreamError: ERROR 3010: Invalid or incomplete input for the model: failed to decode JSON: Request is too large"

Mmichelle i believe it works best if audio files are <30s (whisper model limitation)

K

kingmesal•4/29/24, 11:08 PM

that cannot be a hard limit because billing on this model is per minute of audio ...

S

scotto•4/30/24, 12:19 AM

getting this error with the llama 3 instruct

S

scotto•4/30/24, 12:20 AM

took 30 secs to return error

Mmichelle i believe it works best if audio files are <30s (whisper model limitation)

T

tottocorro•4/30/24, 12:55 AM

I got the same error with this 29s whisper example sound file
https://cdn.openai.com/whisper/draft-20220913a/micro-machines.wav

D

Dragon•4/30/24, 10:13 AM

I am getting this error

D

Dragon•4/30/24, 10:14 AM

D

Dragon•4/30/24, 10:14 AM

I do not see a support for this. Is coludflare community active?

Mmichelle i believe it works best if audio files are <30s (whisper model limitation)

R

Raylight•4/30/24, 2:15 PM

Some feedback on this.
Although whisper might be limited, I believe "Request is too large" is a workers AI problem rather than a problem with the model. I tested and reproduced the exact same error with

@cf/microsoft/resnet-50

@cf/microsoft/resnet-50

@cf/microsoft/resnet-50

@cf/microsoft/resnet-50 and

@cf/runwayml/stable-diffusion-v1-5-img2img

@cf/runwayml/stable-diffusion-v1-5-img2img

@cf/runwayml/stable-diffusion-v1-5-img2img

@cf/runwayml/stable-diffusion-v1-5-img2img, and believe that the same would hold for any model that accepts a large input. I used the following to test and measure:

const srcURL  = "https://cdn.openai.com/whisper/draft-20220913a/micro-machines.wav";
const res     = await fetch(srcURL);
const blob    = await res.arrayBuffer();
const jsArray = [...new Uint8Array(blob)];
const input   = { audio: jsArray };

console.log("Blob size: " + (jsArray.length / (2 << 19)).toFixed(1) + " MB");
console.log("Input array size: " + (jsArray.length / (2 << 17)).toFixed(1) + " MB");

// ai.run() stringifies input array before calling internal fetch:
//   const inpBody = JSON.stringify({ inputs: input });
//   console.log("JSON size: " + (inpBody.length / (2 << 19)).toFixed(1) + " MB");

const response = await ai.run("@cf/openai/whisper", input);

const srcURL  = "https://cdn.openai.com/whisper/draft-20220913a/micro-machines.wav";
const res     = await fetch(srcURL);
const blob    = await res.arrayBuffer();
const jsArray = [...new Uint8Array(blob)];
const input   = { audio: jsArray };

console.log("Blob size: " + (jsArray.length / (2 << 19)).toFixed(1) + " MB");
console.log("Input array size: " + (jsArray.length / (2 << 17)).toFixed(1) + " MB");

// ai.run() stringifies input array before calling internal fetch:
//   const inpBody = JSON.stringify({ inputs: input });
//   console.log("JSON size: " + (inpBody.length / (2 << 19)).toFixed(1) + " MB");

const response = await ai.run("@cf/openai/whisper", input);

const srcURL  = "https://cdn.openai.com/whisper/draft-20220913a/micro-machines.wav";
const res     = await fetch(srcURL);
const blob    = await res.arrayBuffer();
const jsArray = [...new Uint8Array(blob)];
const input   = { audio: jsArray };

console.log("Blob size: " + (jsArray.length / (2 << 19)).toFixed(1) + " MB");
console.log("Input array size: " + (jsArray.length / (2 << 17)).toFixed(1) + " MB");

// ai.run() stringifies input array before calling internal fetch:
//   const inpBody = JSON.stringify({ inputs: input });
//   console.log("JSON size: " + (inpBody.length / (2 << 19)).toFixed(1) + " MB");

const response = await ai.run("@cf/openai/whisper", input);

const srcURL  = "https://cdn.openai.com/whisper/draft-20220913a/micro-machines.wav";
const res     = await fetch(srcURL);
const blob    = await res.arrayBuffer();
const jsArray = [...new Uint8Array(blob)];
const input   = { audio: jsArray };

console.log("Blob size: " + (jsArray.length / (2 << 19)).toFixed(1) + " MB");
console.log("Input array size: " + (jsArray.length / (2 << 17)).toFixed(1) + " MB");

// ai.run() stringifies input array before calling internal fetch:
//   const inpBody = JSON.stringify({ inputs: input });
//   console.log("JSON size: " + (inpBody.length / (2 << 19)).toFixed(1) + " MB");

const response = await ai.run("@cf/openai/whisper", input);

This is with 1.1.0 so that I can insert a logging statement, but changing to env.AI.run doesn't affect the outcome. The issue seems to be size, rather than dimensions (e.g. length, width, height). Changing

{ audio: jsArray }

{ audio: jsArray }

{ audio: jsArray }

{ audio: jsArray } to

{ image: jsArray }

{ image: jsArray }

{ image: jsArray }

{ image: jsArray } and calling resnet-50 would throw the same error.

After fetching a 5 MB file, the worker has to make a copy to turn it into a ~20 MB array, assuming no overhead. The array is then stringified into a ~17 MB string. The receiving end would be faced with potentially parsing 17 MB of json with the format

[123,78,30,255,0,...]

[123,78,30,255,0,...]

[123,78,30,255,0,...]

[123,78,30,255,0,...]. Unless there's a limit somewhere, then at some point something has to give. In this case, there seems to be a limit of just below 10 million bytes.

The immediate and simple part of the problem is that developers typically don't have a good way to handle this. I mean, it's not like the above is common knowledge..

C

Codename_A•4/30/24, 2:20 PM

I'm unsure if using workers for this case would work:
I want to make openai api requests in the backend without showing the user my api key. Something like an api proxy. Ofcourse, then I would need to authenticate to make sure regular people couldn't just use that to make requests. Could workers be a use case for this? Or another product. How would I do this>

RRaylight Some feedback on this. Although whisper might be limited, I believe "Request is ...

K

kingmesal•4/30/24, 2:25 PM

It would be nice if the backend model in these cases could take the URL to process ... it doesn't solve the problem but it does remove 1 more variable from the equation

RRaylight Some feedback on this. Although whisper might be limited, I believe "Request is ...

I

Isaac McFadyen•4/30/24, 2:27 PM

Just to add to this: base64 is more efficient in my testing than an actual stringified array of

[<number>,<number>,<number>]

[<number>,<number>,<number>]

[<number>,<number>,<number>]

[<number>,<number>,<number>] (by a significant amount) but would need to be supported internally by Workers AI which I don't believe it is currently

R

Raylight•4/30/24, 3:19 PM

Yeah.. what's needed (IMO) is a better/faster way to pass large inputs, although optimized internals could go a long way. In some cases, like with a fast model such as

@cf/microsoft/resnet-50

@cf/microsoft/resnet-50

@cf/microsoft/resnet-50

@cf/microsoft/resnet-50, the actual inference can be a fraction of the total latency, where the rest is mainly due to juggling input data. I've done some testing on this.

Mmichelle what does optimized mean here?

R

Raylight•4/30/24, 10:00 PM

1.0.53 has two functions that become expensive on large arrays (serializeType and ensureShape). Additionally, the input is piped through a CompressionStream. "Optimized" is what you'd get if you skipped those operations. SDK 1.1.0 and native have solved much of the issue of high CPU usage attributed to the worker (

), so I've turned my attention to wall time instead.

F

falex•4/30/24, 10:02 PM

I am working with meta-llama-3-8b-instruct and sometimes the API produces this error: InferenceUpstreamError. I am not using the stream option.

Mmichelle phi-3 coming in hot

R

Ryan K•4/30/24, 10:47 PM

with lora adapter too?

D

ddts86•5/1/24, 3:55 AM

What is difference between beta and ga besides the usage?

A

another_User•5/2/24, 5:00 AM

Has anyone tried the image to text recently? I have been trying it for about an hour, I think it's down or I'm doing something very wrong.

F

Fuse•5/2/24, 9:36 AM

why response status alway be 500 and with that response error

Mmichelle how are you using it? not noticing any downtime

A

another_User•5/2/24, 2:16 PM

I first tried by running this code i got from the docs on the dashboard, it responded with an error, but the code and error are below:

import { Ai } from './vendor/@cloudflare/ai.js';

export default {
  async fetch(request, env) {
    const tasks = [];
    const ai = new Ai(env.AI);
    const res = await fetch("https://cataas.com/cat");
  const blob = await res.arrayBuffer();
  console.log(new Uint8Array(blob))
  const input = {
    image: new Uint8Array(blob),
    prompt: "Generate a caption for this image",
    max_tokens: 512
  };

    let response = await ai.run("@cf/unum/uform-gen2-qwen-500m", input);
    console.log(response);
    
    return Response.json(response);
  }
};

import { Ai } from './vendor/@cloudflare/ai.js';

export default {
  async fetch(request, env) {
    const tasks = [];
    const ai = new Ai(env.AI);
    const res = await fetch("https://cataas.com/cat");
  const blob = await res.arrayBuffer();
  console.log(new Uint8Array(blob))
  const input = {
    image: new Uint8Array(blob),
    prompt: "Generate a caption for this image",
    max_tokens: 512
  };

    let response = await ai.run("@cf/unum/uform-gen2-qwen-500m", input);
    console.log(response);
    
    return Response.json(response);
  }
};

import { Ai } from './vendor/@cloudflare/ai.js';

export default {
  async fetch(request, env) {
    const tasks = [];
    const ai = new Ai(env.AI);
    const res = await fetch("https://cataas.com/cat");
  const blob = await res.arrayBuffer();
  console.log(new Uint8Array(blob))
  const input = {
    image: new Uint8Array(blob),
    prompt: "Generate a caption for this image",
    max_tokens: 512
  };

    let response = await ai.run("@cf/unum/uform-gen2-qwen-500m", input);
    console.log(response);
    
    return Response.json(response);
  }
};

import { Ai } from './vendor/@cloudflare/ai.js';

export default {
  async fetch(request, env) {
    const tasks = [];
    const ai = new Ai(env.AI);
    const res = await fetch("https://cataas.com/cat");
  const blob = await res.arrayBuffer();
  console.log(new Uint8Array(blob))
  const input = {
    image: new Uint8Array(blob),
    prompt: "Generate a caption for this image",
    max_tokens: 512
  };

    let response = await ai.run("@cf/unum/uform-gen2-qwen-500m", input);
    console.log(response);
    
    return Response.json(response);
  }
};

i got this:

InferenceUpstreamError: must be string, must be array, must match exactly one schema in oneOf
    at Ai.run (vendor/@cloudflare/ai.js:3084:15)
    at async Object.fetch (index.js:16:20)
    at async jsonError (.internal-57a7950f-b522-44d5-b273-8b1fd9d88826-facade-1.js:12:12)
    at async jsonError (.internal-57a7950f-b522-44d5-b273-8b1fd9d88826-facade-1.js:12:12)

InferenceUpstreamError: must be string, must be array, must match exactly one schema in oneOf
    at Ai.run (vendor/@cloudflare/ai.js:3084:15)
    at async Object.fetch (index.js:16:20)
    at async jsonError (.internal-57a7950f-b522-44d5-b273-8b1fd9d88826-facade-1.js:12:12)
    at async jsonError (.internal-57a7950f-b522-44d5-b273-8b1fd9d88826-facade-1.js:12:12)

InferenceUpstreamError: must be string, must be array, must match exactly one schema in oneOf
    at Ai.run (vendor/@cloudflare/ai.js:3084:15)
    at async Object.fetch (index.js:16:20)
    at async jsonError (.internal-57a7950f-b522-44d5-b273-8b1fd9d88826-facade-1.js:12:12)
    at async jsonError (.internal-57a7950f-b522-44d5-b273-8b1fd9d88826-facade-1.js:12:12)

InferenceUpstreamError: must be string, must be array, must match exactly one schema in oneOf
    at Ai.run (vendor/@cloudflare/ai.js:3084:15)
    at async Object.fetch (index.js:16:20)
    at async jsonError (.internal-57a7950f-b522-44d5-b273-8b1fd9d88826-facade-1.js:12:12)
    at async jsonError (.internal-57a7950f-b522-44d5-b273-8b1fd9d88826-facade-1.js:12:12)

secondly, i tried curl and it was the same thing, I did put the Bearer token in the authentication header.

Mmichelle how are you using it? not noticing any downtime

A

another_User•5/2/24, 2:21 PM

Here is the dart code I also tried :

var image = Uint8List.fromList(
      (await http.get(Uri.parse("https://cataas.com/cat"))).bodyBytes);
  var uri = Uri.https(
    "api.cloudflare.com",
    "/client/v4/accounts/$accountID/ai/run/@cf/unum/uform-gen2-qwen-500m",
  );
  var img_desc = await http.post(uri,
      headers: {
        "Authorization": token
      },
      body: jsonEncode({
        "prompt": "generate a description for this image",
        "image": image,
        "max_tokens": 512
      }));

var image = Uint8List.fromList(
      (await http.get(Uri.parse("https://cataas.com/cat"))).bodyBytes);
  var uri = Uri.https(
    "api.cloudflare.com",
    "/client/v4/accounts/$accountID/ai/run/@cf/unum/uform-gen2-qwen-500m",
  );
  var img_desc = await http.post(uri,
      headers: {
        "Authorization": token
      },
      body: jsonEncode({
        "prompt": "generate a description for this image",
        "image": image,
        "max_tokens": 512
      }));

var image = Uint8List.fromList(
      (await http.get(Uri.parse("https://cataas.com/cat"))).bodyBytes);
  var uri = Uri.https(
    "api.cloudflare.com",
    "/client/v4/accounts/$accountID/ai/run/@cf/unum/uform-gen2-qwen-500m",
  );
  var img_desc = await http.post(uri,
      headers: {
        "Authorization": token
      },
      body: jsonEncode({
        "prompt": "generate a description for this image",
        "image": image,
        "max_tokens": 512
      }));

var image = Uint8List.fromList(
      (await http.get(Uri.parse("https://cataas.com/cat"))).bodyBytes);
  var uri = Uri.https(
    "api.cloudflare.com",
    "/client/v4/accounts/$accountID/ai/run/@cf/unum/uform-gen2-qwen-500m",
  );
  var img_desc = await http.post(uri,
      headers: {
        "Authorization": token
      },
      body: jsonEncode({
        "prompt": "generate a description for this image",
        "image": image,
        "max_tokens": 512
      }));

If returned with:

A

another_User•5/2/24, 2:27 PM

flutter: {"errors":[{"message":"InferenceUpstreamError: must be string, must have required property 'image', must be number, must match exactly one schema in oneOf","code":1000}],"success":false,"result":{},"messages":[]}

flutter: {"errors":[{"message":"InferenceUpstreamError: must be string, must have required property 'image', must be number, must match exactly one schema in oneOf","code":1000}],"success":false,"result":{},"messages":[]}

flutter: {"errors":[{"message":"InferenceUpstreamError: must be string, must have required property 'image', must be number, must match exactly one schema in oneOf","code":1000}],"success":false,"result":{},"messages":[]}

flutter: {"errors":[{"message":"InferenceUpstreamError: must be string, must have required property 'image', must be number, must match exactly one schema in oneOf","code":1000}],"success":false,"result":{},"messages":[]}

LLogan Grasby A simple example using llava 1.5: https://github.com/LoganGrasby/llava-1.5-worke...

W

Wallacy•5/2/24, 5:43 PM

I didint undestant... is @cf/llava-hf/llava-1.5-7b-hf a oficial CF model now or we can just use -hf models or something like that... I was planing to use a llava on my custom VM because resnet-50 is useless for what i need.

WWallacy I didint undestant... is @cf/llava-hf/llava-1.5-7b-hf a oficial CF model now or ...

I

Isaac McFadyen•5/2/24, 5:44 PM

Official model, you can't use HuggingFace models they haven't added.

I

Isaac McFadyen•5/2/24, 5:44 PM

?workers-ai-models

F

Flare•5/2/24, 5:44 PM

Workers AI currently only supports popular open-source models provided by the Cloudflare team, as well as your own LoRAs that can be applied on top of the Cloudflare-provided models. You cannot currently upload your own models or use a model from HuggingFace. See the documentation for the list of Cloudflare-provided models: https://developers.cloudflare.com/workers-ai/models/

W

Wallacy•5/2/24, 5:47 PM

Ah okay, that was i was thinking, because i look on docs and didn't find llava ...

W

Wallacy•5/2/24, 5:47 PM

Theres any place that i can see the updated list?

W

Wallacy•5/2/24, 5:48 PM

I was planing use the lava-llama-3-8b-v1_1-hf but im sure that is not there...

I

Isaac McFadyen•5/2/24, 5:50 PM

Docs: https://developers.cloudflare.com/workers-ai/models/ and also there's a list on the Cloudflare dashboard.

W

Wallacy•5/2/24, 6:13 PM

Like a said... The @cf/llava-hf/llava-1.5-7b-hf is not there... Thats why i asked how that project worked.

W

Wallacy•5/2/24, 6:18 PM

Sure! Thanks... I was just double checking if theres any list out there that i miss out...

W

Wallacy•5/2/24, 6:29 PM

Anyway, just to know... I just deployed one @cf/llava test and worked very fast... good job!

WWallacy Sure! Thanks... I was just double checking if theres any list out there that i m...

C

Chaika•5/2/24, 6:29 PM

there's the list in the dashboard which includes llava

C

Chaika•5/2/24, 6:29 PM

Hello there, Please I'll like to know if it's possible to use a HuggingFace model or even an OpenAI

Similar Threads