You can swap out OpenAIEmbeddings with CloudflareWorkersAIEmbeddings. Definitely not a guide, but..

P

preetramsha•5/5/24, 1:20 PM

Hello I am thinking of publishing an app with workers-ai API. but I am concerned about rate limits.
can my app make thousands of requests per minute?

P

preetramsha•5/5/24, 1:21 PM

please help

RRaylight You can swap out OpenAIEmbeddings with CloudflareWorkersAIEmbeddings. Definitely...

M

Mrinank•5/5/24, 2:11 PM

Thanks it worked :MeowHeartCloudflare:

MMrinank Thanks it worked :MeowHeartCloudflare:

R

RaylightOP•5/5/24, 4:12 PM

Great! Beware that the example with the Brooklyn article consumes a lot of neurons.

RRaylight Great! Beware that the example with the Brooklyn article consumes a *lot* of neu...

M

Mrinank•5/5/24, 6:48 PM

yeah it took 550+ neurons but i am in free tier

M

Mrinank•5/5/24, 6:48 PM

were those from llm or while generating embeddings?

Ppreetramsha Hello I am thinking of publishing an app with workers-ai API. but I am concerned...

R

rob•5/6/24, 1:06 PM

Yes

Rrob Yes

P

preetramsha•5/6/24, 1:11 PM

but how do I handle more requests with than the minits specified?

J

Jens•5/6/24, 1:28 PM

hi, I try to implement the Langchain pdf loader into my application. But if I call

loader.load()

loader.load()

I always get the error message:

RangeError: Maximum call stack size exceeded

RangeError: Maximum call stack size exceeded

Does someone know what could be the cause? I have already tried it with small pdf files, but that didn't help either

notes.post('/', async (c: CustomContext) => {
    // Need to polyfill a method that Cloudflare Workers is missing for the PDF loader
  globalThis.setImmediate = ((fn: () => {}) => setTimeout(fn, 0)) as any
    const cloudflareFetchResponse = await fetch(
        'https://www.cloudflare.com/resources/assets/slt3lc6tev37/3HWObubm6fybC0FWUdFYAJ/5d5e3b0a4d9c5a7619984ed6076f01fe/Cloudflare_for_Campaigns_Security_Guide.pdf'
    )
    const cloudflarePdfBlob = await cloudflareFetchResponse.blob()
    const laoder = new WebPDFLoader(cloudflarePdfBlob)
    const docs = await loader.load() //RangeError: Maximum call stack size exceeded

notes.post('/', async (c: CustomContext) => {
    // Need to polyfill a method that Cloudflare Workers is missing for the PDF loader
  globalThis.setImmediate = ((fn: () => {}) => setTimeout(fn, 0)) as any
    const cloudflareFetchResponse = await fetch(
        'https://www.cloudflare.com/resources/assets/slt3lc6tev37/3HWObubm6fybC0FWUdFYAJ/5d5e3b0a4d9c5a7619984ed6076f01fe/Cloudflare_for_Campaigns_Security_Guide.pdf'
    )
    const cloudflarePdfBlob = await cloudflareFetchResponse.blob()
    const laoder = new WebPDFLoader(cloudflarePdfBlob)
    const docs = await loader.load() //RangeError: Maximum call stack size exceeded

notes.post('/', async (c: CustomContext) => {
    // Need to polyfill a method that Cloudflare Workers is missing for the PDF loader
  globalThis.setImmediate = ((fn: () => {}) => setTimeout(fn, 0)) as any
    const cloudflareFetchResponse = await fetch(
        'https://www.cloudflare.com/resources/assets/slt3lc6tev37/3HWObubm6fybC0FWUdFYAJ/5d5e3b0a4d9c5a7619984ed6076f01fe/Cloudflare_for_Campaigns_Security_Guide.pdf'
    )
    const cloudflarePdfBlob = await cloudflareFetchResponse.blob()
    const laoder = new WebPDFLoader(cloudflarePdfBlob)
    const docs = await loader.load() //RangeError: Maximum call stack size exceeded

notes.post('/', async (c: CustomContext) => {
    // Need to polyfill a method that Cloudflare Workers is missing for the PDF loader
  globalThis.setImmediate = ((fn: () => {}) => setTimeout(fn, 0)) as any
    const cloudflareFetchResponse = await fetch(
        'https://www.cloudflare.com/resources/assets/slt3lc6tev37/3HWObubm6fybC0FWUdFYAJ/5d5e3b0a4d9c5a7619984ed6076f01fe/Cloudflare_for_Campaigns_Security_Guide.pdf'
    )
    const cloudflarePdfBlob = await cloudflareFetchResponse.blob()
    const laoder = new WebPDFLoader(cloudflarePdfBlob)
    const docs = await loader.load() //RangeError: Maximum call stack size exceeded

The part of this code is from the langchain repository https://github.com/jacoblee93/langchain-nuxt-cloudflare-template

Thanks

GitHub

GitHub - jacoblee93/langchain-nuxt-cloudflare-template

Contribute to jacoblee93/langchain-nuxt-cloudflare-template development by creating an account on GitHub.

JJens hi, I try to implement the Langchain pdf loader into my application. But if I ca...

J

Jens•5/6/24, 3:55 PM

with this loader it works

:

  const loader = new CheerioWebBaseLoader('https://developers.cloudflare.com/workers-ai/models/', {
        selector: 'body',
    })

    const docs = await loader.load()

  const loader = new CheerioWebBaseLoader('https://developers.cloudflare.com/workers-ai/models/', {
        selector: 'body',
    })

    const docs = await loader.load()

  const loader = new CheerioWebBaseLoader('https://developers.cloudflare.com/workers-ai/models/', {
        selector: 'body',
    })

    const docs = await loader.load()

  const loader = new CheerioWebBaseLoader('https://developers.cloudflare.com/workers-ai/models/', {
        selector: 'body',
    })

    const docs = await loader.load()

U

Unsmart•5/6/24, 4:04 PM

Whats up with the workers ai usage tab? Cost summary fails to load, daily usage always says 0, the "all models" section says I have only used it on April 11th, and the model specific ones say I have only used them on April 10th. (im a time traveler I guess mustve went ahead a few days used it and broke something my bad)

U

Unsmart•5/6/24, 4:06 PM

oh wait I just realized april was last month I was thinking that wouldve been this month :lul:

UUnsmart Whats up with the workers ai usage tab? Cost summary fails to load, daily usage ...

C

Chaika•5/6/24, 4:06 PM

this month is May lol

C

Chaika•5/6/24, 4:06 PM

only GA models bill you, which is some of the confusion maybe

C

Chaika•5/6/24, 4:06 PM

mine looks sane

U

Unsmart•5/6/24, 4:07 PM

hmm

U

Unsmart•5/6/24, 4:08 PM

Yeah I guess I only do use the beta models atm kinda stupid to not have usage data on them

RRaylight Sounds like you're using the `@cloudflare/ai` npm package. It's depreciated, so ...

K

kingmesal•5/7/24, 12:10 AM

Regarding this https://developers.cloudflare.com/workers-ai/changelog/.
What is the type for TypeScript? I cannot find anything on this

Cloudflare Docs

Changelog · Cloudflare Workers AI docs

Review recent changes to Cloudflare Workers AI.

Kkingmesal Regarding this https://developers.cloudflare.com/workers-ai/changelog/. What is ...

R

RaylightOP•5/7/24, 5:36 AM

They should be in

@cloudflare/workers-types

@cloudflare/workers-types

. https://github.com/cloudflare/workerd/blob/main/types/defines/ai.d.ts if you want to look at them. That's the best I got.

R

Raven•5/8/24, 8:45 AM

Can we serve custom models on Workers AI or do we have to wait for them to get added?

R

Raven•5/8/24, 8:46 AM

And if it is not currently possible to serve custom models via Cloudflare, and it's not prohibited to discuss, what are some good options? I have a list and I'm checking it twice, but curious what people think and/or use

RRaven Can we serve custom models on Workers AI or do we have to wait for them to get a...

I

Isaac McFadyen•5/8/24, 12:54 PM

?workers-ai-models

F

Flare•5/8/24, 12:54 PM

Workers AI currently only supports popular open-source models provided by the Cloudflare team, as well as your own LoRAs that can be applied on top of the Cloudflare-provided models. You cannot currently upload your own models or use a model from HuggingFace. See the documentation for the list of Cloudflare-provided models: https://developers.cloudflare.com/workers-ai/models/

E

element14•5/8/24, 7:22 PM

Hi every one , Im trying to use my own LoRA adapters together with

@cf/mistral/mistral-7b-instruct-v0.2-lora

@cf/mistral/mistral-7b-instruct-v0.2-lora

and worker-ai
I have already uploaded my LoRA adapters successfully , but when I try to run an inference to my LoRAS , Im getting an unknown and undocumented error:

InferenceUpstreamError: ERROR 3028: Unknown internal error

InferenceUpstreamError: ERROR 3028: Unknown internal error

Im using the snippet-code described in the tutorial at the following link : https://developers.cloudflare.com/workers-ai/fine-tunes/loras/


export interface Env {

  AI: any;
}

export default {
  async fetch(request, env): Promise<Response> {


    const response = await env.AI.run(
      "@cf/mistral/mistral-7b-instruct-v0.2-lora", //the model supporting LoRAs
      {
          messages: [{"role": "user", "content": "Hello world"}],
          raw: true,
          lora: "4e900000-0000-0000-0000-000000000",
      }
    );

    return new Response(JSON.stringify(response));
  },
} satisfies ExportedHandler<Env>;


export interface Env {

  AI: any;
}

export default {
  async fetch(request, env): Promise<Response> {


    const response = await env.AI.run(
      "@cf/mistral/mistral-7b-instruct-v0.2-lora", //the model supporting LoRAs
      {
          messages: [{"role": "user", "content": "Hello world"}],
          raw: true,
          lora: "4e900000-0000-0000-0000-000000000",
      }
    );

    return new Response(JSON.stringify(response));
  },
} satisfies ExportedHandler<Env>;


export interface Env {

  AI: any;
}

export default {
  async fetch(request, env): Promise<Response> {


    const response = await env.AI.run(
      "@cf/mistral/mistral-7b-instruct-v0.2-lora", //the model supporting LoRAs
      {
          messages: [{"role": "user", "content": "Hello world"}],
          raw: true,
          lora: "4e900000-0000-0000-0000-000000000",
      }
    );

    return new Response(JSON.stringify(response));
  },
} satisfies ExportedHandler<Env>;


export interface Env {

  AI: any;
}

export default {
  async fetch(request, env): Promise<Response> {


    const response = await env.AI.run(
      "@cf/mistral/mistral-7b-instruct-v0.2-lora", //the model supporting LoRAs
      {
          messages: [{"role": "user", "content": "Hello world"}],
          raw: true,
          lora: "4e900000-0000-0000-0000-000000000",
      }
    );

    return new Response(JSON.stringify(response));
  },
} satisfies ExportedHandler<Env>;

The code is working correctly without my LoRA adapters.
The adapter has been trained with rank <= 8 as per documentation , using

Mistral-7B-Instruct-v0.2

Mistral-7B-Instruct-v0.2

as base Model (https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)
I know the features is in Beta , but any suggestion is appreciated.
PS:
a) The json in the tutorial code is not valid there is a missing

after

Hello world"

Hello world"

b) The suggested model in the same tutorial is not valid model is under the repo

mistral

mistral

and not

mistralai

mistralai

tutorial at the following link https://developers.cloudflare.com/workers-ai/fine-tunes/loras/#running-inference-with-loras

Cloudflare Docs

Using LoRA adapters · Cloudflare Workers AI docs

Upload and use LoRA adapters to get fine-tuned inference on Workers AI.

mistralai/Mistral-7B-Instruct-v0.2 · Hugging Face

I

Isaac McFadyen•5/8/24, 7:48 PM

Also it looks like you're specifying

raw: true

raw: true

but then providing the

messages

messages

field which I don't believe is compatible?

I

Isaac McFadyen•5/8/24, 7:48 PM

If you do

raw: true

raw: true

then you need to format it to ChatML yourself and provide it as an input string if I remember correctly

Mmichelle hey! few things you could try: - did you add `"model_type": "mistral"` to your a...

E

element14•5/8/24, 7:49 PM

yes model_type is defined into the uploaded adapter_config.json

IIsaac McFadyen Also it looks like you're specifying `raw: true` but then providing the `message...

E

element14•5/8/24, 7:50 PM

sorry , forgot to update in the sent message , but yes , I have tried to remove

raw

raw

property also tried true/false using the correct message/prompt format
The result is still the same

InferenceUpstreamError: ERROR 3028: Unknown internal error

InferenceUpstreamError: ERROR 3028: Unknown internal error

Mmichelle hey! few things you could try: - did you add `"model_type": "mistral"` to your a...

E

element14•5/8/24, 7:53 PM

tried also to pass the fine_tune name instead of the id , same result

E

element14•5/8/24, 7:58 PM

maybe something wrong with my adapters , for the record they were generated via mlx_lm , no problem Inferencing with them on my local machine via mlx_lm or Ollama.
Here my adapters_config.json

{
  "adapter_file": null,
  "adapter_path": "adapters",
  "batch_size": 5,
  "config": "lora.yml",
  "data": "./data/",
  "grad_checkpoint": false,
  "iters": 1800,
  "learning_rate": 1e-05,
  "lora_layers": 19,
  "lora_parameters": {
      "keys": [
          "self_attn.q_proj",
          "self_attn.v_proj"
      ],
      "rank": 8,
      "alpha": 16.0,
      "scale": 10.0,
      "dropout": 0.05
  },
  "lr_schedule": {
      "name": "cosine_decay",
      "warmup": 100,
      "warmup_init": 1e-07,
      "arguments": [
          1e-05,
          1000,
          1e-07
      ]
  },
  "max_seq_length": 32768,
  "model": "mistralai/Mistral-7B-Instruct-v0.2",
  "model_type": "mistral",
  "resume_adapter_file": null,
  "save_every": 100,
  "seed": 0,
  "steps_per_eval": 20,
  "steps_per_report": 10,
  "test": false,
  "test_batches": 100,
  "train": true,
  "val_batches": -1
}

{
  "adapter_file": null,
  "adapter_path": "adapters",
  "batch_size": 5,
  "config": "lora.yml",
  "data": "./data/",
  "grad_checkpoint": false,
  "iters": 1800,
  "learning_rate": 1e-05,
  "lora_layers": 19,
  "lora_parameters": {
      "keys": [
          "self_attn.q_proj",
          "self_attn.v_proj"
      ],
      "rank": 8,
      "alpha": 16.0,
      "scale": 10.0,
      "dropout": 0.05
  },
  "lr_schedule": {
      "name": "cosine_decay",
      "warmup": 100,
      "warmup_init": 1e-07,
      "arguments": [
          1e-05,
          1000,
          1e-07
      ]
  },
  "max_seq_length": 32768,
  "model": "mistralai/Mistral-7B-Instruct-v0.2",
  "model_type": "mistral",
  "resume_adapter_file": null,
  "save_every": 100,
  "seed": 0,
  "steps_per_eval": 20,
  "steps_per_report": 10,
  "test": false,
  "test_batches": 100,
  "train": true,
  "val_batches": -1
}

{
  "adapter_file": null,
  "adapter_path": "adapters",
  "batch_size": 5,
  "config": "lora.yml",
  "data": "./data/",
  "grad_checkpoint": false,
  "iters": 1800,
  "learning_rate": 1e-05,
  "lora_layers": 19,
  "lora_parameters": {
      "keys": [
          "self_attn.q_proj",
          "self_attn.v_proj"
      ],
      "rank": 8,
      "alpha": 16.0,
      "scale": 10.0,
      "dropout": 0.05
  },
  "lr_schedule": {
      "name": "cosine_decay",
      "warmup": 100,
      "warmup_init": 1e-07,
      "arguments": [
          1e-05,
          1000,
          1e-07
      ]
  },
  "max_seq_length": 32768,
  "model": "mistralai/Mistral-7B-Instruct-v0.2",
  "model_type": "mistral",
  "resume_adapter_file": null,
  "save_every": 100,
  "seed": 0,
  "steps_per_eval": 20,
  "steps_per_report": 10,
  "test": false,
  "test_batches": 100,
  "train": true,
  "val_batches": -1
}

{
  "adapter_file": null,
  "adapter_path": "adapters",
  "batch_size": 5,
  "config": "lora.yml",
  "data": "./data/",
  "grad_checkpoint": false,
  "iters": 1800,
  "learning_rate": 1e-05,
  "lora_layers": 19,
  "lora_parameters": {
      "keys": [
          "self_attn.q_proj",
          "self_attn.v_proj"
      ],
      "rank": 8,
      "alpha": 16.0,
      "scale": 10.0,
      "dropout": 0.05
  },
  "lr_schedule": {
      "name": "cosine_decay",
      "warmup": 100,
      "warmup_init": 1e-07,
      "arguments": [
          1e-05,
          1000,
          1e-07
      ]
  },
  "max_seq_length": 32768,
  "model": "mistralai/Mistral-7B-Instruct-v0.2",
  "model_type": "mistral",
  "resume_adapter_file": null,
  "save_every": 100,
  "seed": 0,
  "steps_per_eval": 20,
  "steps_per_report": 10,
  "test": false,
  "test_batches": 100,
  "train": true,
  "val_batches": -1
}

E

element14•5/8/24, 7:59 PM

@michelle @Isaac McFadyen | YYZ01 thanks to both of you for the suggestions, appreciated

IIsaac McFadyen ?workers-ai-models

R

Raven•5/8/24, 9:15 PM

Thanks, searching "own model" I realize this is a common question. Is bring-your-own-model on the roadmap?
Also I am looking into cloud compute providers to host custom inference endpoints—does Cloudflare offer any solutions along that axis?

I

Isaac McFadyen•5/8/24, 9:19 PM

I'll let Michelle answer the first. As for the second, they don't currently, it's just Workers AI. I'm not sure whether it's planned but I'd doubt it since currently the focus is on improving the core Workers AI product.

R

Raven•5/8/24, 9:27 PM

Sounds good, for context there are some TTS models that run in the browser here https://hexgrad.com/ (note: may not work on iOS due to low webapp memory budget).
It's free to use but/because it runs on your own CPU. So if you want to churn through a many-hour audiobook, unless you have a solid desktop with good cooling you're probably looking at major heat issues.
Trying to find options to outsource the compute somewhere else. And also potentially combine the TTS with ASR/LLMs to produce conversational/phone agents. Not a novel idea by any stretch, but TTS tends to be a fairly expensive puzzle piece so I think both the standalone TTS and combo agents can be done at relatively competitive prices.

Hexgrad

Speak any text in any voice on your own machine.

R

Raven•5/8/24, 9:28 PM

That website is currently hosted on Cloudflare's free tier via next-on-pages. Satisfied with the experience so far (although it is a static webpage and I saw laurent's caveats this morning)

J

Julian•5/9/24, 10:33 AM

Can I check that this issue is still current? That there's no ability to customise any LLM parameters such as temperature or output format?

https://github.com/cloudflare/cloudflare-docs/issues/11138

It was closed but to me "closed" means "we've done this" not "we're going to ignore it".

GitHub

Add Documentation for Controlling AI Temperature and Related AI Set...

Which Cloudflare product(s) does this pertain to? Workers AI Subject Matter I would like to request documentation that explains how to control the AI temperature and other related AI settings when ...

S

scotto•5/9/24, 2:22 PM

any plans to increase limits of non beta model? i mean having no limits

LLogan Grasby We're expanding the list of models for generating embeddings and taking requests...

S

scotto•5/9/24, 2:22 PM

any multilingual embedding model?

LLogan Grasby Temperature is supported on most LLMs as an input and additional parameters are ...

J

Julian•5/9/24, 2:44 PM

Thanks. Do you have a link to an example of setting temp?

P

Puliczek•5/9/24, 4:51 PM

Is it possible to use llama-3 function calling on workers ai?

R

rob•5/9/24, 8:23 PM

not right now

LLogan Grasby Only PEFT trained loras are compatible. See https://huggingface.co/docs/peft/en/...

E

element14•5/9/24, 11:02 PM

adapter_config.json

adapter_config.json

(click to show the full content)

{
"alpha_pattern": {},
"auto_mapping": null,
"base_model_name_or_path": "mistralai/Mistral-7B-Instruct-v0.2",

"task_type": "CAUSAL_LM",
"model_type": "mistral",
"use_dora": false,
"use_rslora": false
}

LLogan Grasby Only PEFT trained loras are compatible. See https://huggingface.co/docs/peft/en/...

E

element14•5/9/24, 11:02 PM

Thanks for the suggestion.
I have followed the tutorial and re-trained the mistral model using the suggested jupiter notebook
https://github.com/huggingface/autotrain-advanced/blob/main/colabs/AutoTrain_LLM.ipynb

my

adapter_config.json

adapter_config.json

is now: see message above workers-ai

Do you notice anything wrong ? Because now , when I try to upload my

adapter_model.safetensors

adapter_model.safetensors

I recevie a new error from the wrangler

✘ [ERROR] 🚨 Couldn't upload file: A request to the Cloudflare API (/accounts/1111122223334444/ai/finetunes/6a4a4a4a4a4a4a4a4-a5aa5a5a-aaaaaa/finetune-assets) failed. FILE_PARSE_ERROR: 'file' should be of valid safetensors type [code: 1000], quiting...

✘ [ERROR] 🚨 Couldn't upload file: A request to the Cloudflare API (/accounts/1111122223334444/ai/finetunes/6a4a4a4a4a4a4a4a4-a5aa5a5a-aaaaaa/finetune-assets) failed. FILE_PARSE_ERROR: 'file' should be of valid safetensors type [code: 1000], quiting...

✘ [ERROR] 🚨 Couldn't upload file: A request to the Cloudflare API (/accounts/1111122223334444/ai/finetunes/6a4a4a4a4a4a4a4a4-a5aa5a5a-aaaaaa/finetune-assets) failed. FILE_PARSE_ERROR: 'file' should be of valid safetensors type [code: 1000], quiting...

✘ [ERROR] 🚨 Couldn't upload file: A request to the Cloudflare API (/accounts/1111122223334444/ai/finetunes/6a4a4a4a4a4a4a4a4-a5aa5a5a-aaaaaa/finetune-assets) failed. FILE_PARSE_ERROR: 'file' should be of valid safetensors type [code: 1000], quiting...

So to recap my tests:

The adapter previously trained using mlx_lm (https://github.com/ml-explore/mlx-examples , https://huggingface.co/docs/hub/en/mlx) is accepted during the fine-tune upload/creation process , but it generates the error
```
InferenceUpstreamError: ERROR 3028: Unknown internal error
```
```
InferenceUpstreamError: ERROR 3028: Unknown internal error
```
when I try to run an inference.
The adapter trained using autotrain from huggingface is not accepted during the fine-tune upload/creation process , giving me the error described above.

Is there any other required parameter like (

rank r <=8 or quantization = None

rank r <=8 or quantization = None

) that has not been specified into the documentation ?
Thanks again for the support

GitHub

autotrain-advanced/colabs/AutoTrain_LLM.ipynb at main · huggingface...

AutoTrain Advanced. Contribute to huggingface/autotrain-advanced development by creating an account on GitHub.

GitHub

GitHub - ml-explore/mlx-examples: Examples in the MLX framework

Examples in the MLX framework. Contribute to ml-explore/mlx-examples development by creating an account on GitHub.

Sscotto any plans to increase limits of non beta model? i mean having no limits

S

scotto•5/10/24, 12:41 AM

?

Sscotto ?

I

Isaac McFadyen•5/10/24, 1:11 AM

Not sure about plans but one of the people on the Workers AI team, Michelle, has previously said to reach out to her if you are looking for higher limits: https://canary.discord.com/channels/595317990191398933/1138522314594582578/1237041407982698598

You can swap out OpenAIEmbeddings with CloudflareWorkersAIEmbeddings. Definitely not a guide, but..

Similar Threads

Similar Threads

Similar Threads