Ok, that’s how I set my headers. Not sure exactly how to do it with python functions but read this,

Ok, that’s how I set my headers. Not sure exactly how to do it with python functions but read this, it should help https://developers.cloudflare.com/pages/functions/middleware/

Cloudflare Docs

Middleware · Cloudflare Pages docs

Middleware is reusable logic that can be run before your onRequest function. Middlewares are typically utility functions. Error handling, user …

Hhydroxide Ok, that’s how I set my headers. Not sure exactly how to do it with python funct...

vyastj•7/2/24, 10:54 AM

okay so should i replace python with js ?

Vvyastj okay so should i replace python with js ?

hydroxideOP•7/2/24, 12:40 PM

I doubt you need to do that, I just haven’t worked with python on CF so I don’t know how to implement middleware. I’m sure the info is in the link I gave you

qinfengge•7/2/24, 1:35 PM

Hello I use spring AI framework for development and I found a problem

qinfengge•7/2/24, 1:36 PM

In this document I saw that it supports OPENAI’s API format
https://developers.cloudflare.com/workers-ai/configuration/open-ai-compatibility/

Cloudflare Docs

OpenAI compatible API endpoints · Cloudflare Workers AI docs

Workers AI supports OpenAI compatible endpoints for text generation (/v1/chat/completions) and text embedding models (/v1/embeddings). This allows you …

qinfengge•7/2/24, 1:37 PM

I noticed that the interface did not return Usage

DDhravya Shah hi! Yeah, this might be because of a recent API change by OpenAI API

Raylight•7/3/24, 8:04 AM

Token counts aren't included in the response when running Cloudflare's OpenAI compatible endpoint. Perhaps the docs could use some information about supported parameters and the level of compatibility? Tested it with code similar to the example and couldn't get max_tokens to have any effect.

fifaaaron88•7/3/24, 5:14 PM

Unable to add a fine-tune. Error:
https://developers.cloudflare.com/workers-ai/tutorials/fine-tune-models-with-autotrain

FILE_PARSE_ERROR: 'file' should be of valid safetensors type [code: 1000], quiting...

FILE_PARSE_ERROR: 'file' should be of valid safetensors type [code: 1000], quiting...

Receiving this when executing the following command:

npx wrangler ai finetune create @cf/mistral/mistral-7b-instruct-v0.2-lora warp-finetune-mistral folder_location

npx wrangler ai finetune create @cf/mistral/mistral-7b-instruct-v0.2-lora warp-finetune-mistral folder_location

Folder contains adapter_config.json and adapter_model.safetensors.
I used autotrain on Huggingface to build these files: https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/colabs/AutoTrain_LLM.ipynb

Safetensor file size is 80mb. I have another one 100.1 mb. Both fails

Cloudflare Docs

Fine Tune Models With AutoTrain from HuggingFace · Cloudflare Worke...

Fine tuning an AI model gives you the opportunity to add additional training data to the model. Workers AI allows for Low-Rank Adaptation, LoRA, …

Google Colab

Ffifaaaron88 Unable to add a fine-tune. Error: https://developers.cloudflare.com/workers-ai/t...

fifaaaron88•7/3/24, 6:23 PM

Full error:

--- 2024-07-03T18:20:15.757Z debug
🪵  Writing logs to "/Users/agdfdf/.wrangler/logs/wrangler-2024-07-03_18-20-15_678.log"
---

--- 2024-07-03T18:20:15.757Z debug
Failed to load .env file ".env": Error: ENOENT: no such file or directory, open '.env'
    at Object.openSync (node:fs:582:18)
    at Object.readFileSync (node:fs:461:35)
    at tryLoadDotEnv (/Users/ad/node_modules/wrangler/wrangler-dist/cli.js:159106:72)
    at loadDotEnv (/Users/ad/node_modules/wrangler/wrangler-dist/cli.js:159115:12)
    at Array.reduce (<anonymous>) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '.env'
}
---

--- 2024-07-03T18:20:15.764Z debug
Retrieving cached values for account from ../../../node_modules/.cache/wrangler
---

--- 2024-07-03T18:20:15.764Z log
🌀 Creating new finetune "warp-mistral-new" for model "@cf/mistral/mistral-7b-instruct-v0.2-lora"...
---
--- 2024-07-03T18:20:15.765Z debug
-- START CF API REQUEST: POST https://api.cloudflare.com/client/v4/accounts/<account_id_masqueraded>/ai/finetunes
---
--- 2024-07-03T18:20:15.765Z debug
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
---
--- 2024-07-03T18:20:15.765Z debug
INIT: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
---
--- 2024-07-03T18:20:15.765Z debug
-- END CF API REQUEST
---

--- 2024-07-03T18:20:17.189Z debug
-- START CF API RESPONSE: Bad Request 400
---

--- 2024-07-03T18:20:17.190Z debug
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
---

--- 2024-07-03T18:20:17.190Z debug
RESPONSE: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
---

--- 2024-07-03T18:20:17.190Z debug
-- END CF API RESPONSE
---

--- 2024-07-03T18:20:17.229Z error
✘ [ERROR] 🚨 Finetune couldn't be created: A request to the Cloudflare API (/accounts/<account_id_masqueraded>/ai/finetunes) failed. FINETUNE_EXISTS_ERROR: finetune with the name 'warp-mistral-new' already exists [code: 1000]

--- 2024-07-03T18:20:15.757Z debug
🪵  Writing logs to "/Users/agdfdf/.wrangler/logs/wrangler-2024-07-03_18-20-15_678.log"
---

--- 2024-07-03T18:20:15.757Z debug
Failed to load .env file ".env": Error: ENOENT: no such file or directory, open '.env'
    at Object.openSync (node:fs:582:18)
    at Object.readFileSync (node:fs:461:35)
    at tryLoadDotEnv (/Users/ad/node_modules/wrangler/wrangler-dist/cli.js:159106:72)
    at loadDotEnv (/Users/ad/node_modules/wrangler/wrangler-dist/cli.js:159115:12)
    at Array.reduce (<anonymous>) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '.env'
}
---

--- 2024-07-03T18:20:15.764Z debug
Retrieving cached values for account from ../../../node_modules/.cache/wrangler
---

--- 2024-07-03T18:20:15.764Z log
🌀 Creating new finetune "warp-mistral-new" for model "@cf/mistral/mistral-7b-instruct-v0.2-lora"...
---
--- 2024-07-03T18:20:15.765Z debug
-- START CF API REQUEST: POST https://api.cloudflare.com/client/v4/accounts/<account_id_masqueraded>/ai/finetunes
---
--- 2024-07-03T18:20:15.765Z debug
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
---
--- 2024-07-03T18:20:15.765Z debug
INIT: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
---
--- 2024-07-03T18:20:15.765Z debug
-- END CF API REQUEST
---

--- 2024-07-03T18:20:17.189Z debug
-- START CF API RESPONSE: Bad Request 400
---

--- 2024-07-03T18:20:17.190Z debug
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
---

--- 2024-07-03T18:20:17.190Z debug
RESPONSE: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
---

--- 2024-07-03T18:20:17.190Z debug
-- END CF API RESPONSE
---

--- 2024-07-03T18:20:17.229Z error
✘ [ERROR] 🚨 Finetune couldn't be created: A request to the Cloudflare API (/accounts/<account_id_masqueraded>/ai/finetunes) failed. FINETUNE_EXISTS_ERROR: finetune with the name 'warp-mistral-new' already exists [code: 1000]

fifaaaron88•7/3/24, 6:53 PM

error_messages

message.txt3.29KB

Chy•7/5/24, 3:03 PM

request.body.query not working and request.text()??

Chy•7/5/24, 3:22 PM

nvm it changed

tsutsu•7/5/24, 4:59 PM

I understand that currently, developers can only supply LoRAs, rather than arbitrary full models — presumably because, in a multitenant inference setting, you want a few base models everyone is using that can stay always-loaded-in-VRAM, with just the much-smaller LoRAs being loaded/unloaded between calls

but fine-tunes processed with the BitDelta technique (https://www.together.ai/blog/bitdelta; https://arxiv.org/html/2402.10193v2) seem to offer another alternative; the approach has basically the same benefits as LoRAs for multi-tenant settings — keeping the base model(s) always-loaded, and just loading/unloading fine-tune deltas. (In fact, I believe these models would be loaded as adapters by inference frameworks, exactly as LoRAs are — just with these adapter models' layers having the same rank as the original model layers, rather than being low-rank.)

would the Workers AI team potentially consider allowing the use of arbitrary developer-provided fine-tunes (of the fixed set of base models), by computing and caching each fine-tune's BitDelta against the base-model?

(or by uploading pre-computed BitDeltas — but I don't think there's a standard format/encoding for these yet, or even open-source tooling to generate them, as these are a performance/resource optimization mostly relevant only to multi-tenant inference platforms. so the developer pointing at the fine-tune's URL, and CF fetching it + computing a BitDelta from it, is probably better UX for now... unless your team also wants to be the one to open-source the format and tooling for doing this

)

trace•7/8/24, 10:18 AM

Is it possible to know by when the text-images model will be out of beta? Mainly curious about the pricing per generation will be.

Ferdi KIZILTOPRAK•7/8/24, 2:03 PM

Hello, Stability.ai updated SD3 license. Do you plan to add it to your catalog? https://stability.ai/news/license-update?utm_source=www.therundown.ai&utm_medium=newsletter&utm_campaign=openai-s-secret-hack-exposed

Stability AI

Community License — Stability AI

Our new Community License is now free for research, non-commercial, and commercial use. You only need a paid Enterprise license if your yearly revenues exceed USD$1M and you use Stability AI models in commercial products or services.

user•7/10/24, 11:31 AM

Is there a way to set

max_tokens

max_tokens

runWithTools

runWithTools

Name With Exactly 32 Characters!•7/11/24, 1:26 PM

Want to run resnet50 but for image embeddings, not classifications. Is this possible?

Peps•7/14/24, 3:43 PM

Does Cloudflare run some kind of moderation check on input to models?
I'm looking into using Workers AI to identify content which may be CSAM related, as unfortunately OpenAI's free moderation API was not sufficient for my use case and mostly gave me false negatives.
Hopefully the user-generated input I'll give to models won't trigger some kind of flag/check, so just want to make sure that my use case is alright.

akazwz•7/14/24, 4:07 PM

Hello, I build a LLMs/AI chat platform build and design for Cloudflare worker AI. with many features like function calling, image to text, text to image, text to speech. all build with Cloudflare worker. https://github.com/akazwz/antonai

GitHub

GitHub - akazwz/antonai: a LLMs/AI chat platform build and design f...

a LLMs/AI chat platform build and design for Cloudflare worker AI. with many features like function calling, image to text, text to image, text to speech - akazwz/antonai

RRahul is it possible to upload image like chatgpt and ask llama to analyse and answer ...

Craig Dennis•7/15/24, 2:46 PM

If you are looking for an Image-To-Text app, here is one built with Llava + SvelteKit https://floor-is-llava.pages.dev/ (code in the footer)

PPeps Does Cloudflare run some kind of moderation check on input to models? I'm lookin...

Craig Dennis•7/15/24, 2:49 PM

We do offer the LlamaGuard model: https://ai.meta.com/research/publications/llama-guard-llm-based-input-output-safeguard-for-human-ai-conversations/ Does this look like it might assist with what you are after?

Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversa...

We introduce Llama Guard, an LLM-based input-output safeguard model geared towards Human-AI conversation use cases. Our model incorporates a safety risk...

Uuser Is there a way to set `max_tokens` on `runWithTools`?

user•7/16/24, 2:41 PM

Uuser ?

Raylight•7/16/24, 3:47 PM

No. You would have to modify runWithTools to be able to do that, e.g. https://github.com/cloudflare/ai-utils/compare/main...Raylight-PWL:ai-utils:Raylight-patch-maxTokens-option.

user•7/16/24, 3:58 PM

:frown:

DDhravya Shah not right now - not sure how that would work though, Because runwithtools make...

user•7/17/24, 5:56 PM

I think a good idea would be to have either max_tokens be the max tokens for each call, or have the user able to configure the max tokens for each call

user•7/17/24, 5:57 PM

the second option being something like initial_max_tokens, tool_max_tokens, and possibly trim_max_tokens (very scuffed index names, but you get the point)

user•7/17/24, 6:02 PM

Its been really good

user•7/17/24, 6:02 PM

~~its nice to actually have types available unlike on normal cf-AI~~

user•7/17/24, 6:02 PM

on normal CF ai i had to guess so much

user•7/17/24, 6:03 PM

ai-utils is definitely awesome in comparison

user•7/17/24, 6:04 PM

i once tried to read the actual AI library on npm to figure out unknowns

user•7/17/24, 6:04 PM

Never trying that again!

user•7/17/24, 6:04 PM

compiled code :salute:

user•7/17/24, 6:05 PM

np!! :D

user•7/17/24, 6:05 PM

I have a quick question

user•7/17/24, 6:05 PM

considering you do ai-utils shenanigans

user•7/17/24, 6:05 PM

are there any plans on tool usage being supported on the qwen models?

user•7/17/24, 6:06 PM

from what I read, the actual model itself supports it, but I'm obviously not entirely sure on it

DDhravya Shah they _should_ already work with qwen models actually

user•7/17/24, 6:06 PM

It's not documented as so, and when I tried I could not get it to work haha

--- 2024-07-03T18:20:15.757Z debug 🪵 Writing logs to "/Users/agdfdf/.wrangler/logs/wrangler-2024-07-03_18-20-15_678.log" --- --- 2024-07-03T18:20:15.757Z debug Failed to load .env file ".env": Error: ENOENT: no such file or directory, open '.env' at Object.openSync (node:fs:582:18) at Object.readFileSync (node:fs:461:35) at tryLoadDotEnv (/Users/ad/node_modules/wrangler/wrangler-dist/cli.js:159106:72) at loadDotEnv (/Users/ad/node_modules/wrangler/wrangler-dist/cli.js:159115:12) at Array.reduce (<anonymous>) { errno: -2, code: 'ENOENT', syscall: 'open', path: '.env' } --- --- 2024-07-03T18:20:15.764Z debug Retrieving cached values for account from ../../../node_modules/.cache/wrangler --- --- 2024-07-03T18:20:15.764Z log 🌀 Creating new finetune "warp-mistral-new" for model "@cf/mistral/mistral-7b-instruct-v0.2-lora"... --- --- 2024-07-03T18:20:15.765Z debug -- START CF API REQUEST: POST https://api.cloudflare.com/client/v4/accounts/<account_id_masqueraded>/ai/finetunes --- --- 2024-07-03T18:20:15.765Z debug HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data --- --- 2024-07-03T18:20:15.765Z debug INIT: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data --- --- 2024-07-03T18:20:15.765Z debug -- END CF API REQUEST --- --- 2024-07-03T18:20:17.189Z debug -- START CF API RESPONSE: Bad Request 400 --- --- 2024-07-03T18:20:17.190Z debug HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data --- --- 2024-07-03T18:20:17.190Z debug RESPONSE: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data --- --- 2024-07-03T18:20:17.190Z debug -- END CF API RESPONSE --- --- 2024-07-03T18:20:17.229Z error ✘ [ERROR] 🚨 Finetune couldn't be created: A request to the Cloudflare API (/accounts/<account_id_masqueraded>/ai/finetunes) failed. FINETUNE_EXISTS_ERROR: finetune with the name 'warp-mistral-new' already exists [code: 1000]

Ok, that’s how I set my headers. Not sure exactly how to do it with python functions but read this,

Similar Threads

Similar Threads

Similar Threads