qq; is there a limit on the response length from the LLM?
qq; is there a limit on the response length from the LLM?
CAnd, a "less quick" question
Is there a way to do a
Specifically trying to get an LLM to take an input like "dogs" with a system message of "generate a trivia question based on the topic and respond in json format of
Do the worker ai response is
)
Is there a way to do a
json_format response with workers AI? (like what OpenAI supports) Specifically trying to get an LLM to take an input like "dogs" with a system message of "generate a trivia question based on the topic and respond in json format of
{ question: "string", answer: "string" }Do the worker ai response is
{question: "what is the best dog", answer: "all of them"} (made this up )
RYa each model has diff max token output
SBAAI/bge-m3 is ok, from the same family of the already supported
AWhy didn’t Workers AI Playground work?
Phomepage demo is also broken
Pdid you also see my video
Phere
Pmaybe related issues?
Pbut
Psweet, I was finna say "its still not working" but nice ninja edit and nice quick fix!
AI logged all data I had received from a streaming Workers AI call and got a key called
p. What is that?
RThe Cloudflare Blog
The Workers AI and AI Gateway team recently collaborated closely with security researchers at Ben Gurion University regarding a report submitted through our Public Bug Bounty program. Through this process, we discovered and fully patched a vulnerability affecting all LLM providers. Here’s the story
Rthats the p thing ^
云messed around with the descriptions until it used my tool
云
VHello, I want to use CF worker routes to deploy few security headers on all the incoming traffic except many subdomains (huge list). Hence i wrote a worker in
wrangler.toml
with index.py
wrangler.toml
with index.py
VBut this does not seem to work when i do a curl to check for added headers
curl -I https://notexcluded.domain.com
HDo you have a middleware.js?
VNopes i have not set that up. When i was setting the worker, CF gave me python + wrangler as a template
HOk, that’s how I set my headers. Not sure exactly how to do it with python functions but read this, it should help https://developers.cloudflare.com/pages/functions/middleware/
Cloudflare Docs
Middleware is reusable logic that can be run before your onRequest function. Middlewares are typically utility functions. Error handling, user …
Vokay so should i replace python with js ?
HI doubt you need to do that, I just haven’t worked with python on CF so I don’t know how to implement middleware. I’m sure the info is in the link I gave you
QHello I use spring AI framework for development and I found a problem
QIn this document I saw that it supports OPENAI’s API format
https://developers.cloudflare.com/workers-ai/configuration/open-ai-compatibility/
https://developers.cloudflare.com/workers-ai/configuration/open-ai-compatibility/
Cloudflare Docs
Workers AI supports OpenAI compatible endpoints for text generation (/v1/chat/completions) and text embedding models (/v1/embeddings). This allows you …
QI noticed that the interface did not return Usage
RToken counts aren't included in the response when running Cloudflare's OpenAI compatible endpoint. Perhaps the docs could use some information about supported parameters and the level of compatibility? Tested it with code similar to the example and couldn't get max_tokens to have any effect.
FUnable to add a fine-tune. Error:
https://developers.cloudflare.com/workers-ai/tutorials/fine-tune-models-with-autotrain
Receiving this when executing the following command:
Folder contains adapter_config.json and adapter_model.safetensors.
I used autotrain on Huggingface to build these files: https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/colabs/AutoTrain_LLM.ipynb
Safetensor file size is 80mb. I have another one 100.1 mb. Both fails
https://developers.cloudflare.com/workers-ai/tutorials/fine-tune-models-with-autotrain
Receiving this when executing the following command:
Folder contains adapter_config.json and adapter_model.safetensors.
I used autotrain on Huggingface to build these files: https://colab.research.google.com/github/huggingface/autotrain-advanced/blob/main/colabs/AutoTrain_LLM.ipynb
Safetensor file size is 80mb. I have another one 100.1 mb. Both fails
Cloudflare Docs
Fine tuning an AI model gives you the opportunity to add additional training data to the model. Workers AI allows for Low-Rank Adaptation, LoRA, …
FFull error:
Ferror_messages
message.txt3.29KB
Crequest.body.query not working and request.text()??
Cnvm it changed
TI understand that currently, developers can only supply LoRAs, rather than arbitrary full models — presumably because, in a multitenant inference setting, you want a few base models everyone is using that can stay always-loaded-in-VRAM, with just the much-smaller LoRAs being loaded/unloaded between calls
but fine-tunes processed with the BitDelta technique (https://www.together.ai/blog/bitdelta; https://arxiv.org/html/2402.10193v2) seem to offer another alternative; the approach has basically the same benefits as LoRAs for multi-tenant settings — keeping the base model(s) always-loaded, and just loading/unloading fine-tune deltas. (In fact, I believe these models would be loaded as adapters by inference frameworks, exactly as LoRAs are — just with these adapter models' layers having the same rank as the original model layers, rather than being low-rank.)
would the Workers AI team potentially consider allowing the use of arbitrary developer-provided fine-tunes (of the fixed set of base models), by computing and caching each fine-tune's BitDelta against the base-model?
(or by uploading pre-computed BitDeltas — but I don't think there's a standard format/encoding for these yet, or even open-source tooling to generate them, as these are a performance/resource optimization mostly relevant only to multi-tenant inference platforms. so the developer pointing at the fine-tune's URL, and CF fetching it + computing a BitDelta from it, is probably better UX for now... unless your team also wants to be the one to open-source the format and tooling for doing this
)
but fine-tunes processed with the BitDelta technique (https://www.together.ai/blog/bitdelta; https://arxiv.org/html/2402.10193v2) seem to offer another alternative; the approach has basically the same benefits as LoRAs for multi-tenant settings — keeping the base model(s) always-loaded, and just loading/unloading fine-tune deltas. (In fact, I believe these models would be loaded as adapters by inference frameworks, exactly as LoRAs are — just with these adapter models' layers having the same rank as the original model layers, rather than being low-rank.)
would the Workers AI team potentially consider allowing the use of arbitrary developer-provided fine-tunes (of the fixed set of base models), by computing and caching each fine-tune's BitDelta against the base-model?
(or by uploading pre-computed BitDeltas — but I don't think there's a standard format/encoding for these yet, or even open-source tooling to generate them, as these are a performance/resource optimization mostly relevant only to multi-tenant inference platforms. so the developer pointing at the fine-tune's URL, and CF fetching it + computing a BitDelta from it, is probably better UX for now... unless your team also wants to be the one to open-source the format and tooling for doing this
)
TIs it possible to know by when the text-images model will be out of beta? Mainly curious about the pricing per generation will be.
FHello, Stability.ai updated SD3 license. Do you plan to add it to your catalog? https://stability.ai/news/license-update?utm_source=www.therundown.ai&utm_medium=newsletter&utm_campaign=openai-s-secret-hack-exposed
Stability AI
Our new Community License is now free for research, non-commercial, and commercial use. You only need a paid Enterprise license if your yearly revenues exceed USD$1M and you use Stability AI models in commercial products or services.
UIs there a way to set
max_tokens on runWithTools?
NWant to run resnet50 but for image embeddings, not classifications. Is this possible?
PDoes Cloudflare run some kind of moderation check on input to models?
I'm looking into using Workers AI to identify content which may be CSAM related, as unfortunately OpenAI's free moderation API was not sufficient for my use case and mostly gave me false negatives.
Hopefully the user-generated input I'll give to models won't trigger some kind of flag/check, so just want to make sure that my use case is alright.
I'm looking into using Workers AI to identify content which may be CSAM related, as unfortunately OpenAI's free moderation API was not sufficient for my use case and mostly gave me false negatives.
Hopefully the user-generated input I'll give to models won't trigger some kind of flag/check, so just want to make sure that my use case is alright.
AHello, I build a LLMs/AI chat platform build and design for Cloudflare worker AI. with many features like function calling, image to text, text to image, text to speech. all build with Cloudflare worker. https://github.com/akazwz/antonai
GitHub
a LLMs/AI chat platform build and design for Cloudflare worker AI. with many features like function calling, image to text, text to image, text to speech - akazwz/antonai
CIf you are looking for an Image-To-Text app, here is one built with Llava + SvelteKit https://floor-is-llava.pages.dev/ (code in the footer)
CWe do offer the LlamaGuard model: https://ai.meta.com/research/publications/llama-guard-llm-based-input-output-safeguard-for-human-ai-conversations/ Does this look like it might assist with what you are after?
We introduce Llama Guard, an LLM-based input-output safeguard model geared towards Human-AI conversation use cases. Our model incorporates a safety risk...
U?
