Have you got something working with Langchain and successfully deployed to workers? I was running in

M

Hey all, so I have been trying to use different multi-agent frameworks in cloudflare because their AI platform is amazing, but I'm not able to use the big ones such as CrewAI or AutoGen. I want to basically use cloudflare's access to their amazing LLM models because it's such an easy integration done in cloudflare, but I want to deploy/run all my code somehwere else. Does anyone have any experience doing this?

MMaximilian Quindinkus Hey all, so I have been trying to use different multi-agent frameworks in cloudf...

D

Derpenstein69•12/7/24, 4:26 AM

Langchain and Vercel Ai Sdk have adapters for cloudflare that i am aware of

J

JOEY•12/8/24, 8:46 PM

what kind of error is this?

J

JOEY•12/8/24, 8:46 PM

anyone have any idea ? i'm new to AI and cloudflare as well

DDCsan is this a know problem? getting stuck on even running the tutorial apps: ``` Yo...

J

JOEY•12/8/24, 8:47 PM

hey mate this is the same error i got stuck with @DCsan how did it resolved?

E

Erisa•12/9/24, 11:40 AM

the answer is directly underneath the message you replied to

JJOEY hey mate this is the same error i got stuck with <@257212035232825345> how did i...

A

acusti•12/10/24, 5:46 PM

a little more detail: the

@cloudflare/ai

@cloudflare/ai

@cloudflare/ai

@cloudflare/ai package has been deprecated in favor of running AI requests directly via the workers bindings as described in https://developers.cloudflare.com/workers-ai/configuration/bindings/

NNeilBee Have you got something working with Langchain and successfully deployed to worke...

F

Fokke•12/10/24, 10:53 PM

Hah thats my post on Reddit. Glad it worked for you. Quite honestly I would just build it without langchain. Making anything work with Langchain on CF is a big pain and IMO Langchain is a lot of bloat you can skip. We actually build a whole SOTA RAG thing on CF couple blogs in case your interested: https://liquidmetal.ai/casesAndBlogs/sota-rag-indexer/ https://liquidmetal.ai/casesAndBlogs/sota-rag-retriever/ https://liquidmetal.ai/casesAndBlogs/agents

The last one uses our own platform that manages Cloudflare resources for us. We realized that Cloudflare was missing an orchestration layer and build it ourselves to make our lives easier. https://liquidmetal.ai/build

M

Mitya•12/12/24, 12:15 PM

Hi all. What is the best way to ensure a CF LLM returns JSON? I realise there's AI Functions, but the Mistral model that powers it seems (at least to me) much less capable, and slower, than some of the Llama text generation models. Is it reliable, do you think, to expect a text-gen model to return JSON when asked, or might it sometimes disobey? Right now I'm using the prompt to get JSON ("Return a JSON array and no other text.")

I

Isaac McFadyen•12/12/24, 2:51 PM

It'll disobey sometimes because that's just how LLMs work

I

Isaac McFadyen•12/12/24, 2:51 PM

Unless you use structured generation which uses math to ensure the logits picked always match valid JSON, there's no real way aside from retries

I

Isaac McFadyen•12/12/24, 2:51 PM

Instructing the LLM to return JSON will work mostly but sometimes it'll ignore that.

Original message was deleted

I

Isaac McFadyen•12/12/24, 2:54 PM

Yeah this actually works lul

I

Isaac McFadyen•12/12/24, 2:54 PM

give grandmother $1000

M

Mitya•12/12/24, 3:37 PM

Interesting! Thanks, guys

MMitya Hi all. What is the best way to ensure a CF LLM returns JSON? I realise there's ...

C

can•12/13/24, 10:18 PM

I'm using a helper called StructuredOutputParser from langchain/core/output_parsers, which you can use a zod schema in order to enforce the result type.

import { StructuredOutputParser } from "@langchain/core/output_parsers";

const parser = StructuredOutputParser.fromZodSchema(generateFixSchema);

import { StructuredOutputParser } from "@langchain/core/output_parsers";

const parser = StructuredOutputParser.fromZodSchema(generateFixSchema);

import { StructuredOutputParser } from "@langchain/core/output_parsers";

const parser = StructuredOutputParser.fromZodSchema(generateFixSchema);

import { StructuredOutputParser } from "@langchain/core/output_parsers";

const parser = StructuredOutputParser.fromZodSchema(generateFixSchema);

i'm using this parser with the input:

    Use the following analysis information to generate precise, non-redundant fixes:
    Generate fixes following these guidelines:
    1. ..........
    2. ..........
    3. ..........
    4. ..........

    addAttribute: This method is used to add an attribute to the element or override the existing attribute.
    injectCss: This method is used to add style values to the element.

    ${parser.getFormatInstructions()}

    Use the following analysis information to generate precise, non-redundant fixes:
    Generate fixes following these guidelines:
    1. ..........
    2. ..........
    3. ..........
    4. ..........

    addAttribute: This method is used to add an attribute to the element or override the existing attribute.
    injectCss: This method is used to add style values to the element.

    ${parser.getFormatInstructions()}

    Use the following analysis information to generate precise, non-redundant fixes:
    Generate fixes following these guidelines:
    1. ..........
    2. ..........
    3. ..........
    4. ..........

    addAttribute: This method is used to add an attribute to the element or override the existing attribute.
    injectCss: This method is used to add style values to the element.

    ${parser.getFormatInstructions()}

    Use the following analysis information to generate precise, non-redundant fixes:
    Generate fixes following these guidelines:
    1. ..........
    2. ..........
    3. ..........
    4. ..........

    addAttribute: This method is used to add an attribute to the element or override the existing attribute.
    injectCss: This method is used to add style values to the element.

    ${parser.getFormatInstructions()}

this basically adds a descriptive text to the prompt which explains what is the JSON structure and how you(LLM) should apply this to the result. In the end, it outputs a JSON with the correct fields. Maybe you can try this. I'm sure you can just reverse engineer the process and write your own helper.

I'll try to post the result of the

getFormatInstructions

getFormatInstructions

getFormatInstructions

getFormatInstructions when i'm home.

MMitya Interesting! Thanks, guys

R

rob•12/14/24, 1:25 PM

Provide an example and start the assistances response with a { for an object and it would continue the next message as the JSON

R

rob•12/14/24, 1:25 PM

So example send in conversation history / messages of the assistant starting off a json object

Rrob Provide an example and start the assistances response with a { for an object and...

I

Isaac McFadyen•12/14/24, 3:18 PM

While the first part would work, the second part (the prefix of

{

{) would only work with the raw completions API, not the chat completions API.

R

rob•12/15/24, 3:03 AM

i had ut wrking on llama on Workers AI and qwen but ymmv

K

Karim•12/15/24, 6:27 PM

Hi. I am testing llama-3.2 vision model (https://developers.cloudflare.com/workers-ai/models/llama-3.2-11b-vision-instruct/)

image one of

    0 array

    An array of integers that represent the image data constrained to 8-bit unsigned integer values
        items number

        A value between 0 and 255
    1 string

    Binary string representing the image contents.

image one of

    0 array

    An array of integers that represent the image data constrained to 8-bit unsigned integer values
        items number

        A value between 0 and 255
    1 string

    Binary string representing the image contents.

image one of

    0 array

    An array of integers that represent the image data constrained to 8-bit unsigned integer values
        items number

        A value between 0 and 255
    1 string

    Binary string representing the image contents.

image one of

    0 array

    An array of integers that represent the image data constrained to 8-bit unsigned integer values
        items number

        A value between 0 and 255
    1 string

    Binary string representing the image contents.

I am able to include image as array of 8-bit unsigned integers successfully. However, I am not having any such luck with binary string representation.
I can't find any documentation anywhere either.
I tried base64 encoded file content with and without the header (data:image/png;base64).
I get the following error back:

Tensor error: Unknown internal error: failed to decode u8

Tensor error: Unknown internal error: failed to decode u8

Tensor error: Unknown internal error: failed to decode u8

Tensor error: Unknown internal error: failed to decode u8
I also took ChatGPT's hopeless advise and tried "binary" string like 111000... to no success (as was expected).
Python code to help with this issue would be greatly appreciated.

Cloudflare Docs

llama-3.2-11b-vision-instruct · Cloudflare Workers AI docs

The Llama 3.2-Vision instruction-tuned models are optimized for visual recognition, image reasoning, captioning, and answering general questions about an image.

Ccan I'm using a helper called StructuredOutputParser from langchain/core/output_pars...

A

acusti•12/18/24, 5:45 AM

i wrote an extremely lenient parser in typescript (published to npm) specifically to handle bad JSON output from LLMs: https://github.com/acusti/uikit/tree/main/packages/parsing

import { parseAsJSON } from '@acusti/parsing';
const parsedResult = parseAsJSON(resultFromLLM);

import { parseAsJSON } from '@acusti/parsing';
const parsedResult = parseAsJSON(resultFromLLM);

import { parseAsJSON } from '@acusti/parsing';
const parsedResult = parseAsJSON(resultFromLLM);

import { parseAsJSON } from '@acusti/parsing';
const parsedResult = parseAsJSON(resultFromLLM);

i adapted it originally from langchain (https://github.com/langchain-ai/langchainjs/blob/215dd52/langchain-core/src/output_parsers/json.ts#L58) but added a ton of extra parsing to deal with real-world garbage serialized json i’ve received from LLMs over the last year (see examples in the unit tests).

K

kingmesal•12/19/24, 11:25 PM

It has been a while since I've asked, but... here we are...

Any road map updates on text embedding models that can handle more input tokens? something like

BAAI/bge-m3

BAAI/bge-m3

BAAI/bge-m3

BAAI/bge-m3 ... but I'm not super picky...

C

CiRCUiT•12/24/24, 8:06 AM

Using Google Gemini 2.0 Flash Model with Cloudflare Relay
Hi, I want to integrate the Google Gemini 2.0 Flash model with Cloudflare Relay. I'm referring to an attached YouTube video that demonstrates integrating OpenAI's real-time API. However, I want to use Google's Gemini 2.0 Voice API instead. Could someone guide me on how to achieve this?
Youtube Video: https://www.youtube.com/watch?v=xu4Wb-IppmM

YouTubeCloudflare Developers

A Free Relay Server for OpenAI's New Realtime API on Cloudflare Wor...

In this video, Craig Dennis will walk you through the deployment of OpenAI's relay server to use with their realtime API. Using OpenAI's relay server, Craig created a template repository and modified it so it can be deployed easily to Cloudflare Workers. By deploying to the Cloudflare network, you can connect to it from a frontend (like a web ap...

�

😈 Donkey 💫•12/24/24, 11:55 AM

@cf/meta/llama-3.2-11b-vision-instruct
completely failed

please check it

�

😈 Donkey 💫•12/24/24, 11:55 AM

D

Dr. Software•12/25/24, 3:21 PM

is it intentional/normal that i can use some hugging-face models that aren't listed on https://developers.cloudflare.com/workers-ai/models/ ?

Cloudflare Docs

Models · Cloudflare Workers AI docs

D

Dr. Software•12/25/24, 3:21 PM

same for other providers

D

Dr. Software•12/25/24, 3:22 PM

wait nvm i figured it out lol, all good

M

Mitya•12/29/24, 5:58 PM

How can I get embedded function calling to return the response of the attached function? It seems my function's response is fed back into the LLM, for reasons unknown. I want the LLM only to prepare and pass arguments to my function (as with traditional function calling), not to do anything with the function response.

router.get('/fruitInfo2', async (req, env) => {
    return runWithTools(env.AI, model, {
        messages: [{
            role: 'system',
            content: `
                The user message is a cryptic clue about a fruit.
                Extract the name and shape of the fruit.
            `
        }, {
            role: 'user',
            content: 'garden of eden'
        }],
        tools: [{
            name: 'fruitInfo',
            description: 'Extract the fruit name and shape.',
            parameters: {
                type: 'object',
                properties: {
                    name: {type: 'string', description: 'The name of the fruit'},
                    shape: {type: 'string', description: 'The shape of the fruit'}
                },
                required: ['name', 'shape']
            },
            function: ({name, shape}) => `Fruit is ${name} and shape is ${shape}`
        }]
    });
})

router.get('/fruitInfo2', async (req, env) => {
    return runWithTools(env.AI, model, {
        messages: [{
            role: 'system',
            content: `
                The user message is a cryptic clue about a fruit.
                Extract the name and shape of the fruit.
            `
        }, {
            role: 'user',
            content: 'garden of eden'
        }],
        tools: [{
            name: 'fruitInfo',
            description: 'Extract the fruit name and shape.',
            parameters: {
                type: 'object',
                properties: {
                    name: {type: 'string', description: 'The name of the fruit'},
                    shape: {type: 'string', description: 'The shape of the fruit'}
                },
                required: ['name', 'shape']
            },
            function: ({name, shape}) => `Fruit is ${name} and shape is ${shape}`
        }]
    });
})

router.get('/fruitInfo2', async (req, env) => {
    return runWithTools(env.AI, model, {
        messages: [{
            role: 'system',
            content: `
                The user message is a cryptic clue about a fruit.
                Extract the name and shape of the fruit.
            `
        }, {
            role: 'user',
            content: 'garden of eden'
        }],
        tools: [{
            name: 'fruitInfo',
            description: 'Extract the fruit name and shape.',
            parameters: {
                type: 'object',
                properties: {
                    name: {type: 'string', description: 'The name of the fruit'},
                    shape: {type: 'string', description: 'The shape of the fruit'}
                },
                required: ['name', 'shape']
            },
            function: ({name, shape}) => `Fruit is ${name} and shape is ${shape}`
        }]
    });
})

router.get('/fruitInfo2', async (req, env) => {
    return runWithTools(env.AI, model, {
        messages: [{
            role: 'system',
            content: `
                The user message is a cryptic clue about a fruit.
                Extract the name and shape of the fruit.
            `
        }, {
            role: 'user',
            content: 'garden of eden'
        }],
        tools: [{
            name: 'fruitInfo',
            description: 'Extract the fruit name and shape.',
            parameters: {
                type: 'object',
                properties: {
                    name: {type: 'string', description: 'The name of the fruit'},
                    shape: {type: 'string', description: 'The shape of the fruit'}
                },
                required: ['name', 'shape']
            },
            function: ({name, shape}) => `Fruit is ${name} and shape is ${shape}`
        }]
    });
})

M

Mitya•12/29/24, 5:59 PM

Instead I get a response like "The cryptic clue "garden of eden" seems unrelated to a fruit at first, but it ...".

MMitya How can I get embedded function calling to return the response of the attached f...

R

Raylight•12/30/24, 9:29 AM

You're correct that your function's response is fed back.

runWithTools

runWithTools

runWithTools

runWithTools calls

env.AI.run

env.AI.run

env.AI.run

env.AI.run multiple times, and tool calls + tool responses are added to a temporary copy of the message history. The final response is (re)generated by calling

env.AI.run

env.AI.run

env.AI.run

env.AI.run one last time without passing any

tools

tools

tools

tools array.

So you'd have to modify

runWithTools

runWithTools

runWithTools

runWithTools or write your own to get the behavior you want. The implementation is (IMHO) kinda difficult to follow, though. I've written a tiny version (https://gist.github.com/Raylight-PWL/7487b4210e593e00fffcab4f7abe0189#file-3_runwithtoolsx-js) that might be easier, since the part about "embedded" is written as a separate usable component ("runEmbedded") and sounds close to what you want.

RRaylight You're correct that your function's response is fed back. `runWithTools` calls `...

M

Mitya•12/30/24, 9:55 AM

Thanks, that's really interesting (and I don't believe that important difference, i.e. that each function response is fed back into the LLM, is made explicitly clear by the CF docs at all, though perhaps I just missed something.) Appreciate the help.

S

Shawn•12/30/24, 12:37 PM

https://tenor.com/view/excited-i%27m-so-excited-so-excited-yay-happy-gif-16375819648008939887

Tenor

T

Trojan•12/31/24, 2:12 PM

can we finetune an AI on this discord so i can ask it questions?

TTrojan can we finetune an AI on this discord so i can ask it questions?

I

Isaac McFadyen•1/1/25, 3:39 AM

Discord messages are considered property of their senders so you'd need consent from all of the people in the channels. For example, we have AnswerOverflow (the bot) in here to index messages but in order to do that we have a message in the channel description indicating that the bot is doing that and a slash-command for people to opt-out if they wish.

I

Isaac McFadyen•1/1/25, 3:39 AM

It's also against the Discord ToS to self-bot (have a bot that runs on your own account rather than a dedicated bot account with restricted permissions).

T

Trojan•1/1/25, 5:08 AM

oh I meant “we” as a collective, not “we” as in an organization i’m a part of

T

Trojan•1/1/25, 5:09 AM

regardless, surprising take by discord. i always assumed my messages in public servers were like mini open source projects i didn’t attach a license to

T

Trojan•1/3/25, 4:15 PM

does cloudflare use llama.cpp behind the scenes? Would there be any interest in a GBNF API? I would spend so much money… https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md

GitHub

llama.cpp/grammars/README.md at master · ggerganov/llama.cpp

LLM inference in C/C++. Contribute to ggerganov/llama.cpp development by creating an account on GitHub.

TTrojan does cloudflare use llama.cpp behind the scenes? Would there be any interest in ...

I

Isaac McFadyen•1/4/25, 1:25 AM

I believe they currently use vLLM but the exact stack isn't public.

I

Isaac McFadyen•1/4/25, 1:25 AM

As far as I know they do not use llama.cpp.

I

Isaac McFadyen•1/4/25, 1:25 AM

llama.cpp is primarily designed for edge devices or consumer devices whereas other frameworks such as vLLM or Triton are better for serving many models on datacenter-class (40GB+ VRAM) GPUs.

IIsaac McFadyen llama.cpp is primarily designed for edge devices or consumer devices whereas oth...

T

Trojan•1/4/25, 1:53 AM

llama.cpp is indeed more performant which is why it’s better suited for edge and serverless devices

T

Trojan•1/4/25, 1:54 AM

vLLM and Triton were a response to a fast growing ecosystem and the end production inference server of choice will not be written in Python

TTrojan llama.cpp is indeed more performant which is why it’s better suited for edge and...

I

Isaac McFadyen•1/4/25, 2:02 AM

It's entirely use-case based though. Serving at scale is not the same as an end-user device. On an end-user device you're concerned about speed, but with a large-scale deployment you need to be able to run many concurrent requests in parallel. Although they're a touch old, see the benchmarks for 70b models here, for example. With 1 concurrent user, llama.cpp wins, but with 32 concurrent users, vLLM wins by a large margin (40 req/min vs 172). https://github.com/ggerganov/llama.cpp/discussions/6730

TTrojan vLLM and Triton were a response to a fast growing ecosystem and the end producti...

I

Isaac McFadyen•1/4/25, 2:04 AM

As for this argument, that might be the case but vLLM calls into C++ for inference (as does any major framework) for most parts excluding the actual HTTP serving. llama.cpp might be faster on the HTTP side but vLLM is still very much trusted by large organizations for deploying models, as can be seen from their Github where they list organizations that support it: https://github.com/vllm-project/vllm

IIsaac McFadyen It's entirely use-case based though. Serving at scale is not the same as an end-...

T

Trojan•1/4/25, 2:30 AM

right. looks like llama.cpp multi gpu code still needs some work ): https://www.reddit.com/r/LocalLLaMA/comments/1ge1ojk/updated_with_corrected_settings_for_llamacpp/

From the LocalLLaMA community on Reddit: Updated with corrected set...

Explore this post and more from the LocalLLaMA community

IIsaac McFadyen As for this argument, that might be the case but vLLM calls into C++ for inferen...

T

Trojan•1/4/25, 2:31 AM

i wonder how much gains can be seen by better parallelism code and whether we’ll see orgs move to a more maintainable solution with less dependencies

TTrojan i wonder how much gains can be seen by better parallelism code and whether we’ll...

I

Isaac McFadyen•1/4/25, 2:32 AM

Yeah. I'm certainly not arguing that it's nice to have a compiled framework sometimes, I just don't know that we're quite there yet

Have you got something working with Langchain and successfully deployed to workers? I was running in

Similar Threads

Similar Threads

Similar Threads