How can I get embedded function calling to return the response of the attached function? It seems my

How can I get embedded function calling to return the response of the attached function? It seems my function's response is fed back into the LLM, for reasons unknown. I want the LLM only to prepare and pass arguments to my function (as with traditional function calling), not to do anything with the function response.

router.get('/fruitInfo2', async (req, env) => {
    return runWithTools(env.AI, model, {
        messages: [{
            role: 'system',
            content: `
                The user message is a cryptic clue about a fruit.
                Extract the name and shape of the fruit.
            `
        }, {
            role: 'user',
            content: 'garden of eden'
        }],
        tools: [{
            name: 'fruitInfo',
            description: 'Extract the fruit name and shape.',
            parameters: {
                type: 'object',
                properties: {
                    name: {type: 'string', description: 'The name of the fruit'},
                    shape: {type: 'string', description: 'The shape of the fruit'}
                },
                required: ['name', 'shape']
            },
            function: ({name, shape}) => `Fruit is ${name} and shape is ${shape}`
        }]
    });
})

router.get('/fruitInfo2', async (req, env) => {
    return runWithTools(env.AI, model, {
        messages: [{
            role: 'system',
            content: `
                The user message is a cryptic clue about a fruit.
                Extract the name and shape of the fruit.
            `
        }, {
            role: 'user',
            content: 'garden of eden'
        }],
        tools: [{
            name: 'fruitInfo',
            description: 'Extract the fruit name and shape.',
            parameters: {
                type: 'object',
                properties: {
                    name: {type: 'string', description: 'The name of the fruit'},
                    shape: {type: 'string', description: 'The shape of the fruit'}
                },
                required: ['name', 'shape']
            },
            function: ({name, shape}) => `Fruit is ${name} and shape is ${shape}`
        }]
    });
})

router.get('/fruitInfo2', async (req, env) => {
    return runWithTools(env.AI, model, {
        messages: [{
            role: 'system',
            content: `
                The user message is a cryptic clue about a fruit.
                Extract the name and shape of the fruit.
            `
        }, {
            role: 'user',
            content: 'garden of eden'
        }],
        tools: [{
            name: 'fruitInfo',
            description: 'Extract the fruit name and shape.',
            parameters: {
                type: 'object',
                properties: {
                    name: {type: 'string', description: 'The name of the fruit'},
                    shape: {type: 'string', description: 'The shape of the fruit'}
                },
                required: ['name', 'shape']
            },
            function: ({name, shape}) => `Fruit is ${name} and shape is ${shape}`
        }]
    });
})

router.get('/fruitInfo2', async (req, env) => {
    return runWithTools(env.AI, model, {
        messages: [{
            role: 'system',
            content: `
                The user message is a cryptic clue about a fruit.
                Extract the name and shape of the fruit.
            `
        }, {
            role: 'user',
            content: 'garden of eden'
        }],
        tools: [{
            name: 'fruitInfo',
            description: 'Extract the fruit name and shape.',
            parameters: {
                type: 'object',
                properties: {
                    name: {type: 'string', description: 'The name of the fruit'},
                    shape: {type: 'string', description: 'The shape of the fruit'}
                },
                required: ['name', 'shape']
            },
            function: ({name, shape}) => `Fruit is ${name} and shape is ${shape}`
        }]
    });
})

MityaOP•12/29/24, 5:59 PM

Instead I get a response like "The cryptic clue "garden of eden" seems unrelated to a fruit at first, but it ...".

MMitya How can I get embedded function calling to return the response of the attached f...

Raylight•12/30/24, 9:29 AM

You're correct that your function's response is fed back. runWithToolsrunWithTools calls env.AI.runenv.AI.run multiple times, and tool calls + tool responses are added to a temporary copy of the message history. The final response is (re)generated by calling env.AI.runenv.AI.run one last time without passing any toolstools array.

So you'd have to modify runWithToolsrunWithTools or write your own to get the behavior you want. The implementation is (IMHO) kinda difficult to follow, though. I've written a tiny version (https://gist.github.com/Raylight-PWL/7487b4210e593e00fffcab4f7abe0189#file-3_runwithtoolsx-js) that might be easier, since the part about "embedded" is written as a separate usable component ("runEmbedded") and sounds close to what you want.

RRaylight You're correct that your function's response is fed back. `runWithTools` calls `...

MityaOP•12/30/24, 9:55 AM

Thanks, that's really interesting (and I don't believe that important difference, i.e. that each function response is fed back into the LLM, is made explicitly clear by the CF docs at all, though perhaps I just missed something.) Appreciate the help.

Shawn•12/30/24, 12:37 PM

https://tenor.com/view/excited-i%27m-so-excited-so-excited-yay-happy-gif-16375819648008939887

Tenor

Trojan•12/31/24, 2:12 PM

can we finetune an AI on this discord so i can ask it questions?

TTrojan can we finetune an AI on this discord so i can ask it questions?

Isaac McFadyen•1/1/25, 3:39 AM

Discord messages are considered property of their senders so you'd need consent from all of the people in the channels. For example, we have AnswerOverflow (the bot) in here to index messages but in order to do that we have a message in the channel description indicating that the bot is doing that and a slash-command for people to opt-out if they wish.

Isaac McFadyen•1/1/25, 3:39 AM

It's also against the Discord ToS to self-bot (have a bot that runs on your own account rather than a dedicated bot account with restricted permissions).

Trojan•1/1/25, 5:08 AM

oh I meant “we” as a collective, not “we” as in an organization i’m a part of

Trojan•1/1/25, 5:09 AM

regardless, surprising take by discord. i always assumed my messages in public servers were like mini open source projects i didn’t attach a license to

Trojan•1/3/25, 4:15 PM

does cloudflare use llama.cpp behind the scenes? Would there be any interest in a GBNF API? I would spend so much money… https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md

GitHub

llama.cpp/grammars/README.md at master · ggerganov/llama.cpp

LLM inference in C/C++. Contribute to ggerganov/llama.cpp development by creating an account on GitHub.

TTrojan does cloudflare use llama.cpp behind the scenes? Would there be any interest in ...

Isaac McFadyen•1/4/25, 1:25 AM

I believe they currently use vLLM but the exact stack isn't public.

Isaac McFadyen•1/4/25, 1:25 AM

As far as I know they do not use llama.cpp.

Isaac McFadyen•1/4/25, 1:25 AM

llama.cpp is primarily designed for edge devices or consumer devices whereas other frameworks such as vLLM or Triton are better for serving many models on datacenter-class (40GB+ VRAM) GPUs.

IIsaac McFadyen llama.cpp is primarily designed for edge devices or consumer devices whereas oth...

Trojan•1/4/25, 1:53 AM

llama.cpp is indeed more performant which is why it’s better suited for edge and serverless devices

Trojan•1/4/25, 1:54 AM

vLLM and Triton were a response to a fast growing ecosystem and the end production inference server of choice will not be written in Python

TTrojan llama.cpp is indeed more performant which is why it’s better suited for edge and...

Isaac McFadyen•1/4/25, 2:02 AM

It's entirely use-case based though. Serving at scale is not the same as an end-user device. On an end-user device you're concerned about speed, but with a large-scale deployment you need to be able to run many concurrent requests in parallel. Although they're a touch old, see the benchmarks for 70b models here, for example. With 1 concurrent user, llama.cpp wins, but with 32 concurrent users, vLLM wins by a large margin (40 req/min vs 172). https://github.com/ggerganov/llama.cpp/discussions/6730

TTrojan vLLM and Triton were a response to a fast growing ecosystem and the end producti...

Isaac McFadyen•1/4/25, 2:04 AM

As for this argument, that might be the case but vLLM calls into C++ for inference (as does any major framework) for most parts excluding the actual HTTP serving. llama.cpp might be faster on the HTTP side but vLLM is still very much trusted by large organizations for deploying models, as can be seen from their Github where they list organizations that support it: https://github.com/vllm-project/vllm

IIsaac McFadyen It's entirely use-case based though. Serving at scale is not the same as an end-...

Trojan•1/4/25, 2:30 AM

right. looks like llama.cpp multi gpu code still needs some work ): https://www.reddit.com/r/LocalLLaMA/comments/1ge1ojk/updated_with_corrected_settings_for_llamacpp/

From the LocalLLaMA community on Reddit: Updated with corrected set...

Explore this post and more from the LocalLLaMA community

IIsaac McFadyen As for this argument, that might be the case but vLLM calls into C++ for inferen...

Trojan•1/4/25, 2:31 AM

i wonder how much gains can be seen by better parallelism code and whether we’ll see orgs move to a more maintainable solution with less dependencies

TTrojan i wonder how much gains can be seen by better parallelism code and whether we’ll...

Isaac McFadyen•1/4/25, 2:32 AM

Yeah. I'm certainly not arguing that it's nice to have a compiled framework sometimes, I just don't know that we're quite there yet

IIsaac McFadyen Yeah. I'm certainly not arguing that it's nice to have a compiled framework some...

Trojan•1/4/25, 2:33 AM

i’ve certainly suffered from dependency hell. i encourage you to check out GBNF though. these grammars are a hugely underutilized feature. people are still stuck using langchain when there’s so much more expressive power out there

c11network•1/6/25, 1:42 AM

Hey there,
I posted in General Support forum but I'm not sure where my post went from earlier, should I post my question in here?
I even tried to search the title but nothing came up?

c11network•1/6/25, 1:46 AM

Nevermind, found it... :/ It's on CF 0T + OLLAMA API if anyone has any suggestions

Trojan•1/6/25, 1:24 PM

will we see cloudflare train any native models?

Trojan•1/6/25, 1:25 PM

even if it was something like a network traffic sentiment analyzer. it would be really cool to me

SomeyAI•1/14/25, 10:06 PM

Hey, quick question, where is the best place to inform Cloudflare about jailbreaks for the AI's? Some users of my app discovered a disturbing jailbreak, I fixed it on my app but where can I report it to cluodflare so other apps don't suffer the same?

SSomeyAI Hey, quick question, where is the best place to inform Cloudflare about jailbrea...

Isaac McFadyen•1/14/25, 10:14 PM

Cloudflare doesn't produce the AI models they host, so they can't do anything to fix it really. You'd have to look and see if there's a specific spot to report those for the models themselves. Honestly though, it's entirely up to how a model is prompted and I doubt there's a method to report jailbreaks because they're kind of a given with LLMs at the moment (any LLM can be jailbroken with enough time and effort).

IIsaac McFadyen Cloudflare doesn't produce the AI models they host, so they can't do anything to...

SomeyAI•1/14/25, 10:16 PM

Yes, I understand that. However the models Cloudflare is providing are capable of producing content which should 100% not be possible with any model out there. I don't want to exactly discuss this in public, however my website got targeted by multiple prompts generating very very bad images.

SSomeyAI Yes, I understand that. However the models Cloudflare is providing are capable o...

Isaac McFadyen•1/14/25, 10:17 PM

Yes, but even then, Cloudflare cannot do anything about it. They are given the models by the people who train/produce them (Meta, independent authors, etc) and just host the models. They have no control over anything the model does/does not generate.

SomeyAI•1/14/25, 10:17 PM

Is there any way I can check images Cloudflare produces against prohibited contents?

SSomeyAI Is there any way I can check images Cloudflare produces against prohibited conte...

Isaac McFadyen•1/14/25, 10:19 PM

If it's related to CSAM then you can enable the CSAM scanning tool on your site https://developers.cloudflare.com/cache/reference/csam-scanning/ but that isn't realtime/instant. Other than that, there's models like LLaMa Guard (on Workers AI) for filtering text content but I don't believe there's a model on Workers AI for filtering image content.

Muhammadreza Haghiri•1/20/25, 8:19 PM

Hello. Just a question, it is still impossible to host our own models. Will it be possible anytime soon?

Original message was deleted

Isaac McFadyen•1/27/25, 4:34 PM

Try cloudflare-go, this channel is for Cloudflare's AI products.

Spotnag•1/27/25, 4:36 PM

did you have to delete it? i can't copy paste it there now and wil have to rewrite it... -.-

SSpotnag did you have to delete it? i can't copy paste it there now and wil have to rewri...

Isaac McFadyen•1/27/25, 4:39 PM

Oh whoops, sorry.

lfdepombo•1/27/25, 10:40 PM

Hi, I ran the example here and it works without issue - https://developers.cloudflare.com/workers-ai/models/llava-1.5-7b-hf/ I then used an image downloaded from the whatsapp api and I can download it correctly without a problem in my local machine or KV. However, the AI worker returns

  (error) Error processing message: AiError: 3010: Invalid or incomplete input for the model: model returned: Failed to decode image: cannot identify image file <_io.BytesIO object at 0x7ee1b5d57a10>

  (error) Error processing message: AiError: 3010: Invalid or incomplete input for the model: model returned: Failed to decode image: cannot identify image file <_io.BytesIO object at 0x7ee1b5d57a10>

Cloudflare Docs

llava-1.5-7b-hf · Cloudflare Workers AI docs

LLaVA is an open-source chatbot trained by fine-tuning LLaMA/Vicuna on GPT-generated multimodal instruction-following data. It is an auto-regressive language model, based on the transformer architecture.

lfdepombo•1/27/25, 10:46 PM

any idea why this would happen in the worker, but not locally?

Vero•1/28/25, 9:31 AM

------------------------------

This channel is archived, please use workers-ai ai-gateway vectorize

------------------------------

How can I get embedded function calling to return the response of the attached function? It seems my

Similar Threads

Similar Threads

Similar Threads