Going to ping the AI gateway PM here because this issue has come up before (cc <@1136363561472491551

Going to ping the AI gateway PM here because this issue has come up before (cc @Kathy).

TLDR: Gemini isn't allowed to be access from Hong Kong but AI gateway is routing requests through HKG, causing them to be denied. I believe this came up with OpenAI before? Not sure what the resolution was or if you forced the routing to avoid certain countries internally?

digendee•9/15/24, 1:54 AM

Feature request is here in case you want to contribute / vote:

digendee•9/15/24, 1:54 AM

https://community.cloudflare.com/t/do-not-put-workers-in-llm-forbidden-locations/711754

digendee•9/15/24, 1:58 AM

My solution for the time being is to move my LLM-access APIs from Cloudflare Workers to something like Google's Cloud Functions, however my preference is to use Cloudflare Workers if this issue / problem can be resolved in the near future.

digendee•9/15/24, 1:59 AM

@Isaac McFadyen do you know who the Workers PM is? The issue could also be resolved with changes to Worker locations.

Ddigendee <@575799975116800030> do you know who the Workers PM is? The issue could also be...

Isaac McFadyenOP•9/15/24, 2:06 AM

It's more likely that this would be resolved specific to AI Gateway, because this happened before with OpenAI and I believe it was resolved, and I don't believe pinning a Worker to a specific (user-defined) location is planned or going to be implemented any time soon.

ahlstrominfo•9/16/24, 1:47 PM

When I use @cf/baai/bge-base-en-v1.5 and compare output to locally using the same embeddings with pyton or ollama or anything. I get a different embedding for the same word. How can that be?

oof2win2•9/16/24, 4:15 PM

each model creates a different embedding based on its training and understanding of the meaning

Ooof2win2 each model creates a different embedding based on its training and understanding...

ahlstrominfo•9/16/24, 5:52 PM

Completely understandable. But if I use the same model on other device should it not return the same result?

oof2win2•9/16/24, 5:53 PM

not necessarily. they will be similar (even across various models stuff will be similar) but you most likely wont always get the exact same answer

oof2win2•9/16/24, 5:53 PM

ai is just not deterministic

oof2win2•9/16/24, 5:55 PM

like its designed to randomly pick the next token. you can reduce the temperature to make it be less creative (tho idk if embeddings will have temperature, as embeddings are input and not output)

IIsaac McFadyen Going to ping the AI gateway PM here because this issue has come up before (cc <...

digendee•9/17/24, 12:47 PM

@Kathy any updates?

a4amaan•9/17/24, 5:06 PM

Its generating near about 300 words is there any limit of neurons for free users?

const answer = await ctx.env.AI.run('@cf/meta/llama-3-8b-instruct', {
        prompt: "Write 2000 words article",
        stream: true
    });

const answer = await ctx.env.AI.run('@cf/meta/llama-3-8b-instruct', {
        prompt: "Write 2000 words article",
        stream: true
    });

Sunny•9/17/24, 5:09 PM

i think its 10,000 neurons per day for free plan

SSunny i think its 10,000 neurons per day for free plan

a4amaan•9/17/24, 5:12 PM

yes i've read that.. i have not used neurons and getting about 300 words of article? is there any limit for single call?

Sunny•9/17/24, 5:15 PM

don't think so, it doesn't seem that you are setting a max_token limit here so it might be something to do with the llama 3 model?

Sunny•9/17/24, 5:15 PM

found this on hugging face forums for the llama 3 model

Sunny•9/17/24, 5:15 PM

"The output takes the same space as the input. The model completes the 8k token space with the response. You could slide the contxt window and get more output, but then you're losing context at the beginning.

Example, if your input is 100 tokens, you have ~7900 tokens for completion. But if your input is 7900 tokens, you have ~100 tokens left for response until your input starts getting trimmed out. The model can only pay attention to 8k tokens at most."

Sunny•9/17/24, 5:15 PM

I use llama 3.1 which has 128k token context window, it might be worth the upgrade in your use case if you are going over limits

SSunny I use llama 3.1 which has 128k token context window, it might be worth the upgra...

a4amaan•9/17/24, 5:18 PM

that suites good, thanks

a4amaan•9/17/24, 6:01 PM

@Sunny still not getting more than 300 words with 3.1

a4amaan•9/17/24, 6:04 PM

at the end it look like something stopped it, incomplete article

Sunny•9/17/24, 6:09 PM

it seems like there's a limit being set, maybe try manually setting max_tokens/max_length to be a higher limit? You could just try arbitrarily setting it to 100k?

SSunny it seems like there's a limit being set, maybe try manually setting max_tokens/m...

a4amaan•9/17/24, 6:21 PM

Thanks it worked:

const answer = await ctx.env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
        messages,
        stream: true,
        max_tokens:10000
    });

const answer = await ctx.env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
        messages,
        stream: true,
        max_tokens:10000
    });

Avocadio•9/18/24, 11:48 AM

Hey. I uploaded my finetuned model to cloudflare successfully but typescript gave that error and it request gives 3028 unknown internal error

Avocadio•9/18/24, 11:49 AM

It gives that error when i add lora parameter

Avocadio•9/18/24, 11:49 AM

Doesnt work with tsignore neither

Avocadio•9/18/24, 11:49 AM

Can you guys help

Avocadio•9/18/24, 11:55 AM

https://community.cloudflare.com/t/unknown-internal-error-for-lora-adapters/692895/3
Some other people faced the same issue

Ddigendee <@1136363561472491551> any updates?

Kathy•9/18/24, 8:38 PM

could i get your account ID and script name?

KKathy could i get your account ID and script name?

ajgeiss0702•9/19/24, 1:27 AM

i am currently getting the same issue (the 3028 error, not typescript). My account ID is f55b85c8a963663b11036975203c63c0f55b85c8a963663b11036975203c63c0. I am using the http api to run it so i dont have a script name for this

ajgeiss0702•9/19/24, 1:28 AM

I am trying with the @cf/mistral/mistral-7b-instruct-v0.1@cf/mistral/mistral-7b-instruct-v0.1 model because i didnt know the v0.2 model existed on cloudflare until 2 minutes ago

ajgeiss0702•9/19/24, 1:29 AM

{"errors":[{"message":"AiError: AiError: Unknown internal error","code":3028}],"success":false,"result":{},"messages":[]}

{"errors":[{"message":"AiError: AiError: Unknown internal error","code":3028}],"success":false,"result":{},"messages":[]}

Aajgeiss0702 i am currently getting the same issue (the 3028 error, not typescript). My accou...

Kathy•9/19/24, 10:17 AM

wait i dont think this is the same issue at all. @digendee 's issue is because their worker is getting routed through HKG (which is blocked by Google Gemini) even though they're based in Melbourne (allowed by Google Gemini)

Mr.Spok•9/19/24, 5:44 PM

Hello everyone, please tell me about the application for Windows Cloudflare Warp - how to enable autoload? I didn't find anything on this topic in the FAQ, settings and documentation on the site.

Isaac McFadyenOP•9/20/24, 2:18 AM

Both of these questions probably should go in general-discussions , they're not related to Workers AI.

raj.js•9/20/24, 6:53 PM

is there any issue with vision models on cloudflare ai when trying to use from openai sdk? getting the below error

Error processing data: BadRequestError: 400 status code (no body)
at APIError.generate (file:/.........modules/openai/src/error.ts:70:14)

KKathy could i get your account ID and script name?

digendee•9/24/24, 6:55 AM

Sure, I'll DM you

�

😈 Donkey 💫•9/26/24, 6:51 AM

https://developers.cloudflare.com/workers-ai/models/llama-3.2-11b-vision-instruct
Hi,
docs 404 not found, please check

�😈 Donkey 💫https://developers.cloudflare.com/workers-ai/models/llama-3.2-11b-vision-instruc...

cookie•9/26/24, 6:59 AM

they haven't announced it yet, so very soon probably

akazwz•9/26/24, 7:46 AM

soon, today blog will announced it ?

Moixllik•9/26/24, 10:28 AM

Are there models for Text-to-Speech and Text-to-Video? Will it be coming in the future? plus1green

Keebs•9/26/24, 3:11 PM

anyone here trained their own and has some pointers to get a quicker start to use it with workers? anything is appropriated ;)

Fernando Dilland•9/26/24, 7:53 PM

I am starting to consume the lightweight llama model using the Cloudflare API, but where can I see my consumption? I do not want to exceed 10,000 neurons per day, and if I exceed that, what happens? (I'm not using worker, I'm consuming the cloudflare API directly)

FFernando Dilland I am starting to consume the lightweight llama model using the Cloudflare API, b...

Fernando Dilland•9/26/24, 7:55 PM

My question is, do you request them for example: curl https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/ai/run/@cf/meta/llama-3.2-11b-v...

Where or how can I see my consumption?

Stanleasy•9/27/24, 3:04 AM

the flux model documentation on API schema for input parameters all a sudden got a lot shorter in the couple hours i was afk. there were a bunch of parameters like image size and negative prompts that i don't see anymore
https://developers.cloudflare.com/workers-ai/models/flux-1-schnell

Cloudflare Docs

flux-1-schnell | Cloudflare Workers AI docs

FLUX.1 [schnell] is a 12 billion parameter rectified flow transformer capable of generating images from text descriptions.

Chaika•9/27/24, 3:24 AM

hmm yea https://github.com/cloudflare/cloudflare-docs/pull/17153/files 5h ago

GitHub

update Flux schema by LoganGrasby · Pull Request #17153 · cloudflar...

Summary

Screenshots (optional)

Documentation checklist

The documentation style guide has been adhered to.
If a larger change - such as adding a new page- an issue has been opened in relation ...

twentytwohalo•9/27/24, 6:11 AM

Hi I need an urgent support from you guys.
I have setup cloudflare on my domain (https://www.twentytwohalo.com/) and on Cloudflare dashboard it ask me Nameserver update pending.
I have updated the nameservers on my domain registrar. And now the cloudflare dashboard is showing an active status.

But when I update the nameserver of my hosting company and added A record and CNAME records in cloudflare DNS records (my site goes down and never up again even after waiting several hours).

Below are my domain and hosting companies:
Domain Registrar: Namecheap
Hosting: Dreamhost

Any help would be appreciated.
Thanks

22° H Δ L Oadmin

Home – EVENTS

Ttwentytwohalo Hi I need an urgent support from you guys. I have setup cloudflare on my domain...

dexter•9/27/24, 12:24 PM

I recommend you use another channel for nameservers/domains related question

SStanleasy the flux model documentation on API schema for input parameters all a sudden got...

dexter•9/27/24, 12:24 PM

Flux also returns a 500 error for me and other users in this discord, any idea why?

Going to ping the AI gateway PM here because this issue has come up before (cc <@1136363561472491551

Similar Threads

Similar Threads

Similar Threads