It's possible because of some network routing intricacies, you're being routed out of Australia and

It's possible because of some network routing intricacies, you're being routed out of Australia and so Gemini is blocking that.

IIsaac McFadyen It's possible because of some network routing intricacies, you're being routed o...

digendee•9/15/24, 1:23 AM

Thank you. Yes I crossposted because I wasn't sure which channel to post the question in.

When visiting that trace URL you posted, my "colo" is MEL and "loc" is AU - both of which are allowed to access Gemini.

Any other ideas?

digendee•9/15/24, 1:26 AM

Actually, in the trace link you posted above, I get "MEL" as the colo, but with my Pages Function / worker, I get "HKG" as the colo.

digendee•9/15/24, 1:31 AM

Indeed, Hong Kong (HKG) is NOT in the allowed access to Google Gemini list. This would be the reason I am getting that Gemini error. Thanks again @Isaac McFadyen

Ddigendee Indeed, Hong Kong (HKG) is NOT in the allowed access to Google Gemini list. This...

Isaac McFadyenOP•9/15/24, 1:45 AM

No problem... unfortunately because of the way Cloudflare works there's no way to "force" your Worker to use MEL.

digendee•9/15/24, 1:46 AM

I am submitting a feature request now to Cloudflare to avoid putting workers in these types of locations, as this problem will only get worse as LLM API adoption grows

Isaac McFadyenOP•9/15/24, 1:46 AM

Going to ping the AI gateway PM here because this issue has come up before (cc @Kathy).

TLDR: Gemini isn't allowed to be access from Hong Kong but AI gateway is routing requests through HKG, causing them to be denied. I believe this came up with OpenAI before? Not sure what the resolution was or if you forced the routing to avoid certain countries internally?

digendee•9/15/24, 1:54 AM

Feature request is here in case you want to contribute / vote:

digendee•9/15/24, 1:54 AM

https://community.cloudflare.com/t/do-not-put-workers-in-llm-forbidden-locations/711754

digendee•9/15/24, 1:58 AM

My solution for the time being is to move my LLM-access APIs from Cloudflare Workers to something like Google's Cloud Functions, however my preference is to use Cloudflare Workers if this issue / problem can be resolved in the near future.

digendee•9/15/24, 1:59 AM

@Isaac McFadyen do you know who the Workers PM is? The issue could also be resolved with changes to Worker locations.

Ddigendee <@575799975116800030> do you know who the Workers PM is? The issue could also be...

Isaac McFadyenOP•9/15/24, 2:06 AM

It's more likely that this would be resolved specific to AI Gateway, because this happened before with OpenAI and I believe it was resolved, and I don't believe pinning a Worker to a specific (user-defined) location is planned or going to be implemented any time soon.

ahlstrominfo•9/16/24, 1:47 PM

When I use @cf/baai/bge-base-en-v1.5 and compare output to locally using the same embeddings with pyton or ollama or anything. I get a different embedding for the same word. How can that be?

oof2win2•9/16/24, 4:15 PM

each model creates a different embedding based on its training and understanding of the meaning

Ooof2win2 each model creates a different embedding based on its training and understanding...

ahlstrominfo•9/16/24, 5:52 PM

Completely understandable. But if I use the same model on other device should it not return the same result?

oof2win2•9/16/24, 5:53 PM

not necessarily. they will be similar (even across various models stuff will be similar) but you most likely wont always get the exact same answer

oof2win2•9/16/24, 5:53 PM

ai is just not deterministic

oof2win2•9/16/24, 5:55 PM

like its designed to randomly pick the next token. you can reduce the temperature to make it be less creative (tho idk if embeddings will have temperature, as embeddings are input and not output)

IIsaac McFadyen Going to ping the AI gateway PM here because this issue has come up before (cc <...

digendee•9/17/24, 12:47 PM

@Kathy any updates?

a4amaan•9/17/24, 5:06 PM

Its generating near about 300 words is there any limit of neurons for free users?

const answer = await ctx.env.AI.run('@cf/meta/llama-3-8b-instruct', {
        prompt: "Write 2000 words article",
        stream: true
    });

const answer = await ctx.env.AI.run('@cf/meta/llama-3-8b-instruct', {
        prompt: "Write 2000 words article",
        stream: true
    });

Sunny•9/17/24, 5:09 PM

i think its 10,000 neurons per day for free plan

SSunny i think its 10,000 neurons per day for free plan

a4amaan•9/17/24, 5:12 PM

yes i've read that.. i have not used neurons and getting about 300 words of article? is there any limit for single call?

Sunny•9/17/24, 5:15 PM

don't think so, it doesn't seem that you are setting a max_token limit here so it might be something to do with the llama 3 model?

Sunny•9/17/24, 5:15 PM

found this on hugging face forums for the llama 3 model

Sunny•9/17/24, 5:15 PM

"The output takes the same space as the input. The model completes the 8k token space with the response. You could slide the contxt window and get more output, but then you're losing context at the beginning.

Example, if your input is 100 tokens, you have ~7900 tokens for completion. But if your input is 7900 tokens, you have ~100 tokens left for response until your input starts getting trimmed out. The model can only pay attention to 8k tokens at most."

Sunny•9/17/24, 5:15 PM

I use llama 3.1 which has 128k token context window, it might be worth the upgrade in your use case if you are going over limits

SSunny I use llama 3.1 which has 128k token context window, it might be worth the upgra...

a4amaan•9/17/24, 5:18 PM

that suites good, thanks

a4amaan•9/17/24, 6:01 PM

@Sunny still not getting more than 300 words with 3.1

a4amaan•9/17/24, 6:04 PM

at the end it look like something stopped it, incomplete article

Sunny•9/17/24, 6:09 PM

it seems like there's a limit being set, maybe try manually setting max_tokens/max_length to be a higher limit? You could just try arbitrarily setting it to 100k?

SSunny it seems like there's a limit being set, maybe try manually setting max_tokens/m...

a4amaan•9/17/24, 6:21 PM

Thanks it worked:

const answer = await ctx.env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
        messages,
        stream: true,
        max_tokens:10000
    });

const answer = await ctx.env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
        messages,
        stream: true,
        max_tokens:10000
    });

Avocadio•9/18/24, 11:48 AM

Hey. I uploaded my finetuned model to cloudflare successfully but typescript gave that error and it request gives 3028 unknown internal error

Avocadio•9/18/24, 11:49 AM

It gives that error when i add lora parameter

Avocadio•9/18/24, 11:49 AM

Doesnt work with tsignore neither

Avocadio•9/18/24, 11:49 AM

Can you guys help

Avocadio•9/18/24, 11:55 AM

https://community.cloudflare.com/t/unknown-internal-error-for-lora-adapters/692895/3
Some other people faced the same issue

Ddigendee <@1136363561472491551> any updates?

Kathy•9/18/24, 8:38 PM

could i get your account ID and script name?

KKathy could i get your account ID and script name?

ajgeiss0702•9/19/24, 1:27 AM

i am currently getting the same issue (the 3028 error, not typescript). My account ID is f55b85c8a963663b11036975203c63c0f55b85c8a963663b11036975203c63c0. I am using the http api to run it so i dont have a script name for this

ajgeiss0702•9/19/24, 1:28 AM

I am trying with the @cf/mistral/mistral-7b-instruct-v0.1@cf/mistral/mistral-7b-instruct-v0.1 model because i didnt know the v0.2 model existed on cloudflare until 2 minutes ago

ajgeiss0702•9/19/24, 1:29 AM

{"errors":[{"message":"AiError: AiError: Unknown internal error","code":3028}],"success":false,"result":{},"messages":[]}

{"errors":[{"message":"AiError: AiError: Unknown internal error","code":3028}],"success":false,"result":{},"messages":[]}

Aajgeiss0702 i am currently getting the same issue (the 3028 error, not typescript). My accou...

Kathy•9/19/24, 10:17 AM

wait i dont think this is the same issue at all. @digendee 's issue is because their worker is getting routed through HKG (which is blocked by Google Gemini) even though they're based in Melbourne (allowed by Google Gemini)

Mr.Spok•9/19/24, 5:44 PM

Hello everyone, please tell me about the application for Windows Cloudflare Warp - how to enable autoload? I didn't find anything on this topic in the FAQ, settings and documentation on the site.

Isaac McFadyenOP•9/20/24, 2:18 AM

Both of these questions probably should go in general-discussions , they're not related to Workers AI.

raj.js•9/20/24, 6:53 PM

is there any issue with vision models on cloudflare ai when trying to use from openai sdk? getting the below error

Error processing data: BadRequestError: 400 status code (no body)
at APIError.generate (file:/.........modules/openai/src/error.ts:70:14)

KKathy could i get your account ID and script name?

digendee•9/24/24, 6:55 AM

Sure, I'll DM you

�

😈 Donkey 💫•9/26/24, 6:51 AM

https://developers.cloudflare.com/workers-ai/models/llama-3.2-11b-vision-instruct
Hi,
docs 404 not found, please check

�😈 Donkey 💫https://developers.cloudflare.com/workers-ai/models/llama-3.2-11b-vision-instruc...

cookie•9/26/24, 6:59 AM

they haven't announced it yet, so very soon probably

akazwz•9/26/24, 7:46 AM

soon, today blog will announced it ?

Moixllik•9/26/24, 10:28 AM

Are there models for Text-to-Speech and Text-to-Video? Will it be coming in the future? plus1green

Keebs•9/26/24, 3:11 PM

anyone here trained their own and has some pointers to get a quicker start to use it with workers? anything is appropriated ;)

Fernando Dilland•9/26/24, 7:53 PM

I am starting to consume the lightweight llama model using the Cloudflare API, but where can I see my consumption? I do not want to exceed 10,000 neurons per day, and if I exceed that, what happens? (I'm not using worker, I'm consuming the cloudflare API directly)

It's possible because of some network routing intricacies, you're being routed out of Australia and

Similar Threads

Similar Threads

Similar Threads