Hello, I am trying to use Cloudflare OpenAi compatible API, but I am getting this error: "could not

Hello, I am trying to use Cloudflare OpenAi compatible API, but I am getting this error: "could not route to /accounts/xxxx/ai/v1". Note xxxx is my account ID

can anyone please help.
Thanks

Nnet2income Hello, I am trying to use Cloudflare OpenAi compatible API, but I am getting thi...

Isaac McFadyen•9/12/24, 9:02 PM

You need to add one of the OpenAI endpoints to the end of the URL.

Isaac McFadyen•9/12/24, 9:02 PM

For example, /v1/chat/completions/v1/chat/completions or /v1/completions/v1/completions

Artucuno•9/13/24, 12:57 AM

Hey! Is there a way to do real time transcription with Whisper?

AArtucuno Hey! Is there a way to do real time transcription with Whisper?

Isaac McFadyen•9/13/24, 1:50 AM

Not really, because Whisper isn't a streaming transcription model - it needs all of the content in advance. What you can do, and what most things that are "streaming" with Whisper do, is chunk the audio into smaller chunks and process each one with a slight overlap.

Isaac McFadyen•9/13/24, 1:50 AM

The accuracy won't be quite as good but it will probably be as close as you can get to 'streaming' with Whisper.

digendee•9/14/24, 10:26 AM

Hi all. We are having issues trying to access Google's Gemini API from a Cloudflare Pages Function / Worker.

In the worker code, we have tried accessing Google's Gemini API using both direct requests to:

https://generativelanguage.googleapis.com/...

and through the Cloudflare AI gateway:
https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_name}/google-ai-studio/...

Both methods above result in the following error being returned from the Google Gemini API:
{
"error": {
"code": 400,
"message": "User location is not supported for the API use.",
"status": "FAILED_PRECONDITION"
}
}

Upon investigation, this error is supposedly returned when the user is not in a supported location listed here:
https://ai.google.dev/gemini-api/docs/available-regions

However, we ARE in a supported location (Melbourne, Australia). When we access the Gemini API from our local dev env (even if we access it via the Cloudflare AI Gateway), the Gemini API works correctly.

Anyone know what is going on?

Google AI for Developers

Available regions for Google AI Studio and Gemini API | Google AI...

Ddigendee Hi all. We are having issues trying to access Google's Gemini API from a Cloudfl...

Isaac McFadyen•9/14/24, 1:52 PM

First, please don't crosspost - you posted in both cloudflare-ai and ai-gateway

Isaac McFadyen•9/14/24, 1:52 PM

Second, can you try going to https://cloudflare.com/cdn-cgi/trace and see what the colocolo field is?

Isaac McFadyen•9/14/24, 1:53 PM

It's possible because of some network routing intricacies, you're being routed out of Australia and so Gemini is blocking that.

IIsaac McFadyen It's possible because of some network routing intricacies, you're being routed o...

digendee•9/15/24, 1:23 AM

Thank you. Yes I crossposted because I wasn't sure which channel to post the question in.

When visiting that trace URL you posted, my "colo" is MEL and "loc" is AU - both of which are allowed to access Gemini.

Any other ideas?

digendee•9/15/24, 1:26 AM

Actually, in the trace link you posted above, I get "MEL" as the colo, but with my Pages Function / worker, I get "HKG" as the colo.

digendee•9/15/24, 1:31 AM

Indeed, Hong Kong (HKG) is NOT in the allowed access to Google Gemini list. This would be the reason I am getting that Gemini error. Thanks again @Isaac McFadyen

Ddigendee Indeed, Hong Kong (HKG) is NOT in the allowed access to Google Gemini list. This...

Isaac McFadyen•9/15/24, 1:45 AM

No problem... unfortunately because of the way Cloudflare works there's no way to "force" your Worker to use MEL.

digendee•9/15/24, 1:46 AM

I am submitting a feature request now to Cloudflare to avoid putting workers in these types of locations, as this problem will only get worse as LLM API adoption grows

Isaac McFadyen•9/15/24, 1:46 AM

Going to ping the AI gateway PM here because this issue has come up before (cc @Kathy).

TLDR: Gemini isn't allowed to be access from Hong Kong but AI gateway is routing requests through HKG, causing them to be denied. I believe this came up with OpenAI before? Not sure what the resolution was or if you forced the routing to avoid certain countries internally?

digendee•9/15/24, 1:54 AM

Feature request is here in case you want to contribute / vote:

digendee•9/15/24, 1:54 AM

https://community.cloudflare.com/t/do-not-put-workers-in-llm-forbidden-locations/711754

digendee•9/15/24, 1:58 AM

My solution for the time being is to move my LLM-access APIs from Cloudflare Workers to something like Google's Cloud Functions, however my preference is to use Cloudflare Workers if this issue / problem can be resolved in the near future.

digendee•9/15/24, 1:59 AM

@Isaac McFadyen do you know who the Workers PM is? The issue could also be resolved with changes to Worker locations.

Ddigendee <@575799975116800030> do you know who the Workers PM is? The issue could also be...

Isaac McFadyen•9/15/24, 2:06 AM

It's more likely that this would be resolved specific to AI Gateway, because this happened before with OpenAI and I believe it was resolved, and I don't believe pinning a Worker to a specific (user-defined) location is planned or going to be implemented any time soon.

ahlstrominfo•9/16/24, 1:47 PM

When I use @cf/baai/bge-base-en-v1.5 and compare output to locally using the same embeddings with pyton or ollama or anything. I get a different embedding for the same word. How can that be?

oof2win2•9/16/24, 4:15 PM

each model creates a different embedding based on its training and understanding of the meaning

Ooof2win2 each model creates a different embedding based on its training and understanding...

ahlstrominfo•9/16/24, 5:52 PM

Completely understandable. But if I use the same model on other device should it not return the same result?

oof2win2•9/16/24, 5:53 PM

not necessarily. they will be similar (even across various models stuff will be similar) but you most likely wont always get the exact same answer

oof2win2•9/16/24, 5:53 PM

ai is just not deterministic

oof2win2•9/16/24, 5:55 PM

like its designed to randomly pick the next token. you can reduce the temperature to make it be less creative (tho idk if embeddings will have temperature, as embeddings are input and not output)

IIsaac McFadyen Going to ping the AI gateway PM here because this issue has come up before (cc <...

digendee•9/17/24, 12:47 PM

@Kathy any updates?

a4amaan•9/17/24, 5:06 PM

Its generating near about 300 words is there any limit of neurons for free users?

const answer = await ctx.env.AI.run('@cf/meta/llama-3-8b-instruct', {
        prompt: "Write 2000 words article",
        stream: true
    });

const answer = await ctx.env.AI.run('@cf/meta/llama-3-8b-instruct', {
        prompt: "Write 2000 words article",
        stream: true
    });

Sunny•9/17/24, 5:09 PM

i think its 10,000 neurons per day for free plan

SSunny i think its 10,000 neurons per day for free plan

a4amaan•9/17/24, 5:12 PM

yes i've read that.. i have not used neurons and getting about 300 words of article? is there any limit for single call?

Sunny•9/17/24, 5:15 PM

don't think so, it doesn't seem that you are setting a max_token limit here so it might be something to do with the llama 3 model?

Sunny•9/17/24, 5:15 PM

found this on hugging face forums for the llama 3 model

Sunny•9/17/24, 5:15 PM

"The output takes the same space as the input. The model completes the 8k token space with the response. You could slide the contxt window and get more output, but then you're losing context at the beginning.

Example, if your input is 100 tokens, you have ~7900 tokens for completion. But if your input is 7900 tokens, you have ~100 tokens left for response until your input starts getting trimmed out. The model can only pay attention to 8k tokens at most."

Sunny•9/17/24, 5:15 PM

I use llama 3.1 which has 128k token context window, it might be worth the upgrade in your use case if you are going over limits

SSunny I use llama 3.1 which has 128k token context window, it might be worth the upgra...

a4amaan•9/17/24, 5:18 PM

that suites good, thanks

a4amaan•9/17/24, 6:01 PM

@Sunny still not getting more than 300 words with 3.1

a4amaan•9/17/24, 6:04 PM

at the end it look like something stopped it, incomplete article

Sunny•9/17/24, 6:09 PM

it seems like there's a limit being set, maybe try manually setting max_tokens/max_length to be a higher limit? You could just try arbitrarily setting it to 100k?

SSunny it seems like there's a limit being set, maybe try manually setting max_tokens/m...

a4amaan•9/17/24, 6:21 PM

Thanks it worked:

const answer = await ctx.env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
        messages,
        stream: true,
        max_tokens:10000
    });

const answer = await ctx.env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
        messages,
        stream: true,
        max_tokens:10000
    });

Avocadio•9/18/24, 11:48 AM

Hey. I uploaded my finetuned model to cloudflare successfully but typescript gave that error and it request gives 3028 unknown internal error

Avocadio•9/18/24, 11:49 AM

It gives that error when i add lora parameter

Avocadio•9/18/24, 11:49 AM

Doesnt work with tsignore neither

Avocadio•9/18/24, 11:49 AM

Can you guys help

Avocadio•9/18/24, 11:55 AM

https://community.cloudflare.com/t/unknown-internal-error-for-lora-adapters/692895/3
Some other people faced the same issue

Ddigendee <@1136363561472491551> any updates?

Kathy•9/18/24, 8:38 PM

could i get your account ID and script name?

KKathy could i get your account ID and script name?

ajgeiss0702•9/19/24, 1:27 AM

i am currently getting the same issue (the 3028 error, not typescript). My account ID is f55b85c8a963663b11036975203c63c0f55b85c8a963663b11036975203c63c0. I am using the http api to run it so i dont have a script name for this

ajgeiss0702•9/19/24, 1:28 AM

I am trying with the @cf/mistral/mistral-7b-instruct-v0.1@cf/mistral/mistral-7b-instruct-v0.1 model because i didnt know the v0.2 model existed on cloudflare until 2 minutes ago

ajgeiss0702•9/19/24, 1:29 AM

{"errors":[{"message":"AiError: AiError: Unknown internal error","code":3028}],"success":false,"result":{},"messages":[]}

{"errors":[{"message":"AiError: AiError: Unknown internal error","code":3028}],"success":false,"result":{},"messages":[]}

Aajgeiss0702 i am currently getting the same issue (the 3028 error, not typescript). My accou...

Kathy•9/19/24, 10:17 AM

wait i dont think this is the same issue at all. @digendee 's issue is because their worker is getting routed through HKG (which is blocked by Google Gemini) even though they're based in Melbourne (allowed by Google Gemini)

Mr.Spok•9/19/24, 5:44 PM

Hello everyone, please tell me about the application for Windows Cloudflare Warp - how to enable autoload? I didn't find anything on this topic in the FAQ, settings and documentation on the site.

Hello, I am trying to use Cloudflare OpenAi compatible API, but I am getting this error: "could not

Similar Threads

Similar Threads

Similar Threads