<@1136363561472491551> any updates?

A

a4amaan•9/17/24, 5:06 PM

Its generating near about 300 words is there any limit of neurons for free users?

const answer = await ctx.env.AI.run('@cf/meta/llama-3-8b-instruct', {
        prompt: "Write 2000 words article",
        stream: true
    });

const answer = await ctx.env.AI.run('@cf/meta/llama-3-8b-instruct', {
        prompt: "Write 2000 words article",
        stream: true
    });

const answer = await ctx.env.AI.run('@cf/meta/llama-3-8b-instruct', {
        prompt: "Write 2000 words article",
        stream: true
    });

const answer = await ctx.env.AI.run('@cf/meta/llama-3-8b-instruct', {
        prompt: "Write 2000 words article",
        stream: true
    });

S

Sunny•9/17/24, 5:09 PM

i think its 10,000 neurons per day for free plan

SSunny i think its 10,000 neurons per day for free plan

A

a4amaan•9/17/24, 5:12 PM

yes i've read that.. i have not used neurons and getting about 300 words of article? is there any limit for single call?

S

Sunny•9/17/24, 5:15 PM

don't think so, it doesn't seem that you are setting a max_token limit here so it might be something to do with the llama 3 model?

S

Sunny•9/17/24, 5:15 PM

found this on hugging face forums for the llama 3 model

S

Sunny•9/17/24, 5:15 PM

"The output takes the same space as the input. The model completes the 8k token space with the response. You could slide the contxt window and get more output, but then you're losing context at the beginning.

Example, if your input is 100 tokens, you have ~7900 tokens for completion. But if your input is 7900 tokens, you have ~100 tokens left for response until your input starts getting trimmed out. The model can only pay attention to 8k tokens at most."

S

Sunny•9/17/24, 5:15 PM

I use llama 3.1 which has 128k token context window, it might be worth the upgrade in your use case if you are going over limits

SSunny I use llama 3.1 which has 128k token context window, it might be worth the upgra...

A

a4amaan•9/17/24, 5:18 PM

that suites good, thanks

A

a4amaan•9/17/24, 6:01 PM

@Sunny still not getting more than 300 words with 3.1

A

a4amaan•9/17/24, 6:04 PM

at the end it look like something stopped it, incomplete article

S

Sunny•9/17/24, 6:09 PM

it seems like there's a limit being set, maybe try manually setting max_tokens/max_length to be a higher limit? You could just try arbitrarily setting it to 100k?

SSunny it seems like there's a limit being set, maybe try manually setting max_tokens/m...

A

a4amaan•9/17/24, 6:21 PM

Thanks it worked:

const answer = await ctx.env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
        messages,
        stream: true,
        max_tokens:10000
    });

const answer = await ctx.env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
        messages,
        stream: true,
        max_tokens:10000
    });

const answer = await ctx.env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
        messages,
        stream: true,
        max_tokens:10000
    });

const answer = await ctx.env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
        messages,
        stream: true,
        max_tokens:10000
    });

A

Avocadio•9/18/24, 11:48 AM

Hey. I uploaded my finetuned model to cloudflare successfully but typescript gave that error and it request gives 3028 unknown internal error

A

Avocadio•9/18/24, 11:49 AM

It gives that error when i add lora parameter

A

Avocadio•9/18/24, 11:49 AM

Doesnt work with tsignore neither

A

Avocadio•9/18/24, 11:49 AM