11 replies

Llama Token Counter - a Hugging Face Spa...

Hi, using llama2 from a cloudflare worker using the

ai.run

ai.run

binding, and finding that the responses I get back get cut off after < 300 tokens. Is there a way to set the token limit for a response to something higher than whatever it's set to?

A silly example, to illustrate, where I ask for a recipe for potatoes au gratin with bubble gum syrup, gets cut off midway through the instructions...

const response = await ai.run('@cf/meta/llama-2-7b-chat-int8', {
        prompt: `Create a recipe for potatoes au gratin that includes the special ingredient Bubble Gum Syrup\n\n`,
      }
return new Response(JSON.stringify(response));

const response = await ai.run('@cf/meta/llama-2-7b-chat-int8', {
        prompt: `Create a recipe for potatoes au gratin that includes the special ingredient Bubble Gum Syrup\n\n`,
      }
return new Response(JSON.stringify(response));

{"response":"Potatoes au gratin is a classic French dish that consists of thinly sliced potatoes layered in a baking dish and topped with cheese, cream, and breadcrumbs. To give this dish a unique twist, you can add a special ingredient like Bubble Gum Syrup. This sweet and tangy syrup is made with real bubble gum and adds a fun and playful touch to the dish. Here's a recipe for potatoes au gratin with Bubble Gum Syrup:\nIngredients:\n* 3-4 large potatoes, peeled and thinly sliced\n* 1/4 cup Bubble Gum Syrup\n* 1/4 cup grated cheese (such as cheddar or Parmesan)\n* 1/4 cup heavy cream\n* 1/4 cup breadcrumbs\n* Salt and pepper to taste\nInstructions:\n1. Preheat the oven to 375°F (190°C).\n2. In a large baking dish, arrange a layer of over"}

{"response":"Potatoes au gratin is a classic French dish that consists of thinly sliced potatoes layered in a baking dish and topped with cheese, cream, and breadcrumbs. To give this dish a unique twist, you can add a special ingredient like Bubble Gum Syrup. This sweet and tangy syrup is made with real bubble gum and adds a fun and playful touch to the dish. Here's a recipe for potatoes au gratin with Bubble Gum Syrup:\nIngredients:\n* 3-4 large potatoes, peeled and thinly sliced\n* 1/4 cup Bubble Gum Syrup\n* 1/4 cup grated cheese (such as cheddar or Parmesan)\n* 1/4 cup heavy cream\n* 1/4 cup breadcrumbs\n* Salt and pepper to taste\nInstructions:\n1. Preheat the oven to 375°F (190°C).\n2. In a large baking dish, arrange a layer of over"}

If I take that response into a llama token counter:
https://huggingface.co/spaces/Xanthius/llama-token-counter
It's only 259 tokens but cut off, and my understanding is that llama2 is supposed to have to context window of 4096 tokens, so there should be no reason it couldn't have finished the instructions.

When using the OpenAI API's we can pass an argument

max_tokens

max_tokens

to the chat completions API. I don't see an equivalent in workers ai. Is this something that you might add?

Llama Token Counter - a Hugging Face Space by Xanthius

Llama Token Counter - a Hugging Face Spa...

Hi, using llama2 from a cloudflare worker using the

ai.run

ai.run

const response = await ai.run('@cf/meta/llama-2-7b-chat-int8', {
        prompt: `Create a recipe for potatoes au gratin that includes the special ingredient Bubble Gum Syrup\n\n`,
      }
return new Response(JSON.stringify(response));

const response = await ai.run('@cf/meta/llama-2-7b-chat-int8', {
        prompt: `Create a recipe for potatoes au gratin that includes the special ingredient Bubble Gum Syrup\n\n`,
      }
return new Response(JSON.stringify(response));

{"response":"Potatoes au gratin is a classic French dish that consists of thinly sliced potatoes layered in a baking dish and topped with cheese, cream, and breadcrumbs. To give this dish a unique twist, you can add a special ingredient like Bubble Gum Syrup. This sweet and tangy syrup is made with real bubble gum and adds a fun and playful touch to the dish. Here's a recipe for potatoes au gratin with Bubble Gum Syrup:\nIngredients:\n* 3-4 large potatoes, peeled and thinly sliced\n* 1/4 cup Bubble Gum Syrup\n* 1/4 cup grated cheese (such as cheddar or Parmesan)\n* 1/4 cup heavy cream\n* 1/4 cup breadcrumbs\n* Salt and pepper to taste\nInstructions:\n1. Preheat the oven to 375°F (190°C).\n2. In a large baking dish, arrange a layer of over"}

{"response":"Potatoes au gratin is a classic French dish that consists of thinly sliced potatoes layered in a baking dish and topped with cheese, cream, and breadcrumbs. To give this dish a unique twist, you can add a special ingredient like Bubble Gum Syrup. This sweet and tangy syrup is made with real bubble gum and adds a fun and playful touch to the dish. Here's a recipe for potatoes au gratin with Bubble Gum Syrup:\nIngredients:\n* 3-4 large potatoes, peeled and thinly sliced\n* 1/4 cup Bubble Gum Syrup\n* 1/4 cup grated cheese (such as cheddar or Parmesan)\n* 1/4 cup heavy cream\n* 1/4 cup breadcrumbs\n* Salt and pepper to taste\nInstructions:\n1. Preheat the oven to 375°F (190°C).\n2. In a large baking dish, arrange a layer of over"}

max_tokens

max_tokens

to the chat completions API. I don't see an equivalent in workers ai. Is this something that you might add?

Llama Token Counter - a Hugging Face Space by Xanthius

Llama Token Counter - a Hugging Face Spa...

Llama Token Counter - a Hugging Face Spa...

Similar Threads

Similar Threads

Similar Threads