Batch API Access for `@cf/meta/llama-3.3-70b-instruct-batch`
Hello,
I'm currently testing the Workers AI Batch API functionality with LLM models, and I've encountered some issues that I’d like clarification on.
Attempt 1:
I tried to batch multiple requests to this model using the following code:
However, I received this error:
This seems to indicate that batching is not supported for this model, even though it's one of the newest.
Attempt 2:
After reviewing your blog post, I attempted to use this alternative model for batching. However, I received the following error:
This makes me wonder:
1. Is
2. When will batching be supported for
3. Is there a recommended LLM model that supports batching for general instruction-following tasks?
I'm currently testing the Workers AI Batch API functionality with LLM models, and I've encountered some issues that I’d like clarification on.
Attempt 1: @cf/meta/llama-3.3-70b-instruct-fp8-fast
@cf/meta/llama-3.3-70b-instruct-fp8-fastI tried to batch multiple requests to this model using the following code:
However, I received this error:
This seems to indicate that batching is not supported for this model, even though it's one of the newest.
Attempt 2: @cf/meta/llama-3.3-70b-instruct-batch
@cf/meta/llama-3.3-70b-instruct-batchAfter reviewing your blog post, I attempted to use this alternative model for batching. However, I received the following error:
This makes me wonder:
1. Is
@cf/meta/llama-3.3-70b-instruct-batch2. When will batching be supported for
@cf/meta/llama-3.3-70b-instruct-fp8-fast3. Is there a recommended LLM model that supports batching for general instruction-following tasks?
The Cloudflare Blog
We just made Workers AI inference faster with speculative decoding & prefix caching. Use our new batch inference for handling large request volumes seamlessly. Build tailored AI apps with more LoRA options. Lastly, new models and a refreshed dashboard round out this Developer Week update for Workers AI.
Welcome to the official Cloudflare Developers server. Here you can ask for help and stay updated with the latest news
85,042Members
Resources
Was this page helpful?
