OpenAI Serverless Endpoint Docs

Y

yhlong00000•10/12/24, 2:47 PM

https://github.com/runpod-workers/worker-vllm

GitHub

GitHub - runpod-workers/worker-vllm: The RunPod worker template for...

The RunPod worker template for serving our large language model endpoints. Powered by vLLM. - runpod-workers/worker-vllm

Y

yhlong00000•10/12/24, 2:47 PM

This read me has some examples

Yyhlong00000 https://github.com/runpod-workers/worker-vllm

3

3WaDOP•10/12/24, 3:08 PM

Yes. I've read through that source code before asking. Based on that, it should be how I wrote it. But someone also mentioned it might be allowed just for that official worker, so I wanted to make sure the /openai path also works for custom images before I write the whole code around it.

Does it also mean we can send any raw "non-wrapped" payload to that endpoint even when it's not openai related? It should pass any content to the "openai_input" and any route after to the "openai_route", right? Having a setting or another documented endpoint on serverless allowing us to send raw payloads would solve such problems with predefined APIs.

33WaD Yes. I've read through that source code before asking. Based on that, it should ...

O

OBJay•10/21/24, 8:02 AM

Did you figure this out?

O

OBJay•10/21/24, 8:05 AM

it's still not clear whether custom, model baked in images can use the openai route or not

O

OBJay•10/21/24, 8:06 AM

when I send a request to it, it seems to still treat it as a 'normal' request, but maybe I need to pass in openai_route myself?

OOBJay Did you figure this out?

3

3WaDOP•10/21/24, 10:53 AM

I've only tested that calling /openai/ path does indeed produce different responses even on my existing non-openai endpoints.

Response with /openai/... path:

{"error": "Error processing the request"}

{"error": "Error processing the request"}

{"error": "Error processing the request"}

{"error": "Error processing the request"}

Response with any other path (e.g. /testpath/):

404 page not found

404 page not found

404 page not found

404 page not found

Based on this behaviour, I started writing my worker code. I hope I'll be able to test it soon and that it will work.

J

Jason•10/24/24, 3:51 AM

So in custom images you haven't been able to use the /openai route?

O

OBJay•10/24/24, 5:29 PM

it received mine but it still treats it like a normal request

O

OBJay•10/24/24, 5:30 PM

it treats it like an openai request if i did a normal request with use_openai_format, openai_route and openai_input

O

OBJay•10/24/24, 5:31 PM

but then that kinda defeats the point, i think the template vllm worker automatically adds those to the request or something

O

OBJay•10/24/24, 5:35 PM

like they do here but then with openai_route : "chat/completion" and openai_input : messages

O

OBJay•10/24/24, 5:35 PM

https://docs.runpod.io/sdks/python/endpoints

Endpoints | RunPod Documentation

Learn how to use the RunPod Python SDK to interact with various endpoints, perform synchronous and asynchronous operations, stream data, and check endpoint health. Discover how to set your Endpoint Id, run jobs, and cancel or purge queues with this comprehensive guide.

OOBJay but then that kinda defeats the point, i think the template vllm worker automati...

J

Jason•10/24/24, 11:20 PM

Yep yep if it doesn't work well like the official endpoint does

Yyhlong00000 https://github.com/runpod-workers/worker-vllm

J

Jason•10/24/24, 11:22 PM

Do you have the same handler code logic as this, 3WaD and OBJay?

JJason Do you have the same handler code logic as this, 3WaD and OBJay?

3

3WaDOP•10/24/24, 11:29 PM

Uhmm, no. I am asking because I want to build a custom image, not a fork of the vLLM image. Do you know which part of that source code is making the path work exactly?

J

Jason•10/24/24, 11:30 PM

No i don't, but somehow you must have the handling logic to make it work like that

J

Jason•10/24/24, 11:30 PM

Like the input handling at least

J

Jason•10/24/24, 11:31 PM

Well have you tried to fork it first instead , then try deploy it

J

Jason•10/24/24, 11:31 PM

If it doesn't work on your own image, same code then report it to runpod, if not, match it first

3

3WaDOP•10/25/24, 12:08 AM

Just to make sure - is this the correct place to reach someone from RunPod who knows about their endpoints and might have a short definite answer to this question? I appreciate the input and your time so far guys, but to summarize it so far, I've got a link to the official repo using the thing I am asking about, and got told to "hack around and find out"

J

Jason•10/25/24, 12:35 AM

This is community server, if you want to reach out to official staffs use ticketing. Sometimes staffs also check here

33WaD Just to make sure - is this the correct place to reach someone from RunPod who k...

J

Jason•10/25/24, 12:36 AM

Not hack around and find out, you're building with the runpod sdk, and runpod endpoints, just use the starting base from the one that runpod team has built for receiving requests from openai endpoint

JJason This is community server, if you want to reach out to official staffs use ticket...

3

3WaDOP•10/25/24, 12:41 AM

Thank you

J

Jason•10/25/24, 12:41 AM

I assume there is not enough documentation from runpod about receiving inputs from openai endpoints so the best thing you can do is use that as a starting point

J

Jason•10/25/24, 12:41 AM

Yup your welcome

33WaD Hello. From what I could find in the support threads here, you should be able to...

D

deanQ•10/25/24, 4:52 PM

Can someone please confirm that this works in custom images too?

Yes, our serverless API transforms
input
input
to
openai_route
openai_route
openai_route
openai_route +
openai_input
openai_input
openai_input
openai_input as long as you send the request to
/openai/*
/openai/*
/openai/*
/openai/*

Is there any way I can develop this locally?

This happens on our platform only. As of now, there is nothing in the SDK to simulate this during local development.

DdeanQ Can someone please confirm that this works in custom images too? > Yes, our serv...

3

3WaDOP•10/25/24, 6:12 PM

Ahhh. So it's not

{"input": {"openai_input": {}, "openai_route": {}}}

{"input": {"openai_input": {}, "openai_route": {}}}

{"input": {"openai_input": {}, "openai_route": {}}}

{"input": {"openai_input": {}, "openai_route": {}}}

but just

{"openai_input": {}, "openai_route": {}}

{"openai_input": {}, "openai_route": {}}

{"openai_input": {}, "openai_route": {}}

{"openai_input": {}, "openai_route": {}}

Thank you very much for the confirmation!

D

deanQ•10/25/24, 9:26 PM

Just

{"input":{"model": "...", "prompt": "..."}}

{"input":{"model": "...", "prompt": "..."}}

{"input":{"model": "...", "prompt": "..."}}

{"input":{"model": "...", "prompt": "..."}} to pass to

/openai/*

/openai/*

/openai/*

/openai/* and that essentially gets passed to vllm as

{"openai_input": {"model": "...", "prompt": "..."}, "openai_route": {}}

{"openai_input": {"model": "...", "prompt": "..."}, "openai_route": {}}

{"openai_input": {"model": "...", "prompt": "..."}, "openai_route": {}}

{"openai_input": {"model": "...", "prompt": "..."}, "openai_route": {}}

3

3WaDOP•10/25/24, 9:58 PM

Passed to vllm? What if there's no vllm? I'll put it simply - when I send

{"foo":"bar"}

{"foo":"bar"}

{"foo":"bar"}

{"foo":"bar"} to https://api.runpod.ai/v2/<ENDPOINT ID>/openai/abc, will ANY handler function receive the payload (input and path) so we can work with it further or it's not possible on RunPod?

33WaD Passed to vllm? What if there's no vllm? I'll put it simply - when I send `{"foo...

D

deanQ•10/26/24, 8:17 PM

"input" is key here

DdeanQ Just `{"input":{"model": "...", "prompt": "..."}}` to pass to `/openai/*` and th...

O

OBJay•10/26/24, 9:10 PM

but then we cant use the openai client.chat.completions.create right? since it doesn't format it with the {"input":{....

D

deanQ•10/26/24, 10:39 PM

I’m not sure if I’m missing something here. “input” is what you provide us. We take anything inside that and put it inside “openai_input”.

DdeanQ I’m not sure if I’m missing something here. “input” is what you provide us. We t...

3

3WaDOP•10/26/24, 11:54 PM

Yes. We're asking if it's possible to send the json without wrapping it with "input" or not. Because that's how openAI standard requires it.

33WaD Yes. We're asking if it's possible to send the json without wrapping it with "in...

D

deanQ•10/27/24, 12:02 AM

I see. Yes, the endpoint can be treated like an openai server. Your endpoint should have a value for

OPENAI BASE URL

OPENAI BASE URL

OPENAI BASE URL

OPENAI BASE URL You just have to make sure you send it to that path like so...

import openai
import asyncio

async def run():
  runpod_endpoint_id = "vllm-1234567890"
  runpod_api_key = "xxxxxxxxxxxx"
  runpod_base_url = f"https://api.runpod.ai/v2/{runpod_endpoint_id}/openai/v1"

  openai_client = openai.AsyncOpenAI(
      base_url=runpod_base_url,
      api_key=runpod_api_key,
  )

  completion = openai_client.completions.create(
    model="NousResearch/Meta-Llama-3-8B-Instruct",
    messages=[
      {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
    ],
    extra_body={
      "guided_choice": ["positive", "negative"]
    },
  )
  
  return await asyncio.create_task(completion)

import openai
import asyncio

async def run():
  runpod_endpoint_id = "vllm-1234567890"
  runpod_api_key = "xxxxxxxxxxxx"
  runpod_base_url = f"https://api.runpod.ai/v2/{runpod_endpoint_id}/openai/v1"

  openai_client = openai.AsyncOpenAI(
      base_url=runpod_base_url,
      api_key=runpod_api_key,
  )

  completion = openai_client.completions.create(
    model="NousResearch/Meta-Llama-3-8B-Instruct",
    messages=[
      {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
    ],
    extra_body={
      "guided_choice": ["positive", "negative"]
    },
  )
  
  return await asyncio.create_task(completion)

import openai
import asyncio

async def run():
  runpod_endpoint_id = "vllm-1234567890"
  runpod_api_key = "xxxxxxxxxxxx"
  runpod_base_url = f"https://api.runpod.ai/v2/{runpod_endpoint_id}/openai/v1"

  openai_client = openai.AsyncOpenAI(
      base_url=runpod_base_url,
      api_key=runpod_api_key,
  )

  completion = openai_client.completions.create(
    model="NousResearch/Meta-Llama-3-8B-Instruct",
    messages=[
      {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
    ],
    extra_body={
      "guided_choice": ["positive", "negative"]
    },
  )
  
  return await asyncio.create_task(completion)

import openai
import asyncio

async def run():
  runpod_endpoint_id = "vllm-1234567890"
  runpod_api_key = "xxxxxxxxxxxx"
  runpod_base_url = f"https://api.runpod.ai/v2/{runpod_endpoint_id}/openai/v1"

  openai_client = openai.AsyncOpenAI(
      base_url=runpod_base_url,
      api_key=runpod_api_key,
  )

  completion = openai_client.completions.create(
    model="NousResearch/Meta-Llama-3-8B-Instruct",
    messages=[
      {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
    ],
    extra_body={
      "guided_choice": ["positive", "negative"]
    },
  )
  
  return await asyncio.create_task(completion)

For more information, please go through https://github.com/runpod-workers/worker-vllm/

GitHub

GitHub - runpod-workers/worker-vllm: The RunPod worker template for...

The RunPod worker template for serving our large language model endpoints. Powered by vLLM. - runpod-workers/worker-vllm

DdeanQ I see. Yes, the endpoint can be treated like an openai server. Your endpoint sho...

O

OBJay•10/27/24, 2:31 PM

you're referring us to the default vllm template usage again, we were asking how to do this when you make a custom docker image with the model built in.

O

OBJay•10/27/24, 2:33 PM

also, when I do that it seems to default it to a sync request (the id will be like sync-xxxxx) even though I use AsyncOpenAI

O

OBJay•10/27/24, 2:36 PM

I don't understand how there's no documentation about this at all lol

D

deanQ•10/27/24, 4:56 PM

We refer you to that page because that’s where the answers are. Sending the requests to that specific path to your endpoint is the answer to your question. Have you tried it? Are you having issues with vllm blocking sync? It doesn’t do that. Our requests to vllm will always be async. You’ll notice that when you make requests to a vllm endpoint on our serverless. There is nothing we’re doing that changes that. We’re essentially proxying the requests. When you send sync requests, they come back immediately. It’s non-blocking due to vllm’s multi-concurrent processing. If there are issues with this, please file a support ticket so we can help you. CS will be asking for additional information that shouldn’t be divulged here on a public forum.

O

OBJay•10/27/24, 5:17 PM

Yes of course we have tried it, we're asking this question because it doesn't work the way it should

O

OBJay•10/27/24, 5:20 PM

So even though it shows the request as "sync-93450349539'', it still processes it async?

O

OBJay•10/27/24, 5:30 PM

When I use a custom image it does NOT show the link like this here:

OOBJay When I use a **custom image** it does **NOT** show the link like this here:

D

deanQ•10/27/24, 9:44 PM

Despite that, are you able to send requests to the /openai/v1 path?

OOBJay So even though it shows the request as "sync-93450349539'', it still processes i...

D

deanQ•10/27/24, 10:03 PM

Yes. When you’re testing these with longer-running jobs, do they block like they are sync requests?

O

OBJay•10/28/24, 7:37 PM

didn't test with longer running jobs, but good to know that it doesn't matter even if it shows as sync

3

3WaDOP•11/3/24, 2:50 AM

I am finally back home, so I can test this myself. And for anyone coming here in the future and wondering what's the answer to this simple question:

Yes, you can send non-nested payloads to api.runpod.ai/v2/<ENDPOINT ID>/openai/* path that is using any custom async handler or software internally, and it will be available in the handler params.

That means when you send

{"foo":"bar"}

{"foo":"bar"}

{"foo":"bar"}

{"foo":"bar"} to

.../openai/abc

.../openai/abc

.../openai/abc

.../openai/abc:

async def handler(job):
# job["input"] will include {'openai_input': {'foo': 'bar'}, 'openai_route': '/abc'}
#...
runpod.serverless.start({"handler": handler,"return_aggregate_stream": True})

async def handler(job):
# job["input"] will include {'openai_input': {'foo': 'bar'}, 'openai_route': '/abc'}
#...
runpod.serverless.start({"handler": handler,"return_aggregate_stream": True})

async def handler(job):
# job["input"] will include {'openai_input': {'foo': 'bar'}, 'openai_route': '/abc'}
#...
runpod.serverless.start({"handler": handler,"return_aggregate_stream": True})

async def handler(job):
# job["input"] will include {'openai_input': {'foo': 'bar'}, 'openai_route': '/abc'}
#...
runpod.serverless.start({"handler": handler,"return_aggregate_stream": True})

I really wish this would be mentioned somewhere in the official docs, in the ask-ai knowledge base, or at least widely known to the team when asked. But thank you anyways.

OpenAI Serverless Endpoint Docs

Similar Threads

Similar Threads

Similar Threads