Runpod•2y ago•

28 replies

Running serverless endpoint locally

Hi,
I'm trying to run locally, but can't find a solution in documentation. I tried running handler with docker and outside docker.
running with

--rp_serve_api

--rp_serve_api

--rp_serve_api

--rp_serve_api

ERROR:    [Errno 99] error while attempting to bind on address ('::1', 8000, 0, 0): cannot assign requested address

ERROR: [Errno 99] error while attempting to bind on address ('::1', 8000, 0, 0): cannot assign requested address

ERROR:    [Errno 99] error while attempting to bind on address ('::1', 8000, 0, 0): cannot assign requested address

ERROR: [Errno 99] error while attempting to bind on address ('::1', 8000, 0, 0): cannot assign requested address

running with

--rp_serve_api --rp_api_host='0.0.0.0'

--rp_serve_api --rp_api_host='0.0.0.0'

--rp_serve_api --rp_api_host='0.0.0.0'

--rp_serve_api --rp_api_host='0.0.0.0' exits with error:

ERROR:    [Errno -2] Name or service not known

ERROR: [Errno -2] Name or service not known

ERROR:    [Errno -2] Name or service not known

ERROR: [Errno -2] Name or service not known

Is there some other sources I can look to solve this?

Have a nice day!
Mateusz

Solution

python3 -u rp_handler.py --rp_serve_api --rp_api_port 8010 --rp_api_host 0.0.0.0

python3 -u rp_handler.py --rp_serve_api --rp_api_port 8010 --rp_api_host 0.0.0.0

python3 -u rp_handler.py --rp_serve_api --rp_api_port 8010 --rp_api_host 0.0.0.0

python3 -u rp_handler.py --rp_serve_api --rp_api_port 8010 --rp_api_host 0.0.0.0

Jump to solution

MMateusz Hi, I'm trying to run locally, but can't find a solution in documentation. I tri...

Jason•4/17/24, 8:17 AM

another app is using the address on ('::1', 8000)

digigoblin•4/17/24, 8:19 AM

Yeah you can either stop the other app or use a different port for running the handler

Solution

digigoblin•4/17/24, 8:19 AM

python3 -u rp_handler.py --rp_serve_api --rp_api_port 8010 --rp_api_host 0.0.0.0

python3 -u rp_handler.py --rp_serve_api --rp_api_port 8010 --rp_api_host 0.0.0.0

digigoblin•4/17/24, 8:19 AM

that will make it listen on port 8010 instead of 8000

MateuszOP•4/17/24, 8:33 AM

@digigoblin thanks! in docs and arts passing value is after ==, I didn't try without it, my brain didn't notice that it may be wrong. It's running when passing value without == sign

MateuszOP•4/17/24, 8:45 AM

@digigoblin is there a way to define types for input and output to be generated in openapi file?

MMateusz @digigoblin is there a way to define types for input and output to be generated ...

digigoblin•4/17/24, 8:52 AM

Not sure what you mean.

Ddigigoblin Not sure what you mean.

MateuszOP•4/17/24, 8:56 AM

openapi schema generation based on code.

by default openapi file is generated when running serverless handler. It generates schemas and for request DefaultRequestDefaultRequest by default. It defines input as object. When creating some custom endpoint we can make complex structures and it would be nice to put it into schema

MMateusz openapi schema generation based on code. by default openapi file is generated w...

Jason•4/17/24, 9:56 AM

you can check an example of schema for rp's handler in here
https://github.com/ashleykleynhans/runpod-worker-a1111/tree/main/schemas
https://github.com/ashleykleynhans/runpod-worker-a1111/blob/main/rp_handler.py#L87

GitHub

runpod-worker-a1111/schemas at main · ashleykleynhans/runpod-worker...

RunPod Serverless Worker for the Automatic1111 Stable Diffusion API - ashleykleynhans/runpod-worker-a1111

GitHub

runpod-worker-a1111/rp_handler.py at main · ashleykleynhans/runpod-...

RunPod Serverless Worker for the Automatic1111 Stable Diffusion API - ashleykleynhans/runpod-worker-a1111

Jason•4/17/24, 9:58 AM

it doesnt generate openapi file i guess but its for validation

digigoblin•4/17/24, 9:59 AM

RunPod allows OpenAPI compatible endpoints for the vllm worker, so you may want to look at using that.

digigoblin•4/17/24, 10:00 AM

https://github.com/runpod-workers/worker-vllm

GitHub

GitHub - runpod-workers/worker-vllm: The RunPod worker template for...

The RunPod worker template for serving our large language model endpoints. Powered by vLLM. - runpod-workers/worker-vllm

Ddigigoblin RunPod allows OpenAPI compatible endpoints for the vllm worker, so you may want ...

Jason•4/17/24, 10:00 AM

wdym openapi compatible endpoints?

JJason wdym openapi compatible endpoints?

digigoblin•4/17/24, 10:01 AM

You don't have to hack your worker to support them, its supported natively in RunPod serverless

digigoblin•4/17/24, 10:03 AM

https://github.com/runpod-workers/worker-vllm?tab=readme-ov-file#examples-using-your-runpod-endpoint-with-openai

GitHub

GitHub - runpod-workers/worker-vllm: The RunPod worker template for...

The RunPod worker template for serving our large language model endpoints. Powered by vLLM. - runpod-workers/worker-vllm

Jason•4/17/24, 10:03 AM

Oh you mean openai? I think that's not what he mean

MMateusz @digigoblin is there a way to define types for input and output to be generated ...

digigoblin•4/17/24, 10:08 AM

He asked about it here.

digigoblin•4/17/24, 10:09 AM

Sorry he said openapi not openai, my bad.

digigoblin•4/17/24, 10:10 AM

You don't need openapi stuff with serverless.

JJason it doesnt generate openapi file i guess but its for validation

MateuszOP•4/17/24, 12:04 PM

yeah, I was analysing all of those workers, that's why I'm asking questions here, because I can't find answers in other places

Thanks!

MMateusz yeah, I was analysing all of those workers, that's why I'm asking questions here...

Jason•4/17/24, 12:05 PM

yeah i think validation hasnt been documented yet..

Jason•4/17/24, 12:05 PM

Ddigigoblin You don't need openapi stuff with serverless.

MateuszOP•4/17/24, 12:07 PM

why? They are still endpoints and in a team it would be nice to have openapi to share contracts automatically. you then can do some automated tests to make sure contracts are still valid. especially with proper CI/CD pipelines across the project and multiple repositories

MMateusz why? They are still endpoints and in a team it would be nice to have openapi to ...

digigoblin•4/17/24, 12:33 PM

You can't do this with serverless, only with an API on a pod. Serverless has fixed endpoints like /run/run and /runsync/runsync, it doesn't use custom endpoints, and also its already an API so you don't implement an API in your handler.

Ddigigoblin You can't do this with serverless, only with an API on a pod. Serverless has fi...

MateuszOP•4/17/24, 2:07 PM

I was just wandering if some overwriting would be possible to extend base definition, because in most cases inputinput will have some structure defined. If don't then I understand. Thanks a lot for help!

MMateusz I was just wandering if some overwriting would be possible to extend base defini...

digigoblin•4/17/24, 2:13 PM

You have to put the entire contents of your request within the inputinput key of the payload, and then your response will be in the outputoutput key fo the payload. Its not possible to extend it.

Ddigigoblin You have to put the entire contents of your request within the `input` key of th...

MateuszOP•4/17/24, 2:39 PM

yes, and you can put something in this inputinput and it can be an object, so it may be complex structure which you can define. so you could narrow input type generated currently with your type. so in the end it could be somehow "overwritten" or "extend" (you name it, it's just semantics) to have in openapi file (which is currently generated) more precise definition what intputintput and outputoutput are, not some generic list of every type.

Running serverless endpoint locally

Similar Threads

Similar Threads

Similar Threads