ozzie Posts - Answer Overflow

ozzie

Explore posts from servers

•Created by ozzie on 4/28/2025 in #⛅｜pods-clusters

trl vllm-serve not binding to port

I have a pod with two A6000 and I am trying to run vLLM on one of them via:

VLLM_LOGGING_LEVEL=DEBUG NCCL_DEBUG=TRACE trl vllm-serve --model meta-llama/Meta-Llama-3-8B-Instruct --gpu_memory_utilization=0.75 --max_model_len 2048 --host 0.0.0.0 --port 8000

VLLM_LOGGING_LEVEL=DEBUG NCCL_DEBUG=TRACE trl vllm-serve --model meta-llama/Meta-Llama-3-8B-Instruct --gpu_memory_utilization=0.75 --max_model_len 2048 --host 0.0.0.0 --port 8000

AFAICT, the model launches fine however it seems like there is a problem with binding to the port. I see nothing when doing lsof -i :8000 Is there any obvious additional configuration I need todo?

2 replies

RRunPod

•Created by ozzie on 3/18/2025 in #⚡｜serverless

Delete Serverless Endpoint via the API?

I am trying to delete the serverless endpoint via the API, but everytime I make a request to the endpoint, I get an internal error: Via the Python API:

delete_endpoint_graphql = """mutation {{
  deleteEndpoint(id: "{ENDPOINT_ID}") {{
    id
  }}
}}
"""


async def delete_endpoint(endpoint_id: str):
    try:
        runpod.api.graphql.run_graphql_query(
            delete_endpoint_graphql.format(ENDPOINT_ID=endpoint_id)
        )
    except runpod.error.QueryError as e:
        print(e)
        print_warning(
            f"Failed to delete endpoint {endpoint_id}: {e}Please delete it manually."
        )

delete_endpoint_graphql = """mutation {{
  deleteEndpoint(id: "{ENDPOINT_ID}") {{
    id
  }}
}}
"""


async def delete_endpoint(endpoint_id: str):
    try:
        runpod.api.graphql.run_graphql_query(
            delete_endpoint_graphql.format(ENDPOINT_ID=endpoint_id)
        )
    except runpod.error.QueryError as e:
        print(e)
        print_warning(
            f"Failed to delete endpoint {endpoint_id}: {e}Please delete it manually."
        )

Response:

Something went wrong. Please try again later or contact support.

Something went wrong. Please try again later or contact support.

I confirmed that my min and max is set to zero as per the GraphQL docs:

min and max workers must both be set to zero for your call to work.

min and max workers must both be set to zero for your call to work.

11 replies

DDeno

•Created by ozzie on 1/13/2025 in #help

`npm workspace`'s support with Deno v2

I am trying to follow the tutorial of using Deno Workspaces w/ npm workspaces https://docs.deno.com/runtime/fundamentals/workspaces/#migrating-from-npm-workspaces If I copy all the code and then run deno check main.ts, I get the following error:

error: Failed resolving types. [ERR_TYPES_NOT_FOUND] Could not find types for 'file://log/index.js' imported from 'file://hi/mod.ts'

error: Failed resolving types. [ERR_TYPES_NOT_FOUND] Could not find types for 'file://log/index.js' imported from 'file://hi/mod.ts'

If I try todo deno install , it also cannot find my local npm dependency. Is this a known limitation?

1 replies

Gaming

Programming