Request Format Runpod VLLM Worker

{
  "conversation": {
    "id": "some_conversation_id",
    "messages": [
      {
        "source": "USER",
        "content": "Previous messages in the conversation..."
      }
    ]
  },
  "message": {
    "content": "Tell me why RunPod is the best GPU provider",
    "source": "USER"
  }
}

{
  "conversation": {
    "id": "some_conversation_id",
    "messages": [
      {
        "source": "USER",
        "content": "Previous messages in the conversation..."
      }
    ]
  },
  "message": {
    "content": "Tell me why RunPod is the best GPU provider",
    "source": "USER"
  }
}

I have been using the above format with Runpod VLLM worker to utilize the chat history functionality.

I've been getting the error that input is missing in the JSON request so this works.

{
"input": {
"prompt": "Tell me why RunPod is the best GPU provider",
"sampling_params": {
"max_tokens": 100
},
"apply_chat_template": true,
"stream": true
}
}

Did the input change recently?

Runpod•3y ago•

10 replies

Concept

Request Format Runpod VLLM Worker

{
  "conversation": {
    "id": "some_conversation_id",
    "messages": [
      {
        "source": "USER",
        "content": "Previous messages in the conversation..."
      }
    ]
  },
  "message": {
    "content": "Tell me why RunPod is the best GPU provider",
    "source": "USER"
  }
}

{
  "conversation": {
    "id": "some_conversation_id",
    "messages": [
      {
        "source": "USER",
        "content": "Previous messages in the conversation..."
      }
    ]
  },
  "message": {
    "content": "Tell me why RunPod is the best GPU provider",
    "source": "USER"
  }
}

Request Format Runpod VLLM Worker

Similar Threads

Request Format Runpod VLLM Worker

Similar Threads

Similar Threads

Similar Threads