3 replies

Connect via WebSocket connection to OpenAI Realtime API - wss://api.openai.com/v1/realtime

I watched @Craig 's YT video connecting WebRTC to the OpenAI Realtime API. Exciting stuff!

However, I tried my own Worker solution and was not as successful.
Why? Cloudflare workers does not support NPM WS library. Why can't I just use the Worker-supported WebSocket class? Well, because the OpenAI realtime API requires HTTP header

Authorization : Bearer $OAI_API_KEY

Authorization : Bearer $OAI_API_KEY

On OAI Realtime API Docs page, they just added this code to establish vanilla javascript WebSocket connection. Thank the higher powers! I was stuck on this for a week!!

/*
Note that in client-side environments like web browsers, we recommend
using WebRTC instead. It is possible, however, to use the standard 
WebSocket interface in browser-like environments like Deno and 
Cloudflare Workers.
*/

const ws = new WebSocket(
  "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-12-17",
  [
    "realtime",
    // Auth
    "openai-insecure-api-key." + OPENAI_API_KEY, 
    // Optional
    "openai-organization." + OPENAI_ORG_ID,
    "openai-project." + OPENAI_PROJECT_ID,
    // Beta protocol, required
    "openai-beta.realtime-v1"
  ]
);

ws.on("open", function open() {
  console.log("Connected to server.");
});

ws.on("message", function incoming(message) {
  console.log(message.data);
});

/*
Note that in client-side environments like web browsers, we recommend
using WebRTC instead. It is possible, however, to use the standard 
WebSocket interface in browser-like environments like Deno and 
Cloudflare Workers.
*/

const ws = new WebSocket(
  "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-12-17",
  [
    "realtime",
    // Auth
    "openai-insecure-api-key." + OPENAI_API_KEY, 
    // Optional
    "openai-organization." + OPENAI_ORG_ID,
    "openai-project." + OPENAI_PROJECT_ID,
    // Beta protocol, required
    "openai-beta.realtime-v1"
  ]
);

ws.on("open", function open() {
  console.log("Connected to server.");
});

ws.on("message", function incoming(message) {
  console.log(message.data);
});

YouTubeCloudflare Developers

Client Side Tool Calling with the OpenAI WebRTC Realtime API

OpenAI recently released the WebRTC interface for their realtime API. It supports tool calling, which means you can use it to do client side JavaScript function calling. This opens up a whole new set of use cases.

In this video, Craig shows off how he used Cloudflare Worker to relay the WebRTC call securely. Then he uses function calling to co...

MDN Web Docs

WebSocket - Web APIs | MDN

The WebSocket object provides the API for creating and managing a WebSocket connection to a server, as well as for sending and receiving data on the connection.

Connect via WebSocket connection to OpenAI Realtime API - wss://api.openai.com/v1/realtime

Authorization : Bearer $OAI_API_KEY

Authorization : Bearer $OAI_API_KEY

On OAI Realtime API Docs page, they just added this code to establish vanilla javascript WebSocket connection. Thank the higher powers! I was stuck on this for a week!!

/*
Note that in client-side environments like web browsers, we recommend
using WebRTC instead. It is possible, however, to use the standard 
WebSocket interface in browser-like environments like Deno and 
Cloudflare Workers.
*/

const ws = new WebSocket(
  "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-12-17",
  [
    "realtime",
    // Auth
    "openai-insecure-api-key." + OPENAI_API_KEY, 
    // Optional
    "openai-organization." + OPENAI_ORG_ID,
    "openai-project." + OPENAI_PROJECT_ID,
    // Beta protocol, required
    "openai-beta.realtime-v1"
  ]
);

ws.on("open", function open() {
  console.log("Connected to server.");
});

ws.on("message", function incoming(message) {
  console.log(message.data);
});

/*
Note that in client-side environments like web browsers, we recommend
using WebRTC instead. It is possible, however, to use the standard 
WebSocket interface in browser-like environments like Deno and 
Cloudflare Workers.
*/

const ws = new WebSocket(
  "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-12-17",
  [
    "realtime",
    // Auth
    "openai-insecure-api-key." + OPENAI_API_KEY, 
    // Optional
    "openai-organization." + OPENAI_ORG_ID,
    "openai-project." + OPENAI_PROJECT_ID,
    // Beta protocol, required
    "openai-beta.realtime-v1"
  ]
);

ws.on("open", function open() {
  console.log("Connected to server.");
});

ws.on("message", function incoming(message) {
  console.log(message.data);
});

YouTubeCloudflare Developers

Client Side Tool Calling with the OpenAI WebRTC Realtime API

MDN Web Docs

WebSocket - Web APIs | MDN

The WebSocket object provides the API for creating and managing a WebSocket connection to a server, as well as for sending and receiving data on the connection.

Connect via WebSocket connection to OpenAI Realtime API - wss://api.openai.com/v1/realtime

Connect via WebSocket connection to OpenAI Realtime API - wss://api.openai.com/v1/realtime

Similar Threads

Similar Threads

Similar Threads