HeyGen and OpenAI release open infrastructure for real-time voice agents with faces

The two companies collaborated on a reference integration that connects GPT-Live-1, a full-duplex speech-to-speech model, with HeyGen's LiveAvatar and adds an animated display layer called HyperFrames. The code is fully open source and intended as a starting point for developers building voice experiences with visual presence, without reinventing the plumbing.

Minimal architecture by design

The server (Node 20.12 or later, managed with pnpm) acts as orchestrator: it receives audio from the browser over WebSocket, streams it to GPT-Live via the Responses API, and gets back text, tool calls, and audio. The avatar's audio is routed directly to a LiveKit media server; the browser receives only a LiveKit view token and a WebSocket channel for transcripts and visual layers. An API key never leaves the client.

Demo: a Japanese teacher with flashcards and a live summary

The bundled demo is a Japanese teacher. She teaches vocabulary by voice, displays a flashcard — word, pronunciation, meaning — exactly when she says it, and every few words shrinks to a corner and runs a summary panel drawn from the server-side conversation log, not the model's memory. The entire personality is defined in two Markdown files under `server/prompts/`, and every visual element is a single tool and a single composition; swapping them turns the demo into anything else.

Self-validating install

`pnpm run setup` prompts for the two keys (LiveAvatar and OpenAI with GPT-Live access), validates them against the live APIs, and writes `.env` at the repository root. Re-running preserves existing values and only re-validates. Developers who prefer manual setup can copy `.env.example` and fill it in. Both `pnpm dev` and `pnpm start` run a pre-flight check that refuses to start until every required variable exists, listing exactly what is missing.

What the developer still has to do

The project does not pick an avatar or write a prompt. A default avatar ID is already in `.env.example`, and the personas live in `server/prompts/`. Editing `instructions.md` defines who the avatar is; editing `greeting.md` determines how it opens (an empty file means the user speaks first). The idea: the code is the wiring, not a framework that hides it. Fork, swap the personality, keep the face.