Voice Agent Workshop

A voice agent you can phone. Call a number, an AI answers, you have a conversation, you can talk over it.

About 200 lines of TypeScript. Fastify + Twilio Media Streams + OpenAI Realtime.

The point of this repo is that you can read all of it in one sitting.

---

Setup

npm install
cp .env.example .env

Fill in .env. You need an OpenAI API key, a Twilio account, and a Twilio phone number.

Then expose your local server to the internet — Twilio has to be able to reach it:

ngrok http 3000

Paste the https:// forwarding URL into SERVER_URL in .env, then:

npm run dev

Point your Twilio number at it (for inbound calls)

Twilio Console → Phone Numbers → your number → Voice Configuration → A call comes in:

Webhook   POST   https://<your-ngrok>.ngrok-free.app/twilio/voice

Now call your Twilio number from a phone. The agent picks up.

Or have it call you (outbound)

curl -X POST http://localhost:3000/calls/run-call \
  -H 'content-type: application/json' \
  -d '{"to":"+972500000000"}'

Your phone rings. Same agent, same code path.

Or open requests.http and click Send Request — it has the call, the health check, and a simulated Twilio webhook you can fire without spending a call. Needs the VS Code REST Client extension, or use it as-is in JetBrains.

---

How it works

The same diagram as Mermaid, if you'd rather edit it inline

sequenceDiagram
    actor You
    participant P as Caller phone
    participant T as Twilio
    participant S as Your server
    participant O as OpenAI Realtime

    You->>S: POST /calls/run-call
    S->>T: calls.create
    T->>P: rings the phone
    P-->>T: answers
    T->>S: POST /twilio/voice
    S-->>T: TwiML with wss:// url
    T->>S: opens WS /twilio/media-stream
    S->>O: opens WS
    O-->>S: session.created
    S->>O: session.update (audio/pcmu, server_vad)
    S->>O: response.create, so it greets first

    loop rest of the call
        P->>T: caller speaks
        T->>S: media event, base64 payload
        S->>O: input_audio_buffer.append
        O-->>S: response.output_audio.delta
        S-->>T: media event, base64 payload
        T-->>P: agent speaks
    end

Loading

Inbound calls join at POST /twilio/voice and follow the identical path from there — the only difference is that Twilio reaches it via the number's webhook rather than via calls.create.

Four files do the work:

File	Job
src/server.ts	Fastify bootstrap
src/env.ts	Env vars, and derives wss:// from SERVER_URL
src/realtime.ts	The OpenAI connection and session config
src/routes/twilio.ts	The webhook and the audio bridge — read this one
src/routes/calls.ts	POST /calls/run-call

1. There is no audio processing in this project

This is the thing worth taking away.

Twilio Media Streams speak G.711 µ-law, 8 kHz, mono, base64-encoded. OpenAI Realtime accepts and emits exactly that format if you ask it to:

audio: {
  input:  { format: { type: "audio/pcmu" }, ... },
  output: { format: { type: "audio/pcmu" }, ... },
}

So the bridge is a pipe. Twilio's payload goes into OpenAI untouched:

openai.send(JSON.stringify({
  type: "input_audio_buffer.append",
  audio: msg.media.payload,      // verbatim
}));

and OpenAI's audio goes back to Twilio untouched:

twilio.send(JSON.stringify({
  event: "media",
  streamSid,
  media: { payload: event.delta },  // verbatim
}));

No resampling, no µ-law decoding, no PCM16 buffers, no ffmpeg. Plenty of tutorials still decode µ-law → PCM16 → re-encode; that was a workaround for an early limitation and it is no longer needed. It's the difference between this repo and a 400-line one.

2. <Connect>, not <Start>

<Response>
  <Connect>
    <Stream url="wss://.../twilio/media-stream" />
  </Connect>
</Response>

<Connect><Stream> is bidirectional. <Start><Stream> is send-only — you'd hear the caller perfectly and they would never hear the agent. This single tag is the most common way to end up with a silent demo.

3. Barge-in: there are two buffers, and OpenAI only knows about one

Interrupting the agent mid-sentence is what makes a voice agent feel real. It's also where this stack is quietly tricky.

server_vad gets you most of the way. OpenAI detects the caller starting to speak and cancels its own in-flight response. No manual VAD, no response.cancel plumbing on your side.

But Twilio has its own playback buffer, and OpenAI has no idea it exists. By the time the caller interrupts, you have already forwarded several seconds of audio to Twilio, and Twilio's docs are explicit that media is "buffered and played in the order received." OpenAI stopping generation does not un-send bytes that already left your server. The caller keeps hearing the agent talk over them.

The fix is one message:

case "input_audio_buffer.speech_started":
  twilio.send(JSON.stringify({ event: "clear", streamSid }));
  break;

clear empties Twilio's buffer. Three lines, hanging off an event server_vad already emits for you.

Try it: comment those lines out and call again. Talk over the agent. It keeps going for seconds. That gap between "the model stopped" and "the caller stopped hearing it" is the whole lesson.

---

Gotchas

Each of these has cost someone an afternoon.

• Free ngrok URLs change on every restart. When it changes, update it in two places: SERVER_URL in .env and the webhook on your Twilio number.
• Twilio trial accounts can only call verified numbers. Verify the destination in the Console first, or outbound calls fail with error 21219.
• Twilio posts webhooks as application/x-www-form-urlencoded. Fastify has no parser for that out of the box and returns 415 before your handler runs. Hence @fastify/formbody in server.ts.
• Event names changed when the Realtime API went GA. It's response.output_audio.delta now, not response.audio.delta. Most tutorials online still use the old name — nothing errors, the agent is just silent forever. Same story for session config: GA wants nested audio.input.format and rejects the old flat input_audio_format.
• Loud static instead of a voice? It's the audio format name. GA takes audio/pcm, audio/pcmu (µ-law), audio/pcma (A-law). The beta's g711_ulaw is rejected, and it fails nastily: the entire session.update is refused, so the session silently keeps its default 24 kHz PCM16, and Twilio renders that as noise. The tell is an openai realtime error line in your server log with Invalid value: 'g711_ulaw'. Any invalid field in session.update sinks the whole message this way — always read that error log.
• Not every OpenAI voice works here. Realtime models accept alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin, cedar — and OpenAI recommends marin or cedar, which is what this repo uses. The older TTS voices (fable, onyx, nova) are not supported by realtime models; picking one doesn't cleanly error, you just get unusable audio.
• The model name moves. Pinned to gpt-realtime-2.1 in realtime.ts. A 404 on the WebSocket handshake means that constant needs bumping.

---

Deliberately left out

This is a demo, not a product. In rough priority order, here's what you'd add next:

1. Twilio request signature validation. Right now anyone who finds your ngrok URL can trigger the agent. twilio.validateRequest with your auth token.
2. conversation.item.truncate on interruption. OpenAI currently believes it played its whole response, when the caller only heard part. Doesn't affect anything audible — it makes the transcript honest.
3. Tool calling. The natural next workshop step: let the agent look something up mid-call.
4. Call state / transcripts. Nothing is persisted; the conversation lives and dies with the socket.
5. Reconnect handling. If the OpenAI socket drops mid-call, the call just ends.

---

Reference

• Twilio — Media Streams WebSocket messages
• Twilio — <Connect><Stream> TwiML
• OpenAI — Realtime API
• OpenAI — Realtime over WebSocket