Voice Controlled Interface

Introduction

Hand this document to any coding agent (or developer) to give an app real-time voice control. It is domain-agnostic: plug in your data model + actions and the pattern works identically for a todo list, a note-taker, an expense tracker, a kanban board, a habit tracker, or anything else with a bounded action vocabulary.

VCI is built on OpenAI's Realtime API over WebRTC — a single bidirectional streaming connection carrying mic audio in and assistant audio out, with tool calls in between. No browser-native STT/TTS (robotic and unreliable). No separate chat/completions + audio/speech round-trips (slow and expensive).

Philosophy

When to Use VCI

Getting Started

The 5 steps to go from empty folder to working VCI app:

1. Define the data model and storage mutations.
2. Write tool schemas — one per action.
3. Fill in the instructions template with tone + matching rules.
4. Wire the WebRTC connection and tool handler.
5. Render the read-only UI and the three session buttons.

Architecture

Real-time voice over WebRTC to OpenAI's Realtime API. STT, intent parsing, and TTS happen in a single bidirectional streaming connection. Tool calls travel over a WebRTC data channel as JSON.

┌────────────┐   mic audio    ┌──────────────────────┐
│  Browser   │───────────────>│ OpenAI Realtime API  │
│            │<───────────────│ (gpt-realtime,WebRTC)│
│            │  assistant     └──────────┬───────────┘
│  ┌──────┐  │  audio                    │
│  │ UI   │  │                           │ function calls
│  │(read │  │                           │ (JSON on data
│  │ only)│  │                           │  channel)
│  └──────┘  │                           ▼
│  ┌──────┐  │                  executed locally
│  │Store │◄─┼──────────────────against storage
│  └──────┘  │
└────────────┘

Required Modules

Every VCI app has four modules. Names and language are flexible — contracts are not.

storage Required

• readState() → current domain state (typically an array of items).
• One mutation function per state-changing action (add, remove, update, etc.). Each returns the mutated entity or null.
• Backend: localStorage (default), IndexedDB, or a remote API.

ui Required

• renderState(state) — repaint UI from state. No interactive controls for domain actions.
• appendLog(userText, assistantText, kind) — visible conversation log.
• setStatus(status, label) — see Interaction States.
• setMicListening(bool), setMicEnabled(bool), showKeyPanel(bool).

realtime Required

• connect({ apiKey, onTool, getContext, onEvent }) — resolves when connected; throws on failure.
• disconnect() — closes peer connection, data channel, mic tracks.
• isConnected() → boolean.
• refreshContext() — resend session.update with fresh state.

app Required

• Wire the three session buttons (Start Session, Push to Talk, End Session) — see § Three-Button Model.
• Run the 3-minute inactivity timer — see § Inactivity Timeout.
• Implement the tool-call handler — dispatch to storage mutations, re-render, return fresh state.
• Manage the API-key entry flow.

File Layout

<app>/
  index.html
  css/styles.css
  js/
    app.js       orchestration + tool handler
    realtime.js  WebRTC + Realtime API + tool schemas
    storage.js   domain persistence + mutations
    ui.js        DOM rendering (read-only)

Keep each file ≤ ~300 lines. If realtime.js grows beyond that, domain logic is leaking in — move it to app.js.

Data Model

One canonical shape per entity. Give each a stable id (UUID). Example (notes domain):

{
  id:        string,   // crypto.randomUUID()
  title:     string,
  body:      string,
  createdAt: number
}

The model uses id to reference existing entities across turns. Do not expose position-only ids (like array indices) — they shift when items are added/removed.

Storage Module

One function per meaningful action. Keep them boring — no clever aggregation, no side effects beyond persistence.

addNote(title, body)         → note
updateNote(id, fields)       → note | null
deleteNote(id)               → note | null
readState()                  → note[]

Tool Schemas

One tool per action, using the Realtime API's flat function schema. Note: this is not the nested Chat Completions shape.

[
  {
    type: "function",
    name: "add_note",
    description: "Create a new note.",
    parameters: {
      type: "object",
      properties: {
        title: { type: "string" },
        body:  { type: "string" }
      },
      required: ["title"]
    }
  },
  {
    type: "function",
    name: "delete_note",
    description: "Delete a note by id from the current list.",
    parameters: {
      type: "object",
      properties: { note_id: { type: "string" } },
      required: ["note_id"]
    }
  }
  // one per domain action; ideally ≤ 8 total
]

Instructions Template

The session.instructions string tells the model its role, tone, matching rules, and current state. Fill in <DOMAIN>, <verbs>, and inject the state snapshot.

You are a warm, natural voice assistant for a <DOMAIN> app.
Speak briefly and conversationally — never sound robotic.

When the user asks to <verbs>, call the matching tool.
After a tool call, briefly confirm what happened in one short sentence
(e.g. "Added buy milk." or "Deleted the dentist one.").
For small talk or unclear commands, respond briefly without a tool.

Rules for resolving references to existing items:
- Use the state below to match phrases like "the X one" or "the second Y".
- "The first/second/nth" refers to position in the list, 1-indexed.
- Match by loose semantic similarity, but only when unambiguous.
- If you cannot confidently match, say so — never guess.
- After every tool call, the response includes the fresh state.

Current state:
<JSON dump of storage.readState()>

Tool Handler

async function handleToolCall(name, args) {
  switch (name) {
    case "add_note": {
      const note = Storage.addNote(args.title, args.body);
      UI.renderState(Storage.readState());
      return { ok: true, action: "added", note,
               current_state: Storage.readState() };
    }
    // ... one branch per tool
  }
  return { ok: false, reason: "unknown_tool" };
}

Endpoints

Models & Voices

Model: gpt-realtime (GA).

Voices: alloy, ash, ballad, cedar, coral, echo, marin, sage, shimmer, verse.

Recommended defaults: marin (warm, natural) or coral (friendly).

Token request

POST /v1/realtime/client_secrets
Authorization: Bearer <apiKey>
Content-Type: application/json

{
  "session": {
    "type":  "realtime",
    "model": "gpt-realtime",
    "audio": { "output": { "voice": "marin" } }
  }
}

Response (top-level value is the ephemeral token, prefixed ek_):

{ "value": "ek_...", "expires_at": 1234567890, "session": { ... } }

session.update

Send this the moment the data channel opens, and again after every state mutation. VCI standardizes on manual turn detection — the client demarcates every user turn via input_audio_buffer.clear on PTT press and input_audio_buffer.commit on release.

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "instructions": "<see Instructions Template>",
    "tools": [ /* see Tool Schemas */ ],
    "tool_choice": "auto",
    "audio": {
      "input": {
        "transcription":  { "model": "whisper-1" },
        "turn_detection": null
      },
      "output": { "voice": "marin" }
    }
  }
}

"turn_detection": {
  "type": "server_vad",
  "threshold": 0.5,
  "prefix_padding_ms": 300,
  "silence_duration_ms": 500
}

Function Calls

When the model calls a tool, the data channel emits:

{
  "type": "response.function_call_arguments.done",
  "call_id": "call_...",
  "name": "add_note",
  "arguments": "{\"title\":\"...\"}"
}

Reply with the tool result:

{
  "type": "conversation.item.create",
  "item": {
    "type":    "function_call_output",
    "call_id": "<same call_id>",
    "output":  "<JSON.stringify(result)>"
  }
}

Then trigger the spoken confirmation:

{ "type": "response.create" }

WebRTC Recipe

Copy-paste starter for the browser side:

// 1. Mint ephemeral token
const tokenRes = await fetch(
  "https://api.openai.com/v1/realtime/client_secrets",
  {
    method: "POST",
    headers: {
      "Authorization": "Bearer " + apiKey,
      "Content-Type":  "application/json"
    },
    body: JSON.stringify({
      session: {
        type: "realtime",
        model: MODEL,
        audio: { output: { voice: VOICE } }
      }
    })
  }
);
const { value: ephemeral } = await tokenRes.json();

// 2. Peer connection
const pc = new RTCPeerConnection();

// 3. Remote audio playback
const audioEl = document.getElementById("assistant-audio");
pc.ontrack = e => { audioEl.srcObject = e.streams[0]; };

// 4. Mic input
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
stream.getTracks().forEach(t => pc.addTrack(t, stream));

// 5. Data channel for JSON events
const dc = pc.createDataChannel("oai-events");
dc.onopen    = () => sendSessionUpdate(dc);
dc.onmessage = e  => handleEvent(JSON.parse(e.data));

// 6. SDP offer/answer
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);

const answerSDP = await fetch(
  `https://api.openai.com/v1/realtime/calls?model=${encodeURIComponent(MODEL)}`,
  {
    method: "POST",
    body: offer.sdp,
    headers: {
      "Authorization": "Bearer " + ephemeral,
      "Content-Type":  "application/sdp"
    }
  }
).then(r => r.text());

await pc.setRemoteDescription({ type: "answer", sdp: answerSDP });

Event Reference

What is MCP? Optional

Model Context Protocol (MCP) is an open standard for connecting AI models to external tools and data sources. An MCP server exposes a set of tools the model can call — think of it as a plugin system for the AI.

A VCI app's native tool surface is intentionally small (≤ 8 domain tools). MCP lets you extend that surface without polluting the core: plug in Notion for note sync, Google Calendar for events, a web search server, GitHub, Linear, and so on.

Enabling MCP

OpenAI's Realtime API supports remote MCP servers natively as a tool type. You add them to the same tools array in the session.update event, alongside your function tools:

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "tools": [
      // Your native function tools (unchanged)
      { "type": "function", "name": "add_note", ... },
      { "type": "function", "name": "delete_note", ... },

      // MCP servers added inline
      {
        "type": "mcp",
        "server_label": "notion",
        "server_url":   "https://mcp.notion.com/sse",
        "authorization": "Bearer <notion-oauth-token>",
        "require_approval": "always"
      },
      {
        "type": "mcp",
        "server_label": "search",
        "server_url":   "https://mcp.search.example.com/sse",
        "require_approval": "never",
        "allowed_tools": ["web_search"]
      }
    ]
  }
}

The model can now call MCP tools as naturally as native ones. Voice command: "add this to my Notion inbox" → the model calls the Notion MCP server → confirmation spoken back.

Remote MCP Servers

Remote MCP servers speak MCP over HTTP + SSE (or streamable HTTP). They are the only flavor usable directly from a browser VCI app — the model connects to them from OpenAI's side, not yours.

Finding servers

• Public directories: mcpservers.org, modelcontextprotocol/servers.
• First-party: many SaaS vendors now ship their own MCP endpoint (Notion, Linear, GitHub, Slack, Sanity, Vercel, etc.).
• Roll your own: MCP is spec-based; a minimal server is ~150 lines of code.

Local (stdio) MCP servers

Local MCP servers (spawned as subprocesses over stdio) cannot be used directly from a browser. To integrate them, you need a small backend proxy that connects to the local server and exposes an SSE endpoint — at which point they are functionally remote MCP servers again.

Wiring MCP Tools

Recommended flow for optional MCP support in your VCI app:

1. In storage, add a getMcpServers() function returning an array of configured MCP server entries (URL, label, token, approval mode).
2. In the UI, add a small settings panel (still not for domain actions — this is a config surface) where the user can enable/disable MCP servers and paste tokens.
3. In realtime.js, when building the tools array for session.update, concatenate native function tools with configured MCP entries.
4. Re-send session.update whenever MCP config changes so the live session picks up the new tools.

function buildTools() {
  const native = NATIVE_FUNCTION_TOOLS;
  const mcp = Storage.getMcpServers().map(s => ({
    type: "mcp",
    server_label: s.label,
    server_url:   s.url,
    authorization: s.token ? "Bearer " + s.token : undefined,
    require_approval: s.approvalMode || "always",
    allowed_tools: s.allowedTools // optional whitelist
  }));
  return [...native, ...mcp];
}

MCP Security & Approvals

Approval modes

Token handling

• Personal-use apps: store MCP auth tokens in localStorage, same trade-off as the OpenAI key.
• Public apps: mint short-lived MCP tokens on your backend, hand them to the browser alongside the ephemeral OpenAI token.
• Whitelist tools per server via allowed_tools whenever possible — reduces blast radius if the server misbehaves.

When to Add MCP

Three-Button Model Standard

VCI apps use three buttons — Start Session, Push to Talk, and End Session. Implement this verbatim unless you have a specific reason to deviate.

Session Phases

The app tracks three phases; button visibility swaps based on the current phase.

Push-to-Talk

Press (pointerdown or Space keydown)

• Call Realtime.startTurn() — sends input_audio_buffer.clear and sets micTrack.enabled = true.
• Status → listening. Apply the .is-recording visual state.

Release (pointerup/pointercancel/pointerleave or Space keyup)

• Call Realtime.endTurn() — sets micTrack.enabled = false, then sends input_audio_buffer.commit and response.create.
• Status → thinking. Disable PTT briefly (re-enable on response.done).

Barge-in

If the user presses PTT while the model is still speaking, call Realtime.interruptResponse() (sends response.cancel) before startTurn().

Skeleton

btn.addEventListener("pointerdown", e => {
  e.preventDefault();
  btn.setPointerCapture(e.pointerId);
  if (responseInFlight) Realtime.interruptResponse();
  Realtime.startTurn();
  UI.setPttRecording(true);
  UI.setStatus("listening", "Listening");
});

const release = e => {
  Realtime.endTurn();
  UI.setPttRecording(false);
  UI.setPttEnabled(false); // re-enabled on response.done
  UI.setStatus("thinking", "Thinking");
  resetIdleTimer();
};
btn.addEventListener("pointerup",     release);
btn.addEventListener("pointercancel", release);

// Space bar accessibility fallback
document.addEventListener("keydown", e => {
  if (e.code !== "Space" || e.repeat) return;
  if (isTextInput(e.target)) return;
  e.preventDefault();
  startPtt();
});
document.addEventListener("keyup", e => {
  if (e.code !== "Space") return;
  e.preventDefault();
  stopPtt();
});

Inactivity Timeout Standard

Sessions auto-close after 3 minutes with no PTT activity.

• Start a setTimeout when the phase enters connected.
• Reset the timeout every time PTT is released (i.e. after a completed user turn).
• On timeout: call endSession(), log "Session auto-closed after 3 minutes of inactivity."
• Clear the timeout when the phase leaves connected.

const IDLE_TIMEOUT_MS = 3 * 60 * 1000;

function resetIdleTimer() {
  clearIdleTimer();
  if (phase !== "connected") return;
  idleTimer = setTimeout(() => {
    UI.appendLog("(system)",
      "Session auto-closed after 3 minutes of inactivity.", "info");
    endSession();
  }, IDLE_TIMEOUT_MS);
}

Status Pill

The status pill reflects the moment-to-moment state of the assistant during a session.

UI Requirements

• State panel — renders storage.readState(). Visually distinguish entity states. No click handlers for domain actions.
• Conversation log — most recent at bottom, auto-scroll. Populated from transcription + assistant events.
• Three session buttons — Start Session, Push to Talk, End Session (see § Three-Button Model). Visibility swaps based on phase; the PTT button pulses while recording.
• PTT hint text — one line under the buttons: "Hold Push to Talk (or Space) to speak. Session auto-closes after 3 minutes of silence."
• Status pill — reflects Status Pill states.
• API-key panel — shown on first load; hidden once a key is stored. Include a "reset key" link.
• Hidden audio element — <audio id="assistant-audio" autoplay playsinline>. The peer connection's remote track attaches here.

Accessibility

• Give each button an aria-label matching its visible text.
• Space bar acts as PTT (skip when a text input has focus).
• Use aria-live="polite" on the status pill and log so screen readers announce changes.
• The recording state must not rely on color alone — the label text ("Recording…" vs "Hold to Talk") and animation both change.
• Ensure the read-only state panel is still fully readable by screen readers.

Personal Use

• Store the OpenAI API key in localStorage. Show it in a masked input on first load. Provide a reset link.
• The browser mints ephemeral tokens directly; acceptable trade-off for single-user tools.
• Document this trade-off in the README so future users know.

Public Deployment

• Put a minimal backend in front (Cloudflare Worker, Vercel Function, Express endpoint).
• The backend holds the API key and calls POST /v1/realtime/client_secrets.
• Return only the value field to the browser. The browser never sees the real key.
• Bind an OpenAI-Safety-Identifier header on the server-side token request to attribute usage per end-user.
• For MCP: mint MCP tokens on the backend, too. Never ship long-lived OAuth tokens to the browser.

Implementation Checklist

Hand this list to the coder alongside a description of the target domain.

• [ ] Define the domain data model (single shape per entity, stable id).
• [ ] Build storage.js: readState() + one mutation per action.
• [ ] Build tool schemas — flat schema, ≤ 8 tools.
• [ ] Write session instructions with tone + matching rules + state placeholder.
• [ ] Build realtime.js. Use gpt-realtime, a natural voice, and manual turn detection.
• [ ] Expose startTurn(), endTurn(), interruptResponse() from realtime.js for PTT wiring.
• [ ] Implement the three-button model (Start Session, Push to Talk, End Session) per § Three-Button Model.
• [ ] Implement the 3-minute inactivity auto-close per § Inactivity Timeout.
• [ ] Build app.js tool handler — always return current_state.
• [ ] Call refreshContext() after every mutation.
• [ ] Build ui.js with all UI Requirements elements.
• [ ] Add API-key entry (sk- validation, localStorage). Add reset link.
• [ ] Include the hidden assistant audio element.
• [ ] Decide whether to include MCP support. If yes, follow Wiring MCP Tools.
• [ ] Handle 401 / 403 / 429 / network errors gracefully.
• [ ] Handle mic permission denial + unsupported-browser paths.
• [ ] Verify state persists across reload.
• [ ] Document browser support + security stance in README.

Manual Test Script

1. Golden path — unambiguous command → correct tool call → state mutated → assistant confirms briefly.
2. Ambiguous reference — assistant asks for clarification, no mutation.
3. Unrelated chit-chat — assistant chats briefly, no tool call.
4. Empty state read-back — assistant says something meaningful, not silence.
5. Reload — UI shows last confirmed state.
6. Invalid API key — session start fails cleanly, error shown in log.
7. Mic denied — session start fails cleanly, error shown in log.
8. Interrupt — user talks over assistant → server VAD interrupts, new turn begins.
9. MCP tool (if enabled) — voice command triggers MCP call → approval surfaced → executed → confirmed.

Environment Requirements

• Chrome or Edge on desktop (WebRTC + mic autoplay policy). Firefox and Safari are unofficial.
• Secure context: http://localhost, file://, or HTTPS.
• OpenAI API key with Realtime API access.
• Live internet connection during sessions.

Known Limitations

• Cost. Realtime API bills per minute of input and output audio. Short bursts are cheap; long continuous sessions add up.
• No offline mode. Every session requires OpenAI.
• Occasional hallucination of references. The current_state-in-every-tool-response pattern mitigates but does not eliminate this. When it matters (irreversible actions), have the model confirm first.
• Non-English accents. For non-English users, set audio.input.transcription.language if quality drops.
• API drift. Endpoints have already changed once — verify against official docs before implementing.

Extension Patterns

• Multi-entity domains — one tool schema per entity type, prefix tool names with the entity (add_task, add_note). Keep total ≤ 8.
• Confirmations for destructive actions — add a confirm: boolean argument, or split into soft_delete + restore.
• Search/filter as a tool — expose search(query); the model can then reference results by id in follow-up turns.
• Rich responses — for actions like "read my week", let the model speak longer; server VAD still allows interruption.
• Client-side wake word — for "always listening" UX, add a small wake-word detector (e.g. Porcupine) before starting the Realtime session. Do not stream mic to OpenAI 24/7.
• Multi-modal — Realtime supports images via input_image items. Useful for a receipt-capture voice app or a "describe this photo" note-taker.