Realtime Agents

glove-realtime runs a built Glove agent on a realtime multimodal model: OpenAI Realtime, Gemini Live or GPT-Live. The agent is authored as usual. Its tools, permission gates, store, context providers, inbox, goals and subscribers all work on the live session. Depending on the provider, the session takes audio, typed text, images and camera frames, and answers in speech or text. GPT-Live accepts audio and typed text and produces audio.

pnpm add glove-realtime

import { Displaymanager, Glove, MemoryStore } from "glove-core";
import { RealtimeSession, realtimeModel } from "glove-realtime";

const agent = new Glove({
  model: realtimeModel({ provider: "gemini", voice: "Puck" }),   // or "openai", "openai-live"
  store: new MemoryStore("call-42"),
  displayManager: new Displaymanager(),
  systemPrompt: "You are Nova, a support agent. Speak briefly.",
  compaction_config: { compaction_instructions: "Summarize the call." },
})
  .fold(lookupOrder)
  .fold(refund)                       // requiresPermission: true, enforced on the call
  .build();

const session = new RealtimeSession({
  agent,
  compaction: { contextLimit: 24_000 },
  permissions: async ({ tool, input }) => confirmWithSupervisor(tool, input),
});

session.on("audio", (pcm, format) => speaker.play(pcm, format.sampleRate));
session.on("interrupted", () => speaker.flush());
await session.start();

mic.on("pcm", (pcm) => session.sendAudio(pcm));      // in session.adapter.inputFormat
camera.on("frame", (jpeg) => session.sendVideoFrame(jpeg));

Who owns what

A realtime model keeps part of the agent on its own side of the socket. It holds the conversation, decides what a tool schema may contain, and manages its own context window. The providers disagree on every one of those: OpenAI can delete conversation items, Gemini can only append and must reconnect to change its tools, and GPT-Live has no history at all. So the adapter owns the provider's half and the session owns Glove's half.

Adapters declare capabilities and report through one event stream (RealtimeEvent), so nothing on the session side switches on the provider.

Providers

createRealtimeAdapter(args) resolves explicit arguments first, then REALTIME_PROVIDER, REALTIME_MODEL, REALTIME_VOICE and the provider's API key. It fails when the adapter is built if the key is missing.

Glove primitives on a live session

• Tools run through glove-core's Executor: validation, summaries, subagent events, tool_use_result. Every outcome reaches the provider, refusals and errors included. renderData never does. Retries default to 0, because a retry on a call is silence. If a tool folds new tools or changes the system prompt, the provider hears about it before the result goes back.
• requiresPermission is enforced, async predicates included. The store's decision comes first. Otherwise the permissions policy decides: "display" (the display stack's permission_request), "deny", "allow", or a function. The default is "deny" for a serverMode agent.
• The store gets each turn, each tool call together with its result, and compaction summaries. A fresh provider session is seeded from it, so text turns and calls share one history.
• Runtime context follows each provider's delivery. "iteration" context is read on open, after tools and when the user starts speaking. "request" context is read on open and when the user starts speaking, and is frozen in between. On OpenAI each snapshot replaces the last one. "history" context is saved to the store when it changes and told to the call once. Resolved inbox items are announced once and consumed.
• Events (agent.emitEvent, Foundry delegation updates) join the call. The session holds the agent's turn queue, so they never wake the text model. An event goes in silently at the next user turn or tool result. If nobody is talking when it arrives, the session requests a spoken response at a quiet pause; the provider decides what to say. Each event is saved to the store once. While the call runs, processRequest on the same agent waits. events: false leaves events to the agent's policy.
• Subscribers on the agent (agent.addSubscriber) get the same events from the call that a text turn emits, through notifySubscribers. session.addSubscriber() adds one that only hears this session.

Goals, forms and other per-turn state

Goals and forms work on a call with nothing extra to wire. Their tools are on the call, and their status reaches the model through runtime context. Their per-turn preparation (preparing state, resuming interrupted lifecycle hooks, re-applying configure) is registered with glove-core's addTurnPreparer. The session runs it before it opens and when each user turn starts. Any prompt or tool change it makes is sent to the provider.

useGoalRunner(agent, goalAdapter, { scope, configure });
const session = new RealtimeSession({ agent });   // goals prepare on every turn

// Per-turn work that isn't registered on the agent:
new RealtimeSession({ agent, beforeTurn: () => syncCrm(callerId) });

A goal-driven sales conversation

Goals can guide what the agent asks and does next. In the runnable sales demo, the agent opens the conversation, discovers the prospect's workflow, appointment volume and pain points, consults product tools, recommends a fit, and agrees on a next step. It records progress through the real glove_goal_status and glove_goal_updatetools, with version checks. A refusal is a valid outcome.

const { runner } = useGoalRunner(agent, goalAdapter, {
  scope: { subject: "demo-prospect", key: "sales-discovery" },
  delivery: "history",
  tools: { allow: ["status", "update", "history"] },
});
await runner.start(salesProgram); // discover needs → recommend a fit → agree a next step
const call = new RealtimeSession({ agent });
await call.start();
call.respond("Introduce yourself and ask the first discovery question.");

// A product update arrives while the same call is running:
await agent.emitEvent({
  id: "pilot-offer:1", source: "product-catalog",
  content: "Growth now costs 39 USD per month instead of 49 USD.",
});

With delivery: "history", a changed goal snapshot is saved before the next user turn. Progress within a turn reaches the model through tool results. Events are saved once and enter the live session at a user/tool boundary or a quiet pause. On GPT-Live, context appends reach the voice model; backend goal tools read authoritative Glove state. Appends do not remove older provider context.

Full runnable demo and source: browser audio, typed input, tool activity, stored goal snapshots, spontaneous offer updates and a real-provider verification script. The fictional sales call does not dial a phone number or schedule a meeting. In-memory adapters preserve progress across provider reconnects, but durable adapters are needed across server restarts.

# From this repository, with OPENAI_API_KEY in ignored .env.local:
pnpm --filter glove-realtime... build
pnpm --filter glove-realtime demo:live
# Optional real-provider test (uses API credits):
pnpm --filter glove-realtime test:live

Session lifecycle

session.on("state", (state) => hud.show(state));   // connecting → open ⇄ reconnecting → closed
session.on("resumable", (handle) => saveHandle(callId, handle));
await session.start({ resumeHandle: loadHandle(callId) });

• A dropped session is retried with backoff. The first attempt resumes with the latest handle. Later attempts open a fresh session seeded from the store.
• On Gemini's goAway, the session moves to a resumed connection before the provider ends the current one.
• With compaction.contextLimit, the session compacts when the provider reports a larger context. OpenAI writes the summary itself, out of band. Otherwise compaction.model writes it, or the transcript is used.

Synthetic prompts and media

session.inject({ kind: "user", text: "The warranty check came back: covered until 2031." }); // spoken about
session.inject({ kind: "context", text: "Caller is a VIP." });                                // silent
session.respond("Greet the caller in one short sentence.");
await session.say("typed message from the chat box");
session.sendImage(pngBytes, "image/png");

GPT-Live host integration

Select provider: "openai-live" and model: "gpt-live-1" on the server. Feed continuous, paced PCM, including silence. GPT-Live emits continuous transcript fragments; the host supplies speech boundaries and playback state.

// Connect these to local VAD or application speech boundaries.
await session.notifyUserSpeech(true);  // prepare goals and request/history context
await session.notifyUserSpeech(false);
session.notifyPlayback(true);          // actual audio playback starts
session.notifyPlayback(false);         // playback queue drains
session.on("transcript", ({ role, text, startMs, endMs }) => {
  // Keep both speakers' fragments and timestamps: speech may overlap.
});
session.interrupt();                   // flush playback on the interrupted event
session.resumeOutput();                // allow output again
await session.stop();                  // wait for final cumulative usage

Input notifications refresh context and control when queued events reach the call. They do not commit audio or cancel backend work. Transcript packet gaps do not establish a speech boundary. Typed messages also run preparation and refresh context before sending. Startup waits for the agent's turn queue; concurrent stops wait for the same finalization, and failures reject after cleanup.

Related

• Realtime Voice & Avatars: the voice-only S2S path, avatars and LiveKit
• Goals: obligations with versioned progress
• Voice agent architectures