Voice agent

A voice harness has one hard constraint: silence is a bug. Whatever the agent is doing, it has to answer within a breath. That puts the weight on the surface (speech in and out, barge-in, turn-taking) and on the team: a fast front agent that talks, and a heavier worker that does the tool work behind it. Durable work, the kind that should survive the call, goes to a Foundry instance.

How it fits together

Pick the front first. A pipeline (glove-voice) runs VAD, speech-to-text, the Glove agent and text-to-speech, with barge-in and push-to-talk; any chat model can be the agent. A realtime model (glove-voice-s2s: OpenAI Realtime, Gemini Live, GPT-Live) owns turn-taking itself, and RealtimeAgent hands it the same Glove tools. Either way the tools are written once.

A caller asks for something that needs real work: “what's the warranty on hull KES-0007?”. The front agent has almost no tools. In the same turn it says “one sec, let me pull that up” and calls glove_mesh_send_message to the worker with blocking: true. The message lands in the worker's inbox as a resolved item; the host sees it and runs the worker, which researches with its tools and replies with in_reply_to set. The reply lands in the front agent's inbox, which Glove injects on its next turn, and the front relays it out loud. While the request is pending, the blocking item keeps the front from inventing an answer.

App state the caller should not have to repeat (who is calling, what is on screen, the order status) reaches the voice model through runtime context. A realtime session gets changed snapshots silently at start and after each tool; call refreshContext() when something changes outside a tool. Forms work the same over speech as over text, so collecting a delivery address is a form, not a prompt.

Skeleton: a fast front and a worker

Adapted from examples/layered-voice, which adds speaker labels, a <speech> tag protocol (only text inside the tags is spoken) and ElevenLabs streaming.

import { Displaymanager, Glove, MemoryStore, createAdapter } from "glove-core";
import { InMemoryMeshAdapter, MeshNetwork, mountMesh } from "glove-mesh";
import { workerTools } from "./tools.js"; // your code: the data the worker can reach

const network = new MeshNetwork(); // in-process; use a transport-backed MeshAdapter across processes

const front = new Glove({
  store: new MemoryStore("front"), // inbox-capable: mesh messages land here
  model: createAdapter({ provider: "openrouter", model: "google/gemini-2.5-flash" }), // small and fast
  displayManager: new Displaymanager(),
  systemPrompt: `You are the voice at the front desk. Speak in one or two short sentences.
You cannot look anything up. For any lookup, call glove_mesh_send_message
to "worker" with blocking: true, and say a short acknowledgement in the same turn.`,
  serverMode: true,
  compaction_config: { compaction_instructions: "Keep who is calling and what is pending." },
}).build();

const worker = new Glove({
  store: new MemoryStore("worker"),
  model: createAdapter({ provider: "anthropic", model: "claude-sonnet-5-5" }), // heavier
  displayManager: new Displaymanager(),
  systemPrompt: "Handle each delegated request with your tools, then reply to \"front\" with in_reply_to set. Never just acknowledge.",
  serverMode: true,
  compaction_config: { compaction_instructions: "Keep open requests and findings." },
});
for (const tool of workerTools) worker.fold(tool);
const backOffice = worker.build();

await mountMesh(front, {
  adapter: new InMemoryMeshAdapter(network, "front"),
  identity: { id: "front", name: "Voice front", description: "Talks to the caller", capabilities: ["voice"] },
});
await mountMesh(backOffice, {
  adapter: new InMemoryMeshAdapter(network, "worker"),
  identity: { id: "worker", name: "Back office", description: "Looks things up and acts", capabilities: ["research"] },
});

// The host wakes the worker when a delegation arrives, and the front when the reply does.
export async function onUtterance(text: string) {
  await front.processRequest(text); // your voice front streams its text to TTS
  const pending = (await backOffice.store.getInboxItems?.()) ?? [];
  if (pending.some((item) => item.status === "resolved" && item.tag.startsWith("mesh:from:"))) {
    await backOffice.processRequest("You have new delegated requests in your inbox.");
    await front.processRequest("[The worker replied. Relay it.]"); // the reply is injected from the inbox
  }
}

Skeleton: a voice agent in Foundry

In Foundry the call is a run. voiceCall hosts the realtime session over the assembled agent, saves the transcript, and speaks delegates' results at the next pause. The front stays a fast talker; the work goes to sibling agents that can take their time, ask a question back, and use tools a call should not.

import { defineAgent } from "glove-foundry";
import { voiceCall, webSocketRoom } from "glove-foundry/voice";
import { s2sDrivenModel } from "glove-voice-s2s";
import { FileStore } from "../../lib/file-store.js";
import { conciergePrompt } from "./prompt.js";
import { roomFor } from "./room.js";

export default defineAgent({
  description: "Answers the phone and hands longer work to the back office",
  model: () => s2sDrivenModel({ label: "concierge", provider: "openai", voice: "marin" }),
  systemPrompt: conciergePrompt,
  store: ({ conversationId }) => FileStore.open(conversationId),
  delegates: [
    { agent: "bookings", description: "Books tables and travel. Takes a few minutes." },
    { agent: "research", description: "Looks things up on the web and cites sources." },
  ],
  run: voiceCall({
    room: (context) => webSocketRoom(roomFor(context)),   // { port, token } from the request
    greeting: "Greet the caller in one short sentence.",
  }),
});

The bookings agent is an ordinary definition. If it needs to know something (indoors or outside?), glove_foundry_ask brings the question back into the call, where the concierge answers from what it knows or asks the caller. Its runs show up in the inspector and can be approved or steered, and a result that lands after the caller hangs up waits in the conversation. LiveKit rooms and phone bridges plug in as rooms. The full version, with personal memory and a text chat on the same agent, is examples/foundry-layered-assistant.

What not to build

• A second implementation of your tools for the realtime provider. RealtimeAgent reads the agent's registered tools and runs each call through the same Tool.run.
• Speech-to-text, agent and text-to-speech glue. The voice pipeline handles VAD, turn commits, barge-in and push-to-talk, in the browser and on React Native.
• A message bus between the voice agent and the worker. Mesh delivers through each agent's inbox, and a blocking send keeps the front from answering before the worker does.
• Rewriting the session instructions every time app state changes. Runtime context is refreshed silently without a reply and without growing the prompt.
• A job tracker for work started on a call. A Foundry run handle waits, streams, cancels and steers, and the run outlives the call.

Deploy on Kuiper

The durable half deploys like any Foundry app: one unit, its data on /data, a requestAuthorization adapter, models through the gateway. The voice host is where audio lives, so its shape depends on the platform:

• Browser voice (a pipeline or WebRTC realtime): a Next.js unit that serves the page and mints short-lived provider tokens (createVoiceTokenHandler from glove-next). Keep the provider keys as project secrets bound to that unit.
• Server-side voice (LiveKit rooms, phone bridges): a Node service unit that joins the room and runs RealtimeAgent.

Either way the host reaches Foundry over the project's private network: kuiper env set -u voice 'FOUNDRY_URL=${{concierge.INTERNAL_URL}}'. See Deploy on Kuiper and Next.js with Foundry.

Go further

• Voice agent architectures: pipeline versus realtime, single versus layered, and the <speech> protocol, measured.
• examples/foundry-braind-storm: a Gemini Live briefing line that launches durable Foundry workforces mid-call.
• Showcase: Coffee Shop and Lola, voice-first apps with tool-rendered UI.
• Build the parts one at a time: show your work, split the work.
• Reference: Voice pipeline, Realtime voice & avatars, Mesh, Foundry voice hosts.