Voice agents

A voice agent is an ordinary definition with a speech model and run: voiceCall(...). The call uses the same prompt, tools, memory and context as the agent's text turns, saves what both sides say, and stays quick because anything slow goes to a delegate. When a delegate finishes mid-call, the result is spoken at the next pause.

Define the call

The model is s2sDrivenModel(...) from glove-voice-s2s: a realtime provider drives the turns. voiceCall from glove-foundry/voice is the run. Give it a room, which is where the audio comes from and goes to:

import type { AgentAssemblyContext } from "glove-foundry";
import { voiceCall, webSocketRoom } from "glove-foundry/voice";
import { z } from "zod";

/** How the app starts a call: the port the page will connect to, and a one-time token. */
export const callRequest = z.object({
  call: z.object({ port: z.number().int().min(1024).max(65_535), token: z.string().min(24) }),
});

export function isCall(context: Pick<AgentAssemblyContext, "request">): boolean {
  return callRequest.safeParse(context.request.payload).success;
}

export const call = voiceCall({
  room: (context) => {
    const { call: room } = callRequest.parse(context.request.payload);
    return webSocketRoom({ port: room.port, token: room.token, agentName: "Iris" });
  },
  greeting: "The person just called. Greet them in one short sentence, by name if you know it.",
});

One agent can take both calls and chat. Pick the model and the run by the request, as the layered assistant does:

model: (_agent, context) => (isCall(context) ? callModel() : interactorModel()),
systemPrompt: (_agent, context) => interactorPrompt(isCall(context)),
run: (agent, context) => (isCall(context) ? call(agent, context) : context.defaultRun()),

// lib/models.ts
export function callModel(): ModelAdapter {
  return s2sDrivenModel({ label: "assistant-call", provider: "openai", voice: "marin" });
}

Start a call from your app by sending a message whose payload names the room:

const token = randomBytes(24).toString("base64url");
await foundry.createConversation(personId, { id: `${personId}-call` });
await foundry.send(personId, `${personId}-call`, "call", { payload: { call: { port, token } } });
return Response.json({ url: `ws://127.0.0.1:${port}/call?token=${token}` });

What the call runs

• A RealtimeAgent over the assembled agent: its system prompt, its tools and its runtime context (memory, goals, open delegations), refreshed after every tool call.
• What the caller says and what the agent says are appended to the conversation's store, so the next text turn or call has the history.
• The greeting is sent once, when the first caller joins. Leave it out to wait for the caller to speak.
• The call ends when the room calls end(), nobody is connected for idleMs (2 minutes), maxDurationMs passes (30 minutes), or the run is cancelled.

Which tools are on the call

A realtime session calls tools directly, mid-sentence. Two kinds don't belong there, so voiceCall keeps them off:

• Foundry's bookkeeping tools: spawn, sleep, background, the workspace, shared inbox, tasks and environment tools. glove_foundry_schedule, glove_foundry_delegate and glove_foundry_reply stay.
• Any tool with requiresPermission. The realtime path has no approval step, so the tool would run ungated. Delegate that work to a worker, where it pauses on an approval as usual.

Add more with realtime: { excludeTools: [...] }: anything that takes long enough to leave the caller in silence.

Results during and after the call

The call listens on its conversation, so delegation results and questions reach it directly instead of starting a new run. It holds each one until neither side is speaking (eight seconds at most), then tells the model, which tells the caller. A question is put the same way: the model answers from what it knows with glove_foundry_reply, or asks the caller first. The full update is also saved to the conversation. Change the wording with announce: (update) => string | null.

When the caller has hung up, an update starts an ordinary run in that conversation with source.kind "delegation". If your run sends it to voiceCall, the update is saved for the next turn; pass offCall to handle it another way. The layered assistant routes it to a text turn instead, so the result is waiting in the chat.

Rooms

A room connects the session to callers: { open(call), close() }. It feeds caller audio to call.realtime, plays the agent's audio, and calls call.joined() and call.left() as people come and go.

webSocketRoom

One WebSocket per call, served by the run on its own port. A caller connects to ws://host:port/call?token=…; a wrong token is refused, and a new connection replaces the old one. GET /health says whether someone is connected.

The braind-storm example's use-briefing-call.ts is a browser client for this protocol, with audio worklets for capture and playback.

A LiveKit room

glove-voice-livekit joins a LiveKit room from the run. The connection goes out from your server, so it works anywhere the run can reach LiveKit:

import type { FoundryVoiceRoom } from "glove-foundry/voice";
import { LiveKitTransport, attachRealtime, mintParticipantToken } from "glove-voice-livekit";

export async function liveKitRoom(roomName: string): Promise<FoundryVoiceRoom> {
  const transport = new LiveKitTransport({
    url: process.env.LIVEKIT_URL!,
    token: await mintParticipantToken(
      { apiKey: process.env.LIVEKIT_API_KEY!, apiSecret: process.env.LIVEKIT_API_SECRET! },
      { roomName, identity: "agent" },
    ),
  });
  let detach = () => {};
  return {
    async open(call) {
      transport.on("participant_connected", () => call.joined());
      transport.on("participant_disconnected", () => call.left());
      await transport.connect();
      for (const _ of transport.room.remoteParticipants.values()) call.joined();
      detach = attachRealtime(call.realtime, transport);
    },
    async close() {
      detach();
      await transport.disconnect();
    },
  };
}

// call.ts
export const call = voiceCall({ room: (context) => liveKitRoom(roomNameFor(context)) });

Options

The run returns { endedBecause, durationMs, turns, updatesSpoken }, and the trace shows foundry.voice.call.open, caller.joined, caller.left, tool, updates.spoken and call.ended.

Deploying

• webSocketRoom listens on its own port, separate from Foundry's. On a host with one public port, such as a Kuiper unit, put the room behind your own proxy or use a LiveKit room, which connects outward.
• A call holds a run, and one of execution.maxConcurrent's slots, for as long as it lasts. Size the limit for calls plus the workers they start.
• Provider keys stay on the server. Browsers only get the room's one-time token.