The release post said what shipped. This one is the working notes: five preserved examples, three avatar integrations, and a run of bugs that every test suite passed straight through. The decisions that held, and the ones the wire corrected.
Realtime voice is unusually punishing to build because the failure mode is silence. A REST integration that is wrong throws. A voice session that is wrong connects, streams your microphone at a provider, and returns nothing — with every green check still green. Almost everything below follows from that one property.
The obvious way to build a voice agent is to point a realtime model at your tools and let it work. We tried that first and it is wrong for any agent that has to be correct. Realtime models are tuned for conversational latency, not for reasoning over a database, and the moment one is asked to do both it either stalls mid-sentence or invents an answer confidently.
So the front agent is deliberately thin: it owns the conversation, the voice, and the turn-taking, and it knows almost nothing. Anything requiring real work is delegated over glove-mesh to a capable worker model running as a separate job, whose answer is injected back into the live session when it lands.
// The front agent talks. The worker knows things.
const front = buildS2SFrontAgent(store, s2sConfig);
await mountMesh(front, { adapter: meshAdapter, identity: FRONT_IDENTITY });
const rt = new RealtimeAgent({ agent: front });
// …and when the worker replies, it arrives mid-conversation:
rt.inject(`<worker-result>${message.content}</worker-result>`, { respond: true });The prompt rule that makes it work is blunt: never invent a number; if a lookup is pending, say you are still checking. A thin agent that admits it is waiting is better company than a capable one that guesses. This split also means the expensive model is only paying for the turns that need it.
An early version wired the realtime session separately from the agent: build the agent, then build an adapter, then keep them in sync by hand. Every drift between them was a silent behaviour change. The fix was to make the model slot itself carry the realtime configuration, so the agent definition stays the single source of truth and RealtimeAgent derives the session from it.
The same instinct produced typed turn-taking knobs rather than a passthrough JSON blob. These values decide whether your agent feels patient or feels like it talks over people, and they are static per provider — exactly the thing a type should catch. A typo in a raw blob is not an error, it is a personality change you find out about from a user.
Every avatar vendor wants to sell you the whole stack — perception, an LLM, a voice, a face. We use exactly one quarter of that. The agent stays the brain, the realtime model stays the voice, and the avatar is handed finished PCM to lip-sync. Providers call this echo or passthrough mode, and it is the only mode that preserves the layering.
That yields one contract — AvatarAdapter: connect and get a view a client can attach to, feed it audio, end the utterance, interrupt. Interruption is conformance-enforced to be safe at any time, including when nothing is playing, because the voice side treats every user speech-start as a potential barge-in and the face must follow the voice without asking questions.
The reward for the contract came later: the second provider, and then the same providers over a different transport, were all attachAvatar(rt, avatar) and done.
Both avatar vendors turned out to hide the same structural surprise, in different places. Tavus interaction events travel only over the Daily data channel — there is no REST endpoint, no matter what an eager reading of the docs suggests. Anam's passthrough audio input lives only on the browser SDK — there is no server-side audio API at all.
In both cases the server has the audio and no way to deliver it. Rather than smuggle a media client into the room process, both adapters make the gap explicit: a required sendInteraction / sendCommand function the host supplies, which ferries frames to whoever is joined — in our examples, the browser already on the call.
// Required, not optional — the transport genuinely does not exist
// server-side, and pretending otherwise fails at runtime instead of here.
const avatar = new AnamPassthroughAdapter({
apiKey, avatarId,
sendCommand: (command) => duct.send({ t: "avatar_command", command }),
});Making it a required constructor argument was deliberate. An optional courier with a plausible-looking default would have produced an adapter that connects, reports healthy, and never renders a frame.
Worth stating plainly because it is the most natural wrong assumption: adding LiveKit does not remove the need for a Tavus or Anam key. LiveKit has no avatars of its own. What its ecosystem standardises is the wire — a published protocol by which the provider's renderer joins your room as an ordinary participant and reads agent audio off a byte stream.
So the trade is not fewer vendors, it is one transport you own instead of one bespoke surface per vendor. Concretely, in our examples: the browser hook went from about 500 lines to about 250 — audio worklets, a playback ring buffer, a local VAD reflex and a pause/resume/clear protocol all deleted, because WebRTC already does that. Barge-in became server-authoritative: the room flushes its own outbound queue, so there is no client buffer to chase.
Every one of these shipped with a passing test suite. They are grouped because the pattern matters more than any single fix.
The first Tavus call greeted the caller in a stranger's voice before switching to ours. Suppressing the greeting parameter did not fix it: the greeting was configured on the persona, and conversation parameters do not override persona defaults. The fix is to stop reusing a dashboard-made persona and have the adapter ensure a minimal one — echo mode, no greeting, no TTS layer — reused by name. Both LiveKit's and Pipecat's integrations do the same thing, which was the hint that we were fighting the platform rather than a bug.
Anam sessions kept dying a few minutes in. The first fix — raising every timeout the API exposes — was wrong, and the give-away was that it didn't work: the real limit is a plan cap that force-ends conversations regardless of session configuration. Once that is true, the cap is routine rather than exceptional, and the correct handling is renewal, not prevention: the session ends, the host mints a new one, the client re-attaches, and the face blinks instead of dying.
The Gemini path connected, streamed microphone audio, and returned nothing at all. Gemini delivers its JSON over binary WebSocket frames; the adapter parsed them with String(raw), which for a Blob is the literal text "[object Blob]". Every inbound message was dropped, forever, silently.
The conformance suite missed it because the fake socket fired strings. That is the whole lesson: a test double that is more convenient than the real wire tests your convenience. The fake now emits binary frames like the endpoint does, which turns the entire existing suite into a regression test for that class of bug.
Then it still did not work. Gemini's schema type is an OpenAPI 3.0 subset, not JSON Schema, and it rejects the entire session over a single unrecognised key. Zod's toJSONSchema() — which every Glove tool passes through — emits $schema and additionalProperties by default. Any agent with tools could never open a Gemini session.
The sanitizer that fixes it uses an allowlist of supported keys rather than a blocklist of known-bad ones, deliberately: a blocklist would have to be updated every time Zod learns a new keyword, and the failure mode of being out of date is the voice disappearing.
And then the model itself was "not found" — except the model id was correct. The WebSocket URL pinned v1beta, and newer preview models land on v1alpha first. That error message has three indistinguishable causes (wrong id, wrong version, no access on the key), so alongside making the version configurable we added a lookup that asks the provider which models this key can actually open a session with, and prints the environment lines to paste.
Four of those bugs were quick to fix and slow to find, for one shared reason. The adapter's close handler looked like this:
ws.addEventListener("close", () => {
this.connected = false;
this.emit("disconnected");
});The provider was explaining itself the entire time — 1007: Unknown name "$schema" at 'setup.tools[0]…' — and we were dropping it on the floor and reporting a tidy disconnect. Once the close code and reason were surfaced as errors, the next two bugs were diagnosed from a single log line each.
Error plumbing is not hygiene work you get to after the feature. In a stack whose failure mode is silence, the diagnostics are the feature — they are what converts "it doesn't work" into a specific, fixable claim.
The same reasoning produced a probe script in the example: it drives the real agent with its real tools through the real code path, minus the microphone, and prints either working audio or the provider's own complaint. Voice bugs are expensive to reproduce by talking to a browser; a five-second command that returns the truth is worth more than the hour it saves each time.
Every adapter family in this stack ships a behavioural suite that runs against fakes with no credentials — and the honest framing, repeated in every README, is that passing proves the adapter is wired correctly against its own reading of the protocol. Only a live call proves the reading.
This session was an extended demonstration. The suites were genuinely valuable: they caught an utterance-boundary race where audio chunks merged into one stream, and they made the second and third providers cheap to add. They were also, simultaneously, entirely green while the Gemini path could not hear, could not open a session with tools, and pointed at the wrong API version.
So the seams that let a suite run credential-free — an injectable fetch, an injectable socket, an injectable courier — are the same seams that let a fake drift from reality. Keep them, and keep the fake shaped like the wire: binary where the wire is binary, strict where the provider is strict.
Each step is preserved as its own runnable example rather than an upgrade of the previous one, with ports shifted so they run side by side. The instinct to keep them came from the person testing them: being able to return to the last thing that worked is worth more than a tidy repository.
| Example | Pipeline | Transport |
|---|---|---|
layered-voice | Cascade (VAD → STT → LLM → TTS) | Browser-hosted |
server-voice | Cascade | Server rooms, WebSocket audio duct |
s2s-rooms | Speech-to-speech | Server rooms, WebSocket duct |
avatar-rooms | Speech-to-speech + a face | Duct up, provider session down |
livekit-rooms | Speech-to-speech (+ optional face) | LiveKit, both directions |
They are the same starship dealership throughout — a salesperson who knows nothing about ships and a worker who knows everything, so the delegation is visible in the conversation rather than buried in a latency graph. Reading them in order is the fastest way to see what each transport actually costs and buys.
The packages are glove-voice-s2s, glove-voice-avatar and glove-voice-livekit, and the examples are in the repository. All MIT.