Durability

Foundry is built so that a process can die at any moment (the controller, the execution daemon, a run) and nothing it accepted is lost or silently done twice. It gets there by writing what it owes before it acts, and by delivering again, under the same identity, anything it cannot prove finished. That only holds if the stores underneath are durable: in production that means a real database, not the in-memory or file adapters the examples use.

This page covers what Foundry guarantees, how it keeps those guarantees, what your code must do in return, and what a database needs to provide. For configuring each store, see Persistence.

The model

1. Journal first. Foundry records an obligation before it acts on it: an inbound event before its run is queued, a schedule occurrence before the schedule moves past it, a voice notice's receipt before the provider is asked. A crash leaves either nothing (so the cause fires again) or a stored obligation (so the restart delivers it).
2. Claims decide races. Wherever two processes, retries or webhooks could do the same thing twice, the first atomic insert-if-absent wins and the rest see the existing record. These claims are the operations a database adapter must get exactly right.
3. Stable identities. Every obligation has an id derived from what it is, not from when it ran: the same webhook event, schedule occurrence or late-chat effect has the same id on every retry.
4. At least once. When Foundry cannot prove something finished, it delivers it again. A crash after a side effect but before its acknowledgement repeats the work, so side effects must be idempotent on those stable ids. Nothing in Foundry makes an arbitrary external effect exactly-once.
5. Recover on startup. A starting controller reads the journal and finishes what was left: pending inbound events, delegated tasks, the event journal, approvals, schedule occurrences, then activations from their stored cursors.

What is durable, and where

An interrupted model turn is never resumed mid-instruction: the work is delivered again as a new run, which starts from the conversation as stored.

What your code must make idempotent

Anything that leaves Foundry (a message sent, a card charged, a ticket filed) can happen twice when a crash lands between doing it and recording it. Key each effect on the id Foundry gives you, and let the receiving system refuse a repeat:

Most providers accept an idempotency key (Stripe, most email and messaging APIs). Where one does not, record the effect yourself under that key before acting, or check for it first, in the same database.

What a crash does, step by step

These guarantees assume an acknowledged write is durable. If your database can lose writes it already confirmed (an unflushed Redis append-only file, an asynchronous replica that takes over), a crash can undo a claim or a journal entry and the guarantee goes with it.

What the database must provide

• Durable acknowledged writes. When a write returns, it survives a crash of the database process and of the host.
• Atomic insert-if-absent for claims, and a unique constraint for provisioning keys.
• An atomic conditional update for workspace compare-and-set.
• Multi-record atomicity (a transaction, or an idempotent sequence) for purging a worker's data.
• Read-your-writes. Startup lists the journal; it must see everything already acknowledged, so read from the primary or with strong consistency.
• Lossless documents. Foundry treats records as opaque JSON. Store them whole and hand them back unchanged; index only the fields the adapter queries.

The atomic operations, by database

PostgreSQL (recommended)

PostgreSQL gives every guarantee above with its defaults, and every adapter method is a single statement except the purge. A complete, tested implementation of the data adapter and a transcript store is in examples/foundry-postgres. Its tests run the adapter contract against a real server, including concurrent claims from separate connections, and restart a real Foundry runtime on the same database. Every claim has the same shape:

/** Insert-if-absent: the obligation exists before the activation's cursor moves past it. */
claimScheduleOccurrence(occurrence: FoundryScheduleOccurrence) {
  return run(async () => {
    const result = await this.pool.query(
      `insert into foundry_occurrences (id, activation_id, workspace_id, agent_id, status, scheduled_for, doc)
       values ($1, $2, $3, $4, $5, $6, $7) on conflict (id) do nothing`,
      [occurrence.id, occurrence.activationId, occurrence.workspaceId, occurrence.agentId,
       occurrence.status, occurrence.scheduledFor, JSON.stringify(occurrence)]);
    return (result.rowCount ?? 0) > 0;
  });
}

• Store each record in a json column, which keeps the text as written, with the queried fields in their own columns. jsonb reorders keys; that is fine for an index, not for a document you hand back.
• The default isolation level (read committed) is enough: each method is one statement whose constraint decides the race.
• Create tables in a deploy step or migration, not when foundry.application.ts loads: that module loads in every run's process.
• Back it up like any production database (point-in-time recovery). A restore rewinds everything Foundry owes to that point.

Redis

Redis can back the data adapter, but only when it is configured to keep what it acknowledges. With appendfsync everysec (the common default) a crash can lose about a second of confirmed writes: claims and journal entries included. Use appendonly yes with appendfsync always, and remember that replication is asynchronous, so a promoted replica can be missing the newest writes. Many teams keep the record of truth in PostgreSQL and use Redis only for caches.

Keep one key per record (foundry:event:<id>, holding the JSON) and a set per listing (foundry:events), written together in MULTI/EXEC. Claims are SET … NX. Provisioning and compare-and-set need a script, so the check and the write happen together. These scripts were checked against a real server:

-- KEYS[1] = provisioning key index, KEYS[2] = the new agent's record key
-- ARGV[1] = the new agent id, ARGV[2] = the new agent document
local existing = redis.call('GET', KEYS[1])
if existing then return redis.call('GET', 'foundry:agent:' .. existing) end
if redis.call('EXISTS', KEYS[2]) == 1 then return redis.error_reply('agent id taken by another provisioning key') end
redis.call('SET', KEYS[1], ARGV[1])
redis.call('SET', KEYS[2], ARGV[2])
return ARGV[2]

-- KEYS[1] = workspace entry key
-- ARGV[1] = expected updatedAt ('' = must be absent), ARGV[2] = new updatedAt, ARGV[3] = new document
local current = redis.call('HGET', KEYS[1], 'updatedAt')
if ARGV[1] == '' then
  if current then return 0 end
elseif current ~= ARGV[1] then
  return 0
end
redis.call('HSET', KEYS[1], 'updatedAt', ARGV[2], 'doc', ARGV[3])
return 1

On Redis Cluster, every key one script touches must share a hash slot: wrap the common part in braces (foundry:{agent}:…).

MongoDB

• One collection per record kind, with the record id as _id. A claim is insertOne; a duplicate-key error (code 11000) means another caller won.
• A unique index on provisioningKey (sparse or partial, since most agents have none) makes provisioning atomic.
• Compare-and-set is updateOne({ _id, updatedAt: expected }, { $set: … }); it won when matchedCount is 1.
• Write with { w: "majority", j: true } and read from the primary. A purge in one transaction needs a replica set, which production deployments have anyway.
• Documents are limited to 16 MB: keep large VFS snapshots in object storage and store a reference.

async function claim(collection: Collection, record: { id: string }): Promise<boolean> {
  try {
    await collection.insertOne({ _id: record.id, doc: record }, { writeConcern: { w: "majority", j: true } });
    return true;
  } catch (error) {
    if ((error as { code?: number }).code === 11000) return false; // someone else claimed it
    throw error;
  }
}

DynamoDB

• Claims are PutItem with ConditionExpression: "attribute_not_exists(pk)"; a ConditionalCheckFailedException means another caller won.
• Compare-and-set is UpdateItem with ConditionExpression: "updatedAt = :expected".
• Provisioning writes a key item (pk = "provisioning#<key>") and the agent in one TransactWriteItems, conditioned on the key item not existing.
• Listings Foundry reads at startup (the event journal, pending inbound events, open occurrences) must see every acknowledged write. Model them as queries on the base table (one partition per kind, for instance) with ConsistentRead: true. Global secondary indexes are eventually consistent and can miss a write made a moment before a crash.
• Items are limited to 400 KB: store VFS snapshots, and very long records, in S3 with a reference.

SQLite and files

FileFoundryDataAdapter and the SQLite memory adapters are durable on one host (atomic rename and fsync, SQLite's write-ahead log) and are what the examples use, because they need no server. They suit development, demos and single-machine deployments with a persistent disk. They are not for several hosts, and the file adapter rewrites its whole state on every change, so it slows as state grows. See Persistence.

Processes and connections

• Every run executes in its own process, which loads foundry.application.ts and the agent module again. Build the database client there, from configuration (an environment variable). Never pass a live connection or pool from the controller: each process opens its own.
• Size the database for it: the controller plus up to execution.maxConcurrent run processes, each with its own pool. Keep per-process pools small, and let idle connections close (pg's allowExitOnIdle) so a finished run's process can exit.
• Run one Foundry controller per data store. Two controllers on one store would each re-arm every schedule and replay every pending obligation. For availability, restart the controller (a supervisor, a container platform); it recovers from the store.

Production checklist

• A database-backed data adapter with all the optional durability methods: event journal, pending inbound, approvals, schedule occurrences, delivery receipts, purge.
• A database-backed transcript StoreAdapter, with permissions if you use approvals.
• Durable memory adapters and VFS persistence if your agents use them.
• The database configured for durable acknowledged writes and read from the primary.
• Side effects keyed on the ids above.
• One controller per store, restarted automatically.
• Backups, and a restore you have tried.
• The adapter contract tests from examples/foundry-postgres, run against your adapter.
• A kill test: kill -9 the controller while work is due or running, restart it, and check that nothing was lost and every repeat carried the same id.