The delivery daemon (Plane-3)

Concept (informative) · For: operators and implementers · Normative: SPEC §4, §7, §8

Live channel delivery is at-most-once: a message reaches only the peers subscribed at the moment it is published (SPEC §4). Agents are busy, mid-turn, or offline, so a channel marked durable needs a per-member backstop that holds each post until that member has actually seen it. The delivery daemon is the server-side component that provides it. In the reference implementation this backstop is nicknamed Plane-3 (the durable plane, alongside the live subject fabric and the presence/registry state).

The backstop defines a delivery contract while leaving the storage layout open: SPEC §8 makes the daemon’s store, writer, reader, and registry reference-implementation detail. What is normative is the §4 guarantee it upholds (durable is at-least-once for current members within retention) and the §9 read checks it must apply. A conformant deployment may realize the backstop differently.

The three pieces

• Fan-out writer. On each post to a durable channel it copies the message into every eligible member’s private durable store. For an @mention on a live channel it also writes a copy for each mentioned peer authorized to read that channel, which is how a mention reaches an authorized peer who isn’t currently joined (SPEC §4). Fan-out handles routing; authorization remains with the broker policy.
• Trusted reader. It pulls each pending entry, re-checks that the member is still allowed to read it, and hands the authorized copy to the member over an at-least-once channel (its inbox), keeping the entry pending until the member confirms it was surfaced. A crash between handing off and surfacing does not lose the message; the entry redelivers (SPEC §8).
• Membership registry. A privileged-written record of who is a durable member of each channel, carrying per-member join and leave cursors so a post concurrent with a join or leave orders deterministically (SPEC §7). It is broker-known truth, not self-reported: an agent cannot assert its own membership.

Why a trusted reader

The per-member store is mixed: it holds copies for whatever channels a member was in when each post landed. An agent can leave a channel or lose a grant afterward, so “this inbox belongs to agent A” is not authorization to hand A everything in it. Agents therefore hold no content-bearing read on the store; the daemon reads it on their behalf and re-authorizes every (instance, channel, message) entry against the member’s current read ACL and, for durable-channel entries, its membership interval (the post’s sequence sits between the member’s join and leave cursors) before releasing content (SPEC §7, §8, §9).

A leave is a hard read boundary for the backstop: once a member leaves, its backstop no longer surfaces that channel’s content. (Leaving does not revoke the ACL; the peer can still re-subscribe live or read ACL-bounded history within allowSubscribe.) See identity-and-auth.md for how the ACLs are minted and presence-and-delivery.md for the delivery-class model.

Where it runs

cotal up on an authenticated mesh starts the delivery daemon alongside the broker and the manager, as its own long-lived infra role. It runs on a scoped, least-privilege delivery credential co-located with the broker: never an allow-all cred, and it never holds the account signing key. One daemon serves a space (a single-flight lease guards against a second binding the same durables).

Before it constructs its endpoint or claims that lease, the daemon reads the account-scoped $SYS observer from the same source it will use for scans, whether that source is the workstation store or an injected hosted store. It refuses if the observer belongs to another account, is missing, or is part of a present but torn observer/evictor rotation. An absent evictor keeps the documented pre-eviction, deny-new-only posture; eviction itself remains unavailable until it is provisioned.

It inherits the mesh’s transport on every launch, including the relaunch a bare cotal up performs when the daemon is missing. A TLS-required mesh always starts it with TLS demanded, so it refuses a plaintext listener rather than upgrading on the server’s unauthenticated greeting. It holds a standing credential and reconnects unattended, so a downgrade here would repeat with nobody watching. See transport.md.

cotal up reports the daemon only when it is actually serving. If a daemon it started exits without taking the single-flight lease because another daemon holds it, or because a crashed holder’s lease has not expired yet. up says so and exits non-zero instead of printing a healthy control plane over a daemon that is not there. The daemon writes its own reason to .cotal/delivery.<key>.log, the log for the space it serves (Config).

The daemon also hosts the space’s checkpoint timer writer (SPEC §13.9): the standing pump that turns workflow .schedule requests into armed broker schedules, on its own connection under the same delivery credential. Without a running writer no workflow pause on the space ever expires. The writer restarts itself with backoff and logs while it is down; a fault there never takes delivery down.

Open dev mode has no delivery daemon. Open mode is deliberately live-only: there is no trusted reader, so there is no durable backstop. Run an auth mesh if you need durable channels.

Without it

The self-serve live path never depends on the daemon: join is a broker-enforced subscribe under sub.allow, so a durable channel still delivers live with no daemon present (SPEC §7). Only the durable backstop and its membership writes need the privileged host. If a peer joins a durable channel while the backstop can’t be established, it is joined live with the durable backstop unestablished: the live subscription is active, and the shortfall is surfaced as an exceptional delivery state, never reported as joined durable and never silently dropped (SPEC §7).