# Endpoint discovery ## Request routes One kind, `ep`, carries every request under a mode token that says where the request routes, never which verb it is (the verb rides the envelope): `one` (queue-group anycast, one and only one class member), `all` (scatter, every instance), and `inst` (one instance by its stable address). Replies come back on a `reply` rail keyed to the serving instance and its epoch. Around these sit the sibling planes the composites use: per-goal events, timers, sessions, and the journal that holds durable facts. Every request carries the caller as three forge-locked tokens, `owner`, `actor`, and lifecycle `uid`, plus an unguessable nonce, so the broker polices who is calling in the subject grammar itself. See [SPEC §13.2](/docs/spec/#132-grammar) for the grammar and [§13.5](/docs/spec/#135-verbs) for the verbs (`call`, `cast`, `watch`, `claim`, `scatter`). ## Lifecycle identity A principal `owner.actor` is a reusable routing alias: a despawn frees the actor name and a later spawn may legitimately reuse it, so the alias alone is never authority. Two further coordinates make an identity durable: a **lifecycle uid**, an unguessable, never-reused id for one managed lifecycle under a principal, and a **process epoch**, the fenced ownership epoch of the process currently animating it, advanced on every restart or takeover. At most one live epoch owns an identity, and a superseded epoch must stop serving. Durables and credentials key on the lifecycle uid, not the reusable name, which is what lets a supervised restart recover the same lifecycle instead of minting a new one. See [SPEC §13.1](/docs/spec/#131-lifecycle-identity) and [identity & auth](/docs/identity-and-auth/). ## Service discovery No client has compile-time knowledge of any endpoint's commands. `cotal describe ` resolves a registered endpoint's command set off the wire: the reserved `describe` command answers the registered contract digests, the schemas are fetched from the space's content-addressed contract store, recompiled, and verified against those digests. Each command prints with its capability class and targeting shape. `cotal invoke --args ''` then calls one command by name, validating the arguments as they will be sent (JSON drops a key whose value is undefined) against the fetched input schema before publish. A refusal at that check means nothing was sent, and it is marked `not-executed`. An endpoint that refuses a request before running the command states `not-executed` in the reply's `error.outcome` too, for example on a wrong envelope version, sender, contract digest or target mode, or a body that does not parse. A refusal the command itself raises carries only the outcome it states. A refusal that does not serialize comes back as `internal`, and one too large to publish as `resource-exhausted`, and both keep that outcome. A signed-in user invokes the same surface through their bearer, and the broker enforces each command's existing capability grant. A manager alias supplied through `--name` resolves through its name-keyed `inspect` command, so an authorized targeted call does not need the manager-wide `ps` enumeration grant. Every built-in manager command uses this same trust chain, so there is nothing the built-ins can reach that a described contract cannot. The registered `auth` endpoint is describable the same way `manager` is: `cotal describe auth` lists `retire-lifecycle` and its exact-mode target shape. See [SPEC §13.7](/docs/spec/#137-contracts-and-discovery) and [cli.md](/api-reference/cli/). ### Listing what a space offers An agent asks the space's registry what it can call with the `cotal_endpoints` tool. The registry is a registered endpoint named `registry`. Its `endpoints` command lists each registered endpoint where the caller holds at least one command, with the commands and request forms the caller's recorded ceiling covers, and leaves the other endpoints out. It reads the ceiling the issuer recorded for the caller's credential and does not check that the ceiling's sources are still live, so the broker or the endpoint can still refuse a listed command. The answer says this with `source: "recorded-ceiling"`. On an open mesh every registered endpoint is listed and `source` is `no-permission-system`. Each endpoint row also lists its registered instances, the state each one recorded, and whether its command declarations verified. A page holds up to 100 endpoints. When the registry does not answer, the tool says so and lists nothing; it does not build a list from `describe` or presence. A listing drops the session's cached `describe` of every endpoint it lists. On a user-auth mesh manager commands ride a separate manager-call credential, so `cotal_endpoints` with `view: "manager"` opens that credential as a manager call does and lists what it can call. A session whose bearer comes from its host has no manager-call credential, and the tool says so. See [SPEC §13.20](/docs/spec/#1320-registry). ## Machine vitals The manager's `vitals` command reports one invocation-time sample of host CPU count and load, available and total memory, and bounded local filesystem capacity. It is untargeted but resolves to one exact Manager instance. The separate `manager.vitals` capability is issued only when the actor scope contains `machine-vitals`; `spawn`, `run` and `admin` do not imply it. The response contains mount points and aggregate byte counts, never device paths, remote endpoints, process data, command lines, environment values or persisted history. Unsupported measurement refuses without a partial or zero-valued fallback. ## Host directory resolution The manager's `resolve-cwd` command is in the `manager.spawn` capability class. It accepts an absolute path on that manager's host and returns its canonical directory plus the host name. It refuses a relative, missing or non-directory path with `failed-precondition`; it creates nothing. `spawn` applies the same check at admission, before any credentials or durables are minted. It is minted beside `spawn` in the operator instruments, which are the caller on a static mesh, and in a `spawn` holder's `manager-caller` view. The standing agent `spawn` capability does not carry it. A runtime that delegates seats outside the manager's host cannot use a host cwd. Its manager refuses `resolve-cwd` with `failed-precondition` before any filesystem lookup, with the same execution-mode refusal for every path. It returns no host name or canonical path. ## Spawn input resolution The manager's `resolve-agent` command is minted beside `spawn` in the spawn capability and the operator instruments. It accepts the persona fields `spawn` takes (`name`, `config`, `agent`, `defaultAgent`) and returns `{agent}`, the harness `spawn` on that manager would launch: the flag, then the persona's `agent:` pin, then the caller's default, then the manager's own default. It loads the persona with the same visibility check as `spawn` and launches nothing. A detached `--resume` without `--agent` asks it before carrying a session, and a manager that does not serve it refuses the call. On a delegated runtime, both `resolve-agent` and `spawn` refuse an explicit persona `config` before any lookup of that path. Absolute, relative and bare-name config overrides receive the same `failed-precondition` refusal. Select a named catalog persona through `name` instead. Locally executing managers retain their config-path behavior. A resolved manifest uses its own validated, host-materialized persona resource and is not a caller-selected config override. `spawn` refuses an empty or whitespace-only value in any of its optional string fields, such as `role`, `agent` or `identity`, with `bad-request` naming the field. To take the persona file's value, omit the field. ## Instance routing A space can run more than one manager. Each manager persists a stable logical instance id across restarts and advances its process epoch when it comes back, so callers address a specific manager without caring which process currently serves it. A start serves only at the epoch its own registration committed, never at the epoch of a later start of the same instance. On a static mesh or from an unbound open caller, an untargeted spawn rides class anycast (any manager may accept, and the acceptance records which one did). An open managed seat's persona definition, list, show and spawn tools default to its launching manager. Its generic manager describe and self-targeted invoke use that launch address too. An explicit existing persona or spawn selector still selects that instance. Static credential grants and user-auth issuer selection are unchanged. For self-stop, persona definition and direct spawn, a static managed seat keeps its class-only credential and accepts only its launching manager's attributed describe reply. These commands retain that bind on the class rail, so a sibling refuses before effects and the existing bounded split repair asks again. An absent launching manager never selects a replacement. The library's `describeService` accepts an optional `instanceId`; a pinned describe does not read or populate the endpoint-only class cache. `cotal spawn --detach --on ` and `cotal_spawn(instance: "")` pin one instance by its exact id, as `--on` does for `cotal ps`, `cotal models` and every hosted `cotal run` verb, and `instance` for `cotal_run`, `cotal_persona` and `cotal_personas`. A foreground CLI spawn has no manager to pin and refuses the flag. An MCP pin that does not resolve is refused without falling back to class anycast. There are no ordinal aliases and no short forms: wherever a display names an instance you can address, it prints the whole id, because both surfaces take nothing else. On a user-auth mesh, manager commands obtain a short-lived `manager-caller` view from the exchange. It authorizes one concrete manager instance using the caller's current actor grant and the host's registered service records. Discovery and invocation both use that instance's `inst` route. This view grants no registry scan or class queue. It adds no command capability except the three reads a launch needs: a view whose actor holds `spawn` may call `status`, `models` and `resolve-cwd` on that instance, subject to the execution-mode limit above. A `run`-only or no-scope view may not, and `ps` and `slots` stay with `admin`. A user mesh's co-located manager answers every owner's view, so its `status` counts only the caller's own seats, as `ps` would list them, unless the caller holds `admin`. An absent, ambiguous or unauthorized selection refuses before the command is sent. An ambiguous one lists the owner's live manager instance ids and names the caller's way to pick one: `--on ` for the CLI, `--manager-instance ` for `cotal agent-bearer --manager-call`, and the `instance` argument for an agent's tools. A pin changes routing, not the caller's grant. Each `status.connectors` row includes `eventPlane: boolean`. The value is true only when the connector declares the event channel it publishes. Installed-extension rows carry the cached fact before the package is imported, and an old inventory without it is refused rather than treated as true. A client that reads a required event policy can use this field to disable an incompatible connector before spawn. The Manager still owns the launch refusal text and applies it at admission. Managed launches carry `COTAL_MANAGER_INSTANCE` so their tools address the manager that launched them; a tool's `instance` argument must name that same manager. The `cotal` commands run from their shell select it too, unless `--on` names another instance. Existing unbound sessions can use the exchange's unique authorized selection without replacing their actor or conversation. The connector uses a separate control connection; the standing message connection and its credential source are unchanged. Accepted spawn goals are followed on that renewing connection, using its existing caller-scoped progress grant, so a long readiness budget does not depend on the short-lived control credential. The follower confirms its progress subscription with the broker before submitting on the separate connection. A caller still checks the resolved instance and epoch, and never retries an ambiguous mutation outcome. The reference `cotal_spawn` tool retains the published request id when its submission wait ends before acceptance is observed. That id is the manager's goal id if the submission is accepted later. The tool reports an unknown outcome with the id and a same-caller `goal-result` read, without claiming that acceptance occurred. Do not repeat the spawn. An unanswered describe or a broker-attested no-responder refusal does not create a goal handle. A missing readback is inconclusive while the original submission may still arrive. ## Goal result reads The manager's `goal-result` command accepts `{goalId}` and returns `{goalId, result?}`. It reads only the authenticated caller's owner, actor and lifecycle through the manager's separate trusted goal-writer connection. The caller receives an attributed reply, never a raw JetStream reader grant. Each read is admitted by the connection's broker-enforced command grant. A live user-auth connection remains bounded by its bearer expiry after revocation; a renewed connection is checked against fresh authority. There is no separate per-read ledger check. An absent `result` means no terminal is recorded; it does not prove the goal is running or permit another submission. The existing trusted goal-writer's leader-served EPF read is space-wide at the broker; the handler confines it to this endpoint and caller triple. ## Goal cancellation The manager's reserved `cancel` command accepts `{goalId, mode?}` and returns `{goalId, state}`. It is served for a turn the manager relays. The goal is the authenticated caller's own, so a caller withdraws only a turn it submitted. The turn ends `cancelled`, its seat is not shown it again, and a later yield of it is answered with that terminal. A goal that already ended is refused `failed-precondition` with its cached outcome attached, and a goal this manager does not relay is refused without being changed. So is a second cancel that arrives while a first is still ending the turn; a first that fails leaves the turn pending unless something ended it meanwhile. A workflow run sends it for the turn, ask attempt or escalation of a branch it cancelled. ## Request outcomes A followed mutation requires a manager whose attributed describe includes `goal-result`. If it is absent, preflight refuses with `failed-precondition`, an upgrade notice and `not-executed` before submitting the mutation. Narrowed discovery preserves that refusal. Update the manager, issuer and client together before using that recovery path. Reloading an issuer alone cannot change an already-running participant manager. Recovery re-resolves the accepting instance's epoch, preserves the caller lifecycle and validates the result against the accepted goal and any acceptance fingerprint. Stopping the caller ends its observation, not the already accepted goal. A followed call resolves the endpoint within its deadline before the submission starts, so a refused or unanswered describe surfaces as its own error. On a user-auth mesh that preparation also obtains the caller's `manager-caller` view and opens its connection, and a failure there surfaces unchanged. The control connection uses the remaining preparation deadline, including time spent obtaining the bearer, rather than a separate two-second limit. A deadline that passes during those steps reports that the request was not run. The describe then spends what is left of the deadline, so a describe that draws no reply still reports itself as unanswered. A describe or command publish that the broker refuses reports `not-executed`. Cancellation before submission reports `not-executed`. Once submission starts, cancellation or a lost reply reports an unknown outcome unless an attributed refusal proves otherwise. A received refusal remains a refusal even when stop races it. Local failures do not invent responder identities. The follower owns its subscription, timers and read cancellation signal. Reconciliation begins before the wait deadline, and late read completions cannot settle an expired observation. Its read callback receives the accepting caller triple, remaining budget and abort signal; borrowed bearer commands and control connections use that signal. An in-flight dial that finishes after cancellation closes without publishing. A local reply-subscription failure prevents publication and is observed by the same request promise, including when the transport is closing or draining. Request cancellation does not revoke or resubmit the accepted operation. ## Split topology "Only one manager per space" is not the current invariant. A split topology that keeps the broker host manager-free is still a topology choice: `cotal up --no-manager` on that host starts no manager, and `cotal supervise --server` runs the manager elsewhere ([Run a mesh](/docs/run-a-mesh/)). A broker host that already runs a manager stops it only after `✓ manager up` in `.cotal/manager..log` (detach stdout listing `manager` is pidfile liveness, not a teardown boundary). A controller stack stops it with `cotal service stop stack-manager --space --yes`; a legacy stack has no command that stops only its manager. Extra live managers are addressable, not an error. ## Discovery deadlines The reserved `describe` bootstrap is the one request the resolver may repeat while waiting: it is read-only, it is re-published under the same request binding, and every attempt stays inside the original deadline. This covers the startup window where Core NATS discards the first request before the manager has subscribed. If the connection closes while the resolver waits, the describe fails cleanly instead of throwing from the retry timer. The resolved command is never repeated by this readiness behavior. The describe and the contract store reads after it share the resolver's one deadline, so a slow store read fails the resolve with `deadline-exceeded` when that deadline expires. `cotal invoke` passes its `--timeout` as that deadline (default 10000 ms) and gives the reply what the resolve leaves of it. `cotal describe` has no `--timeout` and passes 10000 ms. Cancelling a resolve settles a store read in flight instead of waiting for it to return. When one store read fails, the resolve fails with it and fetches no further command contracts; fetches already under way finish on their own. A call for one command fetches only that command's cluster and schema closures. `cotal invoke --name` also fetches `inspect`'s, because it resolves the target through `inspect`. `describe` still fetches the complete visible surface because its result lists every command. ## Instance mismatch The resolve and the invoke are separate trips through the same anycast queue, so in a multi-manager space an unpinned call can land on an instance the caller did not resolve. Every call carries the incarnation it resolved against, and a manager that is not that incarnation **refuses before running the command**, so the failure an operator sees says the command did not run, and re-issuing it cannot duplicate the effect. That is the difference that matters for a mutation: the older behaviour detected the mismatch on the reply, after the manager had already acted, and could only tell you to go and check. `--on` still matters for reaching a specific manager (`ps`, `stop`, `attach`, `spawn --detach`), but it is no longer what stands between a split and a duplicated spawn. Against a manager older than this fence the refusal is still after the fact, and its message says so. The re-issue is automatic only when the refusal states `not-executed` in its `outcome` field; a refusal that omits the field, or states `unknown`, is surfaced to the caller instead of repaired, because neither proves the command did not run. The CLI's manager commands, `cotal invoke`, the `cotal run` verbs and the manager row of `cotal status` re-describe and re-issue an unpinned call after each such refusal, up to 16 times, so a split reaches the operator only when every attempt split. A hosted run's own manager calls use the same bound. A pinned call is never re-issued. An agent's own manager tools, such as `cotal_spawn` and `cotal_despawn`, re-describe and re-issue with the same bound, including the goal-result read that follows a spawn to its outcome. An unpinned targeted call, such as `cotal_despawn` or a hosted run's turn relay, can also reach a manager that does not host its target, because each manager resolves targets against the agents it runs. That manager refuses with `expired` and `not-executed` and says it holds no mapping for the target, and the same re-issue repairs it within the same bound. An agent that no manager hosts still ends in that refusal once the re-issues run out. A pinned call gets the refusal of the instance it named. ## Manager availability A verified Manager `status` reply includes `host`, the answering process's host name, and `version`, the running `@cotal-ai/manager` package version captured when its module loaded. These fields are advisory display metadata. They do not select a Manager, authorize a caller or determine placement. Use the verified responder's `instanceId` for attribution. Clients must accept status payloads that omit either field and leave the missing value unknown. The optional fields advance the Manager cluster revision to 28. A manager whose boot inventory marked every declared connector unavailable does not subscribe `spawn` or `launch` on the class `one` rail. Those commands stay on scatter and on this instance's `inst` rail, so a sibling that can launch them can take an unpinned spawn, and a caller that pins this instance with `--on` still gets a named harness refusal. `describe` still lists the commands: the instance rail serves them, and `describe` itself stays on the class rail (SPEC 13.7). An unpinned `spawn` can therefore bind-fence: `describe` may land on the skip member while `spawn` lands on a sibling, the command was not run, and the caller re-issues or pins `--on`. `status` reports `classSpawn: false` when that skip is in effect. When no manager takes the class queue, the broker answers an unpinned `spawn` with no responders, and the caller classifies that `not-executed` (SPEC 13.3). A manager that can launch some connectors keeps the class rail. If the queue hands it a harness its inventory marked unavailable, the refusal names `--on` because the standing serve credential cannot read sibling inventories. Pin the capable instance (the whole id, as `ps` prints it). ## Scatter liveness `ps` and `status` become a **scatter** across every registered instance: the caller freezes the expected set from the service registry, invokes each under a shared deadline, and merges the results with per-instance attribution. A non-answering instance is labelled as registered with no answer within the deadline, never silently omitted. See [SPEC §13.5](/docs/spec/#135-verbs) (scatter) and [cli.md](/api-reference/cli/). The expected set comes from the **registry**, which records registration rather than liveness. An instance that crashes never deregisters, so it stays in the set and the gather has nothing left to wait for but an answer that cannot come. It pays the whole deadline, on every scatter, indefinitely. A scatter can therefore be given a per-instance liveness probe: when the broker itself reports that an instance holds no subscription on its own instance rail, the gather stops waiting for it. Only that affirmative report counts. A lapsed presence entry, a probe that timed out, and a probe that failed are all *absence of evidence*, and treating any of them as death would turn a slow correct answer into a fast wrong one, so they leave the full deadline standing. Nothing about the outcome changes either way: an instance that did not answer is still unreachable, still surfaced, and the scatter is still not complete. The probe is supplied by the **caller**, not invented by the scatter. Asking about an instance is a publish on that instance's rail, and a credential that holds no row for it is refused by the broker asynchronously, while the publish itself returns normally. The probe verb watches for that refusal and raises it as `permission-denied` naming the rail, so it is never mistaken for a quiet instance, and it never burns the probe budget waiting out a refusal. Only the layer that minted the credential knows which ids it may ask about, so that layer asks about those and no others. `cotal ps` freezes the class on its first connection, re-mints an instrument pinned only to the frozen ids, and scatters on a second; a refusal the broker raises anyway is printed and the instance's row says the probe was refused, which is a fact about the credential, not about the instance. This does not help against an instance that is **connected but not answering**. A hung manager holds its subscriptions, so it is indistinguishable from a slow one, and it still costs the full deadline. That is the correct result, not a gap in the probe. ## Deregistration A probe makes a dead registration cheap to skip; it does not remove it. Removal is the registration's own exit, and there are two explicit routes to it ([SPEC §13.5](/docs/spec/#135-verbs): a deleted `svc` spec *is* the deregistration). A manager that stops cleanly removes its own registration, so an ordinary shutdown leaves no stale row. The delete is pinned to the registration revision that process wrote. When a successor has registered the same instance since then, the stop logs that and leaves the successor's registration alone. It refuses that delete while this instance holds the endpoint governance slot at the live issuance-gate generation (a registration still completing its reopen). A leftover slot whose generation is behind that live generation is not in-flight and does not block the stop. A manager that cannot renew or read its lease keeps serving, stays registered, and retries. If another process holds the same instance key, that process has taken the instance over, so this one logs the conflict and exits without deregistering, leaving the successor's registration alone. A restart that died *mid-registration* is a different residue: the issuance gate stays frozen under that op. The successor completes the dead registration on boot when the freeze-holder is affirmatively gone under a complete CONNZ sweep (the same composition as [`cotal reconcile-gate`](/api-reference/cli/#reconcile-gate)). A committed spec write is finished under that same freeze; only a definite no-commit abort-reopens and then runs the normal takeover. Once that reopen commits, the registration has completed: a governance slot it then fails to release is left behind the live generation, as on a normal registration, and does not fail the restart. It does not invent a TTL and it does not start a new freeze over a still-held one. A registration's own reopen fails in one of two ways, and its `unavailable` error says which. A reopen that lost its compare-and-set wrote nothing: a reconciler or newer barrier moved the gate. A reopen that threw, such as one whose acknowledgement was lost, has an unknown outcome and may have committed. Either way, re-read the gate before retrying. That residue has a second half, and it is the endpoint governance slot rather than the gate. Every registration takes the endpoint-wide slot before it publishes its spec and holds it until its own gate reopens, which is what serializes registration for the endpoint. An instance that stopped between those two points leaves the slot held with no registration behind it, so the endpoint refuses new registrations while nothing is actually in flight. The slot is stamped with the generation of the gate its holder had frozen when it took it, and a slot is promoted only at that same generation. So once the holder's gate has reopened past the stamp, the slot can never be promoted by anyone, and the next registration for that endpoint replaces it. That reclaim is part of an ordinary start and needs no operator step. A slot whose holder's gate is still at the stamped generation is a registration that is genuinely in flight, and it keeps refusing. The two states read differently only in the holder's gate coordinate, so reopening that gate is what separates them: the holder's own restart heals it on boot, and [`cotal reconcile-gate`](/api-reference/cli/#reconcile-gate) is the operator's route when the boot path cannot run. The registration path is the slot's only writer, and neither repair command writes it. A registration that cannot read the holder's gate at all refuses, because an unreadable gate does not distinguish the two states either. Each of these refusals carries `kind = ai.cotal.ep.foreign-slot-held` in `error.details[]` with the holder's instance id and the `condition` that refused: `in-flight` for a holder gate still at the stamp, or `no-seam`, `unreadable`, `garbled` or `behind` when the registration could not read that gate or read it below the stamp. A remote manager asks its host to reconcile the holder only on `in-flight`, the one condition a gate repair can clear. For the instance that cannot cooperate, an operator names it: `cotal endpoint deregister ` ([cli.md](/api-reference/cli/#endpoint-deregister)). It removes the record only on the same evidence `cotal ps` acts on: the broker reporting nothing subscribed on that instance's own rail. It refuses if the instance answers a describe, refuses if the probe could not run at all, and refuses if the instance is merely quiet, because a hung process still holds its subscriptions and is therefore not affirmed gone. It also refuses while that instance holds the endpoint governance slot at the live issuance-gate generation (a registration still completing); a leftover slot behind that generation is not in-flight and does not block. Nothing sweeps the registry on an age threshold or on silence. An instance that is deregistered while it is merely wedged re-registers over the tombstone on its next start, which is what makes the operator's decision a recoverable one.