Inside orchOnline edition English
PDFEPUB
orch Appendix B43

Appendix B

7 min read Section 43 of 47

Appendix B - Lifecycle States and Invariant Reference

A lifecycle reference answers two separate questions: whether an edge is valid, and who is allowed to request it. The server owns desired task state. Assigned agents report actual state through an authorization and transition boundary. Node monitors and policy logic can also change task records for explicit reasons. A state name alone never grants ownership, and a condition such as node_lost is not a replacement state machine.

The tables below were generated by actually invoking the pinned public lifecycle validators in a disposable source copy. They include same-state acceptance separately, making repeated observations visible without pretending they are new transitions. Their confidence is the local helper contract: constructor execution and finite state checks do not prove a distributed execution will converge under every failure.

Allowed transitions

Service

From Different allowed destinations Duplicate
active deleting accepted
deleted none accepted
deleting deleted accepted

Task actual

From Different allowed destinations Duplicate
assigned pulling, created, running, healthy, unhealthy, stopping, stopped, removed, failed accepted
created starting, running, stopping, stopped, removed, failed accepted
failed removed accepted
healthy running, unhealthy, stopping, stopped, removed, failed accepted
pending assigned, stopped, removed, failed accepted
pulling created, running, stopping, stopped, removed, failed accepted
removed none accepted
running healthy, unhealthy, stopping, stopped, removed, failed accepted
starting running, stopping, stopped, removed, failed accepted
stopped removed accepted
stopping stopped, removed, failed accepted
unhealthy running, healthy, stopping, stopped, removed, failed accepted

Task desired

From Different allowed destinations Duplicate
assigned none accepted
created none accepted
failed none accepted
healthy none accepted
pending none accepted
pulling none accepted
removed none accepted
running stopped, removed accepted
starting none accepted
stopped removed accepted
stopping none accepted
unhealthy none accepted

Node

From Different allowed destinations Duplicate
draining ready, offline accepted
offline ready accepted
ready draining, offline accepted
unknown ready, offline accepted

Deployment

From Different allowed destinations Duplicate
failed none accepted
paused running, failed accepted
pending running, succeeded, failed, rolling_back, rolled_back accepted
rolled_back none accepted
rolling_back paused, failed, rolled_back accepted
running paused, succeeded, failed accepted
succeeded none accepted

Desired-state construction normally uses running, stopped and removed. The generic desired validator first accepts any identical known TaskStatus values; therefore its matrix includes identity edges for other enum values. That is helper behavior, not permission for an agent to introduce those desired states. Starting and stopping are also recognized actual enums, but the ordinary agent report whitelist does not admit them. Their presence reserves lifecycle vocabulary without inventing a runtime progress feature.

Service deleted is terminal: restoring an old deleted ID to active is forbidden. Task failed and stopped are terminal for execution, but the generic actual-state validator permits cleanup to removed. Removed cannot leave itself. Deployment succeeded, failed and rolled_back are immutable terminal histories; rollback creates another deployment with its own ID and rolling_back state. Paused is accepted by the enum/validator but lacks an operator pause/resume CLI feature at this revision.

The agent guard narrows the generic task matrix. An actual failed task with desired running cannot report removed merely because ValidateTaskTransition permits that edge; AgentCanReportTaskStatus accepts only a duplicate terminal report in that branch. Once desired state is stopped or removed, the cleanup branch permits stopped, removed or failed. An already removed task still accepts only removed. This order matters when interpreting a late worker message.

Jobs and cron are separate mechanisms

Jobs have pending, running, succeeded and failed states. They do not call a generic ValidateJobTransition helper: MemoryService derives their state from associated task reports. Running or healthy work marks a job running. A stopped task uses an explicit exit code when available and otherwise zero; zero means succeeded. A failed task defaults its missing exit code to one. Retry budget checks compare Attempts to BackoffLimit before increasing attempts and creating a new pending task identity.

These rules must not be inferred from the service restart state machine. A retry creates another task; it does not resurrect a terminal task in place. Cron records scheduling metadata such as LastScheduleAt and LastJobID rather than a lifecycle enum equivalent to deployment status. Allow, Forbid and Replace govern overlapping jobs; occurrence selection, catch-up and clock behavior belong to the cron controller. A suspended cron entry does not imply its already running job was canceled.

Invariant index and scope

ID Invariant Enforcing boundary Counterexample outside scope
INV-01 Replica convergence reconcileServiceTasksLocked Zero capacity or restart never can leave intent unfulfilled; initial immediate placement bypasses richer Plan constraints.
INV-02 Atomic task claim AssignTask One-winner claim does not establish default server persistence or remote process fencing.
INV-03 State-change event ordering ReportTaskStatus Memory mutation and append are not a durable transaction across process failure; do not generalize to every controller append.
INV-04 Assigned node owns reports ReportTaskStatus Credential validity alone cannot authorize a different node’s task; physical old-worker activity is not fenced.
INV-05 Namespace secrecy GetService A global identifier must not be returned before scope validation; inspect every newer workload surface separately.
INV-06 Rollout availability budget reconcileDeployment Running new tasks alone do not satisfy newHealthy; concurrent physical failure can invalidate a snapshot budget.
INV-07 Owned container cleanup removeAssignedTask Server deleting status is not proof removal happened; cleanup can fail or old agent can remain unreachable.

Each companion invariant records assumptions, a precise source symbol and either a real test name or an explicit test gap. An inspected test is not labeled as a new run. Replica convergence requires an active service, an applicable restart policy, capacity and eventually successful operations. Atomic claim safety establishes one successful assignment transition for one pending record; it does not fence a worker that lost contact after starting a process.

Event emission follows validated state changes in the memory handler, and duplicate terminal reports avoid duplicate state-change events. This is not durable atomicity between a database commit and an external event stream. The store-backed reconciler has different transaction/error windows and requires its own constructor and backing store. Default process restart destroys MemoryService intent even if a PostgreSQL package test previously passed.

Namespace secrecy is a lookup and return-value property, not a naming convention. An implementation must verify context before returning data associated with a guessed identifier. Safe cleanup similarly depends on managed, node and task labels, not just a user-provided container name. Neither invariant should be extended to new handlers or runtime adapters without tracing those checks again.

Three source-derived traces

Trace 1: a repeated healthy report. Preconditions: the task belongs to the authenticated node, DesiredStatus is running, ActualStatus is healthy, and its service restart policy need not act. (1) The agent submits healthy again. (2) Node/task ownership is verified. (3) The guard accepts the identity edge. (4) The control plane retains healthy without creating another state-change event. A new health observation may carry metadata; it must not be counted as a new lifecycle transition. The companion validator harness observed guard acceptance; full handler event behavior is established by the cited source test.

Trace 2: deletion during an active rollout. Preconditions: an active service has an in-progress deployment and running tasks. (1) DeleteService marks the service deleting. (2) Desired task state changes to stopped. (3) A late healthy report is rejected by the cleanup-intent guard. (4) The assigned agent stops and removes its owned container, then reports removed. (5) Finalization requires every associated actual task to be removed before the service is deleted. A previously captured rollout snapshot cannot make that service active again. Offline workers can leave deleting visible indefinitely; failed or stopped alone does not satisfy the finalization predicate. This is a reconstructed path, not a new Docker observation.

Trace 3: an offline stateless versus stateful task. Preconditions: two services differ by the Stateful flag and both have tasks on a node whose nonzero heartbeat is older than the monitor cutoff. (1) The node becomes offline. (2) Tasks receive a node_lost condition independently of their status. (3) Stateless work is failed with desired removed so replacement intent can be created elsewhere. (4) Stateful work retains assignment for manual recovery rather than pretending local data moved. (5) A later authenticated worker response must still satisfy current ownership and cleanup rules. Neither branch physically stops a partitioned process; there is no remote fencing token enforced by Docker. This trace describes source behavior and explicitly leaves split-brain prevention unproven.

Using the matrix in reviews

When adding a state, enumerate its owner, outgoing edges, duplicate behavior, cleanup path and every consumer switch. Start with an invalid representative edge such as failed to healthy, then inspect whether a higher-level policy replaces the task instead. Review stale reports against both current desired state and authenticated assignment. A valid generic edge can remain forbidden at the agent boundary, and a valid store transition can remain unused by the shipped binary.

The matrix is a finite reference for finding missing cases, not a formal proof of distributed correctness. Its most useful review question is concrete: after this message is delayed, retried or delivered to a restarted process, which identity and owner still exist to authorize it?

Source: lifecycle validators and report guard, transition tests, MemoryService transitions, agent ownership/cleanup tests, job bookkeeping, rollout controller, and store claim tests.

Source-backed boundary statements for this appendix are indexed in the appendix claim record. Detailed command, state, exercise and reference tables retain their own source provenance.

Aleksandar Popovic43 / 47

Aleksandar Popovic · Copyright © 2026 Aleksandar Popovic · All rights reserved. Licensing and attribution