← Inside orch contents

Lab 04 - Rollout, Rollback and Modeled Availability

Atlas has three useful replicas. The fourth slot is deliberately temporary: a new version must earn a healthy report before the controller can ask an old replica to stop. In this lab you observe the real rollout controller and scheduler against MemoryService, then separate their arithmetic from application traffic. No Docker daemon, image pull, database or external endpoint is used. Agent reports are synthetic inputs, explicitly supplied by the teaching harness.

Read Chapters 10, 13, 14 and 17 first. Use the pinned source and Go 1.25 toolchain described in the lab environment. The complete teaching extension is additional book code, not a patched application. Its service and node IDs are generated by the unchanged store; captured IDs and UTC timestamps identify this particular execution. Logical names, action order, CPU quantities and assertions are fixed. Do not describe the random identifiers or elapsed timings as deterministic.

Inputs and Cleanup

The baseline creates one node with CPU 300 milli-units and memory 1 GiB. Three service tasks each request CPU 100 and memory 1 MiB. Their image labels are nginx:1.27-alpine and nginx:1.28-alpine, but neither image is pulled. A restart-never policy keeps injected failure distinct from automatic replacement. The rollout uses maxUnavailable zero and maxSurge one. Health is an explicit report, not inferred from running.

Before running, verify all copied baseline files against source-lock.json. Copy the extension into .book-labs/rollout inside a disposable Orch module. Keep the original snapshot untouched. The process owns only in-memory objects and its selected CSV output. Cleanup is process exit and, when no longer needed, removal of that exact copied extension directory. Never remove a real service or container to imitate a synthetic failure.

From that disposable module, with the explicit pinned GOROOT, caches and GOTOOLCHAIN=local selected:

GOFLAGS=-buildvcs=false go run ./.book-labs/rollout /tmp/inside-orch-rollout-timeline.csv

The command writes a CSV and prints a JSON observation document. Choose an output file owned by this run. Save both with the actual command, tool version, source SHA, UTC times and exit code. The author's recorded command, raw observations and timeline are existing execution evidence; reviewing them is different from executing a new run.

Predict Each Pass

First predict whether a pass creates, waits, stops, fails or completes. The first controller pass changes the deployment from pending to running and creates one pending target. That fills the modeled four-task ceiling. The scheduler cannot assign it because the three original requests consume all advertised CPU. An additional controller pass must retain the same target rather than create a fifth task.

The fixture then explicitly changes advertised node capacity to CPU 400. This is a controlled input change, not a claim that physical hardware expanded. The scheduler assigns the pending target. A running report still leaves all three old replicas active: running is insufficient for target readiness. A healthy report authorizes one stop directive, leaving three modeled available tasks. The harness acknowledges each old stop with stopped then removed before advancing the next cohort.

The captured baseline has seven checkpoints: requested; target-created; capacity-exhausted; running-but-not-healthy; and three healthy-controller passes. For each, inspect old_active, target_active, target_healthy, pending_stop, modeled_available and surge_bound. Add old_active and target_active: the sum must not exceed four. Whenever old_active decreases, enough targets must already be healthy to satisfy the stop gate. Do not substitute the count of all stored historical task objects for active work.

The final observed checkpoint is succeeded while one final stop directive is still unacknowledged. That is an especially useful boundary. The active predicate excludes a task whose desired status is stopped, although its last actual report may still be healthy. Deployment success here is the controller's model; it is not proof that Docker removed every old container. The harness deliberately retains that observation rather than retroactively calling it physical cleanup. There were no containers in this track.

Failure and Conflicting Operations

The harness repeats an identical active rollout and asserts the same deployment ID. A changed image conflicts. Scale during the rollout conflicts, and an active rollback also conflicts. These outcomes are actual MemoryService operation checks, not invented CLI force flags. A stale deployment status write using the original UpdatedAt is rejected after the controller advances the object. That is the store's optimistic concurrency guard. Agent task reports have no equivalent caller-supplied timestamp field; the separate late healthy report tests terminal-state rejection instead.

A fresh fixture creates a target, assigns it explicitly and reports failed. This assignment is an injection seam, not scheduler proof of available capacity. The controller marks the deployment failed. A later healthy report cannot revive that failed task. Rollback creates a distinct rolling_back deployment from version two to version one and preserves the failed deployment's ID, completion timestamp and status. Version one qualifies through the initial-baseline special case; no previous version on an untouched service produces invalid state. A separate fixture verifies that delete cancels active deployment work by marking it failed.

Compare these observations with the unchanged operation_lock_test.go, rollback_test.go and rollout_test.go source. The executed package selection also includes these fixtures. Maintenance windows and an audited emergency bypass are separate policy checks in operations_test.go. The passing TestMaintenanceWindowAndForcedBypass does not make force a normal rollout requirement. Do not invent pause/resume, an automatic rollback or a progress deadline: the baseline does not provide those controls.

Worked Questions

  1. Why does full capacity wait safely? The controller has already consumed the one surge slot with a pending task. The scheduler finds no CPU fit. No new healthy target exists, so the old-task stop gate refuses progress. Safe waiting may be indefinite; an operator must diagnose capacity or health rather than assume a deadline will recover it.
  2. Does maxUnavailable zero prove zero downtime? No. It constrains reported task counts in this implementation. A traffic claim additionally needs owned endpoints, a real probe stream, timestamp resolution, sampling intervals, response criteria and routing convergence observations. None were collected here.
  3. Why retain failed history when rolling back? It preserves the failed decision and its terminal evidence while a new deployment owns recovery. Rewriting the failed record to success would destroy the distinction between the incident and its response.
  4. What survives a controller restart? Reconstructing a controller over the same store retains objects. A fresh default server constructs a new MemoryService and loses them. Durable restart requires a separately integrated persistent store and new-process test; neither an in-memory timeline nor the existence of PostgreSQL migration files establishes that composition.

The machine acceptance record checks the captured budget, health gates, negative operations and terminal history. It certifies this controlled execution scope, not external availability or production readiness.