Lab 03 — Reconciliation and ownership under drift

Atlas has three separate questions: how many task records should exist, which local containers belong to this agent, and whether the control plane will accept the agent's next status report. Predict each answer before running the fixtures. A container that disappeared is not sufficient permission to reset a running task to pulling.

This lab is an original teaching extension against source cbfc94ade5b57e449d6b73bf1485b1c5b1647022. Its observed track uses upstream FakeRuntime, the actual in-process MemoryService lifecycle, and the separate store-backed reconciler with its upstream test store. It uses no HTTP listeners, Docker daemon, PostgreSQL database, or external images. Source files copied for private-package access remain byte-identical; the only additions are two test files. The shipped server does not start the store-backed reconciler.

Predict the two passes

Start with the five intent cases in fixtures/cases.json. Write the expected new task count, ordered stop directives, service status, and transition-event count after two unchanged passes.

The last case then receives explicit removed-task observations in the store fixture. Only the next pass marks the service deleted. This fixture is not a physical runtime acknowledgement transport. It isolates the reconciler's decision after that observation becomes available.

The exact stop ordering prioritizes lifecycle state, then descending task ID. Desired stopped tasks stop counting as active immediately; this does not prove that their physical containers have already stopped. Deletion logic has the stronger terminal removed requirement.

Run the isolated fixtures

Prerequisites are the locked Go 1.25.0 toolchain, its cached dependencies, Python 3, and a disposable copy of the locked application source. Do not execute against evidence/orch. The runner verifies all 361 baseline file hashes before making its two private .book-labs/lab03* package overlays. It never edits baseline files. Set the paths to your actual source copy and Go installation; the example values describe the generation environment.

python3 books/inside-orch/scripts/run_recorded.py \
  --id lab03-reader-run \
  --cwd /home/popac/alekpopovic.github.io \
  --output validation/labs/lab03-reader-command.json \
  -- python3 books/inside-orch/scripts/run_drift_lab.py \
  --source-copy /tmp/inside-orch-labs-cbfc94ade5b5/orch \
  --go-root /tmp/inside-orch-gomod/golang.org/toolchain@v0.0.1-go1.25.0.linux-amd64 \
  --output-prefix validation/labs/lab03-reader-observed

Use fresh output names: existing evidence is deliberately not overwritten. run_recorded.py captures UTC start/end, command, working directory, exit code and raw stdout/stderr. run_drift_lab.py copies unchanged agent/reconciler package and test files, adds the original tests, and executes only TestBook* with go test -count=1. The runner's fixed cache paths are generation defaults; adapt those environment locations for another host. The canonical tests contain executable fixture setup; the JSON fixture file describes and checks the exact case ID set.

Observe lifecycle and ownership together

The ten agent cases all include an unmanaged container with a similar name and a managed container belonging to another node. Both must survive each of the two passes, including all cleanup cases. Matching uses labels, never name similarity. Exact ownership predicates distinguish adoption from cleanup eligibility: matching additionally requires service, version and image consistency; cleanup eligibility requires managed, matching node and matching task identity.

The precise UUID-bearing rejection is retained per case in the actual check. Its stable suffix is cannot accept agent status "pulling". The running task remains running in control-plane state. This is a passing test of an observed failure boundary, not a successful recovery claim. Upstream agent unit tests use a more permissive fake client; connecting the unchanged agent to actual MemoryService.ReportTaskStatus exposes the invalid running-to-pulling transition.

Inspect currentTaskContainer, then ensureTask. Ordering matters: cleanup of an owned mismatch can occur before the later rejected lifecycle report. The lab leaves that result visible rather than modifying the agent or allowing an illegal transition. A second unchanged pass must add no transition events; repeated drift observations and rejected report attempts may still occur and are not transition-event duplicates.

Optional reader Docker track

This track was not executed for this lab. Use a disposable host, inspect socket authority and docker info, and read the pinned runtime integration test before opting in. From the disposable application copy, the actual gate and test target are:

ORCH_DOCKER_INTEGRATION=1 go test -count=1 -run '^TestDockerRuntimeIntegration$' ./internal/docker

The daemon can grant host-level authority. Confirm the test's ownership labels, capture each created container ID, and retain the cleanup result. Clean only those captured IDs or an equally strict lab-owned label set. Never run broad container deletion or target another node's containers. A Docker runtime test still does not run this lab's MemoryService/agent lifecycle comparison; the fake track and daemon track establish different facts. The book's separately recorded real Docker flow is in Lab 01, with its own captured IDs and cleanup.

Explain the boundary

Explain why changing only the fake client would hide the most useful finding. Propose a separate recovery design with legal lifecycle transitions, retry behavior and failure ownership; label it design work until tested. Do not patch the immutable baseline. Cleanup for this track is process exit plus the two owned overlay directories; it creates no external containers or database objects.