Record bounded Ace NVMe power-loss and offline tests #95

Merged

ams-tech merged 13 commits from codex/offline-qualification-evidence into main

2026-10-01 12:18:27 -04:00

ams-tech commented

2026-09-29 00:50:54 -04:00

(Migrated from github.com)

The kit includes a pinned Pi 5 ARM64 image with a read-only recovery system, separate synthetic test state, an exact-drive guarded writer with full readback verification, and console/SSH operations for clock refusal, retained SPIRE identity, Fleet revocation crash boundaries and backup restoration. Independent receipts on Malak distinguish an acknowledged revocation from an interrupted transaction. Old-backup detection uses an external expectation and does not claim hardware rollback prevention.

The earlier inventory, temporary SPIRE smoke and persistent-pilot observations remain as historical evidence. This new image contains synthetic identities and no live pilot private state. The operator programmed the spare with full image readback and completed the bounded physical campaign on Ace. Both PoE crash boundaries, retained identities, synthetic backup/restore, external stale-backup detection, and offline clock refusal followed by online recovery passed. Both planned swaps are complete, with no reflash round trip. Ace returned to its original encrypted NVMe and booted generation 14; all 17 protected services, retained identity/state, current workload credentials, authenticated device access, 12 Ace/Mako DNS queries and boot-order/tmpfiles checks passed. Mako renewed its node and recovered its scheduled workload probe automatically; Malak remains fenced.

Validation:

Native ARM64 image build; complete image checksum, partition/filesystem/ownership and firmware-kernel-initrd checks.
Exact image booted with its ARM64 kernel in QEMU using a disposable NVMe overlay and no network; initialization refused without synchronized time. Pi EEPROM is not emulated.
Seven operation-guard tests and 28 existing CI tests passed.
Real-service campaign exercised both transaction boundaries, retained keys, current and obsolete restores, plus 11 lifecycle and 17 same-key recovery/cutover scenarios.

Full hardware qualification remains false. Spare-drive durability will not qualify the original pilot drive, protected keys, secure boot, rollback prevention or trusted offline time.

Operator runbook: deploy/nvme-qualification/README.md. Build evidence: docs/observations/2026-09-30-nvme-qualification-preparation.json. Physical observations: docs/observations/2026-09-30-nvme-physical-qualification.json. One physical cut was performed per transaction boundary; exact cut timing and duration were operator-controlled, not independently instrumented.

Stacked on #94. Companion plan: https://github.com/PseudoDesign/kaiba-infra/pull/6. Fleet runtime remains pinned to 0bd55c576536c29825aada2f7ce6fa052a877402.

Prepare an isolated NVMe installation and operator workflow for Ace's pending offline-start, interrupted-write and restore tests. Program a spare on Malak, keep the original pilot NVMe disconnected throughout the campaign, then restore it: two planned swaps, with an extra reflash round trip only if the test disk becomes unbootable. The kit includes a pinned Pi 5 ARM64 image with a read-only recovery system, separate synthetic test state, an exact-drive guarded writer with full readback verification, and console/SSH operations for clock refusal, retained SPIRE identity, Fleet revocation crash boundaries and backup restoration. Independent receipts on Malak distinguish an acknowledged revocation from an interrupted transaction. Old-backup detection uses an external expectation and does not claim hardware rollback prevention. The earlier inventory, temporary SPIRE smoke and persistent-pilot observations remain as historical evidence. This new image contains synthetic identities and no live pilot private state. The operator programmed the spare with full image readback and completed the bounded physical campaign on Ace. Both PoE crash boundaries, retained identities, synthetic backup/restore, external stale-backup detection, and offline clock refusal followed by online recovery passed. Both planned swaps are complete, with no reflash round trip. Ace returned to its original encrypted NVMe and booted generation 14; all 17 protected services, retained identity/state, current workload credentials, authenticated device access, 12 Ace/Mako DNS queries and boot-order/tmpfiles checks passed. Mako renewed its node and recovered its scheduled workload probe automatically; Malak remains fenced. Validation: - Native ARM64 image build; complete image checksum, partition/filesystem/ownership and firmware-kernel-initrd checks. - Exact image booted with its ARM64 kernel in QEMU using a disposable NVMe overlay and no network; initialization refused without synchronized time. Pi EEPROM is not emulated. - Seven operation-guard tests and 28 existing CI tests passed. - Real-service campaign exercised both transaction boundaries, retained keys, current and obsolete restores, plus 11 lifecycle and 17 same-key recovery/cutover scenarios. Full hardware qualification remains false. Spare-drive durability will not qualify the original pilot drive, protected keys, secure boot, rollback prevention or trusted offline time. Operator runbook: `deploy/nvme-qualification/README.md`. Build evidence: `docs/observations/2026-09-30-nvme-qualification-preparation.json`. Physical observations: `docs/observations/2026-09-30-nvme-physical-qualification.json`. One physical cut was performed per transaction boundary; exact cut timing and duration were operator-controlled, not independently instrumented. Stacked on #94. Companion plan: https://github.com/PseudoDesign/kaiba-infra/pull/6. Fleet runtime remains pinned to `0bd55c576536c29825aada2f7ce6fa052a877402`.