Create immutable Hydra jobsets for PR and manual CI runs #4

Merged

ams-tech merged 2 commits from codex/hydra-pr-manual into main

2026-09-28 16:00:19 -04:00

ams-tech commented

2026-09-28 00:15:57 -04:00

(Migrated from github.com)

The service uses systemd credentials and a separate dynamic user. Hydra administration credentials stay on Ace. Main's status and Cachix publication rules remain unchanged; run outputs are disposable and historical results stay visible. One rejected request does not block independent runs. Timestamp checks make trigger retries safe, and explicit API triggering avoids a child-reaping race in the deployed Hydra version's native one-shot mode.

Validation: all 32 Python tests and the full Hydra integration VM pass. Coverage includes admission, attempts, stale merges, pagination, credential handling, retry recovery, a successful real evaluation without an evaluator restart, and backup restoration. Both live PR and manual trials evaluated exactly ten ARM64 jobs once, all ten passed, and their GitHub aggregate gates passed. The evaluator remained stable throughout qualification. Ace's complete candidate test activation also passed; the temporary executable override has been removed.

Companions: https://github.com/PseudoDesign/kaiba-provisioning/pull/93 and https://github.com/PseudoDesign/nix-pseudo-design/pull/11.

PR and manually dispatched CI runs need Hydra jobsets pinned to their own commits. Add an Ace service that discovers running provisioning CI waiter jobs through GitHub's API, creates immutable `ci-<run-id>-<attempt>` jobsets, and triggers each evaluation once with automatic polling disabled. PR merge commits must match the run's head; manual requests must match the dispatched SHA. Existing jobsets cannot be repointed or re-enabled. The service uses systemd credentials and a separate dynamic user. Hydra administration credentials stay on Ace. Main's status and Cachix publication rules remain unchanged; run outputs are disposable and historical results stay visible. One rejected request does not block independent runs. Timestamp checks make trigger retries safe, and explicit API triggering avoids a child-reaping race in the deployed Hydra version's native one-shot mode. Validation: all 32 Python tests and the full Hydra integration VM pass. Coverage includes admission, attempts, stale merges, pagination, credential handling, retry recovery, a successful real evaluation without an evaluator restart, and backup restoration. Both live PR and manual trials evaluated exactly ten ARM64 jobs once, all ten passed, and their GitHub aggregate gates passed. The evaluator remained stable throughout qualification. Ace's complete candidate test activation also passed; the temporary executable override has been removed. - PR: https://github.com/PseudoDesign/kaiba-provisioning/actions/runs/36384748248 - Manual: https://github.com/PseudoDesign/kaiba-provisioning/actions/runs/36384755213 Companions: https://github.com/PseudoDesign/kaiba-provisioning/pull/93 and https://github.com/PseudoDesign/nix-pseudo-design/pull/11.