Alex/kaido-core
SHA256
Watch
1
0
Fork
You've already forked kaido-core
 0
Code Issues 10 Pull requests 1 Releases Activity Actions

Fix installed startup diagnostics and safe recovery progression #70

Merged
Alex merged 5 commits from issue-28-startup-recovery into issue-65-doltlite-poc 2026-11-05 20:42:24 +01:00
Conversation 2 Commits 5 Files changed 67 +6263 -279
Alex commented 2026-11-05 18:54:01 +01:00
Owner

Problem

  • Installed startup failures could remove management access, prevent safe retries, or strand restore activation after releasing the native-service fence.
  • Configuration-only restore could be rejected because native leases were incompatible. WireGuard bootstrap DNS and management/proxy rollback could also prevent recovery.
  • Native QEMU acceptance additionally exposed invalid ownership/modes on restored device-local TLS files and repeated publication of already durable restore files across authentication normalization.

Changes

  • Preserve authenticated administrator diagnostics and explicitly configured listeners while real startup/readiness remains blocked.
  • Serialize bounded retries of eligible startup phases, persist final restore activation before fence release, and drain startup work before shutdown.
  • Repair confirmed management dependencies additively, including verified managed DHCPv4 renewal; preserve interface snapshots and HA ownership. Bound WireGuard DNS lookups while retaining protection.
  • Separate native lease eligibility from configuration restore validity; preserve structural and checksum validation. Preflight deployment before replacement and reject offline management changes conflicting with recovery journals.
  • Preserve the target's reserved device-local TLS identity and its root-helper trust boundary. Keep archived identity bytes accepted and verified; custom TLS files continue to restore. Resume durable awaiting_restart without republishing archived files.
  • Update OpenAPI and the proposed ADR. Accompanying EN/DE site documentation remains local under repository policy. No new configuration or TOML fields.

Validation

  • Reviewed revision: d3aaca2967f656244151f6d652b9f4c83ea26bd2b388791eb928f7bfee8c6da3.
  • Full make check GO_TEST_FLAGS=-timeout=30m passed: race tests, vet/staticcheck, all 442 UI tests, Svelte/TypeScript and non-CGO checks. Only the cumulative timeout allowance changed.
  • Full make check-build passed on the exact final source manifest, including required Alpine integration with no skips.
  • Behavioral regressions reproduce the corrected failures against prior source and pass after the fixes. The existing restart fixtures now accurately model unpublished files; assertions are unchanged.
  • Installed Alpine QEMU/KVM, Alpine 3.24.3 / kernel 6.18.58, Kea 3.0.4 / Knot Resolver 6.5.1:
    • Uninstrumented release-agent API configuration-only restore + reboot preserved all three current leases, including the lease absent from the older archive; restored hostname, DNS, management and real readiness; preserved local TLS identity.
    • Older native snapshot was refused specifically for conflicting current authoritative allocations without mutation. Eligible native lease restore + reboot passed.
    • Actual QEMU process power cuts before fence release, after release/before activation, and after activation/before completion all recovered on the same overlay. The last case retained a fourth real lease acquired after activation.
    • A completed native control-socket timeout kept readiness and mutations at 503 with activation_pending, then recovered automatically with the same agent PID/UID and boot ID. Kea itself restarted during retry.
    • Real root TLS identity engine/journal regression passed in an isolated guest mount namespace. Earlier native WireGuard, DHCP renewal and diagnostic smoke evidence remains scoped to the revisions recorded in those reports.
  • Independent static review covers code, regressions and recorded evidence; final formal review approved this exact head. Task VM resources, overlays, binaries and private credentials were removed; reusable base checksum unchanged.

Scope and limits

  • Crash checkpoints used a same-source optimized symbol-bearing executable and a temporary fixture GDB start command with the normal service UID/capabilities. Its machine code differs from the stripped release executable; the normal restore controls separately used the uninstrumented release agent. No product fault hooks were added.
  • Only the newly acquired native lease file was explicitly fsynced in the post-activation crash case; the product restore journal was not manually flushed.
  • Repeatedly killing Kea during OpenRC start interrupts the service transition itself. It remained safely blocked in starting and recovered with intact leases after reboot; automatic recovery is not claimed for that case.
  • This is installed Alpine QEMU acceptance, not validation on physical/remote project routers or clustered HA peers. Those targets were not accessed. The PR remains a draft; no merge, auto-merge or issue closure is authorized by this test run.

Refs #28

## Problem - Installed startup failures could remove management access, prevent safe retries, or strand restore activation after releasing the native-service fence. - Configuration-only restore could be rejected because native leases were incompatible. WireGuard bootstrap DNS and management/proxy rollback could also prevent recovery. - Native QEMU acceptance additionally exposed invalid ownership/modes on restored device-local TLS files and repeated publication of already durable restore files across authentication normalization. ## Changes - Preserve authenticated administrator diagnostics and explicitly configured listeners while real startup/readiness remains blocked. - Serialize bounded retries of eligible startup phases, persist final restore activation before fence release, and drain startup work before shutdown. - Repair confirmed management dependencies additively, including verified managed DHCPv4 renewal; preserve interface snapshots and HA ownership. Bound WireGuard DNS lookups while retaining protection. - Separate native lease eligibility from configuration restore validity; preserve structural and checksum validation. Preflight deployment before replacement and reject offline management changes conflicting with recovery journals. - Preserve the target's reserved device-local TLS identity and its root-helper trust boundary. Keep archived identity bytes accepted and verified; custom TLS files continue to restore. Resume durable `awaiting_restart` without republishing archived files. - Update OpenAPI and the proposed ADR. Accompanying EN/DE site documentation remains local under repository policy. No new configuration or TOML fields. ## Validation - Reviewed revision: `d3aaca2967f656244151f6d652b9f4c83ea26bd2b388791eb928f7bfee8c6da3`. - Full `make check GO_TEST_FLAGS=-timeout=30m` passed: race tests, vet/staticcheck, all 442 UI tests, Svelte/TypeScript and non-CGO checks. Only the cumulative timeout allowance changed. - Full `make check-build` passed on the exact final source manifest, including required Alpine integration with no skips. - Behavioral regressions reproduce the corrected failures against prior source and pass after the fixes. The existing restart fixtures now accurately model unpublished files; assertions are unchanged. - Installed Alpine QEMU/KVM, Alpine 3.24.3 / kernel 6.18.58, Kea 3.0.4 / Knot Resolver 6.5.1: - Uninstrumented release-agent API configuration-only restore + reboot preserved all three current leases, including the lease absent from the older archive; restored hostname, DNS, management and real readiness; preserved local TLS identity. - Older native snapshot was refused specifically for conflicting current authoritative allocations without mutation. Eligible native lease restore + reboot passed. - Actual QEMU process power cuts before fence release, after release/before activation, and after activation/before completion all recovered on the same overlay. The last case retained a fourth real lease acquired after activation. - A completed native control-socket timeout kept readiness and mutations at 503 with `activation_pending`, then recovered automatically with the same agent PID/UID and boot ID. Kea itself restarted during retry. - Real root TLS identity engine/journal regression passed in an isolated guest mount namespace. Earlier native WireGuard, DHCP renewal and diagnostic smoke evidence remains scoped to the revisions recorded in those reports. - Independent static review covers code, regressions and recorded evidence; [final formal review approved this exact head](https://git.knoll-systems.de/Alex/kaido-core/pulls/70#issuecomment-679). Task VM resources, overlays, binaries and private credentials were removed; reusable base checksum unchanged. ## Scope and limits - Crash checkpoints used a same-source optimized symbol-bearing executable and a temporary fixture GDB start command with the normal service UID/capabilities. Its machine code differs from the stripped release executable; the normal restore controls separately used the uninstrumented release agent. No product fault hooks were added. - Only the newly acquired native lease file was explicitly fsynced in the post-activation crash case; the product restore journal was not manually flushed. - Repeatedly killing Kea during OpenRC start interrupts the service transition itself. It remained safely blocked in `starting` and recovered with intact leases after reboot; automatic recovery is not claimed for that case. - This is installed Alpine QEMU acceptance, not validation on physical/remote project routers or clustered HA peers. Those targets were not accessed. The PR remains a draft; no merge, auto-merge or issue closure is authorized by this test run. Refs #28
kaido-reviewer approved these changes 2026-11-05 19:10:49 +01:00
Dismissed
kaido-reviewer left a comment

Static review

  • Reviewed implementation, regression tests, and existing validation evidence for commit 06cd258230e70ed9a8fb338cd8533978c627a7f3fdc76800dfc39eeab75e3d30, tree c6e8676021fb376bf9b9512a041b495ea199f39332c090c5f4d3af669972c1cc.
  • No remaining substantive code findings. Earlier findings concerning native archive structure, stale listener identities, completed native timeouts, retry shutdown, unrelated proxy dependencies, and running-client DHCP renewal are addressed.
  • The reviewer executed no tests, builds, browser sessions, or router actions.

Validation evidence

  • Final make-check-passed.log records Go race tests, vet, staticcheck, all 442 UI tests, Svelte checks with zero errors/warnings, and non-CGO tests/builds passing.
  • Final make-check-build-approved.log passes the build integrations; the accompanying 2,502-file manifest binds the exact source snapshot.
  • Scoped regression evidence includes expected failures against previous behavior. Native evidence covers WireGuard guards on the reserved isolated VM and Alpine kernel, real packaged udhcpc renewal restoring a removed address with the same pinned client and a new callback, and the separately scoped installed-diagnostics agent smoke.

Acceptance limits

  • The native evidence does not establish the full backup-restore/reboot/crash lifecycle or acceptance on the other routers. These limits remain tracked by issue #28.
  • This approval covers the reviewed code and stated evidence. The PR remains a draft; landing still requires the user's explicit approval. No merge is authorized by this review.
## Static review - Reviewed implementation, regression tests, and existing validation evidence for commit `06cd258230e70ed9a8fb338cd8533978c627a7f3fdc76800dfc39eeab75e3d30`, tree `c6e8676021fb376bf9b9512a041b495ea199f39332c090c5f4d3af669972c1cc`. - No remaining substantive code findings. Earlier findings concerning native archive structure, stale listener identities, completed native timeouts, retry shutdown, unrelated proxy dependencies, and running-client DHCP renewal are addressed. - The reviewer executed no tests, builds, browser sessions, or router actions. ## Validation evidence - Final `make-check-passed.log` records Go race tests, vet, staticcheck, all 442 UI tests, Svelte checks with zero errors/warnings, and non-CGO tests/builds passing. - Final `make-check-build-approved.log` passes the build integrations; the accompanying 2,502-file manifest binds the exact source snapshot. - Scoped regression evidence includes expected failures against previous behavior. Native evidence covers WireGuard guards on the reserved isolated VM and Alpine kernel, real packaged udhcpc renewal restoring a removed address with the same pinned client and a new callback, and the separately scoped installed-diagnostics agent smoke. ## Acceptance limits - The native evidence does not establish the full backup-restore/reboot/crash lifecycle or acceptance on the other routers. These limits remain tracked by issue #28. - This approval covers the reviewed code and stated evidence. The PR remains a draft; landing still requires the user's explicit approval. No merge is authorized by this review.
kaido-reviewer approved these changes 2026-11-05 20:09:48 +01:00
kaido-reviewer left a comment

Independent static review

  • Approved exact revision d3aaca2967f656244151f6d652b9f4c83ea26bd2b388791eb928f7bfee8c6da3. No blocking findings remain after reviewing product changes, regression tests and recorded validation. The reviewer executed no tests, builds or router actions.
  • Reserved target-local TLS identity is preserved while custom TLS restoration and archive integrity checks remain active. Durable restart checkpoints no longer replay already-published configuration over authentication normalization.

Validation evidence

  • Full make check GO_TEST_FLAGS=-timeout=30m and exact-source make check-build passed; includes race tests, vet/staticcheck, 442 UI tests, Svelte checks and non-CGO checks.
  • Recorded Alpine QEMU release controls cover config-only restore preserving newer leases, native conflict refusal without mutation, eligible native restore and reboot. Same-source optimized debugger cases cover power loss before/after fence release, completed native readiness timeout with automatic recovery in the same agent process, and power loss after activation preserving a fourth newly allocated lease. Debugger machine code differs from the stripped release binary; separate release controls passed.
  • Final guest evidence confirms restored original service, release executable and root-owned TLS permissions. Cleanup evidence records stopped task processes and removal of disposable overlay, binaries, archives, credentials and sockets; reusable base retained unchanged.

Limits

  • Repeated interruption during OpenRC service start remains fail-closed with a pending checkpoint and requires reboot in the recorded case; it is not claimed as automatic recovery.
  • Acceptance is scoped to the disposable installed Alpine QEMU fixture; physical routers and clustered HA peers were not tested. This review does not authorize merge or issue closure.
  • Evidence: artifacts/tasks/28-qemu-acceptance-20261105/REPORT.md and its referenced reports/logs (local artifacts).
## Independent static review - Approved exact revision `d3aaca2967f656244151f6d652b9f4c83ea26bd2b388791eb928f7bfee8c6da3`. No blocking findings remain after reviewing product changes, regression tests and recorded validation. The reviewer executed no tests, builds or router actions. - Reserved target-local TLS identity is preserved while custom TLS restoration and archive integrity checks remain active. Durable restart checkpoints no longer replay already-published configuration over authentication normalization. ## Validation evidence - Full `make check GO_TEST_FLAGS=-timeout=30m` and exact-source `make check-build` passed; includes race tests, vet/staticcheck, 442 UI tests, Svelte checks and non-CGO checks. - Recorded Alpine QEMU release controls cover config-only restore preserving newer leases, native conflict refusal without mutation, eligible native restore and reboot. Same-source optimized debugger cases cover power loss before/after fence release, completed native readiness timeout with automatic recovery in the same agent process, and power loss after activation preserving a fourth newly allocated lease. Debugger machine code differs from the stripped release binary; separate release controls passed. - Final guest evidence confirms restored original service, release executable and root-owned TLS permissions. Cleanup evidence records stopped task processes and removal of disposable overlay, binaries, archives, credentials and sockets; reusable base retained unchanged. ## Limits - Repeated interruption during OpenRC service start remains fail-closed with a pending checkpoint and requires reboot in the recorded case; it is not claimed as automatic recovery. - Acceptance is scoped to the disposable installed Alpine QEMU fixture; physical routers and clustered HA peers were not tested. This review does not authorize merge or issue closure. - Evidence: `artifacts/tasks/28-qemu-acceptance-20261105/REPORT.md` and its referenced reports/logs (local artifacts).
Alex changed title from WIP: Fix installed startup diagnostics and safe recovery progression to Fix installed startup diagnostics and safe recovery progression 2026-11-05 20:09:57 +01:00
Alex changed title from Fix installed startup diagnostics and safe recovery progression to WIP: Fix installed startup diagnostics and safe recovery progression 2026-11-05 20:10:21 +01:00
Alex changed title from WIP: Fix installed startup diagnostics and safe recovery progression to Fix installed startup diagnostics and safe recovery progression 2026-11-05 20:42:06 +01:00
Alex merged commit 6ac074c3d6 into issue-65-doltlite-poc 2026-11-05 20:42:24 +01:00
Sign in to join this conversation.
Reviewers
No reviewers
kaido-reviewer
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
Alex/kaido-core!70
No description provided.