Alex/kaido-core

SHA256

Watch

Fork 0

Restore and startup availability failures with stale runtime state #28

New issue

Open

opened 2026-10-21 17:38:44 +02:00 by Alex · 1 comment

Alex commented

2026-10-21 17:38:44 +02:00

Owner

Preserve native leases and allocator fencing. Allow the pending restore to inspect an already stopped allocator without activating or stopping an HA primary through the normal backup path. Verify that management configuration survives the complete restore and restart. Also diagnose the console management-address failure before further device refreshes.

Discovered during the separately authorized three-device refresh after issue #25; do not merge without explicit approval. Keep issue #25 source removal isolated.

Restore and startup availability findings

Source revision: 6553a5f5b8c930a27a268b8429c53b3ffd8761f7e3339b470a4719a0877a3cac,
based on issue25 revision 0767af8. The user requested that these findings stay
in issue28 as deferred work. Do not close issue28 merely because the immediate
stopped-HA inspection fix lands. No general startup redesign is implemented.

Reproduced incident: pending restore loses management reachability

Scope7's configuration-only restore was accepted and requested a reboot. The
running and committed backup configurations had no HA cluster, but the native
Kea configuration still identified this router as HA primary. The restore
journal reached awaiting_restart and established the durable allocator fence.
During boot, seedPendingRecoveryDNS called normal native backup capture,
which returned HA primary lease capture requires maintenance backup preparation.
This happened before interface reconciliation. The agent process remained
running, protected API endpoints returned503, and the management IP was absent.
The installer network file uses manual addressing and relies on the agent to
restore the address. The user had to add the address locally using ip.

The user also reported that console option4 could not restore the address.
The installed menu persists the address using --set-mgmt-ip, starts the agent
and waits for the address. The same pending recovery failure prevents that
agent from applying the saved address. The precise console error output was
not captured, so a separate console defect is not established. During the
maintenance automation, a different observed stop/start race briefly left an
exiting agent child detectable by the offline command. That attempt rolled
back the original databases; the maintenance script now waits for process exit
before moving any database files.

Immediate correction: a read-only stopped-allocator lease inspection path is
used solely for pending restore DNS seeding after durable fencing. It refuses
a running allocator and does not stop/start one. The normal HA-primary backup
capture guard remains in place. It does not replace native lease files.

Validation so far:

Provider regression fails with the pre-fix capture behavior and the incident
error; corrected Kea, API and agent-command test packages pass.
Independent static review found no substantive code defect.
On Scope7 the corrected agent became API-ready and cleared the existing
pending restore journal. Native lease-file hashes were unchanged.
Management configuration matches the pre-purge backup. Whole-state comparison
matches the documented full-restore contract, including native proxy files
captured into explicit desired configuration by captureProxyFiles. The native
proxy contents are unchanged; JSON representation differs from the old export.
A full reboot persistence test passed: changed boot ID, management address
restored automatically, API ready and journal absent. Full make check
passed on the unchanged rerun (Go race/vet,325 UI tests, Svelte0errors/0warnings).
The first run hit the existing leaseevent socket-mode timing failure755; its
log is retained. Optional staticcheck was unavailable and skipped by Makefile.

Desired follow-up behavior (deferred)

A stale HA address or failed optional service must not make management,
diagnostics or configuration recovery inaccessible. Keep service health
separate from management availability. Preserve allocator/HA fencing and
transaction recovery: management availability must not grant unsafe DHCP
allocation authority or skip an incomplete recovery transaction.

Additional source-confirmed startup blockers

These findings are static source analysis; they have not been reproduced on
routers. They are recorded for later work, not represented as fixed.

Management listener binding precedes recovery.
internal/api/server.go:687 initializes listeners before boot reconciliation.
Interface/address resolution can fail after its retry period (:1090,
:1108). Explicit BindAddress is returned unchanged (:1155); a missing
old HA address can cause listener bind failure (:728, :739) and terminate
the agent (internal/agent/agent.go:85) before it can repair networking.
Interface-based binding chooses the first IPv4 address (server.go:1169)
without distinguishing a configured management address from incidental VIPs.
Reproduce with a saved listener address missing after a cold boot.
WireGuard preparation precedes all subsystem reconciliation.
internal/state/manager.go:1193 prepares VPN protection before the normal
per-subsystem loop. Endpoint hostname resolution
(internal/service/wireguard_protection.go:77) can fail and return no
successful subsystems (manager.go:1197). Networking is then not repaired,
and required DHCP/DNS startup success fails the global gate
(internal/api/server.go:892). This preparation is outside the normal
per-applier90-second timeout. Reproduce with an unavailable endpoint DNS
resolver and otherwise valid management configuration.
Pending native reverse-proxy rollback precedes interface recovery.
internal/state/manager.go:986 replays native rollback before interfaces.
internal/state/native_rollback.go:148 propagates failure. Proxy rollback
may start/reload an originally running Caddy/HAProxy configuration
(internal/service/reverseproxy_beforeimage.go:195). If it binds an address
requiring interface recovery, failure blocks the recovery that would
restore that address. Reproduce interrupted proxy activation followed by
cold boot without the old bound address.
Feature lifecycle errors become global readiness failures.
internal/service/feature_lifecycle.go:49, :58, :68 aggregate disable,
RAUC-start and autostart errors. internal/api/server.go:896 turns the
aggregate into global startup failure even after subsystem reconciliation.
The readiness middleware (server.go:1003) rejects protected API endpoints,
including repair/restore operations, while the gate fails.
Reproduce a failing optional service start with otherwise healthy management.
Warm restarts preserve runtime rather than repairing all stale state.
internal/state/startup.go:105-163 adopts successful current-boot metadata.
Interfaces, routing, firewall and WireGuard have no general restart restorer.
HA preserves VIP ownership (internal/service/startup_restore.go:33);
proxies normally restore caches without restarting (:69); logging restores
its cache (:125). This intentionally avoids disruption, but a running
process does not prove that crash residues match desired configuration.
Test deliberate stale runtime independently of the persisted startup marker.

Existing isolation and retained safeguards

Normal applier errors from interfaces, routing/FRR, firewall, gateways, VPN,
HA, proxies, notifications and logging are logged and do not stop remaining
appliers (internal/state/manager.go:1215, :1250, :1039). Networking failures
have dependency retries (internal/state/startup.go:202). These protections
do not cover the global preparation/replay/listener paths above.

On interface apply, unwanted static addresses are removed and desired addresses
are added, subject to explicit preserved-address exceptions
(internal/service/interfaces.go:829, :846). HA apply stops an empty desired
configuration (internal/service/appliers.go:1201). Pending native rollback,
candidate and command transactions still act as global replay fences
(internal/state/manager.go:986-995); later work must preserve their safety
without making management diagnostics inaccessible.

Additional findings observed during the Minis-Forum refresh (deferred)

An unselected lease archive can block configuration-only restore analysis.
The original full backup's ZIP/manifest verified and its configurations were
readable, but analysis returned400 (native snapshot/configuration generation mismatch) before the user could select config-only recovery. Native lease
installation was not requested. A derivative configuration-only archive,
preserving every configuration/provenance entry byte-for-byte while explicitly
omitting the lease archive and its manifest claim, was accepted. The original
full backup is retained. Later work should distinguish configuration-only
eligibility from native lease installation eligibility, preserving all lease
validation for actual lease recovery. Never assume ZIP CRC proves API restore
eligibility; analyze before purging a store.
Transient link readiness can become global unready state after restore.
On Minis-Forum after reboot, Kea logged that the configured physical interface
was not running, then failed its control-socket readiness check. The agent
reported degraded startup/API503 although the management address and SSH were
reachable. This is an observed early-service failure, not evidence of a stale
HA address. The final allocator activation occurs after journal completion,
so later work must verify retry/recovery semantics as well as persistent
management access. The incident logs are retained privately.

RZ-VM native runtime prerequisite finding (deferred)

Package compatibility does not prove native agent boot compatibility.
RZ-VM used an older installation without the required protected A/B boot
policy payload. The user approved matching signed native Kea/Knot packages
if compatible. Bundle/recipe verification, target APK signatures and offline
solver simulation passed without downgrades (one libexpat patch upgrade).
The packages installed successfully. The new agent then failed in API server
construction, before opening SQLite stores: DHCP/DNS boot protection: allocator cutover boot policy: open /usr/share/kaido/grub-policy.cfg.in: no such file or directory.

Source: internal/api/server.go:149-160 calls
PrepareAllocatorCutover whenever native services are installed;
internal/provider/rauc/backend.go:285 propagates boot policy failure;
CheckBackendBootPolicy requires the packaged reference policy and the
deployed GRUB policy to match, plus actual slot/layout checks. Merely
installing a reference text file would not establish a safe deployed policy.
No bootloader/layout changes were authorized or made. RZ-VM databases were
not purged. The prior agent/service were restored; API readiness and unchanged running
configuration were verified. Consider a separate OS/boot prerequisite update
before retrying the new agent. Native packages
remain installed; do not report the new RZ agent refresh as completed.

Later work should include this whole startup prerequisite in agent-update
preflight before replacing a working agent, and keep management diagnostics
available when native allocator activation is blocked. Do not remove boot
fencing or silently mark an old boot layout compatible.

Suggested acceptance evidence for later work

Cold boot and same-boot restart with a removed former VIP.
Backup restore to a router whose old native service still references a VIP.
DNS, proxy and VPN failures remain visible while management stays reachable.
Offline management recovery works without depending on failed optional services.
Interrupted restore/rollback resumes or reports a recoverable state; journal
and durable allocator fences are retained until safe completion.
No unsafe allocator activation, no native lease replacement for config-only
restore, and no automatic acceptance of unresolved interface mappings.

Authorized RZ-VM completion follow-up

The user subsequently authorized adapting RZ-VM. Read-only comparison proved that the deployed GRUB policy already exactly matched the current reviewed template, including both actual GPT partition identities. Only /usr/share/kaido/grub-policy.cfg.in was missing. The reference was installed; the deployed bootloader and partition layout were unchanged and the boot check passed.

The next real blocker was a fixed old lease file containing exactly zero bytes, with no old allocator process, no configured DHCP server or HA, and no factory/upgrade-origin marker. The native cutover refused it solely because an executable could not establish the timestamp format, although no timestamps existed. No factory marker or upgrade checkpoint was minted, and no lease file was removed.

Commit bd561eb introduces an explicit empty-source retirement proof: the root helper must first retire/mask the old service, verify the fixed source and no running allocator, and read a regular NOFOLLOW file containing exactly zero bytes. Root-receipt status and protected handoff recheck the empty source, and consumed/corrupt native import receipts remain rejected. Nonempty, absent and unknown-origin data remain protected. Tests reproduce the pre-fix failure, cover active/alternate/populated/symlink/directory/changed-source cases, and affected cutover/service/API packages pass. Static review has no substantive finding; full checks and real RZ protected-handoff evidence are still pending. Broader startup availability work remains deferred.

A configuration-only restore can leave the router without its management address after reboot. On Scope7 the pending restore journal resumes and establishes the allocator fence, but lease DNS seeding rejects an existing HA-primary native configuration before network reconciliation. The persisted configuration has no HA cluster. The API remains degraded (503) and the installer network file uses manual addressing, so local recovery is needed. Preserve native leases and allocator fencing. Allow the pending restore to inspect an already stopped allocator without activating or stopping an HA primary through the normal backup path. Verify that management configuration survives the complete restore and restart. Also diagnose the console management-address failure before further device refreshes. Discovered during the separately authorized three-device refresh after issue #25; do not merge without explicit approval. Keep issue #25 source removal isolated. --- # Restore and startup availability findings Source revision: `6553a5f5b8c930a27a268b8429c53b3ffd8761f7e3339b470a4719a0877a3cac`, based on issue25 revision `0767af8`. The user requested that these findings stay in issue28 as deferred work. Do not close issue28 merely because the immediate stopped-HA inspection fix lands. No general startup redesign is implemented. ## Reproduced incident: pending restore loses management reachability Scope7's configuration-only restore was accepted and requested a reboot. The running and committed backup configurations had no HA cluster, but the native Kea configuration still identified this router as HA primary. The restore journal reached `awaiting_restart` and established the durable allocator fence. During boot, `seedPendingRecoveryDNS` called normal native backup capture, which returned `HA primary lease capture requires maintenance backup preparation`. This happened before interface reconciliation. The agent process remained running, protected API endpoints returned503, and the management IP was absent. The installer network file uses manual addressing and relies on the agent to restore the address. The user had to add the address locally using `ip`. The user also reported that console option4 could not restore the address. The installed menu persists the address using `--set-mgmt-ip`, starts the agent and waits for the address. The same pending recovery failure prevents that agent from applying the saved address. The precise console error output was not captured, so a separate console defect is not established. During the maintenance automation, a different observed stop/start race briefly left an exiting agent child detectable by the offline command. That attempt rolled back the original databases; the maintenance script now waits for process exit before moving any database files. Immediate correction: a read-only stopped-allocator lease inspection path is used solely for pending restore DNS seeding after durable fencing. It refuses a running allocator and does not stop/start one. The normal HA-primary backup capture guard remains in place. It does not replace native lease files. Validation so far: - Provider regression fails with the pre-fix capture behavior and the incident error; corrected Kea, API and agent-command test packages pass. - Independent static review found no substantive code defect. - On Scope7 the corrected agent became API-ready and cleared the existing pending restore journal. Native lease-file hashes were unchanged. - Management configuration matches the pre-purge backup. Whole-state comparison matches the documented full-restore contract, including native proxy files captured into explicit desired configuration by captureProxyFiles. The native proxy contents are unchanged; JSON representation differs from the old export. - A full reboot persistence test passed: changed boot ID, management address restored automatically, API ready and journal absent. Full make check passed on the unchanged rerun (Go race/vet,325 UI tests, Svelte0errors/0warnings). The first run hit the existing leaseevent socket-mode timing failure755; its log is retained. Optional staticcheck was unavailable and skipped by Makefile. ## Desired follow-up behavior (deferred) A stale HA address or failed optional service must not make management, diagnostics or configuration recovery inaccessible. Keep service health separate from management availability. Preserve allocator/HA fencing and transaction recovery: management availability must not grant unsafe DHCP allocation authority or skip an incomplete recovery transaction. ## Additional source-confirmed startup blockers These findings are static source analysis; they have not been reproduced on routers. They are recorded for later work, not represented as fixed. 1. **Management listener binding precedes recovery.** `internal/api/server.go:687` initializes listeners before boot reconciliation. Interface/address resolution can fail after its retry period (`:1090`, `:1108`). Explicit `BindAddress` is returned unchanged (`:1155`); a missing old HA address can cause listener bind failure (`:728`, `:739`) and terminate the agent (`internal/agent/agent.go:85`) before it can repair networking. Interface-based binding chooses the first IPv4 address (`server.go:1169`) without distinguishing a configured management address from incidental VIPs. Reproduce with a saved listener address missing after a cold boot. 2. **WireGuard preparation precedes all subsystem reconciliation.** `internal/state/manager.go:1193` prepares VPN protection before the normal per-subsystem loop. Endpoint hostname resolution (`internal/service/wireguard_protection.go:77`) can fail and return no successful subsystems (`manager.go:1197`). Networking is then not repaired, and required DHCP/DNS startup success fails the global gate (`internal/api/server.go:892`). This preparation is outside the normal per-applier90-second timeout. Reproduce with an unavailable endpoint DNS resolver and otherwise valid management configuration. 3. **Pending native reverse-proxy rollback precedes interface recovery.** `internal/state/manager.go:986` replays native rollback before interfaces. `internal/state/native_rollback.go:148` propagates failure. Proxy rollback may start/reload an originally running Caddy/HAProxy configuration (`internal/service/reverseproxy_beforeimage.go:195`). If it binds an address requiring interface recovery, failure blocks the recovery that would restore that address. Reproduce interrupted proxy activation followed by cold boot without the old bound address. 4. **Feature lifecycle errors become global readiness failures.** `internal/service/feature_lifecycle.go:49`, `:58`, `:68` aggregate disable, RAUC-start and autostart errors. `internal/api/server.go:896` turns the aggregate into global startup failure even after subsystem reconciliation. The readiness middleware (`server.go:1003`) rejects protected API endpoints, including repair/restore operations, while the gate fails. Reproduce a failing optional service start with otherwise healthy management. 5. **Warm restarts preserve runtime rather than repairing all stale state.** `internal/state/startup.go:105-163` adopts successful current-boot metadata. Interfaces, routing, firewall and WireGuard have no general restart restorer. HA preserves VIP ownership (`internal/service/startup_restore.go:33`); proxies normally restore caches without restarting (`:69`); logging restores its cache (`:125`). This intentionally avoids disruption, but a running process does not prove that crash residues match desired configuration. Test deliberate stale runtime independently of the persisted startup marker. ## Existing isolation and retained safeguards Normal applier errors from interfaces, routing/FRR, firewall, gateways, VPN, HA, proxies, notifications and logging are logged and do not stop remaining appliers (`internal/state/manager.go:1215`, `:1250`, `:1039`). Networking failures have dependency retries (`internal/state/startup.go:202`). These protections do not cover the global preparation/replay/listener paths above. On interface apply, unwanted static addresses are removed and desired addresses are added, subject to explicit preserved-address exceptions (`internal/service/interfaces.go:829`, `:846`). HA apply stops an empty desired configuration (`internal/service/appliers.go:1201`). Pending native rollback, candidate and command transactions still act as global replay fences (`internal/state/manager.go:986-995`); later work must preserve their safety without making management diagnostics inaccessible. ## Additional findings observed during the Minis-Forum refresh (deferred) 6. **An unselected lease archive can block configuration-only restore analysis.** The original full backup's ZIP/manifest verified and its configurations were readable, but analysis returned400 (`native snapshot/configuration generation mismatch`) before the user could select config-only recovery. Native lease installation was not requested. A derivative configuration-only archive, preserving every configuration/provenance entry byte-for-byte while explicitly omitting the lease archive and its manifest claim, was accepted. The original full backup is retained. Later work should distinguish configuration-only eligibility from native lease installation eligibility, preserving all lease validation for actual lease recovery. Never assume ZIP CRC proves API restore eligibility; analyze before purging a store. 7. **Transient link readiness can become global unready state after restore.** On Minis-Forum after reboot, Kea logged that the configured physical interface was not running, then failed its control-socket readiness check. The agent reported degraded startup/API503 although the management address and SSH were reachable. This is an observed early-service failure, not evidence of a stale HA address. The final allocator activation occurs after journal completion, so later work must verify retry/recovery semantics as well as persistent management access. The incident logs are retained privately. ## RZ-VM native runtime prerequisite finding (deferred) 8. **Package compatibility does not prove native agent boot compatibility.** RZ-VM used an older installation without the required protected A/B boot policy payload. The user approved matching signed native Kea/Knot packages if compatible. Bundle/recipe verification, target APK signatures and offline solver simulation passed without downgrades (one libexpat patch upgrade). The packages installed successfully. The new agent then failed in API server construction, before opening SQLite stores: `DHCP/DNS boot protection: allocator cutover boot policy: open /usr/share/kaido/grub-policy.cfg.in: no such file or directory`. Source: `internal/api/server.go:149-160` calls `PrepareAllocatorCutover` whenever native services are installed; `internal/provider/rauc/backend.go:285` propagates boot policy failure; `CheckBackendBootPolicy` requires the packaged reference policy and the deployed GRUB policy to match, plus actual slot/layout checks. Merely installing a reference text file would not establish a safe deployed policy. No bootloader/layout changes were authorized or made. RZ-VM databases were not purged. The prior agent/service were restored; API readiness and unchanged running configuration were verified. Consider a separate OS/boot prerequisite update before retrying the new agent. Native packages remain installed; do not report the new RZ agent refresh as completed. Later work should include this whole startup prerequisite in agent-update preflight before replacing a working agent, and keep management diagnostics available when native allocator activation is blocked. Do not remove boot fencing or silently mark an old boot layout compatible. ## Suggested acceptance evidence for later work - Cold boot and same-boot restart with a removed former VIP. - Backup restore to a router whose old native service still references a VIP. - DNS, proxy and VPN failures remain visible while management stays reachable. - Offline management recovery works without depending on failed optional services. - Interrupted restore/rollback resumes or reports a recoverable state; journal and durable allocator fences are retained until safe completion. - No unsafe allocator activation, no native lease replacement for config-only restore, and no automatic acceptance of unresolved interface mappings. ## Authorized RZ-VM completion follow-up The user subsequently authorized adapting RZ-VM. Read-only comparison proved that the deployed GRUB policy already exactly matched the current reviewed template, including both actual GPT partition identities. Only `/usr/share/kaido/grub-policy.cfg.in` was missing. The reference was installed; the deployed bootloader and partition layout were unchanged and the boot check passed. The next real blocker was a fixed old lease file containing exactly zero bytes, with no old allocator process, no configured DHCP server or HA, and no factory/upgrade-origin marker. The native cutover refused it solely because an executable could not establish the timestamp format, although no timestamps existed. No factory marker or upgrade checkpoint was minted, and no lease file was removed. Commit bd561eb introduces an explicit empty-source retirement proof: the root helper must first retire/mask the old service, verify the fixed source and no running allocator, and read a regular NOFOLLOW file containing exactly zero bytes. Root-receipt status and protected handoff recheck the empty source, and consumed/corrupt native import receipts remain rejected. Nonempty, absent and unknown-origin data remain protected. Tests reproduce the pre-fix failure, cover active/alternate/populated/symlink/directory/changed-source cases, and affected cutover/service/API packages pass. Static review has no substantive finding; full checks and real RZ protected-handoff evidence are still pending. Broader startup availability work remains deferred.

Alex changed title from ~~Fix pending restore boot blocked by stopped HA lease inspection~~ to Restore and startup availability failures with stale runtime state

2026-10-21 17:48:12 +02:00

Alex referenced this issue

2026-10-21 17:53:26 +02:00

Recover stopped HA leases and empty allocator sources #29

Alex referenced this issue from a pull request that will close it,

2026-10-21 19:07:36 +02:00

Remove legacy DHCP/DNS conversion and recover stopped restore sources #27

Alex commented

2026-10-21 19:15:39 +02:00

Author

Owner

The scoped stopped-lease recovery and empty allocator source fixes have landed through PRs #29/#27 (main e8df456). Reopening this issue because the broader startup/restore availability findings documented above were explicitly deferred and remain unresolved. All three devices passed final agent readiness/configuration checks; that evidence does not establish fixes for the deferred failure scenarios.

The scoped stopped-lease recovery and empty allocator source fixes have landed through PRs #29/#27 (main e8df456). Reopening this issue because the broader startup/restore availability findings documented above were explicitly deferred and remain unresolved. All three devices passed final agent readiness/configuration checks; that evidence does not establish fixes for the deferred failure scenarios.

Alex referenced this issue

2026-11-05 18:54:01 +01:00

Fix installed startup diagnostics and safe recovery progression #70

No labels

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Alex/kaido-core#28

No description provided.

Rows
Columns