Source routing cleanup deletes Kaido's own default routes in main #133

New issue

Closed

opened 2026-11-12 08:24:11 +01:00 by Alex · 0 comments

Alex commented

2026-11-12 08:24:11 +01:00

Owner

Severity: High. Found in the code review of the commits from 2026-11-04 to 2026-11-11 (main at 6076be32). Still present on main at adbeb2f4. Reproduced with a kernel test and in a router VM.

Context

Return routing consists of reply-to (ADR 0026) and source routing (ADR 0027), with an off switch (ADR 0028).
Source routing applies to DHCP-discovered uplink gateways. It gives each uplink address its own policy rules and a source table (2000+).
53a0d664 ("Let main's specific routes win for traffic from uplink addresses", 2026-11-11) added a second rule per address, from <addr> lookup main suppress_prefixlength 0. It lets LAN, VPN and static routes win for router traffic from an uplink address.
Kaido also owns default routes in main (proto 201):
- the configured default (reconcileDefaults, metric 1);
- the fail-closed unreachable default (installMainUnreachableRoute, internal/service/gateway_group.go:421–433).

Problem

isManagedSourceRule (internal/service/gateway_kernel.go:378–386) now also counts the main-suppress rule as one of Kaido's own source rules:
```
if rule.Table == unix.RT_TABLE_MAIN { return rule.SuppressPrefixlen == 0 && rule.Src != nil }
```
reconcileSourceRouting (internal/service/gateway_source_routing.go:110–118) adds the table of every managed rule to the cleanup candidates, so main (254) becomes a candidate.
The cleanup loop (gateway_source_routing.go:271–292) deletes every proto-201 default route in a candidate table that is not a desired source-table route.
- The desired routes all live in tables 2000+, so the loop deletes Kaido's configured main default and the fail-closed unreachable default.
- The gateway-group reconcile installs them again.
The predicate "this rule is ours" was widened, but other code reads it as "this table is ours and may be cleaned up".

Impact

Each audit tick (30 s): the main default is deleted and re-added. This causes route churn and short gaps without the configured default.
After a reconfigure or a discovery change: source routing runs after the default reconcile. The router can then be without its configured default for up to 30 s.
- Traffic leaves through DHCP/RA defaults, i.e. through an uplink the operator did not choose.
- Or the fail-closed protection is missing for that time.
Condition: default_gateways is set, and at least one DHCP-discovered uplink with an address has return routing on, which is the default. This is the usual setup for internet uplinks.

Evidence

Kernel test: TestGatewayKernelAuditNoMutationsAndRepairsDrift (internal/service/gateway_route_state_kernel_test.go:15–55) expects zero kernel changes from a healthy audit in exactly this setup. On the build VM:
- At dd77ca02 it fails 4 of 4 runs with healthy audit made 1 kernel mutations and source routing reconciliation failed error="no such process".
- At c38df1d6 (the parent of 53a0d664) it passes 4 of 4.
Router VM:
- Setup: image 6076be32, Alpine kernel 6.18.63-virt, two DHCP uplinks, default_gateways = ['WAN1_GW4'], both uplinks up.
- ip monitor route shows Deleted default via 198.51.100.1 dev eth1 proto 201 metric 1 followed by a re-add at 06:24:08, 06:24:38 and 06:25:08.
- Each deletion coincides with the agent log line gateway audit finished. The DHCP defaults are not touched.
The kernel tests were evidently not run before 53a0d664 was committed. The existing audit test above would have caught this.

Proposed change

Never treat table 254 as a route-cleanup candidate. Limit the route cleanup to the source-table range (srcRouteTableBase …).
The main-suppress rule itself stays a managed rule. It is still created, checked and removed as a rule.

Validation

TestGatewayKernelAuditNoMutationsAndRepairsDrift must pass. It is a kernel test that needs conntrack labels, so it runs on the build VM, not in the sandbox.
A unit test for the candidate selection: main is never a candidate.
go vet on internal/service.

References

Commit 53a0d664, ADR 0027.
Fixed together with #134, which concerns the same rule.

Severity: **High**. Found in the code review of the commits from 2026-11-04 to 2026-11-11 (`main` at `6076be32`). Still present on `main` at `adbeb2f4`. Reproduced with a kernel test and in a router VM. ## Context - Return routing consists of reply-to (ADR 0026) and source routing (ADR 0027), with an off switch (ADR 0028). - Source routing applies to DHCP-discovered uplink gateways. It gives each uplink address its own policy rules and a source table (`2000+`). - `53a0d664` ("Let main's specific routes win for traffic from uplink addresses", 2026-11-11) added a second rule per address, `from <addr> lookup main suppress_prefixlength 0`. It lets LAN, VPN and static routes win for router traffic from an uplink address. - Kaido also owns default routes in main (proto 201): - the configured default (`reconcileDefaults`, metric 1); - the fail-closed `unreachable default` (`installMainUnreachableRoute`, `internal/service/gateway_group.go:421–433`). ## Problem - `isManagedSourceRule` (`internal/service/gateway_kernel.go:378–386`) now also counts the main-suppress rule as one of Kaido's own source rules: ```go if rule.Table == unix.RT_TABLE_MAIN { return rule.SuppressPrefixlen == 0 && rule.Src != nil } ``` - `reconcileSourceRouting` (`internal/service/gateway_source_routing.go:110–118`) adds the table of every managed rule to the cleanup `candidates`, so **main (254) becomes a candidate**. - The cleanup loop (`gateway_source_routing.go:271–292`) deletes every proto-201 default route in a candidate table that is not a desired source-table route. - The desired routes all live in tables `2000+`, so the loop deletes Kaido's configured main default and the fail-closed unreachable default. - The gateway-group reconcile installs them again. - The predicate "this rule is ours" was widened, but other code reads it as "this table is ours and may be cleaned up". ## Impact - **Each audit tick (30 s):** the main default is deleted and re-added. This causes route churn and short gaps without the configured default. - **After a reconfigure or a discovery change:** source routing runs after the default reconcile. The router can then be without its configured default for up to 30 s. - Traffic leaves through DHCP/RA defaults, i.e. through an uplink the operator did not choose. - Or the fail-closed protection is missing for that time. - **Condition:** `default_gateways` is set, and at least one DHCP-discovered uplink with an address has return routing on, which is the default. This is the usual setup for internet uplinks. ## Evidence - **Kernel test:** `TestGatewayKernelAuditNoMutationsAndRepairsDrift` (`internal/service/gateway_route_state_kernel_test.go:15–55`) expects zero kernel changes from a healthy audit in exactly this setup. On the build VM: - At `dd77ca02` it **fails 4 of 4** runs with `healthy audit made 1 kernel mutations` and `source routing reconciliation failed error="no such process"`. - At `c38df1d6` (the parent of `53a0d664`) it **passes 4 of 4**. - **Router VM:** - Setup: image `6076be32`, Alpine kernel 6.18.63-virt, two DHCP uplinks, `default_gateways = ['WAN1_GW4']`, both uplinks up. - `ip monitor route` shows `Deleted default via 198.51.100.1 dev eth1 proto 201 metric 1` followed by a re-add at 06:24:08, 06:24:38 and 06:25:08. - Each deletion coincides with the agent log line `gateway audit finished`. The DHCP defaults are not touched. - The kernel tests were evidently not run before `53a0d664` was committed. The existing audit test above would have caught this. ## Proposed change - Never treat table 254 as a route-cleanup candidate. Limit the route cleanup to the source-table range (`srcRouteTableBase` …). - The main-suppress rule itself stays a managed rule. It is still created, checked and removed as a rule. ## Validation - `TestGatewayKernelAuditNoMutationsAndRepairsDrift` must pass. It is a kernel test that needs conntrack labels, so it runs on the build VM, not in the sandbox. - A unit test for the candidate selection: main is never a candidate. - `go vet` on `internal/service`. ## References - Commit `53a0d664`, ADR 0027. - Fixed together with #134, which concerns the same rule.

Alex referenced this issue

2026-11-12 08:24:26 +01:00

Remote access via the second uplink fails while the configured default gateway is down #134

Alex referenced this issue from a pull request that will close it,

2026-11-12 11:03:04 +01:00

Keep main out of the source routing route cleanup #154

Alex referenced this issue from a commit

2026-11-12 11:03:05 +01:00

Keep main out of the source routing route cleanup

Alex closed this issue

2026-11-12 11:03:10 +01:00