Remote access via the second uplink fails while the configured default gateway is down #134

New issue

Open

opened 2026-11-12 08:24:26 +01:00 by Alex · 1 comment

Alex commented 2026-11-12 08:24:26 +01:00

Owner

• Copy link
• Copy Markdown

Severity: High. Found in the code review of the commits from 2026-11-04 to 2026-11-11 (main at 6076be32). Still present on main at adbeb2f4. Reproduced in a router VM, with the cause confirmed.

Requirement (from the user)

• The dashboard on port 9443 is allowed from any source to any destination, so it can be reached via WAN1 and WAN2.
• It is reachable on the WAN1 address and on the WAN2 address.
• If WAN1 fails, it stays reachable on WAN2, and the same the other way round.

Context

• Reply-to (f96fcd5e, ADR 0026): connections that arrive on an uplink get a conntrack label. Their replies get the uplink's policy mark, and the fwmark rule at priority 100 selects the gateway table.

• Shared subnet (c38df1d6, ADR 0027, #120): uplink tables carry their connected routes, and source routing installs rules per uplink address.

• 53a0d664 made it three rules per address on consecutive priorities:

  1. from A lookup <source table> suppress_prefixlength 0
  2. from A lookup main suppress_prefixlength 0 (new)
  3. from A lookup <source table>

  This fixed LAN clients that connect to an uplink address. Their replies had left through the uplink's gateway.

• Fail-closed: when all configured default gateways are down, Kaido installs unreachable default proto 201 metric 1 in main (internal/service/gateway_group.go:421–433). Traffic then does not escape through lower-priority DHCP/RA defaults.

Problem

• The kernel applies suppress_prefixlength only when the table lookup succeeds (net/core/fib_rules.c:342–352).
• An unreachable route returns -EHOSTUNREACH (net/ipv4/fib_semantics.c:117, net/ipv4/fib_trie.c:1566), so the rule walk ends with an error and never reaches rule 3.
• So while WAN1 is down, every locally generated reply from the WAN2 address (SYN-ACK, echo reply) ends at main's unreachable default, and no reply is sent.
• Reply-to cannot help. A local reply first needs a successful route lookup, and only after that does the output chain set the reply-to mark and reroute.
• Before 53a0d664, the source table answered these packets correctly.

Impact

• Remote management (dashboard, API, SSH, VPN endpoints) via the second uplink fails exactly during failover, when it is needed most.
• Condition: the configured default has no working member while another uplink is still up, e.g. default_gateways = WAN1 only and WAN2 as a secondary uplink. The uplinks are DHCP-discovered, because source routing applies only to them.
• Not affected: a default that is a failover group containing WAN2, because the default moves to WAN2. Also not affected: WAN2 down while WAN1 is the working default.

Evidence (router VM, image 6076be32, two DHCP uplinks, default_gateways = ['WAN1_GW4'], remote client 192.0.2.10 behind WAN2)

• Control, WAN1 up: ip route get 192.0.2.10 from 203.0.113.2 gives via 203.0.113.1 dev eth2 table 2001. From the remote client, ping gets 3 of 3 and TCP 8443 connects.
• WAN1 down at 06:26:25:
  • Kaido installs unreachable default proto 201 metric 1.
  • ip route get … from 203.0.113.2 then gives RTNETLINK answers: Host is unreachable in 52 of 52 samples.
  • From the remote client, ping and TCP fail.
• Causal control:
  • Deleting only rule 204: from 203.0.113.2 lookup main suppress_prefixlength 0 immediately sends the route via table 2001, and ping and TCP from the remote client work.
  • Kaido re-adds the rule 18 s later, and access fails again.
• Recovery: with WAN1 up again, everything works.

Proposed change

• Make the fail-closed default in main a throw route instead of unreachable. This touches installMainUnreachableRoute and the audit/reconcile predicates for it.
• Why this works, from the kernel source:
  • RTN_THROW returns -EAGAIN (net/ipv4/fib_semantics.c:125, net/ipv6/route.c:1101), and the rule walk then continues (net/core/fib_rules.c:348, net/ipv6/fib6_rules.c:250–259).
  • A reply from an uplink address therefore reaches rule 3, its source table.
  • Normal traffic finds no later default, because table default is empty. The lookup ends with ENETUNREACH, so fail-closed still holds.
  • At metric 1 the throw route still hides the DHCP/RA defaults.
• This must be proven with kernel tests for IPv4 and IPv6 before merging.
• Unchanged: reply-to (ADR 0026), the shared-subnet tables and per-address rules (ADR 0027), and the off switch (ADR 0028).

Rejected alternatives

• Revert 53a0d664: brings back the bug where a LAN client cannot reach the dashboard on an uplink address (#120 context).
• Copy all of main's specific routes (LAN, VPN, static, FRR) into every source table: a large change that must also follow dynamic routes. It contradicts the "bounded data per reconciliation" rationale of ADR 0027.

Open question: manual gateways

• Source routing covers discovered gateways only. That has been the case since 88813470 (April 2026), without a documented reason; ADR 0028 only records the consequence.
• Suspicion (derived from the kernel source, not tested): with a manually configured WAN2 gateway, the same failover also fails, even after the throw change.
  • There is no source rule, so throw falls through to the empty default table, and the first lookup still fails.
  • Loose rp_filter=2 may also drop the incoming request.
• A baseline comparison on the build VM is running: plain Alpine 3.24 against Kaido with DHCP uplinks, manual gateways and a failover group. The results will be added here.
• If the suspicion is confirmed, extend source routing to manual gateways and update ADR 0028 and the gateway form text.

Acceptance tests

• Router VM. Setup: dashboard on 0.0.0.0:9443 (not bind_interface = 'lan' as in example-router.toml:3106) and a firewall that allows 9443 from any source. Checks:
  • the dashboard is reachable on the WAN1 and the WAN2 address with both uplinks up, with WAN1 down and with WAN2 down;
  • each case once with default_gateways = WAN1 only and once with a WAN1→WAN2 failover group;
  • a LAN client reaches the dashboard via a WAN address (the 53a0d664 case).
• Kernel tests on the build VM:
  • a new lab case "configured default down, other uplink up":
    • (a) a reply from the uplink address leaves through its source table;
    • (b) forwarded LAN traffic and router traffic still find no route, even with a DHCP default present.
  • The existing TestIsolatedReplyTo* (including shared subnet, #120) and TestIsolatedSourceRouting* stay green.

References

• Commits 53a0d664, c38df1d6, f96fcd5e, 6076be32, 88813470.
• ADR 0026, 0027 and 0028; #120.
• Same code as #133; both are fixed in one PR.

Severity: **High**. Found in the code review of the commits from 2026-11-04 to 2026-11-11 (`main` at `6076be32`). Still present on `main` at `adbeb2f4`. Reproduced in a router VM, with the cause confirmed. ## Requirement (from the user) - The dashboard on port 9443 is allowed from any source to any destination, so it can be reached via WAN1 and WAN2. - It is reachable on the WAN1 address and on the WAN2 address. - If WAN1 fails, it stays reachable on WAN2, and the same the other way round. ## Context - **Reply-to** (`f96fcd5e`, ADR 0026): connections that arrive on an uplink get a conntrack label. Their replies get the uplink's policy mark, and the fwmark rule at priority 100 selects the gateway table. - **Shared subnet** (`c38df1d6`, ADR 0027, #120): uplink tables carry their connected routes, and source routing installs rules per uplink address. - **`53a0d664`** made it three rules per address on consecutive priorities: 1. `from A lookup <source table> suppress_prefixlength 0` 2. `from A lookup main suppress_prefixlength 0` (new) 3. `from A lookup <source table>` This fixed LAN clients that connect to an uplink address. Their replies had left through the uplink's gateway. - **Fail-closed:** when all configured default gateways are down, Kaido installs `unreachable default proto 201 metric 1` in main (`internal/service/gateway_group.go:421–433`). Traffic then does not escape through lower-priority DHCP/RA defaults. ## Problem - The kernel applies `suppress_prefixlength` only when the table lookup **succeeds** (`net/core/fib_rules.c:342–352`). - An `unreachable` route returns `-EHOSTUNREACH` (`net/ipv4/fib_semantics.c:117`, `net/ipv4/fib_trie.c:1566`), so the rule walk ends with an error and never reaches rule 3. - So while WAN1 is down, every locally generated reply from the WAN2 address (SYN-ACK, echo reply) ends at main's unreachable default, and no reply is sent. - Reply-to cannot help. A local reply first needs a successful route lookup, and only after that does the output chain set the reply-to mark and reroute. - Before `53a0d664`, the source table answered these packets correctly. ## Impact - Remote management (dashboard, API, SSH, VPN endpoints) via the second uplink fails **exactly during failover**, when it is needed most. - **Condition:** the configured default has no working member while another uplink is still up, e.g. `default_gateways` = WAN1 only and WAN2 as a secondary uplink. The uplinks are DHCP-discovered, because source routing applies only to them. - **Not affected:** a default that is a failover group containing WAN2, because the default moves to WAN2. Also not affected: WAN2 down while WAN1 is the working default. ## Evidence (router VM, image `6076be32`, two DHCP uplinks, `default_gateways = ['WAN1_GW4']`, remote client 192.0.2.10 behind WAN2) - **Control, WAN1 up:** `ip route get 192.0.2.10 from 203.0.113.2` gives `via 203.0.113.1 dev eth2 table 2001`. From the remote client, ping gets 3 of 3 and TCP 8443 connects. - **WAN1 down at 06:26:25:** - Kaido installs `unreachable default proto 201 metric 1`. - `ip route get … from 203.0.113.2` then gives `RTNETLINK answers: Host is unreachable` in **52 of 52** samples. - From the remote client, ping and TCP fail. - **Causal control:** - Deleting only rule `204: from 203.0.113.2 lookup main suppress_prefixlength 0` immediately sends the route via `table 2001`, and ping and TCP from the remote client work. - Kaido re-adds the rule 18 s later, and access fails again. - **Recovery:** with WAN1 up again, everything works. ## Proposed change - Make the fail-closed default in main a **`throw`** route instead of `unreachable`. This touches `installMainUnreachableRoute` and the audit/reconcile predicates for it. - **Why this works**, from the kernel source: - `RTN_THROW` returns `-EAGAIN` (`net/ipv4/fib_semantics.c:125`, `net/ipv6/route.c:1101`), and the rule walk then continues (`net/core/fib_rules.c:348`, `net/ipv6/fib6_rules.c:250–259`). - A reply from an uplink address therefore reaches rule 3, its source table. - Normal traffic finds no later default, because table `default` is empty. The lookup ends with `ENETUNREACH`, so fail-closed still holds. - At metric 1 the throw route still hides the DHCP/RA defaults. - This must be proven with kernel tests for IPv4 and IPv6 before merging. - **Unchanged:** reply-to (ADR 0026), the shared-subnet tables and per-address rules (ADR 0027), and the off switch (ADR 0028). ## Rejected alternatives - **Revert `53a0d664`:** brings back the bug where a LAN client cannot reach the dashboard on an uplink address (#120 context). - **Copy all of main's specific routes (LAN, VPN, static, FRR) into every source table:** a large change that must also follow dynamic routes. It contradicts the "bounded data per reconciliation" rationale of ADR 0027. ## Open question: manual gateways - Source routing covers **discovered gateways only**. That has been the case since `88813470` (April 2026), without a documented reason; ADR 0028 only records the consequence. - **Suspicion (derived from the kernel source, not tested):** with a manually configured WAN2 gateway, the same failover also fails, even after the throw change. - There is no source rule, so `throw` falls through to the empty `default` table, and the first lookup still fails. - Loose `rp_filter=2` may also drop the incoming request. - A baseline comparison on the build VM is running: plain Alpine 3.24 against Kaido with DHCP uplinks, manual gateways and a failover group. The results will be added here. - If the suspicion is confirmed, extend source routing to manual gateways and update ADR 0028 and the gateway form text. ## Acceptance tests - **Router VM.** Setup: dashboard on `0.0.0.0:9443` (not `bind_interface = 'lan'` as in `example-router.toml:3106`) and a firewall that allows 9443 from any source. Checks: - the dashboard is reachable on the WAN1 and the WAN2 address with both uplinks up, with WAN1 down and with WAN2 down; - each case once with `default_gateways` = WAN1 only and once with a WAN1→WAN2 failover group; - a LAN client reaches the dashboard via a WAN address (the `53a0d664` case). - **Kernel tests on the build VM:** - a new lab case "configured default down, other uplink up": - (a) a reply from the uplink address leaves through its source table; - (b) forwarded LAN traffic and router traffic still find no route, even with a DHCP default present. - The existing `TestIsolatedReplyTo*` (including shared subnet, #120) and `TestIsolatedSourceRouting*` stay green. ## References - Commits `53a0d664`, `c38df1d6`, `f96fcd5e`, `6076be32`, `88813470`. - ADR 0026, 0027 and 0028; #120. - Same code as #133; both are fixed in one PR.

Alex referenced this issue 2026-11-12 08:24:27 +01:00

Source routing cleanup deletes Kaido's own default routes in main #133

Alex commented 2026-11-12 09:24:08 +01:00

Author

Owner

• Copy link
• Copy Markdown

Baseline comparison on the build VM (2026-11-12)

Setup.

• Two uplinks: WAN1 198.51.100.2/24 and WAN2 203.0.113.2/24, each behind its own provider namespace.
• Remote client 192.0.2.10 in an "internet" namespace, reachable through both providers.
• Each provider drops packets whose source is not its customer prefix (anti-spoofing).
• A TCP listener on 0.0.0.0:9443, and the firewall allows 9443 and ICMP from any source.

States:

• S0: both up.
• S1: WAN1's provider dead, guest link up.
• S2: WAN1 restored, WAN2's provider dead.
• S3: WAN1 link down via QEMU set_link.

Systems:

• Plain Alpine: official cloud image 3.24.4, kernel 6.18.60-virt, defaults unchanged.
• Kaido: image 6076be32, Alpine 3.24.4, kernel 6.18.63-virt.

Results (TCP connects from the remote client to the dashboard port, 10 attempts each)

System	S0 WAN1 / WAN2	S1 (WAN1 dead) WAN2	S2 (WAN2 dead) WAN1
Plain Alpine, dhcpcd (image default)	ok / fail	fail	ok
Plain Alpine, udhcpc	ok / fail	fail	ok
Kaido, DHCP uplinks, default_gateways = WAN1	ok / ok	fail	ok
Kaido, manual gateways, default_gateways = gw-wan1	ok / ok	fail	ok
Kaido, failover group gw-wan1 → gw-wan2	ok / ok	ok	ok

Further results:

• In S3, Alpine with dhcpcd drops the WAN1 default on link loss, so WAN2 then works (10/10). Kaido with manual gateways fails in S3 the same way as in S1.
• The anti-spoofing drop counters stayed 0 in every state of every system.
• LAN clients reached the WAN addresses in all Kaido states.

Findings

• Plain Alpine fails WAN2 even with both uplinks up.
  • There is one main table, and the udhcpc/dhcpcd defaults use metric 200+ifindex or 1000+ifindex, so WAN1 wins.
  • rp_filter=1 drops every request that arrives on WAN2. The guest's IPReversePathFilter counter rose by 33 per probe run.
  • Alpine has no gateway monitoring. A dead provider keeps the default for good; only a link loss under dhcpcd removes it.
• Kaido S0 works for both uplinks through reply-to and loose rp_filter=2. This is exactly what Alpine lacks.
• Kaido, DHCP uplinks, S1: reproduces this issue.
  • unreachable default proto 201 metric 1 appears in main.
  • ip route get 192.0.2.10 from 203.0.113.2 gives RTNETLINK answers: Host is unreachable.
  • The rp counter rose by 33.
• Kaido, manual gateways, S1: the suspicion is CONFIRMED.
  • There are no source rules at all (rules are only 0, 66/67, 100/101, 32766, 32767), and route get from the WAN2 address is unreachable.
  • Diagnostic with all.rp_filter=0 set by hand: the SYNs reach the VM, but there is still no SYN-ACK and route get stays unreachable.
  • So there are two independent gaps: the missing first route for the local reply, and the loose rp_filter drop, whose reverse lookup also finds no route.
• Kaido with a failover group meets the requirement. The default moves to default via 203.0.113.1 dev eth2 proto 201 metric 1.

Consequence for the fix

throw alone is not enough:

• With DHCP uplinks, the source rules then reach the source table. This covers the reply and the rp_filter reverse lookup, because its source is the uplink address.
• With manual gateways there is no source rule, so throw falls through to the empty default table and both gaps remain.

The fix therefore has two parts:

1. Make the fail-closed default in main a throw route instead of unreachable.
2. Extend source routing to manual gateways with an address on their interface. Respect return_routing. Update ADR 0028, which today says "source routing covers discovered gateways only", and the gateway form text.

The acceptance test runs all five Kaido variants above. In S1 and S2, the uplink that is still up must answer.

## Baseline comparison on the build VM (2026-11-12) **Setup.** - Two uplinks: WAN1 198.51.100.2/24 and WAN2 203.0.113.2/24, each behind its own provider namespace. - Remote client 192.0.2.10 in an "internet" namespace, reachable through both providers. - Each provider drops packets whose source is not its customer prefix (anti-spoofing). - A TCP listener on `0.0.0.0:9443`, and the firewall allows 9443 and ICMP from any source. **States:** - S0: both up. - S1: WAN1's provider dead, guest link up. - S2: WAN1 restored, WAN2's provider dead. - S3: WAN1 link down via QEMU `set_link`. **Systems:** - Plain Alpine: official cloud image 3.24.4, kernel 6.18.60-virt, defaults unchanged. - Kaido: image `6076be32`, Alpine 3.24.4, kernel 6.18.63-virt. ### Results (TCP connects from the remote client to the dashboard port, 10 attempts each) | System | S0 WAN1 / WAN2 | S1 (WAN1 dead) WAN2 | S2 (WAN2 dead) WAN1 | |---|---|---|---| | Plain Alpine, dhcpcd (image default) | ok / **fail** | **fail** | ok | | Plain Alpine, udhcpc | ok / **fail** | **fail** | ok | | Kaido, DHCP uplinks, `default_gateways` = WAN1 | ok / ok | **fail** | ok | | Kaido, manual gateways, `default_gateways` = gw-wan1 | ok / ok | **fail** | ok | | Kaido, failover group gw-wan1 → gw-wan2 | ok / ok | ok | ok | Further results: - In S3, Alpine with dhcpcd drops the WAN1 default on link loss, so WAN2 then works (10/10). Kaido with manual gateways fails in S3 the same way as in S1. - The anti-spoofing drop counters stayed 0 in every state of every system. - LAN clients reached the WAN addresses in all Kaido states. ### Findings - **Plain Alpine** fails WAN2 even with both uplinks up. - There is one main table, and the udhcpc/dhcpcd defaults use metric 200+ifindex or 1000+ifindex, so WAN1 wins. - `rp_filter=1` drops every request that arrives on WAN2. The guest's `IPReversePathFilter` counter rose by 33 per probe run. - Alpine has no gateway monitoring. A dead provider keeps the default for good; only a link loss under dhcpcd removes it. - **Kaido S0** works for both uplinks through reply-to and loose `rp_filter=2`. This is exactly what Alpine lacks. - **Kaido, DHCP uplinks, S1:** reproduces this issue. - `unreachable default proto 201 metric 1` appears in main. - `ip route get 192.0.2.10 from 203.0.113.2` gives `RTNETLINK answers: Host is unreachable`. - The rp counter rose by 33. - **Kaido, manual gateways, S1: the suspicion is CONFIRMED.** - There are no source rules at all (rules are only 0, 66/67, 100/101, 32766, 32767), and route get from the WAN2 address is unreachable. - Diagnostic with `all.rp_filter=0` set by hand: the SYNs reach the VM, but there is still no SYN-ACK and route get stays unreachable. - So there are two independent gaps: the missing first route for the local reply, and the loose rp_filter drop, whose reverse lookup also finds no route. - **Kaido with a failover group** meets the requirement. The default moves to `default via 203.0.113.1 dev eth2 proto 201 metric 1`. ### Consequence for the fix `throw` alone is not enough: - With DHCP uplinks, the source rules then reach the source table. This covers the reply and the rp_filter reverse lookup, because its source is the uplink address. - With manual gateways there is no source rule, so `throw` falls through to the empty `default` table and both gaps remain. The fix therefore has two parts: 1. Make the fail-closed default in main a `throw` route instead of `unreachable`. 2. **Extend source routing to manual gateways** with an address on their interface. Respect `return_routing`. Update ADR 0028, which today says "source routing covers discovered gateways only", and the gateway form text. The acceptance test runs all five Kaido variants above. In S1 and S2, the uplink that is still up must answer.

Alex referenced this issue from a commit 2026-11-12 14:19:23 +01:00

Keep main's fail-closed default in place on reconciliation

Alex referenced this issue from a commit 2026-11-12 14:19:23 +01:00

Mark TCP connections to the router before routing

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

134-reply-to-before-routing

119-example-firstboot

No results found.

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Alex/kaido-core#134

No description provided.