From 6d9a05406f0875d7c1d4c9b90b4ac08749412d62 Mon Sep 17 00:00:00 2001 From: Jose Celano Date: Mon, 9 Mar 2026 18:23:58 +0000 Subject: [PATCH 1/7] fix: enable Docker ip6tables to restore IPv6 UDP tracker visibility MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Docker's default ip6tables: false causes its chain rewrites to wipe ufw's live ip6tables rules after every container restart. This silently dropped all IPv6 UDP port 6969 traffic, causing the UDP tracker to show as down on newTrackon after the nightly backup restart. Fix: add /etc/docker/daemon.json with ip6tables: true. Docker now manages ip6tables rules for all published ports automatically, mirroring its IPv4 behaviour. The fix is systemic — no per-port or per-container configuration needed, and it covers future UDP trackers on any port. Verified on 2026-03-09: - ufw6-user-input rule for udp/6969 survives docker compose restart - udp://udp1.torrust-tracker-demo.com:6969/announce back up on newTrackon Refs: #2 --- project-words.txt | 1 + server/etc/docker/daemon.json | 3 +++ 2 files changed, 4 insertions(+) create mode 100644 server/etc/docker/daemon.json diff --git a/project-words.txt b/project-words.txt index 582a1bb..c745096 100644 --- a/project-words.txt +++ b/project-words.txt @@ -30,6 +30,7 @@ noall noprefixroute noqueue oneshot +post-mortems prometheus qdisc qlen diff --git a/server/etc/docker/daemon.json b/server/etc/docker/daemon.json new file mode 100644 index 0000000..9f66c67 --- /dev/null +++ b/server/etc/docker/daemon.json @@ -0,0 +1,3 @@ +{ + "ip6tables": true +} \ No newline at end of file From cd607be787f62e1e9a1d78af7ab79375cff75445 Mon Sep 17 00:00:00 2001 From: Jose Celano Date: Mon, 9 Mar 2026 18:24:16 +0000 Subject: [PATCH 2/7] docs: document Docker IPv6 fix, post-mortem and server config notes - Add docs/docker-ipv6.md: operations reference explaining why ip6tables: true is required, its scope, what it does not cover (floating IP routing), and server application + verification steps - Add docs/post-mortems/2026-03-09-udp-ipv6-docker.md: full investigation log with commands, outputs, root cause analysis and fix decision rationale - Update docs/issues/ISSUE-2-udp-tracker-down-on-newtrackon.md: add Fix Applied section, tick off completed acceptance criteria - Update server/README.md: add Key Configuration Notes section cross-referencing daemon.json and netplan floating IP config Refs: #2 --- docs/docker-ipv6.md | 123 +++++++++++++++ .../ISSUE-2-udp-tracker-down-on-newtrackon.md | 34 +++- .../2026-03-09-udp-ipv6-docker.md | 149 ++++++++++++++++++ server/README.md | 18 +++ 4 files changed, 320 insertions(+), 4 deletions(-) create mode 100644 docs/docker-ipv6.md create mode 100644 docs/post-mortems/2026-03-09-udp-ipv6-docker.md diff --git a/docs/docker-ipv6.md b/docs/docker-ipv6.md new file mode 100644 index 0000000..f973102 --- /dev/null +++ b/docs/docker-ipv6.md @@ -0,0 +1,123 @@ +# Docker IPv6 Configuration + +## Overview + +By default, Docker has `ip6tables` support disabled. This means Docker does not insert +ip6tables rules when containers publish ports — unlike IPv4, where Docker automatically +creates iptables DNAT rules that route traffic to containers and bypass the ufw INPUT chain. + +For IPv6, without `ip6tables` support enabled in Docker: + +- Incoming IPv6 packets must pass through the ufw INPUT chain to reach a container. +- When Docker starts or restarts any container it rewrites its own iptables/ip6tables + chains (`DOCKER`, `DOCKER-FORWARD`, `DOCKER-USER`). This flush removes ufw's live + ip6tables rules from the kernel, even though they remain stored on disk in `/etc/ufw/`. +- ufw does not automatically reload its ip6tables rules after a Docker chain rewrite. +- Result: IPv6 UDP traffic is silently dropped after every container restart. + +This is documented in [post-mortems/2026-03-09-udp-ipv6-docker.md](post-mortems/2026-03-09-udp-ipv6-docker.md). + +## Fix + +Enable `ip6tables` in the Docker daemon configuration: + +```json +// /etc/docker/daemon.json +{ + "ip6tables": true +} +``` + +With this setting, Docker inserts its own ip6tables DNAT and FORWARD rules when containers +publish ports, exactly mirroring its IPv4 behaviour. ufw's INPUT chain is bypassed for +Docker-published ports, and the rules survive Docker chain rewrites. + +The configuration file is tracked in this repository at +[server/etc/docker/daemon.json](../server/etc/docker/daemon.json). + +## Scope + +This is a **daemon-level** setting. It applies automatically to: + +- All containers, on all Docker-published ports, without any per-container or per-port + configuration. +- Future UDP trackers on new ports (e.g. 6868, 7979) — no additional action needed. + +## What this does NOT cover + +**Floating IP asymmetric routing** is a separate concern. When a published port is +reachable via a Hetzner floating IP, replies must leave via the same floating IP rather +than the server's primary IP. This is handled by policy routing tables in netplan and must +be configured per floating IP. + +See [server/etc/netplan/60-floating-ip.yaml](../server/etc/netplan/60-floating-ip.yaml). + +When adding a **new floating IP**, repeat the following in that file and run +`sudo netplan apply`: + +```yaml +routing-policy: + - from: + table: + - from: + table: +routes: + - to: default + via: 172.31.1.1 + table: + - to: default + via: fe80::1 + table: +``` + +## Applying on the server + +### One-time setup + +```bash +# Copy daemon.json to the server +sudo cp /path/to/daemon.json /etc/docker/daemon.json + +# Restart the Docker daemon +# Containers with restart: unless-stopped will come back up automatically +sudo systemctl restart docker + +# Verify Docker is running +sudo systemctl status docker +``` + +### Verification + +After the Docker daemon restarts, confirm ufw's ip6tables rules are present in the +live kernel: + +```bash +sudo ip6tables -L ufw6-user-input -n +``` + +Expected output includes: + +```text +Chain ufw6-user-input (1 references) +target prot opt source destination +ACCEPT 6 -- ::/0 ::/0 tcp dpt:22 +ACCEPT 17 -- ::/0 ::/0 udp dpt:6969 +``` + +Note: Docker with `ip6tables: true` uses FORWARD chain rules (via `DOCKER-FORWARD` → +`DOCKER-BRIDGE`) rather than NAT/DNAT for IPv6 published ports. The ufw INPUT chain +rules for port 6969 are preserved after Docker chain rewrites, which is the fix. + +### Simulating a nightly restart + +To verify the fix survives a container restart without waiting for the cron job: + +```bash +cd /opt/torrust +docker compose stop tracker +docker compose up -d tracker +sudo ip6tables -L ufw6-user-input -n +``` + +The `ACCEPT udp dpt:6969` rule must still be present after `docker compose up`. +This was verified on 2026-03-09 and the rule survived. diff --git a/docs/issues/ISSUE-2-udp-tracker-down-on-newtrackon.md b/docs/issues/ISSUE-2-udp-tracker-down-on-newtrackon.md index 8ccb217..96aecc5 100644 --- a/docs/issues/ISSUE-2-udp-tracker-down-on-newtrackon.md +++ b/docs/issues/ISSUE-2-udp-tracker-down-on-newtrackon.md @@ -187,12 +187,38 @@ Document the root cause and fix in [torrust-tracker-deployer](https://github.com/torrust/torrust-tracker-deployer) repository (the canonical home for the deployment investigation docs for this server). +## Fix Applied + +**Post-mortem**: [post-mortems/2026-03-09-udp-ipv6-docker.md](../post-mortems/2026-03-09-udp-ipv6-docker.md) + +Root cause confirmed on 2026-03-09: + +- Policy routing rules (Fix B) were **not** affected — `ip rule` and `ip route` entries + were still present. +- The live ip6tables INPUT chain had **no ACCEPT rule for 6969** immediately after the + nightly Docker restart, even though ufw's on-disk rules were intact. Docker's chain + rewrite wiped the live ip6tables rules and ufw never reloaded them. + +**Chosen fix**: Enable `ip6tables: true` in `/etc/docker/daemon.json`. + +With this setting Docker manages ip6tables rules for all published ports automatically, +mirroring its IPv4 behaviour. The fix is systemic — it applies to all containers and all +ports without per-port configuration and survives any container restart. + +See [docs/docker-ipv6.md](../docker-ipv6.md) for full documentation and server +application instructions. + +Files changed: + +- `server/etc/docker/daemon.json` — added with `{"ip6tables": true}` + ## Acceptance Criteria -- [ ] Root cause confirmed (which of the hypotheses applies) -- [ ] `udp://udp1.torrust-tracker-demo.com:6969/announce` accepted by newTrackon ✅ -- [ ] The fix survives a nightly Docker Compose restart (verified by waiting 24 h or simulating - with `docker compose restart tracker`) +- [x] Root cause confirmed — Docker's default `ip6tables: false` causes its chain rewrites + to wipe ufw's live ip6tables rules after every container restart +- [x] `udp://udp1.torrust-tracker-demo.com:6969/announce` accepted by newTrackon ✅ (confirmed 2026-03-09) +- [x] The fix survives a nightly Docker Compose restart (verified by simulating + with `docker compose stop tracker && docker compose up -d tracker` on 2026-03-09) - [ ] Root cause and fix documented in the deployer repo's deployment journal - [ ] If a deployer-level fix is needed, a follow-up issue is opened in [torrust-tracker-deployer](https://github.com/torrust/torrust-tracker-deployer) diff --git a/docs/post-mortems/2026-03-09-udp-ipv6-docker.md b/docs/post-mortems/2026-03-09-udp-ipv6-docker.md new file mode 100644 index 0000000..918d181 --- /dev/null +++ b/docs/post-mortems/2026-03-09-udp-ipv6-docker.md @@ -0,0 +1,149 @@ +# Post-Mortem: UDP Tracker Down on newTrackon After Nightly Restart + +**Date**: 2026-03-09 +**Issue**: [#2](https://github.com/torrust/torrust-tracker-demo/issues/2) +**Issue doc**: [issues/ISSUE-2-udp-tracker-down-on-newtrackon.md](../issues/ISSUE-2-udp-tracker-down-on-newtrackon.md) +**Server**: `udp1.torrust-tracker-demo.com` / `2a01:4f8:1c0c:828e::1` + +--- + +## Investigation + +### Step 1 — Check live ip6tables INPUT chain + +Note: `ufw reload` had NOT been run manually since the last nightly restart when these +commands were executed. + +Command: + +```bash +sudo ip6tables -L INPUT -n --line-numbers | grep -E "6969|ACCEPT|Chain" +``` + +Output: + +```text +Chain INPUT (policy DROP) +``` + +**Finding**: No ACCEPT rule for port 6969 in the live ip6tables. Policy is DROP. +The bug was already active — IPv6 UDP 6969 was being dropped without needing to +simulate a restart. + +--- + +### Step 2 — Check ufw status (stored rules vs live rules) + +Command: + +```bash +sudo ufw status verbose +``` + +Output: + +```text +Status: active +Logging: on (low) +Default: deny (incoming), allow (outgoing), deny (routed) +New profiles: skip + +To Action From +-- ------ ---- +22/tcp ALLOW IN Anywhere # SSH access (configured port 22) +6969/udp ALLOW IN Anywhere +22/tcp (v6) ALLOW IN Anywhere (v6) # SSH access (configured port 22) +6969/udp (v6) ALLOW IN Anywhere (v6) +``` + +**Finding**: ufw has the 6969/udp (v6) rule stored on disk but it is absent from the live +ip6tables. Docker's chain rewrite after the nightly container restart wiped ufw's live +ip6tables rules and ufw did not reload them. Root cause confirmed. + +--- + +### Step 3 — Check Docker daemon.json (ip6tables setting) + +Command: + +```bash +cat /etc/docker/daemon.json 2>/dev/null || echo "(no daemon.json)" +``` + +Output: + +```text +(no daemon.json) +``` + +**Finding**: Docker is using all defaults. `ip6tables` is disabled by default in Docker, +so Docker never inserts ip6tables rules for published ports. IPv6 UDP traffic on port 6969 +must pass through the ufw INPUT chain, but Docker's chain rewrite after each container +start wipes ufw's live ip6tables rules. + +--- + +### Step 4 — Verify fix survives a container restart + +After adding `/etc/docker/daemon.json` with `{"ip6tables": true}` and restarting the +Docker daemon, the container restart was simulated to confirm the fix: + +Command: + +```bash +cd /opt/torrust && docker compose stop tracker && docker compose up -d tracker && sudo ip6tables -L ufw6-user-input -n +``` + +Output: + +```text +[+] Stopping 1/1 + ✔ Container tracker Stopped +[+] Running 2/2 + ✔ Container mysql Healthy + ✔ Container tracker Started +Chain ufw6-user-input (1 references) +target prot opt source destination +ACCEPT 6 -- ::/0 ::/0 tcp dpt:22 +ACCEPT 17 -- ::/0 ::/0 udp dpt:6969 +``` + +**Finding**: The `ACCEPT udp dpt:6969` rule survived the container restart. Fix confirmed. + +--- + +## Root Cause + +1. ufw has `6969/udp (v6) ALLOW IN` stored on disk in `/etc/ufw/` ✅ +2. The live ip6tables INPUT chain had **no rule** for 6969 and policy is DROP ❌ +3. Docker has no `daemon.json` → `ip6tables` is disabled by default +4. When Docker starts/restarts the tracker container it rewrites its chains, flushing + ufw's live ip6tables rules. ufw does not automatically reload after this. +5. For IPv4, Docker inserts its own DNAT rules that bypass ufw INPUT entirely → unaffected. + For IPv6, no equivalent DNAT rules are inserted → traffic must go through ufw INPUT → + silently dropped. + +**Root cause**: Docker's default `ip6tables: false` combined with Docker chain rewrites on +container restart — ufw's ip6tables rules are wiped and never restored. + +--- + +## Fix Decision + +Options considered: + +1. **`ufw reload` in backup script** — targeted, but only covers the nightly cron restart. + Any other container restart (manual, OOM, redeploy) would re-expose the bug. + +2. **Enable `ip6tables: true` in `/etc/docker/daemon.json`** — Docker manages ip6tables + rules the same way it does for IPv4. Published port 6969/udp gets proper FORWARD chain + rules that survive Docker chain rewrites. No changes to the backup script needed. + Requires one-time Docker daemon restart; containers restart automatically via + `restart: unless-stopped`. + +**Chosen fix: Option 2.** It is the systemic fix — Docker becomes responsible for its own +ip6tables rules for published ports, exactly mirroring the IPv4 behaviour. The fix applies +globally to all Docker-published ports on all containers without any per-port or +per-container configuration. + +See [docs/docker-ipv6.md](../docker-ipv6.md) for the full configuration reference. diff --git a/server/README.md b/server/README.md index 88aa6e3..53dac98 100644 --- a/server/README.md +++ b/server/README.md @@ -50,3 +50,21 @@ The following are excluded — they contain runtime data, large binaries, or add | `/opt/torrust/storage/mysql/data/` | MySQL data files | | `/opt/torrust/storage/tracker/lib/` | SQLite database | | `/opt/torrust/storage/tracker/log/` | Log files | + +## Key Configuration Notes + +### Docker IPv6 (`etc/docker/daemon.json`) + +`ip6tables: true` is required for IPv6 UDP traffic to reach Docker containers. Without it +Docker does not insert ip6tables rules for published ports and its chain rewrites wipe ufw's +live ip6tables rules after every container restart, silently dropping all IPv6 UDP traffic. + +See [docs/docker-ipv6.md](../docs/docker-ipv6.md) for the full explanation. + +### Floating IP routing (`etc/netplan/60-floating-ip.yaml`) + +Each Hetzner floating IP requires a policy routing table entry so that replies leave via the +same floating IP rather than the primary server IP (asymmetric routing fix). Adding a new +floating IP requires updating this file and running `sudo netplan apply`. + +See [docs/docker-ipv6.md](../docs/docker-ipv6.md) for details. From 6ffbc3963799536fd040f394118ec9102d14020b Mon Sep 17 00:00:00 2001 From: Jose Celano Date: Mon, 9 Mar 2026 18:35:11 +0000 Subject: [PATCH 3/7] docs: add post-deployment manual steps guide MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The deployer handles basic provisioning but does not cover floating IP routing or Docker IPv6 ip6tables configuration. Add docs/post-deployment.md as the canonical checklist of manual steps required after running the deployer, with a section for each concern: 1. Floating IP routing (netplan policy routing — required per floating IP) 2. Docker IPv6 for UDP trackers (daemon.json ip6tables: true — required only for IPv6 UDP announces) - Add deployer callout to docs/docker-ipv6.md pointing to the checklist - Add post-deployment link to README.md Refs: #2 --- README.md | 2 ++ docs/docker-ipv6.md | 5 +++ docs/post-deployment.md | 71 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 78 insertions(+) create mode 100644 docs/post-deployment.md diff --git a/README.md b/README.md index 8dc5eb1..f4f865b 100644 --- a/README.md +++ b/README.md @@ -75,6 +75,8 @@ supports both **IPv4 and IPv6**. The server was provisioned using [torrust-tracker-deployer](https://github.com/torrust/torrust-tracker-deployer). +Some features (floating IP routing, Docker IPv6) require +[manual post-deployment steps](docs/post-deployment.md) not covered by the deployer. ## Related projects diff --git a/docs/docker-ipv6.md b/docs/docker-ipv6.md index f973102..4fd0a70 100644 --- a/docs/docker-ipv6.md +++ b/docs/docker-ipv6.md @@ -1,5 +1,10 @@ # Docker IPv6 Configuration +> **Post-deployment step**: This configuration is not applied by the +> [torrust-tracker-deployer](https://github.com/torrust/torrust-tracker-deployer). +> It must be applied manually after provisioning. See +> [post-deployment.md](post-deployment.md) for all manual steps. + ## Overview By default, Docker has `ip6tables` support disabled. This means Docker does not insert diff --git a/docs/post-deployment.md b/docs/post-deployment.md new file mode 100644 index 0000000..5deb8c2 --- /dev/null +++ b/docs/post-deployment.md @@ -0,0 +1,71 @@ +# Post-Deployment Manual Steps + +The server was provisioned using +[torrust-tracker-deployer](https://github.com/torrust/torrust-tracker-deployer), +which sets up the complete Docker Compose stack (tracker, MySQL, Caddy, Prometheus, +Grafana) and configures the firewall and basic networking. + +The following steps are **not handled by the deployer** and must be applied manually +after provisioning. They represent customizations specific to this demo's use of +Hetzner floating IPs and IPv6 UDP. + +--- + +## 1. Floating IP routing (required for each floating IP) + +The deployer configures the tracker to listen on the server's primary public IP only. +If you want tracker endpoints to be reachable via a **Hetzner floating IP**, you must +add policy routing rules so that replies leave via the same floating IP rather than the +primary server IP (asymmetric routing fix). + +This is required for **both IPv4 and IPv6** floating IPs. + +**Configuration file**: [`server/etc/netplan/60-floating-ip.yaml`](../server/etc/netplan/60-floating-ip.yaml) + +For each floating IP pair (IPv4 + IPv6), add a routing policy and a route entry, then +apply: + +```bash +sudo netplan apply +``` + +Verify the rules are active: + +```bash +ip rule list +ip route show table 100 +ip -6 rule list +ip -6 route show table 200 +``` + +> This step must be repeated for every new floating IP you add. The deployer has no +> support for floating IP routing and will not generate or apply netplan configuration. + +--- + +## 2. Docker IPv6 for UDP trackers (required only for IPv6 UDP announces) + +The deployer does not configure Docker to manage `ip6tables`. Without this, Docker's +chain rewrites wipe ufw's live ip6tables rules after every container restart, silently +dropping all IPv6 UDP traffic on port 6969. + +This step is only needed if you want the UDP tracker to be reachable over **IPv6**. +IPv4 UDP and all HTTP traffic are unaffected. + +**Configuration file**: [`server/etc/docker/daemon.json`](../server/etc/docker/daemon.json) + +Apply on the server: + +```bash +sudo cp server/etc/docker/daemon.json /etc/docker/daemon.json +sudo systemctl restart docker +``` + +Verify: + +```bash +sudo ip6tables -L ufw6-user-input -n +# Must show: ACCEPT 17 -- ::/0 ::/0 udp dpt:6969 +``` + +See [docker-ipv6.md](docker-ipv6.md) for the full explanation and verification steps. From eb99428519418cc93509a67f9c5814935105194f Mon Sep 17 00:00:00 2001 From: Jose Celano Date: Mon, 9 Mar 2026 19:34:44 +0000 Subject: [PATCH 4/7] fix: enable IPv6 on Docker proxy_network for native IPv6 UDP support Root Cause B: Docker bridge networks were IPv4-only, so no container IPv6 address existed. Docker created no ip6tables DNAT rules, leaving docker-proxy as sole handler. docker-proxy attempted cross-AF UDP relay (IPv6 host socket to IPv4 container backend 172.21.0.3), which silently drops all native IPv6 UDP packets. Changes: - docker-compose.yml: add enable_ipv6: true + subnet fd01:db8:1::/64 to proxy_network so the container receives a ULA IPv6 address and Docker generates ip6tables DNAT rules, bypassing docker-proxy entirely - docs/post-mortems/2026-03-09-udp-ipv6-docker.md: add Steps 9-12 (docker logs grep, ps aux smoking gun), Root Cause B section, Fix B decision rationale - docs/docker-ipv6.md: rewrite to cover both problems (chain wipe and docker-proxy cross-AF) and both fixes (daemon.json and enable_ipv6 + SNAT) - docs/post-deployment.md: add Step 3 with exact SNAT rule for before6.rules to override Docker MASQUERADE and pin UDP 6969 replies to the floating IPv6 2a01:4f8:1c0c:828e::1 - docs/issues/ISSUE-2-udp-tracker-down-on-newtrackon.md: tick off newTrackon acceptance criterion (confirmed 2026-03-09), update survival criterion - server/README.md: expand Key Configuration Notes with daemon.json, enable_ipv6, and SNAT subsections - project-words.txt: add SNAT, SNATs, POSTROUTING, PREROUTING, userland, ulnp, UNCONN, misrouted, ADDRTYPE, pkts Fix verified: newTrackon reports tracker working for both IPv6 and IPv4 (2a01:4f8:1c0c:828e::1 and 116.202.177.184) as of 2026-03-09. A manual server-side step is still required on fresh deploys: prepend a *nat POSTROUTING SNAT rule to /etc/ufw/before6.rules and run sudo ufw reload. See docs/post-deployment.md Step 3 for the exact lines. --- docs/docker-ipv6.md | 196 ++++++++--- .../ISSUE-2-udp-tracker-down-on-newtrackon.md | 41 ++- docs/post-deployment.md | 45 +++ .../2026-03-09-udp-ipv6-docker.md | 319 +++++++++++++++++- project-words.txt | 10 + server/README.md | 32 +- server/opt/torrust/docker-compose.yml | 9 + 7 files changed, 576 insertions(+), 76 deletions(-) diff --git a/docs/docker-ipv6.md b/docs/docker-ipv6.md index 4fd0a70..04d18b0 100644 --- a/docs/docker-ipv6.md +++ b/docs/docker-ipv6.md @@ -7,6 +7,12 @@ ## Overview +Docker requires two distinct configuration changes to make IPv6 UDP tracker traffic work +correctly end-to-end. Both problems were discovered during the 2026-03-09 incident +(see [post-mortems/2026-03-09-udp-ipv6-docker.md](post-mortems/2026-03-09-udp-ipv6-docker.md)). + +### Problem 1 — Docker chain rewrites wipe ufw ip6tables rules + By default, Docker has `ip6tables` support disabled. This means Docker does not insert ip6tables rules when containers publish ports — unlike IPv4, where Docker automatically creates iptables DNAT rules that route traffic to containers and bypass the ufw INPUT chain. @@ -20,64 +26,110 @@ For IPv6, without `ip6tables` support enabled in Docker: - ufw does not automatically reload its ip6tables rules after a Docker chain rewrite. - Result: IPv6 UDP traffic is silently dropped after every container restart. -This is documented in [post-mortems/2026-03-09-udp-ipv6-docker.md](post-mortems/2026-03-09-udp-ipv6-docker.md). +### Problem 2 — docker-proxy cannot relay native IPv6 UDP to an IPv4 container + +Even after Problem 1 is fixed, native IPv6 UDP still does not work. Docker spawns two +`docker-proxy` processes for each published UDP port: + +- IPv4: `-host-ip 0.0.0.0 … -container-ip 172.x.x.x` (same address family — works) +- IPv6: `-host-ip :: … -container-ip 172.x.x.x` (cross-address-family — silently fails) + +docker-proxy receives native IPv6 UDP on its `::` socket but cannot forward to an IPv4 +container backend. Packets are accepted by ip6tables, reach docker-proxy, and are then +silently dropped with no reply and no error log. ## Fix -Enable `ip6tables` in the Docker daemon configuration: +Two configuration changes are required. + +### Fix 1 — Enable `ip6tables` in the Docker daemon + +Create `/etc/docker/daemon.json`: ```json -// /etc/docker/daemon.json { "ip6tables": true } ``` -With this setting, Docker inserts its own ip6tables DNAT and FORWARD rules when containers -publish ports, exactly mirroring its IPv4 behaviour. ufw's INPUT chain is bypassed for -Docker-published ports, and the rules survive Docker chain rewrites. +With this setting, Docker manages ip6tables rules for published ports (same as it already +does for IPv4 with iptables). ufw's INPUT chain rules are preserved after Docker chain +rewrites because Docker now inserts its own FORWARD chain rules for published ports. The configuration file is tracked in this repository at [server/etc/docker/daemon.json](../server/etc/docker/daemon.json). +### Fix 2 — Enable IPv6 on the Docker bridge network + SNAT for reply source + +Enabling `ip6tables: true` alone is not sufficient. Docker only creates ip6tables DNAT +rules when the container has an IPv6 address. Without IPv6 on the Docker network, the +container has only IPv4 bridge addresses and docker-proxy is still the only IPv6 path +(which silently drops native IPv6 UDP, see Problem 2 above). + +**Step 2a — Enable IPv6 on `proxy_network`** in `docker-compose.yml`: + +```yaml +proxy_network: + driver: bridge + enable_ipv6: true + ipam: + config: + - subnet: "fd01:db8:1::/64" +``` + +With an IPv6 address on the container, Docker creates ip6tables DNAT rules that bypass +docker-proxy entirely for native IPv6 traffic, exactly as iptables DNAT does for IPv4. + +This is tracked in +[server/opt/torrust/docker-compose.yml](../server/opt/torrust/docker-compose.yml). + +**Step 2b — Add SNAT to `/etc/ufw/before6.rules`** to rewrite reply source to floating IP: + +Docker's MASQUERADE rule rewrites container reply sources to the server's primary IPv6 +(`2a01:4f8:1c19:620b::1`). Clients probing the floating IPv6 (`2a01:4f8:1c0c:828e::1`) +would receive a reply from the wrong address and time out. Add this block at the **very +top** of `/etc/ufw/before6.rules`, before the existing `*filter` section: + +```text +# NAT: rewrite source of Docker UDP tracker IPv6 replies to the floating IP +*nat +:POSTROUTING ACCEPT [0:0] +-A POSTROUTING -s fd01:db8:1::/64 -o eth0 -p udp --sport 6969 \ + -j SNAT --to-source 2a01:4f8:1c0c:828e::1 +COMMIT +``` + +This rule fires before Docker's MASQUERADE because ufw loads `before6.rules` at startup, +before Docker starts. The SNAT takes precedence and replies leave via the correct floating +IP. + ## Scope -This is a **daemon-level** setting. It applies automatically to: +Fix 1 (`ip6tables: true`) is a **daemon-level** setting. It applies automatically to: - All containers, on all Docker-published ports, without any per-container or per-port configuration. -- Future UDP trackers on new ports (e.g. 6868, 7979) — no additional action needed. -## What this does NOT cover +Fix 2 (`enable_ipv6` on `proxy_network` + SNAT) is **per-`proxy_network`** and +**per-floating-IP**: -**Floating IP asymmetric routing** is a separate concern. When a published port is -reachable via a Hetzner floating IP, replies must leave via the same floating IP rather -than the server's primary IP. This is handled by policy routing tables in netplan and must -be configured per floating IP. +- Future UDP trackers on new ports that use `proxy_network` get IPv6 DNAT automatically. +- If a new floating IPv6 is added, an additional SNAT rule for that address is required + in `before6.rules`. -See [server/etc/netplan/60-floating-ip.yaml](../server/etc/netplan/60-floating-ip.yaml). +## What this does NOT cover -When adding a **new floating IP**, repeat the following in that file and run -`sudo netplan apply`: +**Floating IP asymmetric routing** is a separate concern for IPv4. When a published port +is reachable via a Hetzner floating IPv4, replies must leave via the same floating IPv4 +rather than the server's primary IP. This is handled by policy routing tables in netplan. -```yaml -routing-policy: - - from: - table: - - from: - table: -routes: - - to: default - via: 172.31.1.1 - table: - - to: default - via: fe80::1 - table: -``` +For IPv6, the floating-IP reply source is handled by the SNAT rule in Fix 2b above. + +See [server/etc/netplan/60-floating-ip.yaml](../server/etc/netplan/60-floating-ip.yaml). ## Applying on the server -### One-time setup +### Fix 1 — Docker daemon (one-time setup) ```bash # Copy daemon.json to the server @@ -91,16 +143,56 @@ sudo systemctl restart docker sudo systemctl status docker ``` +### Fix 2a — Enable IPv6 on proxy_network + +Update `docker-compose.yml` on the server to match +[server/opt/torrust/docker-compose.yml](../server/opt/torrust/docker-compose.yml), then +recreate the network: + +```bash +cd /opt/torrust +docker compose down +docker compose up -d +``` + +> `docker compose down` removes the old IPv4-only `proxy_network`. `up` recreates it with +> IPv6 enabled and a new ULA subnet (`fd01:db8:1::/64`). The tracker container will +> receive a new IPv6 address on this bridge. + +### Fix 2b — SNAT in before6.rules + +Add the following block at the **very top** of `/etc/ufw/before6.rules` (before the +existing `*filter` line): + +```bash +sudo sed -i '1s/^/# NAT: Docker UDP tracker IPv6 reply source → floating IP\n*nat\n:POSTROUTING ACCEPT [0:0]\n-A POSTROUTING -s fd01:db8:1:\/64 -o eth0 -p udp --sport 6969 -j SNAT --to-source 2a01:4f8:1c0c:828e::1\nCOMMIT\n\n/' /etc/ufw/before6.rules +sudo ufw reload +``` + +Or edit manually — the file should begin with: + +```text +*nat +:POSTROUTING ACCEPT [0:0] +-A POSTROUTING -s fd01:db8:1::/64 -o eth0 -p udp --sport 6969 \ + -j SNAT --to-source 2a01:4f8:1c0c:828e::1 +COMMIT + +# rules.before +# ...(existing content follows) +``` + ### Verification -After the Docker daemon restarts, confirm ufw's ip6tables rules are present in the -live kernel: +After applying all three steps, verify end-to-end: + +**1. ufw ip6tables rules still present:** ```bash sudo ip6tables -L ufw6-user-input -n ``` -Expected output includes: +Expected: ```text Chain ufw6-user-input (1 references) @@ -109,20 +201,42 @@ ACCEPT 6 -- ::/0 ::/0 tcp dpt:22 ACCEPT 17 -- ::/0 ::/0 udp dpt:6969 ``` -Note: Docker with `ip6tables: true` uses FORWARD chain rules (via `DOCKER-FORWARD` → -`DOCKER-BRIDGE`) rather than NAT/DNAT for IPv6 published ports. The ufw INPUT chain -rules for port 6969 are preserved after Docker chain rewrites, which is the fix. +**2. Container has an IPv6 address on the Docker bridge:** + +```bash +docker inspect tracker --format '{{range .NetworkSettings.Networks}}{{.GlobalIPv6Address}} {{end}}' +``` + +Expected: a non-empty address starting with `fd01:db8:1::` (the ULA subnet). + +**3. ip6tables DNAT rule exists for port 6969:** + +```bash +sudo ip6tables -t nat -L PREROUTING -n -v | grep 6969 +``` + +Expected: a DNAT rule redirecting to the container's IPv6 address. -### Simulating a nightly restart +**4. SNAT rule exists in POSTROUTING:** + +```bash +sudo ip6tables -t nat -L POSTROUTING -n -v | grep 6969 +``` + +Expected: + +```text +SNAT 17 -- fd01:db8:1::/64 ::/0 udp spt:6969 to:2a01:4f8:1c0c:828e::1 +``` -To verify the fix survives a container restart without waiting for the cron job: +**5. Simulate a nightly restart and verify all rules survive:** ```bash cd /opt/torrust docker compose stop tracker docker compose up -d tracker sudo ip6tables -L ufw6-user-input -n +sudo ip6tables -t nat -L POSTROUTING -n -v | grep 6969 ``` -The `ACCEPT udp dpt:6969` rule must still be present after `docker compose up`. -This was verified on 2026-03-09 and the rule survived. +Both the INPUT ACCEPT rule and the SNAT rule must still be present. diff --git a/docs/issues/ISSUE-2-udp-tracker-down-on-newtrackon.md b/docs/issues/ISSUE-2-udp-tracker-down-on-newtrackon.md index 96aecc5..17ada52 100644 --- a/docs/issues/ISSUE-2-udp-tracker-down-on-newtrackon.md +++ b/docs/issues/ISSUE-2-udp-tracker-down-on-newtrackon.md @@ -191,7 +191,7 @@ Document the root cause and fix in **Post-mortem**: [post-mortems/2026-03-09-udp-ipv6-docker.md](../post-mortems/2026-03-09-udp-ipv6-docker.md) -Root cause confirmed on 2026-03-09: +Root cause A confirmed on 2026-03-09: - Policy routing rules (Fix B) were **not** affected — `ip rule` and `ip route` entries were still present. @@ -199,26 +199,45 @@ Root cause confirmed on 2026-03-09: nightly Docker restart, even though ufw's on-disk rules were intact. Docker's chain rewrite wiped the live ip6tables rules and ufw never reloaded them. -**Chosen fix**: Enable `ip6tables: true` in `/etc/docker/daemon.json`. +**Fix A**: Enable `ip6tables: true` in `/etc/docker/daemon.json`. Verified working — +tracker accepted by newTrackon on 2026-03-09. -With this setting Docker manages ip6tables rules for all published ports automatically, -mirroring its IPv4 behaviour. The fix is systemic — it applies to all containers and all -ports without per-port configuration and survives any container restart. +Root cause B confirmed same day (~24 min after fix A confirmed): + +Even with `ip6tables: true`, native IPv6 UDP still failed. `docker-proxy` for IPv6 +(`-host-ip ::`) had an IPv4 container backend (`-container-ip 172.21.0.3`). Cross- +address-family UDP forwarding silently drops all native IPv6 packets. Docker only creates +ip6tables DNAT rules (which bypass docker-proxy) when the container has an IPv6 address — +which requires IPv6 to be enabled on the Docker bridge network. + +**Fix B**: + +1. Add `enable_ipv6: true` with subnet `fd01:db8:1::/64` to `proxy_network` in + `docker-compose.yml` so the container gets an IPv6 address and Docker creates DNAT + rules. +2. Add a SNAT rule to `/etc/ufw/before6.rules` to rewrite reply source addresses from + the Docker ULA subnet to the floating IPv6 (`2a01:4f8:1c0c:828e::1`), overriding + Docker's MASQUERADE. See [docs/docker-ipv6.md](../docker-ipv6.md) for full documentation and server application instructions. Files changed: -- `server/etc/docker/daemon.json` — added with `{"ip6tables": true}` +- `server/etc/docker/daemon.json` — added with `{"ip6tables": true}` (Fix A) +- `server/opt/torrust/docker-compose.yml` — `proxy_network` now has `enable_ipv6: true` + with subnet `fd01:db8:1::/64` (Fix B1) +- Manual server step: prepend SNAT nat section to `/etc/ufw/before6.rules` (Fix B2) ## Acceptance Criteria -- [x] Root cause confirmed — Docker's default `ip6tables: false` causes its chain rewrites - to wipe ufw's live ip6tables rules after every container restart -- [x] `udp://udp1.torrust-tracker-demo.com:6969/announce` accepted by newTrackon ✅ (confirmed 2026-03-09) -- [x] The fix survives a nightly Docker Compose restart (verified by simulating - with `docker compose stop tracker && docker compose up -d tracker` on 2026-03-09) +- [x] Root cause A confirmed — Docker's default `ip6tables: false` causes its chain + rewrites to wipe ufw's live ip6tables rules after every container restart +- [x] Root cause B confirmed — docker-proxy cross-AF UDP drops native IPv6 packets + silently when no container IPv6 address exists +- [x] `udp://udp1.torrust-tracker-demo.com:6969/announce` accepted by newTrackon ✅ + (confirmed 2026-03-09) +- [ ] The fix survives a nightly Docker Compose restart (to verify after next nightly restart) - [ ] Root cause and fix documented in the deployer repo's deployment journal - [ ] If a deployer-level fix is needed, a follow-up issue is opened in [torrust-tracker-deployer](https://github.com/torrust/torrust-tracker-deployer) diff --git a/docs/post-deployment.md b/docs/post-deployment.md index 5deb8c2..2941825 100644 --- a/docs/post-deployment.md +++ b/docs/post-deployment.md @@ -69,3 +69,48 @@ sudo ip6tables -L ufw6-user-input -n ``` See [docker-ipv6.md](docker-ipv6.md) for the full explanation and verification steps. + +--- + +## 3. Docker IPv6 — SNAT for UDP tracker replies via floating IPv6 (required only for IPv6 UDP) + +Enabling `ip6tables` in Docker (Step 2) and adding `enable_ipv6: true` to the Docker +Compose `proxy_network` allows Docker to create ip6tables DNAT rules for the container. +However, when the container replies, Docker's MASQUERADE rule rewrites the source address +to the server's **primary** IPv6 (`2a01:4f8:1c19:620b::1`) rather than the **floating** +IPv6 (`2a01:4f8:1c0c:828e::1`). Clients that probed the floating IP receive a reply from +the wrong source and treat it as a timeout. + +Fix: add a SNAT rule to `/etc/ufw/before6.rules` **before** the existing `*filter` section +so that replies from the Docker IPv6 bridge are rewritten to use the floating IP. + +Add the following block to the **very top** of `/etc/ufw/before6.rules`: + +```text +# NAT: rewrite source of Docker UDP tracker IPv6 replies to the floating IP +*nat +:POSTROUTING ACCEPT [0:0] +-A POSTROUTING -s fd01:db8:1::/64 -o eth0 -p udp --sport 6969 \ + -j SNAT --to-source 2a01:4f8:1c0c:828e::1 +COMMIT +``` + +Then reload ufw: + +```bash +sudo ufw reload +``` + +Verify the rule is loaded: + +```bash +sudo ip6tables -t nat -L POSTROUTING -n -v +# Must show: SNAT 17 -- fd01:db8:1::/64 ::/0 udp spt:6969 to: 2a01:4f8:1c0c:828e::1 +``` + +> `fd01:db8:1::/64` is the IPv6 subnet assigned to the `proxy_network` Docker bridge +> in [`server/opt/torrust/docker-compose.yml`](../server/opt/torrust/docker-compose.yml). +> If you change that subnet, update this SNAT rule to match. +> This rule must be in `before6.rules` so it is applied before Docker's MASQUERADE rule +> at ufw startup. Docker's MASQUERADE is added at container start; our SNAT fires first +> and takes precedence, so the correct source address is used. diff --git a/docs/post-mortems/2026-03-09-udp-ipv6-docker.md b/docs/post-mortems/2026-03-09-udp-ipv6-docker.md index 918d181..d55bdad 100644 --- a/docs/post-mortems/2026-03-09-udp-ipv6-docker.md +++ b/docs/post-mortems/2026-03-09-udp-ipv6-docker.md @@ -123,27 +123,316 @@ ACCEPT 17 -- ::/0 ::/0 udp dpt:6969 For IPv6, no equivalent DNAT rules are inserted → traffic must go through ufw INPUT → silently dropped. -**Root cause**: Docker's default `ip6tables: false` combined with Docker chain rewrites on -container restart — ufw's ip6tables rules are wiped and never restored. +**Root cause A**: Docker's default `ip6tables: false` combined with Docker chain rewrites +on container restart — ufw's ip6tables rules are wiped and never restored after the +nightly `docker compose stop/start` in the backup cron job. + +## Root Cause B (follow-up, same incident) + +After Root Cause A was fixed, the tracker still failed for IPv6 UDP. + +Docker's userland proxy (`docker-proxy`) spawns two processes for each published UDP port: + +- `-host-ip 0.0.0.0 … -container-ip 172.21.0.3` — IPv4 relay (same AF) ✅ +- `-host-ip :: … -container-ip 172.21.0.3` — IPv6 relay with IPv4 backend (cross-AF) ❌ + +docker-proxy cannot relay native IPv6 UDP packets to an IPv4 backend. It receives native +IPv6 packets on its `::` socket but silently drops them. No forwarding, no reply. + +Root Cause B was only reachable because Root Cause A had been masking it: when ip6tables +dropped all IPv6 UDP at the INPUT chain, packets never reached docker-proxy at all. + +**Root cause B**: Docker networks are IPv4-only by default. No container has an IPv6 +address, so Docker creates no ip6tables DNAT rules. All IPv6 UDP is handled by +docker-proxy's cross-AF path, which silently fails. --- ## Fix Decision -Options considered: +### Fix A — Enable `ip6tables: true` in `/etc/docker/daemon.json` + +Prevents Docker chain rewrites from wiping ufw's live ip6tables rules. Systemic fix — +applies to all containers and ports, no per-port changes needed. -1. **`ufw reload` in backup script** — targeted, but only covers the nightly cron restart. - Any other container restart (manual, OOM, redeploy) would re-expose the bug. +See [docs/docker-ipv6.md](../docker-ipv6.md) for the full configuration reference. -2. **Enable `ip6tables: true` in `/etc/docker/daemon.json`** — Docker manages ip6tables - rules the same way it does for IPv4. Published port 6969/udp gets proper FORWARD chain - rules that survive Docker chain rewrites. No changes to the backup script needed. - Requires one-time Docker daemon restart; containers restart automatically via - `restart: unless-stopped`. +### Fix B — Enable IPv6 on Docker's `proxy_network` + SNAT for reply source address -**Chosen fix: Option 2.** It is the systemic fix — Docker becomes responsible for its own -ip6tables rules for published ports, exactly mirroring the IPv4 behaviour. The fix applies -globally to all Docker-published ports on all containers without any per-port or -per-container configuration. +Two sub-steps: -See [docs/docker-ipv6.md](../docker-ipv6.md) for the full configuration reference. +**B1**: Add `enable_ipv6: true` with a fixed ULA subnet to `proxy_network` in +`docker-compose.yml`. This gives the tracker container an IPv6 address on the bridge +network, which causes Docker to create ip6tables DNAT rules for published ports — +bypassing docker-proxy for native IPv6 traffic entirely, exactly as iptables DNAT +already does for IPv4. + +```yaml +proxy_network: + driver: bridge + enable_ipv6: true + ipam: + config: + - subnet: "fd01:db8:1::/64" +``` + +**B2**: Docker's MASQUERADE rule rewrites container reply source addresses to the +server's primary IPv6 (`2a01:4f8:1c19:620b::1`). Clients probing the floating IPv6 +(`2a01:4f8:1c0c:828e::1`) receive a reply from the wrong source and treat it as a +timeout. Add a SNAT rule to `/etc/ufw/before6.rules` before the `*filter` section: + +```text +*nat +:POSTROUTING ACCEPT [0:0] +-A POSTROUTING -s fd01:db8:1::/64 -o eth0 -p udp --sport 6969 \ + -j SNAT --to-source 2a01:4f8:1c0c:828e::1 +COMMIT +``` + +This rule fires before Docker's MASQUERADE (it is added by ufw at boot, before Docker +starts) and SNATs replies to the floating IPv6. + +--- + +## Follow-up Investigation (same day, ~24 min after fix confirmed) + +newTrackon reported the tracker down again with `UDP timeout` on the IPv6 address. The +daemon.json fix was confirmed still in place and the container was running healthy. + +### Step 5 — Confirm docker-proxy is listening on IPv6 + +Command: + +```bash +sudo ss -6 -ulnp | grep 6969 +``` + +Output: + +```text +UNCONN 0 0 [::]:6969 [::]:* users:(("docker-proxy",pid=1343459,fd=7)) +``` + +**Finding**: docker-proxy is listening on `[::]` (wildcard), not on the floating IP +specifically. This was initially suspected to cause source-address asymmetry (replies +leaving from the primary IPv6 `2a01:4f8:1c19:620b::1` rather than the floating IP +`2a01:4f8:1c0c:828e::1`). + +### Step 6 — Check all global IPv6 addresses on eth0 + +Command: + +```bash +ip -6 addr show dev eth0 scope global +``` + +Output: + +```text +2: eth0: mtu 1500 qdisc fq_codel state UP group default qlen 1000 + inet6 2a01:4f8:1c0c:828e::1/64 scope global + inet6 2a01:4f8:1c0c:9aae::1/64 scope global + inet6 2a01:4f8:1c19:620b::1/64 scope global +``` + +**Finding**: Three global IPv6 addresses: the two floating IPs and the primary +Hetzner-assigned address `2a01:4f8:1c19:620b::1`. This identified a suspected +source-address asymmetry: docker-proxy on `[::]` would send replies with the kernel's +preferred source, which could be the primary IP rather than the floating IP. + +### Step 7 — Capture on-wire traffic (tcpdump) + +Command: + +```bash +sudo tcpdump -i eth0 -n udp port 6969 -v +``` + +Output (excerpt, ~10 seconds of capture): + +```text +18:46:06.152974 IP ... 176.124.202.52.34035 > 116.202.177.184.6969: UDP, length 55 +18:46:06.153389 IP ... 116.202.177.184.6969 > 176.124.202.52.34035: UDP, length 137 + ... +18:46:06.838585 IP6 ... 2409:8a5e:dc81:5bf0:e40:9a4a:2b89:5a64.31550 > 2a01:4f8:1c0c:828e::1.6969: ... + (no reply) +18:46:11.554713 IP6 ... 2408:8207:1921:9e20:ac83:6ed9:f11:1641.9346 > 2a01:4f8:1c0c:828e::1.6969: ... + (no reply) +``` + +**Finding**: IPv4 works correctly — requests arrive and replies leave immediately. IPv6 +requests arrive but **zero replies** leave eth0 for any IPv6 client. This rules out the +source-address asymmetry hypothesis (a misrouted reply would still appear in tcpdump). +The replies are not being generated or not reaching eth0 at all. + +### Step 8 — Full ip6tables filter table inspection + +Command: + +```bash +sudo ip6tables -L -n -v +``` + +Key findings from output: + +- `Chain ufw6-user-input`: `315K 22M ACCEPT 17 udp dpt:6969` — the firewall IS accepting + incoming IPv6 UDP 6969 packets. The INPUT chain is **not** the problem. +- `Chain FORWARD (policy DROP 0 packets, 0 bytes)` — zero packets through FORWARD. Docker's + FORWARD path for IPv6 is not being used at all. +- `Chain DOCKER-BRIDGE` and `Chain DOCKER-CT`: both empty (no rules, no traffic). + Docker set up no container routing rules for IPv6. +- `Chain OUTPUT (policy ACCEPT)` — output policy is ACCEPT and the ufw output chains have + no blocking rules for UDP. + +**Finding**: The firewall accepts IPv6 UDP 6969 incoming. docker-proxy on `[::]:6969` +should receive those packets. The OUTPUT chain does not block replies. Yet tcpdump +confirms no replies leave eth0. The failure point is between docker-proxy receiving +the packet and the reply being emitted on eth0. + +**Remaining hypotheses**: + +1. Docker's NAT table (`ip6tables -t nat`) has an interfering rule added with + `ip6tables: true`. +2. docker-proxy is not forwarding to the container (or the container is not responding to + the forwarded packet). + +**Next diagnostic commands**: + +```bash +sudo ip6tables -t nat -L -n -v +docker logs tracker --tail 50 2>&1 +``` + +### Step 9 — Inspect ip6tables NAT table + +Command: + +```bash +sudo ip6tables -t nat -L -n -v +``` + +Output (abridged): + +```text +Chain PREROUTING (policy ACCEPT 2254K packets, 171M bytes) + 21090 1619K DOCKER 0 -- * * ::/0 ::/0 ADDRTYPE match dst-type LOCAL + +Chain DOCKER (2 references) + pkts bytes target prot opt in out source destination +``` + +**Finding**: The `DOCKER` chain in the ip6tables NAT table is **empty**. Docker with +`ip6tables: true` has NOT created any IPv6 DNAT rules for published ports. + +For IPv4, Docker creates iptables DNAT rules for every published port (e.g. +`-j DNAT --to-destination 172.17.0.x:6969`), routing incoming packets directly to the +container via the kernel. For IPv6, no equivalent DNAT rules exist. The IPv6 DOCKER chain +(in the nat table) is empty because none of the tracker's Docker networks have IPv6 +enabled, so containers have no IPv6 addresses. Docker cannot create DNAT rules pointing to +container IPv6 addresses that do not exist. + +Without ip6tables DNAT, the **only** IPv6 forwarding path is docker-proxy. + +### Step 10 — Inspect container logs + +Command: + +```bash +docker logs tracker --tail 50 2>&1 +``` + +The initial `--tail 50` only showed HTTP entries because those 50 lines happened to be +HTTP requests. Filtering with `docker logs tracker | grep ":6969"` revealed a different +picture: + +```text +... ERROR ... UDP TRACKER: response error + client_socket_addr=[::ffff:213.155.193.247]:42881 + server_socket_addr=[::]:6969 ... +... ERROR ... UDP TRACKER: response error + client_socket_addr=[::ffff:95.25.50.20]:50711 + server_socket_addr=[::]:6969 ... +(many more similar lines, all with [::ffff:x.x.x.x] addresses) +``` + +**Finding**: The tracker IS receiving UDP traffic on port 6969. However, every single +`client_socket_addr` is of the form `[::ffff:x.x.x.x]` — IPv4-mapped IPv6 addresses. +This is how the Linux kernel represents IPv4 connections received on a dual-stack UDP +socket (`[::]:6969`). **Not one entry contains a native IPv6 address** such as +`[2a01:...]`, `[2409:...]`, etc. + +The errors are all `"Invalid action"` — scanner/bot traffic sending malformed packets, +unrelated to the newTrackon failure. + +### Step 11 — Review Docker network configuration and docker-proxy behaviour + +Command: + +```bash +docker logs tracker | grep ":6969" | grep -v "ffff" | head +``` + +(No output — zero native IPv6 UDP entries.) + +Reviewing `server/opt/torrust/docker-compose.yml`: + +```yaml +networks: + database_network: + driver: bridge + metrics_network: + driver: bridge + proxy_network: + driver: bridge + visualization_network: + driver: bridge +``` + +All four Docker networks are IPv4-only bridge networks. The tracker container's bridge +addresses are in the `172.x.x.x` range. + +**Finding**: Docker's userland UDP proxy (`docker-proxy`) listens on `[::]:6969` (a +dual-stack IPv6 socket that also accepts IPv4-mapped connections). It relays IPv4 traffic +(as `::ffff:x.x.x.x`) to the container correctly. However, it **silently drops native +IPv6 UDP packets** — it receives them on the frontend socket but cannot relay them to the +IPv4 container backend. No error is emitted and no reply is generated. + +This is a known limitation of Docker's userland UDP proxy: it does not support relaying +native IPv6 clients to an IPv4-only container backend. + +### Step 12 — Confirm docker-proxy backend address (smoking gun) + +Command: + +```bash +ps aux | grep docker-proxy | grep 6969 +``` + +Output: + +```text +root 1343452 ... /usr/bin/docker-proxy -proto udp -host-ip 0.0.0.0 -host-port 6969 \ + -container-ip 172.21.0.3 -container-port 6969 -use-listen-fd +root 1343459 ... /usr/bin/docker-proxy -proto udp -host-ip :: -host-port 6969 \ + -container-ip 172.21.0.3 -container-port 6969 -use-listen-fd +``` + +**Finding**: Two docker-proxy processes for port 6969: + +1. IPv4 proxy: `-host-ip 0.0.0.0` → `-container-ip 172.21.0.3` — same address family ✅ +2. IPv6 proxy: `-host-ip ::` → **`-container-ip 172.21.0.3`** — cross-address-family ❌ + +The IPv6 proxy has a native IPv6 frontend socket (`::`) but an IPv4 container backend +(`172.21.0.3`). This is cross-address-family UDP forwarding. docker-proxy cannot relay +native IPv6 UDP packets to an IPv4 backend — it silently drops them without reply. + +This is the definitive root cause. The fix requires giving the container an IPv6 address +so that Docker creates ip6tables DNAT rules (bypassing docker-proxy entirely, exactly as +it works for IPv4 today via iptables DNAT). + +This explains all observed behaviour: + +- ip6tables INPUT accepts 315K IPv6 UDP 6969 packets ✅ (they reach docker-proxy) +- docker logs show UDP traffic only with `::ffff:` IPv4-mapped addresses ✅ (IPv4 works) +- docker logs show zero native IPv6 UDP entries ✅ (docker-proxy cross-AF drop) +- tcpdump shows zero IPv6 replies ✅ (no reply is ever generated) diff --git a/project-words.txt b/project-words.txt index c745096..38220ee 100644 --- a/project-words.txt +++ b/project-words.txt @@ -1,3 +1,4 @@ +ADDRTYPE agentskills augmentedcode behaviour @@ -20,6 +21,7 @@ healthcheck leecher Leechers logfile +misrouted mkpath mysqladmin netnsid @@ -30,13 +32,18 @@ noall noprefixroute noqueue oneshot +pkts post-mortems +POSTROUTING +PREROUTING prometheus qdisc qlen QUIC repomix rgba +SNAT +SNATs sourceable tcpdump tera @@ -44,4 +51,7 @@ timepicker tmpfs torrust Torrust +ulnp +UNCONN +userland veth diff --git a/server/README.md b/server/README.md index 53dac98..2ce7855 100644 --- a/server/README.md +++ b/server/README.md @@ -55,16 +55,30 @@ The following are excluded — they contain runtime data, large binaries, or add ### Docker IPv6 (`etc/docker/daemon.json`) -`ip6tables: true` is required for IPv6 UDP traffic to reach Docker containers. Without it -Docker does not insert ip6tables rules for published ports and its chain rewrites wipe ufw's -live ip6tables rules after every container restart, silently dropping all IPv6 UDP traffic. +`ip6tables: true` is required to prevent Docker chain rewrites from wiping ufw's live +ip6tables rules after every container restart. Without it, all IPv6 UDP traffic is +silently dropped after each restart. -See [docs/docker-ipv6.md](../docs/docker-ipv6.md) for the full explanation. +### Docker IPv6 \u2014 `proxy_network` (`opt/torrust/docker-compose.yml`) -### Floating IP routing (`etc/netplan/60-floating-ip.yaml`) +`enable_ipv6: true` with subnet `fd01:db8:1::/64` on `proxy_network` gives the tracker +container an IPv6 address, which causes Docker to create ip6tables DNAT rules that bypass +docker-proxy for native IPv6 traffic. Without this, docker-proxy silently drops all native +IPv6 UDP packets because it cannot relay them to an IPv4-only container backend. + +### Docker IPv6 \u2014 SNAT for floating IPv6 (manual server step) + +A `*nat POSTROUTING SNAT` rule must be prepended to `/etc/ufw/before6.rules` on the +server to rewrite reply source addresses from the Docker ULA bridge subnet +(`fd01:db8:1::/64`) to the floating IPv6 (`2a01:4f8:1c0c:828e::1`). Without it, Docker's +MASQUERADE rewrites replies to the primary server IPv6 and clients probing the floating IP +time out. -Each Hetzner floating IP requires a policy routing table entry so that replies leave via the -same floating IP rather than the primary server IP (asymmetric routing fix). Adding a new -floating IP requires updating this file and running `sudo netplan apply`. +See [docs/docker-ipv6.md](../docs/docker-ipv6.md) for the full explanation and all +application steps. + +### Floating IP routing (`etc/netplan/60-floating-ip.yaml`) -See [docs/docker-ipv6.md](../docs/docker-ipv6.md) for details. +Each Hetzner floating IP requires a policy routing table entry so that replies leave via +the same floating IP rather than the primary server IP. Adding a new floating IP requires +updating this file and running `sudo netplan apply`. diff --git a/server/opt/torrust/docker-compose.yml b/server/opt/torrust/docker-compose.yml index f9208c4..bb54f8e 100644 --- a/server/opt/torrust/docker-compose.yml +++ b/server/opt/torrust/docker-compose.yml @@ -215,8 +215,17 @@ networks: metrics_network: driver: bridge # TLS termination: Caddy ↔ backend services + # IPv6 is enabled so Docker creates ip6tables DNAT rules for published ports, + # giving the tracker container an IPv6 address. This bypasses docker-proxy for + # native IPv6 traffic. Without this, docker-proxy silently drops native IPv6 + # UDP packets because it cannot relay them to an IPv4-only container backend. + # See: docs/docker-ipv6.md proxy_network: driver: bridge + enable_ipv6: true + ipam: + config: + - subnet: "fd01:db8:1::/64" # Dashboard queries: Prometheus ↔ Grafana visualization_network: driver: bridge From 24b17aa519f291f4efafbaa96a4137519e432e48 Mon Sep 17 00:00:00 2001 From: Jose Celano Date: Mon, 9 Mar 2026 19:41:52 +0000 Subject: [PATCH 5/7] docs: add before6.rules to server config snapshot Add the full /etc/ufw/before6.rules file to server/etc/ufw/ so the repo captures the complete server UFW configuration, including the custom *nat POSTROUTING SNAT rule added as part of Fix B. Also exclude the three UFW rules files from cspell (verbatim system files with non-prose content like icmpv6 type names) and update docs/infrastructure.md and docs/post-deployment.md to reference the new file. --- cspell.json | 5 +- docs/infrastructure.md | 2 + docs/post-deployment.md | 2 + server/etc/ufw/before6.rules | 148 +++++++++++++++++++++++++++++++++++ 4 files changed, 156 insertions(+), 1 deletion(-) create mode 100644 server/etc/ufw/before6.rules diff --git a/cspell.json b/cspell.json index d33f3f4..2dfc9e2 100644 --- a/cspell.json +++ b/cspell.json @@ -13,7 +13,10 @@ ], "ignorePaths": [ "/project-words.txt", - "repomix-output.xml" + "repomix-output.xml", + "server/etc/ufw/before6.rules", + "server/etc/ufw/user.rules", + "server/etc/ufw/user6.rules" ], "files": [ "**/*", diff --git a/docs/infrastructure.md b/docs/infrastructure.md index e691849..dac33eb 100644 --- a/docs/infrastructure.md +++ b/docs/infrastructure.md @@ -72,6 +72,8 @@ Defined in [`server/etc/netplan/60-floating-ip.yaml`](../server/etc/netplan/60-f Managed by UFW. Rules are in [`server/etc/ufw/user.rules`](../server/etc/ufw/user.rules) (IPv4) and [`server/etc/ufw/user6.rules`](../server/etc/ufw/user6.rules) (IPv6). +Custom pre-filter rules (including the SNAT for IPv6 UDP) are in +[`server/etc/ufw/before6.rules`](../server/etc/ufw/before6.rules). Port 443 (HTTPS) is not in the UFW user rules — it is exposed directly by the Caddy container via Docker's `iptables` integration. diff --git a/docs/post-deployment.md b/docs/post-deployment.md index 2941825..93416bf 100644 --- a/docs/post-deployment.md +++ b/docs/post-deployment.md @@ -84,6 +84,8 @@ the wrong source and treat it as a timeout. Fix: add a SNAT rule to `/etc/ufw/before6.rules` **before** the existing `*filter` section so that replies from the Docker IPv6 bridge are rewritten to use the floating IP. +**Configuration file**: [`server/etc/ufw/before6.rules`](../server/etc/ufw/before6.rules) + Add the following block to the **very top** of `/etc/ufw/before6.rules`: ```text diff --git a/server/etc/ufw/before6.rules b/server/etc/ufw/before6.rules new file mode 100644 index 0000000..37209ec --- /dev/null +++ b/server/etc/ufw/before6.rules @@ -0,0 +1,148 @@ +# +# rules.before +# +# Rules that should be run before the ufw command line added rules. Custom +# rules should be added to one of these chains: +# ufw6-before-input +# ufw6-before-output +# ufw6-before-forward +# + +# NAT: rewrite source of Docker UDP tracker IPv6 replies to the floating IP +*nat +:POSTROUTING ACCEPT [0:0] +-A POSTROUTING -s fd01:db8:1::/64 -o eth0 -p udp --sport 6969 -j SNAT --to-source 2a01:4f8:1c0c:828e::1 +COMMIT + +# Don't delete these required lines, otherwise there will be errors +*filter +:ufw6-before-input - [0:0] +:ufw6-before-output - [0:0] +:ufw6-before-forward - [0:0] +# End required lines + + +# allow all on loopback +-A ufw6-before-input -i lo -j ACCEPT +-A ufw6-before-output -o lo -j ACCEPT + +# drop packets with RH0 headers +-A ufw6-before-input -m rt --rt-type 0 -j DROP +-A ufw6-before-forward -m rt --rt-type 0 -j DROP +-A ufw6-before-output -m rt --rt-type 0 -j DROP + +# quickly process packets for which we already have a connection +-A ufw6-before-input -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A ufw6-before-output -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A ufw6-before-forward -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + +# multicast ping replies are part of the ok icmp codes for INPUT (rfc4890, +# 4.4.1 and 4.4.2), but don't have an associated connection and are otherwise +# be marked INVALID, so allow here instead. +-A ufw6-before-input -p icmpv6 --icmpv6-type echo-reply -j ACCEPT + +# drop INVALID packets (logs these in loglevel medium and higher) +-A ufw6-before-input -m conntrack --ctstate INVALID -j ufw6-logging-deny +-A ufw6-before-input -m conntrack --ctstate INVALID -j DROP + +# ok icmp codes for INPUT (rfc4890, 4.4.1 and 4.4.2) +-A ufw6-before-input -p icmpv6 --icmpv6-type destination-unreachable -j ACCEPT +-A ufw6-before-input -p icmpv6 --icmpv6-type packet-too-big -j ACCEPT +# codes 0 and 1 +-A ufw6-before-input -p icmpv6 --icmpv6-type time-exceeded -j ACCEPT +# codes 0-2 (echo-reply needs to be before INVALID, see above) +-A ufw6-before-input -p icmpv6 --icmpv6-type parameter-problem -j ACCEPT +-A ufw6-before-input -p icmpv6 --icmpv6-type echo-request -j ACCEPT +-A ufw6-before-input -p icmpv6 --icmpv6-type router-solicitation -m hl --hl-eq 255 -j ACCEPT +-A ufw6-before-input -p icmpv6 --icmpv6-type router-advertisement -m hl --hl-eq 255 -j ACCEPT +-A ufw6-before-input -p icmpv6 --icmpv6-type neighbor-solicitation -m hl --hl-eq 255 -j ACCEPT +-A ufw6-before-input -p icmpv6 --icmpv6-type neighbor-advertisement -m hl --hl-eq 255 -j ACCEPT +# IND solicitation +-A ufw6-before-input -p icmpv6 --icmpv6-type 141 -m hl --hl-eq 255 -j ACCEPT +# IND advertisement +-A ufw6-before-input -p icmpv6 --icmpv6-type 142 -m hl --hl-eq 255 -j ACCEPT +# MLD query +-A ufw6-before-input -p icmpv6 --icmpv6-type 130 -s fe80::/10 -j ACCEPT +# MLD report +-A ufw6-before-input -p icmpv6 --icmpv6-type 131 -s fe80::/10 -j ACCEPT +# MLD done +-A ufw6-before-input -p icmpv6 --icmpv6-type 132 -s fe80::/10 -j ACCEPT +# MLD report v2 +-A ufw6-before-input -p icmpv6 --icmpv6-type 143 -s fe80::/10 -j ACCEPT +# SEND certificate path solicitation +-A ufw6-before-input -p icmpv6 --icmpv6-type 148 -m hl --hl-eq 255 -j ACCEPT +# SEND certificate path advertisement +-A ufw6-before-input -p icmpv6 --icmpv6-type 149 -m hl --hl-eq 255 -j ACCEPT +# MR advertisement +-A ufw6-before-input -p icmpv6 --icmpv6-type 151 -s fe80::/10 -m hl --hl-eq 1 -j ACCEPT +# MR solicitation +-A ufw6-before-input -p icmpv6 --icmpv6-type 152 -s fe80::/10 -m hl --hl-eq 1 -j ACCEPT +# MR termination +-A ufw6-before-input -p icmpv6 --icmpv6-type 153 -s fe80::/10 -m hl --hl-eq 1 -j ACCEPT + +# ok icmp codes for OUTPUT (rfc4890, 4.4.1 and 4.4.2) +-A ufw6-before-output -p icmpv6 --icmpv6-type destination-unreachable -j ACCEPT +-A ufw6-before-output -p icmpv6 --icmpv6-type packet-too-big -j ACCEPT +# codes 0 and 1 +-A ufw6-before-output -p icmpv6 --icmpv6-type time-exceeded -j ACCEPT +# codes 0-2 +-A ufw6-before-output -p icmpv6 --icmpv6-type parameter-problem -j ACCEPT +-A ufw6-before-output -p icmpv6 --icmpv6-type echo-request -j ACCEPT +-A ufw6-before-output -p icmpv6 --icmpv6-type echo-reply -j ACCEPT +-A ufw6-before-output -p icmpv6 --icmpv6-type router-solicitation -m hl --hl-eq 255 -j ACCEPT +-A ufw6-before-output -p icmpv6 --icmpv6-type neighbor-advertisement -m hl --hl-eq 255 -j ACCEPT +-A ufw6-before-output -p icmpv6 --icmpv6-type neighbor-solicitation -m hl --hl-eq 255 -j ACCEPT +-A ufw6-before-output -p icmpv6 --icmpv6-type router-advertisement -m hl --hl-eq 255 -j ACCEPT +# IND solicitation +-A ufw6-before-output -p icmpv6 --icmpv6-type 141 -m hl --hl-eq 255 -j ACCEPT +# IND advertisement +-A ufw6-before-output -p icmpv6 --icmpv6-type 142 -m hl --hl-eq 255 -j ACCEPT +# MLD query +-A ufw6-before-output -p icmpv6 --icmpv6-type 130 -s fe80::/10 -j ACCEPT +# MLD report +-A ufw6-before-output -p icmpv6 --icmpv6-type 131 -s fe80::/10 -j ACCEPT +# MLD done +-A ufw6-before-output -p icmpv6 --icmpv6-type 132 -s fe80::/10 -j ACCEPT +# MLD report v2 +-A ufw6-before-output -p icmpv6 --icmpv6-type 143 -s fe80::/10 -j ACCEPT +# SEND certificate path solicitation +-A ufw6-before-output -p icmpv6 --icmpv6-type 148 -m hl --hl-eq 255 -j ACCEPT +# SEND certificate path advertisement +-A ufw6-before-output -p icmpv6 --icmpv6-type 149 -m hl --hl-eq 255 -j ACCEPT +# MR advertisement +-A ufw6-before-output -p icmpv6 --icmpv6-type 151 -s fe80::/10 -m hl --hl-eq 1 -j ACCEPT +# MR solicitation +-A ufw6-before-output -p icmpv6 --icmpv6-type 152 -s fe80::/10 -m hl --hl-eq 1 -j ACCEPT +# MR termination +-A ufw6-before-output -p icmpv6 --icmpv6-type 153 -s fe80::/10 -m hl --hl-eq 1 -j ACCEPT + +# ok icmp codes for FORWARD (rfc4890, 4.3.1) +-A ufw6-before-forward -p icmpv6 --icmpv6-type destination-unreachable -j ACCEPT +-A ufw6-before-forward -p icmpv6 --icmpv6-type packet-too-big -j ACCEPT +# codes 0 and 1 +-A ufw6-before-forward -p icmpv6 --icmpv6-type time-exceeded -j ACCEPT +# codes 0-2 +-A ufw6-before-forward -p icmpv6 --icmpv6-type parameter-problem -j ACCEPT +-A ufw6-before-forward -p icmpv6 --icmpv6-type echo-request -j ACCEPT +-A ufw6-before-forward -p icmpv6 --icmpv6-type echo-reply -j ACCEPT +# ok icmp codes for FORWARD (rfc4890, 4.3.2) +# Home Agent Address Discovery Reques +-A ufw6-before-input -p icmpv6 --icmpv6-type 144 -j ACCEPT +# Home Agent Address Discovery Reply +-A ufw6-before-input -p icmpv6 --icmpv6-type 145 -j ACCEPT +# Mobile Prefix Solicitation +-A ufw6-before-input -p icmpv6 --icmpv6-type 146 -j ACCEPT +# Mobile Prefix Advertisement +-A ufw6-before-input -p icmpv6 --icmpv6-type 147 -j ACCEPT + +# allow dhcp client to work +-A ufw6-before-input -p udp -s fe80::/10 --sport 547 -d fe80::/10 --dport 546 -j ACCEPT + +# allow MULTICAST mDNS for service discovery +-A ufw6-before-input -p udp -d ff02::fb --dport 5353 -j ACCEPT + +# allow MULTICAST UPnP for service discovery +-A ufw6-before-input -p udp -d ff02::f --dport 1900 -j ACCEPT + +# don't delete the 'COMMIT' line or these rules won't be processed +COMMIT From f326ae6cd5c96d8bbc866b1d4b71c2bb8b7c1dcc Mon Sep 17 00:00:00 2001 From: Jose Celano Date: Mon, 9 Mar 2026 19:48:59 +0000 Subject: [PATCH 6/7] docs: add packet-flow diagram to docker-ipv6.md --- docs/docker-ipv6.md | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/docs/docker-ipv6.md b/docs/docker-ipv6.md index 04d18b0..48367fa 100644 --- a/docs/docker-ipv6.md +++ b/docs/docker-ipv6.md @@ -38,6 +38,43 @@ docker-proxy receives native IPv6 UDP on its `::` socket but cannot forward to a container backend. Packets are accepted by ip6tables, reach docker-proxy, and are then silently dropped with no reply and no error log. +## Packet Flow + +```text + SERVER (eth0) + ┌──────────────────────────────────────────────────────────┐ + │ │ + CLIENT │ ip6tables nat PREROUTING │ + 2409:8a5e::1 │ ┌─────────────────────────────────┐ │ + │ │ │ DOCKER chain │ │ + │ UDP :6969 │ │ DNAT → fd01:db8:1::3:6969 │ │ + ▼ │ └────────────────┬────────────────┘ │ + 2a01:4f8:1c0c:828e::1 │ │ │ + (floating IPv6) │ ▼ │ + │ │ Docker bridge (fd01:db8:1::/64) │ + │ │ ┌──────────────────────────────┐ │ + │ │ │ TRACKER CONTAINER │ │ + │ │ │ fd01:db8:1::3 │ │ + │ │ │ │ │ + │ │ │ sends reply from │ │ + │ │ │ fd01:db8:1::3 │ │ + │ │ └──────────────┬───────────────┘ │ + │ │ │ │ + │ │ ip6tables nat POSTROUTING │ + │ │ ┌──────────────────────────────────────────────┐ │ + │ │ │ our SNAT rule (before6.rules) │ │ + │ │ │ fd01:db8:1::/64 → 2a01:4f8:1c0c:828e::1 │ │ + │ │ │ │ │ + │ │ │ (without this, Docker MASQUERADE would use │ │ + │ │ │ primary IPv6 2a01:4f8:1c19:620b::1 instead) │ │ + │ │ └──────────────────────────────────────────────┘ │ + │ │ │ │ + └───────────────┼─────────────────┘ │ + reply from │ source = 2a01:4f8:1c0c:828e::1 ✅ │ + correct IP │ │ + └──────────────────────────────────────────────────────────┘ +``` + ## Fix Two configuration changes are required. From 3b331cd7f7626931f63e018fcd372d171639a2b3 Mon Sep 17 00:00:00 2001 From: Jose Celano Date: Mon, 9 Mar 2026 20:56:33 +0000 Subject: [PATCH 7/7] docs: clarify dual-stack sockets and IPv4-mapped IPv6 addresses --- docs/docker-ipv6.md | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/docs/docker-ipv6.md b/docs/docker-ipv6.md index 48367fa..c6accaf 100644 --- a/docs/docker-ipv6.md +++ b/docs/docker-ipv6.md @@ -164,6 +164,46 @@ For IPv6, the floating-IP reply source is handled by the SNAT rule in Fix 2b abo See [server/etc/netplan/60-floating-ip.yaml](../server/etc/netplan/60-floating-ip.yaml). +## Note on dual-stack sockets and IPv4-mapped IPv6 addresses + +### What are IPv4-mapped addresses? + +When a process opens an IPv6 socket and binds to `::` (all interfaces), the Linux kernel +has a feature called **dual-stack sockets** (controlled by the `IPV6_V6ONLY` flag). By +default on Linux, `IPV6_V6ONLY` is `0`, meaning an IPv6 socket also accepts IPv4 +connections. When an IPv4 packet arrives, the kernel transparently represents its source +address as `::ffff:x.x.x.x` before handing it to the socket. + +This is entirely a kernel-level mechanism — Docker does nothing special here. It emerges +whenever any process uses a dual-stack socket. + +### What you saw in the logs before the fix + +Before Fix 2, docker-proxy was the only path for incoming traffic. It listened on a `::` dual-stack socket, so IPv4 clients were transparently mapped to `::ffff:x.x.x.x` by the +kernel before docker-proxy forwarded them. The tracker logs therefore showed entries like +`[::ffff:116.202.177.184]` for what were actually plain IPv4 clients. + +After Fix 2, docker-proxy is bypassed entirely. iptables DNAT handles IPv4 and ip6tables +DNAT handles IPv6 — each family takes its own path — so IPv4 clients now appear as plain +IPv4 addresses in the tracker logs. + +### When would you actually need dual-stack? + +IPv4-mapped addresses matter when the **server has no public IPv4 address at all** +(IPv6-only server). In that scenario: + +- The server has one or more IPv6 addresses but no IPv4 address on `eth0`. +- IPv4 clients reach the server through a hosting-provider NAT64/DNS64 gateway that + translates their IPv4 packets into IPv6 before delivery. +- The tracker container receives all traffic as IPv6, with IPv4 client addresses + represented as `::ffff:x.x.x.x`. + +In such a setup there is only one IP family to manage, so the SNAT complexity described +in this document does not apply. The motivation for our multi-IP setup is different: +we use **two separate floating IPs** (one for HTTP, one for UDP) so that both tracker +endpoints can be listed independently on [newTrackon](https://newtrackon.com/), which +tracks one tracker per IP address. + ## Applying on the server ### Fix 1 — Docker daemon (one-time setup)