diff --git a/.github/skills/usage/operations/debug-command-failure/skill.md b/.github/skills/usage/operations/debug-command-failure/skill.md new file mode 100644 index 000000000..b2a8766d4 --- /dev/null +++ b/.github/skills/usage/operations/debug-command-failure/skill.md @@ -0,0 +1,231 @@ +--- +name: debug-command-failure +description: Guide for debugging and investigating deployer command failures. Covers reading error output, locating trace files, inspecting environment state, examining build artifacts, and running manual verification steps. Use when any deployer command (provision, configure, release, run, etc.) fails. Triggers on "command failed", "debug failure", "investigate error", "why did it fail", "trace", "deployer error", or "command error". +metadata: + author: torrust + version: "1.0" +--- + +# Debugging Deployer Command Failures + +This skill walks through collecting and interpreting diagnostic information when any deployer +command fails. + +## Investigation Layers (in order) + +```text +1. Console error output → immediate symptom + tip +2. Environment state → data/{env}/environment.json +3. Trace log → data/{env}/traces/{timestamp}-{command}.log +4. Build artifacts → build/{env}/ +5. Manual verification → SSH, curl, provider console +``` + +Work top-to-bottom. Each layer provides richer context than the previous. + +--- + +## Layer 1 — Console Error Output + +A failed command prints: + +```text +❌ command failed: +Tip: +Tip: Check logs and try running with --log-output file-and-stderr for more details +``` + +Note the **error summary** and the **tip** lines. The summary often names the failed step and the +kind of error. + +--- + +## Layer 2 — Environment State + +After any command failure, the deployer writes machine-readable state: + +```text +data/{env-name}/environment.json +``` + +Key fields to inspect: + +```json +{ + "state": { + "context": { + "failed_step": "WaitSshConnectivity", + "error_kind": "NetworkConnectivity", + "error_summary": "SSH connectivity failed: ...", + "failed_at": "2026-03-03T15:33:32Z", + "execution_started_at": "2026-03-03T15:30:00Z", + "execution_duration": { "secs": 212, "nanos": 885591647 }, + "trace_id": "bcba0ee9-b2cf-4302-be0e-5ed04c665141", + "trace_file_path": "./data/{env-name}/traces/20260303-153332-provision.log" + } + } +} +``` + +| Field | What it tells you | +| -------------------- | ---------------------------------------------------------- | +| `failed_step` | Which internal step failed (maps to deployer source code) | +| `error_kind` | Category: `NetworkConnectivity`, `TemplateRendering`, etc. | +| `error_summary` | Human-readable description of the error | +| `execution_duration` | How long the command ran before failing | +| `trace_file_path` | Exact path to the full trace log | + +```bash +# Quick inspection +cat data/{env-name}/environment.json | python3 -m json.tool +# or +jq '.state.context' data/{env-name}/environment.json +``` + +--- + +## Layer 3 — Trace Log + +The trace log records every step, sub-step, and decision the deployer made: + +```text +data/{env-name}/traces/{YYYYMMDD-HHMMSS}-{command}.log +``` + +The exact path is in `environment.json → state.context.trace_file_path`. + +```bash +# Read the full log +cat data/{env-name}/traces/20260303-153332-provision.log + +# Focus on errors and warnings +grep -E 'ERROR|WARN|failed|error' data/{env-name}/traces/20260303-153332-provision.log + +# Show the last 50 lines (where failures are usually recorded) +tail -50 data/{env-name}/traces/20260303-153332-provision.log +``` + +The trace contains structured log lines with timestamps, log levels, and context fields. Look for +`ERROR` lines and the step names that precede them. + +--- + +## Layer 4 — Build Artifacts + +The `build/` directory holds rendered templates and intermediate files generated before +infrastructure is touched: + +```text +build/{env-name}/ +├── tofu/ +│ └── hetzner/ (or lxd/) +│ ├── main.tf # OpenTofu infrastructure definition +│ ├── cloud-init.yml # cloud-init script run on first boot +│ └── *.tf # Other Terraform/OpenTofu files +└── ansible/ + ├── inventory.ini # Ansible inventory + └── playbooks/ # Ansible playbooks +``` + +Common inspections: + +```bash +# Verify SSH public key was correctly injected into cloud-init +grep -A3 'ssh_authorized_keys' build/{env-name}/tofu/hetzner/cloud-init.yml + +# Compare with the actual public key +cat ~/.ssh/torrust_tracker_deployer_ed25519.pub + +# Inspect the infrastructure definition +cat build/{env-name}/tofu/hetzner/main.tf +``` + +**Why this matters**: Build artifacts are generated from your config file without touching the +cloud provider. If the artifact is wrong, the root cause is in the environment config or a +template bug — not in the network or provider. + +--- + +## Layer 5 — Manual Verification + +When the deployer fails but the cloud resource appears to be up, verify the resource directly. + +### SSH connectivity + +```bash +# Test SSH manually with verbose output (-v shows handshake details) +ssh -v -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@{server-ip} "whoami && cloud-init status" +``` + +A successful response looks like: + +```text +torrust +status: done +``` + +If `cloud-init status` returns `status: running`, cloud-init is still executing — wait and retry. + +### Cloud-init timing + +```bash +# Check cloud-init completion and timing +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@{server-ip} \ + "cloud-init status --long && sudo journalctl -u ssh --since '5 minutes ago' | tail -20" +``` + +**Note**: If the clock timestamp shows `1970-01-01`, the system clock was not yet NTP-synced when +cloud-init completed — this is normal and does not indicate a failure. + +### Port availability + +```bash +# Check if SSH port is open (times out quickly if no service is listening) +nc -zv {server-ip} 22 + +# Check if HTTP tracker port is open +nc -zv {server-ip} 6969 +``` + +--- + +## Common Error Patterns + +| `failed_step` | `error_kind` | Likely Cause | +| ------------------------- | --------------------- | -------------------------------------------------------------------- | +| `RenderOpenTofuTemplates` | `TemplateRendering` | SSH key path not found — check container vs host path in config | +| `WaitSshConnectivity` | `NetworkConnectivity` | Server SSH not ready within timeout — server may need more boot time | +| `RunAnsiblePlaybook` | `Ansible` | SSH key rejected or unreachable — verify `~/.ssh/known_hosts` | +| `CreateServer` | `ProviderApi` | API token invalid or quota exceeded — check Hetzner console | + +--- + +## After Investigation + +Once the root cause is identified, the recovery path depends on how far the command progressed: + +- **Failed before any cloud resources were created** (e.g., `TemplateRendering`): fix the config, + `purge --force`, `create environment`, retry command. + +- **Failed after cloud resources were created** (e.g., `WaitSshConnectivity`): the deployer state + is `ProvisionFailed` or `ConfigureFailed`. Resources exist in the cloud. Must `destroy` to clean + up both cloud resources and local state, then `create environment` and retry. + +```bash +# Destroy cloud resources + local state +docker run --rm \ + -v $(pwd)/data:/var/lib/torrust/deployer/data \ + -v $(pwd)/build:/var/lib/torrust/deployer/build \ + -v $(pwd)/envs:/var/lib/torrust/deployer/envs \ + -v ~/.ssh:/home/deployer/.ssh:ro \ + torrust/tracker-deployer:latest \ + destroy {env-name} + +# Recreate local environment +docker run --rm \ + -v $(pwd)/data:/var/lib/torrust/deployer/data \ + -v $(pwd)/build:/var/lib/torrust/deployer/build \ + -v $(pwd)/envs:/var/lib/torrust/deployer/envs \ + torrust/tracker-deployer:latest \ + create environment --env-file envs/{env-name}.json +``` diff --git a/AGENTS.md b/AGENTS.md index f0b803708..f1d16c84d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -179,6 +179,7 @@ Available skills: | Creating issues | `.github/skills/dev/planning/create-issue/skill.md` | | Creating new skills | `.github/skills/add-new-skill/skill.md` | | Creating refactor plans | `.github/skills/dev/planning/create-refactor-plan/skill.md` | +| Debugging command failures | `.github/skills/usage/operations/debug-command-failure/skill.md` | | Debugging test errors | `.github/skills/dev/testing/debug-test-errors/skill.md` | | Handling errors in code | `.github/skills/dev/rust-code-quality/handle-errors-in-code/skill.md` | | Handling secrets | `.github/skills/dev/rust-code-quality/handle-secrets/skill.md` | diff --git a/docs/deployments/hetzner-demo-tracker/README.md b/docs/deployments/hetzner-demo-tracker/README.md new file mode 100644 index 000000000..83424dd1c --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/README.md @@ -0,0 +1,125 @@ +# Deployment Journal: Hetzner Demo Tracker + +**Issue**: [#405](https://github.com/torrust/torrust-tracker-deployer/issues/405) +**Date started**: 2026-03-03 +**Domain**: `torrust-tracker-demo.com` +**Provider**: Hetzner Cloud + +## Purpose + +Deploy a public Torrust Tracker demo instance to Hetzner Cloud and document every step of the process. This journal will serve as the source material for a blog post on [torrust.com](https://torrust.com). + +## Table of Contents + +1. [Prerequisites](prerequisites.md) — Account setup, tools, SSH keys +2. [Deployment Specification](deployment-spec.md) — What we want to deploy: config decisions, + endpoints, sanitized config +3. Deployment commands — step-by-step per deployer command: + - [create](commands/create/README.md) — generate template, validate, create environment + - [provision](commands/provision/README.md) — create the Hetzner VM + - [configure](commands/configure/README.md) — install Docker and Docker Compose on the server + - [release](commands/release/README.md) — pull and stage Docker images + - [run](commands/run/README.md) — start all services +4. Post-provision manual steps (done once, before `configure`): + - [DNS setup](post-provision/dns-setup.md) — assign floating IPs, create DNS records, verify + - [Volume setup](post-provision/volume-setup.md) — create and mount Hetzner volume for storage + - [Hetzner Backups](post-provision/hetzner-backups.md) — enable automated server backups (can be done any time after provisioning) +5. [Service Verification](verify/README.md) — verifying all services after deployment: + - [HTTP Tracker](verify/http-tracker.md) + - [UDP Tracker](verify/udp-tracker.md) + - [Tracker API](verify/api.md) + - [Grafana](verify/grafana.md) + - [Health Check](verify/health-check.md) + - [Docker Services](verify/docker-services.md) + - [MySQL Database](verify/mysql.md) + - [Storage Volume](verify/storage.md) + - [Backup](verify/backup.md) +6. Problems — issues encountered, per command: + - [create problems](commands/create/problems.md) + - [provision problems](commands/provision/problems.md) +7. Improvements — recommended deployer improvements found during this deployment: + - [provision improvements](commands/provision/improvements.md) +8. [Observations](observations.md) — cross-cutting insights and learnings about the deployer +9. [Maintenance](maintenance/README.md) — post-deployment operational tasks: + - [Secrets rotation](maintenance/secrets-rotation.md) — rotate all secrets after AI-assisted deployment +10. [Tracker Registry](tracker-registry.md) — submit the tracker to public registries (newTrackon) +11. [Bugs](bugs.md) — all deployer bugs discovered during this deployment (11 bugs, 1 fixed) +12. [Improvements](improvements.md) — all improvement recommendations collected in one place (13 items) + +## Deployment + +> This section will be filled in as we execute each deployment phase. + +### Phase 1: Setup and Prerequisites + +See [prerequisites.md](prerequisites.md) for the complete checklist. + +### Phase 2: Create and Configure Environment + +See [deployment-spec.md](deployment-spec.md) for config decisions and the sanitized config. +See [commands/create/README.md](commands/create/README.md) for running the `create template`, `validate`, and +`create environment` commands. + +### Phase 3: Provision Infrastructure + +See [commands/provision/README.md](commands/provision/README.md) for running the `provision` command and server +details. + +### Phase 3.5: Post-Provision Setup + +Manual steps done once after provisioning, required before `configure`: + +1. [DNS setup](post-provision/dns-setup.md) — assign floating IPs to the server and create DNS + records for all six domains. +2. [Volume setup](post-provision/volume-setup.md) — create a 50 GB Hetzner volume and mount it + at `/opt/torrust/storage` so persistent data lives on a separate disk. +3. [Hetzner Backups](post-provision/hetzner-backups.md) — enable automated daily server backups + via the Hetzner Console (can be done at any time after provisioning). + +See [post-provision/README.md](post-provision/README.md) for the full overview. + +### Phase 4: Configure Instance + +See [commands/configure/README.md](commands/configure/README.md) for running the `configure` +command. Installs Docker 28.2.2 and Docker Compose v2.29.2. + +### Phase 5: Release Application + +See [commands/release/README.md](commands/release/README.md) for running the `release` +command. Pulled and staged all Docker images (~134 s, state=`Released`). + +### Phase 6: Run Services + +See [commands/run/README.md](commands/run/README.md) for running the `run` +command. All services started successfully (state=`Running`). + +### Phase 7: Verify Deployment + +See [verify/README.md](verify/README.md) for the full verification index. +All 9 services verified — HTTP tracker, UDP tracker, Tracker API, Grafana, +health check, Docker services, MySQL database, storage volume, and backup. +Verification included end-to-end announce tests using the Torrust reference +client (`http_tracker_client` and `udp_tracker_client`). + +## Service Endpoints + +> Will be filled after deployment. + +| Service | URL | Status | +| -------------- | ------------------------------------------------- | ---------- | +| HTTP Tracker 1 | `https://http1.torrust-tracker-demo.com/announce` | ✅ Running | +| HTTP Tracker 2 | `https://http2.torrust-tracker-demo.com/announce` | ✅ Running | +| UDP Tracker 1 | `udp://udp1.torrust-tracker-demo.com:6969` | ✅ Running | +| UDP Tracker 2 | `udp://udp2.torrust-tracker-demo.com:6868` | ✅ Running | +| Tracker API | `https://api.torrust-tracker-demo.com/api/v1` | ✅ Running | +| Health Check | `http://127.0.0.1:1313/health_check` (internal) | ✅ Running | +| Grafana | `https://grafana.torrust-tracker-demo.com` | ✅ Running | + +## Cost + +> Will be documented after choosing server type. + +| Resource | Monthly Cost (EUR) | +| -------- | ------------------ | +| Server | TBD | +| Total | TBD | diff --git a/docs/deployments/hetzner-demo-tracker/bugs.md b/docs/deployments/hetzner-demo-tracker/bugs.md new file mode 100644 index 000000000..6ed0c23e5 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/bugs.md @@ -0,0 +1,177 @@ +# Bugs Found During Hetzner Demo Tracker Deployment + +All deployer bugs discovered during this deployment, collected in one place. +Each entry links to the full description in the relevant command document. + +> **Legend**: 🔴 Open  |  🟢 Fixed (in this branch, pending merge) + +--- + +## `create` command + +### B-01 — Template defaults bind addresses to `0.0.0.0` (IPv4 only) + +🔴 Open  |  **Severity**: Major + +The `create template` command hard-codes `0.0.0.0` as the default bind address for +all UDP and HTTP tracker sockets. IPv6 clients cannot connect. Public trackers +should bind to `[::]` by default. + +Full description: [commands/create/problems.md — Template defaults bind addresses to `0.0.0.0`](commands/create/problems.md#problem-template-defaults-bind-addresses-to-0000-ipv4-only) + +--- + +### B-02 — Template silently defaults to SQLite with no database choice + +🔴 Open  |  **Severity**: Major + +The `create template` command uses SQLite without prompting the user or noting that +MySQL is the recommended choice for production. Users may not notice until +deep into the deployment. + +Full description: [commands/create/problems.md — Template silently defaults to SQLite](commands/create/problems.md#problem-template-silently-defaults-to-sqlite--no-database-choice-presented) + +--- + +### B-03 — `instance_name: null` generated with no explanation + +🔴 Open  |  **Severity**: Minor + +The `create template` command outputs `"instance_name": null` with no +documentation of what the deployer will do with a null value (it auto-generates +`torrust-tracker-vm-{env_name}`). The template should explain this. + +Full description: [commands/create/problems.md — Template generates `instance_name: null`](commands/create/problems.md#problem-template-generates-instance_name-null-with-no-explanation) + +--- + +## `provision` command + +### B-04 — SSH probe budget too short for Hetzner (120 s hardcoded) + +🔴 Open  |  **Severity**: Major + +The deployer polls SSH with a fixed budget of 60 × 2 s = 120 s. +On Hetzner `ccx23`, cloud-init user provisioning can take over 3 minutes, +causing `provision` to fail even when the server eventually comes up healthy. + +Full description: [commands/provision/problems.md — SSH connectivity times out](commands/provision/problems.md#problem-provisioning-fails--ssh-connectivity-to-newly-created-server-times-out) + +--- + +### B-05 — Passphrase-protected SSH keys fail silently inside Docker + +🔴 Open  |  **Severity**: Major + +When the deployer runs inside Docker (the standard production workflow), there is +no SSH agent. If the configured deployment key has a passphrase, every probe +attempt silently returns `Permission denied`. Neither `create environment` nor +`validate` warns about this. The user sees repeated SSH failures with no +diagnostic pointing at the passphrase as the cause. + +Full description: [commands/provision/problems.md — SSH probe always fails — passphrase-protected key](commands/provision/problems.md#problem-ssh-probe-always-fails-from-docker-container--passphrase-protected-private-key) + +--- + +### B-06 — UDP tracker domains missing from `provision` output + +🔴 Open  |  **Severity**: Minor + +The `domains` array in the `provision` JSON output only contains HTTP-based +domains. The two UDP tracker domains configured in the environment are silently +omitted. + +Full description: [commands/provision/bugs.md — UDP tracker domains missing](commands/provision/bugs.md#bug-udp-tracker-domains-missing-from-provision-output) + +--- + +## `release` command + +### B-07 — `release` fails when `docker` is not in PATH (Docker-based usage) + +🟢 Fixed (this branch)  |  **Severity**: High + +The `release` command validates the rendered `docker-compose.yml` locally by +running `docker compose config`. When the deployer runs inside its own Docker +container (where `docker` is not installed), this validation fails with +`ENOENT`. The `release` command is completely broken for Docker-based usage. + +**Fix**: the validator now skips with a warning when `docker` is not in PATH +instead of hard-failing. + +Full description: [commands/release/bugs.md — `release` fails when docker not in PATH](commands/release/bugs.md#bug-release-fails-when-deployer-runs-inside-docker-docker-not-in-path) + +--- + +## `run` command + +### B-08 — MySQL `"root"` username not rejected at creation time + +🔴 Open  |  **Severity**: High + +The deployer accepts `"root"` as the MySQL application username. MySQL 8.4 +explicitly rejects `MYSQL_USER=root`. The error only surfaces at `run` time +when MySQL fails to start, not at `create environment` when the bad value is first +accepted. + +Full description: [commands/run/bugs.md — MySQL `"root"` not rejected at creation time](commands/run/bugs.md#bug-1-mysql-app-username-root-is-not-rejected-at-environment-creation-time) + +--- + +### B-09 — MySQL root password silently diverges from operator-configured password + +🔴 Open  |  **Severity**: Medium + +The deployer auto-derives the MySQL root password by appending `"_root"` to the +configured application password (`format!("{password}_root")`). The effective +root password is never surfaced to the operator, making it impossible to connect +to MySQL as `root` using the value from the env config. + +Full description: [commands/run/bugs.md — MySQL root password silently diverges](commands/run/bugs.md#bug-2-mysql-root-password-silently-diverges-from-the-configured-password) + +--- + +### B-10 — MySQL password not URL-encoded in `tracker.toml` connection string + +🔴 Open  |  **Severity**: High + +The tracker `tracker.toml` template renders the MySQL password raw into the +database connection URL. If the password contains URL-special characters (e.g. +`/`, `@`, `#`), the URL is malformed and the tracker crashes at startup with a +misleading `InvalidPort` error. Worked around in this deployment by manually +URL-encoding the password. + +Full description: [commands/run/bugs.md — MySQL password not URL-encoded](commands/run/bugs.md#bug-3-mysql-password-is-not-url-encoded-in-the-tracker-connection-string) + +--- + +## `test` command + +### B-11 — DNS check produces false positives when a floating IP is in use + +🔴 Open  |  **Severity**: Minor + +The `test` command resolves each domain and compares the result against the +**instance IP**. When a floating IP is assigned and DNS points to it instead, +every domain triggers a false-positive warning (`expected 46.225.234.201, got +116.202.176.169`). The deployer has no concept of floating IPs. + +Full description: [commands/improvements.md — Deployer is not aware of floating IPs](commands/improvements.md#improvement-deployer-is-not-aware-of-floating-ips) + +--- + +## Summary + +| ID | Command | Description | Severity | Status | +| ---- | ----------- | ----------------------------------------------------- | -------- | -------- | +| B-01 | `create` | Template binds to `0.0.0.0` (IPv4 only) | Major | 🔴 Open | +| B-02 | `create` | Template defaults to SQLite silently | Major | 🔴 Open | +| B-03 | `create` | `instance_name: null` unexplained | Minor | 🔴 Open | +| B-04 | `provision` | SSH probe budget too short for Hetzner (120 s) | Major | 🔴 Open | +| B-05 | `provision` | Passphrase-protected SSH keys fail silently in Docker | Major | 🔴 Open | +| B-06 | `provision` | UDP tracker domains missing from output | Minor | 🔴 Open | +| B-07 | `release` | Fails when `docker` not in PATH (Docker-based usage) | High | 🟢 Fixed | +| B-08 | `run` | MySQL `"root"` username not rejected at creation time | High | 🔴 Open | +| B-09 | `run` | MySQL root password silently diverges | Medium | 🔴 Open | +| B-10 | `run` | MySQL password not URL-encoded in connection string | High | 🔴 Open | +| B-11 | `test` | DNS check false positives with floating IP | Minor | 🔴 Open | diff --git a/docs/deployments/hetzner-demo-tracker/commands/configure/README.md b/docs/deployments/hetzner-demo-tracker/commands/configure/README.md new file mode 100644 index 000000000..ac66da79b --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/commands/configure/README.md @@ -0,0 +1,64 @@ +# Command: configure + +> **Status**: ✅ Configured successfully (2026-03-04, took ~103 s). + +## What `configure` does + +The `configure` command: + +1. Renders Ansible playbook templates into `build//ansible/`. +2. Runs the Ansible playbook over SSH to set up the server: + - Installs Docker and Docker Compose. + - Creates the application user and directory structure under `/opt/torrust/`. + - Writes the tracker configuration files (`.env`, `docker-compose.yml`, + `tracker.toml`, Prometheus config, Grafana provisioning) to + `/opt/torrust/` on the server. +3. Marks the environment as `Configured` on success. + +It does **not** pull Docker images or start any services — that is done by `release` and `run`. + +## Command + +```bash +docker run --rm \ + -v $(pwd)/data:/var/lib/torrust/deployer/data \ + -v $(pwd)/build:/var/lib/torrust/deployer/build \ + -v $(pwd)/envs:/var/lib/torrust/deployer/envs \ + -v ~/.ssh:/home/deployer/.ssh:ro \ + torrust/tracker-deployer:latest \ + configure torrust-tracker-demo 2>&1 | tee -a data/logs/log.txt +``` + +## Output + +```text +[2026-03-04 11:52:13] Starting Torrust Tracker Deployer container... +[2026-03-04 11:52:13] Verifying installed tools... +[2026-03-04 11:52:13] Tool versions: +[2026-03-04 11:52:13] - OpenTofu: OpenTofu v1.11.5 +[2026-03-04 11:52:13] - Ansible: ansible [core 2.20.3] +[2026-03-04 11:52:13] - SSH: OpenSSH_10.0p2 Debian-7, OpenSSL 3.5.4 30 Sep 2025 +[2026-03-04 11:52:13] - Git: git version 2.47.3 +[2026-03-04 11:52:13] SSH directory found, checking permissions... +[2026-03-04 11:52:13] Container initialization complete. Executing command... +⏳ [1/3] Validating environment... +⏳ ✓ Environment name validated: torrust-tracker-demo (took 0ms) +⏳ [2/3] Creating command handler... +⏳ ✓ Done (took 0ms) +⏳ [3/3] Configuring infrastructure... +⏳ ✓ Infrastructure configured (took 103.5s) +✅ Environment 'torrust-tracker-demo' configured successfully + +{ + "environment_name": "torrust-tracker-demo", + "instance_name": "torrust-tracker-vm-torrust-tracker-demo", + "provider": "hetzner", + "state": "Configured", + "instance_ip": "46.225.234.201", + "created_at": "2026-03-03T19:00:42.481676821Z" +} +``` + +## Problems + +None. diff --git a/docs/deployments/hetzner-demo-tracker/commands/create/README.md b/docs/deployments/hetzner-demo-tracker/commands/create/README.md new file mode 100644 index 000000000..07d9f9c39 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/commands/create/README.md @@ -0,0 +1,96 @@ +# Command: create + +The `create` command has two sub-commands used before provisioning: + +1. `create template` — generate a starter config file for a given provider +2. `create environment` — register the config with the deployer and create local state + +See [problems.md](problems.md) for issues encountered when running these commands. + +## create template + +Generates a fully-featured JSON template for the chosen provider with placeholders for all +required fields. + +```bash +docker run --rm \ + -v $(pwd)/envs:/var/lib/torrust/deployer/envs \ + torrust/tracker-deployer:latest \ + create template --provider hetzner /var/lib/torrust/deployer/envs/torrust-tracker-demo.json +``` + +The generated file is written to `envs/torrust-tracker-demo.json`. Edit it and replace: + +| Placeholder | Value | +| -------------------------------------------- | --------------------------------------------------------------------------- | +| `REPLACE_WITH_ENVIRONMENT_NAME` | `torrust-tracker-demo` | +| `REPLACE_WITH_SSH_PRIVATE_KEY_ABSOLUTE_PATH` | `/home/deployer/.ssh/torrust_tracker_deployer_ed25519` (container path) | +| `REPLACE_WITH_SSH_PUBLIC_KEY_ABSOLUTE_PATH` | `/home/deployer/.ssh/torrust_tracker_deployer_ed25519.pub` (container path) | +| `REPLACE_WITH_HETZNER_API_TOKEN` | Your Hetzner API token (never commit this) | + +> **Important**: When running via Docker, all file paths in the config must be container-internal +> paths (e.g. `/home/deployer/.ssh/...`), not host paths. See [problems.md](problems.md). + +See [deployment-spec.md](../../deployment-spec.md) for the full set of decisions and the sanitized +config used for this deployment. + +## validate + +After editing the config, validate it before registering the environment: + +```bash +docker run --rm \ + -v $(pwd)/envs:/var/lib/torrust/deployer/envs \ + torrust/tracker-deployer:latest \ + validate --env-file /var/lib/torrust/deployer/envs/torrust-tracker-demo.json +``` + +Output confirming the config is valid: + +```json +{ + "environment_name": "torrust-tracker-demo", + "config_file": "envs/torrust-tracker-demo.json", + "provider": "hetzner", + "is_valid": true, + "has_prometheus": true, + "has_grafana": true, + "has_https": true, + "has_backup": true +} +``` + +> **Note**: In this deployment we used `cargo run --bin torrust-tracker-deployer validate ...` +> because we ran from source. For end-users the Docker command above is equivalent and simpler. + +## create environment + +Once the config is validated, register the environment with the deployer: + +```bash +docker run --rm \ + -v $(pwd)/data:/var/lib/torrust/deployer/data \ + -v $(pwd)/build:/var/lib/torrust/deployer/build \ + -v $(pwd)/envs:/var/lib/torrust/deployer/envs \ + torrust/tracker-deployer:latest \ + create environment --env-file /var/lib/torrust/deployer/envs/torrust-tracker-demo.json +``` + +Output: + +```json +{ + "environment_name": "torrust-tracker-demo", + "instance_name": "torrust-tracker-vm-torrust-tracker-demo", + "data_dir": "./data/torrust-tracker-demo", + "build_dir": "./build/torrust-tracker-demo", + "created_at": "2026-03-03T13:59:22.635908188Z" +} +``` + +The deployer creates `data/torrust-tracker-demo/environment.json` with the internal state. This +file holds the environment's domain model — it is managed exclusively by the deployer and must +never be edited manually. + +As shown in the output, `instance_name: null` in the config results in an auto-generated name: +`torrust-tracker-vm-torrust-tracker-demo`. This will be the name of the Hetzner VM. diff --git a/docs/deployments/hetzner-demo-tracker/commands/create/problems.md b/docs/deployments/hetzner-demo-tracker/commands/create/problems.md new file mode 100644 index 000000000..287752ad7 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/commands/create/problems.md @@ -0,0 +1,166 @@ +# Problems: create + +Issues encountered while running the `create template` and `create environment` commands. + +> This is a living document — problems are added as they occur. + + + +## Problem: Template generates `instance_name: null` with no explanation + +**Command**: `create template` +**Severity**: Minor + +### Symptom + +Running `create template --provider hetzner` generates a config with `"instance_name": null` in the +`environment` section. It is unclear whether this field is optional or required, and what value the +deployer will use at runtime if it is left as `null`. + +```json +{ + "environment": { + "name": "REPLACE_WITH_ENVIRONMENT_NAME", + "instance_name": null + } +} +``` + +### Root Cause + +`instance_name` is an optional field. When `null`, the deployer auto-generates the server name as +`torrust-tracker-vm-{env_name}` (for example, `torrust-tracker-vm-torrust-tracker-demo`). The +template does not document this, leaving users uncertain. + +### Resolution + +Leave the field as `null` to accept the auto-generated name, or provide a custom value following +instance naming rules (1–63 characters, ASCII letters/numbers/dashes, cannot start or end with a +dash). + +For this deployment we left it as `null`, so the server will be named +`torrust-tracker-vm-torrust-tracker-demo`. + +### Prevention + +The template generator should add an inline comment or a companion `_comment` field explaining +the auto-generation behavior. + +--- + +## Problem: Template defaults bind addresses to `0.0.0.0` (IPv4 only) + +**Command**: `create template` +**Severity**: Major + +### Symptom + +The generated template binds all UDP and HTTP tracker sockets to `0.0.0.0`: + +```json +{ "bind_address": "0.0.0.0:6969" } +``` + +This works for IPv4 clients but completely ignores IPv6 connections, which is unacceptable for a +public-facing tracker in 2026. + +### Root Cause + +The `create template` command uses `0.0.0.0` as the hardcoded default, which is the safe minimum +for local/LXD environments. No prompt or note about IPv6 is shown. + +### Resolution + +Change all public-facing bind addresses to `[::]` (the IPv6 any-address). On Linux, `[::]` also +accepts IPv4 connections via the IPv4-mapped address mechanism (`::ffff:x.x.x.x`), so only a +single socket per port is required: + +```json +{ "bind_address": "[::]:6969", "domain": "udp1.torrust-tracker-demo.com" } +``` + +The `health_check_api` is internal (used only by Docker health probes) and remains on +`127.0.0.1:1313`. + +### Prevention + +The template generator should default to `[::]` for public-facing sockets, or at minimum include a +note explaining the dual-stack trade-off. A `validate` warning for `0.0.0.0` on Hetzner +deployments would also help. + +--- + +## Problem: Template silently defaults to SQLite — no database choice presented + +**Command**: `create template` +**Severity**: Major + +### Symptom + +The generated template uses SQLite without asking the user: + +```json +{ + "database": { + "driver": "sqlite3", + "database_name": "tracker.db" + } +} +``` + +For a production-facing demo tracker, SQLite has limitations (no concurrent writes, no easy +remote inspection). Users may not notice the default until they are deep into the deployment. + +### Root Cause + +The `create template` command uses SQLite as the default database because it requires no additional +configuration fields (no host, port, username, password). It is the simpler default for local +development. No prompt or warning is shown. + +### Resolution + +Manually change the database section to MySQL after generating the template: + +```json +{ + "database": { + "driver": "mysql", + "host": "mysql", + "port": 3306, + "database_name": "torrust", + "username": "root", + "password": "" + } +} +``` + +Use a securely generated password (e.g., `openssl rand -base64 24`). + +### Prevention + +The `create template` command should either prompt the user for a database choice, or include a +prominent comment in the generated file noting that SQLite is the dev default and MySQL is +recommended for production. diff --git a/docs/deployments/hetzner-demo-tracker/commands/improvements.md b/docs/deployments/hetzner-demo-tracker/commands/improvements.md new file mode 100644 index 000000000..1a099fdd7 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/commands/improvements.md @@ -0,0 +1,75 @@ +# Deployer Commands — Improvements + +Improvements identified during the Hetzner demo tracker deployment that apply +across multiple commands, not just one specific command. + +--- + +## Improvement: Deployer is not aware of floating IPs + +**Status**: Open +**Affects**: `test` command (DNS checks), potentially `configure`, `release`, `run` +**Observed in**: `test` command DNS warnings + +### Observation + +Hetzner and other providers support **floating IPs** — IP addresses that are +owned independently of any single server and can be reassigned between servers +instantly without changing DNS. In this deployment: + +- **Instance IP**: `46.225.234.201` — the bare VM's IP, not published in DNS +- **Floating IP**: `116.202.176.169` — the IP published in all DNS A records + +The deployer `test` command resolves each configured domain and compares the +result against the **instance IP**. Because DNS points to the floating IP, every +domain triggers a warning: + +```text +⚠️ DNS check: api.torrust-tracker-demo.com resolves to [116.202.176.169] + but expected 46.225.234.201 +``` + +This is not a real problem — DNS is configured correctly and traffic reaches the +server. The warning is a false positive produced by the deployer's lack of +floating IP awareness. + +### Why Floating IPs Matter + +Using a floating IP provides two key operational benefits: + +1. **Zero-downtime failover**: If the primary server fails, the floating IP can + be reassigned to a standby server in seconds. DNS does not need to change and + clients are not affected by TTL delays. +2. **Maintenance without downtime**: A new server can be fully provisioned and + configured before cutting traffic over by reassigning the floating IP. + +These benefits are lost if DNS records point directly to the instance IP. + +### Proposed Fix + +The environment JSON config (or a separate provider config section) should allow +specifying a **floating IP** (or more generally, a **public IP**) that is +distinct from the instance IP. For example: + +```json +"provider": { + ... + "floating_ip": "116.202.176.169" +} +``` + +The deployer should then: + +- Use the floating IP (when present) as the expected DNS target in `test` DNS + checks, treating a match as a pass rather than a warning. +- Optionally, during provisioning, automatically assign the floating IP to the + newly created instance. +- Optionally, expose the floating IP in the `Running` state output so operators + can confirm which IP is serving traffic. + +### Current Workaround + +The DNS warnings in `test` output can be safely ignored when a floating IP is in +use. Verify manually that the domains resolve to the expected floating IP and +that the services are reachable through those domains (see +[../verify/](../verify/) for service verification procedures). diff --git a/docs/deployments/hetzner-demo-tracker/commands/provision/README.md b/docs/deployments/hetzner-demo-tracker/commands/provision/README.md new file mode 100644 index 000000000..2757eb174 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/commands/provision/README.md @@ -0,0 +1,165 @@ +# Command: provision + +> **Status**: ✅ Provisioned successfully (attempt 4, 2026-03-03 ~19:01 UTC). +> See [problems.md](problems.md) for the full failure history and root cause analysis. +> See [improvements.md](improvements.md) for deployer improvements applied during this process. +> See [cleanup-between-attempts.md](cleanup-between-attempts.md) for the cleanup procedure used between attempts. + +## What `provision` does + +The `provision` command: + +1. Renders OpenTofu templates (Terraform HCL + cloud-init YAML) into `build//tofu/hetzner/`. +2. Runs `tofu init` + `tofu apply` to create the Hetzner server. +3. Waits for SSH connectivity (up to 300 seconds, 60 × 5 s intervals). +4. Marks the environment as `Provisioned` once SSH responds. + +It does **not** install Docker, configure the tracker, or deploy any software — that is done by +the `configure`, `release`, and `run` commands. + +## Command + +```bash +docker run --rm \ + -v $(pwd)/data:/var/lib/torrust/deployer/data \ + -v $(pwd)/build:/var/lib/torrust/deployer/build \ + -v $(pwd)/envs:/var/lib/torrust/deployer/envs \ + -v ~/.ssh:/home/deployer/.ssh:ro \ + torrust/tracker-deployer:latest \ + provision torrust-tracker-demo +``` + +## Provisioned Server Details + +The Hetzner server was created successfully by OpenTofu on 2026-03-03 at ~19:00 UTC (attempt 4). + +| Property | Value | +| ------------ | ----------------------------------------- | +| Server name | `torrust-tracker-vm-torrust-tracker-demo` | +| IPv4 | `46.225.234.201` | +| IPv6 | `2a01:4f8:1c19:620b::/64` | +| IPv6 address | `2a01:4f8:1c19:620b::1` | +| Location | `nbg1` (Nuremberg, Germany) | +| Server type | `ccx23` (4 vCPU, 16 GB RAM) | +| Image | `ubuntu-24.04` | +| SSH user | `torrust` | + +> **Note**: These are the server's own IPs assigned by Hetzner. The floating IPs +> (`116.202.176.169` IPv4, `2a01:4f8:1c0c:9aae::/64` IPv6) are separate and must be +> assigned to this server manually in the Hetzner Console after successful provisioning. +> +> **Note**: The `provision` command output currently only reports the IPv4 address. The IPv6 +> address and network must be looked up in the Hetzner console or `terraform.tfstate`. See +> [improvements.md](improvements.md) for the tracked improvement. + +![Hetzner console showing the provisioned server details — attempt 4 (current)](../../media/hetzner-console-provisioned-server-details-attempt-4.png) + +> **Note**: The Hetzner activity log shows "Server is being created" as the last event even +> after the server is fully created and accessible. Hetzner does not emit a matching +> "Server creation finished" event, so this entry can be misleading. See +> [problems.md](problems.md) for full context. + +For reference, the first server created in attempt 1: + +![Hetzner console showing the provisioned server details — attempt 1](../../media/hetzner-console-provisioned-server-details-attempt-1.png) + +## Generated Artifacts + +After provisioning the following build artifacts are created: + +- `build/torrust-tracker-demo/tofu/hetzner/` — rendered OpenTofu project (HCL + cloud-init) +- `build/torrust-tracker-demo/tofu/hetzner/cloud-init.yml` — cloud-init config with SSH key +- `data/torrust-tracker-demo/environment.json` — state updated to `Provisioned` (or + `ProvisionFailed` if something went wrong) +- `data/torrust-tracker-demo/traces/` — failure trace files (generated automatically on error) + +## Verifying the SSH Key Injection + +The rendered `build/torrust-tracker-demo/tofu/hetzner/cloud-init.yml` should contain the +public SSH key in the `ssh_authorized_keys` section: + +```yaml +users: + - name: torrust + groups: sudo + shell: /bin/bash + sudo: ["ALL=(ALL) NOPASSWD:ALL"] + ssh_authorized_keys: + - ssh-ed25519 torrust-tracker-deployer +``` + +To verify the key matches your local public key: + +```bash +grep -A1 "ssh_authorized_keys" build/torrust-tracker-demo/tofu/hetzner/cloud-init.yml +cat ~/.ssh/torrust_tracker_deployer_ed25519.pub +``` + +## Successful Provision Output (Attempt 4) + +Final output of the `provision` command on attempt 4 (2026-03-03 ~19:01 UTC, 50.9 seconds): + +```text +⏳ [1/3] Validating environment... +⏳ ✓ Environment name validated: torrust-tracker-demo (took 0ms) +⏳ [2/3] Creating command handler... +⏳ ✓ Done (took 0ms) +⏳ [3/3] Provisioning infrastructure... +⏳ ✓ Infrastructure provisioned (took 50.9s) +✅ Environment 'torrust-tracker-demo' provisioned successfully + +{ + "environment_name": "torrust-tracker-demo", + "instance_name": "torrust-tracker-vm-torrust-tracker-demo", + "instance_ip": "46.225.234.201", + "ssh_username": "torrust", + "ssh_port": 22, + "ssh_private_key_path": "/home/deployer/.ssh/torrust_tracker_deployer_ed25519", + "provider": "hetzner", + "provisioned_at": "2026-03-03T19:00:42.481676821Z", + "domains": [ + "http1.torrust-tracker-demo.com", + "http2.torrust-tracker-demo.com", + "api.torrust-tracker-demo.com", + "grafana.torrust-tracker-demo.com" + ] +} +``` + +> **Note**: The output includes `instance_ip` (IPv4 only). The IPv6 address +> (`2a01:4f8:1c19:620b::1`) and network (`2a01:4f8:1c19:620b::/64`) are not included. See +> [improvements.md](improvements.md) for the tracked improvement. + +## Key Learnings + +After four attempts, the following were identified as the root causes of all failures: + +1. **SSH key paths must be container-internal paths** — paths in `envs/*.json` must reflect + paths inside the Docker container (`/home/deployer/.ssh/...`), not host paths. + +2. **The deployment SSH key must not have a passphrase** — when running inside Docker there is + no SSH agent and no TTY, so a passphrase-protected key cannot be used for authentication + even if the key file is readable. This was the root cause of all three `Permission denied` + failures in attempts 2, 3, and 4. See [problems.md](problems.md) for full analysis. + +3. **Cloud-init on Hetzner `ccx23` can take 15+ minutes** — the SSH probe budget must account + for this. In attempt 4 the server was ready well within the 300-second window (50.9s total + including `tofu apply`), so this was not an issue once the passphrase was removed. But in + the earlier attempts where the server lingered in "Server is being created" for 15+ minutes + in the Hetzner console, a longer timeout would still not have helped because the fundamental + auth problem was the passphrase. + +## Manual SSH Verification (After Provisioning) + +Once the server is up, verify SSH access directly from the host: + +```bash +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@ "whoami && cloud-init status" +``` + +Expected output: + +```text +torrust +status: done +``` diff --git a/docs/deployments/hetzner-demo-tracker/commands/provision/bugs.md b/docs/deployments/hetzner-demo-tracker/commands/provision/bugs.md new file mode 100644 index 000000000..ee1e25320 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/commands/provision/bugs.md @@ -0,0 +1,128 @@ +# Bugs: provision + +Issues found in the `provision` command output during the Hetzner demo tracker deployment. + +> This is a living document — bugs are added as they are discovered. + + + +## Bug: UDP tracker domains missing from `provision` output + +**Command**: `provision` +**Severity**: Minor + +### Symptom + +The `domains` array in the successful `provision` output only contains HTTP-based domains. +UDP tracker domains are not included: + +```json +{ + "domains": [ + "http1.torrust-tracker-demo.com", + "http2.torrust-tracker-demo.com", + "api.torrust-tracker-demo.com", + "grafana.torrust-tracker-demo.com" + ] +} +``` + +The two UDP tracker domains configured in `envs/torrust-tracker-demo.json` are absent: + +```text +udp1.torrust-tracker-demo.com (port 6969) +udp2.torrust-tracker-demo.com (port 6868) +``` + +### Expected Behavior + +All service domains — HTTP and UDP — should appear in the `domains` array: + +```json +{ + "domains": [ + "http1.torrust-tracker-demo.com", + "http2.torrust-tracker-demo.com", + "udp1.torrust-tracker-demo.com", + "udp2.torrust-tracker-demo.com", + "api.torrust-tracker-demo.com", + "grafana.torrust-tracker-demo.com" + ] +} +``` + +These domains are defined in `envs/torrust-tracker-demo.json` under `tracker.udp_trackers`: + +```json +"udp_trackers": [ + { "bind_address": "[::]:6969", "domain": "udp1.torrust-tracker-demo.com" }, + { "bind_address": "[::]:6868", "domain": "udp2.torrust-tracker-demo.com" } +] +``` + +And in the deployment spec at +[`docs/deployments/hetzner-demo-tracker/deployment-spec.md`](../../deployment-spec.md): + +```text +UDP Tracker 1: udp://udp1.torrust-tracker-demo.com:6969/announce +UDP Tracker 2: udp://udp2.torrust-tracker-demo.com:6868/announce +``` + +### Root Cause + +The `provision` command output is built from the environment configuration. It likely only +collects domains from HTTP-based tracker configs (`http_trackers`, `http_api`, `grafana`) and +does not iterate over `udp_trackers` when assembling the `domains` list. + +### Workaround + +The UDP tracker domains are defined in the environment config file and can be read directly: + +```bash +cat envs/torrust-tracker-demo.json \ + | python3 -c "import json,sys; d=json.load(sys.stdin); \ + [print(t['domain']) for t in d['tracker']['udp_trackers']]" +``` + +Expected output: + +```text +udp1.torrust-tracker-demo.com +udp2.torrust-tracker-demo.com +``` + +### Fix + +The domain collection logic in the `provision` output builder should include domains from all +service types. Specifically, `udp_trackers[].domain` entries should be added to the `domains` +list alongside the HTTP tracker and API domains. + +**Likely location**: the `ProvisionOutput` struct and the code that populates its `domains` +field, in `src/application/commands/provision/`. diff --git a/docs/deployments/hetzner-demo-tracker/commands/provision/cleanup-between-attempts.md b/docs/deployments/hetzner-demo-tracker/commands/provision/cleanup-between-attempts.md new file mode 100644 index 000000000..9e65e0749 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/commands/provision/cleanup-between-attempts.md @@ -0,0 +1,76 @@ +# Cleanup Between Provision Attempts + +When `provision` fails, the environment is left in a `ProvisionFailed` state with a live +server still running on Hetzner. Before retrying, the failed environment must be fully +cleaned up — both on the provider and locally. + +## Steps + +### 1. Build the Docker image + +If you have made code changes since the last attempt, rebuild the image first: + +```bash +docker build --target release \ + --tag torrust/tracker-deployer:latest \ + --file docker/deployer/Dockerfile . +``` + +Skip this step if no code has changed since the last Docker build. + +### 2. Destroy the failed environment on the provider + +This runs `tofu destroy` to remove the Hetzner server: + +```bash +docker run --rm \ + -v $(pwd)/data:/var/lib/torrust/deployer/data \ + -v $(pwd)/build:/var/lib/torrust/deployer/build \ + -v $(pwd)/envs:/var/lib/torrust/deployer/envs \ + -v ~/.ssh:/home/deployer/.ssh:ro \ + torrust/tracker-deployer:latest \ + destroy torrust-tracker-demo +``` + +### 3. Verify the server is gone on Hetzner + +Log in to the [Hetzner Cloud Console](https://console.hetzner.cloud/) and confirm the +server no longer appears under the project. Also verify the floating IPs are unassigned +but still present (they are not destroyed by `destroy`). + +### 4. Purge local environment data + +This removes `build/torrust-tracker-demo/` and `data/torrust-tracker-demo/` and clears +the environment from the local registry: + +```bash +docker run --rm \ + -v $(pwd)/data:/var/lib/torrust/deployer/data \ + -v $(pwd)/build:/var/lib/torrust/deployer/build \ + -v $(pwd)/envs:/var/lib/torrust/deployer/envs \ + torrust/tracker-deployer:latest \ + purge torrust-tracker-demo --force +``` + +> **Note**: `--force` skips the interactive confirmation prompt. Omit it if you want to +> confirm interactively. + +### 5. Clear the global log file + +The global log at `data/logs/log.txt` accumulates entries across all environments and +sessions. Clearing it before a retry makes it much easier to isolate errors from the +new attempt: + +```bash +> data/logs/log.txt +``` + +> **Note**: This truncates the file in-place (preserving the file itself). Git will not +> track the change since `data/logs/` is git-ignored. + +--- + +After completing all steps, the environment is fully clean. You can now proceed to: + +1. Re-create the environment: see [README.md](README.md) +2. Retry provision: see [README.md](README.md) diff --git a/docs/deployments/hetzner-demo-tracker/commands/provision/improvements.md b/docs/deployments/hetzner-demo-tracker/commands/provision/improvements.md new file mode 100644 index 000000000..8c449e1dc --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/commands/provision/improvements.md @@ -0,0 +1,251 @@ +# Deployer Improvements: provision + +Recommendations for improving the deployer based on problems encountered while running the +`provision` command against a Hetzner server. Each improvement is grounded in a specific +observation from the deployment logs. + +> This document captures field experience. The issues described here are not blockers for +> day-to-day use but would significantly reduce debugging time and false-alarm incidents. + +--- + +## 1. Distinguish SSH failure reason in the probe loop + +### Current behavior + +`wait_for_connectivity` treats all failed attempts the same way: `test_connectivity()` returns +`Ok(false)` for any failure — whether the port is not yet open (TCP timeout) or sshd is running +but rejects the connection (authentication failure). The log only says "still waiting": + +```text +INFO ... Still waiting for SSH connectivity attempt=10 max_attempts=60 +``` + +### Observed impact + +When investigating Problem 5, we could not determine from the trace whether the probe was even +reaching the server. We had to cross-reference `data/logs/log.txt` and measure the time between +log lines manually to discover that TCP connections started succeeding at attempt 4 while +authentication was failing until cloud-init finished. + +### Recommendation + +`test_connectivity` (or `check_command`) should capture the stderr output of the ssh process and +expose it alongside the boolean result. `wait_for_connectivity` should then log a different +message depending on the failure type: + +```text +INFO ... SSH probe: connection timeout (port 22 not open yet) attempt=3 +INFO ... SSH probe: authentication rejected (user not ready) attempt=10 +``` + +This can be inferred from stderr: + +| stderr contains | Meaning | +| ----------------------- | --------------------------------- | +| `Connection timed out` | Port 22 not listening | +| `Connection refused` | Port 22 not listening | +| `Permission denied` | sshd up, user/key not ready yet | +| `Host key verification` | Wrong host key (misconfiguration) | + +The ssh exit code is `255` in all failure cases, so stderr is the only signal. + +**Related source**: `src/adapters/ssh/client.rs` — `test_connectivity` and `wait_for_connectivity` + +--- + +## 2. Classify `error_kind` more precisely for auth failures + +### Current behavior + +When `WaitSshConnectivity` fails (for any reason), the `error_kind` in +`data/{env}/environment.json` is always `NetworkConnectivity`. This includes authentication +failures, which are not network problems. + +```json +{ "error_kind": "NetworkConnectivity" } +``` + +### Observed impact + +The label directed investigation towards the network layer (firewall? wrong IP? routing?) when +the actual problem was an application-level issue (cloud-init not yet finished creating the user). + +### Recommendation + +Add a more specific `error_kind` variant such as `SshAuthenticationFailed` or +`SshUserNotReady` for the case where TCP connects but the remote rejects the credentials. Reserve +`NetworkConnectivity` for cases where the TCP connection itself cannot be established. + +**Related source**: `src/adapters/ssh/` — `SshError` variants and how they map to `error_kind` +in the provision command handler + +--- + +## 3. Include per-attempt failure details in the trace file + +### Current behavior + +The provision trace file (`data/{env}/traces/{timestamp}-provision.log`) contains only a summary: + +```text +Error Summary: SSH connectivity failed: ... after 60 attempts (120s total) +``` + +### Observed impact + +Diagnosing Problem 5 required reading `data/logs/log.txt` and manually measuring timestamps +between lines — a non-obvious step that is not documented anywhere in the error output. The trace +file is the first place an operator looks and it contained no actionable detail. + +### Recommendation + +The trace file should include a condensed per-phase summary of the SSH probe, for example: + +```text +SSH Probe Summary: + Attempts 1–3 : connection timeout (port 22 not open) + Attempts 4–60 : authentication rejected (user not ready) + Total duration : 181s + Conclusion : sshd was ready but cloud-init did not finish within the probe window +``` + +This can be generated by tracking failure reason counts inside `wait_for_connectivity` and +passing them to the trace writer on failure. + +--- + +## 4. Support configurable SSH connectivity timeout + +### Current behavior + +The probe loop is hardcoded at 60 attempts × 2-second intervals = 120 seconds. This is sized for +fast LXD VMs (typically ready in under 30 seconds) and is too short for Hetzner servers with +cloud-init user provisioning. + +### Observed impact + +The deployer gave up after 120 seconds. Cloud-init finished more than 3.5 minutes after the +server appeared, meaning a minimum probe budget of ~250 seconds was needed to succeed. + +### Recommendation + +Expose the probe parameters as configuration, ordered by preference: + +1. **Per-provider default** — Hetzner profile uses a longer timeout (e.g. 300s) than LXD (60s) +2. **Environment config field** — `ssh_connectivity_timeout_secs` in `envs/*.json` +3. **CLI override** — `provision --ssh-timeout 300` + +Until provider-level defaults are implemented, document in the Hetzner provider guide that a +timeout of at least 300 seconds (150 attempts × 2s) is recommended. + +**Related source**: `src/adapters/ssh/` — `SshConnectionConfig` struct (likely contains +`max_retry_attempts` and `retry_interval_secs` fields); `src/application/steps/connectivity/` + +--- + +## 6. Include IPv6 address in `provision` command output + +### Current behavior + +The JSON output of a successful `provision` command only includes `instance_ip` (IPv4): + +```json +{ + "instance_ip": "46.225.234.201" +} +``` + +The IPv6 address and network assigned by Hetzner are not reported anywhere in the command +output. To find them, the operator must either log into the Hetzner console or read the raw +`build/torrust-tracker-demo/tofu/hetzner/terraform.tfstate` file: + +```json +"ipv6_address": "2a01:4f8:1c19:620b::1", +"ipv6_network": "2a01:4f8:1c19:620b::/64" +``` + +### Observed impact + +After successful provisioning, the floating IP assignment step requires knowing the server's +IPv6 network. Without it in the output, the operator cannot complete the setup without +consulting additional sources. + +### Recommendation + +Extend the `ProvisionOutput` JSON to include both IPv4 and IPv6: + +```json +{ + "instance_ip": "46.225.234.201", + "instance_ipv6": "2a01:4f8:1c19:620b::1", + "instance_ipv6_network": "2a01:4f8:1c19:620b::/64" +} +``` + +**Related source**: `src/application/commands/provision/` — output struct; OpenTofu outputs +defined in `templates/tofu/hetzner/main.tf.tera` + +--- + +## 7. Detect passphrase-protected SSH keys early and warn the user + +### Current behavior + +The deployer does not validate whether the configured SSH private key is passphrase-protected. +A key with a passphrase is loaded and offered to the server, but signing fails silently because +there is no agent and no TTY inside the Docker container. The user sees repeated +`Permission denied (publickey,password)` errors with no indication that the key itself is the +problem. + +### Observed impact + +All three provision failures (attempts 2, 3, 4) were caused by this. Investigation consumed +multiple hours chasing unrelated hypotheses (wrong key in cloud-init, SSH agent key exhaustion, +`IdentitiesOnly` flag, cloud-init timing) before the passphrase was identified as the +root cause. + +### Recommendation + +In `create environment` or early in `provision`, attempt to load the private key with an empty +passphrase. If it fails, emit a warning before any infrastructure is created: + +```text +⚠ Warning: The SSH private key at '/home/deployer/.ssh/torrust_tracker_deployer_ed25519' + appears to be passphrase-protected. When running via Docker (no SSH agent, no TTY), + the key cannot be used for authentication. + → Remove the passphrase: ssh-keygen -p -f + → Or add a passphrase-free key and update ssh_credentials in your env config. +``` + +This can be done using `ssh2` or `openssl` bindings in Rust to probe whether the key decrypts +with an empty passphrase. + +**Related source**: `src/application/commands/create/` or `src/adapters/ssh/` — key validation +step before any remote operations + +--- + +## 5. Add a `wait-for-ssh` command (or `--resume` flag on provision) + +### Current behavior + +When `provision` fails at `WaitSshConnectivity`, the environment transitions to `ProvisionFailed`. +The only recovery path is `destroy` → `create environment` → `provision` again — which destroys +and re-creates a server that is otherwise healthy. + +### Observed impact + +The server created in attempt 2 was fully functional (SSH accessible, cloud-init complete) but +had to be discarded. All the time spent on `tofu apply` and cloud-init startup was wasted. + +### Recommendation + +Two complementary options: + +- **`provision --resume`**: allow re-running `WaitSshConnectivity` (and subsequent steps) against + an already-created server without re-running `tofu apply`. State machine would need a new + `ProvisionPartiallyFailed` state that distinguishes "server exists" from "server never created". +- **`wait-for-ssh `**: a standalone command that only runs the SSH probe and, on + success, transitions the environment to `Provisioned`. Useful for manual recovery without + modifying the main `provision` command. diff --git a/docs/deployments/hetzner-demo-tracker/commands/provision/problems.md b/docs/deployments/hetzner-demo-tracker/commands/provision/problems.md new file mode 100644 index 000000000..c3d436cd0 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/commands/provision/problems.md @@ -0,0 +1,421 @@ +# Problems: provision + +Issues encountered while running the `provision` command. + +> This is a living document — problems are added as they occur. + + + +## Problem: SSH key paths in config must be container-internal paths when running via Docker + +**Command**: `provision` +**Severity**: Blocker + +### Symptom + +Running `provision` via Docker fails immediately with: + +```text +❌ OpenTofu template rendering failed: Failed to render cloud-init template: + SSH public key file not found or unreadable +``` + +The SSH key files exist on the host but the deployer cannot find them. + +Trace file: `data/torrust-tracker-demo/traces/20260303-152806-provision.log` + +```text +Error Summary: OpenTofu template rendering failed: Failed to render cloud-init template: + SSH public key file not found or unreadable +Failed Step: RenderOpenTofuTemplates +Error Kind: TemplateRendering +``` + +### Root Cause + +The `ssh_credentials` paths in the environment config file (`envs/torrust-tracker-demo.json`) were +set to the host machine's paths (e.g. `/home/josecelano/.ssh/torrust_tracker_deployer_ed25519`). +When running the deployer via `docker run`, the SSH directory is mounted into the container at a +different location: + +```bash +-v ~/.ssh:/home/deployer/.ssh:ro +``` + +Inside the container, all SSH keys are under `/home/deployer/.ssh/`, not +`/home//.ssh/`. The deployer reads the path literally from the config, so it looks for +a file that does not exist inside the container. + +### Resolution + +Always use the container-internal path in `ssh_credentials` when running via Docker: + +```json +{ + "ssh_credentials": { + "private_key_path": "/home/deployer/.ssh/torrust_tracker_deployer_ed25519", + "public_key_path": "/home/deployer/.ssh/torrust_tracker_deployer_ed25519.pub" + } +} +``` + +After fixing the paths, the environment must be recreated (the previous environment was purged +with `--force` because no infrastructure had been created yet): + +```bash +docker run --rm \ + -v $(pwd)/data:/var/lib/torrust/deployer/data \ + -v $(pwd)/build:/var/lib/torrust/deployer/build \ + -v $(pwd)/envs:/var/lib/torrust/deployer/envs \ + torrust/tracker-deployer:latest \ + purge torrust-tracker-demo --force + +docker run --rm \ + -v $(pwd)/data:/var/lib/torrust/deployer/data \ + -v $(pwd)/build:/var/lib/torrust/deployer/build \ + -v $(pwd)/envs:/var/lib/torrust/deployer/envs \ + torrust/tracker-deployer:latest \ + create environment --env-file envs/torrust-tracker-demo.json +``` + +### Prevention + +The documentation for Docker-based usage should explicitly state that all paths in the config file +(SSH keys, etc.) must be the paths as seen **inside the container**, not on the host. The +`validate` command could also check that the referenced SSH key files are readable from within the +process's filesystem context. + +--- + +## Problem: Provisioning fails — SSH connectivity to newly created server times out + +**Command**: `provision` +**Severity**: Blocker + +### Symptom + +The deployer creates the Hetzner server successfully (the VM appears in the Hetzner console) but +then fails while waiting for SSH to become available: + +```text +❌ Provision command failed: Failed to provision environment 'torrust-tracker-demo': + SSH connectivity failed: Failed to establish SSH connectivity to 46.225.234.201 + after 60 attempts (120s total) +Tip: Check if instance is fully booted and SSH service is running +Tip: Check logs and try running with --log-output file-and-stderr for more details +``` + +The server IP `46.225.234.201` is reachable (Hetzner reports it as running) but SSH on port 22 +never responds within the 120-second window. + +Trace file: `data/torrust-tracker-demo/traces/20260303-153332-provision.log` + +The deployment environment state recorded in `data/torrust-tracker-demo/environment.json`: + +```json +{ + "state": { + "context": { + "failed_step": "WaitSshConnectivity", + "error_kind": "NetworkConnectivity", + "error_summary": "SSH connectivity failed: ...", + "failed_at": "2026-03-03T15:33:32.933487060Z", + "execution_started_at": "2026-03-03T15:30:00.047895413Z", + "execution_duration": { "secs": 212, "nanos": 885591647 }, + "trace_id": "bcba0ee9-b2cf-4302-be0e-5ed04c665141", + "trace_file_path": "./data/torrust-tracker-demo/traces/20260303-153332-provision.log" + } + } +} +``` + +### Root Cause + +**The SSH public key was correctly injected** — confirmed by inspecting the rendered +`build/torrust-tracker-demo/tofu/hetzner/cloud-init.yml`, which contains the correct public key: + +```yaml +ssh_authorized_keys: + - ssh-ed25519 torrust-tracker-deployer +``` + +This matches the content of `~/.ssh/torrust_tracker_deployer_ed25519.pub` exactly. A manual SSH +from the host confirms the server accepted the key: + +```bash +$ ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@46.225.234.201 "whoami && cloud-init status" +torrust +status: done +``` + +**Precise timeline reconstructed from `data/logs/log.txt`:** + +| Time (UTC) | Event | +| ------------- | --------------------------------------------------------- | +| `15:30:13` | `tofu apply` started | +| `15:30:32` | `tofu apply` done — server `46.225.234.201` created (19s) | +| `15:30:32` | SSH probe starts (attempt 1) | +| `15:30:32–46` | Attempts 1–3: ~7s each (`ConnectTimeout=5` + 2s sleep) | +| `15:30:49+` | Attempts 4–60: ~2–3s each | +| `15:33:32` | Probe exhausted — 60 attempts, 120s total | +| `> 15:44:00` | Manual SSH succeeds (`cloud-init status: done`) | + +The change in per-attempt duration is the key signal: + +- **Attempts 1–3 (~7s each)**: port 22 not yet open — the `ssh` process hangs for the full + `ConnectTimeout=5` seconds before returning. sshd was not listening. +- **Attempts 4–60 (~2–3s each)**: TCP connection to port 22 **succeeds** but sshd rejects the + authentication immediately. sshd is running but the `torrust` user and `~/.ssh/authorized_keys` + do not exist yet — cloud-init has not finished creating them. + +**The real bottleneck is cloud-init user provisioning**, not sshd startup. sshd was listening +within ~17 seconds of the server appearing. But cloud-init had not yet created the `torrust` user +and written their `authorized_keys`. Every one of the 60 probe attempts was rejected with +"permission denied" for this reason. + +Cloud-init finished some time between `15:33:32` (when the deployer gave up) and `~15:44:00` +(when manual SSH succeeded) — meaning cloud-init took **more than 3 minutes 32 seconds** on this +`ccx23` server. The 120-second probe budget was not enough. + +**cloud-init timing note**: `cloud-init status --long` reports +`last_update: Thu, 01 Jan 1970 00:00:13 +0000` — the epoch-based timestamp shows the system clock +was not yet NTP-synced when cloud-init completed, which is consistent with cloud-init completing +very early in the boot sequence before network time was established. + +The deployer's SSH probe loop (60 × 2-second intervals = 120 seconds total) is designed for +fast LXD VMs which are typically ready in under 30 seconds. Hetzner `ccx23` servers with +cloud-init user provisioning take significantly longer. + +### Resolution + +The environment is in `ProvisionFailed` state. The deployer does not allow re-running provision +from a failed state; the environment must be destroyed and recreated: + +```bash +# Destroy the Hetzner server (cleans up cloud resources and local state) +docker run --rm \ + -v $(pwd)/data:/var/lib/torrust/deployer/data \ + -v $(pwd)/build:/var/lib/torrust/deployer/build \ + -v $(pwd)/envs:/var/lib/torrust/deployer/envs \ + -v ~/.ssh:/home/deployer/.ssh:ro \ + torrust/tracker-deployer:latest \ + destroy torrust-tracker-demo + +# Recreate the environment +docker run --rm \ + -v $(pwd)/data:/var/lib/torrust/deployer/data \ + -v $(pwd)/build:/var/lib/torrust/deployer/build \ + -v $(pwd)/envs:/var/lib/torrust/deployer/envs \ + torrust/tracker-deployer:latest \ + create environment --env-file envs/torrust-tracker-demo.json + +# Retry provision +docker run --rm \ + -v $(pwd)/data:/var/lib/torrust/deployer/data \ + -v $(pwd)/build:/var/lib/torrust/deployer/build \ + -v $(pwd)/envs:/var/lib/torrust/deployer/envs \ + -v ~/.ssh:/home/deployer/.ssh:ro \ + torrust/tracker-deployer:latest \ + provision torrust-tracker-demo +``` + +The retry may still time out if cloud-init on the new server also takes more than 120 seconds. +If it does, see the Prevention section for options to increase the probe budget. + +### Prevention + +The deployer should expose a configurable SSH connectivity timeout (currently hardcoded at 60 +attempts / 120 seconds). For Hetzner servers with cloud-init scripts, a longer timeout is needed. + +Potential improvements: + +- A `--ssh-timeout` parameter on the `provision` command. +- A `wait-for-ssh` command to decouple the retry loop (useful when provision partially succeeds). +- Document in the Hetzner provider guide that the first SSH probe may take 3–5 minutes. + +--- + +## Problem: SSH probe always fails from Docker container — passphrase-protected private key + +**Command**: `provision` +**Severity**: Blocker + +### Symptom + +All 60 SSH probe attempts fail with `Permission denied (publickey,password)` even after the server +is fully running and cloud-init has completed. The error appears in `data/logs/log.txt`: + +```text +Permission denied, please try again. +Permission denied, please try again. +torrust@46.225.234.201: Permission denied (publickey,password). +``` + +Manual SSH from the host succeeds with the same key: + +```bash +$ ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@46.225.234.201 "whoami && cloud-init status" +torrust +status: done +``` + +But the same command from inside the Docker container fails: + +```bash +$ docker run --rm --entrypoint bash \ + -v ~/.ssh:/home/deployer/.ssh:ro \ + torrust/tracker-deployer:latest \ + -c "ssh -i /home/deployer/.ssh/torrust_tracker_deployer_ed25519 \ + -o StrictHostKeyChecking=no -o IdentitiesOnly=yes \ + torrust@46.225.234.201 'echo connected'" +Permission denied (publickey,password). +``` + +### Root Cause + +**The private key `~/.ssh/torrust_tracker_deployer_ed25519` is protected by a passphrase.** + +When SSH is invoked without an agent and without a TTY, it cannot decrypt the private key to +sign the authentication challenge. The connection reaches the public-key offer phase (the server +responds PK_OK), but signing fails silently because there is no passphrase source. The server +then rejects the unauthenticated attempt. + +From the host, SSH works because the **GNOME Keyring / SSH agent** has already decrypted the key +and holds it in memory. The agent handles the signing transparently. + +Inside the Docker container there is no SSH agent — and the key file, though readable, cannot be +used for authentication without its passphrase. + +**This was the true root cause of all three provision failures.** The cloud-init timing issue +was an additional factor in attempts 1 and 2, but even with the 300-second probe budget of +attempt 3 (where cloud-init had finished in time), SSH from Docker still failed every attempt +because the passphrase prevented signing. Forwarding the SSH agent socket into the container +confirms this — with the agent socket forwarded, SSH succeeds immediately: + +```bash +docker run --rm --entrypoint bash \ + -v ~/.ssh:/home/deployer/.ssh:ro \ + -v "$SSH_AUTH_SOCK:/tmp/ssh-agent.sock" \ + -e SSH_AUTH_SOCK=/tmp/ssh-agent.sock \ + torrust/tracker-deployer:latest \ + -c "ssh -i /home/deployer/.ssh/torrust_tracker_deployer_ed25519 \ + -o StrictHostKeyChecking=no -o IdentitiesOnly=yes \ + torrust@46.225.234.201 'echo connected'" +# → CONNECTED +``` + +### Resolution + +Remove the passphrase from the deployment key. Deployment keys used for automation do not benefit +from passphrases because: + +- There is no interactive session to prompt for the passphrase. +- The key is already protected by filesystem permissions (`0600`). +- The server already restricts access to injected public keys only. + +```bash +ssh-keygen -p -f ~/.ssh/torrust_tracker_deployer_ed25519 +# Enter current passphrase when prompted +# Leave new passphrase empty (press Enter twice) +``` + +After removing the passphrase, provision works without any agent forwarding: + +```bash +docker run --rm --entrypoint bash \ + -v ~/.ssh:/home/deployer/.ssh:ro \ + torrust/tracker-deployer:latest \ + -c "ssh -i /home/deployer/.ssh/torrust_tracker_deployer_ed25519 \ + -o StrictHostKeyChecking=no -o IdentitiesOnly=yes \ + torrust@46.225.234.201 'echo connected'" +# → CONNECTED +``` + +### Notes + +- Users may prefer to keep the passphrase on their key (e.g. if the same key is also used for + interactive logins). In that case, the alternative is to generate a separate passphrase-free key + specifically for deployment automation and configure it in `ssh_credentials`. +- Hetzner allows setting a default SSH key in the Hetzner console that is automatically added to + all new servers. If a user has already done this (for a key that may be different), SSH may + succeed with the wrong key if `IdentitiesOnly=yes` is not set. Keeping `IdentitiesOnly=yes` + ensures only the explicitly configured deployment key is tried. + +### Prevention + +The deployer `create environment` or `validate` command should detect passphrase-protected keys +early and warn the user: + +```text +⚠ Warning: SSH private key appears to be passphrase-protected. When running via Docker, + no SSH agent is available and the key cannot be used for authentication. + Consider removing the passphrase from the deployment key, or document that the + SSH agent socket must be forwarded into the container. +``` + +--- + +## Problem: Hetzner activity log shows "Server is being created" indefinitely — misleading status + +**Command**: `provision` +**Severity**: Minor (documentation / UX confusion) + +### Symptom + +After a server is fully created, accessible via SSH, and cloud-init is complete, the Hetzner +console activity log still shows **"Server is being created"** as the last event — even 15–20 +minutes later. There is no follow-up event such as "Server creation finished". + +This was observed during provision attempts 3 and 4. In attempt 3, we assumed the server was +still being provisioned by Hetzner ("stuck") because the activity log never updated, which +led to unnecessary investigation into cloud-init delays and SSH timeouts. + +### Root Cause + +This is Hetzner's expected (if unintuitive) behavior. The event **"Server is being created"** +is a point-in-time action record — it records that the creation action was initiated — not a +status that transitions when the action completes. Hetzner does not emit a corresponding +"Server creation completed" event. + +The actual completion can be inferred from: + +- The server **status** in the Hetzner console changing from `Initializing` to `Running` +- SSH connectivity being established +- `cloud-init status` returning `done` on the server + +### Resolution + +No action needed — the "Server is being created" entry is normal and permanent. Use server +**status** (`Running`) and direct SSH access to confirm the server is ready, not the activity +log. + +### Prevention + +Document this behavior in the Hetzner provider guide so operators are not misled. Specifically: + +- The Hetzner activity log records what happened, not the current state. +- "Server is being created" is the last event and does not update when creation finishes. +- Use `hcloud server describe ` or SSH connectivity as the authoritative readiness signal. diff --git a/docs/deployments/hetzner-demo-tracker/commands/release/README.md b/docs/deployments/hetzner-demo-tracker/commands/release/README.md new file mode 100644 index 000000000..5a683a3c6 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/commands/release/README.md @@ -0,0 +1,59 @@ +# Command: release + +> **Status**: ✅ Done (2026-03-04) + +## What `release` does + +The `release` command: + +1. Pulls the latest Docker images for the tracker and monitoring stack on the server. +2. Stages the release artifacts. +3. Marks the environment as `Released` on success. + +It does **not** start the services — that is done by the `run` command. + +## Command + +```bash +docker run --rm \ + -v $(pwd)/data:/var/lib/torrust/deployer/data \ + -v $(pwd)/build:/var/lib/torrust/deployer/build \ + -v $(pwd)/envs:/var/lib/torrust/deployer/envs \ + -v ~/.ssh:/home/deployer/.ssh:ro \ + torrust/tracker-deployer:latest \ + release torrust-tracker-demo 2>&1 | tee -a data/logs/log.txt +``` + +## Output + +```text +[2026-03-04 12:35:20] Starting Torrust Tracker Deployer container... +[2026-03-04 12:35:20] Verifying installed tools... +[2026-03-04 12:35:20] Tool versions: +[2026-03-04 12:35:20] - OpenTofu: OpenTofu v1.11.5 +[2026-03-04 12:35:20] - Ansible: ansible [core 2.20.3] +[2026-03-04 12:35:20] - SSH: OpenSSH_10.0p2 Debian-7, OpenSSL 3.5.4 30 Sep 2025 +[2026-03-04 12:35:20] - Git: git version 2.47.3 +[2026-03-04 12:35:20] SSH directory found, checking permissions... +[2026-03-04 12:35:20] Container initialization complete. Executing command... +⏳ [1/2] Validating environment... +⏳ ✓ Environment name validated: torrust-tracker-demo (took 0ms) +⏳ [2/2] Releasing application... +⏳ ✓ Application released successfully (took 133.5s) +{ + "environment_name": "torrust-tracker-demo", + "instance_name": "torrust-tracker-vm-torrust-tracker-demo", + "provider": "hetzner", + "state": "Released", + "instance_ip": "46.225.234.201", + "created_at": "2026-03-03T19:00:42.481676821Z" +} +``` + +**Duration**: ~134 seconds (image pulls dominate) + +## Problems + +See [bugs.md](bugs.md) for the issue encountered on the first attempt: +`release` failed with `docker not in PATH` when running via the deployer container. +The fix was applied and the image was rebuilt before the successful run above. diff --git a/docs/deployments/hetzner-demo-tracker/commands/release/bugs.md b/docs/deployments/hetzner-demo-tracker/commands/release/bugs.md new file mode 100644 index 000000000..80926ea2d --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/commands/release/bugs.md @@ -0,0 +1,81 @@ +# Release Command — Bugs + +## Bug: `release` fails when deployer runs inside Docker (docker not in PATH) + +**Status**: Fixed in this branch (pending merge) +**Related PR**: [#384](https://github.com/torrust/torrust-tracker-deployer/pull/384) introduced the local validator +**Severity**: High — `release` is completely broken when running the deployer via Docker container + +### Symptom + +Running the `release` command via the Docker container fails immediately with: + +```text +❌ Release command failed: Release command failed: Template rendering failed: +Docker Compose template rendering failed: Rendered docker-compose.yml failed +local validation: Failed to run 'docker compose config --quiet' — is Docker +installed and in PATH? +Source: No such file or directory (os error 2) +``` + +The environment transitions to `ReleaseFailed` state and the deployment cannot continue. + +### Root Cause + +PR [#384](https://github.com/torrust/torrust-tracker-deployer/pull/384) added a +local validation step that runs `docker compose config --quiet` against the rendered +`docker-compose.yml` before uploading it to the remote server. This is a useful check +when the deployer runs natively on a machine where Docker is installed. + +However, the standard production usage is to run the deployer **inside a Docker +container** (`torrust/tracker-deployer:latest`). The deployer container does not +contain a `docker` binary — it has no need to run Docker locally. When `docker` is not +in PATH, the OS returns `ENOENT` (error 2, "No such file or directory"), which the +validator treated as a hard failure. + +### Fix Applied + +The validator in +`src/infrastructure/templating/docker_compose/local_validator.rs` was updated to +handle `io::ErrorKind::NotFound` gracefully: when `docker` is not in PATH, validation +is **skipped** with a warning log rather than failing the command. + +The warning logged when running inside a container: + +```text +WARN Skipping local docker-compose.yml validation: 'docker' is not available +in PATH (deployer may be running inside a container). The rendered file will be +validated by Docker Compose on the remote host. +``` + +Any other OS error (e.g. `PermissionDenied`) is still treated as a hard failure, +since that indicates a real system problem. + +### State Recovery Applied + +After the failed `release`, the environment state was manually reset from +`ReleaseFailed` back to `Configured` so that `release` could be retried: + +```bash +# Before resetting: back up the failed state +cp data/torrust-tracker-demo/environment.json \ + data/torrust-tracker-demo/environment.json.release-failed-bak + +# Reset to Configured (the state serialized as {"Configured": {"context": ..., "state": null}}) +python3 -c " +import json +with open('data/torrust-tracker-demo/environment.json') as f: + data = json.load(f) +context = data['ReleaseFailed']['context'] +new_data = {'Configured': {'context': context, 'state': None}} +with open('data/torrust-tracker-demo/environment.json', 'w') as f: + json.dump(new_data, f, indent=2) +" +``` + +This is the manual state recovery approach described in +[observations.md](../../observations.md#potential-manual-recovery-via-state-snapshot-untested). +It was safe here because: + +- The failure happened at template rendering, **before** any remote action was taken +- The server state was not modified at all during the failed `release` attempt diff --git a/docs/deployments/hetzner-demo-tracker/commands/run/README.md b/docs/deployments/hetzner-demo-tracker/commands/run/README.md new file mode 100644 index 000000000..3bd3b0bf6 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/commands/run/README.md @@ -0,0 +1,108 @@ +# Command: run + +> **Status**: ✅ Completed (2026-03-04) — state: `Running` +> +> **Note**: The `run` command succeeded and the environment transitioned to +> `Running`, but the tracker container was in a restart loop due to +> [Bug 3](bugs.md#bug-3-mysql-password-is-not-url-encoded-in-the-tracker-connection-string) +> (URL encoding). A manual fix was applied before the `test` command was run. +> See [bugs.md](bugs.md) for details. + +## What `run` does + +The `run` command: + +1. Validates that the environment is in `Released` state. +2. Runs the `run-compose-services` Ansible playbook on the remote server. +3. The playbook pulls any updated Docker images, then runs `docker compose up -d`. +4. Transitions the environment state to `Running` on success. + +It requires the environment to already be in a `Released` state (i.e., `release` +must have been run first). + +**Important**: `Running` state only guarantees that `docker compose up -d` +returned successfully — not that all services are healthy and reachable. Use the +`test` command immediately after `run` to verify the stack. +See [improvements.md](improvements.md) for more context. + +## Command + +```bash +docker run --rm \ + -v $(pwd)/data:/var/lib/torrust/deployer/data \ + -v $(pwd)/build:/var/lib/torrust/deployer/build \ + -v ~/.ssh:/home/deployer/.ssh:ro \ + torrust/tracker-deployer:latest \ + run torrust-tracker-demo 2>&1 | tee -a data/logs/log.txt +``` + +## Output + +```text +[2026-03-04 13:19:16] Starting Torrust Tracker Deployer container... +[2026-03-04 13:19:16] Verifying installed tools... +[2026-03-04 13:19:16] Tool versions: +[2026-03-04 13:19:16] - OpenTofu: OpenTofu v1.11.5 +[2026-03-04 13:19:16] - Ansible: ansible [core 2.20.3] +[2026-03-04 13:19:16] - SSH: OpenSSH_10.0p2 Debian-7, OpenSSL 3.5.4 30 Sep 2025 +[2026-03-04 13:19:16] - Git: git version 2.47.3 +[2026-03-04 13:19:16] SSH directory found, checking permissions... +[2026-03-04 13:19:16] Container initialization complete. Executing command... +⏳ [1/2] Validating environment... +⏳ ✓ Environment name validated: torrust-tracker-demo (took 0ms) +⏳ [2/2] Running application services... +⏳ ✓ Services started (took 31.6s) +✅ Run command completed for 'torrust-tracker-demo' +{ + "environment_name": "torrust-tracker-demo", + "state": "Running", + "services": { + "udp_trackers": [ + "udp://udp1.torrust-tracker-demo.com:6969/announce", + "udp://udp2.torrust-tracker-demo.com:6868/announce" + ], + "https_http_trackers": [ + "https://http1.torrust-tracker-demo.com/announce", + "https://http2.torrust-tracker-demo.com/announce" + ], + "direct_http_trackers": [], + "localhost_http_trackers": [], + "api_endpoint": "https://api.torrust-tracker-demo.com/api", + "api_uses_https": true, + "api_is_localhost_only": false, + "health_check_url": "http://46.225.234.201:1313/health_check", + "health_check_uses_https": false, + "health_check_is_localhost_only": true, + "tls_domains": [ + { + "domain": "http1.torrust-tracker-demo.com", + "internal_port": 7070 + }, + { + "domain": "http2.torrust-tracker-demo.com", + "internal_port": 7071 + }, + { + "domain": "api.torrust-tracker-demo.com", + "internal_port": 1212 + }, + { + "domain": "grafana.torrust-tracker-demo.com", + "internal_port": 3000 + } + ] + }, + "grafana": { + "url": "https://grafana.torrust-tracker-demo.com/", + "uses_https": true + } +} +``` + +### Duration + +- Total: ~31.6s (Ansible `docker compose up -d` + container startup) + +### State Transition + +`Released` → `Running` diff --git a/docs/deployments/hetzner-demo-tracker/commands/run/bugs.md b/docs/deployments/hetzner-demo-tracker/commands/run/bugs.md new file mode 100644 index 000000000..240b1206d --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/commands/run/bugs.md @@ -0,0 +1,231 @@ +# Run Command — Bugs + +## Bug 1: MySQL app username `"root"` is not rejected at environment creation time + +**Status**: Open — no fix applied yet +**Severity**: High — silently generates an invalid configuration that prevents MySQL from starting + +### Symptom + +The `run` command fails because the MySQL container is unhealthy. MySQL 8.4 +refuses to start with `MYSQL_USER="root"` and prints the following error +repeatedly in its logs: + +```text +[ERROR] [Entrypoint]: MYSQL_USER="root", MYSQL_USER and MYSQL_PASSWORD are for +configuring a regular user and cannot be used for the root user. + Remove MYSQL_USER="root" and use one of the following to control the root + user password: + - MYSQL_ROOT_PASSWORD + - MYSQL_ALLOW_EMPTY_PASSWORD + - MYSQL_RANDOM_ROOT_PASSWORD +``` + +Because MySQL never reaches a healthy state, the dependent containers (tracker, +caddy, prometheus, grafana) all fail to start. + +### Root Cause + +The deployer accepts any string as `username` in the MySQL database block of the +environment JSON config. When `username` is `"root"`, the template renderer +places it into `MYSQL_USER` in the generated `.env` file: + +```env +MYSQL_USER='root' ← MySQL 8.4 rejects this +``` + +MySQL's `MYSQL_USER` environment variable is intended solely for creating a new +**non-root application user** on first boot. The root user is managed exclusively +via `MYSQL_ROOT_PASSWORD`. Passing `MYSQL_USER=root` is explicitly rejected since +MySQL 8.4. + +The deployer does not validate this constraint. It silently renders an unusable +configuration, which only fails later at `run` time (when MySQL actually starts), +not at creation or release time. + +### Affected Code + +- `src/application/services/rendering/docker_compose.rs` — + `create_mysql_contexts()` passes `username` directly to `MYSQL_USER` without + checking for the reserved value `"root"`. +- `templates/docker-compose/.env.tera` — renders `MYSQL_USER='{{ mysql.user }}'` + verbatim. + +### Proposed Fix + +The deployer should reject `"root"` as a MySQL application username at +**environment creation time**, returning a clear user-facing error such as: + +```text +❌ Invalid MySQL configuration: username "root" is reserved. + Use a non-root application username (e.g. "torrust"). + The MySQL root user is managed automatically via MYSQL_ROOT_PASSWORD. +``` + +The validation should live in the domain layer, close to the environment config +parsing step, so that it fails early and never reaches the rendering stage. + +--- + +## Bug 2: MySQL root password silently diverges from the configured password + +**Status**: Open — no fix applied yet +**Severity**: Medium — the root password is unguessable from the env config, but +the effective root password is inconsistent with what the operator provided + +### Symptom + +The MySQL root user password set inside the container is **not** the password +specified in the environment JSON config. If any tool or script tries to connect +to MySQL as `root` using the configured password it will be denied. + +With `password = "secret"` in the env config, the `.env` file on the server +contains: + +```env +MYSQL_ROOT_PASSWORD='secret_root' ← not "secret" +MYSQL_PASSWORD='secret' +``` + +The auto-derived root password (`secret_root`) is never surfaced to the operator. + +### Root Cause + +The feature was never implemented. In +`src/application/services/rendering/docker_compose.rs`, `create_mysql_contexts()` +contains a placeholder that derives the root password by appending `"_root"` to +the configured password, with a comment acknowledging the gap: + +```rust +// For MySQL, generate a secure root password (in production, this should be managed securely) +let root_password = format!("{password}_root"); +``` + +The comment explicitly says this should be managed securely in production, but +no proper implementation was ever added. The `"_root"` suffix is a stub, not a +deliberate design decision. As a result the root password silently diverges from +the configured password and the gap is invisible to the operator. + +### Affected Code + +- `src/application/services/rendering/docker_compose.rs` — + the `format!("{password}_root")` derivation inside `create_mysql_contexts()`. + +### Proposed Fix + +The root password should be either: + +1. **Explicitly configurable** — expose a separate `root_password` field in the + environment JSON config so the operator controls it directly; or +2. **Randomly generated at environment creation time** and stored in the + environment state so it can be retrieved if needed. + +Option 2 is preferred because it eliminates any predictable relationship between +the app password and the root password. + +A migration path for existing deployments should be documented and the old +`format!("{password}_root")` derivation removed. + +--- + +## Bug 3: MySQL password is not URL-encoded in the tracker connection string + +**Status**: Open — no fix applied yet (worked around manually in this deployment) +**Severity**: High — tracker crashes at startup whenever the MySQL password +contains URL-special characters (e.g. `/`, `@`, `#`, `?`, `+`) + +### Symptom + +The `run` command completes and the state transitions to `Running`, but the +tracker container immediately enters a restart loop with exit code 101. Its +logs show: + +```text +thread 'main' (1) panicked at packages/configuration/src/v2_0_0/database.rs:50:54: +path for MySQL driver should be a valid URL: InvalidPort +``` + +The `InvalidPort` error is misleading — it is caused by the password containing +a `/` character, which the URL parser treats as the start of the database-name +path segment, making the port field invalid. + +### Root Cause + +The `tracker.toml` template renders the MySQL password raw into the connection +URL: + +```text +mysql://{username}:{password}@{host}:{port}/{database} +``` + +When the password contains characters that have special meaning in a URL (such as +`/`, `@`, `#`, `?`, `+`, `%`), the resulting URL is malformed. The tracker +configuration parser rejects it at startup. + +The deployer template does not URL-encode the password before substituting it +into the connection string. + +The generated `tracker.toml` in this deployment contained: + +```toml +# The password contains a '/' character: +path = "mysql://torrust:@mysql:3306/torrust_tracker" +``` + +The `/` in the password was interpreted as a URL path separator, corrupting +the URL structure. + +### Affected Code + +The template or the Rust wrapper that renders `tracker.toml` — somewhere in the +rendering pipeline the password value is inserted as a raw string into the URL. +The exact file is in `templates/tracker/` or the corresponding Rust wrapper +under `src/infrastructure/templating/`. + +### Recovery Applied (Manual Workaround) + +After `run` transitioned the state to `Running`, the tracker restart loop was +diagnosed from SSH: + +```bash +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@46.225.234.201 +cd /opt/torrust && docker compose logs tracker --tail 30 +``` + +The `tracker.toml` file was manually fixed locally by URL-encoding the `/` as +`%2F` in the connection string: + +```toml +# Before (broken — '/' in password breaks URL parsing): +path = "mysql://torrust:@mysql:3306/torrust_tracker" + +# After (fixed — '/' encoded as '%2F'): +path = "mysql://torrust:@mysql:3306/torrust_tracker" +``` + +The fixed file was uploaded and the tracker container restarted: + +```bash +scp -i ~/.ssh/torrust_tracker_deployer_ed25519 \ + build/torrust-tracker-demo/tracker/tracker.toml \ + torrust@46.225.234.201:/tmp/ +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@46.225.234.201 " + sudo cp /tmp/tracker.toml /opt/torrust/storage/tracker/etc/tracker.toml + cd /opt/torrust && sudo docker compose restart tracker +" +``` + +The tracker came up healthy after the restart. The environment state was already +`Running` (the state transition happened when `docker compose up -d` returned +successfully), so no state reset was required. + +### Proposed Fix + +The rendering layer must percent-encode any characters with special URL meaning +in the password before substituting it into the MySQL connection URL. At minimum +the following characters must be encoded: `/`, `@`, `#`, `?`, `+`, `%`, `&`, +`=`, ` `. + +The correct place to apply this is in the Rust wrapper or Tera template that +builds the `tracker.toml` connection string, not in the raw password value +stored in the env config. diff --git a/docs/deployments/hetzner-demo-tracker/commands/run/improvements.md b/docs/deployments/hetzner-demo-tracker/commands/run/improvements.md new file mode 100644 index 000000000..fd1422f85 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/commands/run/improvements.md @@ -0,0 +1,63 @@ +# Run Command — Improvements + +## Improvement: `Running` state does not guarantee services are healthy + +**Status**: Documented — existing `test` command is the recommended solution +**Type**: Clarification / UX + +### Observation + +After the `run` command completes and the environment transitions to `Running`, +the individual containers are not necessarily healthy. In this deployment the +state became `Running` while the tracker container was stuck in a restart loop +due to [Bug 3](bugs.md#bug-3-mysql-password-is-not-url-encoded-in-the-tracker-connection-string). +The deployer gave no indication that anything was wrong. + +The `Running` state only means that `docker compose up -d` exited with code 0 — +i.e. Docker accepted the request to start the stack. It says nothing about +whether each service is actually reachable and functioning. + +### Why the `run` Command Does Not Wait for Health + +Waiting inside `run` for all containers to become healthy is difficult in +practice because: + +- Health checks vary per service and are defined in `docker-compose.yml` with + service-specific commands and timeouts. +- Some services (e.g. Caddy obtaining TLS certificates) can take tens of seconds + or minutes to become fully operational depending on DNS propagation and the + ACME provider. +- The deployer has no deep knowledge of what "healthy" means for each service + beyond what Docker's own health-check reports, and Docker's health-check for + the tracker does not distinguish between "still starting" and "crashed". + +Blocking the `run` command until everything is provably healthy would require +duplicating the logic that is already expressed in the `test` command (smoke +tests). + +### Recommended Approach + +The deployer already provides a separate `test` command that runs smoke tests +against the deployed stack. That is the right tool for verifying that services +are actually reachable and responding correctly after `run` completes. + +The intended workflow is: + +```text +release → run → test +``` + +`run` starts the stack; `test` confirms it works. Operators should always run +`test` after `run` and treat a passing `test` as the definitive signal that the +deployment is functional. + +### Possible Future Improvement + +A lightweight post-start check could be added to `run` that waits for Docker's +own health status (not full smoke-test level) to settle — for example polling +`docker compose ps` until no container is in `starting` or `restarting` state, +with a configurable timeout. This would catch fast-failing containers like the +tracker URL-encoding crash without requiring the full `test` logic inside `run`. + +This should be considered only if the cost of running `test` immediately after +`run` becomes a significant friction point for operators. diff --git a/docs/deployments/hetzner-demo-tracker/commands/run/problems.md b/docs/deployments/hetzner-demo-tracker/commands/run/problems.md new file mode 100644 index 000000000..c44b3e3d0 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/commands/run/problems.md @@ -0,0 +1,101 @@ +# Run Command — Problems + +## Problem: `run` fails — MySQL container is unhealthy (`MYSQL_USER="root"` rejected) + +**Status**: Root cause identified (2026-03-04) +**Severity**: High — `run` is blocked + +### Symptom + +The `run` command fails at the "Start Docker Compose services" Ansible task: + +```text +❌ Run command failed: Failed to start services in environment 'environment': +Ansible playbook 'run-compose-services' failed: Command +'ansible-playbook -v run-compose-services.yml' failed with exit code 2 +``` + +The Docker Compose `up -d` output shows all networks and containers were created, +but `mysql` failed its health check and the dependent containers could not start: + +```text + Container mysql Error +dependency failed to start: container mysql is unhealthy +``` + +### Root Cause + +MySQL 8.4 refuses to start when `MYSQL_USER="root"` is set, because `root` is a +reserved/privileged MySQL user that cannot be created via the `MYSQL_USER` +environment variable. Its log shows this repeatedly: + +```text +[ERROR] [Entrypoint]: MYSQL_USER="root", MYSQL_USER and MYSQL_PASSWORD are for +configuring a regular user and cannot be used for the root user + Remove MYSQL_USER="root" and use one of the following to control the root + user password: + - MYSQL_ROOT_PASSWORD + - MYSQL_ALLOW_EMPTY_PASSWORD + - MYSQL_RANDOM_ROOT_PASSWORD +``` + +The generated `.env` file on the server contains: + +```env +MYSQL_ROOT_PASSWORD='_root' +MYSQL_DATABASE='torrust' +MYSQL_USER='root' ← INVALID: MySQL 8.4 rejects this +MYSQL_PASSWORD='' +``` + +### Why This Happened + +The environment config (`envs/lxd-local-example.json`) was used as the basis for +the Hetzner deployment and had `"username": "root"` for the MySQL database +config. + +The deployer's template rendering service +(`src/application/services/rendering/docker_compose.rs`) passes `username` +directly as `MYSQL_USER`. It is designed to work with a **non-root application +user** — the `root` MySQL user is managed separately via `MYSQL_ROOT_PASSWORD`, +which is auto-derived as `{configured_password}_root`. + +Using `"username": "root"` in the environment config is an unsupported +configuration: it produces `MYSQL_USER=root` which MySQL 8.4 rejects. + +### Fix Required + +The environment config must use a **non-root MySQL username** (e.g. `torrust`). + +**Required change in the env config** (`envs/lxd-local-example.json` → +`envs/hetzner-demo.json`): + +```json +"database": { + "driver": "mysql", + "config": { + "host": "mysql", + "port": 3306, + "database_name": "torrust", + "username": "torrust", ← was "root" + "password": "..." + } +} +``` + +After updating the env config: + +1. Re-run `release` to regenerate the `.env` and `docker-compose.yml` with the + correct username. +2. Re-run `run` to start the services. + +Note: the environment state must first be reset from `RunFailed` to `Released` +before `release` can be retried. See +[observations.md](../../observations.md#potential-manual-recovery-via-state-snapshot-untested) +for the state recovery procedure. + +### Related Deployer Improvement + +The deployer should validate that `MYSQL_USER` is not `root` at environment +creation or template rendering time, and return a clear error instead of +silently generating an invalid configuration. diff --git a/docs/deployments/hetzner-demo-tracker/commands/test/README.md b/docs/deployments/hetzner-demo-tracker/commands/test/README.md new file mode 100644 index 000000000..0f7a383e7 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/commands/test/README.md @@ -0,0 +1,111 @@ +# Command: test + +> **Status**: ✅ Completed (2026-03-04) — result: `pass` (with DNS warnings) + +## What `test` does + +The `test` command runs smoke tests against a deployed environment to verify +that the infrastructure and services are reachable and responding correctly. +It is the recommended step to run immediately after `run`. + +The test command: + +1. Validates that the environment is in `Running` state. +2. Performs DNS resolution checks for each configured domain, comparing the + resolved IP against the server's instance IP. +3. Reports a `pass` result if the infrastructure tests succeed, even if there + are DNS warnings. + +**Note**: These are infrastructure-level smoke tests. They do not test tracker +protocol logic (announce, scrape, etc.) — those are covered by deeper integration +tests. + +## Command + +```bash +docker run --rm \ + -v $(pwd)/data:/var/lib/torrust/deployer/data \ + -v $(pwd)/build:/var/lib/torrust/deployer/build \ + -v ~/.ssh:/home/deployer/.ssh:ro \ + torrust/tracker-deployer:latest \ + test torrust-tracker-demo 2>&1 | tee -a data/logs/log.txt +``` + +## Output + +```text +[2026-03-04 14:06:48] Starting Torrust Tracker Deployer container... +[2026-03-04 14:06:48] Verifying installed tools... +[2026-03-04 14:06:48] Tool versions: +[2026-03-04 14:06:48] - OpenTofu: OpenTofu v1.11.5 +[2026-03-04 14:06:48] - Ansible: ansible [core 2.20.3] +[2026-03-04 14:06:48] - SSH: OpenSSH_10.0p2 Debian-7, OpenSSL 3.5.4 30 Sep 2025 +[2026-03-04 14:06:48] - Git: git version 2.47.3 +[2026-03-04 14:06:48] SSH directory found, checking permissions... +[2026-03-04 14:06:48] Container initialization complete. Executing command... +⏳ [1/3] Validating environment... +⏳ ✓ Environment name validated: torrust-tracker-demo (took 0ms) +⏳ [2/3] Creating command handler... +⏳ ✓ Done (took 0ms) +⏳ [3/3] Testing infrastructure... +⚠️ DNS check: api.torrust-tracker-demo.com resolves to [116.202.176.169] but expected 46.225.234.201 +⚠️ DNS check: http1.torrust-tracker-demo.com resolves to [116.202.176.169] but expected 46.225.234.201 +⚠️ DNS check: http2.torrust-tracker-demo.com resolves to [116.202.176.169] but expected 46.225.234.201 +⚠️ DNS check: grafana.torrust-tracker-demo.com resolves to [116.202.176.169] but expected 46.225.234.201 +⏳ ✓ Infrastructure tests passed (with DNS warnings) (took 1.0s) +{ + "environment_name": "torrust-tracker-demo", + "instance_ip": "46.225.234.201", + "result": "pass", + "dns_warnings": [ + { + "domain": "api.torrust-tracker-demo.com", + "expected_ip": "46.225.234.201", + "issue": "api.torrust-tracker-demo.com resolves to [116.202.176.169] but expected 46.225.234.201" + }, + { + "domain": "http1.torrust-tracker-demo.com", + "expected_ip": "46.225.234.201", + "issue": "http1.torrust-tracker-demo.com resolves to [116.202.176.169] but expected 46.225.234.201" + }, + { + "domain": "http2.torrust-tracker-demo.com", + "expected_ip": "46.225.234.201", + "issue": "http2.torrust-tracker-demo.com resolves to [116.202.176.169] but expected 46.225.234.201" + }, + { + "domain": "grafana.torrust-tracker-demo.com", + "expected_ip": "46.225.234.201", + "issue": "grafana.torrust-tracker-demo.com resolves to [116.202.176.169] but expected 46.225.234.201" + } + ] +} +``` + +### Duration + +- Total: ~1.0s + +### State Transition + +No state transition — `test` is a read-only verification step. The environment +remains in `Running` state. + +## DNS Warnings Explained + +The test checks that each domain resolves to the server's **instance IP** +(`46.225.234.201`). All four domains instead resolve to `116.202.176.169`, which +is the **floating IP** assigned to this deployment. + +This is expected and correct behavior for this setup. The DNS records +deliberately point to the **floating IP**, not the instance IP. The floating IP +is a separate Hetzner resource that can be reassigned to a different server +instance without changing any DNS records — enabling zero-downtime failover. The +instance IP (`46.225.234.201`) is the bare VM's IP and is not published in DNS. + +The deployer's `test` command currently expects the domain to resolve to the +instance IP, so it raises a warning whenever a floating IP is in use. This is a +deployer limitation, not a problem with the deployment. + +See [../improvements.md](../improvements.md) for a proposed improvement to make +the deployer aware of floating IPs. diff --git a/docs/deployments/hetzner-demo-tracker/deployment-spec.md b/docs/deployments/hetzner-demo-tracker/deployment-spec.md new file mode 100644 index 000000000..262221881 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/deployment-spec.md @@ -0,0 +1,166 @@ +# Deployment Specification + +Decisions and desired end-state for the demo tracker at `torrust-tracker-demo.com`. + +> **Note**: This document describes _what_ we want to deploy. For step-by-step command +> documentation see the per-command sub-directories under `commands/` (`commands/create/`, +> `commands/provision/`, etc.). +> +> All secrets, API tokens, and passwords shown here are placeholders. Never commit real +> credentials. + +## Configuration Decisions + +| Decision | Choice | Rationale | +| ----------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------ | +| Deployment method | Docker (`torrust/tracker-deployer:latest`) | Reproducible, no local dependency management; Docker is the recommended approach | +| Server type | `ccx23` — 4 vCPU, 16 GB RAM | Sufficient headroom for a demo with full monitoring stack (tracker + Prometheus + Grafana) | +| Location | `nbg1` (Nuremberg, Germany) | Close to our development machines; matches the floating IP location | +| OS image | `ubuntu-24.04` | LTS release, well-tested with Ansible playbooks | +| Database | MySQL | Production-ready; better suited for a public demo with real traffic | +| HTTPS | Yes — Let's Encrypt (`use_staging: false`) | Production certificates via Caddy reverse proxy; domain is fully registered | +| Backup | Daily at 03:00 UTC, 7-day retention | Good default for a demo; protects against accidental data loss | + +## Service Endpoints + +| Service | URL / Address | +| ---------------- | --------------------------------------------------- | +| HTTP Tracker 1 | `https://http1.torrust-tracker-demo.com/announce` | +| HTTP Tracker 2 | `https://http2.torrust-tracker-demo.com/announce` | +| UDP Tracker 1 | `udp://udp1.torrust-tracker-demo.com:6969/announce` | +| UDP Tracker 2 | `udp://udp2.torrust-tracker-demo.com:6868/announce` | +| REST API | `https://api.torrust-tracker-demo.com` | +| Grafana | `https://grafana.torrust-tracker-demo.com` | +| Health Check API | `127.0.0.1:1313` (internal only) | + +## How Decisions Were Made + +Configuration decisions were made interactively with an AI coding agent (GitHub Copilot). The +agent asked targeted questions before generating the config file. The screenshots below capture +those questions and our answers. + +**Should HTTPS with Let's Encrypt be enabled?** + +![AI agent question: Let's Encrypt](media/configuratrion-ai-agent-questions/question-lets-encrypt.png) + +Answer: Yes — production certificates via Caddy. + +**Which Hetzner server type?** + +![AI agent question: server type](media/configuratrion-ai-agent-questions/question-server-type.png) + +Answer: Custom entry `ccx23` — 4 vCPU, 16 GB RAM. + +**What email for Let's Encrypt certificate notifications?** + +![AI agent question: Let's Encrypt email](media/configuratrion-ai-agent-questions/question-lets-encrypt-email.png) + +Answer: `jose.celano@nautilus-cyberneering.dev` + +**What subdomains for each service?** + +![AI agent question: service subdomains](media/configuratrion-ai-agent-questions/question-service-subdomains.png) + +Answer: + +- HTTP Tracker 1: `https://http1.torrust-tracker-demo.com/announce` +- HTTP Tracker 2: `https://http2.torrust-tracker-demo.com/announce` +- UDP Tracker 1: `udp://udp1.torrust-tracker-demo.com:6969/announce` +- UDP Tracker 2: `udp://udp2.torrust-tracker-demo.com:6868/announce` +- REST API: `https://api.torrust-tracker-demo.com` +- Grafana: `https://grafana.torrust-tracker-demo.com` + +> **Note**: The agent also silently used SQLite by default. We caught this and switched to MySQL. +> The bind addresses were also defaulted to `0.0.0.0` (IPv4 only) — we changed them to `[::]` for +> dual-stack. Both issues are documented in [commands/create/problems.md](commands/create/problems.md). + +## Environment Configuration File (sanitized) + +The file is stored at `envs/torrust-tracker-demo.json` (git-ignored — contains the real API token). + +```json +{ + "environment": { + "name": "torrust-tracker-demo", + "description": "Torrust Tracker demo instance at torrust-tracker-demo.com" + }, + "ssh_credentials": { + "private_key_path": "/home/deployer/.ssh/torrust_tracker_deployer_ed25519", + "public_key_path": "/home/deployer/.ssh/torrust_tracker_deployer_ed25519.pub", + "username": "torrust", + "port": 22 + }, + "provider": { + "provider": "hetzner", + "api_token": "", + "server_type": "ccx23", + "location": "nbg1", + "image": "ubuntu-24.04" + }, + "tracker": { + "core": { + "database": { + "driver": "mysql", + "host": "mysql", + "port": 3306, + "database_name": "torrust", + "username": "root", + "password": "" + }, + "private": false + }, + "udp_trackers": [ + { + "bind_address": "[::]:6969", + "domain": "udp1.torrust-tracker-demo.com" + }, + { "bind_address": "[::]:6868", "domain": "udp2.torrust-tracker-demo.com" } + ], + "http_trackers": [ + { + "bind_address": "[::]:7070", + "domain": "http1.torrust-tracker-demo.com", + "use_tls_proxy": true + }, + { + "bind_address": "[::]:7071", + "domain": "http2.torrust-tracker-demo.com", + "use_tls_proxy": true + } + ], + "http_api": { + "bind_address": "[::]:1212", + "admin_token": "", + "domain": "api.torrust-tracker-demo.com", + "use_tls_proxy": true + }, + "health_check_api": { + "bind_address": "127.0.0.1:1313" + } + }, + "prometheus": { + "scrape_interval_in_secs": 15 + }, + "grafana": { + "admin_user": "admin", + "admin_password": "", + "domain": "grafana.torrust-tracker-demo.com", + "use_tls_proxy": true + }, + "https": { + "admin_email": "", + "use_staging": false + }, + "backup": { + "schedule": "0 3 * * *", + "retention_days": 7 + } +} +``` + +## Related Documentation + +- [Hetzner server types and pricing](../../user-guide/providers/hetzner.md#available-server-types) +- [HTTPS/TLS configuration](../../user-guide/services/https.md) +- [Grafana service](../../user-guide/services/grafana.md) +- [Prometheus service](../../user-guide/services/prometheus.md) diff --git a/docs/deployments/hetzner-demo-tracker/improvements.md b/docs/deployments/hetzner-demo-tracker/improvements.md new file mode 100644 index 000000000..d6f3f5914 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/improvements.md @@ -0,0 +1,180 @@ +# Improvements & Recommendations from Hetzner Demo Tracker Deployment + +All deployer improvements and recommendations identified during this deployment, +collected in one place. Each entry links to the full description in the relevant +document. + +--- + +## `create` command + +### I-01 — Document `instance_name: null` auto-generation in template + +The `create template` output contains `"instance_name": null` with no explanation +of the auto-generated value (`torrust-tracker-vm-{env_name}`). The template should +include an inline comment or a `_comment` field describing this behavior. + +Full description: [commands/create/problems.md — instance_name: null unexplained](commands/create/problems.md#problem-template-generates-instance_name-null-with-no-explanation) + +--- + +### I-02 — Default bind addresses to `[::]` (dual-stack) for public trackers + +The `create template` command defaults to `0.0.0.0` (IPv4 only). Public trackers +should bind to `[::]`, which accepts both IPv4 and IPv6 on Linux. The template +generator should either default to `[::]` or include a note about the trade-off. + +Full description: [commands/create/problems.md — Template defaults to `0.0.0.0`](commands/create/problems.md#problem-template-defaults-bind-addresses-to-0000-ipv4-only) + +--- + +### I-03 — Prompt for database choice or note SQLite dev default + +The `create template` command silently selects SQLite without informing the user. +It should either prompt for a database choice interactively or include a comment +noting that SQLite is the development default and MySQL is recommended for +production. + +Full description: [commands/create/problems.md — Template silently defaults to SQLite](commands/create/problems.md#problem-template-silently-defaults-to-sqlite--no-database-choice-presented) + +--- + +## `provision` command + +### I-04 — Distinguish SSH failure reason in the probe loop + +The SSH probe logs a generic "still waiting" message for every failed attempt +regardless of whether the port is unreachable (TCP timeout) or sshd is up +but authentication is rejected. Logging a different message per failure type +would significantly reduce investigation time. + +Full description: [commands/provision/improvements.md — Distinguish SSH failure reason](commands/provision/improvements.md#1-distinguish-ssh-failure-reason-in-the-probe-loop) + +--- + +### I-05 — Classify `error_kind` more precisely for SSH auth failures + +A `WaitSshConnectivity` failure is always recorded as `NetworkConnectivity` in +the environment JSON, even when the root cause is authentication rejection (not +a network problem). A more specific `SshAuthenticationFailed` variant would +direct investigation to the right layer immediately. + +Full description: [commands/provision/improvements.md — Classify error_kind more precisely](commands/provision/improvements.md#2-classify-error_kind-more-precisely-for-auth-failures) + +--- + +### I-06 — Include per-attempt failure details in the provision trace file + +The trace file only records a final summary. A condensed per-phase breakdown +of the SSH probe (how many attempts timed out vs. were rejected by sshd) would +be immediately actionable for operators without requiring analysis of +`data/logs/log.txt`. + +Full description: [commands/provision/improvements.md — Per-attempt details in trace file](commands/provision/improvements.md#3-include-per-attempt-failure-details-in-the-trace-file) + +--- + +### I-07 — Make SSH connectivity timeout configurable + +The probe budget is hardcoded at 60 × 2 s = 120 s. Hetzner servers with +cloud-init user provisioning require over 3 minutes. This should be +configurable per provider, per env config, and via a CLI flag, with a longer +default for Hetzner. + +Full description: [commands/provision/improvements.md — Configurable SSH connectivity timeout](commands/provision/improvements.md#4-support-configurable-ssh-connectivity-timeout) + +--- + +### I-08 — Detect passphrase-protected SSH keys early and warn + +The deployer does not check whether the configured SSH private key has a +passphrase. When running inside Docker (no agent, no TTY), a passphrase-protected +key silently fails every attempt. This should be caught at `create environment` +or `validate` time, with a clear actionable warning. + +Full description: [commands/provision/improvements.md — Detect passphrase-protected keys early](commands/provision/improvements.md#7-detect-passphrase-protected-ssh-keys-early-and-warn-the-user) + +--- + +### I-09 — Add `wait-for-ssh` command or `provision --resume` flag + +When `provision` fails at `WaitSshConnectivity`, the server itself is healthy +but the environment must be destroyed and recreated from scratch. A +`wait-for-ssh` command or `provision --resume` flag would allow retrying only +the SSH probe step against an already-created server, saving the full +`tofu apply` + cloud-init cycle. + +Full description: [commands/provision/improvements.md — `wait-for-ssh` command](commands/provision/improvements.md#5-add-a-wait-for-ssh-command-or---resume-flag-on-provision) + +--- + +### I-10 — Include IPv6 address in `provision` output + +The `provision` JSON output only includes the IPv4 instance IP. Hetzner +assigns an IPv6 address and /64 network to every server, but these are only +visible in the raw Tofu state file. Exposing them in the output avoids +operators having to consult the state file during post-provision steps like +floating IP setup. + +Full description: [commands/provision/improvements.md — Include IPv6 in provision output](commands/provision/improvements.md#6-include-ipv6-address-in-provision-command-output) + +--- + +## `run` command + +### I-11 — Add lightweight post-start health check to `run` + +The `Running` state only indicates that `docker compose up -d` returned exit +code 0, not that services are healthy. A lightweight poll of `docker compose ps` +after startup (until no container is in `starting` or `restarting` state) would +catch fast-failing containers — such as the tracker URL-encoding crash in this +deployment — without duplicating the full `test` smoke-test logic. + +Full description: [commands/run/improvements.md — `Running` state does not guarantee healthy services](commands/run/improvements.md#improvement-running-state-does-not-guarantee-services-are-healthy) + +--- + +## Cross-cutting + +### I-12 — Add floating IP support to environment config and DNS checks + +The `test` command compares resolved DNS addresses against the bare instance IP. +When a floating IP is in use (the recommended production setup), every domain +produces a false-positive DNS warning. The environment config should allow +specifying a `floating_ip` so the deployer uses it as the expected DNS target +and can optionally auto-assign it during provisioning. + +Full description: [commands/improvements.md — Deployer not aware of floating IPs](commands/improvements.md#improvement-deployer-is-not-aware-of-floating-ips) + +--- + +## Post-provision / operational + +### I-13 — Write netplan floating IP config with correct permissions from the start + +The post-provision guide writes the netplan file with `sudo tee`, which creates +it world-readable. Netplan then logs a warning requiring `chmod 600`. Using +`sudo install -m 600 /dev/stdin /etc/netplan/60-floating-ip.yaml` instead +avoids the warning and the manual fix step. + +Full description: [post-provision/dns-setup.md — Improvements](post-provision/dns-setup.md#improvements) + +--- + +## Summary + +| ID | Area | Description | +| ---- | -------------- | ------------------------------------------------------------ | +| I-01 | `create` | Document `instance_name` auto-generation in template | +| I-02 | `create` | Default bind addresses to `[::]` for public trackers | +| I-03 | `create` | Prompt for database choice or note SQLite dev default | +| I-04 | `provision` | Distinguish SSH failure reason in probe loop | +| I-05 | `provision` | Classify `error_kind` more precisely for SSH auth failures | +| I-06 | `provision` | Include per-attempt SSH failure details in trace file | +| I-07 | `provision` | Make SSH connectivity timeout configurable | +| I-08 | `provision` | Detect passphrase-protected SSH keys early and warn | +| I-09 | `provision` | Add `wait-for-ssh` command or `provision --resume` flag | +| I-10 | `provision` | Include IPv6 address in provision output | +| I-11 | `run` | Add lightweight post-start health check | +| I-12 | Cross-cutting | Add floating IP support to env config and DNS checks | +| I-13 | Post-provision | Write netplan config with correct permissions from the start | diff --git a/docs/deployments/hetzner-demo-tracker/maintenance/README.md b/docs/deployments/hetzner-demo-tracker/maintenance/README.md new file mode 100644 index 000000000..2b3d028ca --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/maintenance/README.md @@ -0,0 +1,11 @@ +# Maintenance + +Post-deployment maintenance procedures for the Hetzner demo tracker instance. + +## Tasks + +| Task | Guide | Status | +| ----------------- | -------------------------------------------- | ---------- | +| Secrets Rotation | [secrets-rotation.md](secrets-rotation.md) | ✅ Done | +| OS Updates | [os-updates.md](os-updates.md) | ⏳ Pending | +| Uptime Monitoring | [uptime-monitoring.md](uptime-monitoring.md) | ⏳ Pending | diff --git a/docs/deployments/hetzner-demo-tracker/maintenance/os-updates.md b/docs/deployments/hetzner-demo-tracker/maintenance/os-updates.md new file mode 100644 index 000000000..b671a57f0 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/maintenance/os-updates.md @@ -0,0 +1,77 @@ +# OS Updates + +**Instance**: Hetzner demo tracker (`46.225.234.201`) +**OS**: Ubuntu 24.04.3 LTS + +Apply OS security and package updates periodically to keep the server patched. + +## When to Run + +- After initial deployment (first login often shows pending updates) +- When the login banner reports security updates available +- At least once a month as routine maintenance + +## Procedure + +### 1. SSH into the server + +```bash +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@46.225.234.201 +``` + +The login banner shows the number of pending updates: + +```text +59 updates can be applied immediately. +37 of these updates are standard security updates. +``` + +### 2. Update package lists and upgrade + +```bash +sudo apt update && sudo apt upgrade -y +``` + +This upgrades all installed packages. It may prompt to restart services — accept +the defaults (press Enter or choose "Ok"). + +### 3. Remove unused packages + +```bash +sudo apt autoremove -y +``` + +### 4. Check if a reboot is required + +```bash +[ -f /var/run/reboot-required ] && echo "REBOOT REQUIRED" || echo "No reboot needed" +``` + +Kernel and libc updates typically require a reboot. + +### 5. Reboot if required + +```bash +sudo reboot +``` + +Wait ~30 seconds for the server to come back up, then reconnect and verify +services: + +```bash +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@46.225.234.201 +cd /opt/torrust && sudo docker compose ps +``` + +All services should show `running` or `healthy`. If any container failed to +start, check the logs: + +```bash +sudo docker compose logs --tail=30 +``` + +## Log + +| Date | Updates Applied | Reboot Required | Notes | +| ---------- | ---------------- | --------------- | ----------------------------------------------------------------------------------------------------------------- | +| 2026-03-04 | 59 (37 security) | Yes | First post-deployment update run — all services healthy after reboot (caddy, grafana, mysql, prometheus, tracker) | diff --git a/docs/deployments/hetzner-demo-tracker/maintenance/secrets-rotation.md b/docs/deployments/hetzner-demo-tracker/maintenance/secrets-rotation.md new file mode 100644 index 000000000..5b2d0bdb7 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/maintenance/secrets-rotation.md @@ -0,0 +1,476 @@ +# Secrets Rotation + +**Date**: 2026-03-04 +**Reason**: Full deployment was performed with an AI coding agent (Claude Sonnet 4.6 +via GitHub Copilot in Visual Studio Code). All secrets that appeared in terminal output, +configuration files, or SSH sessions were potentially sent to Microsoft (GitHub +Copilot), Anthropic (Claude), and processed by cloud infrastructure operated by +those companies. All live secrets must be rotated. + +## What to Rotate vs Delete + +| Secret | Action | Reason | +| ----------------------------- | ------- | ------------------------------------------------------- | +| Tracker admin token | ✅ Done | Rotated 2026-03-04 — tracker and Prometheus scraping OK | +| MySQL `torrust` user password | ✅ Done | Rotated 2026-03-04 | +| MySQL `root` user password | ✅ Done | Rotated 2026-03-04 | +| Grafana admin password | ✅ Done | Rotated 2026-03-04 | +| SSH deployer key | ✅ Done | Rotated 2026-03-04 | +| Hetzner Cloud API token | ✅ Done | Deleted 2026-03-04 — no longer needed after deployment | +| Hetzner DNS API token | ✅ Done | Deleted 2026-03-04 — no longer needed after DNS setup | +| Local sensitive files | ✅ Done | Archived and removed 2026-03-04 | + +> **Nothing to delete**: all tokens and keys are still needed for ongoing +> administration of the running instance, **except** the Hetzner Cloud and DNS +> API tokens which are only needed during deployment and can be deleted. + +## Secret-to-Files Relationship Map + +The same secret can appear in multiple configuration files. Every location must +be updated when rotating — missing one will cause service failures. + +| Secret | Server file path | Variable / field | +| ------------------------ | ---------------------------------------------------- | ----------------------------------------------------------------------------- | +| Tracker admin token | `/opt/torrust/.env` | `TORRUST_TRACKER_CONFIG_OVERRIDE_HTTP_API__ACCESS_TOKENS__ADMIN` | +| Tracker admin token | `/opt/torrust/storage/prometheus/etc/prometheus.yml` | `params.token` in `tracker_stats` scrape job | +| Tracker admin token | `/opt/torrust/storage/prometheus/etc/prometheus.yml` | `params.token` in `tracker_metrics` scrape job | +| MySQL `torrust` password | `/opt/torrust/.env` | `MYSQL_PASSWORD` | +| MySQL `torrust` password | `/opt/torrust/storage/tracker/etc/tracker.toml` | `core.database.path` connection string (password is URL-encoded: `%2F` → `/`) | +| MySQL `torrust` password | `/opt/torrust/storage/backup/etc/backup.conf` | `DB_PASSWORD` | +| MySQL `root` password | `/opt/torrust/.env` | `MYSQL_ROOT_PASSWORD` | +| Grafana admin password | `/opt/torrust/.env` | `GF_SECURITY_ADMIN_PASSWORD` | + +> **Rule**: whenever Step 1 updates the tracker admin token in `.env`, you must +> **also** update `prometheus.yml` (step 1b below) — otherwise Prometheus can no +> longer scrape tracker metrics. + +## Suggested Password Generation Commands + +Run these locally to generate new secrets before starting rotation. Use the +output as your `` values below. + +```bash +# New tracker admin token (URL-safe base64) +openssl rand -base64 32 | tr -d '\n=' + +# New MySQL torrust password (hex — no special chars, no URL-encoding needed) +openssl rand -hex 24 + +# New MySQL root password (hex) +openssl rand -hex 24 + +# New Grafana admin password +openssl rand -base64 24 | tr -d '\n=' +``` + +Keep these values in a local password manager — do not share them with any AI +agent or paste them into a chat window. + +--- + +## Step 1: Rotate the Tracker Admin Token ✅ Done (2026-03-04) + +The admin token appears in **three places**: `.env` (used by the tracker +container at startup) and **two scrape jobs** in `prometheus.yml` (used by +Prometheus to poll `/api/v1/stats` and `/api/v1/metrics`). All three must be +updated together. + +**On the server:** + +```bash +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@46.225.234.201 +``` + +### 1a. Update `.env` + +```bash +sudo vim /opt/torrust/.env +``` + +Change: + +```text +TORRUST_TRACKER_CONFIG_OVERRIDE_HTTP_API__ACCESS_TOKENS__ADMIN='' +``` + +To: + +```text +TORRUST_TRACKER_CONFIG_OVERRIDE_HTTP_API__ACCESS_TOKENS__ADMIN='' +``` + +### 1b. Update `prometheus.yml` + +```bash +sudo vim /opt/torrust/storage/prometheus/etc/prometheus.yml +``` + +Find and replace the token in **both** scrape jobs (`tracker_stats` and +`tracker_metrics`): + +```yaml +params: + token: [""] +``` + +Change to: + +```yaml +params: + token: [""] +``` + +There are two occurrences — update both. + +### 1c. Recreate tracker and Prometheus containers + +> **Important**: `docker compose restart` is **not enough** here. The admin +> token is injected as an environment variable when the container is first +> created. `restart` only restarts the process inside the existing container — +> it does not re-read `.env`. You must recreate the containers to pick up the +> new value. + +```bash +cd /opt/torrust && sudo docker compose up -d --force-recreate tracker prometheus +``` + +Verify the new token works: + +```bash +curl -s "https://api.torrust-tracker-demo.com/api/v1/stats?token=" | head -c 200 +``` + +Verify Prometheus is scraping again (wait ~30 s then check targets): + +```bash +curl -s http://localhost:9090/api/v1/targets | python3 -m json.tool | grep -A3 'scrapePool' +``` + +--- + +## Step 2: Rotate the MySQL Passwords ✅ Done (2026-03-04) + +### 2a. Change passwords in MySQL + +```bash +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@46.225.234.201 +cd /opt/torrust +sudo docker compose exec mysql mysql -u root -p +``` + +Enter the current root password when prompted. Then inside MySQL: + +```sql +ALTER USER 'torrust'@'%' IDENTIFIED BY ''; +ALTER USER 'root'@'%' IDENTIFIED BY ''; +ALTER USER 'root'@'localhost' IDENTIFIED BY ''; +FLUSH PRIVILEGES; +EXIT; +``` + +Verify the new torrust user password works: + +```bash +sudo docker compose exec -e MYSQL_PWD="" mysql \ + mysql -u torrust torrust_tracker -e "SELECT COUNT(*) FROM torrents;" +``` + +### 2b. Update `/opt/torrust/.env` + +```bash +sudo vim /opt/torrust/.env +``` + +Change: + +```text +MYSQL_ROOT_PASSWORD='' +MYSQL_PASSWORD='' +``` + +To: + +```text +MYSQL_ROOT_PASSWORD='' +MYSQL_PASSWORD='' +``` + +> **Note**: These vars only take effect when the MySQL container is first created. +> Since MySQL passwords are now changed directly in the database (step 2a), the +> `.env` update is for documentation and future container rebuilds only. + +### 2c. Update `tracker.toml` + +```bash +sudo vim /opt/torrust/storage/tracker/etc/tracker.toml +``` + +Find the line like: + +```toml +path = "mysql://torrust:@mysql:3306/torrust_tracker" +``` + +Change to (if your new password contains no special characters, no URL-encoding +needed): + +```toml +path = "mysql://torrust:@mysql:3306/torrust_tracker" +``` + +> **Note**: If the new password contains `/`, encode it as `%2F`. If it contains +> `@`, encode it as `%40`. Using a hex password avoids this entirely. + +### 2d. Update `backup.conf` + +```bash +sudo vim /opt/torrust/storage/backup/etc/backup.conf +``` + +Change: + +```text +DB_PASSWORD= +``` + +To: + +```text +DB_PASSWORD= +``` + +### 2e. Restart tracker and backup containers + +> **Note**: `restart` is sufficient here. Unlike the admin token (an env var), +> `tracker.toml` and `backup.conf` are bind-mounted files — the process re-reads +> them on each startup, so no container recreation is needed. + +```bash +cd /opt/torrust +sudo docker compose restart tracker backup +``` + +Verify tracker is up and connected: + +```bash +sudo docker compose ps tracker +sudo docker compose logs --tail=20 tracker +``` + +--- + +## Step 3: Rotate the Grafana Admin Password ✅ Done (2026-03-04) + +### Option A: Change via the Grafana UI (simplest) + +1. Log in at `https://grafana.torrust-tracker-demo.com` with `admin` / current + password +2. Click your profile avatar (bottom-left) → **Profile** +3. Scroll to **Change Password** +4. Enter the current password and the new one, then click **Change Password** + +Then update `.env` to keep it in sync: + +```bash +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@46.225.234.201 +sudo vim /opt/torrust/.env +``` + +Change: + +```text +GF_SECURITY_ADMIN_PASSWORD='' +``` + +To: + +```text +GF_SECURITY_ADMIN_PASSWORD='' +``` + +### Option B: Change via the Grafana CLI (headless) + +```bash +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@46.225.234.201 +cd /opt/torrust +sudo docker compose exec grafana grafana-cli admin reset-admin-password '' +``` + +Then update `.env` as shown in Option A. + +Verify login at `https://grafana.torrust-tracker-demo.com` with credentials +`admin` / ``. + +--- + +## Step 4: Rotate the SSH Deployer Key ✅ Done (2026-03-04) + +The `torrust_tracker_deployer_ed25519` key was used by the AI agent to SSH into +the server and run `scp` commands. Generate a new key pair and replace it. + +### 4a. Generate the new key pair (on your local machine) + +```bash +ssh-keygen -t ed25519 -C "torrust-tracker-deployer" \ + -f ~/.ssh/torrust_tracker_deployer_ed25519_new +``` + +Leave the passphrase empty (or add one if you prefer). + +### 4b. Add the new public key to the server + +```bash +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@46.225.234.201 \ + "echo '$(cat ~/.ssh/torrust_tracker_deployer_ed25519_new.pub)' >> ~/.ssh/authorized_keys" +``` + +### 4c. Test the new key + +```bash +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519_new torrust@46.225.234.201 "echo OK" +``` + +### 4d. Remove the old public key from the server + +```bash +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519_new torrust@46.225.234.201 +``` + +Then inside the server: + +```bash +vim ~/.ssh/authorized_keys +``` + +Delete the line with `torrust-tracker-deployer` that contains the **old** key +fingerprint (the first line — there should now be two entries, old and new). + +### 4e. Replace the local key files + +```bash +mv ~/.ssh/torrust_tracker_deployer_ed25519_new ~/.ssh/torrust_tracker_deployer_ed25519 +mv ~/.ssh/torrust_tracker_deployer_ed25519_new.pub ~/.ssh/torrust_tracker_deployer_ed25519.pub +``` + +### 4f. Update the deployer env file + +Edit `envs/torrust-tracker-demo.json` and update the `ssh_private_key_path` +field (or equivalent) to point to the same path — no change needed if the path +did not change. + +### 4g. Delete the old SSH key from the Hetzner console ✅ Done (2026-03-04) + +The deployer uploaded the old public key to Hetzner during provisioning so +cloud-init could inject it into the server. Now that the key has been rotated, +the Hetzner console entry is stale and should be removed. + +1. Open [Hetzner Cloud Console](https://console.hetzner.cloud/) +2. Go to **Security → SSH Keys** +3. Find the old `torrust-tracker-deployer` key and click **Delete** + +> **Note**: Deleting the key from the Hetzner console has no effect on the +> running server — `authorized_keys` on the server is independent and was +> already updated in steps 4b and 4d. + +--- + +## Step 5: Delete the Hetzner Cloud API Token ✅ Done (2026-03-04) + +The token was only needed during provisioning (`provision` command). The server +is running and no further OpenTofu operations are planned, so it can be deleted. + +1. Open [Hetzner Cloud Console](https://console.hetzner.cloud/) +2. Go to **Security → API Tokens** +3. Find the token used by the deployer (e.g. `torrust-tracker-deployer`) +4. Click **Delete** + +If you need to run deployer commands again in the future, create a new token at +that point. + +--- + +## Step 6: Delete the Hetzner DNS API Token ✅ Done (2026-03-04) + +The token was only needed during DNS setup. All DNS records are in place and no +changes are planned, so it can be deleted. + +1. Open [Hetzner DNS Console](https://dns.hetzner.com/) +2. Go to **API Tokens** +3. Delete the token used during setup + +If you need to manage DNS records again in the future, create a new token at +that point. + +--- + +## Step 7: Archive and Remove Local Sensitive Files ✅ Done (2026-03-04) + +The following local directories and files were generated by the deployer and +contain real secrets (API tokens, passwords, SSH key paths). They are +git-ignored and must be archived in a safe place before being removed from +the local machine. + +| Path | Sensitive contents | +| -------------------------------- | ----------------------------------------------------------- | +| `build/torrust-tracker-demo/` | Generated configs with real passwords and tokens | +| `data/torrust-tracker-demo/` | Deployment state including any cached credentials | +| `envs/torrust-tracker-demo.json` | Full environment config: Hetzner tokens, DB passwords, etc. | + +### 7a. Archive to a safe location + +Move the files to an encrypted vault or password manager attachment before +deleting them. At minimum, store `envs/torrust-tracker-demo.json` — it is the +source of truth for recreating the deployment. + +Example using a local encrypted directory (adjust path to your vault): + +```bash +cp -r build/torrust-tracker-demo ~/vault/torrust-tracker-demo-build-2026-03-04 +cp -r data/torrust-tracker-demo ~/vault/torrust-tracker-demo-data-2026-03-04 +cp envs/torrust-tracker-demo.json ~/vault/torrust-tracker-demo-env-2026-03-04.json +``` + +> **Note**: Do not commit these files to git or store them in any cloud service +> accessible without encryption. + +### 7b. Remove the local copies + +Once archived safely: + +```bash +rm -rf build/torrust-tracker-demo +rm -rf data/torrust-tracker-demo +rm envs/torrust-tracker-demo.json +``` + +> **When to do this**: After all secrets are rotated on the server (steps 1–4), +> so the archived files reflect the **old** credentials only. If you archive +> before rotation, you archive the same compromised secrets — which is still +> useful as a record, but make sure the archived copy is clearly labelled as +> pre-rotation. + +--- + +## Verification Checklist + +After completing all steps, run through this checklist: + +- [ ] HTTP tracker responds: `curl -s https://http1.torrust-tracker-demo.com/announce?...` +- [ ] UDP tracker responds: BEP 15 handshake or `udp_tracker_client` +- [x] API responds with new token: `curl .../api/v1/stats?token=` +- [ ] Grafana login works with new password +- [x] SSH access works with new key +- [ ] Application backup runs successfully — SSH in and run: + `cd /opt/torrust && sudo docker compose run --rm backup` +- [ ] MySQL dumps are not empty (check last backup file size) +- [x] Local sensitive files archived to safe storage (step 7) +- [x] `build/torrust-tracker-demo/`, `data/torrust-tracker-demo/`, `envs/torrust-tracker-demo.json` removed from local machine + +--- + +## Local Files to Update + +| File | What to update | +| ----------------------------------------- | ------------------------------------------------------ | +| `envs/torrust-tracker-demo.json` | Tracker admin token, MySQL passwords (if stored there) | +| `~/.ssh/torrust_tracker_deployer_ed25519` | New private key (step 4e) | diff --git a/docs/deployments/hetzner-demo-tracker/maintenance/uptime-monitoring.md b/docs/deployments/hetzner-demo-tracker/maintenance/uptime-monitoring.md new file mode 100644 index 000000000..f6f15a8c7 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/maintenance/uptime-monitoring.md @@ -0,0 +1,45 @@ +# Uptime Monitoring + +## Context + +The Torrust demo instance previously hosted on DigitalOcean used DigitalOcean's +built-in **Monitoring** feature, which allows configuring uptime checks against +HTTP/HTTPS URLs. When a check fails, DigitalOcean sends an automatic email +alert. This made it easy to detect when services went down without any +additional infrastructure. + +**Hetzner does not offer an equivalent native monitoring feature.** The Hetzner +Cloud Console has no uptime check or alert capability. + +## Recommended External Tools + +The following free or low-cost external services can replicate this +functionality: + +| Tool | Free tier | Notes | +| -------------------------------------------------------------- | ------------------ | ---------------------------------- | +| [UptimeRobot](https://uptimerobot.com/) | 50 monitors, 5 min | HTTP/HTTPS, email + webhook alerts | +| [Freshping](https://www.freshping.io/) | 50 monitors, 1 min | HTTP, email alerts | +| [Better Uptime](https://betterstack.com/) | Limited free tier | HTTP, phone/SMS/email alerts | +| [Checkly](https://www.checklyhq.com/) | Limited free tier | HTTP + browser checks | +| [statuspage.io](https://www.atlassian.com/software/statuspage) | Paid | Public status page | + +## What to Monitor + +These are the public endpoints that should be checked: + +| Endpoint | Expected response | +| ------------------------------------------------------- | ----------------- | +| `https://api.torrust-tracker-demo.com/api/health_check` | HTTP 200 | +| `https://grafana.torrust-tracker-demo.com` | HTTP 200 | +| `https://http1.torrust-tracker-demo.com/health_check` | HTTP 200 | +| `https://http2.torrust-tracker-demo.com/health_check` | HTTP 200 | + +> **Note**: UDP tracker health cannot be checked by standard HTTP monitoring +> tools. Monitoring the HTTP API health check endpoint is sufficient as a +> proxy for overall service availability, since the tracker container runs +> all services (UDP + HTTP + API) as a single process. + +## Status + +⏳ Uptime monitoring not yet configured — tracked as a future improvement. diff --git a/docs/deployments/hetzner-demo-tracker/media/configuratrion-ai-agent-questions/question-lets-encrypt-email.png b/docs/deployments/hetzner-demo-tracker/media/configuratrion-ai-agent-questions/question-lets-encrypt-email.png new file mode 100644 index 000000000..411dc3281 Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/configuratrion-ai-agent-questions/question-lets-encrypt-email.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/configuratrion-ai-agent-questions/question-lets-encrypt.png b/docs/deployments/hetzner-demo-tracker/media/configuratrion-ai-agent-questions/question-lets-encrypt.png new file mode 100644 index 000000000..e936189a9 Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/configuratrion-ai-agent-questions/question-lets-encrypt.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/configuratrion-ai-agent-questions/question-server-type.png b/docs/deployments/hetzner-demo-tracker/media/configuratrion-ai-agent-questions/question-server-type.png new file mode 100644 index 000000000..61257b898 Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/configuratrion-ai-agent-questions/question-server-type.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/configuratrion-ai-agent-questions/question-service-subdomains.png b/docs/deployments/hetzner-demo-tracker/media/configuratrion-ai-agent-questions/question-service-subdomains.png new file mode 100644 index 000000000..c2223311c Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/configuratrion-ai-agent-questions/question-service-subdomains.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/hetzner-console-api-token-read-write-permissions.png b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-api-token-read-write-permissions.png new file mode 100644 index 000000000..1a14ab49d Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-api-token-read-write-permissions.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/hetzner-console-assign-floating-ipv4-popup.png b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-assign-floating-ipv4-popup.png new file mode 100644 index 000000000..96f5f3cee Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-assign-floating-ipv4-popup.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/hetzner-console-assign-floating-ipv6-popup.png b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-assign-floating-ipv6-popup.png new file mode 100644 index 000000000..8da29f15e Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-assign-floating-ipv6-popup.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/hetzner-console-backups-enabled-no-backups-yet.png b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-backups-enabled-no-backups-yet.png new file mode 100644 index 000000000..3b1909ff8 Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-backups-enabled-no-backups-yet.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/hetzner-console-backups-tab.png b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-backups-tab.png new file mode 100644 index 000000000..cb6e9316c Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-backups-tab.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/hetzner-console-create-floating-ip-ipv4-form.png b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-create-floating-ip-ipv4-form.png new file mode 100644 index 000000000..5edd2d0e5 Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-create-floating-ip-ipv4-form.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/hetzner-console-create-floating-ip-ipv6-form.png b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-create-floating-ip-ipv6-form.png new file mode 100644 index 000000000..7d527f256 Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-create-floating-ip-ipv6-form.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/hetzner-console-dns-zone-initial-state.png b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-dns-zone-initial-state.png new file mode 100644 index 000000000..e3359de39 Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-dns-zone-initial-state.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/hetzner-console-dns-zones-list.png b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-dns-zones-list.png new file mode 100644 index 000000000..9aaeae98b Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-dns-zones-list.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/hetzner-console-floating-ips-assigned-to-server.png b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-floating-ips-assigned-to-server.png new file mode 100644 index 000000000..d5040adcf Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-floating-ips-assigned-to-server.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/hetzner-console-floating-ips-list.png b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-floating-ips-list.png new file mode 100644 index 000000000..6ec0fb9d4 Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-floating-ips-list.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/hetzner-console-generate-api-token.png b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-generate-api-token.png new file mode 100644 index 000000000..2978a662f Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-generate-api-token.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/hetzner-console-provisioned-server-details-attempt-1.png b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-provisioned-server-details-attempt-1.png new file mode 100644 index 000000000..22c611ca6 Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-provisioned-server-details-attempt-1.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/hetzner-console-provisioned-server-details-attempt-4.png b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-provisioned-server-details-attempt-4.png new file mode 100644 index 000000000..f7a7f2d6c Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-provisioned-server-details-attempt-4.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/hetzner-console-volume-configure-popup.png b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-volume-configure-popup.png new file mode 100644 index 000000000..0bd16fbba Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-volume-configure-popup.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/hetzner-console-volumes-list.png b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-volumes-list.png new file mode 100644 index 000000000..e8d6fddac Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/hetzner-console-volumes-list.png differ diff --git a/docs/deployments/hetzner-demo-tracker/media/hetzner-dns-create-api-token-form.png b/docs/deployments/hetzner-demo-tracker/media/hetzner-dns-create-api-token-form.png new file mode 100644 index 000000000..24ce2bd55 Binary files /dev/null and b/docs/deployments/hetzner-demo-tracker/media/hetzner-dns-create-api-token-form.png differ diff --git a/docs/deployments/hetzner-demo-tracker/observations.md b/docs/deployments/hetzner-demo-tracker/observations.md new file mode 100644 index 000000000..f6f4e2305 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/observations.md @@ -0,0 +1,82 @@ +# Deployment Observations + +Cross-cutting learnings and insights gathered during this deployment that apply to the deployer +in general, not to any specific command or step. + +## Deployer State and Recovery + +### No built-in failure recovery (by design) + +The deployer has no mechanism to recover from a failed state. If a command fails (e.g. +`provision` fails halfway through), the environment is left in a failed state +(`ProvisioningFailed`, `ConfigureFailed`, etc.) and the only supported path forward is to clean +up and restart from scratch. + +This is intentional — see +[post-provision/README.md § Design Notes](post-provision/README.md#design-notes-tradeoffs-in-setup-sequencing) +for the rationale (recovery complexity vs. fast server recreation). + +### Potential manual recovery via state snapshot (untested) + +> ⚠️ **Warning**: This approach has not been tested or verified. It is a theoretical recovery +> path. Only attempt it if you understand exactly why the command failed and are confident the +> server is in a state that can be manually completed. Getting this wrong may leave the +> environment in a worse, harder-to-diagnose state. + +The deployer stores the environment state in a JSON file at: + +```text +data//environment.json +``` + +For this deployment: + +```text +data/torrust-tracker-demo/environment.json +``` + +This file tracks the current state (`Provisioned`, `Configured`, `Released`, `Running`, etc.) +and metadata like the server IP and creation timestamp. + +**Theoretical recovery procedure:** + +1. **Before running a command**, take a snapshot of the state file: + + ```bash + cp data/torrust-tracker-demo/environment.json \ + data/torrust-tracker-demo/environment.json.bak-before-configure + ``` + +2. **If the command fails**, identify exactly what the command did before failing. For example: + - `configure` installs Docker, Docker Compose, and then writes application config files. + If it failed after installing Docker but before writing config files, Docker is installed + but the config is incomplete. + - `release` pulls Docker images and stages release artifacts. If it failed midway, some + images may be present, others not. + +3. **Manually complete or undo the partial work** on the server via SSH so the server is in a + consistent state matching the target state of the command. + +4. **Restore the pre-command snapshot**: + + ```bash + cp data/torrust-tracker-demo/environment.json.bak-before-configure \ + data/torrust-tracker-demo/environment.json + ``` + +5. **Retry the command.** + +**When this is safe to attempt:** + +- The failure reason is clear and the root cause is understood. +- The partial work done by the command is reversible or completable manually. +- You can verify the server state via SSH before retrying. + +**When NOT to attempt this:** + +- The failure cause is unknown. +- Infrastructure state (e.g. Hetzner resources created by OpenTofu) is out of sync with + the local Tofu state files — in that case, restoring the environment JSON will not fix the + inconsistency and may cause further problems. +- The command involved OpenTofu (`provision`, `destroy`) — Tofu manages its own state in + `build//tofu/` and that state must also be consistent. diff --git a/docs/deployments/hetzner-demo-tracker/post-provision/README.md b/docs/deployments/hetzner-demo-tracker/post-provision/README.md new file mode 100644 index 000000000..0a1cedfcc --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/post-provision/README.md @@ -0,0 +1,84 @@ +# Post-Provision Steps + +These are manual steps performed **after** `provision` succeeds and **before** running `configure`. + +They are not automated by the deployer CLI — they require actions in the Hetzner Console and +on the server via SSH. + +## Steps + +| Step | Guide | Status | +| ------------------ | ---------------------------------------- | ------- | +| 1. DNS Setup | [dns-setup.md](dns-setup.md) | ✅ Done | +| 2. Volume Setup | [volume-setup.md](volume-setup.md) | ✅ Done | +| 3. Hetzner Backups | [hetzner-backups.md](hetzner-backups.md) | ✅ Done | + +## Why Before `configure`? + +- **DNS**: The `configure` command installs Caddy as a TLS reverse proxy. Caddy uses + Let's Encrypt to obtain TLS certificates for the domains in the environment config. This + requires that all DNS records already resolve to the server's IP **before** `configure` runs, + otherwise the ACME challenge will fail and TLS setup will break. + +- **Volume**: The `configure` command creates `/opt/torrust/` and its subdirectories on the + server. If the external volume is attached and mounted at `/opt/torrust/storage` **before** + `configure` runs, Ansible writes all persistent data directly onto the volume — so nothing + needs to be migrated afterwards. + +## Design Notes: Tradeoffs in Setup Sequencing + +### Deployer state recovery is intentionally absent + +The deployer has no failure-recovery mechanism — if a command fails (e.g. `provision`), the +environment is left in a failed state (e.g. `ProvisioningFailed`) and the only path forward is +to clean up and start from scratch. This was a deliberate design decision: + +1. **Complexity**: Implementing robust recovery logic for partial infrastructure states + (half-created servers, partial Ansible runs, etc.) is significantly complex. +2. **Speed**: Recreating a server from scratch takes less than 5 minutes. + +This tradeoff works well for the core deployment flow (`provision` → `configure` → `release` +→ `run`). However, it interacts poorly with the extra setup steps introduced by this +deployment's use of floating IPs and an external storage volume. + +### Volume attachment timing: a sequencing dilemma + +For this demo deployment we chose to add two infrastructure extras: + +- **Floating IPs** — so DNS records stay stable if the server is ever recreated. +- **External storage volume** — so tracker data survives server recreation and can be + backed up independently. + +These extras must currently be set up **before** `configure` (see "Why Before `configure`?") +because Ansible writes data directly to `/opt/torrust/storage/` during `configure`. If the +volume is not mounted at that path beforehand, data lands on the root disk and would need to +be migrated later. + +This creates a problem when the deployment fails and must be restarted from scratch: + +- **Floating IPs**: No problem — the IPs are already assigned and the DNS records already + point to them. When a new server is created, you simply reassign the floating IPs to it. + No DNS changes needed. +- **Volume**: The volume must be detached from the old server, then reattached and remounted + on the new server. All the fstab and mount point setup must be repeated. This is not + complex, but it is manual work. + +### Alternative: defer volume setup until after `run` succeeds + +An alternative approach would be to: + +1. Run the full deployment (`provision` → `configure` → `release` → `run`) without the + external volume — all data lands on the root disk at `/opt/torrust/storage/`. +2. Only after `run` succeeds and the tracker is confirmed working, attach the volume and + migrate the data directory to it. + +**Pros**: If `provision` fails and you need to start over, there is no volume to reattach +— just provision a new server and rerun the deployment commands. + +**Cons**: Requires a data migration step (copy `/opt/torrust/storage/` from the root disk to +the volume, update the mount point) after the deployment is confirmed working. Slightly +more complex as a one-time operation. + +For a long-running production server this migration cost is worth it. For a demo that may be +re-provisioned many times during development, the current approach (volume before `configure`) +is acceptable since it only requires rerunning the volume setup script. diff --git a/docs/deployments/hetzner-demo-tracker/post-provision/dns-setup.md b/docs/deployments/hetzner-demo-tracker/post-provision/dns-setup.md new file mode 100644 index 000000000..c0aa97cee --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/post-provision/dns-setup.md @@ -0,0 +1,363 @@ +# DNS Setup + +> **Status**: ✅ Done — floating IPs assigned and configured on VM; all 12 DNS records created and verified. + +Set up DNS records so that all domain names in the environment config resolve to +the floating IP before running `configure`. + +## Why Floating IPs? + +The deployment uses **Hetzner floating IPs** (static IPs that can be reassigned across servers) +rather than the server's direct IP (`46.225.234.201`). This means: + +- DNS records always point to the same IP, even if the underlying server is ever recreated. +- To rebuild the server, you reassign the floating IP — no DNS changes needed. + +## Floating IPs + +| Type | IP | Notes | +| ---- | ----------------------- | ------------------------------------------------------------------------------------ | +| IPv4 | `116.202.176.169` | Assign as A records | +| IPv6 | `2a01:4f8:1c0c:9aae::1` | First usable address from `/64` block `2a01:4f8:1c0c:9aae::/64`; use as AAAA records | + +## Step 1: Assign Floating IPs to the Server + +In the [Hetzner Console](https://console.hetzner.cloud/): + +1. Open the project `torrust-tracker-demo.com`. +2. Go to **Networking → Floating IPs**. +3. For the IPv4 floating IP (`116.202.176.169`): + - Click **⋯ → Assign**. + - Select server `torrust-tracker-vm-torrust-tracker-demo`. + - Confirm. +4. For the IPv6 floating IP (`2a01:4f8:1c0c:9aae::/64`): + - Same procedure — assign to the same server. + +### IPv4 Assigned (2026-03-04) + +The IPv4 floating IP was assigned successfully. Hetzner showed a confirmation popup: + +![Hetzner console popup after assigning the IPv4 floating IP](../media/hetzner-console-assign-floating-ipv4-popup.png) + +The popup text: + +> **Configure Floating IP** +> +> The Floating IP has been successfully assigned. You now need to configure it on +> your server in order for it to work. +> +> **Command for temporary configuration** +> +> `sudo ip addr add 116.202.176.169 dev eth0` +> +> A temporary configuration will only work until the next reboot. To permanently +> configure the IP have a look at our Docs. + +### IPv6 Assigned (2026-03-04) + +The IPv6 floating IP was assigned successfully. Hetzner showed a confirmation popup: + +![Hetzner console popup after assigning the IPv6 floating IP](../media/hetzner-console-assign-floating-ipv6-popup.png) + +The popup text: + +> **Configure Floating IP** +> +> The Floating IP has been successfully assigned. You now need to configure it on +> your server in order for it to work. +> +> **Command for temporary configuration** +> +> `sudo ip addr add 2a01:4f8:1c0c:9aae::1 dev eth0` +> +> A temporary configuration will only work until the next reboot. To permanently +> configure the IP have a look at our Docs. + +### Both Floating IPs Assigned + +Both IPs now appear in the Hetzner console Floating IPs list assigned to the server: + +![Hetzner console showing both floating IPs assigned to the server](../media/hetzner-console-floating-ips-assigned-to-server.png) + +### Step 1.5: Configure the Floating IPs Inside the VM (2026-03-04) + +Hetzner's assignment only updates their routing — the VM's network interface still needs to +know about the new IPs. The Hetzner console popup shows a **temporary** command that works +until the next reboot. We need the **permanent** configuration instead. + +Reference: [Hetzner — Persistent Floating IP Configuration](https://docs.hetzner.com/cloud/floating-ips/persistent-configuration/) + +**Temporary (shown by Hetzner popup — lost on reboot):** + +```bash +sudo ip addr add 116.202.176.169 dev eth0 +sudo ip addr add 2a01:4f8:1c0c:9aae::1 dev eth0 +``` + +**Permanent (survives reboot) — what we did:** + +First, we checked the current state of eth0 to confirm the floating IPs were not yet +configured on the VM: + +```bash +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 -o StrictHostKeyChecking=accept-new torrust@46.225.234.201 'ip addr show eth0' +``` + +Output: + +```text +2: eth0: mtu 1500 qdisc fq_codel state UP group default qlen 1000 + link/ether 92:00:07:4f:b3:4f brd ff:ff:ff:ff:ff:ff + inet 46.225.234.201/32 metric 100 scope global dynamic eth0 + valid_lft 71163sec preferred_lft 71163sec + inet6 2a01:4f8:1c19:620b::1/64 scope global + valid_lft forever preferred_lft forever + inet6 fe80::9000:7ff:fe4f:b34f/64 scope link + valid_lft forever preferred_lft forever +``` + +Only the server's own IPs are present — no floating IPs yet. + +> **Note**: We got a `WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED` error because this +> is a new server with the same IP as a previously provisioned server (from an earlier attempt). +> Fixed by removing the stale known_hosts entry: +> +> ```bash +> ssh-keygen -f '/home/josecelano/.ssh/known_hosts' -R '46.225.234.201' +> ``` + +Wrote the netplan config file on the server: + +```bash +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@46.225.234.201 \ + 'printf "network:\n version: 2\n renderer: networkd\n ethernets:\n eth0:\n addresses:\n - 116.202.176.169/32\n - 2a01:4f8:1c0c:9aae::1/64\n" | sudo tee /etc/netplan/60-floating-ip.yaml' +``` + +Fixed file permissions (netplan requires `600`) and applied: + +```bash +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@46.225.234.201 \ + 'sudo chmod 600 /etc/netplan/60-floating-ip.yaml && sudo netplan apply' +``` + +Verified both floating IPs are now on eth0: + +```bash +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@46.225.234.201 'ip addr show eth0' +``` + +Output: + +```text +2: eth0: mtu 1500 qdisc fq_codel state UP group default qlen 1000 + link/ether 92:00:07:4f:b3:4f brd ff:ff:ff:ff:ff:ff + inet 116.202.176.169/32 scope global eth0 + valid_lft forever preferred_lft forever + inet 46.225.234.201/32 metric 100 scope global dynamic eth0 + valid_lft 86399sec preferred_lft 86399sec + inet6 2a01:4f8:1c0c:9aae::1/64 scope global + valid_lft forever preferred_lft forever + inet6 2a01:4f8:1c19:620b::1/64 scope global + valid_lft forever preferred_lft forever + inet6 fe80::9000:7ff:fe4f:b34f/64 scope link + valid_lft forever preferred_lft forever +``` + +Both `116.202.176.169/32` and `2a01:4f8:1c0c:9aae::1/64` are present — `valid_lft forever` +confirms they are permanently configured. + +> **Note**: The `configure` command does not configure floating IPs — this must be done +> manually before running `configure`. + +## Step 2: Create DNS Records (2026-03-04) + +The DNS zone `torrust-tracker-demo.com` was created in the **Hetzner Cloud Console** +(new system, [console.hetzner.cloud](https://console.hetzner.cloud/)). Hetzner is +[migrating DNS management](https://docs.hetzner.com/networking/dns/migration-to-hetzner-console/process/) +from the old [dns.hetzner.com](https://dns.hetzner.com) to the Cloud Console. Zones created +in the Cloud Console are only accessible via the **Hetzner Cloud API** — the old DNS API at +`dns.hetzner.com/api/v1` cannot see them. + +> **Note**: If you create a DNS API token at `dns.hetzner.com`, it will return empty results +> (`"zones": []`) for zones created in the Cloud Console. Use the **Cloud API token** instead +> (from **Security → API Tokens** in the Cloud Console project). + +### Records to Create + +| Subdomain | Type | Value | +| --------- | ---- | ----------------------- | +| `http1` | A | `116.202.176.169` | +| `http1` | AAAA | `2a01:4f8:1c0c:9aae::1` | +| `http2` | A | `116.202.176.169` | +| `http2` | AAAA | `2a01:4f8:1c0c:9aae::1` | +| `api` | A | `116.202.176.169` | +| `api` | AAAA | `2a01:4f8:1c0c:9aae::1` | +| `grafana` | A | `116.202.176.169` | +| `grafana` | AAAA | `2a01:4f8:1c0c:9aae::1` | +| `udp1` | A | `116.202.176.169` | +| `udp1` | AAAA | `2a01:4f8:1c0c:9aae::1` | +| `udp2` | A | `116.202.176.169` | +| `udp2` | AAAA | `2a01:4f8:1c0c:9aae::1` | + +### API Approach + +First, find the zone ID (or use the zone name directly): + +```bash +curl -s -H "Authorization: Bearer $HCLOUD_TOKEN" \ + "https://api.hetzner.cloud/v1/zones" +``` + +Output showed zone `torrust-tracker-demo.com` with ID `944360`. + +Create each record with `POST /v1/zones/{zone_name}/rrsets`: + +```bash +HCLOUD_TOKEN="" +ZONE="torrust-tracker-demo.com" +IPV4="116.202.176.169" +IPV6="2a01:4f8:1c0c:9aae::1" + +for sub in http1 http2 api grafana udp1 udp2; do + # A record + curl -s -X POST \ + -H "Authorization: Bearer $HCLOUD_TOKEN" \ + -H "Content-Type: application/json" \ + "https://api.hetzner.cloud/v1/zones/$ZONE/rrsets" \ + -d "{\"name\": \"$sub\", \"type\": \"A\", \"records\": [{\"value\": \"$IPV4\"}], \"ttl\": 300}" + # AAAA record + curl -s -X POST \ + -H "Authorization: Bearer $HCLOUD_TOKEN" \ + -H "Content-Type: application/json" \ + "https://api.hetzner.cloud/v1/zones/$ZONE/rrsets" \ + -d "{\"name\": \"$sub\", \"type\": \"AAAA\", \"records\": [{\"value\": \"$IPV6\"}], \"ttl\": 300}" +done +``` + +Each successful response contains an `rrset` object, for example: + +```json +{ + "action": { + "id": 615271254051174, + "status": "running", + "command": "create_rrset", + ... + }, + "rrset": { + "id": "http1/A", + "name": "http1", + "type": "A", + "ttl": 300, + "records": [{"value": "116.202.176.169", "comment": ""}], + "zone": 944360 + } +} +``` + +Verify all records were created: + +```bash +curl -s -H "Authorization: Bearer $HCLOUD_TOKEN" \ + "https://api.hetzner.cloud/v1/zones/$ZONE/rrsets" +``` + +All 12 RRSets (6 × A + 6 × AAAA) plus the default NS and SOA were present. + +## Step 3: Verify DNS Propagation (2026-03-04) + +Verify against the authoritative nameservers first (no propagation delay): + +```bash +for sub in http1 http2 api grafana udp1 udp2; do + echo "=== $sub ===" + dig +short A "$sub.torrust-tracker-demo.com" @hydrogen.ns.hetzner.com + dig +short AAAA "$sub.torrust-tracker-demo.com" @hydrogen.ns.hetzner.com +done +``` + +Then verify global resolution (system resolver): + +```bash +for sub in http1 http2 api grafana udp1 udp2; do + A=$(dig +short A "$sub.torrust-tracker-demo.com") + AAAA=$(dig +short AAAA "$sub.torrust-tracker-demo.com") + echo "$sub: A=$A AAAA=$AAAA" +done +``` + +Actual output (2026-03-04): + +```text +http1: A=116.202.176.169 AAAA=2a01:4f8:1c0c:9aae::1 +http2: A=116.202.176.169 AAAA=2a01:4f8:1c0c:9aae::1 +api: A=116.202.176.169 AAAA=2a01:4f8:1c0c:9aae::1 +grafana: A=116.202.176.169 AAAA=2a01:4f8:1c0c:9aae::1 +udp1: A=116.202.176.169 AAAA=2a01:4f8:1c0c:9aae::1 +udp2: A=116.202.176.169 AAAA=2a01:4f8:1c0c:9aae::1 +``` + +✅ All 12 records resolve correctly globally. + +> DNS propagation with Hetzner's nameservers (`helium.ns.hetzner.de`, `hydrogen.ns.hetzner.com`, +> `oxygen.ns.hetzner.com`) is typically fast (under 1 minute). If you get `NXDOMAIN` or empty +> results, wait a minute and retry. + +## Outcome + +✅ All subdomains resolve to `116.202.176.169` (A) and `2a01:4f8:1c0c:9aae::1` (AAAA). DNS +setup is complete. The next step is [volume-setup.md](volume-setup.md). + +## Problems + +### DNS API token from dns.hetzner.com does not see Cloud Console zones + +**Symptom**: Creating a token at [dns.hetzner.com](https://dns.hetzner.com) and querying +`https://dns.hetzner.com/api/v1/zones` returns `{"zones": [], "error": {"message": "zone not found", "code": 404}}`. + +**Cause**: Hetzner is migrating DNS management from the old `dns.hetzner.com` console to the +new Cloud Console ([console.hetzner.cloud](https://console.hetzner.cloud)). Zones **created in +the Cloud Console** live in the Cloud API only and are invisible to the old DNS API. See the +[migration docs](https://docs.hetzner.com/networking/dns/migration-to-hetzner-console/process/). + +Additionally, the Cloud API does **not** have a `/v1/dns/zones` path — +`GET https://api.hetzner.cloud/v1/dns/zones` returns `{"error": {"code": "not_found", ...}}`. +The correct path is `/v1/zones`. + +**Fix**: Use the Hetzner Cloud API token (from **Security → API Tokens** in the project) and +the `/v1/zones` endpoint: + +```bash +curl -H "Authorization: Bearer $HCLOUD_TOKEN" https://api.hetzner.cloud/v1/zones +``` + +### SSH host key mismatch when connecting to the new server + +**Symptom**: `WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED` when SSHing to `46.225.234.201`. + +**Cause**: A previous server was provisioned at the same IP during an earlier attempt (see +[provision/problems.md](../commands/provision/problems.md)). The old host key is still in +`~/.ssh/known_hosts`. + +**Fix**: + +```bash +ssh-keygen -f '~/.ssh/known_hosts' -R '46.225.234.201' +``` + +Then reconnect — SSH will accept and store the new host key. + +### Netplan file permissions warning + +**Symptom**: `WARNING: Permissions for /etc/netplan/60-floating-ip.yaml are too open` when +running `sudo netplan apply`. + +**Cause**: Writing with `sudo tee` creates the file world-readable. Netplan requires `600`. + +**Fix**: `sudo chmod 600 /etc/netplan/60-floating-ip.yaml` before or after `netplan apply`. + +## Improvements + +- The netplan file should be written with correct permissions from the start. Use + `sudo install -m 600 /dev/stdin /etc/netplan/60-floating-ip.yaml` instead of `tee` to + avoid the permissions warning. diff --git a/docs/deployments/hetzner-demo-tracker/post-provision/hetzner-backups.md b/docs/deployments/hetzner-demo-tracker/post-provision/hetzner-backups.md new file mode 100644 index 000000000..852eedb8a --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/post-provision/hetzner-backups.md @@ -0,0 +1,89 @@ +# Hetzner Backups Setup + +> **Status**: ✅ Done — Hetzner automated backups enabled (2026-03-04). + +Enable Hetzner's automated server backup feature so that a full server image is +taken once daily. This complements the application-level backup (MySQL dump + +config files) that runs inside the container. + +> **This step can be done at any time** after provisioning — it does not depend +> on `configure`, `release`, or `run`. + +## Why Hetzner Backups in Addition to the Application Backup? + +The application-level backup (`backup` container + cron job) captures only the +application data stored on the Hetzner volume: + +- MySQL database dumps (`mysql_*.sql.gz`) +- Config file archives (`config_*.tar.gz`) + +It does **not** capture the OS, Docker installation, Caddy TLS certificates, +container images, or any data outside `/opt/torrust/storage/`. + +Hetzner Backups capture the **entire server root disk** as a snapshot image. +This protects against catastrophic failures such as: + +- OS corruption +- Accidental deletion of system files or Docker config +- Need to roll back to a known-good server state + +| Layer | What it covers | Storage location | +| --------------- | --------------------------------------- | ------------------------- | +| App backup | MySQL dump + config files | Hetzner volume (internal) | +| Hetzner Backups | Full root disk (OS, Docker, everything) | Hetzner infrastructure | + +> **Note**: Hetzner Backups do **not** include attached volumes. The 50 GB +> storage volume (`torrust-tracker-demo-storage`) is not captured by this +> feature. Application-level backups remain essential for data recovery. + +## Hetzner Backup Options + +Hetzner offers two image-based backup mechanisms for servers: + +| Feature | Backups | Snapshots | +| ------------- | ------------------------------------- | ------------------------------------ | +| Trigger | Automatic (daily, Hetzner picks time) | Manual or via API | +| Retention | Last 7 kept automatically | Kept until deleted manually | +| Cost | 20% of server price (~€0.76/mo) | €0.0119/GB/month of compressed image | +| Configuration | Enable/disable toggle only | Fully manual | +| Use case | Ongoing safety net | Point-in-time capture before changes | + +For this deployment we enable **Backups** (automated daily) as the ongoing +safety net. Snapshots can be taken manually before risky operations like +secrets rotation. + +## Step 1: Enable Backups via the Hetzner Console (2026-03-04) + +1. Open [Hetzner Cloud Console](https://console.hetzner.cloud/) +2. Navigate to **Projects → torrust-tracker-demo → Servers → + torrust-tracker-demo** +3. Click the **Backups** tab +4. Click **Enable backups** + + The console presents a confirmation showing the cost: **+20% of server price**. + + ![Hetzner Console — Backups tab before enabling](../media/hetzner-console-backups-tab.png) + +5. Confirm by clicking **Enable backups** in the dialog + +Once enabled, the Backups tab immediately shows **Backups enabled** with an +empty list — no backups exist yet. The first backup will be taken automatically +during the next maintenance window (usually within 24 hours). + +![Hetzner Console — Backups enabled, no backups yet](../media/hetzner-console-backups-enabled-no-backups-yet.png) + +## What Happens After Enabling + +- Hetzner takes one backup per day, automatically +- The last **7 backups** are retained; the oldest is deleted when a new one is + created +- Each backup appears as a server image in **Images → Backups** in the console +- To restore: navigate to the server → **Backups** tab → select a backup → + **Rebuild server from image** (this replaces the current root disk) + +## Cost + +| Resource | Price | Notes | +| --------------- | ------------------- | ---------------------------- | +| Hetzner Backups | 20% of server price | CX22 = ~€3.79/mo → ~€0.76/mo | +| Snapshots | €0.0119/GB/month | Only if taken manually | diff --git a/docs/deployments/hetzner-demo-tracker/post-provision/volume-setup.md b/docs/deployments/hetzner-demo-tracker/post-provision/volume-setup.md new file mode 100644 index 000000000..5606dc34a --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/post-provision/volume-setup.md @@ -0,0 +1,234 @@ +# Volume Setup + +> **Status**: ✅ Done — volume created, attached, formatted, mounted and verified. + +Create a Hetzner volume and mount it at `/opt/torrust/storage` on the server before running +`configure`. + +## Why a Separate Volume? + +The server's root disk contains the OS and application binaries. Persistent tracker data +(database, logs, Grafana state, Prometheus data) lives under `/opt/torrust/storage/`. + +Putting that data on a separate Hetzner volume means: + +- **Easy migration**: detach the volume and reattach to a new server if the VM is recreated. +- **Independent lifecycle**: resize the volume without touching the server. +- **Application-level backups**: back up only the data directory, not the entire server disk + (see the application's backup commands). + +> **Note**: Hetzner does **not** support volume snapshots. Their snapshot feature only captures +> the server's root disk, not attached volumes. Data backup must be done at the application +> level (e.g. the deployer's `backup` command) or via filesystem-level tools (rsync, tar). + +## Volume Specification + +| Property | Value | +| ----------- | ----------------------------------------------------- | +| Name | `torrust-tracker-demo-storage` | +| Size | 50 GB | +| Location | `nbg1` (same as the server — required for attachment) | +| Format | `ext4` | +| Mount point | `/opt/torrust/storage` | + +## Step 1: Create the Volume via Cloud API (2026-03-04) + +The volume was created and attached using the **Hetzner Cloud API** — no UI needed. + +### 1a: Create the volume + +```bash +curl -s -X POST \ + -H "Authorization: Bearer $HCLOUD_TOKEN" \ + -H "Content-Type: application/json" \ + "https://api.hetzner.cloud/v1/volumes" \ + -d '{ + "name": "torrust-tracker-demo-storage", + "size": 50, + "location": "nbg1", + "format": "ext4", + "labels": {"project": "torrust-tracker-demo"} + }' +``` + +Key fields in the response: + +```json +{ + "volume": { + "id": 104927743, + "name": "torrust-tracker-demo-storage", + "size": 50, + "format": "ext4", + "status": "creating", + "linux_device": "/dev/disk/by-id/scsi-0HC_Volume_104927743", + "location": { "name": "nbg1" }, + "server": null + } +} +``` + +> **Note**: Passing `"format": "ext4"` in the create request tells Hetzner to format the +> volume automatically. This means **`mkfs.ext4` does not need to be run manually** — the +> volume arrives with a ready-to-use ext4 filesystem including a UUID. + +### 1b: Find the server ID + +```bash +curl -s -H "Authorization: Bearer $HCLOUD_TOKEN" \ + "https://api.hetzner.cloud/v1/servers?name=torrust-tracker-vm-torrust-tracker-demo" +``` + +Returned server `id=122663759`. + +### 1c: Attach the volume to the server + +```bash +curl -s -X POST \ + -H "Authorization: Bearer $HCLOUD_TOKEN" \ + -H "Content-Type: application/json" \ + "https://api.hetzner.cloud/v1/volumes/104927743/actions/attach" \ + -d '{"server": 122663759, "automount": false}' +``` + +Confirmed attached: + +```bash +curl -s -H "Authorization: Bearer $HCLOUD_TOKEN" \ + "https://api.hetzner.cloud/v1/volumes/104927743" | \ + python3 -c "import sys,json; v=json.load(sys.stdin)['volume']; \ + print(f\"status={v['status']} server={v['server']} device={v['linux_device']}\")" +# status=available server=122663759 device=/dev/disk/by-id/scsi-0HC_Volume_104927743 +``` + +The volume list is visible in the Hetzner Console under **Storage → Volumes**: + +![Hetzner console volumes list showing torrust-tracker-demo-storage](../media/hetzner-console-volumes-list.png) + +The console also shows a **Configure volume** popup with the recommended commands: + +![Hetzner console volume configure popup](../media/hetzner-console-volume-configure-popup.png) + +The popup suggests mounting at `/mnt/torrust-tracker-demo-storage`. We use `/opt/torrust/storage` +instead, to match the application's expected directory layout. + +## Step 2: Verify Device on the Server (2026-03-04) + +```bash +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@46.225.234.201 \ + 'lsblk && echo "---blkid---" && sudo blkid /dev/disk/by-id/scsi-0HC_Volume_104927743' +``` + +Output: + +```text +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS +sda 8:0 0 152.6G 0 disk +|-sda1 8:1 0 152.3G 0 part / +|-sda14 8:14 0 1M 0 part +`-sda15 8:15 0 256M 0 part /boot/efi +sdb 8:16 0 50G 0 disk +sr0 11:0 1 1024M 0 rom +---blkid--- +/dev/disk/by-id/scsi-0HC_Volume_104927743: UUID="6fb9df14-c744-4e50-a48d-9ca4522a02de" BLOCK_SIZE="4096" TYPE="ext4" +``` + +The volume appears as `/dev/sdb`, accessible via the stable symlink +`/dev/disk/by-id/scsi-0HC_Volume_104927743`. It is already formatted as `ext4` — Hetzner +handled that when we passed `"format": "ext4"` in the API create call. + +## Step 3: Format the Volume + +**Skipped** — the volume was formatted automatically by Hetzner when we passed `"format": "ext4"` +in the create request. UUID `6fb9df14-c744-4e50-a48d-9ca4522a02de` was confirmed by `blkid` above. + +> If you create a volume **without** specifying `format` in the API (or via the UI with +> "leave unformatted"), run: +> +> ```bash +> sudo mkfs.ext4 -F /dev/disk/by-id/scsi-0HC_Volume_ +> ``` +> +> This matches the command shown in the Hetzner Console "Configure volume" popup. + +## Steps 4–8: Mount, fstab, Ownership, Verify (2026-03-04) + +All remaining steps were run in a single SSH session: + +```bash +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@46.225.234.201 'bash -s' << 'ENDSSH' +set -e + +DEVICE="/dev/disk/by-id/scsi-0HC_Volume_104927743" +MOUNT_POINT="/opt/torrust/storage" +UUID="6fb9df14-c744-4e50-a48d-9ca4522a02de" + +# Step 4: Create mount point +sudo mkdir -p "$MOUNT_POINT" + +# Step 5: Mount with discard,defaults (matches Hetzner's recommendation) +sudo mount -o discard,defaults "$DEVICE" "$MOUNT_POINT" +df -h "$MOUNT_POINT" + +# Step 6: Add to fstab (UUID + discard,nofail,defaults) +echo "UUID=$UUID $MOUNT_POINT ext4 discard,nofail,defaults 0 2" | sudo tee -a /etc/fstab +grep "$UUID" /etc/fstab + +# Step 7: Set ownership +sudo chown -R torrust:torrust "$MOUNT_POINT" +ls -la /opt/torrust/ | grep storage + +# Step 8: Verify +mountpoint -q "$MOUNT_POINT" && echo "Mounted OK" +touch "$MOUNT_POINT/.volume-test" && echo "Write OK" && rm "$MOUNT_POINT/.volume-test" +ENDSSH +``` + +Output: + +```text +Filesystem Size Used Avail Use% Mounted on +/dev/sdb 49G 24K 47G 1% /opt/torrust/storage +UUID=6fb9df14-c744-4e50-a48d-9ca4522a02de /opt/torrust/storage ext4 discard,nofail,defaults 0 2 +drwxr-xr-x 3 torrust torrust 4096 Mar 4 11:38 storage +Mounted OK +Write OK +``` + +Then confirmed fstab survives reboot by unmounting and remounting via `mount -a`: + +```bash +ssh -i ~/.ssh/torrust_tracker_deployer_ed25519 torrust@46.225.234.201 \ + 'sudo umount /opt/torrust/storage && sudo mount -a && df -h /opt/torrust/storage && echo "fstab remount OK"' +``` + +Output: + +```text +Filesystem Size Used Avail Use% Mounted on +/dev/sdb 49G 24K 47G 1% /opt/torrust/storage +fstab remount OK +``` + +## Outcome + +✅ Volume `torrust-tracker-demo-storage` (50 GB, ext4) is mounted at `/opt/torrust/storage`, +owned by `torrust:torrust`, and will remount automatically on reboot. The next step is +[running the `configure` command](../commands/configure/). + +## Problems + +### Hetzner volumes cannot be snapshotted + +The Hetzner Console has no snapshot option for volumes — only for server root disks. There is +no API endpoint for volume snapshots either. + +Data backup must be handled at the application level. The deployer provides a `backup` command +for this purpose. Alternatively, use filesystem tools (`rsync`, `tar`) to copy the volume +data to an off-server location. + +## Improvements + +- The `discard` mount option enables TRIM for SSD-backed volumes (Hetzner volumes are + SSD-backed), which helps maintain performance over time. It is already included in both + the mount command and the fstab entry above. diff --git a/docs/deployments/hetzner-demo-tracker/prerequisites.md b/docs/deployments/hetzner-demo-tracker/prerequisites.md new file mode 100644 index 000000000..b8971d698 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/prerequisites.md @@ -0,0 +1,200 @@ +# Prerequisites + +Checklist of everything needed before deploying the demo tracker to Hetzner Cloud. + +## System + +Commands in this guide were run on: + +| | | +| ---------------- | ----------------------- | +| **OS** | Ubuntu 25.10 (Questing) | +| **Kernel** | Linux 6.17.0-14-generic | +| **Architecture** | x86_64 | + +## Accounts + +- [x] **Hetzner Cloud account** — [Sign up](https://www.hetzner.com/cloud) if you don't have one +- [x] **Hetzner Cloud project** — Created project `torrust-tracker-demo.com` in [Hetzner Console](https://console.hetzner.cloud/) to group all resources for this deployment +- [x] **Hetzner API token** — Created in Hetzner Console → project `torrust-tracker-demo.com` → Security → API Tokens (Read & Write). Description: `Torrust Tracker Deployer - torrust-tracker-demo.com` +- [x] **Domain registrar access** — Domain `torrust-tracker-demo.com` registered at [cdmon.com](https://cdmon.com/). DNS servers changed to Hetzner: + - `helium.ns.hetzner.de` + - `hydrogen.ns.hetzner.com` + - `oxygen.ns.hetzner.com` +- [x] **Hetzner DNS Zone** — DNS zone `torrust-tracker-demo.com` created in Hetzner Console + +## SSH Keys + +An SSH key pair is required for VM access. The deployer uses this to connect to the provisioned server. + +```bash +# Generate a dedicated key pair (if you don't have one) +ssh-keygen -t ed25519 -C "torrust-tracker-deployer" -f ~/.ssh/torrust_tracker_deployer_ed25519 +``` + +- [x] SSH private key available (`~/.ssh/torrust_tracker_deployer_ed25519`) +- [x] SSH public key available (`~/.ssh/torrust_tracker_deployer_ed25519.pub`) +- [x] Key permissions are correct (`chmod 600` on private key) +- [x] Passphrase set on private key + +## Tools + +The deployer supports two modes: + +| Method | Best for | Requires | +| ---------- | ------------------------------------ | ----------------------- | +| **Docker** | End-users, reproducible environments | Docker only | +| **Native** | Developers working from source | Rust, OpenTofu, Ansible | + +**Recommendation for end-users: use Docker** — one tool, no dependency management. + +We are running from source (native), so we verified both. + +### Docker + +Pull the latest image first to ensure you have the most recent version: + +```bash +docker pull torrust/tracker-deployer:latest +``` + +Output confirmed a fresh download: + +```text +Status: Downloaded newer image for torrust/tracker-deployer:latest +Digest: sha256:01e3735b52bba1e733d422fe7cac918808d8e72da34c45b06c1a86ba8f33e119 +``` + +Then verify the image works: + +```bash +docker run --rm torrust/tracker-deployer:latest --help +``` + +The image starts, prints tool versions, and shows the CLI help. Tool versions inside the container: + +- **OpenTofu**: `v1.11.2` +- **Ansible**: `core 2.19.5` +- **SSH**: `OpenSSH_9.2p1` + +> **Note**: The Docker image only supports cloud providers (Hetzner). The LXD provider requires native installation. + +- [x] **Docker** — `28.3.3` installed, latest image pulled and verified + +### Native (used for this deployment) + +All dependencies verified with: + +```bash +cargo run -p torrust-dependency-installer --bin dependency-installer check +``` + +- [x] **Rust toolchain** — `rustc 1.96.0-nightly (ec818fda3 2026-03-02)` +- [x] **OpenTofu** — `v1.10.5` +- [x] **Ansible** — `core 2.19.0` +- [x] **cargo-machete** — `0.9.1` + +## Working Directories + +All three directories already existed in the repository. `envs/` permissions were tightened to `700` since it contains the environment config with the Hetzner API token. + +```bash +chmod 700 envs +``` + +- [x] `data/` directory exists (`775`) +- [x] `build/` directory exists (`775`) +- [x] `envs/` directory exists with restricted permissions (`700`) + +## Hetzner Infrastructure Resources + +Beyond the server itself, we are setting up two additional Hetzner resources to make the deployment more robust and operationally flexible. + +### Floating IPs (IPv4 and IPv6) + +**Why floating IPs?** +Floating IPs are static IPs that are independent of the server. This means: + +- If we resize or replace the server, we reassign the floating IP — no DNS change needed. +- The DNS records point to the floating IP permanently. + +Floating IPs are created in Hetzner Console → project → Networking → Floating IPs, then assigned to the server after provisioning. + +> **Important**: Floating IPs must be created in the **same location** (datacenter) as the server. + +**Location chosen**: Nuremberg (`nbg1`) — same location we'll use for the server. + +**Pricing**: + +| Type | Monthly cost (excl. VAT) | +| ---- | ------------------------ | +| IPv4 | €3.00 | +| IPv6 | €1.00 | + +**Names and addresses**: + +| Name | Type | Address | +| --------------------------- | ---- | ------------------------- | +| `torrust-tracker-demo-ipv4` | IPv4 | `116.202.176.169` | +| `torrust-tracker-demo-ipv6` | IPv6 | `2a01:4f8:1c0c:9aae::/64` | + +![Hetzner Console — Create floating IPv4 form](media/hetzner-console-create-floating-ip-ipv4-form.png) + +![Hetzner Console — Create floating IPv6 form](media/hetzner-console-create-floating-ip-ipv6-form.png) + +![Hetzner Console — Floating IPs list](media/hetzner-console-floating-ips-list.png) + +- [x] IPv4 floating IP created in Hetzner project (`torrust-tracker-demo-ipv4`, `nbg1`, `116.202.176.169`) +- [x] IPv6 floating IP created in Hetzner project (`torrust-tracker-demo-ipv6`, `nbg1`, `2a01:4f8:1c0c:9aae::/64`) +- [ ] Both IPs assigned to the server (after provisioning) + +### Volume for Storage (⚠️ deferred — do after `release`) + +**Why a separate volume?** +The tracker's `storage/` directory holds all persistent data (SQLite database, logs, Grafana data, Prometheus data). Placing it on a dedicated Hetzner Volume means: + +- Resize the server without touching the data. +- Create incremental volume backups without needing a full VM snapshot. +- Detach and reattach to a new server for disaster recovery. + +> ⚠️ **This step is deferred** — the volume must be attached to a running server and mounted at `/opt/torrust/storage/` before the `release` command is run. See the deployment journal for the exact step. + +- [ ] Volume created in Hetzner project (after server is provisioned) +- [ ] Volume attached to server and mounted at `/opt/torrust/storage/` (before `release`) + +## DNS (after provisioning) + +DNS records will be configured after we know the floating IP addresses. We need: + +- [ ] A records for `torrust-tracker-demo.com` subdomains pointing to the **floating IPv4** address +- [ ] AAAA records for subdomains pointing to the **floating IPv6** address + +### Initial DNS State (before deployment) + +The zone was freshly created in Hetzner DNS with only the default SOA and NS records — no A records yet. + +![Hetzner Console — DNS zone initial state](media/hetzner-console-dns-zone-initial-state.png) + +Records at zone creation (queried from `helium.ns.hetzner.de`): + +```text +;; NS records +torrust-tracker-demo.com. 3600 IN NS helium.ns.hetzner.de. +torrust-tracker-demo.com. 3600 IN NS hydrogen.ns.hetzner.com. +torrust-tracker-demo.com. 3600 IN NS oxygen.ns.hetzner.com. + +;; SOA record +torrust-tracker-demo.com. 3600 IN SOA hydrogen.ns.hetzner.com. dns.hetzner.com. ( + 2026030300 ; serial + 86400 ; refresh (1 day) + 10800 ; retry (3 hours) + 3600000 ; expire (5 weeks 6 days 16 hours) + 3600 ; minimum (1 hour) + ) +``` + +## Related Documentation + +- [Hetzner Cloud Provider guide](../../user-guide/providers/hetzner.md) +- [Quick Start: Docker Deployment](../../user-guide/quick-start/docker.md) +- [Quick Start: Native Installation](../../user-guide/quick-start/native.md) diff --git a/docs/deployments/hetzner-demo-tracker/tracker-registry.md b/docs/deployments/hetzner-demo-tracker/tracker-registry.md new file mode 100644 index 000000000..c2195456d --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/tracker-registry.md @@ -0,0 +1,41 @@ +# Tracker Registry + +Public tracker registries list open BitTorrent trackers so clients can +discover and use them. Submitting the demo tracker improves community +visibility and provides a passive uptime signal. + +## newTrackon + +[newTrackon](https://newtrackon.com/) continuously monitors the health of +submitted trackers and publishes them in public lists. + +The previous Torrust demo tracker (`udp://tracker.torrust-demo.com:6969/announce`) +was already listed there. The new Hetzner demo tracker should be submitted as +well. + +### Which tracker to submit + +Only **UDP Tracker 1** is submitted to public registries: + +```text +udp://udp1.torrust-tracker-demo.com:6969/announce +``` + +**UDP Tracker 2** (`udp://udp2.torrust-tracker-demo.com:6868/announce`) is +intentionally kept off all public tracker lists. Once a tracker appears in +public lists it receives a continuous stream of announces from BitTorrent +clients worldwide. That background noise makes it very hard to read logs +and debug issues when testing something in production. Keeping `udp2` quiet +reserves it as a low-traffic endpoint for manual testing and investigation. + +### How to submit + +1. Go to +2. Paste `udp://udp1.torrust-tracker-demo.com:6969/announce` into the submission box +3. Click **Submit** +4. Wait a few minutes while newTrackon gathers data +5. Verify it appears in the [Submitted](https://newtrackon.com/submitted) section + +### Status + +✅ Submitted (2026-03-04) — pending appearance in the public list. diff --git a/docs/deployments/hetzner-demo-tracker/verify/README.md b/docs/deployments/hetzner-demo-tracker/verify/README.md new file mode 100644 index 000000000..e1fd5a2f6 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/verify/README.md @@ -0,0 +1,47 @@ +# Service Verification + +Manual verification procedures for all services in the Hetzner demo tracker +deployment. Run these after the `test` command to confirm that each service +is fully functional end-to-end. + +## Services + +| Service | URL | File | +| --------------- | ------------------------------------------------- | ---------------------------------------- | +| HTTP Tracker | `https://http1.torrust-tracker-demo.com/announce` | [http-tracker.md](http-tracker.md) | +| UDP Tracker | `udp://udp1.torrust-tracker-demo.com:6969` | [udp-tracker.md](udp-tracker.md) | +| Tracker API | `https://api.torrust-tracker-demo.com/api/v1` | [api.md](api.md) | +| Grafana | `https://grafana.torrust-tracker-demo.com` | [grafana.md](grafana.md) | +| Health Check | `http://127.0.0.1:1313/health_check` (internal) | [health-check.md](health-check.md) | +| Docker Services | All containers | [docker-services.md](docker-services.md) | +| MySQL Database | `torrust_tracker` DB (internal) | [mysql.md](mysql.md) | +| Storage Volume | `/opt/torrust/storage` on `sdb` (internal) | [storage.md](storage.md) | +| Backup | `storage/backup/` on volume (internal) | [backup.md](backup.md) | + +## Status + +| Service | Status | +| --------------- | ----------- | +| HTTP Tracker | ✅ Verified | +| UDP Tracker | ✅ Verified | +| Tracker API | ✅ Verified | +| Grafana | ✅ Verified | +| Health Check | ✅ Verified | +| Docker Services | ✅ Verified | +| MySQL Database | ✅ Verified | +| Storage Volume | ✅ Verified | +| Backup | ✅ Verified | + +## Prerequisites + +- The environment must be in `Running` state. +- The `test` command must have passed (even with DNS warnings). +- For UDP tracker tests: `openssl` and `xxd` must be available locally, or use + a BitTorrent client. +- For API tests: `curl` must be available locally. + +## Network Notes + +All domain names resolve to the floating IP `116.202.176.169`. The instance IP +`46.225.234.201` is only used for direct SSH access. The health check endpoint +is bound to `localhost` on the server and is only accessible via SSH. diff --git a/docs/deployments/hetzner-demo-tracker/verify/api.md b/docs/deployments/hetzner-demo-tracker/verify/api.md new file mode 100644 index 000000000..3de04edca --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/verify/api.md @@ -0,0 +1,130 @@ +# Tracker API Verification + +**Status**: ✅ Verified (2026-03-04) + +## Endpoint + +`https://api.torrust-tracker-demo.com/api/v1` + +## Authentication + +All API endpoints require the admin token as a query parameter or header. + +Admin token: see `envs/torrust-tracker-demo.json` → `tracker.http_api.admin_token` + +```bash +# Set token for reuse in the commands below +TOKEN="" +``` + +## 1. TLS Certificate Check + +```bash +curl -v --head "https://api.torrust-tracker-demo.com/api/v1/stats?token=$TOKEN" 2>&1 | grep -E "subject|issuer|SSL|HTTP" +``` + +Expected: valid Let's Encrypt certificate, HTTP 200. + +## 2. Tracker Statistics + +Fetch global tracker statistics (total torrents, peers, etc.). + +```bash +TOKEN="" +curl -s "https://api.torrust-tracker-demo.com/api/v1/stats?token=$TOKEN" | python3 -m json.tool +``` + +Expected response structure (counters may be non-zero if there has been any +network activity since deployment): + +```json +{ + "torrents": 0, + "seeders": 0, + "completed": 0, + "leechers": 0, + "tcp4_connections_handled": 0, + "tcp4_announces_handled": 0, + "tcp4_scrapes_handled": 0, + "tcp6_connections_handled": 0, + "tcp6_announces_handled": 0, + "tcp6_scrapes_handled": 0, + "udp_requests_aborted": 0, + "udp_requests_banned": 0, + "udp_banned_ips_total": 0, + "udp_avg_connect_processing_time_ns": 0, + "udp_avg_announce_processing_time_ns": 0, + "udp_avg_scrape_processing_time_ns": 0, + "udp4_requests": 0, + "udp4_connections_handled": 0, + "udp4_announces_handled": 0, + "udp4_scrapes_handled": 0, + "udp4_responses": 0, + "udp4_errors_handled": 0, + "udp6_requests": 0, + "udp6_connections_handled": 0, + "udp6_announces_handled": 0, + "udp6_scrapes_handled": 0, + "udp6_responses": 0, + "udp6_errors_handled": 0 +} +``` + +## 3. List Torrents + +```bash +TOKEN="" +curl -s "https://api.torrust-tracker-demo.com/api/v1/torrents?token=$TOKEN&limit=10&offset=0" | python3 -m json.tool +``` + +Expected: empty list `[]` on a fresh deployment. + +## 4. Add and Remove a Test Torrent (whitelist mode only) + +> **Note**: This deployment runs in **public mode** (`private = false`), so +> whitelisting is not enforced. Adding a torrent via the API is still useful +> to confirm write access works. + +```bash +TOKEN="" +INFO_HASH="0000000000000000000000000000000000000001" + +# Add +curl -s -X POST "https://api.torrust-tracker-demo.com/api/v1/torrent/$INFO_HASH?token=$TOKEN" + +# Verify it appears +curl -s "https://api.torrust-tracker-demo.com/api/v1/torrents?token=$TOKEN" | python3 -m json.tool + +# Remove +curl -s -X DELETE "https://api.torrust-tracker-demo.com/api/v1/torrent/$INFO_HASH?token=$TOKEN" +``` + +## 5. Invalid Token Rejected + +Confirm authentication is enforced. + +```bash +curl -s -o /dev/null -w "%{http_code}" "https://api.torrust-tracker-demo.com/api/v1/stats?token=invalid" +``` + +Expected: `401` (a `500` is currently returned — see [Bug 4](#bug-4-invalid-token-returns-500-instead-of-401)) + +## Results + +| Check | Result | Notes | +| ------------------ | ------ | --------------------------------------------------------- | +| TLS certificate | ✅ | Let's Encrypt, valid until Jun 2, 2026 | +| Tracker statistics | ✅ | Non-zero UDP6 counters from network activity after deploy | +| List torrents | ✅ | Empty list on fresh deployment | +| Add/remove torrent | ✅ | Both return empty body on success | +| Invalid token | ⚠️ | Returns HTTP `500` instead of `401` — see Bug 4 below | + +## Bug 4: Invalid Token Returns 500 Instead of 401 + +When sending an invalid token, the API returns: + +- **HTTP status**: `500` +- **Body**: `Unhandled rejection: Err { reason: "token not valid" }` + +A `401 Unauthorized` would be the correct HTTP status for an authentication +failure. This is a bug in the tracker API error handling. diff --git a/docs/deployments/hetzner-demo-tracker/verify/backup.md b/docs/deployments/hetzner-demo-tracker/verify/backup.md new file mode 100644 index 000000000..4096cf606 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/verify/backup.md @@ -0,0 +1,189 @@ +# Backup Verification + +**Status**: ✅ Verified (2026-03-04) + +## Architecture + +Backups run daily at 03:00 UTC via a host cron job that orchestrates a +graceful maintenance window: + +```text +/etc/cron.d/tracker-backup (03:00 daily) + └─▶ /usr/local/bin/maintenance-backup.sh + ├─ stop tracker container + ├─ docker compose --profile backup run --rm backup + │ ├─ mysqldump torrust_tracker → mysql_.sql.gz + │ └─ tar config files → config_.tar.gz + └─ start tracker container +``` + +The backup container uses the `backup` Docker Compose profile, so it is +**not** started on `docker compose up` — it only runs when explicitly invoked. + +### Retention + +Old backups older than **7 days** are deleted automatically at the end of each +backup cycle. + +## What Gets Backed Up + +### MySQL dump + +The full `torrust_tracker` database is exported with `mysqldump` and compressed +with gzip. Output: `storage/backup/mysql/mysql_.sql.gz`. + +### Config files + +The following files are archived into a tarball: +`storage/backup/config/config_.tar.gz`. + +| File | Description | +| --------------------------------------------------- | ------------------------------- | +| `storage/tracker/etc/tracker.toml` | Tracker configuration | +| `storage/prometheus/etc/prometheus.yml` | Prometheus configuration | +| `storage/grafana/provisioning/datasources/*.yml` | Grafana datasource provisioning | +| `storage/grafana/provisioning/dashboards/*.yml` | Grafana dashboard provisioning | +| `storage/grafana/provisioning/dashboards/torrust/*` | Dashboard JSON definitions | +| `storage/caddy/etc/Caddyfile` | Caddy reverse-proxy config | + +## How to Trigger a Manual Backup + +```bash +ssh -i ~/.ssh/ torrust@46.225.234.201 " + cd /opt/torrust + sudo docker compose --profile backup run --rm backup +" +``` + +## How to List Existing Backups + +```bash +ssh -i ~/.ssh/ torrust@46.225.234.201 " + sudo find /opt/torrust/storage/backup -type f | sort +" +``` + +## How to Inspect a Backup + +```bash +# List SQL dump tables +ssh -i ~/.ssh/ torrust@46.225.234.201 " + sudo zcat /opt/torrust/storage/backup/mysql/.sql.gz | grep '^CREATE TABLE' +" + +# List config archive contents +ssh -i ~/.ssh/ torrust@46.225.234.201 " + sudo tar -tzf /opt/torrust/storage/backup/config/.tar.gz +" +``` + +## Issues Found and Fixed During Verification + +### Oversight: backup.conf not updated after manual credentials fix + +During the initial deployment the `run` command failed because the MySQL root +user password was not URL-encoded in `tracker.toml` (see Bug 1 and Bug 3 in +`commands/improvements.md`). That was fixed manually — the templates were +updated and the `run` command was retried. + +However, `backup.conf` was also generated at the same time with the original +credentials, and was **not updated** when the tracker credentials were fixed. +As a result the backup container could not authenticate to MySQL: + +| Setting | Value in backup.conf | Required value | +| --------- | -------------------- | ----------------- | +| `DB_USER` | `root` | `torrust` | +| `DB_NAME` | `torrust` | `torrust_tracker` | + +The fix was to align `backup.conf` with the credentials that were already +working for the tracker: + +```bash +sudo sed -i \ + -e 's/^DB_USER=root/DB_USER=torrust/' \ + -e 's/^DB_NAME=torrust$/DB_NAME=torrust_tracker/' \ + /opt/torrust/storage/backup/etc/backup.conf +``` + +This is a process gap: whenever credentials are changed or fixed manually +after a deployment, all configuration files that reference those credentials +must be updated together — including `backup.conf`. + +### Known warning: PROCESS privilege + +The backup container uses `mariadb-dump` (from the MariaDB Docker image) +against a MySQL 8.4 server. MariaDB's dump client emits the following +non-critical warning: + +```text +mysqldump: Error: 'Access denied; you need (at least one of) the PROCESS privilege(s) +for this operation' when trying to dump tablespaces +``` + +This only affects tablespace metadata, not the actual table data. All table +structures and rows are dumped correctly. The `torrust` user would need +`GRANT PROCESS ON *.* TO 'torrust'@'%'` or the `--no-tablespaces` flag added +to the `mysqldump` command to suppress this warning. + +## Results (2026-03-04) + +### Test run output + +```text +[2026-03-04 16:07:58] Torrust Backup Container starting +[2026-03-04 16:07:58] Loading configuration from: /etc/backup/backup.conf +[2026-03-04 16:07:58] Configuration: +[2026-03-04 16:07:58] Retention: 7 days +[2026-03-04 16:07:58] Database: mysql +[2026-03-04 16:07:58] Config paths file: /etc/backup/backup-paths.txt +[2026-03-04 16:07:58] Starting backup cycle +[2026-03-04 16:07:58] Starting MySQL backup: torrust_tracker@mysql:3306 +[2026-03-04 16:07:58] MySQL backup completed: /backups/mysql/mysql_20260304_160758.sql.gz +[2026-03-04 16:07:59] Size: 4.0K +[2026-03-04 16:07:59] Starting config files backup +[2026-03-04 16:07:59] Config backup completed: /backups/config/config_20260304_160759.tar.gz +[2026-03-04 16:07:59] Files backed up: 4 +[2026-03-04 16:07:59] Size: 8.0K +[2026-03-04 16:07:59] Cleaning up backups older than 7 days +[2026-03-04 16:07:59] No old backups to delete +[2026-03-04 16:07:59] Backup cycle completed successfully +``` + +### SQL dump content + +```text +-- Host: mysql Database: torrust_tracker +CREATE TABLE `keys` (...) +CREATE TABLE `torrent_aggregate_metrics` (...) +CREATE TABLE `torrents` (...) +CREATE TABLE `whitelist` (...) +``` + +### Config archive content + +```text +data/storage/tracker/etc/tracker.toml +data/storage/prometheus/etc/prometheus.yml +data/storage/grafana/provisioning/datasources/prometheus.yml +data/storage/grafana/provisioning/dashboards/torrust.yml +data/storage/grafana/provisioning/dashboards/torrust/stats.json +data/storage/grafana/provisioning/dashboards/torrust/metrics.json +data/storage/caddy/etc/Caddyfile +``` + +### Verification summary + +| Check | Result | +| -------------------------------- | ------------------------------------------------------- | +| Cron job installed | ✅ `/etc/cron.d/tracker-backup` | +| Schedule | ✅ Daily at 03:00 UTC | +| Maintenance script present | ✅ `/usr/local/bin/maintenance-backup.sh` | +| Manual test run | ✅ Exit code 0 | +| MySQL dump created | ✅ `mysql_20260304_160758.sql.gz` (4 KB) | +| All 4 tables present in dump | ✅ keys, torrent_aggregate_metrics, torrents, whitelist | +| Config archive created | ✅ `config_20260304_160759.tar.gz` (8 KB, 4 paths) | +| Retention policy | ✅ 7 days | +| Backups stored on volume (`sdb`) | ✅ `/opt/torrust/storage/backup/` | +| Wrong DB_USER in backup.conf | ⚠️ Oversight — fixed (root → torrust) | +| Wrong DB_NAME in backup.conf | ⚠️ Oversight — fixed (torrust → torrust_tracker) | +| PROCESS privilege warning | ⚠️ Non-critical (tablespaces only) | diff --git a/docs/deployments/hetzner-demo-tracker/verify/docker-services.md b/docs/deployments/hetzner-demo-tracker/verify/docker-services.md new file mode 100644 index 000000000..fac49a542 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/verify/docker-services.md @@ -0,0 +1,106 @@ +# Docker Services Verification + +**Status**: ✅ Verified (2026-03-04) + +## How to Check + +```bash +ssh -i ~/.ssh/ torrust@46.225.234.201 " + cd /opt/torrust + sudo docker compose ps + sudo docker compose logs --tail=30 tracker + sudo docker compose logs --tail=20 mysql + sudo docker compose logs --tail=20 caddy + sudo docker compose logs --tail=20 prometheus + sudo docker compose logs --tail=20 grafana +" +``` + +## 1. Container Health Status + +All containers must report `(healthy)`. Docker evaluates each service's +health check before setting this status. + +Expected: + +```text +NAME IMAGE SERVICE STATUS +caddy caddy:2.10 caddy Up X hours (healthy) +grafana grafana/grafana:12.3.1 grafana Up X hours (healthy) +mysql mysql:8.4 mysql Up X hours (healthy) +prometheus prom/prometheus:v3.5.0 prometheus Up X hours (healthy) +tracker torrust/tracker:develop tracker Up X hours (healthy) +``` + +### Actual Output (2026-03-04) + +| Container | Image | Status | +| ---------- | ------------------------- | ---------- | +| caddy | `caddy:2.10` | ✅ healthy | +| grafana | `grafana/grafana:12.3.1` | ✅ healthy | +| mysql | `mysql:8.4` | ✅ healthy | +| prometheus | `prom/prometheus:v3.5.0` | ✅ healthy | +| tracker | `torrust/tracker:develop` | ✅ healthy | + +## 2. Service Logs + +### Tracker + +✅ **Clean** — INFO level only. Logs show periodic health check polling and +Prometheus metrics scrapes, all returning `200 OK`. No warnings or errors. + +### MySQL + +⚠️ **Expected warnings at startup** — three cosmetic warnings that appear on +every MySQL 8.4 container start: + +- `Unable to load '/usr/share/zoneinfo/zone.tab' as time zone` — MySQL 8.4 + cosmetic warning; timezone data not installed in the container image. Does + not affect operation. +- `CA certificate ca.pem is self signed` — default self-signed cert for + encrypted connections; not used in this deployment. +- `Insecure configuration for --pid-file` — the `/var/run/mysqld` path is + accessible to all OS users inside the container; harmless in Docker context. + +No errors. Database initialization log shows `torrust_tracker` database and +`torrust` user were created successfully. + +### Caddy + +⚠️ **Two categories of expected noise** — no application errors: + +1. **Transient DNS errors at startup** (`ERROR: dial tcp: lookup grafana on +127.0.0.11:53: server misbehaving`): Docker's internal DNS resolver + was not ready when Caddy first tried to resolve `grafana`. These appear + only during the first seconds after `docker compose up` and self-resolve. + Not present in steady-state operation. + +2. **WARN: aborting with incomplete response**: External bots and scanners + (probing for `/wp-login.php`, `/wp-admin/`, `/administrator/`) dropped + TCP connections before Caddy finished sending the response. No legitimate + traffic is affected. + +### Prometheus + +✅ **Clean** — INFO level only. Startup, WAL replay, and scrape configuration +loaded successfully. No warnings or errors. + +### Grafana + +✅ **Clean** — INFO level only. The only notable entries: + +- Periodic cleanup jobs and plugin update checks — expected background tasks. +- `404` responses for `/api/dashboards/uid/*/public-dashboards` — these are + expected because public dashboards are not configured. Grafana logs these + at INFO (not ERROR), and they do not affect dashboard functionality. + +## Results + +| Check | Result | Notes | +| ---------------------- | ------ | --------------------------------------------------- | +| All containers healthy | ✅ | All 5 report `(healthy)` status | +| Tracker logs clean | ✅ | INFO only | +| MySQL logs clean | ⚠️ | 3 cosmetic startup warnings — expected, harmless | +| Caddy logs clean | ⚠️ | Transient DNS + bot scan WARNs — expected, harmless | +| Prometheus logs clean | ✅ | INFO only | +| Grafana logs clean | ✅ | INFO only; 404 for unconfigured public dashboards | diff --git a/docs/deployments/hetzner-demo-tracker/verify/grafana.md b/docs/deployments/hetzner-demo-tracker/verify/grafana.md new file mode 100644 index 000000000..2b7772da5 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/verify/grafana.md @@ -0,0 +1,100 @@ +# Grafana Verification + +**Status**: ✅ Verified (2026-03-04) + +## Endpoint + +`https://grafana.torrust-tracker-demo.com` + +## Credentials + +- **Username**: `admin` +- **Password**: see `envs/torrust-tracker-demo.json` → `grafana.admin_password` + +## 1. TLS Certificate and Login Page + +Open in a browser or check with curl: + +```bash +curl -sv --head "https://grafana.torrust-tracker-demo.com" 2>&1 | grep -E "HTTP|subject|issuer" +``` + +Expected: HTTP 302 redirect to `/login` (or direct 200), valid Let's Encrypt +certificate for `grafana.torrust-tracker-demo.com`. + +## 2. API Login Check + +Verify the credentials work via the Grafana HTTP API. + +> **Note**: use `-u` flag — URL-embedded credentials (`admin:pass@host`) will +> fail if the password contains `/`, as does this deployment's password. + +```bash +GRAFANA_PASS="" +curl -s -u "admin:${GRAFANA_PASS}" "https://grafana.torrust-tracker-demo.com/api/user" | python3 -m json.tool +``` + +Expected response: + +```json +{ + "id": 1, + "email": "admin@localhost", + "name": "admin", + "login": "admin", + "role": "Admin", + ... +} +``` + +## 3. Data Source — Prometheus Connected + +Confirm Grafana is receiving metrics from Prometheus. + +```bash +GRAFANA_PASS="" +curl -s -u "admin:${GRAFANA_PASS}" "https://grafana.torrust-tracker-demo.com/api/datasources" | python3 -m json.tool +``` + +Expected: a data source named `Prometheus` with `"type": "prometheus"` and +`"url": "http://prometheus:9090"`. + +## 4. Dashboards Provisioned + +Confirm the pre-provisioned dashboards are present. + +```bash +GRAFANA_PASS="" +curl -s -u "admin:${GRAFANA_PASS}" "https://grafana.torrust-tracker-demo.com/api/dashboards/home" | python3 -m json.tool +``` + +For a full list of dashboards: + +```bash +curl -s -u "admin:${GRAFANA_PASS}" "https://grafana.torrust-tracker-demo.com/api/search?type=dash-db" | python3 -m json.tool +``` + +Expected: one or more dashboards from the `torrust` folder as configured in +`templates/grafana/provisioning/dashboards/`. + +## 5. Browser Verification + +Navigate to `https://grafana.torrust-tracker-demo.com` in a browser, log in with +admin credentials, and confirm: + +- Dashboards load without errors +- Prometheus data source shows a green "Data source connected" status + (Settings → Data sources → Prometheus → Test) +- The tracker metrics dashboard shows panels (values may be zero on a fresh + deployment with no traffic) + +## Results + +| Check | Result | Notes | +| ------------------------------ | ------ | ------------------------------------------------------------ | +| TLS certificate valid | ✅ | Let's Encrypt, valid until Jun 2, 2026 | +| Login page reachable | ✅ | HTTP 302 redirect to `/login` | +| API login with credentials | ✅ | Returns admin user details | +| Prometheus data source present | ✅ | `http://prometheus:9090`, default, `readOnly: true` | +| Dashboards provisioned | ✅ | 2 dashboards in "Torrust Tracker" folder (metrics and stats) | +| Browser login and dashboard | ⏳ | Manual browser check not yet performed | diff --git a/docs/deployments/hetzner-demo-tracker/verify/health-check.md b/docs/deployments/hetzner-demo-tracker/verify/health-check.md new file mode 100644 index 000000000..b75dae298 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/verify/health-check.md @@ -0,0 +1,110 @@ +# Health Check Verification + +**Status**: ✅ Verified (2026-03-04) + +## Endpoint + +The tracker exposes a health check API on `127.0.0.1:1313` **inside the +tracker container** (loopback only). It is not accessible from the host or +from outside Docker. It is also not routed through Caddy. + +The `show` command reports it as: + +```text +"health_check_url": "http://46.225.234.201:1313/health_check" +"health_check_is_localhost_only": true +``` + +## How to Access + +Because the endpoint is bound to the container's loopback interface, it must +be reached via `docker compose exec` on the server: + +```bash +ssh -i ~/.ssh/ torrust@46.225.234.201 \ + "cd /opt/torrust && sudo docker compose exec tracker \ + sh -c 'wget -qO- http://127.0.0.1:1313/health_check'" +``` + +## Expected Response + +A healthy deployment returns a JSON object with `"status": "Ok"` and one +entry per configured service: + +```json +{ + "status": "Ok", + "message": "", + "details": [ + { + "service_binding": "http://[::]:7071/", + "binding": "[::]:7071", + "service_type": "http_tracker", + "info": "checking http tracker health check at: http://[::]:7071/health_check", + "result": { "Ok": "200 OK" } + }, + { + "service_binding": "http://[::]:1212/", + "binding": "[::]:1212", + "service_type": "tracker_rest_api", + "info": "checking api health check at: http://[::]:1212/api/health_check", + "result": { "Ok": "200 OK" } + }, + { + "service_binding": "udp://[::]:6969", + "binding": "[::]:6969", + "service_type": "udp_tracker", + "info": "checking the udp tracker health check at: [::]:6969", + "result": { "Ok": "Connected" } + }, + { + "service_binding": "udp://[::]:6868", + "binding": "[::]:6868", + "service_type": "udp_tracker", + "info": "checking the udp tracker health check at: [::]:6868", + "result": { "Ok": "Connected" } + }, + { + "service_binding": "http://[::]:7070/", + "binding": "[::]:7070", + "service_type": "http_tracker", + "info": "checking http tracker health check at: http://[::]:7070/health_check", + "result": { "Ok": "200 OK" } + } + ] +} +``` + +## Actual Output (2026-03-04) + +The above response matches exactly what was returned on verification. All +five services reported healthy: + +| Service | Port | Result | +| ---------------------------- | ---- | ------------ | +| HTTP Tracker 2 (`http2`) | 7071 | ✅ 200 OK | +| Tracker REST API | 1212 | ✅ 200 OK | +| UDP Tracker 1 (`udp1`, 6969) | 6969 | ✅ Connected | +| UDP Tracker 2 (`udp2`, 6868) | 6868 | ✅ Connected | +| HTTP Tracker 1 (`http1`) | 7070 | ✅ 200 OK | + +## Notes + +- `wget` is available in the tracker container image (`torrust/tracker:develop`) + but `curl` is not. +- The health check is a loopback-only endpoint by design — it is not intended + to be exposed externally. +- Port `1313` appears in `docker compose ps` output but is not published to + the host (no `0.0.0.0:1313->1313/tcp` mapping). + +## Public REST API Health Check + +The Tracker REST API (port 1212, routed through Caddy) also exposes its own +health check endpoint publicly: + +```bash +curl -s "https://api.torrust-tracker-demo.com/api/health_check" +``` + +This is a lighter check — it only confirms the REST API itself is responding, +not the individual tracker services. It does not require an auth token. diff --git a/docs/deployments/hetzner-demo-tracker/verify/http-tracker.md b/docs/deployments/hetzner-demo-tracker/verify/http-tracker.md new file mode 100644 index 000000000..f69f7371e --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/verify/http-tracker.md @@ -0,0 +1,120 @@ +# HTTP Tracker Verification + +**Status**: ✅ Verified (2026-03-04) + +## Endpoints + +| Domain | URL | +| -------------- | ------------------------------------------------- | +| HTTP Tracker 1 | `https://http1.torrust-tracker-demo.com/announce` | +| HTTP Tracker 2 | `https://http2.torrust-tracker-demo.com/announce` | + +## 1. Basic Connectivity (scrape request) + +The simplest check — a scrape request with no info hashes returns a valid +bencoded response. + +```bash +curl -v "https://http1.torrust-tracker-demo.com/announce?info_hash=%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00&peer_id=%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00&port=6881&uploaded=0&downloaded=0&left=0&event=started&compact=1" +``` + +Expected: HTTP 200 with a bencoded response body starting with `d` (a bencoded +dictionary). The response will contain `interval` and `peers` fields. + +## 2. TLS Certificate Check + +Confirm the TLS certificate is valid and issued for the correct domain. + +```bash +curl -v --head https://http1.torrust-tracker-demo.com/announce 2>&1 | grep -E "subject|issuer|expire|SSL" +``` + +Expected: Certificate issued by Let's Encrypt for `http1.torrust-tracker-demo.com`, +not expired. + +## 3. Second Endpoint + +Repeat the connectivity check for the second HTTP tracker: + +```bash +curl -sv "https://http2.torrust-tracker-demo.com/announce?info_hash=%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00&peer_id=%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00&port=6881&uploaded=0&downloaded=0&left=0&event=started&compact=1" 2>&1 | head -20 +``` + +## 4. Health Check + +The tracker exposes a health check API bound to the container's loopback +interface (`127.0.0.1:1313`). It is not accessible from the host directly. +See [health-check.md](health-check.md) for the verification procedure and +actual output. + +## 5. Using the Torrust Tracker Client + +The [Torrust Tracker](https://github.com/torrust/torrust-tracker) project ships +a reference `http_tracker_client` binary located in +[`console/tracker-client`](https://github.com/torrust/torrust-tracker/tree/develop/console/tracker-client). +It sends a full BEP 3 HTTP announce request and displays the JSON-decoded +response. + +### HTTP Tracker 1 + +```bash +# From the torrust-tracker console/tracker-client directory +cargo run --bin http_tracker_client announce \ + https://http1.torrust-tracker-demo.com \ + 9c38422213e30bff212b30c360d26f9a02136422 +``` + +Output: + +```json +{ + "complete": 3, + "incomplete": 0, + "interval": 300, + "min interval": 300, + "peers": [ + { "ip": "::ffff:2.137.92.24", "port": 34094 }, + { "ip": "::ffff:2.137.92.24", "port": 48887 } + ] +} +``` + +### HTTP Tracker 2 + +```bash +cargo run --bin http_tracker_client announce \ + https://http2.torrust-tracker-demo.com \ + 9c38422213e30bff212b30c360d26f9a02136422 +``` + +Output: + +```json +{ + "complete": 3, + "incomplete": 0, + "interval": 300, + "min interval": 300, + "peers": [ + { "ip": "::ffff:2.137.92.24", "port": 34094 }, + { "ip": "::ffff:2.137.92.24", "port": 48887 } + ] +} +``` + +Both trackers returned the same response — they share the same backend database. +The IP `::ffff:2.137.92.24` is the IPv4-mapped IPv6 form of the local machine's +public IP (`2.137.92.24`), confirming the peer was registered correctly from the +client's perspective. The two ports (`34094` and `48887`) correspond to peers +registered from previous HTTP and UDP announces during this verification session. + +## Results + +| Check | Result | Notes | +| ----------------------------------- | ------ | ---------------------------------------------- | +| HTTP Tracker 1 connectivity | ✅ | HTTP 200 with bencoded response | +| HTTP Tracker 1 TLS cert | ✅ | Let's Encrypt, valid until Jun 2, 2026 | +| HTTP Tracker 2 connectivity | ✅ | HTTP 200 with bencoded response | +| Health check | ✅ | See [health-check.md](health-check.md) | +| Torrust client announce (tracker 1) | ✅ | `interval=300`, `complete=3`, 2 peers returned | +| Torrust client announce (tracker 2) | ✅ | `interval=300`, `complete=3`, 2 peers returned | diff --git a/docs/deployments/hetzner-demo-tracker/verify/mysql.md b/docs/deployments/hetzner-demo-tracker/verify/mysql.md new file mode 100644 index 000000000..5bbc62619 --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/verify/mysql.md @@ -0,0 +1,112 @@ +# MySQL Database Verification + +**Status**: ✅ Verified (2026-03-04) + +## Overview + +Verifies that the MySQL database is reachable from the tracker container and +that reads and writes work correctly. The tracker uses MySQL as its persistent +store for whitelisted torrents, authentication keys, torrent stats, and +aggregate metrics. + +## Database Schema + +Connect to MySQL using the `MYSQL_PWD` environment variable to avoid shell +escaping issues with the password (which contains a `/`): + +```bash +ssh -i ~/.ssh/ torrust@46.225.234.201 " + cd /opt/torrust + MYSQL_PWD=\$(sudo grep MYSQL_PASSWORD .env | cut -d= -f2 | tr -d \"'\") + sudo docker compose exec -e MYSQL_PWD=\"\$MYSQL_PWD\" mysql \ + mysql -u torrust torrust_tracker -e 'SHOW TABLES;' +" +``` + +### Tables + +```text ++-----------------------------+ +| Tables_in_torrust_tracker | ++-----------------------------+ +| keys | +| torrent_aggregate_metrics | +| torrents | +| whitelist | ++-----------------------------+ +``` + +| Table | Description | +| --------------------------- | ------------------------------------------------------------------ | +| `keys` | Authentication keys (id, key varchar(32), valid_until int) | +| `torrent_aggregate_metrics` | Named aggregate counters (id, metric_name varchar(50), value int) | +| `torrents` | Persisted torrent stats (id, info_hash varchar(40), completed int) | +| `whitelist` | Whitelisted info hashes (id, info_hash varchar(40)) | + +## Read/Write Verification + +Verifying the tracker→MySQL connection requires an actual write. The simplest +approach is to add a torrent to the whitelist via the API and confirm it +appears in the `whitelist` table immediately. + +> **Note**: The tracker is running in open mode (`listed = false`), so the +> whitelist table is not used for access control here. The entry can be safely +> deleted after the test. + +### 1. Add a torrent to the whitelist + +```bash +curl -s -X POST \ + "https://api.torrust-tracker-demo.com/api/v1/whitelist/0000000000000000000000000000000000000001?token=" +``` + +Expected response: + +```json +{ "status": "ok" } +``` + +### 2. Verify the row in the database + +```bash +ssh -i ~/.ssh/ torrust@46.225.234.201 " + cd /opt/torrust + MYSQL_PWD='' + sudo docker compose exec -e MYSQL_PWD=\"\$MYSQL_PWD\" mysql \ + mysql -u torrust torrust_tracker \ + -e 'SELECT * FROM whitelist;' +" +``` + +Expected output: + +```text +id info_hash +1 0000000000000000000000000000000000000001 +``` + +### 3. Clean up + +```bash +curl -s -X DELETE \ + "https://api.torrust-tracker-demo.com/api/v1/whitelist/0000000000000000000000000000000000000001?token=" +``` + +Expected response: + +```json +{ "status": "ok" } +``` + +## Results (2026-03-04) + +| Check | Result | +| ----------------------------- | ------- | +| Tables present (4 tables) | ✅ Pass | +| Whitelist write via API | ✅ Pass | +| Row visible in DB immediately | ✅ Pass | +| Whitelist delete via API | ✅ Pass | +| Row removed from DB | ✅ Pass | + +The tracker→MySQL connection is working correctly. Writes are synchronous — +rows appear in the database immediately after the API call completes. diff --git a/docs/deployments/hetzner-demo-tracker/verify/storage.md b/docs/deployments/hetzner-demo-tracker/verify/storage.md new file mode 100644 index 000000000..6ab80f92b --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/verify/storage.md @@ -0,0 +1,162 @@ +# Storage Volume Verification + +**Status**: ✅ Verified (2026-03-04) + +## Overview + +Verifies that all persistent data is written to the attached Hetzner volume +(`/dev/sdb`, 50 GB) and **not** to the server's internal disk (`/dev/sda`). + +## How to Verify + +### 1. Confirm the volume is mounted + +```bash +ssh -i ~/.ssh/ torrust@46.225.234.201 "lsblk && df -h | grep -v tmpfs | grep -v udev" +``` + +### 2. Confirm the storage directory is on the volume + +```bash +ssh -i ~/.ssh/ torrust@46.225.234.201 "df -h /opt/torrust/storage" +``` + +### 3. Inspect the storage tree + +```bash +ssh -i ~/.ssh/ torrust@46.225.234.201 " + find /opt/torrust/storage -maxdepth 3 | sort + sudo du -sh /opt/torrust/storage/mysql/ +" +``` + +## Results (2026-03-04) + +### Block devices + +```text +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS +sda 8:0 0 152.6G 0 disk +|-sda1 8:1 0 152.3G 0 part / +|-sda14 8:14 0 1M 0 part +`-sda15 8:15 0 256M 0 part /boot/efi +sdb 8:16 0 50G 0 disk /opt/torrust/storage +``` + +`sdb` (the attached Hetzner volume) is mounted at `/opt/torrust/storage`. +`sda` (the internal disk) only holds the OS (`/`) and the EFI partition. + +### Filesystem usage + +```text +Filesystem Size Used Avail Use% Mounted on +/dev/sda1 150G 4.1G 140G 3% / +/dev/sda15 253M 146K 252M 1% /boot/efi +/dev/sdb 49G 261M 47G 1% /opt/torrust/storage +``` + +### Docker Compose volume mounts + +The `docker-compose.yml` uses **relative paths** (`./storage/...`) for all +volume mounts. Because the working directory is `/opt/torrust` and +`/opt/torrust/storage` is on `sdb`, every bind mount resolves to the Hetzner +volume: + +| Service | Host path (→ sdb) | Container path | Purpose | +| ---------- | -------------------------------- | --------------------------- | ------------------ | +| caddy | `./storage/caddy/etc/Caddyfile` | `/etc/caddy/Caddyfile` | Config (read-only) | +| caddy | `./storage/caddy/data` | `/data` | TLS certificates | +| caddy | `./storage/caddy/config` | `/config` | Runtime config | +| tracker | `./storage/tracker/lib` | `/var/lib/torrust/tracker` | Tracker library | +| tracker | `./storage/tracker/log` | `/var/log/torrust/tracker` | Logs | +| tracker | `./storage/tracker/etc` | `/etc/torrust/tracker` | Config | +| mysql | `./storage/mysql/data` | `/var/lib/mysql` | MySQL data files | +| prometheus | `./storage/prometheus/etc` | `/etc/prometheus` | Config | +| grafana | `./storage/grafana/data` | `/var/lib/grafana` | Grafana database | +| grafana | `./storage/grafana/provisioning` | `/etc/grafana/provisioning` | Dashboards/DS | + +### Data on the volume + +Command (run with `sudo` to read MySQL-owned files; excludes MySQL internal +schemas and InnoDB redo/temp directories for brevity): + +```bash +sudo tree /opt/torrust/storage -L 4 --prune \ + -I 'performance_schema|#innodb_redo|#innodb_temp|sys' +``` + +Output (2026-03-04, post-deployment): + +```text +/opt/torrust/storage +├── backup +│ └── etc +│ ├── backup.conf +│ └── backup-paths.txt +├── caddy +│ ├── config +│ │ └── caddy +│ │ └── autosave.json +│ ├── data +│ │ └── caddy +│ │ ├── instance.uuid +│ │ └── last_clean.json +│ └── etc +│ └── Caddyfile +├── grafana +│ ├── data +│ │ └── grafana.db +│ └── provisioning +│ ├── dashboards +│ │ └── torrust.yml +│ └── datasources +│ └── prometheus.yml +├── mysql +│ └── data +│ ├── auto.cnf +│ ├── binlog.000001 +│ ├── binlog.000002 +│ ├── binlog.index +│ ├── ca-key.pem +│ ├── ca.pem +│ ├── client-cert.pem +│ ├── client-key.pem +│ ├── ib_buffer_pool +│ ├── ibdata1 +│ ├── mysql.ibd +│ ├── mysql_upgrade_history +│ ├── private_key.pem +│ ├── public_key.pem +│ ├── server-cert.pem +│ ├── server-key.pem +│ ├── torrust_tracker +│ │ ├── keys.ibd +│ │ ├── torrent_aggregate_metrics.ibd +│ │ ├── torrents.ibd +│ │ └── whitelist.ibd +│ ├── undo_001 +│ └── undo_002 +├── prometheus +│ └── etc +│ └── prometheus.yml +└── tracker + ├── etc + │ └── tracker.toml + └── lib + └── database + └── tracker.db +``` + +> **Note**: TLS certificates are stored deeper inside `caddy/data/caddy/` under +> `acme/` (ACME account) and `certificates/` (one subdirectory per domain) — +> omitted above by the depth limit. + +MySQL occupies ~209 MB on the volume. The volume has 47 GB free (1% used). + +## Conclusion + +All persistent data — MySQL, TLS certificates, Grafana database, tracker +config, and logs — is written to the attached 50 GB Hetzner volume (`sdb`). +The server's internal disk is used only for the OS. If the server is destroyed +and a new one is created with the same volume attached and mounted at +`/opt/torrust/storage`, all data will be preserved. diff --git a/docs/deployments/hetzner-demo-tracker/verify/udp-tracker.md b/docs/deployments/hetzner-demo-tracker/verify/udp-tracker.md new file mode 100644 index 000000000..3b0eb001d --- /dev/null +++ b/docs/deployments/hetzner-demo-tracker/verify/udp-tracker.md @@ -0,0 +1,158 @@ +# UDP Tracker Verification + +**Status**: ✅ Verified (2026-03-04) + +## Endpoints + +| Domain | URL | +| ------------- | ------------------------------------------ | +| UDP Tracker 1 | `udp://udp1.torrust-tracker-demo.com:6969` | +| UDP Tracker 2 | `udp://udp2.torrust-tracker-demo.com:6868` | + +## 1. Port Connectivity + +Check that the UDP ports are open and reachable. + +> **Note**: `nc -u -z` is unreliable for UDP — there is no handshake to confirm +> the port is open. Use the BEP 15 script below as the real connectivity test. + +```bash +# Test port 6969 +nc -u -z -w 3 udp1.torrust-tracker-demo.com 6969 && echo "port 6969 open" || echo "port 6969 closed" + +# Test port 6868 +nc -u -z -w 3 udp2.torrust-tracker-demo.com 6868 && echo "port 6868 open" || echo "port 6868 closed" +``` + +## 2. UDP Tracker Protocol Test (BEP 15) + +The UDP tracker protocol (defined in +[BEP 15](https://www.bittorrent.org/beps/bep_0015.html)) requires a two-step +handshake: a connection request followed by an announce/scrape request. + +Use this Python script to perform a full connection handshake against both +trackers: + +```python +import socket +import struct +import random + +for host, port, label in [ + ("udp1.torrust-tracker-demo.com", 6969, "UDP Tracker 1 (port 6969)"), + ("udp2.torrust-tracker-demo.com", 6868, "UDP Tracker 2 (port 6868)"), +]: + transaction_id = random.randint(0, 0xFFFFFFFF) + packet = struct.pack(">QII", 0x41727101980, 0, transaction_id) + sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + sock.settimeout(5) + try: + sock.sendto(packet, (host, port)) + data, _ = sock.recvfrom(16) + action, resp_tid, connection_id = struct.unpack(">IIQ", data) + assert action == 0, f"unexpected action: {action}" + assert resp_tid == transaction_id, "transaction ID mismatch" + print(f"✅ {label}: Connected! connection_id = {connection_id:#018x}") + except socket.timeout: + print(f"❌ {label}: Timeout — no response") + except Exception as e: + print(f"❌ {label}: Error — {e}") + finally: + sock.close() +``` + +Expected output: + +```text +✅ UDP Tracker 1 (port 6969): Connected! connection_id = 0x<16-digit hex value> +✅ UDP Tracker 2 (port 6868): Connected! connection_id = 0x<16-digit hex value> +``` + +## 3. Using a BitTorrent Client + +The most realistic verification is to add a torrent to a BitTorrent client +(e.g. qBittorrent, Transmission, Deluge) and configure it to use one of the +tracker URLs. The client will send a UDP announce and the tracker will respond +with a peer list. + +Example magnet link using both UDP trackers: + +```text +magnet:?xt=urn:btih:0000000000000000000000000000000000000000 + &tr=udp://udp1.torrust-tracker-demo.com:6969/announce + &tr=udp://udp2.torrust-tracker-demo.com:6868/announce +``` + +Note: a real info hash is needed for a meaningful announce. The zero hash above +will return an empty peer list but confirms the tracker is reachable. + +## 4. Using the Torrust Tracker Client + +The [Torrust Tracker](https://github.com/torrust/torrust-tracker) project ships +a reference `udp_tracker_client` binary located in +[`console/tracker-client`](https://github.com/torrust/torrust-tracker/tree/develop/console/tracker-client). +It implements the full BEP 15 protocol and sends a real `announce` request, +making it a definitive end-to-end test. + +### UDP Tracker 1 (port 6969) + +```bash +# From the torrust-tracker console/tracker-client directory +cargo run --bin udp_tracker_client announce \ + udp://udp1.torrust-tracker-demo.com:6969 \ + 9c38422213e30bff212b30c360d26f9a02136422 +``` + +Output: + +```json +{ + "AnnounceIpv4": { + "transaction_id": -888840697, + "announce_interval": 300, + "leechers": 0, + "seeders": 1, + "peers": [] + } +} +``` + +### UDP Tracker 2 (port 6868) + +```bash +cargo run --bin udp_tracker_client announce \ + udp://udp2.torrust-tracker-demo.com:6868 \ + 9c38422213e30bff212b30c360d26f9a02136422 +``` + +Output: + +```json +{ + "AnnounceIpv4": { + "transaction_id": -888840697, + "announce_interval": 300, + "leechers": 0, + "seeders": 2, + "peers": ["0.0.0.0:0", "0.0.0.0:65535", "2.137.92.24:48887"] + } +} +``` + +Both trackers responded with `announce_interval: 300` (5 minutes), confirming +announces were processed correctly. Tracker 2 returned peers registered from +previous announces: the `0.0.0.0` entries are placeholder peers from the BEP 15 +handshake script, and `2.137.92.24:48887` is the local machine running the +tracker client — confirming the announce was registered correctly and the peer +IP was recorded as seen by the server. + +## Results + +| Check | Result | Notes | +| ----------------------------------- | ------ | ------------------------------------------------------ | +| UDP port 6969 open | ✅ | BEP 15 handshake succeeded | +| UDP port 6868 open | ✅ | BEP 15 handshake succeeded | +| BEP 15 handshake (tracker 1) | ✅ | `connection_id = 0x927bc33b3260b795` | +| BEP 15 handshake (tracker 2) | ✅ | `connection_id = 0x59c13493038e3be3` | +| Torrust client announce (tracker 1) | ✅ | `announce_interval=300`, `seeders=1`, peers=0 | +| Torrust client announce (tracker 2) | ✅ | `announce_interval=300`, `seeders=2`, 3 peers returned | diff --git a/docs/issues/405-deploy-hetzner-demo-tracker-and-document-process.md b/docs/issues/405-deploy-hetzner-demo-tracker-and-document-process.md index 294d2d4b5..5b2fbc2ca 100644 --- a/docs/issues/405-deploy-hetzner-demo-tracker-and-document-process.md +++ b/docs/issues/405-deploy-hetzner-demo-tracker-and-document-process.md @@ -49,23 +49,43 @@ docs/deployments/ ### Phase 1: Setup and Prerequisites -- [ ] Task 1.1: Create `docs/deployments/hetzner-demo-tracker/` directory structure -- [ ] Task 1.2: Document prerequisites (Hetzner account, API token, SSH keys, tool versions) -- [ ] Task 1.3: Verify all required tools are installed and working +- [x] Task 1.1: Create `docs/deployments/hetzner-demo-tracker/` directory structure +- [x] Task 1.2: Document prerequisites (Hetzner account, API token, SSH keys, tool versions) +- [x] Task 1.3: Verify all required tools are installed and working ### Phase 2: Create and Configure Environment -- [ ] Task 2.1: Generate environment configuration template for Hetzner -- [ ] Task 2.2: Document configuration decisions (server type, location, image, credentials) -- [ ] Task 2.3: Create the environment using the deployer +- [x] Task 2.1: Generate environment configuration template for Hetzner +- [x] Task 2.2: Document configuration decisions (server type, location, image, credentials) +- [x] Task 2.3: Create the environment using the deployer ### Phase 3: Deploy the Tracker -- [ ] Task 3.1: Provision infrastructure (create Hetzner VM) -- [ ] Task 3.2: Configure the instance (Docker, SSH, system setup) -- [ ] Task 3.3: Release the application (deploy tracker files) +- [x] Task 3.1: Provision infrastructure (create Hetzner VM) +- [x] Task 3.2: Configure the instance (Docker, SSH, system setup) +- [x] Task 3.3: Release the application (deploy tracker files) - [ ] Task 3.4: Run the services (start the tracker) +### Phase 3.5: Post-Provision Manual Setup + +Steps required after provisioning and before running `configure`. +See [`docs/deployments/hetzner-demo-tracker/post-provision/`](../deployments/hetzner-demo-tracker/post-provision/README.md). + +**DNS Setup** ([dns-setup.md](../deployments/hetzner-demo-tracker/post-provision/dns-setup.md)): + +- [x] Task 3.5.1: Assign IPv4 floating IP (`116.202.176.169`) to the server in Hetzner Console +- [x] Task 3.5.2: Assign IPv6 floating IP (`2a01:4f8:1c0c:9aae::/64`) to the server in Hetzner Console +- [x] Task 3.5.3: Configure floating IPs permanently inside the VM (netplan) +- [x] Task 3.5.4: Create DNS records for all six subdomains via Hetzner Cloud API +- [x] Task 3.5.5: Verify all DNS records resolve correctly + +**Volume Setup** ([volume-setup.md](../deployments/hetzner-demo-tracker/post-provision/volume-setup.md)): + +- [x] Task 3.5.6: Create a 50 GB Hetzner volume (`torrust-tracker-demo-storage`) in `nbg1` +- [x] Task 3.5.7: Format the volume (`ext4`) and mount it at `/opt/torrust/storage` +- [x] Task 3.5.8: Add the volume to `/etc/fstab` for persistent mounting +- [x] Task 3.5.9: Verify volume is correctly mounted and writable + ### Phase 4: Verify and Document - [ ] Task 4.1: Verify tracker is accessible and functioning diff --git a/project-words.txt b/project-words.txt index af3a0c316..fd4f71f3c 100644 --- a/project-words.txt +++ b/project-words.txt @@ -1,52 +1,169 @@ AAAAB AAAAC AAAAI +AGENTS Adapty -addgroup Addrs +Alertmanager +André +Apryse +Aridane +Ashburn +Autorestic +Avalonia +Azevedo +BBDBE +Backblaze +Banuba +Bilu +Bitwarden +Boto +Brinke +Bynder +CIFS +CRCL +Caddyfile +Certbot +Checkly +Ciphernutz +Cockburn +Codegen +Containerfile +Cowan +Crossplane +Cunha +DGRAM +Datagram +Desynchronization +Distrib +Dockerfiles +Dreamfactory +Duplicati +EAAAADAQABAAABAQC +ENDSSH +ENISA +ENOENT +EPEL +Fairchild +Falkenstein +Firestore +Freshping +Frontegg +Geeksfor +Gossman +Grafana +Grafonnet +Graça +HIDS +Herberto +Hillsboro +Hostnames +ISAM +Inno +Kiota +Kopia +LUKS +Liskov +Litestream +Luciq +MAAACBA +MLKEM +MOUNTPOINTS +MTCPS +MVCC +MVVM +Martín +Mermaid +Moreira +MyISAM +NOPASSWD +NXDOMAIN +Nushell +OAAAAN +OCSP +OOXML +OSSEC +Ollama +Ondřej +Osherove +Pikuma +Pingoo +Preinstalling +Promon +Pulumi +Pygame +Pythonic +QUIC +RAII +RUSTDOCFLAGS +Regenerable +Repomix +Repositóri +Rescan +Restic +Runrestic +Rustdoc +SARIF +SCRIPTDIR +SWEBOK +Scanbot +Scribd +Scriptability +Shivavangari +Silverlight +Sophos +StorageGRID +Subissue +Swatinem +Swebok +Tablespace +Taplo +Tera +Testcontain +Testcontainers +Testinfra +Torrust +Trackon +Traefik +Unflushed +Unkey +VARCHAR +Wazuh +XTSTEP +Xtra +Zeroize +Zálešák +addgroup adduser agentic -AGENTS agentskills -Alertmanager alpn -André androiddev appender appendonly -Apryse aquasec aquasecurity architecting -Aridane -Ashburn autocheckpoint autolinks +automount +autoremove autorestart -Autorestic -Avalonia -Azevedo -Backblaze backlinks -Banuba -BBDBE bencoded -Bilu +binlog binstall bitrot -Bitwarden blackbox +blkid bootcmd -Boto -Brinke browsable +btih btrfs buildx -Bynder -Caddyfile cdmon +celano certbot -Certbot certonly chatbots chdir @@ -57,51 +174,40 @@ checkpointing childlogdir chkdsk chrono -CIFS -Ciphernutz clig clippy clonable cloneable cloudinit -Cockburn -Codegen +codel completei concepsts configurator connectionless connrefused containerd -Containerfile copilotignore -Cowan cpus -CRCL creds crond crontabs -Crossplane -Cunha cursorignore custompass customuser -Datagram +cyberneering dcron dearmor debootstrap debuginfo deogmiudufm derefs -Desynchronization devkits devpass dfsg dirmngr -Distrib distro distroless distutils -Dockerfiles dockerhub doctest doctests @@ -109,74 +215,56 @@ downcasted downcasting downloadedi dpkg -Dreamfactory drwxr drwxrwxr dtolnay -Duplicati -EAAAADAQABAAABAQC ehthumbs elif -Émojis endfor endraw -ENISA entr envsubst epel -EPEL eprint eprintln equalto esac +ethernets executability exfiltration exitcode -Fairchild -Falkenstein filesd -Firestore flatlined -Frontegg frontends frontmatter fswc gamedev -Geeksfor getent getopt gobinary goroutines -Gossman gosu gpgv -Graça -Grafana -Grafonnet handleable hashset healthcheck healthchecks -Herberto hetznercloud hexdigit hexdump -HIDS -Hillsboro -Hostnames hotfixes htdocs hugepages +ibdata idna impls incompletei initialisation -Inno +innodb inspectable instructionsets intervali ionice -ISAM isdir isreg josecelano @@ -187,9 +275,7 @@ keepalive keygen keypair keyrings -Kiota kopia -Kopia kutca larstobi leecher @@ -203,32 +289,24 @@ libpython libsqlite lifecycles lineinfile -Liskov listenfd listhost litestream -Litestream loadability logfile logfiles logicaldisk loglevel lspconfig -Luciq -LUKS lvremove lxdbr -MAAACBA -Martín maxbytes memalign -Mermaid mgmt millis minizip mkdir mktemp -MLKEM mlock mlockfile mmin @@ -236,25 +314,23 @@ moby mockall mocksecret momentjs -Moreira +mountpoint mprotect mpsc -MTCPS mtorrust multiprocess -MVCC -MVVM myapp myenv -MyISAM mysqladmin mysqlcheck +mysqld mysqldump mysqlpump nameof namespacing nanos netfilter +netplan networkd newgrp newtype @@ -266,20 +342,13 @@ nmap nocapture noconfirm nodaemon +nofail noninteractive noout -NOPASSWD nslookup nullglob -OAAAAN -OCSP -Ollama -Ondřej oneline ooxml -OOXML -Osherove -OSSEC overpromise pacman parameterizing @@ -294,9 +363,7 @@ peerslee pgzip pidfile pids -Pikuma pingoo -Pingoo pingooio pipeable pipefail @@ -305,101 +372,85 @@ pkcs pkill postconditions preconfigured -Preinstalling preinstalls prereq println procps prodrigestivill -Promon promtool ptracker publickey publicsuffix pullable -Pulumi purgeable -Pygame pypdf pytesseract pytest -Pythonic -QUIC -RAII +qdisc +qlen +randint rclone readlink realpath rebranded recognisable +recvfrom reentrancy -Regenerable reimplementation reissuance -Repomix -Repositóri reprioritize reprovision reprovisioning reqwest -Rescan resolv restic -Restic returncode rgba rootpass rpcinterface +rrset +rrsets rsplit rstest runbooks runcmd runnability runpip -Runrestic rustc -Rustdoc -RUSTDOCFLAGS rustflags rustls rustup rwxrwx sandboxed sarif -SARIF -Scanbot scannability schemafile schemars -Scribd -Scriptability scriptable -SCRIPTDIR secureboot selectattr +sendto serde serialisation serialising serverurl +settimeout shellcheck -Shivavangari -Silverlight smallstep smorimoto -Sophos sourceable -spëcial spki +spëcial sqlx sshpass standardisation startretries +statuspage stdlib -StorageGRID stringly subcontroller subcontrollers subhandlers -Subissue subissues subshell subshells @@ -408,27 +459,17 @@ substep supervisorctl supervisord swappability -Swatinem -Swebok -SWEBOK sysfs sysv -Tablespace tablespaces taiki tamasfe -Taplo taskkill tasklist -Tera terraformrc -tést -Testcontain testcontainer testcontainers -Testcontainers testhost -Testinfra testkey testpass testuser @@ -447,9 +488,7 @@ tokei toolkits toolsets torrust -Torrust touchpoint -Traefik transactionally trixie tryfrom @@ -458,12 +497,12 @@ tulpn turbofish typestate tzdata +tést ulpn +unconfigured undertested unergonomic -Unflushed unittests -Unkey unrepresentable unsubscription userexample @@ -473,30 +512,25 @@ userpass userspace usize utmp -VARCHAR vbqajnc venv venvs versionable viewmodel vulns -Wazuh webservers writability writeln wrongpassword -Xtra xtrabackup -XTSTEP yourdomain youruser -Zálešák zcat zeroize -Zeroize +zoneinfo zstd +Émojis значение ключ конфиг файл -Nushell diff --git a/src/adapters/ssh/client.rs b/src/adapters/ssh/client.rs index 481416fc9..144106f6f 100644 --- a/src/adapters/ssh/client.rs +++ b/src/adapters/ssh/client.rs @@ -158,10 +158,8 @@ impl SshClient { let mut attempt = 0; while attempt < max_attempts { - let result = self.test_connectivity(); - - match result { - Ok(true) => { + match self.execute_with_options("echo 'SSH connected'", &[]) { + Ok(_) => { info!( operation = "ssh_connectivity", host_ip = %self.ssh_config.host_ip(), @@ -170,18 +168,17 @@ impl SshClient { ); return Ok(()); } - Ok(false) => { - // Connection failed, continue trying + Err(CommandError::ExecutionFailed { ref stderr, .. }) => { if (attempt + 1) % conn_config.retry_log_frequency == 0 { info!( operation = "ssh_connectivity", host_ip = %self.ssh_config.host_ip(), attempt = attempt + 1, max_attempts = max_attempts, + reason = %stderr, "Still waiting for SSH connectivity" ); } - tokio::time::sleep(Duration::from_secs(u64::from( conn_config.retry_interval_secs, ))) @@ -211,6 +208,7 @@ impl SshClient { /// - `StrictHostKeyChecking`: `no` (disable host key verification) /// - `UserKnownHostsFile`: `/dev/null` (ignore known hosts file) /// - `ConnectTimeout`: configured timeout (prevents hanging) + /// - `IdentitiesOnly`: `yes` (only use the configured key, ignore SSH agent) /// /// These defaults ensure reliable automation but can be overridden by /// user-provided options in `additional_options`. @@ -225,6 +223,11 @@ impl SshClient { .connect_timeout_secs .to_string(), ); + // Only use the explicitly configured identity file, ignoring any keys + // loaded in the SSH agent. Without this, SSH may exhaust the server's + // MaxAuthTries limit by trying agent keys before the configured key, + // causing "Too many authentication failures" on every attempt. + defaults.insert("IdentitiesOnly".to_string(), "yes".to_string()); defaults } @@ -580,8 +583,8 @@ mod tests { // Act let default_options = ssh_client.build_default_ssh_options(); - // Assert: Should contain 3 key-value pairs - assert_eq!(default_options.len(), 3); + // Assert: Should contain 4 key-value pairs + assert_eq!(default_options.len(), 4); // Verify expected keys and values assert_eq!( @@ -596,6 +599,10 @@ mod tests { default_options.get("ConnectTimeout"), Some(&expected_timeout.to_string()) ); + assert_eq!( + default_options.get("IdentitiesOnly"), + Some(&"yes".to_string()) + ); } #[test] diff --git a/src/adapters/ssh/config.rs b/src/adapters/ssh/config.rs index 269f4fd2f..9664db8e6 100644 --- a/src/adapters/ssh/config.rs +++ b/src/adapters/ssh/config.rs @@ -24,13 +24,14 @@ pub const DEFAULT_SSH_PORT: u16 = 22; /// Default SSH connection timeout in seconds pub const DEFAULT_CONNECT_TIMEOUT_SECS: u32 = 5; -/// Default maximum number of connection retry attempts -/// Set to 60 to allow up to 120 seconds total wait time (60 attempts × 2 second interval) -/// This accounts for cloud-init completion time when custom SSH ports are configured +/// Default maximum number of connection retry attempts. +/// Set to 60 to allow up to 300 seconds total wait time (60 attempts × 5 second interval). +/// Cloud-init provisioning (user creation, SSH key injection) can take over 3 minutes on some +/// providers and small machines, so a 5-minute budget is used as the default. pub const DEFAULT_MAX_RETRY_ATTEMPTS: u32 = 60; /// Default retry interval in seconds -pub const DEFAULT_RETRY_INTERVAL_SECS: u32 = 2; +pub const DEFAULT_RETRY_INTERVAL_SECS: u32 = 5; /// Default retry log frequency (log every N attempts) pub const DEFAULT_RETRY_LOG_FREQUENCY: u32 = 5; @@ -117,7 +118,7 @@ impl SshConnectionConfig { /// use torrust_tracker_deployer_lib::adapters::ssh::SshConnectionConfig; /// /// let config = SshConnectionConfig::default(); - /// assert_eq!(config.total_timeout_secs(), 120); // 60 attempts × 2 seconds + /// assert_eq!(config.total_timeout_secs(), 300); // 60 attempts × 5 seconds /// ``` #[must_use] pub fn total_timeout_secs(&self) -> u32 { @@ -130,10 +131,10 @@ impl Default for SshConnectionConfig { /// /// Uses constants defined at module level: /// - Connection timeout: `DEFAULT_CONNECT_TIMEOUT_SECS` (5 seconds) - /// - Max retry attempts: `DEFAULT_MAX_RETRY_ATTEMPTS` (30) - /// - Retry interval: `DEFAULT_RETRY_INTERVAL_SECS` (2 seconds) + /// - Max retry attempts: `DEFAULT_MAX_RETRY_ATTEMPTS` (60) + /// - Retry interval: `DEFAULT_RETRY_INTERVAL_SECS` (5 seconds) /// - Retry log frequency: `DEFAULT_RETRY_LOG_FREQUENCY` (every 5 attempts) - /// - Total wait time: 30 × 2 = 60 seconds + /// - Total wait time: 60 × 5 = 300 seconds fn default() -> Self { Self { connect_timeout_secs: DEFAULT_CONNECT_TIMEOUT_SECS, diff --git a/src/infrastructure/templating/docker_compose/local_validator.rs b/src/infrastructure/templating/docker_compose/local_validator.rs index 061d7f21e..1d31048a1 100644 --- a/src/infrastructure/templating/docker_compose/local_validator.rs +++ b/src/infrastructure/templating/docker_compose/local_validator.rs @@ -14,6 +14,14 @@ //! failure. This is a cheap, dependency-free check — `docker` is already a //! project requirement. //! +//! ## Graceful degradation when Docker is not available +//! +//! When the deployer itself runs **inside a Docker container** (the typical +//! production usage pattern), `docker` is not installed inside that container. +//! In that case the validator logs a warning and skips the check rather than +//! failing the command — a missing tool cannot indicate a template error. Any +//! other OS error (e.g. permission denied) is still treated as a hard failure. +//! //! ## Example failure caught by this validator //! //! An empty `networks:` key (no list items) produces: @@ -32,6 +40,7 @@ //! validate_docker_compose_file(Path::new("build/my-env/docker-compose")).unwrap(); //! ``` +use std::io; use std::path::Path; use std::process::Command; @@ -40,9 +49,9 @@ use thiserror::Error; /// Errors that can occur when validating a rendered `docker-compose.yml` #[derive(Error, Debug)] pub enum DockerComposeLocalValidationError { - /// The `docker` binary could not be executed (not installed or not in PATH) + /// The `docker` binary could be found but could not be executed (e.g. permission denied) #[error( - "Failed to run 'docker compose config --quiet' — is Docker installed and in PATH?\n\ + "Failed to run 'docker compose config --quiet' — unexpected OS error.\n\ Source: {source}" )] CommandExecutionFailed { @@ -69,7 +78,8 @@ impl DockerComposeLocalValidationError { pub fn help(&self) -> String { match self { Self::CommandExecutionFailed { .. } => { - "Install Docker and ensure it is added to your PATH, then re-run the command." + "An unexpected OS error occurred while trying to run Docker.\n\ + Check that Docker is installed and that the current user has permission to run it." .to_string() } Self::InvalidDockerComposeFile { .. } => { @@ -89,6 +99,11 @@ impl DockerComposeLocalValidationError { /// The command is executed in `compose_dir` so that it picks up the /// `docker-compose.yml` (and optionally `.env`) that live there. /// +/// When `docker` is not installed (e.g. when the deployer itself runs inside a +/// Docker container), validation is **skipped** with a warning rather than +/// treating the missing tool as a template error. Any other OS error is still +/// a hard failure. +/// /// # Arguments /// /// * `compose_dir` — directory containing the rendered `docker-compose.yml` @@ -96,7 +111,7 @@ impl DockerComposeLocalValidationError { /// # Errors /// /// - [`DockerComposeLocalValidationError::CommandExecutionFailed`] if `docker` -/// cannot be executed (not installed, not in PATH, OS error, …) +/// is found but cannot be executed due to an unexpected OS error /// - [`DockerComposeLocalValidationError::InvalidDockerComposeFile`] if the /// file fails structural validation /// @@ -113,11 +128,25 @@ pub fn validate_docker_compose_file( "Validating rendered docker-compose.yml with 'docker compose config --quiet'" ); - let output = Command::new("docker") + let output = match Command::new("docker") .args(["compose", "config", "--quiet"]) .current_dir(compose_dir) .output() - .map_err(|source| DockerComposeLocalValidationError::CommandExecutionFailed { source })?; + { + Ok(out) => out, + Err(err) if err.kind() == io::ErrorKind::NotFound => { + tracing::warn!( + compose_dir = %compose_dir.display(), + "Skipping local docker-compose.yml validation: \ + 'docker' is not available in PATH (deployer may be running inside a container). \ + The rendered file will be validated by Docker Compose on the remote host." + ); + return Ok(()); + } + Err(source) => { + return Err(DockerComposeLocalValidationError::CommandExecutionFailed { source }); + } + }; if output.status.success() { tracing::debug!( @@ -232,13 +261,11 @@ services: #[test] fn it_should_return_help_message_for_command_execution_failed() { let error = DockerComposeLocalValidationError::CommandExecutionFailed { - source: std::io::Error::new(std::io::ErrorKind::NotFound, "not found"), + // Use PermissionDenied — NotFound is handled as a skip (not an error) + source: std::io::Error::new(std::io::ErrorKind::PermissionDenied, "permission denied"), }; let help = error.help(); assert!(!help.is_empty(), "Help message must not be empty"); - assert!( - help.contains("Docker"), - "Help should mention Docker installation" - ); + assert!(help.contains("Docker"), "Help should mention Docker"); } } diff --git a/templates/ansible/ansible.cfg b/templates/ansible/ansible.cfg index d22f9eff9..43bab8cbc 100644 --- a/templates/ansible/ansible.cfg +++ b/templates/ansible/ansible.cfg @@ -57,4 +57,6 @@ timeout = 30 # - ControlPersist=60s: Keep connections alive for 60 seconds after last use # - StrictHostKeyChecking=no: Don't verify SSH host keys (lab environment only) # - UserKnownHostsFile=/dev/null: Don't save host keys to known_hosts file -ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null +# - IdentitiesOnly=yes: Only use the explicitly configured key, ignore SSH agent +# (prevents "Too many authentication failures" when agent has many keys loaded) +ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes