Skip to content

feat: [#220] Tracker Slice - Release and Run Commands#221

Merged
josecelano merged 70 commits intomainfrom
220-tracker-slice-release-run-commands
Dec 11, 2025
Merged

feat: [#220] Tracker Slice - Release and Run Commands#221
josecelano merged 70 commits intomainfrom
220-tracker-slice-release-run-commands

Conversation

@josecelano
Copy link
Copy Markdown
Member

@josecelano josecelano commented Dec 8, 2025

Summary

This PR implements the complete Tracker Slice for Release and Run commands, replacing the demo nginx service with a fully functional Torrust Tracker deployment. The implementation follows an incremental, testable approach with 8 phases, all of which have been successfully completed.

What's Been Implemented

✅ All 8 Phases Complete

Phase 0: Module Refactoring (Completed)

  • Renamed src/infrastructure/external_tools/src/infrastructure/templating/
  • Better reflects the module's purpose (template rendering logic)

Phase 1: Storage Directory Creation (Completed)

  • Ansible playbook creates tracker storage structure: /opt/torrust/storage/tracker/{etc,lib/database,log}
  • Integrated into ReleaseCommand workflow
  • Manual E2E test verified directory creation with correct ownership and permissions

Phase 2: SQLite Database Initialization (Completed)

  • Ansible playbook initializes SQLite database file (tracker.db)
  • Empty database file created with proper permissions (0644, torrust:torrust)
  • Integrated into ReleaseCommand workflow after storage creation

Phase 3: Docker Compose Environment Variables (Completed)

  • Implemented .env file generation using Project Generator pattern
  • Created EnvContext, EnvTemplate, and EnvRenderer components
  • Refactored to three-layer architecture (Wrapper → Renderer → Generator)
  • .env file deployed to /opt/torrust/.env with tracker configuration

Phase 4: Tracker Configuration Template (Completed)

  • Created templates/tracker/tracker.toml.tera with Torrust Tracker configuration
  • Implemented TrackerProjectGenerator following Project Generator pattern
  • Created RenderTrackerTemplatesStep and DeployTrackerConfigStep
  • Configuration deployed to /opt/torrust/storage/tracker/etc/tracker.toml

Phase 5: Docker Compose Service Replacement (Completed)

  • Replaced nginx demo service with Torrust Tracker in docker-compose
  • Configured UDP tracker ports (6969), HTTP tracker (7070), and HTTP API (1212)
  • Volume mounts for storage directories (lib, log, etc)
  • Environment variable integration for configuration

Phase 6: Environment Configuration Support (Completed)

  • Added TrackerConfig schema to environment configuration
  • Users can configure: UDP trackers, HTTP trackers, private/public mode, API admin token
  • All templates dynamically populated from environment config
  • Template generation uses tracker configuration for variable substitution

Phase 7: Firewall Configuration (Completed)

  • Ansible playbook configures UFW firewall for tracker ports
  • Opens UDP tracker ports, HTTP tracker ports, and HTTP API port
  • Port extraction from environment configuration
  • Integrated into ConfigureCommand workflow
  • Manual E2E test verified all firewall rules correctly applied

Phase 8: E2E Test Updates (Completed)

  • Updated RunningServicesValidator for external-only validation
  • Checks tracker HTTP API health endpoint (http://<vm-ip>:1212/api/health_check)
  • Checks HTTP tracker health endpoint (http://<vm-ip>:7070/api/health_check)
  • Simplified to external validation (proves service AND firewall working)
  • Refactored validation code for better maintainability

Architecture Improvements

Domain-Driven Design (DDD)

  • Proper layer separation: Domain DTOs, Infrastructure templates, Application workflows
  • Created tracker DTO module with validation (port 0 not supported, documented in ADR)
  • Multiple HTTP tracker support with external validation for each instance

Infrastructure Module Organization

  • Created external_validators/ module for direct HTTP validation (no SSH required)
  • Better separation: remote_actions/ (via SSH) vs external_validators/ (direct HTTP)
  • Documented in ADR for future maintainability

Template System

  • Consistent Project Generator pattern across all template modules
  • Three-layer architecture: Wrapper → Renderer → Generator
  • Static templates explicitly registered, dynamic templates with .tera extension

Testing

Manual E2E Testing

All phases manually tested with complete verification:

  • ✅ Storage directories created with correct permissions
  • ✅ SQLite database initialized
  • .env file deployed with tracker configuration
  • tracker.toml configuration deployed
  • ✅ Tracker container running with all ports exposed
  • ✅ Firewall rules applied (UDP 6969, HTTP 7070, API 1212)
  • ✅ External HTTP health checks passing

Automated E2E Testing

  • ✅ Full E2E test suite passing (complete workflow validation)
  • ✅ Infrastructure lifecycle tests passing (provisioning and destruction)
  • ✅ Deployment workflow tests passing (configuration, release, run)
  • ✅ All 1424 unit tests passing
  • ✅ All 40 integration tests passing
  • ✅ All pre-commit checks passing

Real-World Validation

  • ✅ Tracker HTTP API accessible: curl http://<vm-ip>:1212/api/health_check returns HTTP 200
  • ✅ HTTP Tracker accessible: curl http://<vm-ip>:7070/api/health_check returns HTTP 200
  • ✅ All tracker ports open in firewall (verified with sudo ufw status)
  • ✅ Docker services running: docker compose ps shows tracker in "running" state

Documentation Updates

  • ✅ Complete implementation plan with all 8 phases documented
  • ✅ Manual testing workflow guide with real command outputs
  • ✅ Architecture Decision Records (ADRs) for key design decisions
  • ✅ Updated quick-start guide with tracker deployment workflow
  • ✅ Updated manual E2E testing documentation

Acceptance Criteria

All acceptance criteria from issue #220 have been met:

Quality Checks

  • ✅ Pre-commit checks pass
  • ✅ All linters passing (markdown, yaml, toml, cspell, clippy, rustfmt, shellcheck)
  • ✅ All unit, integration, and E2E tests passing

Functional Requirements

  • ✅ Storage directories created on VM
  • ✅ SQLite database initialized
  • ✅ Docker compose .env file deployed
  • ✅ Tracker configuration file deployed
  • ✅ Docker compose service updated with tracker
  • ✅ Firewall configured for all tracker ports
  • ✅ Tracker container starts successfully
  • ✅ Tracker HTTP API responds to health checks
  • ✅ Tracker ports accessible externally

Configuration Requirements

  • ✅ Environment config supports tracker configuration
  • ✅ Users can specify UDP/HTTP tracker instances
  • ✅ Private/public mode configurable
  • ✅ Database name configurable
  • ✅ API admin token configurable
  • ✅ Template variables correctly substituted

Architecture Requirements

  • ✅ Tracker templates isolated in separate module
  • ✅ Project Generator pattern implemented
  • ✅ Static templates registered in renderer
  • ✅ No mixing of tracker and docker-compose templates

Testing Requirements

  • ✅ All phases manually testable
  • ✅ E2E tests updated for tracker validation
  • ✅ Example environment files demonstrate configuration

Related Documentation

Deployment Workflow

The complete workflow for deploying a Torrust Tracker:

# 1. Create environment from template
cargo run -- create template --provider lxd > envs/my-tracker.json

# 2. Edit environment configuration with your values

# 3. Create environment
cargo run -- create environment --env-file envs/my-tracker.json

# 4. Provision VM
cargo run -- provision my-tracker

# 5. Configure VM (installs Docker, configures firewall, creates storage)
cargo run -- configure my-tracker

# 6. Release (deploys configurations and docker-compose)
cargo run -- release my-tracker

# 7. Run services
cargo run -- run my-tracker

# 8. Verify deployment
VM_IP=$(cargo run -- show my-tracker | grep 'IP Address' | awk '{print $3}')
curl http://$VM_IP:1212/api/health_check  # Tracker API
curl http://$VM_IP:7070/api/health_check  # HTTP Tracker

Next Steps

This PR is ready to merge. All phases completed, all tests passing, all documentation updated.

Future enhancements (not in this PR):

  • Additional tracker configuration options (database driver, intervals, logging)
  • Support for multiple tracker instances
  • Tracker management commands (status, logs, restart)
  • Integration with Index and Grafana services

Closes #220

- Rename src/infrastructure/external_tools/ to src/infrastructure/templating/
- Update all import paths throughout codebase (src and tests)
- Update module documentation to reflect semantic clarity
- Add backticks for OpenTofu and Docker Compose in docs

Phase 0 complete - module now better reflects its purpose of template
rendering and generation rather than just external tool integration.
Update progress tracking checklist to reflect completion of module
renaming from external_tools to templating.
Add tracker storage directory creation as part of the ReleaseCommand workflow.
This creates the required directory structure for the Torrust Tracker application
on the remote host.

Architecture Changes:
- Moved CreateTrackerStorageStep from system/ to application/ layer
  * Rationale: Creating tracker-specific directories is application deployment,
    not system-level configuration
  * ConfigureCommand: System preparation (Docker, security, firewall)
  * ReleaseCommand: Application deployment (app directories, configs)
  * ProvisionCommand: Infrastructure (VMs, networks, volumes)

Components Added:
- templates/ansible/create-tracker-storage.yml: Ansible playbook to create directories
  * Creates /opt/torrust/storage/tracker/{etc,lib/database,log}
  * Sets ownership to ansible_user with 0755 permissions
  * Idempotent operation (safe to run multiple times)

- src/application/steps/application/create_tracker_storage.rs: Step implementation
  * Wraps Ansible playbook execution
  * Follows established Step pattern (like InstallDockerStep)
  * Comprehensive error handling and tracing

- src/application/command_handlers/release/handler.rs: ReleaseCommand integration
  * Added CreateTrackerStorageStep as first step in execute_release_workflow()
  * Executes before template rendering
  * Added error variant TrackerStorageCreation with troubleshooting guide

- src/domain/environment/state/release_failed.rs: State tracking
  * Added ReleaseStep::CreateTrackerStorage enum variant
  * Enables precise failure tracking and recovery guidance

- src/infrastructure/templating/ansible/template/renderer/project_generator.rs
  * Registered create-tracker-storage.yml in copy_static_templates()
  * Critical: Static playbooks must be explicitly registered

Documentation Updates:
- docs/issues/220-tracker-slice-release-run-commands.md
  * Marked Phase 1 as completed
  * Added architecture note explaining ConfigureCommand vs ReleaseCommand distinction
  * Updated tasks with checkmarks for completed items

Directory Structure Created:

Testing:
- Manual E2E test pending (will verify directories are created on actual VM)
- Unit tests added for CreateTrackerStorageStep construction
- Pre-commit checks: All linters pass, all tests pass

Next Phase: Phase 2 - Initialize SQLite database
- Verified directory structure created on VM
- Confirmed ownership and permissions (torrust:torrust, 0755)
- All verification checks passed
- Updated documentation to use 'torrust' username for future tests
- Create Ansible playbook templates/ansible/init-tracker-database.yml
  * Creates empty SQLite database file at /opt/torrust/storage/tracker/lib/database/tracker.db
  * Sets ownership to ansible_user with 0644 permissions
  * Verifies file creation with assertions
  * Idempotent operation

- Create InitTrackerDatabaseStep in application/ layer
  * Wraps Ansible playbook execution
  * Follows Step pattern (similar to CreateTrackerStorageStep)
  * Comprehensive error handling and tracing
  * Placed in application/ layer (app deployment, not system config)

- Integrate into ReleaseCommand workflow
  * Added as second step in execute_release_workflow() after CreateTrackerStorageStep
  * Added ReleaseStep::InitTrackerDatabase enum variant
  * Added TrackerDatabaseInit error variant with troubleshooting guide
  * Updated error handling with complete help text

- Register playbook in AnsibleProjectGenerator
  * Added to copy_static_templates() method
  * Updated playbook count from 10 to 11

- Add project dictionary word: isreg (Ansible stat module field)

Manual E2E Test Results (✅ PASSED):
- Environment: test-phase2 (LXD VM, IP: 10.140.190.228)
- Database file created: /opt/torrust/storage/tracker/lib/database/tracker.db
- Ownership: torrust:torrust (ansible_user)
- Permissions: 0644 (-rw-r--r--)
- File type: empty (expected for new SQLite database)
- Executed as part of ReleaseCommand workflow
- All linters passing, all tests passing
…pattern

Implements Phase 3 of the tracker deployment workflow by adding support
for rendering Docker Compose environment variables file (.env).

Architecture Changes:
- Refactored to Project Generator pattern (three-layer architecture)
- Replaced monolithic renderer (~700 lines) with modular structure
- Added Wrapper layer: EnvContext and EnvTemplate
- Added Renderer layer: EnvRenderer for .env.tera processing
- Added Generator layer: DockerComposeProjectGenerator orchestrator

Implementation:
- Created templates/docker-compose/.env.tera with tracker variables
- Added wrapper types in template/wrappers/env/ directory
- Implemented EnvRenderer in template/renderer/env.rs
- Created DockerComposeProjectGenerator in template/renderer/project_generator.rs
- Updated RenderDockerComposeTemplatesStep to use new generator
- Extended File type with Format::Env and Extension::Env support

Testing:
- All unit tests passing (1353 tests)
- All linters passing (markdown, yaml, toml, cspell, clippy, rustfmt, shellcheck)
- Manual E2E test completed successfully on LXD VM
- Verified .env file generation in build directory
- Verified .env file deployment to VM at /opt/torrust/.env
- Confirmed hardcoded 'MyAccessToken' renders correctly

Documentation:
- Updated docs/technical/template-system-architecture.md with pattern details
- Updated docs/issues/220-tracker-slice-release-run-commands.md with Phase 3 results
- Documented proper workflow: use 'create template --provider lxd' then customize

Notes:
- Hardcoded 'MyAccessToken' in Phase 3 (will be configurable in Phase 6)
- Template file renamed: env.tera → .env.tera (required for File type)
- Follows established Ansible template architecture pattern
- .env file automatically deployed via existing deploy-compose-files.yml playbook
…le pattern

- Create tracker_config submodule under wrapper/ following ansible inventory pattern
- Move context.rs and template.rs into wrapper/tracker_config/
- Rename TrackerRenderer to TrackerConfigRenderer (remove redundant suffix)
- Rename tracker_renderer.rs to tracker_config.rs in renderer/
- Update all module imports and exports throughout the codebase
- Update documentation to reflect new naming

This refactoring aligns the tracker template structure with the established
ansible inventory pattern, improving consistency and maintainability.
…e (Phase 5)

- Replace demo-app (nginx:alpine) with tracker service (torrust/tracker:develop)
- Add tracker environment variables:
  * TORRUST_TRACKER_CONFIG_TOML_PATH: Path to tracker configuration file
  * TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER: SQLite3 database driver
  * TORRUST_TRACKER_CONFIG_OVERRIDE_HTTP_API__ACCESS_TOKENS__ADMIN: API admin token
- Expose tracker ports: 6868/udp, 6969/udp (UDP trackers), 7070 (HTTP tracker), 1212 (HTTP API)
- Mount storage directories: lib, log, etc
- Configure logging: max-size 10m, max-file 10
- Add backend_network for inter-service communication

Manual E2E test verification:
- Tracker container starts successfully
- Configuration loaded correctly from /etc/torrust/tracker/tracker.toml
- UDP trackers listening on ports 6868 and 6969
- HTTP tracker listening on port 7070
- HTTP API listening on port 1212
- Health checks responding with 200 OK
- API stats endpoint working with authentication: curl -H 'Authorization: Bearer MyAccessToken' http://localhost:1212/api/v1/stats

Full E2E test suite: PASSED (97.9 seconds)
- create → provision → configure → release → run → validate → destroy workflow complete

Phase 5 of tracker slice implementation - tracker service now deployed
instead of nginx placeholder.
@josecelano josecelano self-assigned this Dec 8, 2025
josecelano and others added 19 commits December 8, 2025 19:58
Phase 6: Add Environment Configuration Support - Complete

This commit makes the tracker field mandatory in UserInputs, providing
better type safety and eliminating the need for unwrapping logic throughout
the codebase.

Key Changes:

Domain Layer:
- Added TrackerConfig with comprehensive Default implementation
- Default values: SQLite tracker.db, public mode, UDP 6868+6969, HTTP 7070
- Admin token: MyAccessToken
- Changed tracker field from Option<TrackerConfig> to TrackerConfig
- All tracker submodules (database, core, http_api, etc.) with documentation

Application Layer:
- Simplified admin token extraction (removed Option handling)
- Fixed unused self parameter in render_tracker_templates
- Merged identical match arms in release error handling
- Removed all unwrapping logic for tracker configuration

Infrastructure Layer:
- Updated tracker template rendering to use mandatory field
- Simplified context creation (no as_ref() needed)

Testing:
- Updated all tests to use TrackerConfig::default()
- Added test for Default implementation
- All 1386 tests passing

Code Quality:
- Fixed all clippy warnings (doc_markdown, match_same_arms, unused_self)
- Removed long namespace qualifiers where not needed
- Updated doctest with tracker field

Manual E2E Verification:
- Complete deployment workflow tested (create → provision → configure → release → run)
- Tracker configuration deployed correctly on VM
- Docker services running healthy
- All ports exposed correctly

Phase 6 Status: ✅ COMPLETE
…n (partial)

This commit implements the core infrastructure for Phase 7 (Configure Firewall
for Tracker Ports) but does not yet wire the tracker config through the
provision workflow. This is a partial implementation that will be completed
in a follow-up commit.

What's Implemented:

Infrastructure Layer:
- Updated AnsibleVariablesContext to accept tracker configuration
- Added port extraction logic from tracker bind addresses
- Extract UDP tracker ports, HTTP tracker ports, and API port
- Updated variables.yml.tera to include tracker port variables
- Created configure-tracker-firewall.yml playbook for UFW rules

Application Layer:
- Created ConfigureTrackerFirewallStep following existing patterns
- Added step to ConfigureCommandHandler workflow
- Registered playbook in AnsibleProjectGenerator
- Added ConfigureTrackerFirewall to ConfigureStep enum

Testing:
- Added comprehensive unit tests for port extraction
- Tests for valid/invalid bind addresses
- Tests for empty tracker configurations
- All 1390 tests passing

What's NOT Yet Implemented (TODO):

The tracker configuration needs to be passed from the environment through
the provision workflow to the Ansible template rendering. This requires:

1. Update AnsibleTemplateService to accept environment config
2. Update ProvisionCommandHandler to pass environment config
3. Update RenderAnsibleTemplatesStep to forward tracker config
4. Pass tracker config to AnsibleProjectGenerator::render()

This wiring will be completed in the next commit to finish Phase 7.

Current Behavior:
- Firewall playbook will be copied to build directory
- Playbook will skip all tasks (no tracker ports in variables.yml yet)
- No functional change until tracker config is wired through

Technical Details:
- Port extraction uses helper function to parse bind_address strings
- Supports multiple UDP and HTTP tracker instances
- API port currently hardcoded to 1212 (can be made configurable later)
- Playbook conditional on tracker_*_ports variables existence
- Firewall reload preserves existing SSH rules
Fixed documentation linting issues in Phase 7 implementation:
- Added backticks around code references (ConfigureFirewallStep, bind_address, tuple fields)
- Escaped square brackets in rustdoc to prevent broken intra-doc links
- Updated issue spec with Phase 7 test results and progress status

Changes:
- src/application/steps/system/configure_tracker_firewall.rs: Added backticks and escaped brackets
- src/infrastructure/templating/ansible/template/wrappers/variables/context.rs: Added backticks
- docs/issues/220-tracker-slice-release-run-commands.md: Updated Phase 7 status and added E2E test results

All pre-commit checks now pass:
- cargo machete: ✅ No unused dependencies
- Linters: ✅ All passing (markdown, yaml, toml, cspell, clippy, rustfmt, shellcheck)
- Unit tests: ✅ 1390 tests passing
- E2E tests: ✅ All passing (provision, configure, release, full workflow)
…docs

Updated taplo configuration and linting documentation to properly handle
generated/runtime folders and reflect current linting implementation.

Changes to .taplo.toml:
- Added exclude paths for build/, data/, and envs/ directories
- These folders contain generated artifacts and runtime data that shouldn't be linted

Changes to docs/contributing/linting.md:
- Added taplo (TOML linting) and cspell (spell checking) to tools table
- Added new section documenting excluded directories (build/, data/, envs/)
- Explained why these folders should be excluded from all linters
- Updated implementation section to reflect Rust binary (src/bin/linter.rs)
- Removed outdated references to bash scripts in scripts/linting/ directory
- Removed references to parallel execution (no longer recommended)
- Added cspell to individual linter commands
- Updated performance tips with current timing estimates

Rationale for exclusions:
- build/: Contains generated build artifacts and rendered templates
- data/: Contains runtime application data and test outputs
- envs/: Contains user environment configurations (may not follow conventions)

This fixes the taplo linting error encountered during pre-commit checks:
  ERROR taplo:format_files: the file is not properly formatted
  path=".../build/e2e-config/tracker/tracker.toml"

All pre-commit checks now pass successfully.
…complete)

Complete Phase 7 by wiring tracker configuration from environment through the
provision workflow to enable firewall rules for tracker ports.

Changes to RenderAnsibleTemplatesStep:
- Added tracker_config field (TrackerConfig, not Option since it's mandatory)
- Updated constructor to accept TrackerConfig parameter
- Forward tracker config to AnsibleProjectGenerator.render()

Changes to AnsibleTemplateService:
- Refactored render_templates() to accept UserInputs + instance_ip (2 params)
- Previous: 4 params (ssh_credentials, instance_ip, ssh_port, tracker_config)
- Improvement: Better cohesion, separates UserInputs from RuntimeOutputs
- Extract ssh_credentials and tracker from UserInputs internally
- Updated module documentation to reflect new signature

Changes to ProvisionCommandHandler:
- Pass environment.context().user_inputs to AnsibleTemplateService
- Pass instance_ip as second parameter (runtime output)
- Simplified prepare_for_configuration() method

Changes to RegisterCommandHandler:
- Updated to use new AnsibleTemplateService.render_templates() signature
- Pass environment.context().user_inputs instead of individual params

Testing results:
- All 1390 unit tests passing
- Full E2E test passed (102.0s)
- Manual verification: UFW rules correctly configured for all tracker ports
  * SSH port 22: ✅
  * UDP tracker ports 6868, 6969: ✅
  * HTTP tracker port 7070: ✅
  * HTTP API port 1212: ✅
- Variables.yml correctly populated with extracted tracker ports
- All pre-commit checks passing

Design benefits:
- Reduced parameter list from 4 to 2 (cleaner API)
- Better semantic separation (UserInputs vs RuntimeOutputs)
- More maintainable (adding user inputs doesn't change signatures)
- Respects domain model (UserInputs is a cohesive unit)

Phase 7 Status: ✅ COMPLETE
- Infrastructure: commit 6939553
- Wiring: this commit
- All acceptance criteria met
Simplified default tracker configuration to include only one UDP tracker
on port 6969 instead of two (6868 and 6969).

Changes:
- Updated TrackerConfig::default() to have single UDP tracker (6969)
- Updated documentation to reflect one UDP tracker instance
- Updated code example in module documentation
- Fixed test it_should_create_default_tracker_config to expect 1 UDP tracker

Default configuration now includes:
- 1 UDP tracker: 0.0.0.0:6969
- 1 HTTP tracker: 0.0.0.0:7070
- 1 HTTP API: admin_token (MyAccessToken)
Added validation to ensure .env file exists before running docker compose.
Docker Compose requires the .env file to inject environment variables into
the services.

Changes:
- Added task to verify .env file exists in deploy directory
- Added task to fail with actionable error if .env file is missing
- Error message includes fix instructions (run release command)
- Added link to Docker Compose documentation

This prevents cryptic Docker Compose errors when services fail due to
missing environment variables. The error now clearly tells users to run
the release command first to deploy the .env file.
…alidation with parameterized ports

- Remove internal SSH health checks (dual validation → external-only)
- Remove unused ssh_client field and internal validation methods
- Add tracker port fields (tracker_api_port, http_tracker_port) to validator
- Update constructors to accept port parameters
- Simplify execute() to only call validate_external_accessibility()
- Parameterize all validation methods to use configurable ports
- Thread ports from E2eEnvironmentInfo through validation chain
- Extract build_test_ssh_credentials() helper function

This completes Phase 8 of the Tracker Slice implementation. External-only
validation is appropriate for E2E black-box testing since external accessibility
proves services are both running and properly configured. Ports are now
extracted from environment configuration instead of being hardcoded.
…ridge networking

- Created E2eConfigEnvironment for desired configuration (ports from env file)
- Created E2eRuntimeEnvironment for actual runtime state (mapped ports from Docker)
- Created ContainerPorts type for Docker-assigned mapped ports
- Reverted from host to bridge networking to avoid port conflicts on GitHub Actions
- Added port mapping retrieval for SSH and all tracker service ports
- Updated RunningProvisionedContainer to expose additional_mapped_ports

Architecture Benefits:
- Clean separation between configuration (what we want) and runtime (what we get)
- Type system makes the distinction explicit at compile time
- Bridge networking avoids SSH port 22 conflicts on GitHub runners

Known Issue:
- E2E test currently fails at register step because CLI register command
  doesn't support custom SSH ports (reads port 22 from config but actual
  SSH is on mapped port). Next commit will add --ssh-port argument to register.

Co-authored-by: GitHub Copilot <noreply@github.com>
- Added --ssh-port optional CLI argument to register subcommand
- Updated RegisterCommandController to accept custom SSH port
- Modified RegisterCommandHandler to override config port if provided
- Enhanced AnsibleTemplateService to support SSH port override
- E2E test now passes mapped SSH port to register command
- Fixes E2E test failure with bridge networking on GitHub Actions

This allows the register command to connect to SSH servers on non-standard
ports, which is essential for E2E testing with Docker bridge networking
where ports are dynamically mapped (e.g., 22 → 33061).

The custom SSH port is used for both:
1. SSH connectivity validation during registration
2. Ansible inventory generation for subsequent configuration

Production code changes:
- CLI: Added ssh_port: Option<u16> field to Register command
- Presentation: RegisterCommandController.execute accepts optional ssh_port
- Application: RegisterCommandHandler uses custom port for SSH + Ansible templates
- Services: AnsibleTemplateService.render_templates accepts ssh_port_override

E2E test changes:
- TestRunner.register_instance accepts optional SSH port
- ProcessRunner.run_register_command builds CLI args with --ssh-port
- E2E test passes runtime environment's mapped SSH port

With this enhancement, E2E tests work correctly with Docker bridge networking,
eliminating port conflicts on GitHub Actions runners while maintaining
compatibility with normal provisioning workflows.
…sting guide

Created comprehensive Architectural Decision Record (ADR) documenting:
- Context: SSH port conflict on GitHub Actions runners
- Decision: Add optional --ssh-port argument to register command
- Implementation: Layer-by-layer propagation from CLI to Ansible templates
- Consequences: CI compatibility, production use cases, backward compatibility
- Alternatives: Five rejected approaches with detailed rationale

Updated E2E testing guide with new section on SSH port conflicts:
- Explains GitHub Actions SSH port 22 conflict with test containers
- Documents bridge networking solution with dynamic port mapping
- Shows register command usage with --ssh-port argument
- Links to ADR for complete technical details
- Highlights real-world use cases beyond testing

Updated ADR index in docs/decisions/README.md with new entry.

This documentation ensures future developers understand:
1. Why the --ssh-port argument exists
2. How Docker bridge networking solves CI conflicts
3. When to use custom SSH ports in production
4. Implementation details and architectural considerations
- Added tracker field to EnvironmentCreationConfig struct
- Updated template() method to include default tracker config with:
  * UDP tracker on port 6969
  * HTTP tracker on port 7070
  * API with admin token (bind address fixed at 1212 in template)
  * SQLite database with default settings
- Updated to_environment_params() to return tracker config as 6th parameter
- Added Environment::with_working_dir_and_tracker() method
- Added UserInputs::with_tracker() method
- Updated all test fixtures and helpers with tracker configuration
- Fixed generate_environment_config_with_port to use correct JSON format
- Updated tracker port extraction to support both config and state formats

Generated templates now provide complete, working tracker configuration
example instead of empty config, improving user experience and reducing
configuration errors.
- Changed create_test_environment_config() to return String instead of serde_json::Value
- Removed intermediate parsing step in generate_environment_config_with_port()
- Write JSON string directly to file without re-serialization
- Simplifies code flow and removes unnecessary conversions
- Created tests/e2e/ module directory
- Moved e2e_create_command.rs to tests/e2e/create_command.rs
- Moved e2e_destroy_command.rs to tests/e2e/destroy_command.rs
- Created tests/e2e_integration.rs as entry point for E2E tests
- Updated module imports to use super::super::support
- Converted file prefix pattern (e2e_*) into hierarchical module (e2e::*)
- All 8 E2E tests pass successfully
…ed function

- Made create_test_environment_config() public in src/testing/e2e/tasks/black_box/generate_config.rs
- Changed to use env!(CARGO_MANIFEST_DIR) for reliable project root detection
- Removed duplicate functions from tests/e2e/create_command.rs and tests/e2e/destroy_command.rs
- Updated both test files to import shared function from black_box module
- Compile-time constant ensures SSH keys are found regardless of working directory
- All 8 E2E tests pass with consolidated implementation
- Removed EnvironmentContext::with_working_dir() - not used anywhere
- Removed Environment::with_working_dir() - not used anywhere
- Kept EnvironmentContext::with_working_dir_and_tracker() - actively used by create command
- Kept InternalConfig::with_working_dir() - used by with_working_dir_and_tracker()
- Updated documentation to remove references to removed functions
- All 1395+ tests pass
- Removed update_environment_ssh_port() from generate_config.rs - not used anywhere
- Removed function export from black_box module
- Function was originally for bridge networking SSH port updates but never actually used
- All 1395+ tests pass
… E2E test configuration

- Add E2eConfigEnvironment::new() constructor for direct instantiation
- Add E2eConfigEnvironment::to_json_config() to generate JSON from struct values
- Implement Default trait for TrackerPorts (instead of custom method)
- Refactor build_e2e_test_config() to build struct in-memory without file I/O
- Add write_environment_config() helper to separate file writing from struct creation
- Move config file writing to run_deployer_workflow() where it's actually needed
- Rename generate_environment_config_with_port() to build_e2e_test_config() for clarity

Benefits:
- E2eConfigEnvironment is now the authoritative source, not the file
- Better testability: can work with config in-memory without file I/O
- Clearer data flow: build struct → use it → write file when needed
- Configuration values come from struct, not hardcoded defaults
- Function name clearly indicates E2E scope without misleading port parameter
BREAKING CHANGE: Domain tracker config types now use SocketAddr instead
of String for bind_address fields. This enforces type safety at compile
time and prevents invalid IP:PORT combinations.

- Updated UdpTrackerConfig, HttpTrackerConfig, HttpApiConfig to use
  SocketAddr with custom serde serialization for JSON compatibility
- Simplified port extraction logic (direct .port() instead of parsing)
- Updated all tests and DTOs to parse SocketAddr at the boundary
- Moved helper functions before tests to satisfy clippy lint rules
- Added Panics documentation for test helper functions
- Changed http_tracker_port: Option<u16> to http_tracker_ports: Vec<u16>
- Updated constructor signatures (new and with_deploy_dir)
- Modified validate_external_accessibility to iterate over all HTTP tracker ports
- Updated test command handler to extract all HTTP tracker ports instead of just first
- Updated E2E test task signature to accept Vec<u16>
- Added missing IpAddr import in run_run_validation.rs

This allows the validator to check external accessibility of all configured HTTP trackers
instead of only the first one.
- Added 7 unit tests for RunningServicesValidator
- Tests cover empty, single, and multiple HTTP tracker port scenarios
- Tests verify both new() and with_deploy_dir() constructors
- Created helper function create_test_ssh_config() for test setup
- All 1424 tests passing, including 5 new tests
- Updated to reflect validation of ALL configured HTTP trackers, not just first
- Added note about port 0 (dynamic assignment) not being supported
- Enhanced health check details to clarify multiple HTTP tracker validation
- Added link to ADR for port zero restriction
- Clarified that HTTP tracker checks are optional (warn only) while API check is required
- Added running_services.rs to remote actions validators list
- Documented that it validates all configured HTTP tracker instances
- Clarifies validator is correctly placed in infrastructure layer
… management warnings

- Updated docs/e2e-testing/manual-testing.md:
  * Replaced all fake command output with real output from test runs
  * Removed all references to non-existent 'deploy' command
  * Updated state machine to reflect actual workflow (provision → configure → release → run)
  * Added comprehensive warnings about manual state editing dangers
  * Added 'Checking Logs for Diagnosis' section with examples
  * Updated command durations with real timing data
  * Fixed all command sequences and recovery procedures

- Updated docs/user-guide/quick-start.md:
  * Replaced fake output with real command output
  * Updated complete workflow to show correct 5-step process
  * Changed Step 6 from 'test' to 'release' command
  * Added Step 7 'run' command (was missing)
  * Renumbered cleanup to Step 8
  * Updated Quick Reference with correct command sequence
  * Fixed all timing information with real test data

- Updated docs/e2e-testing/README.md:
  * Added manual-testing.md to documentation structure
  * Added link to manual testing guide in 'Learn More' section
  * Updated description to distinguish automated vs manual testing

- Updated AGENTS.md:
  * Added Manual E2E Testing section with link to manual-testing.md
  * Listed key topics covered in the manual testing guide

All markdown linting checks pass. Documentation now reflects actual system behavior with accurate command outputs, correct workflows, and proper warnings about state management.
@josecelano josecelano marked this pull request as ready for review December 11, 2025 15:35
@josecelano
Copy link
Copy Markdown
Member Author

ACK cff2e0d

@josecelano josecelano merged commit 3a31f9f into main Dec 11, 2025
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tracker Slice - Release and Run Commands

1 participant