Implement MySQL support for tracker deployment (#232)#235
Merged
Conversation
…ditional MySQL support Phase 1: Add MySQL service to docker-compose template Changes: - Convert docker-compose.yml to docker-compose.yml.tera (dynamic template) - Add conditional MySQL 8.0 service (renders only when database.driver == 'mysql') - Add MySQL healthcheck using mysqladmin ping - Add mysql_data volume for persistence - Create DockerComposeContext wrapper (supports SQLite and MySQL) - Create DockerComposeTemplate wrapper with validation - Create DockerComposeRenderer following renderer pattern - Update DockerComposeProjectGenerator to render dynamically - Add comprehensive unit tests for MySQL rendering (1460 tests passing) MySQL Service Features: - MySQL 8.0 with native password authentication - Healthcheck: mysqladmin ping (10s interval, 5s timeout, 5 retries, 30s start_period) - Environment variables: root password, database, user, password - Port mapping: 3306 - Named volume for data persistence - Network: backend_network Testing: - Unit tests verify MySQL service renders with all components - Unit tests verify SQLite doesn't render MySQL service - All 1460 unit tests passing - Pre-commit checks passing Limitation: - Phase 2 needed: Environment configuration doesn't support MySQL yet - Full E2E test with MySQL deferred to Phase 2 Related to #232
Phase 2: Extend environment configuration for MySQL support
Core Changes:
- Add MySQL variant to domain DatabaseConfig enum with host, port, database_name, username, password fields
- Add MySQL variant to application layer DatabaseSection enum
- Update conversion methods to handle MySQL configuration
- Regenerate JSON schema to include MySQL database driver
Environment Variable Injection:
- Extend EnvContext to include optional MySQL credentials
- Add new_with_mysql() constructor for MySQL mode
- Update .env.tera template to conditionally include MySQL variables
- Change docker-compose.yml.tera to use ${MYSQL_*} references instead of hardcoded values
- Update RenderDockerComposeTemplatesStep to create proper contexts based on database driver
Documentation:
- Add comprehensive comment block in docker-compose.yml.tera explaining environment variable injection pattern
- Create ADR docs/decisions/environment-variable-injection-in-docker-compose.md
- Update environment templates with MySQL example comments
Testing:
- Add 6 new unit tests for MySQL configuration (serialization, deserialization, conversion)
- Update docker-compose renderer tests to verify environment variable references
- All 1466 tests passing
Benefits:
- System administrators can modify .env values and restart services without regenerating templates
- Follows Docker Compose best practices for configuration management
- Separates template generation (deploy-time) from configuration (runtime)
- Enables secure credential rotation without redeployment
This commit completes Phase 3 of MySQL support by extending the tracker configuration template to support both SQLite and MySQL databases. Changes: - Extended TrackerContext with database driver and MySQL connection fields - Updated tracker.toml.tera to conditionally render database paths: * SQLite: file path format * MySQL: connection string format (mysql://user:pass@host:port/database) - Added comprehensive tests for both SQLite and MySQL configurations - Fixed long namespace paths following module organization conventions - Created ADR documenting database configuration structure decision The template now exposes structured database configuration (host, port, user, password) rather than pre-resolved connection strings, aligning with planned Torrust Tracker API improvements and hiding ORM implementation details. All 1469 unit tests passing. All pre-commit checks passing.
…ical service dependency fix
This commit completes the MySQL slice implementation for the release and run commands,
adding full MySQL database support to the Torrust Tracker deployment workflow.
CRITICAL FIX: Docker Compose Service Dependencies
==================================================
Added 'depends_on' with 'condition: service_healthy' to the tracker service in the
docker-compose template. This ensures the tracker container waits for the MySQL
healthcheck to pass before starting, preventing startup race conditions and
'Connection refused' errors.
Without this fix, the tracker would attempt to connect to MySQL immediately upon
container start, often before MySQL was ready to accept connections, resulting in:
ERROR r2d2: DriverError { Could not connect to address `mysql:3306': Connection refused }
The healthcheck configuration includes:
- Interval: 10s
- Timeout: 5s
- Retries: 5
- Start period: 30s
This increases total startup time by ~6 seconds (from ~19s to ~25s) but ensures
reliable service initialization.
Template Changes
================
1. Docker Compose Template (docker-compose.yml.tera):
- Added depends_on section to tracker service
- Conditional dependency on mysql service when database.driver == "mysql"
- Uses service_healthy condition for proper startup ordering
- Added standardized TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER env var
- Removed hardcoded 'sqlite3' value, now uses environment variable injection
2. Environment Variables Template (.env.tera):
- Standardized variable: TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER
- Replaced legacy DATABASE_DRIVER with standardized name
- Added explanatory comment about entrypoint script requirement
- Added section headers: "Tracker Service Configuration" and "MySQL Service Configuration"
- Changed conditional from {% if mysql_root_password %} to {% if database_driver == "mysql" %}
- Explicit database_driver check improves readability and correctness
Code Changes
============
3. EnvContext Structure (src/infrastructure/templating/docker_compose/template/wrappers/env/context.rs):
- Added database_driver field to EnvContext struct
- new() method sets database_driver to "sqlite3" (default)
- new_with_mysql() method sets database_driver to "mysql"
- Field controls template conditionals and variable rendering
Documentation
=============
4. Tracker Issues Documentation (docs/tracker-issues/):
- Created README.md explaining folder purpose and scope
- Created database-driver-double-specification.md documenting tracker entrypoint behavior
- Documents that tracker requires driver in both config file AND environment variable
- Explains root cause, impact, workaround, and recommended solution
- References tracker entrypoint script source code
5. Manual Testing Guide (docs/e2e-testing/manual-testing-mysql.md):
- Complete end-to-end manual testing workflow documented
- Updated with actual command outputs from successful test run
- Corrected VM file locations (changed from /srv/torrust to /opt/torrust)
- Added debugging section explaining data/logs/log.txt usage
- Documented depends_on fix and its importance
- Added CRITICAL warning about envs/ vs data/ directories distinction
- Deep verification sections for MySQL, tracker, and HTTP API
6. AGENTS.md Updates:
- Added Rule 1: CRITICAL explanation of envs/ vs data/ directories
- Most frequently violated rule - clearly distinguishes user input from application state
- Renumbered subsequent rules (2-17)
- Maintains complete AI assistant instructions
Environment Variable Standardization
=====================================
The tracker container's develop branch requires the standardized variable name:
TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER
This follows the TORRUST_TRACKER_CONFIG_OVERRIDE_* pattern used by the tracker's
entrypoint script to select the appropriate default configuration template.
The entrypoint script REQUIRES this variable to be set, otherwise the container
Testing
=======
All tests passing:
- Unit tests: 1469 passed
- Integration tests: 40 passed
- E2E infrastructure lifecycle: PASSED
- E2E deployment workflow: PASSED
- Manual MySQL E2E test: Complete and successful
- Pre-commit checks: ALL PASSED
Manual test verified:
- No startup race conditions
- No "Connection refused" errors
- MySQL tables created correctly
- Tracker HTTP API responding
- Both containers healthy
- Environment variables correctly injected
Implementation Details
======================
This implementation follows ADRs:
- Environment Variable Injection in Docker Compose
- Database Configuration Structure in Templates
- Template System Architecture
Backwards compatibility:
- SQLite remains the default driver (database_driver = "sqlite3")
- MySQL slice is opt-in via environment configuration
Related Issue: #232
…ssues folder Consolidates external tool issue documentation into a unified structure for better organization and discoverability. Changes: - Created new docs/external-issues/ directory as parent folder - Moved docs/github-actions-issues/ → docs/external-issues/github-actions/ - Moved docs/tracker-issues/ → docs/external-issues/tracker/ - Created docs/external-issues/README.md with overview and scope guidelines - Updated all relative path references in moved documentation - Updated docs/README.md to reflect new structure - Updated AGENTS.md navigation table - Updated quick navigation: "Fix a CI issue" → "Fix external tool issues" Benefits: - Clearer separation between internal and external issues - Easier to add new external tool issue categories - Consistent naming (external-issues instead of mixed naming) - Better navigation and discoverability Related to #232 (documentation organization during MySQL implementation)
- Create TrackerServiceConfig and MySqlServiceConfig types to mirror .env template structure
- Update .env.tera template to use nested variable syntax (e.g., {{ tracker.api_admin_token }}, {{ mysql.root_password }})
- Add backward-compatible getter methods on EnvContext for existing code
- Update all template tests to use new nested variable syntax
- Add comprehensive tests for context serialization and MySQL configuration
- Benefits: Better separation of concerns, more maintainable code, clearer template variable organization
…testing - Add X-Forwarded-For header usage for HTTP announces from outside VM - Update stats verification to use host-based API calls - Simplify MySQL verification to focus on connectivity and stats counters - Document that tracker uses in-memory storage by default - Add note about when torrents are persisted to MySQL (private mode vs whitelisted) - Remove expectations for database persistence in default public mode - Improve verification examples with actual expected output
Member
Author
|
ACK cb591c5 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR implements full MySQL database support for Torrust Tracker deployment, completing issue #232. The implementation extends the docker-compose stack with MySQL service and environment-driven configuration while maintaining backward compatibility with SQLite (default).
Critical Fix: Service Dependencies
Most Important Change: Added
depends_onwithcondition: service_healthyto the tracker service in docker-compose template. This prevents startup race conditions and "Connection refused" errors by ensuring MySQL is fully healthy before the tracker container starts.Impact:
Trade-off: Increases startup time by ~6 seconds (from ~19s to ~25s) but ensures reliable initialization.
Changes Summary
Templates
docker-compose.yml.tera:
depends_onwithcondition: service_healthyfor tracker service (CRITICAL)TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVERenvironment variablesqlite3value, now uses environment variable injection.env.tera:
TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER(replaces legacyDATABASE_DRIVER){% if mysql_root_password %}to{% if database_driver == "mysql" %}(explicit and correct)Code
src/infrastructure/templating/docker_compose/template/wrappers/env/context.rs):database_driverfield (controls template conditionals)new()method sets"sqlite3"(default)new_with_mysql()method sets"mysql"Documentation
Tracker Issues (
docs/tracker-issues/):database-driver-double-specification.mddocumenting tracker entrypoint behaviorManual Testing Guide (
docs/e2e-testing/manual-testing-mysql.md):/opt/torrust)data/logs/log.txtusage)depends_onfix importanceAGENTS.md:
envs/vsdata/directories distinctionEnvironment Variable Standardization
The tracker container's
developbranch requires:This follows the
TORRUST_TRACKER_CONFIG_OVERRIDE_*pattern used by the tracker's entrypoint script. The entrypoint requires this variable to be set, otherwise the container fails to start.Note: Legacy container versions may have used
DATABASE_DRIVER, but this is no longer supported in current images.Testing
All tests passing:
Manual test verified:
Backward Compatibility
database_driver = "sqlite3")Implementation Details
This implementation follows ADRs:
Closes #232