docs: [#310] research database backup strategies

[josecelano]

Summary

Comprehensive research documentation for database backup strategies as part of Epic #309 (Add backup support).

This PR includes complete research for SQLite and MySQL backup strategies, backup tools evaluation, container backup architectures, a working proof-of-concept backup container with 58 bats-core unit tests, and a recommended solution (Maintenance Window Hybrid approach).

What's Included

Database Backup Strategies

SQLite

• Backup approaches: .backup command (Online Backup API), VACUUM INTO, file copy risks
• WAL mode analysis: Checkpointing behavior, persistence, pros/cons
• Backup verification and restore procedures: Integrity checks, recovery steps
• Torrust Live Demo analysis: Current implementation (unsafe cp), proposed improvements
• ⚠️ Critical Large Database Finding: SQLite .backup stalls at 10% after 16+ hours for 17GB database (~37 MB/hour effective rate). Maintenance window backup completes in 72 seconds.

MySQL

• Backup approaches: mysqldump, physical backups, binary log backups
• Container-specific considerations: Accessing MySQL in Docker containers
• Backup verification and restore procedures

Container Backup Architectures

• 5 patterns documented: Host Crontab, Centralized, Sidecar, Orchestrator, External Tool
• Comparison matrix with pros/cons
• Decision flowchart for pattern selection

Backup Tools Evaluation

• ✅ Restic: Recommended - mature, encrypted, deduplicated, Docker support
• ⚠️ Kopia: Alternative - newer, more features (GUI, ECC, server mode), less mature
• ❌ Rustic: Discarded - beta status, not production-ready
• Two-phase backup approach documented (DB dump → file backup)

Solution Comparison (NEW)

Four backup solutions evaluated with detailed trade-off analysis:

Solution	Best For	Complexity
Continuous Sidecar	Hot backups, simple setup	Low
Maintenance Window	Large DBs, complete consistency	Medium
External Scheduler	Multi-service environments	High
Native Database	WAL-enabled SQLite	Low

Recommended Solution: Maintenance Window Hybrid (95% container, 5% host script)

Maintenance Window Backup POC (Complete - NEW)

A working proof-of-concept with 58 bats-core unit tests supporting both MySQL and SQLite:

Feature	Status
MySQL backup with mysqldump	✅ Complete
SQLite backup with sqlite3	✅ Complete
Config file backup	✅ Complete
Retention policy (delete old backups)	✅ Complete
Single mode (run once, exit)	✅ Complete
Continuous mode (loop)	✅ Complete
Host orchestration script	✅ Complete
Crontab configuration	✅ Complete
58 unit tests	✅ All passing

POC Artifacts:

• Multi-stage Dockerfile with MySQL and SQLite support
• backup.sh script with modular functions
• maintenance-backup.sh host orchestration script
• Docker Compose examples for MySQL and SQLite
• Production and test crontab configurations
• Lessons learned document with implementation concerns

Key Findings

Finding	Details
SQLite Safe Backup	Use .backup command (Online Backup API) - safe during concurrent writes
SQLite Large DB Limitation	.backup impractical for DBs > 1GB due to locking overhead (~37 MB/hour)
Maintenance Window Backup	72 seconds for 17GB SQLite (vs ~17 days with .backup)
Disk I/O Capacity	445 MB/s proven - SQLite locking is bottleneck, not disk
MySQL Backup	mysqldump works reliably for containerized deployments
WAL Mode	Optional for safe backups, useful for read performance under high load
Recommended Tool	Restic - battle-tested, simple, Docker-native, sufficient features
Recommended Solution	Maintenance Window Hybrid - container + host crontab
Sidecar Pattern	Best for single-server deployments with few services

Lessons Learned (Implementation Concerns)

Key pain points discovered during POC that affect future implementation:

Pain Point	Severity	Notes
Template conditionals for DB type	Medium	Docker Compose env vars differ for MySQL vs SQLite
Path translation (host/container)	Medium	Multiple representations of same path
SSH agent key selection	Low	Use IdentitiesOnly=yes
Container exits in single mode	Low	Expected behavior, just surprising
Log rotation missing	Low	Easy to add, often forgotten
Backup verification missing	Medium	Important for production

Related Issues

• Closes Research database backup strategies #310 (Research Database Backup Strategies)
• Part of Epic Add backup support #309
• Created on torrust-demo repo:
  • Issue [EPIC] Coverage & Reporting EPIC - Test coverage checks, workflows, and docs #85: Use .backup instead of cp
  • Issue Add coverage check to pre-commit script #86: Evaluate WAL mode for high-traffic scenario

Checklist

Research Complete

• SQLite backup approaches documented
• SQLite large database findings (17GB test)
• MySQL backup approaches documented
• WAL mode analysis with checkpointing behavior
• Backup verification and restore procedures
• Torrust Live Demo analysis
• Container backup architectures (5 patterns)
• Backup tools evaluation (Restic, Kopia, Rustic)
• Solution comparison (4 approaches)
• Recommended solution documented

POC Complete

• Multi-stage Dockerfile with MySQL and SQLite support
• 58 bats-core unit tests (all passing)
• MySQL backup/restore validated
• SQLite backup/restore validated
• Config file backup
• Retention policy (delete expired backups)
• Single mode (run once, exit)
• Continuous mode (loop with interval)
• Host orchestration script
• Crontab configurations (production + test)
• Docker Compose examples (MySQL + SQLite)
• Lessons learned document
• Issue spec progress updated (all tasks complete)

Future Work (out of scope for this PR)

• Implement backup command in deployer
• Off-site transfer automation (S3, Backblaze B2)
• Backup encryption
• Backup verification command

Documentation Structure

docs/research/backup-strategies/
├── README.md                           # Overview and navigation
├── conclusions.md                      # Key findings and recommendations
├── requirements.md                     # Design preferences
├── architectures/
│   └── container-patterns.md           # 5 architecture patterns
├── databases/
│   ├── mysql/
│   │   ├── README.md
│   │   └── backup-approaches.md
│   └── sqlite/
│       ├── README.md
│       ├── backup-approaches.md
│       ├── large-database-backup.md    # Critical 17GB findings
│       └── torrust-live-demo/
│           ├── README.md
│           ├── current-implementation.md
│           └── proposed-improvements.md
├── tools/
│   ├── README.md                       # Tools overview
│   ├── restic.md                       # Detailed Restic evaluation
│   └── restic-vs-kopia.md              # Comparison document
└── solutions/
    ├── README.md                       # Solution comparison (NEW)
    ├── sidecar-container/              # Original sidecar POC
    └── maintenance-window/             # Recommended solution (NEW)
        ├── README.md                   # Architecture and workflow
        ├── implementation-recommendations.md  # Lessons learned
        └── artifacts/
            ├── backup-container/
            │   ├── Dockerfile
            │   ├── backup.sh
            │   └── backup_test.bats    # 58 tests
            ├── docker-compose-with-backup-mysql.yml
            ├── docker-compose-with-backup-sqlite.yml
            ├── maintenance-backup.sh
            ├── maintenance-backup.cron
            └── maintenance-backup-test.cron

[josecelano]

ACK 1168780