Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
848dbde
docs: [#310] research database backup strategies
josecelano Jan 28, 2026
17b15a8
docs: [#310] add preliminary conclusions for backup research
josecelano Jan 28, 2026
2fcf835
docs: [#310] add MySQL backup research and sidecar container solution
josecelano Jan 29, 2026
8b8aa51
docs: [#310] add PoC plan and answer backup requirements questions
josecelano Jan 29, 2026
1406df4
docs: [#310] complete Phase 1 of sidecar backup PoC
josecelano Jan 29, 2026
37fca8a
docs: [#310] reorganize PoC into structured folder with phase docs
josecelano Jan 29, 2026
7b661aa
docs: [#310] complete Phase 2 - minimal backup container
josecelano Jan 29, 2026
270ef7c
docs: [#310] complete Phase 3 - MySQL backup with mariadb-dump
josecelano Jan 29, 2026
bfbedd9
docs: [#310] reorganize artifacts with backup-container folder
josecelano Jan 29, 2026
550d382
docs: [#310] add backup integrity verification tests to Phase 3
josecelano Jan 29, 2026
fd8c19a
docs: [#310] implement Phase 4 - config files backup
josecelano Jan 29, 2026
745b102
docs: [#310] update Phase 4 documentation with implementation details
josecelano Jan 29, 2026
cb36c17
docs: [#310] add production considerations and fix non-root container
josecelano Jan 29, 2026
672fafc
docs: [#310] reference issue #313 for Ansible ownership fix
josecelano Jan 29, 2026
7076964
docs: [#310] mark Phase 4 as complete in progress table
josecelano Jan 29, 2026
4eb29fe
docs: [#310] update Phase 5 plan with maintenance approach
josecelano Jan 29, 2026
bf7e719
docs: [#310] implement Phase 5 - backup maintenance (packaging & rete…
josecelano Jan 29, 2026
e9b506c
refactor: [#310] add Rust-style documentation to backup script
josecelano Jan 30, 2026
e370b48
test: [#310] add unit tests for backup script with bats-core
josecelano Jan 30, 2026
d040367
docs: [#310] Phase 6 - Restore validation complete
josecelano Jan 30, 2026
7ea89cf
docs: [#310] add SQLite large database backup findings
josecelano Jan 30, 2026
7b3fb8a
docs: [#310] complete Phase 7 documentation and update conclusions
josecelano Jan 30, 2026
6cff3b6
docs: [#310] reorganize backup-strategies folder structure
josecelano Jan 30, 2026
717b4b8
docs: [#310] add alternative backup solutions for large databases
josecelano Jan 30, 2026
b9fd407
docs: [#310] add Torrust Demo database analysis for exclude-statistics
josecelano Jan 30, 2026
1c63945
docs: [#310] add maintenance-window solution artifacts and update def…
josecelano Jan 30, 2026
2d97bf1
docs: [#310] add unit tests and improve documentation clarity
josecelano Jan 30, 2026
c7cc2c5
docs: [#310] recommend maintenance-window as primary backup solution
josecelano Jan 30, 2026
c92c199
feat: [#310] add SQLite backup support to backup container
josecelano Jan 30, 2026
74a082d
docs: [#310] update issue spec progress - all tasks complete
josecelano Jan 30, 2026
1168780
docs: [#310] add lessons learned from backup POC implementation
josecelano Jan 30, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 68 additions & 68 deletions docs/issues/310-research-database-backup-strategies.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,33 +12,33 @@ Research and document backup strategies for SQLite databases, MySQL databases, a

### SQLite Backup

- [ ] Learn how to backup SQLite files safely while used in production (no locks, safe copying)
- [ ] Research tools and techniques for copying and compressing SQLite backups
- [ ] Investigate redundancy strategies for SQLite backups (cloud volumes, S3, backup services, snapshots)
- [ ] Document current Torrust Live Demo SQLite backup implementation
- [x] Learn how to backup SQLite files safely while used in production (no locks, safe copying)
- [x] Research tools and techniques for copying and compressing SQLite backups
- [x] Investigate redundancy strategies for SQLite backups (cloud volumes, S3, backup services, snapshots)
- [x] Document current Torrust Live Demo SQLite backup implementation

### MySQL Backup

- [ ] Research MySQL backup approaches for containerized deployments
- [ ] Learn about MySQL-specific backup tools (mysqldump, hot backup, volume snapshots)
- [ ] Investigate compression and redundancy strategies for MySQL backups
- [x] Research MySQL backup approaches for containerized deployments
- [x] Learn about MySQL-specific backup tools (mysqldump, hot backup, volume snapshots)
- [x] Investigate compression and redundancy strategies for MySQL backups

### Complete Storage Folder Backup

- [ ] Research approaches for backing up the entire deployment storage folder
- [ ] Learn about tools for full directory backups (tar, rsync, volume snapshots)
- [ ] Understand trade-offs between full storage backup and selective approaches
- [x] Research approaches for backing up the entire deployment storage folder
- [x] Learn about tools for full directory backups (tar, rsync, volume snapshots)
- [x] Understand trade-offs between full storage backup and selective approaches

### Selective Files Backup

- [ ] Identify which configuration files and directories need backup
- [ ] Research strategies for backing up specific files (docker-compose, tracker config, etc.)
- [ ] Learn about version control and organization for selective backups
- [x] Identify which configuration files and directories need backup
- [x] Research strategies for backing up specific files (docker-compose, tracker config, etc.)
- [x] Learn about version control and organization for selective backups

### General Research

- [ ] Explore different backup scope strategies and their trade-offs
- [ ] Document all findings in `docs/research/backup-strategies/` (to be created during research)
- [x] Explore different backup scope strategies and their trade-offs
- [x] Document all findings in `docs/research/backup-strategies/` (to be created during research)

## 🏗️ Architecture Requirements

Expand Down Expand Up @@ -221,87 +221,87 @@ Research different approaches to defining backup scope:

### Phase 1: SQLite Research (estimated 4-6 hours)

- [ ] Read SQLite backup documentation
- [ ] Research safe file copy approaches while database is in use
- [ ] Investigate SQLite locking mechanisms and WAL mode
- [ ] Research compression tools and techniques
- [ ] Learn about cloud volume attachment and snapshot strategies
- [ ] Study S3 and backup service integration options
- [ ] Analyze Torrust Live Demo backup script implementation
- [ ] Document all findings in `docs/research/backup-strategies/sqlite-backup-strategies.md` (create folder and file)
- [x] Read SQLite backup documentation
- [x] Research safe file copy approaches while database is in use
- [x] Investigate SQLite locking mechanisms and WAL mode
- [x] Research compression tools and techniques
- [x] Learn about cloud volume attachment and snapshot strategies
- [x] Study S3 and backup service integration options
- [x] Analyze Torrust Live Demo backup script implementation
- [x] Document all findings in `docs/research/backup-strategies/sqlite-backup-strategies.md` (create folder and file)

### Phase 2: MySQL Research (estimated 4-6 hours)

- [ ] Read MySQL backup documentation
- [ ] Research `mysqldump` usage and locking behavior
- [ ] Investigate physical backup tools (Percona XtraBackup)
- [ ] Learn about Docker volume backup strategies
- [ ] Research compression techniques for MySQL dumps
- [ ] Study cloud redundancy options for MySQL backups
- [ ] Test basic mysqldump in Docker container (optional hands-on)
- [ ] Document all findings in `docs/research/backup-strategies/mysql-backup-strategies.md`
- [x] Read MySQL backup documentation
- [x] Research `mysqldump` usage and locking behavior
- [x] Investigate physical backup tools (Percona XtraBackup)
- [x] Learn about Docker volume backup strategies
- [x] Research compression techniques for MySQL dumps
- [x] Study cloud redundancy options for MySQL backups
- [x] Test basic mysqldump in Docker container (optional hands-on)
- [x] Document all findings in `docs/research/backup-strategies/mysql-backup-strategies.md`

### Phase 3: Configuration Research (estimated 2-3 hours)

- [ ] Identify all configuration files and directories
- [ ] Research file copy and archive tools (`tar`, `rsync`)
- [ ] Learn about compression options and trade-offs
- [ ] Study configuration storage strategies
- [ ] Research version control for config backups
- [ ] Document all findings in `docs/research/backup-strategies/configuration-backup-strategies.md`
- [x] Identify all configuration files and directories
- [x] Research file copy and archive tools (`tar`, `rsync`)
- [x] Learn about compression options and trade-offs
- [x] Study configuration storage strategies
- [x] Research version control for config backups
- [x] Document all findings in `docs/research/backup-strategies/configuration-backup-strategies.md`

### Phase 4: Backup Scope Strategies (estimated 2-3 hours)

- [ ] Research full storage backup approaches
- [ ] Compare database-only backup patterns
- [ ] Study selective backup strategies
- [ ] Learn about layered backup approaches
- [ ] Document trade-offs for each strategy
- [ ] Document all findings in `docs/research/backup-strategies/backup-scope-strategies.md`
- [x] Research full storage backup approaches
- [x] Compare database-only backup patterns
- [x] Study selective backup strategies
- [x] Learn about layered backup approaches
- [x] Document trade-offs for each strategy
- [x] Document all findings in `docs/research/backup-strategies/backup-scope-strategies.md`

### Phase 5: Documentation Review (estimated 1 hour)

- [ ] Review all research documents for completeness
- [ ] Create README in research folder with overview
- [ ] Ensure all research questions are addressed
- [ ] Cross-reference with Torrust Live Demo implementation
- [ ] Run linters and ensure documentation quality
- [ ] Update issue with any follow-up questions or findings
- [x] Review all research documents for completeness
- [x] Create README in research folder with overview
- [x] Ensure all research questions are addressed
- [x] Cross-reference with Torrust Live Demo implementation
- [x] Run linters and ensure documentation quality
- [x] Update issue with any follow-up questions or findings

## Acceptance Criteria

> **Note for Contributors**: These criteria define what the PR reviewer will check. Use this as your pre-review checklist before submitting the PR to minimize back-and-forth iterations.

**Quality Checks**:

- [ ] Pre-commit checks pass: `./scripts/pre-commit.sh`
- [x] Pre-commit checks pass: `./scripts/pre-commit.sh`

**Research Documentation**:

- [ ] SQLite backup approaches documented (safe copying, compression, redundancy)
- [ ] MySQL backup approaches documented (tools, techniques, containerization)
- [ ] Configuration backup approaches documented
- [ ] Backup scope strategies compared
- [ ] Torrust Live Demo implementation analyzed and documented
- [ ] All research questions addressed with sufficient detail
- [ ] Cloud redundancy strategies documented (volumes, S3, snapshots)
- [ ] Compression techniques compared
- [x] SQLite backup approaches documented (safe copying, compression, redundancy)
- [x] MySQL backup approaches documented (tools, techniques, containerization)
- [x] Configuration backup approaches documented
- [x] Backup scope strategies compared
- [x] Torrust Live Demo implementation analyzed and documented
- [x] All research questions addressed with sufficient detail
- [x] Cloud redundancy strategies documented (volumes, S3, snapshots)
- [x] Compression techniques compared

**Research Completeness**:

- [ ] All research questions in specifications section answered
- [ ] Tools and techniques identified for each backup type
- [ ] Trade-offs documented for different approaches
- [ ] References to official documentation included
- [ ] Findings organized in `docs/research/backup-strategies/` folder
- [ ] README created in research folder with overview
- [x] All research questions in specifications section answered
- [x] Tools and techniques identified for each backup type
- [x] Trade-offs documented for different approaches
- [x] References to official documentation included
- [x] Findings organized in `docs/research/backup-strategies/` folder
- [x] README created in research folder with overview

**Documentation Quality**:

- [ ] Markdown linting passes (markdownlint)
- [ ] Spell checking passes (cspell)
- [ ] All links valid and properly formatted
- [ ] Code examples properly formatted with syntax highlighting (if any)
- [x] Markdown linting passes (markdownlint)
- [x] Spell checking passes (cspell)
- [x] All links valid and properly formatted
- [x] Code examples properly formatted with syntax highlighting (if any)

## Related Documentation

Expand Down
97 changes: 97 additions & 0 deletions docs/research/backup-strategies/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# Backup Strategies Research

**Issue**: [#310 - Research database backup strategies](https://github.com/torrust/torrust-tracker-deployer/issues/310)
**Parent Epic**: [#309 - Add backup support](https://github.com/torrust/torrust-tracker-deployer/issues/309)

## Overview

This folder contains research documentation for backup strategies in the context of Torrust Tracker deployments. The research covers different backup types, database-specific approaches, and general requirements.

> **📋 See [Conclusions](conclusions.md) for a summary of key findings and recommendations.**

## Backup Types

The research is organized around three backup types:

| Type | Description | Status |
| -------------------- | ----------------------------------------------- | ----------- |
| **Storage Backup** | Complete backup of entire storage (config + DB) | Not started |
| **Database Backup** | Database-specific backup using safe tools | In progress |
| **Selective Backup** | Partial storage backup (e.g., config only) | Not started |

See [requirements.md](requirements.md) for detailed explanation of each type.

## Documents

### General

| Document | Description |
| ------------------------------- | ---------------------------------------------------------------- |
| [Requirements](requirements.md) | Collected requirements, constraints, and backup type definitions |
| [Conclusions](conclusions.md) | Summary of key findings and recommendations |

### Database-Specific

| Folder | Description |
| -------------------------------------- | -------------------------------------------------------------------- |
| [databases/sqlite/](databases/sqlite/) | SQLite backup research (approaches, current implementation analysis) |
| [databases/mysql/](databases/mysql/) | MySQL backup research (mysqldump, hot backups, locking behavior) |

### Architectures

| Document | Description |
| -------------------------------------------------------------------------- | -------------------------------------------- |
| [architectures/container-patterns.md](architectures/container-patterns.md) | Container-based backup architecture patterns |

### Tools

| Document | Description |
| ---------------- | ---------------------------------- |
| [tools/](tools/) | Backup tool research (restic, etc) |

### Solutions

| Folder | Description |
| ------------------------ | ---------------------------------------------------- |
| [solutions/](solutions/) | Proposed backup solutions and architectural patterns |

**⭐ Recommended**: [Maintenance Window Pattern](solutions/maintenance-window/) -
A hybrid approach combining container-based backup with host-level orchestration.
The backup container runs once per day (triggered by crontab), not continuously.
This works for databases of any size and preserves container portability.

**Alternative**: [Sidecar Container Pattern](solutions/sidecar-container/) -
A dedicated backup container that runs continuously. Only practical for small
databases (< 1GB) due to SQLite locking issues under load.

## Research Status

### SQLite

- [x] Document backup approaches
- [x] Analyze Torrust Live Demo implementation
- [x] Investigate journal mode
- [ ] Test in containerized environment

### MySQL

- [x] Research mysqldump approaches
- [x] Research hot backup tools (Percona XtraBackup)
- [x] Document locking behavior (no lock needed for InnoDB)

### General

- [x] Document backup types
- [x] Collect requirements from discussions
- [ ] Research compression strategies
- [ ] Research retention policies
- [ ] Research restore procedures

## Key Decisions Captured

From research discussions:

1. **No data loss acceptable** - Safety is priority over simplicity
2. **User provides backup path** - Deployer is storage-location agnostic
3. **Keep complexity low** - No cloud provider APIs, focus on local backups
4. **Database-aware backups** - Simple `cp` not acceptable for databases
Loading
Loading