Skip to content

feat: [#121] Skip slow tests for Copilot agent with enhanced messaging#158

Merged
josecelano merged 2 commits intomainfrom
121-skip-e2e-tests-for-copilot-agent
Nov 7, 2025
Merged

feat: [#121] Skip slow tests for Copilot agent with enhanced messaging#158
josecelano merged 2 commits intomainfrom
121-skip-e2e-tests-for-copilot-agent

Conversation

@josecelano
Copy link
Copy Markdown
Member

Summary

This PR addresses issue #121 by implementing a solution to prevent GitHub Copilot coding agent from timing out during pre-commit checks.

Changes

1. Environment Variable Renaming

Changed: TORRUST_TD_SKIP_E2E_TESTSTORRUST_TD_SKIP_SLOW_TESTS

Rationale:

  • Action-based naming (SKIP_SLOW_TESTS) is clearer than condition-based (RUNNING_IN_AGENT_ENV)
  • More descriptive of what actually happens
  • Not tied to a specific environment (more flexible)
  • Could be used by developers locally for faster pre-commit checks

2. Expanded Test Skipping

Now skips in fast mode:

  • E2E provision and destroy tests (~44s)
  • E2E configuration tests (~48s)
  • Code coverage check (~1m 29s)
  • Total time saved: ~3 minutes (54.8% reduction)

Still runs in fast mode:

  • cargo machete (~0.08s)
  • All linters (~19s)
  • Unit tests - 1529 tests (~1m 16s)
  • cargo doc (~44s)

3. Enhanced Pre-commit Messaging

When TORRUST_TD_SKIP_SLOW_TESTS=true, the script now displays:

⚠️  Running in fast mode (skipping slow tests)

The following tests are SKIPPED to stay within the 5-minute timeout limit:
  • E2E provision and destroy tests (~44 seconds)
  • E2E configuration tests (~48 seconds)
  • Code coverage check (~1 minute 29 seconds)

💡 These tests will run automatically in CI after PR creation.

If you want to run them manually before committing, use these commands:
  cargo run --bin e2e-provision-and-destroy-tests  # ~44s
  cargo run --bin e2e-config-tests                 # ~48s
  cargo cov-check                                  # ~1m 29s

Fast mode execution time: ~3 minutes 48 seconds

Benefits:

  • Clear list of what was skipped
  • Exact commands to run each test separately (all < 5 min)
  • AI agents can choose to run them individually if time permits
  • Informative guidance about CI safety net

4. Comprehensive Documentation

Added detailed timing tables to docs/contributing/copilot-agent/pre-commit-config.md:

Individual Task Timings:

Task Time % of Total Skipped?
cargo machete 0.08s 0.04% ❌ No
All linters 18.75s 5.7% ❌ No
Unit tests 1m 16s 22.9% ❌ No
cargo doc 44s 13.4% ❌ No
E2E provision 44s 13.4% ✅ Yes
E2E config 48s 14.4% ✅ Yes
Coverage 1m 29s 26.9% ✅ Yes

Unit Tests Breakdown:

  • Shows detailed breakdown of all 1529 tests across 8 test suites
  • Explains why unit tests are NOT skipped despite taking 1m 16s

5. Documentation Organization

  • Created docs/contributing/copilot-agent/ folder
  • Moved copilot-agent-firewall.mdcopilot-agent/firewall.md
  • Added copilot-agent/README.md index
  • Added copilot-agent/pre-commit-config.md comprehensive setup guide
  • Updated all cross-references

Performance Results

  • Full mode: ~5m 30s (exceeds Copilot's ~5-6 min timeout ⚠️)
  • Fast mode: ~3m 48s (31% faster, ~2 min safety margin ✅)
  • Unit tests: 1m 16s (kept in fast mode for quality assurance)

Testing

✅ Fast mode tested: TORRUST_TD_SKIP_SLOW_TESTS=true ./scripts/pre-commit.sh
✅ Full mode tested: ./scripts/pre-commit.sh
✅ All markdown linting passes
✅ Messaging displays correctly with timing information

Next Steps for Repository Admins

Important: Update GitHub environment variable configuration:

  1. Go to Settings > Environments > copilot
  2. Change variable name: TORRUST_TD_SKIP_E2E_TESTSTORRUST_TD_SKIP_SLOW_TESTS
  3. Keep value as true

CI Safety Net

Even though slow tests are skipped in pre-commit for Copilot agent, they still run:

  • In GitHub Actions workflows on PR creation
  • In the full CI pipeline before merging

This ensures no regressions slip through while keeping Copilot agent functional.

Related

- Rename environment variable: TORRUST_TD_SKIP_E2E_TESTS → TORRUST_TD_SKIP_SLOW_TESTS
  * Action-based naming is clearer and more flexible than condition-based
  * Can be used by developers locally for faster pre-commit checks

- Expand what gets skipped in fast mode:
  * E2E provision and destroy tests (~44s)
  * E2E configuration tests (~48s)
  * Code coverage check (~1m 29s)
  * Total time saved: ~3 minutes (54.8% reduction)

- Enhanced pre-commit messaging when slow tests are skipped:
  * Clear list of skipped tests with timing information
  * Exact commands to run each test separately (all < 5 min)
  * AI agents can choose to run them individually if time permits
  * Informative guidance about CI safety net

- Add comprehensive timing tables to documentation:
  * Individual pre-commit task timings (7 tasks breakdown)
  * Detailed cargo test breakdown (8 test suites, 1529 tests)
  * Performance analysis showing what to skip and why

- Reorganize Copilot agent documentation:
  * Created docs/contributing/copilot-agent/ folder
  * Moved firewall.md to new location
  * Added README.md index for all Copilot docs
  * Added pre-commit-config.md with comprehensive setup guide

Performance Results:
- Full mode: ~5m 30s (exceeds Copilot timeout)
- Fast mode: ~3m 48s (31% faster, ~2 min safety margin)
- Unit tests (1m 16s) kept in fast mode for quality assurance

Related:
- Issue #121
- Community Discussion: https://github.com/orgs/community/discussions/178998
@josecelano josecelano self-assigned this Nov 7, 2025
- Created detailed guide on condition-based vs action-based naming
- Includes decision frameworks, real-world examples, and best practices
- Explains rationale for choosing TORRUST_TD_SKIP_SLOW_TESTS (action-based)
- Added essential rule to Copilot instructions for creating new env vars
- References both naming guide and prefix ADR for complete guidance
- Fixed spelling: added 'namespacing' to project dictionary
@josecelano
Copy link
Copy Markdown
Member Author

ACK 48c3293

@josecelano josecelano merged commit ea36005 into main Nov 7, 2025
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Install Git Pre-Commit Hooks for Copilot Agent

1 participant