Skip to content

docs: [#234] Evaluate Pingoo as TLS proxy replacement for nginx+certbot #269

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
josecelano merged 6 commits into from
Jan 13, 2026
Merged

docs: [#234] Evaluate Pingoo as TLS proxy replacement for nginx+certbot #269

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3e47de1
docs: [#234] add Phase 1 environment preparation documentation
josecelano Jan 12, 2026
117376a
docs: [#234] complete Experiment 1 - Pingoo Hello World HTTPS test
josecelano Jan 12, 2026
d395020
docs: [#234] Add Experiment 2 results - Tracker API with Pingoo
josecelano Jan 12, 2026
f27e476
docs: [#234] Add Experiment 3 results - HTTP Tracker with Pingoo
josecelano Jan 12, 2026
289f0f2
docs: [#234] complete Experiment 4 - WebSocket limitation discovered
josecelano Jan 12, 2026
c93ce45
docs: [#234] close Pingoo evaluation - not adopting due to WebSocket …
josecelano Jan 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 128 additions & 0 deletions docs/research/pingoo-tls-proxy-evaluation/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
# Pingoo TLS Proxy Evaluation

**Issue**: [#234](https://github.com/torrust/torrust-tracker-deployer/issues/234)
**Specification**: [docs/issues/234-evaluate-pingoo-for-https-termination.md](../../issues/234-evaluate-pingoo-for-https-termination.md)
**Status**: ✅ CLOSED - Not Adopting
**Started**: 2026-01-12
**Completed**: 2026-01-13

## Overview

This research evaluated [Pingoo](https://pingoo.io/) as a potential replacement for nginx+certbot
for automatic HTTPS/TLS termination in Torrust Tracker deployments.

**Outcome**: Pingoo works excellently for the Tracker (API + HTTP Tracker) but does not
support WebSocket connections required for Grafana Live. We have decided **not to adopt
Pingoo** at this time and will instead evaluate **Caddy** as a simpler alternative that
supports both HTTP proxying and WebSocket.

## Test Environment

- **Server**: Hetzner ccx23, Ubuntu 24.04, nbg1 location
- **IP**: 46.224.206.37
- **Domain**: torrust-tracker.com (with subdomains)

### Subdomains

| Subdomain | Purpose | DNS Status |
| ----------------------------- | -------------------------- | ------------- |
| `test.torrust-tracker.com` | Experiment 1: Hello World | ✅ Propagated |
| `api.torrust-tracker.com` | Experiment 2: Tracker API | ✅ Propagated |
| `http1.torrust-tracker.com` | Experiment 3: HTTP Tracker | ✅ Propagated |
| `grafana.torrust-tracker.com` | Experiment 4: Grafana UI | ✅ Propagated |

## Phases

### Phase 1: Environment Preparation

See [phase-1-environment-preparation.md](phase-1-environment-preparation.md) for:

- DNS configuration and propagation verification
- Server accessibility checks
- Prerequisites for running experiments

**Status**: ✅ Complete

### Phase 2: Experiments

| Experiment | Document | Status |
| ---------------------- | ------------------------------------------------------------ | -------------------- |
| 1. Minimal Hello World | [experiment-1-hello-world.md](experiment-1-hello-world.md) | ✅ Complete |
| 2. Tracker API HTTPS | [experiment-2-tracker-api.md](experiment-2-tracker-api.md) | ✅ Complete |
| 3. HTTP Tracker HTTPS | [experiment-3-http-tracker.md](experiment-3-http-tracker.md) | ✅ Complete |
| 4. Grafana WebSocket | [experiment-4-grafana.md](experiment-4-grafana.md) | ⚠️ Partial (WS fail) |

## Key Questions to Answer

1. Does Pingoo automatically generate Let's Encrypt certificates? **✅ YES**
2. Does certificate renewal work without manual intervention? **⏳ Cannot test (90-day validity)**
3. Does Pingoo support WebSocket connections (needed for Grafana Live)? **❌ NO**
4. How does configuration complexity compare to nginx+certbot? **Much simpler (~10 lines vs ~50+)**
5. Are there any issues with TLS 1.3-only support? **✅ No issues detected**

## Findings Summary

### ✅ Successful Tests

- **Automatic certificate generation** - Pingoo obtained Let's Encrypt certs without manual steps
- **TLS 1.3 with post-quantum cryptography** - Uses X25519MLKEM768 key exchange
- **Minimal configuration** - Only ~10 lines of YAML needed
- **No email required** - Unlike certbot, no email setup needed
- **Tracker API proxying** - Health checks and API endpoints work perfectly
- **HTTP Tracker proxying** - BitTorrent announce/scrape work via HTTPS

### ❌ Failed Test

- **WebSocket support** - Pingoo strips the `Upgrade` header, breaking WebSocket connections
- Root cause: `Upgrade` header treated as hop-by-hop header in `http_proxy_service.rs`
- Impact: Grafana Live (real-time streaming) does not work
- Workaround: Use nginx for services requiring WebSocket

### ⏳ Pending Verification

- **Certificate renewal** - Cannot test yet (cert valid for 90 days)

## Final Decision

**Not adopting Pingoo** - We will evaluate Caddy instead.

### Why Not Adopt Pingoo?

While Pingoo works excellently for the Tracker, the WebSocket limitation means we would
need a **hybrid architecture** (Pingoo for Tracker + another proxy for Grafana). This
adds unnecessary complexity for our simple deployment setup:

1. **Two proxies is overkill** - For a simple Tracker + Grafana setup, maintaining two
different TLS proxies adds operational complexity without significant benefit.

2. **Caddy as alternative** - [Caddy](https://caddyserver.com/) offers similar benefits
to Pingoo (automatic HTTPS, simple config) while also supporting WebSocket natively.
It can handle both Tracker and Grafana with a single proxy.

3. **Maturity considerations** - Caddy is more mature and battle-tested than Pingoo.
While Pingoo has attractive features (post-quantum crypto, Rust implementation),
Caddy's stability may be more valuable for production deployments.

4. **Performance is not the bottleneck** - One concern about Caddy (written in Go) vs
Pingoo (written in Rust) is performance. However, based on running a tracker demo
for a couple of years, the proxy is unlikely to be the bottleneck. Most users prefer
the UDP tracker which doesn't use the proxy at all.

### Open Issue on Pingoo

We filed [pingooio/pingoo#23](https://github.com/pingooio/pingoo/issues/23) to confirm
the WebSocket limitation and discuss potential solutions. If Pingoo adds WebSocket
support in the future, it could be reconsidered.

### Next Steps

A new research issue will be opened to evaluate **Caddy** as an alternative to
nginx+certbot, following the same experimental approach used here.

See [conclusion.md](conclusion.md) for full rationale.

## Timeline

- **2026-01-12**: Research started, Experiments 1-4 completed
- **2026-01-12**: WebSocket limitation discovered, issue filed on Pingoo repo
- **2026-01-13**: Decision made not to adopt Pingoo, research closed
177 changes: 177 additions & 0 deletions docs/research/pingoo-tls-proxy-evaluation/conclusion.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
# Pingoo TLS Proxy Evaluation - Conclusion

**Status**: ✅ EVALUATION COMPLETE - NOT ADOPTING
**Last Updated**: 2026-01-13

## Final Decision

**Not adopting Pingoo.** We will evaluate Caddy as an alternative instead.

Pingoo provides excellent TLS termination for HTTP-based services but **does not support
WebSocket connections**, which are required for Grafana Live. Rather than adopting a
hybrid architecture (Pingoo + another proxy), we have decided to evaluate Caddy, which
can handle both the Tracker and Grafana with a single proxy.

## Why Not Adopt Pingoo?

### The Hybrid Architecture Problem

Initially, we considered using Pingoo for the Tracker and nginx/Caddy for Grafana:

```text
┌─────────────────────────────────────────────────────────────────┐
│ Public Internet │
└─────────────────────────────────────────────────────────────────┘
│ │
▼ ▼
┌──────────────────┐ ┌───────────────────┐
│ Pingoo │ │ nginx or Caddy │
│ (port 443) │ │ (port 3443) │
└────────┬─────────┘ └────────┬──────────┘
│ │
▼ ▼
┌──────────────────┐ ┌───────────────────┐
│ Tracker API │ │ Grafana │
│ HTTP Tracker │ │ (WebSocket) │
└──────────────────┘ └───────────────────┘
```

However, this approach has significant drawbacks for our use case:

1. **Operational complexity** - Two different TLS proxies means two different
configurations, two sets of certificates to manage, and two potential points of
failure.

2. **Overkill for simple setups** - For a typical Torrust deployment (Tracker +
Grafana), maintaining two proxies adds unnecessary complexity without clear benefits.

3. **Better alternatives exist** - Caddy offers similar benefits to Pingoo (automatic
HTTPS, simple configuration) while also supporting WebSocket natively.

### Caddy as Alternative

[Caddy](https://caddyserver.com/) is a more mature alternative that can replace
nginx+certbot entirely:

| Feature | Pingoo | Caddy |
| -------------------- | ----------------- | -------------- |
| Automatic HTTPS | ✅ Yes | ✅ Yes |
| Simple config | ✅ ~10 lines | ✅ ~5-10 lines |
| WebSocket support | ❌ No | ✅ Native |
| Post-quantum crypto | ✅ X25519MLKEM768 | ❌ No |
| TLS versions | 1.3 only | 1.2 and 1.3 |
| Language | Rust | Go |
| Maturity | Newer | Very mature |
| Single proxy for all | ❌ Needs hybrid | ✅ Yes |

### Performance Considerations

One concern about Caddy (Go) vs Pingoo (Rust) is performance. However:

- **Proxy is not the bottleneck** - Based on running a Torrust tracker demo for a couple
of years, the TLS proxy is unlikely to be the performance bottleneck.

- **UDP tracker dominates** - Most BitTorrent clients prefer the UDP tracker protocol,
which doesn't go through the HTTP/HTTPS proxy at all.

- **Stability over raw speed** - For a production deployment, Caddy's maturity and
stability may be more valuable than Pingoo's potential performance advantages.

### What We Lose by Not Using Pingoo

1. **Post-quantum cryptography** - Pingoo's X25519MLKEM768 key exchange provides
protection against future quantum computers. Caddy doesn't have this yet.

2. **TLS 1.3-only enforcement** - Pingoo only supports TLS 1.3, which simplifies
security configuration. Caddy supports both 1.2 and 1.3.

3. **Rust implementation** - Pingoo's Rust codebase may offer better memory safety
and performance characteristics.

These are trade-offs we accept in favor of a simpler, single-proxy architecture.

## Open Issue on Pingoo Repository

We filed [pingooio/pingoo#23](https://github.com/pingooio/pingoo/issues/23) to:

1. Confirm that WebSocket proxying is not currently supported
2. Discuss potential solutions or workarounds
3. Request consideration for adding WebSocket support

If Pingoo adds WebSocket support in the future, it could be reconsidered for Torrust
deployments.

## WebSocket Limitation - Root Cause

Pingoo's HTTP proxy explicitly removes the `Upgrade` header, which is required for
WebSocket protocol upgrades. From the source code:

```rust
// https://github.com/pingooio/pingoo/blob/main/pingoo/services/http_proxy_service.rs
const HOP_HEADERS: &[&str] = &[
"Connection",
// ... other headers ...
"Upgrade", // This breaks WebSocket!
];
```

This means any service requiring WebSocket connections cannot use Pingoo's `http_proxy`.
This is a fundamental limitation, not a configuration issue.

## Experiment Results Summary

| Experiment | Status | Result |
| ---------------------- | ----------- | -------------------------------------------- |
| 1. Hello World | ✅ Complete | SUCCESS - Certificate auto-generated |
| 2. Tracker API | ✅ Complete | SUCCESS - API endpoints work via HTTPS |
| 3. HTTP Tracker | ✅ Complete | SUCCESS - BitTorrent announce/scrape working |
| 4. Grafana (WebSocket) | ⚠️ Partial | HTTP works, WebSocket FAILS |

## Key Findings from Experiments

### Experiment 1: Hello World

- ✅ Automatic Let's Encrypt certificate generation works
- ✅ No email or manual steps required
- ✅ TLS 1.3 with post-quantum key exchange (X25519MLKEM768)
- ✅ ECDSA certificate from Let's Encrypt E8 intermediate
- ✅ Certificate stored with domain-named files for easy identification
- ✅ ACME account persisted for future renewals

### Experiment 2: Tracker API

- ✅ JSON API responses proxied correctly
- ✅ Health check endpoints work
- ✅ No issues with TLS 1.3 for API clients

### Experiment 3: HTTP Tracker

- ✅ BitTorrent `announce` endpoint works via HTTPS
- ✅ BitTorrent `scrape` endpoint works via HTTPS
- ✅ Binary bencoded responses handled correctly

### Experiment 4: Grafana (WebSocket)

- ✅ HTTP dashboard access works
- ✅ Login and navigation work
- ❌ **WebSocket fails** - `Upgrade` header stripped by Pingoo
- ❌ Grafana Live (real-time streaming) does not work

## Next Steps

1. ✅ ~~Complete Experiment 1 (Hello World)~~ - Certificate auto-generation verified
2. ✅ ~~Complete Experiment 2 (Tracker API)~~ - JSON API proxying verified
3. ✅ ~~Complete Experiment 3 (HTTP Tracker)~~ - BitTorrent protocol verified
4. ✅ ~~Complete Experiment 4 (Grafana)~~ - WebSocket limitation discovered
5. ✅ ~~File issue with Pingoo project~~ - [pingooio/pingoo#23](https://github.com/pingooio/pingoo/issues/23)
6. ✅ ~~Decision made~~ - Not adopting Pingoo, will evaluate Caddy instead
7. 🔲 Open new issue to evaluate Caddy as nginx+certbot replacement
8. 🔲 Run Caddy experiments following similar methodology

## References

- [Pingoo Documentation](https://pingoo.io/docs)
- [Pingoo WebSocket Issue](https://github.com/pingooio/pingoo/issues/23) - Our issue asking about WebSocket support
- [Issue #234 - Evaluate Pingoo](https://github.com/torrust/torrust-tracker-deployer/issues/234)
- [Issue Specification](../../issues/234-evaluate-pingoo-for-https-termination.md)
- [Caddy Server](https://caddyserver.com/) - Alternative to evaluate next
Loading
Loading