Overhaul stats: Add extendable-labeled metrics to the `tracker-core` package

[josecelano]

Depends on:

• Overhaul stats: Emit events from the torrent-repository package and collect metrics #1358
• Overhaul stats: Emit events from the tracker-core package #1436

As I have mentioned here I could not add extendable-labeled metrics for torrents and peers in this issue because I first need to add events to the tracker-core package.

The missing metrics are:

Torrents

tracker_core_torrents_total (counter)

#tracker_core_torrents_active_total (counter)

Peers

tracker_core_peers_completed_total (counter)
tracker_core_peers_seeders_total (counter)
tracker_core_peers_leechers_total (counter)

#tracker_core_peers_connected_peers_total (counter)
#tracker_core_peers_peers_seeding_only_total (counter)
#tracker_core_peers_peers_leeching_only_total (counter)
#tracker_core_peers_peers_seeding_and_leeching_total (counter)

NOTE: Commented metrics (starting with #) are the extra ones we want to include that are not included in the current /stats endpoint.

[josecelano]

I've been thinking about this issue. I have some doubts.

1. Refactor events to use labels (proposal 1)

Following recent changes and conventions, it may make sense to use a single metric with multiple labels.

Torrents

• Prefix: tracker_core_torrents_
• Type: Gauge
• Labels:
  • status: active, inactive
• Examples:
  • tracker_core_torrents_total{status="active"}
  • tracker_core_torrents_total{status="inactive"}

A torrent is considered inactive if:

• It has no peers.
• All peers are inactive.

Peers

• Prefix: tracker_core_peers_
• Type: Gauge
• Labels:
  • status: active, inactive
  • seeder: true or false
  • leecher: true or false
• Examples:
  • tracker_core_peers_total{status="active",seeder="true"} <- active seeders
  • tracker_core_peers_total{status="inactive",seeder="true"} <- inactive seeders
  • tracker_core_peers_total{status="active",leecher="true"} <- active leechers
  • tracker_core_peers_total{status="inactive",leecher="true"} <- inactive leechers
  • tracker_core_peers_total{status="active",seeder="true",leecher="true"} <- active seeder and leecher peers
  • tracker_core_peers_total{status="inactive",seeder="true",leecher="true"} <- inactive seeder and leecher peers
  • tracker_core_peers_total{seeder="true",leecher="false"} <- seeding-only peers
  • tracker_core_peers_total{seeder="false",leecher="true"} <- leeching-only peers
  • tracker_core_peers_total{seeder="true",leecher="true"} <- seeder and leecher peers

A peer is considered inactive if its last update timestamp was more than config.tracker_policy.max_peer_timeout seconds ago.

2. Refactor events to use labels (proposal 2)

Torrents

Prefix: tracker_core_torrents_
Type: Counter
Labels:

• peerless: true or false
  Examples:
• tracker_core_torrents_added_total <- Number of torrents added
• tracker_core_torrents_removed_total{peerless="true"} <- Number of peerless torrents removed because they don't have peers

Peers

Prefix: tracker_core_peers_
Type: Counter
Labels:

• inactive: true or false
  Examples:
• tracker_core_peers_added_total <- Number of peers that have been added
• tracker_core_peers_removed_total <- Number of peers that have been removed
• tracker_core_peers_removed_total{inactive="true"} <- Number of peers that have been removed because of inactivity
• tracker_core_peers_completed_total <- Number of peers that have completed

We could include both proposals.

3. Calculate metrics on demand vs keeping metrics updated

Currently, the metrics are calculated on demand. The torrent repository offers several methods for retrieving swarm statistics and aggregate swarm statistics.

On demand

Pros:

• It only affects performance when the API receives a request (currently every 15 seconds from the Prometheus service).

Cons:

• It does not scale well. With millions of torrents and hundreds of peers per torrent, it would be slow O(n*m)

Updated metrics

Pros:

• No time needed to calculate them in the API endpoint.

Cons:

• Overhead for each request to update metrics.

4. How to resolve new metrics for inactive torrents/peers

@devnakx proposed including metrics for inactive torrents and peers.

Torrents and peers are not marked as inactive. We only include the latest update time.

Inactive peers are those whose latest update time is older than the config.tracker_policy.max_peer_timeout value.

We have several options for calculating these metrics.

• Option 1. Iterate over all torrents and peers (on demand).
• Option 2. Add new events for when a torrent or peer is activated or deactivated. They are activated when their "update" time changes, and they go into inactivity when the latest "update" time is longer than config.tracker_policy.max_peer_timeout. We still need a cronjob for "deactivation".

I would use option 1. We can have two types of metrics: those that are maintained when new events are triggered and those that are calculated on demand. These could be of the second type.

5. Metrics accuracy

The current torrents and peers metrics:

{
  "torrents": 706240,
  "seeders": 187753,
  "completed": 1200202,
  "leechers": 306611
}

are calculated by iterating over the torrent repository:

let mut repository_seeders = 0;
let mut repository_leechers = 0;

for each torrent in torrent_repository {
   let mut torrent_peers = 0;
   let mut torrent_seeders = 0;
   let mut torrent_leechers = 0;

   for each peer in torrent.peer_list {
      torrent_peers++;

      if peer.is_seeder() {
         torrent_seeders++;
      }
   }

   torrent_leechers = torrent_peers - torrent_seeders

   repository_seeders += torrent_seeders
   repository_leechers += torrent_leechers
}

These provide duplicate peers in the global torrent repository metrics because some peers are the same seeding and/or leeching different torrents.

I think we should count unique peers (using the peer ID), and we will likely need a secondary data structure to store the list of peers. This can also be calculated on demand.

6. Should we merge torrent-repository and tracker-core packages?

There are some metrics that we can calculate within the torrent-repository package; however, for other metrics, such as those related to inactivity, we depend on configuration values that are only known by the tracker core.

I think the complexity could be reduced by merging these two packages. However, I won't do it for now, since we want to share the torrent-repository as a crate.

6. Plan

I will:

1. Add events to the torrent-repository package.
2. Add maintained metrics updated via events.
3. Add on-demand metrics (inactivity and accurate seeders and leechers counters).

cc @da2ce7

[josecelano]

Relates to point 3: #1456 (comment)

I want to add a counter for all BitTorrent client implementations that are sending the incorrect connection ID.

In the beginning, I wanted to use the labeled metrics (new API metrics endpoint), but the cardinality is too high. That's another example of metrics that maybe we can extract on demand (related to point 3).

Perhaps we should retain the old /stats endpoint for metrics collected from live data structures, and the new /metrics endpoint for metrics that are continuously updated in a separate statistics data structure.

cc @da2ce7

[josecelano]

Relates to point 5: #1437 (comment)

Regarding metrics accuracy (point 5 avobe), I think we should rename the metrics to represent better what they are counting:

✅ Unique peers (real peers):

• seeder — number of unique peers currently seeding any torrent.
• leecher — number of unique peers currently leeching any torrent.

📦 Torrent-peer relationships (per torrent count):

• seeded_torrents — total number of torrents being seeded across all peers (e.g., a peer seeding 3 torrents adds +3).
• leeching_torrents — total number of torrents being leeched across all peers.

The current values for seeder and leecher are the second ones.

cc @da2ce7

[da2ce7]

Here's how MVVM could be applied to metrics architecture:

1. Model Layer (Data Sources):

• Torrent/Peer Repositories (raw data stores)
• Event streams (peer added/removed events)
• Configuration (max_peer_timeout, policies)
• Current State: Direct database-like access to repository data

2. ViewModel Layer (Metrics Presentation):

• Metrics Calculator (transforms raw data → prometheus format)
• Label Manager (handles status, seeder/leecher labels)
• Caching Layer (maintains pre-computed metric states)
• State Management (active/inactive status computation)

3. View Layer (Metrics Exposure):

• Prometheus HTTP endpoint (/metrics)
• Text-based format serializer
• Exposition protocol handler

Key MVVM Advantages for Metrics:

1. Separation of Concerns:

• Model: Pure data storage/retrieval ("What we track")
• ViewModel: Metric business logic ("How we interpret")
• View: Presentation/formatting ("How we expose")

2. Performance Handling:

• Precompute metrics incrementally (on events)
• Cache expiration strategies for inactive peers
• Batch updates for high-frequency events

3. Label Management:

struct PeerLabels {
    status: Arc<AtomicBool>,
    seeder: AtomicBool,
    leecher: AtomicBool,
    last_updated: AtomicU64
}

impl LabelStrategy {
    fn update(&self, config: &Config) {
        // Atomic updates based on config thresholds
    }
}

4. Hybrid Approach:

• Core metrics (counters) updated in real-time
• Complex metrics (inactive peers) computed on-demand
• Scheduled background jobs for expensive aggregations

Your Case Considerations:

1. Proposal 1 vs 2 → Implement as different ViewModel strategies
2. Unique Peer Counting → ViewModel maintains de-duplication index
3. Package Structure:

• Model: torrent-repository (data structure focused)
• ViewModel: tracker-core (business logic aware)
• View: Separate exposition module

4. Performance Optimization:

metrics.on_peer_update(|peer| {
    vm.handle_peer_update(peer)  // Incremental update
});

metrics.on_scrape(|| {
    vm.precompute_complex_metrics()  // Expensive ops only during scrape
});

This would allow mixing event-driven updates (for basic counters) with on-demand computation (for inactive status checks) while keeping configuration-dependent logic contained in the ViewModel.

[da2ce7]

The ViewModel prepares metric states (not raw data formats)
It defines logical representations like:

enum MetricType {
    Counter { value: AtomicU64 },     // Proposal 2-style
    Gauge {
        labels: HashMap<LabelKey, LabelValue>,
        value: AtomicU64              // Proposal 1-style
    }
}

Key Responsibilities:

1. Type Conversion
   Transforms raw repository data into metric primitives:

   fn torrent_viewmodel(&self) -> MetricType {
       if use_proposal_1 {
           Gauge::with_labels(torrent.status) // Label-based gauge
       } else {
           Counter::new(torrent_count)         // Simple counter
       }
   }

2. Label Orchestration
   Handles combinatorial cases like seeder="true",leecher="true":

   fn peer_labels(peer: &Peer) -> Vec<(&'static str, String)> {
       vec![
           ("status", peer.status()),
           ("seeder", peer.is_seeder().to_string()),
           ("leecher", peer.is_leecher().to_string())
       ]
   }

3. Format Abstraction
   Maintains metric states agnostic to output format:

   trait MetricRenderer {
       fn render_counter(&self, name: &str, value: u64);
       fn render_gauge(&self, name: &str, labels: &[(&str, &str)], value: u64);
   }
   
   struct PrometheusRenderer; // Implements MetricRenderer
   struct JsonRenderer;       // Alternate implementation

Why This Matters for Your Design:

1. Multiple Proposal Support
   You could implement switchable strategies:

   trait MetricsStrategy {
       fn handle_peer_metric(&self, peer: PeerEvent);
       fn handle_torrent_metric(&self, torrent: TorrentEvent);
   }
   
   struct Proposal1Strategy; // Uses gauge+labels
   struct Proposal2Strategy; // Uses counters

2. Performance Isolation
   Heavy operations like inactive peer detection can be hidden behind async boundaries:

   impl ViewModel {
       async fn inactive_peers(&self) -> u64 {
           // Runs in background thread
           self.peer_repository.find_inactive().await.count()
       }
   }

3. Consistent Label Handling
   Prevents invalid combinations like seeder="false",leecher="false" at the type level:

   struct PeerState {
       seeder: bool,
       leecher: bool
   }
   
   impl PeerState {
       fn new() -> Self {
           // At least one must be true
           unimplemented!()
       }
   }

This architecture lets you:

• Support both proposals simultaneously via different strategies
• Mix real-time (event-driven) and computed (on-demand) metrics
• Maintain type safety for complex label combinations
• Decouple metric computation from Prometheus specifics

[josecelano]

Today, in our weekly meeting @da2ce7 and I decided to:

1. Use these new names for more accurate peer metrics:

✅ Unique peers (real peers):

• unique_seeders — number of unique peers currently seeding any torrent.
• unique_leechers — number of unique peers currently leeching any torrent.

📦 Torrent-peer relationships (per torrent count):

• torrent_seeders — total number of torrents being seeded across all peers (e.g., a peer seeding 3 torrents adds +3).
• torrent_leechers — total number of torrents being leeched across all peers.

2. Implement proposals 2 and 1

Start by adding "event" metrics:

• peers added
• peers removed
• seeder added
• seeder removed

And continue adding "aggregate" metrics like:

• current number of peers
• current number of seeders
• current number of leechers

The second group of metrics (proposal 1) could be calculated by the metrics consumer from the first group, but to make consumers' lives easier, we would also include the second type, which we think is even more useful than the first. In fact, we want to include the first to expose info that otherwise would not be accessible to external consumers of the metrics unless they poll very often.

3. Priorize event-driven metrics over on-demand (real-time) metrics

The tracker has two main responsibilities:

• Serve peer list to clients.
• Return internal statistics to third-party systems like indexes.

We consider the first to be the central tracker's responsibility and the other a secondary responsibility. Observability is essential, but whenever we have two choices, we always prefer the one that does not affect the performance of the tracker's main purpose.

On-demand metrics affect the performance more than the asynchronous metrics if you have many API metrics consumers.