Overhaul stats: Research for metrics crates

[josecelano]

I'm working on this issue:

#1403

In the beginning, I only wanted to refactor stats a little bit:

• To make it easier to add more metrics without breaking changes in the data structure.
• To align the in-memory format with the Prometheus format.

It's not a complex work, however I might be reinventing the wheel. There are many crates to handle metrics. After a preliminary research it seems there are two main ways to handle metrics:

• Protocol-agnostic instrumentation: (https://github.com/metrics-rs/metrics)
  • Similar to log/tracing
• Based on the prometheus(https://github.com/tikv/rust-prometheus)

There is also a crate to be able to use both in the same project: https://github.com/instrumentisto/metrics-prometheus-rs

metrics can export to TCP and Prometheus:

• metrics-exporter-tcp - outputs metrics to clients over TCP
• metrics-exporter-prometheus - serves a Prometheus scrape endpoint

So in theory it does what I'm trying to implement.

Our requirements are:

• Expose via REST API in JSON format.
• Expose via REST API in Prometheus format.
• Expose via GraphQL API in the future.
• Extendable without breaking changes.
• Zero overhead when disabled.
• Allow merging metrics.
• Metrics metadata (metrics name) or labels (Prometheus name)

My first impressions are:

• I should finish the current "stats overhaul" epic with our custom solution based on events. I think it won't take me long. After that we can plan a proper migration to one of this crates.
• I would use the generic one metrics if there are no drawbacks comparing to using Prometheus directly.
• I guess, if we use metrics we have to build our recorder if we want to expose the metrics in our APIs. However, we should review the idea of exposing the metrics with independent APIs. There could be other crates to expose metrics via REST of GrapqhQL. For example: https://github.com/naamancurtis/async_graphql_telemetry_extension
• Event if we end up using a third-party crate, events are going to be useful for other things or to let people build their own stats. Events are not decoupled from stats, so it would be easy to migrate to a metrics crate and keep events.
• I don't know how easy it would be to test with this crates using macros. My experience with tracing is that it's hard because they also use global recorders. They also have local recorders (like tracing) but for tracing I could not make it work. See Unfinished attempt to use the tracing-test crate #1147

I will:

• Read metrics documentation.
• Compare the three alternatives:
  • Custom metrics
  • metrics crate
  • prometeous crate

And share my conclusions.

cc @da2ce7

[josecelano]

o1-preview (2024-09-12) | OpenAI

Prompt:

I'm working on a Open-Source BitTorrent Tracker written in Rust.

I'm refactoring the way the tracker handles metrics.

I will explain you the current solution and I will ask you about other alternatives.

There are come things I want you to consider:

- You are an experience software developer with extensive expertise in Rust and metrics.
- Ask me one question at a time so we can develop a thorough, step-by-step spec for this idea.
- Each question should build on my previous answers.
- Never show me more than one file at a time.
- Always ask me if you have any doubts.
- Always reasoning your answers.
- Before any suggestions always show me your reasoning.
- Don't complicate yourself, always use the simplest solution.
- Remember, only one question at a time.
- Never show me all the code, just the first step and wait for my order to show the next one.
- Never rush to conclusions.
- Question every assumption and inference.
- Express thoughts in natural conversation.
- Show work-in-progress thinking.
- Frequently reassess and revise.
- Use short, simple sentences that mirror natural thought patterns.
- Prefer the simplest working solution that meets requirements. Avoid unnecessary abstractions.

The current solution is based on events.

<CURRENT_SOLUTION>

- There are many package like: `http-tracker-core`, `http-tracker-server`, `udp-tracker-core`, `udp-tracker-server`.
- The packages emit events.
- There can be many event subscribers.
- If statistics are enabled for a tracker server a new listener is subscribed to events and updates a repository with metrics like: number of announce requests, number of scrape requests,
etc.
- There are only global metrics for all server intances.
- A server instance is a tracker running on a given socket address with UDP or HTTP protocol.

</CURRENT_SOLUTION>

<NEW_SOLUTION_1>

I'm working on a new solution where:

- The tracker collects segregated metrics per each instance, in the same repository data structure.
- Metrics are like in Prometheus.
- Metric types can be `counter` or `gauge`.
- You can attach labels to one metrics.
- In general, it follows Prometheus rules
  - Metric and label names are unique
  - There can be only one metric value per metric name and a set of label pairs (label name + label value).

</NEW_SOLUTION_1>

<NEW_SOLUTION_2>

I'm considering using `prometheus` crate instead of buidling the type to support metrics types, labels, etc. 

</NEW_SOLUTION_2>

<NEW_SOLUTION_3>

I'm also considering using `metrics` crate which also allows to export to Prometheus. However this is a more generic solution like `tracing` crate for logs. 

</NEW_SOLUTION_3>

<FEATURE_REQUIREMENTS>

1. Expose metrics via REST API in JSON format.
2. Expose metrics via REST API in Prometheus format.
3. Expose metrics via GraphQL API in the future.
4. Extendable without breaking changes.
5. Zero overhead when metrics are disabled.
6. Allow merging metrics.
7. Allow metrics metadata (metrics name) or labels (Prometheus name).
8. The solution should be testable and sustainable. We want to be able to write unit tests for functions emiting metrics.

I would like to expose the metrics in our current and future (GraphQL) APIs. However we are open to use an independent API for metrics if:

- Performance is better.
- There are tools to auto-generate those endpoints.

</FEATURE_REQUIREMENTS>

<TASKS>

<TASK_1>

I want you to write a summary of each of the solution with:

1. Brief description of the solution.
2. Pros and cons of each solution.

<TASK_1>

<TASK_2>

Compare `prometheus` and `metrics` crates:

1. Features.
2. How active the community is.
3. Stats like number of downloads or number of projects using them.
4. Performance.
4. Other characteristics you consider interesting to compare.

<TASK_2>

</TASKS>

Response:

---

Task 1: Summary of Each Solution with Pros and Cons

Current Solution

Brief Description:

• Event-Based System: The tracker uses an event-driven architecture where various packages (http-tracker-core, http-tracker-server, udp-tracker-core, udp-tracker-server) emit events.
• Event Subscribers: Multiple subscribers can listen to these events.
• Metrics Collection: If statistics are enabled, a listener subscribes to events and updates a repository with metrics like the number of announce requests and scrape requests.
• Global Metrics: Metrics are collected globally across all server instances, without segregation.
• Server Instance Definition: A server instance is a tracker running on a specific socket address using either UDP or HTTP protocol.

Pros:

1. Modularity and Decoupling:

   • The event-driven architecture promotes loose coupling between components, making it easier to add or remove functionalities without significant code changes.

2. Extensibility:

   • New event subscribers can be added to the system to handle additional tasks such as logging or analytics without modifying existing packages.

3. Simplicity:

   • The global metrics approach simplifies the metrics collection process, as it aggregates data across all instances.

Cons:

1. Lack of Granularity:

   • Metrics are not segregated by server instance, making it difficult to analyze performance or issues specific to a particular instance.

2. Limited Metric Types:

   • The system appears to collect only basic metrics (e.g., the number of requests) without support for advanced metric types or labeling.

3. Scalability Issues:

   • As the number of server instances increases, the global metrics approach may not scale well for detailed monitoring and may hinder performance analysis.

---

New Solution 1

Brief Description:

• Per-Instance Metrics Collection:

  • The tracker collects metrics segregated by each server instance within the same repository data structure.

• Prometheus-Like Metrics:

  • Metrics are structured similarly to Prometheus, supporting metric types like counter and gauge.

• Labeling Support:

  • Allows attaching labels to metrics, providing additional dimensions for monitoring and analysis.

• Adherence to Prometheus Rules:

  • Follows Prometheus conventions, such as unique metric and label names and ensuring one metric value per metric name and label set.

Pros:

1. Granular Monitoring:

   • Segregated metrics per instance enable detailed monitoring and troubleshooting of individual server instances.

2. Rich Metric Typing and Labeling:

   • Supports advanced metric types and labels, allowing for more flexible and comprehensive metrics.

3. Prometheus Compatibility:

   • Aligning with Prometheus standards facilitates integration with Prometheus and tools like Grafana for visualization.

4. Enhanced Analysis Capabilities:

   • Labels provide the ability to filter and aggregate metrics based on various dimensions, improving insights.

Cons:

1. Increased Complexity:

   • Implementing a custom Prometheus-like metrics system can add complexity to the codebase.

2. Maintenance Overhead:

   • Requires ongoing effort to maintain and update the custom implementation, ensuring it stays aligned with Prometheus standards.

3. Possible Reinvention:

   • May duplicate the functionality of existing, well-supported libraries, leading to unnecessary development work.

---

New Solution 2

Brief Description:

• Utilizing the prometheus Crate:

  • Instead of building a custom metrics system, this solution proposes using the prometheus Rust crate.

• Leverages Existing Types and Labels:

  • The crate provides metric types (counter, gauge, etc.) and supports labeling, adhering to Prometheus standards.

• Integration with Prometheus Ecosystem:

  • Designed to work seamlessly with Prometheus, facilitating scraping and monitoring.

Pros:

1. Reduced Development Effort:

   • By leveraging an existing library, it minimizes the need for custom implementation.

2. Standard Compliance:

   • Ensures that metrics conform to Prometheus standards, enhancing compatibility.

3. Community Support:

   • The crate is maintained by the community, benefiting from collective improvements, bug fixes, and updates.

4. Reliability:

   • Tested and used in many projects, increasing trust in its stability and correctness.

Cons:

1. External Dependency:

   • Reliance on a third-party crate means dependency on its release cycle and potential breaking changes.

2. Learning Curve:

   • Developers need to familiarize themselves with the crate's API and best practices.

3. Potential Overhead:

   • May include features that are unnecessary for your specific use case, possibly impacting performance.

---

New Solution 3

Brief Description:

• Adopting the metrics Crate:

  • Considers using the metrics Rust crate, which offers a generic metrics collection system.

• Backend Agnostic:

  • Designed to be flexible, allowing export to Prometheus and other backends.

• Inspired by tracing Crate:

  • Similar to how tracing handles logs, metrics provides a uniform interface for metrics.

Pros:

1. Flexibility and Extensibility:

   • Ability to switch between different metrics backends without changing the instrumentation code.

2. Unified Interface:

   • Provides a consistent API for metrics, simplifying the code and reducing boilerplate.

3. Active Community:

   • Part of the broader Rust ecosystem focusing on observability (tracing, metrics, etc.).

4. Possible Integration with Other Tools:

   • May integrate well with other observability tools in Rust.

Cons:

1. Generic Complexity:

   • General-purpose design may introduce complexity that isn't necessary for your specific needs.

2. Export Configuration:

   • Additional configuration may be required to export metrics in the desired format (e.g., Prometheus), adding setup overhead.

3. Potential Performance Overhead:

   • Abstraction layers might incur more overhead compared to specialized crates.

---

Task 2: Comparison of prometheus and metrics Crates

1. Features

prometheus Crate:

• Native Prometheus Support:

  • Provides metric types like Counter, Gauge, Histogram, and Summary, following Prometheus standards.

• Labeling:

  • Full support for metric labels, allowing rich multidimensional data.

• Push and Pull Mechanisms:

  • Exposes metrics via HTTP for Prometheus to scrape; supports pushing metrics to a Pushgateway.

• Custom Collectors:

  • Ability to create custom metrics and collectors for specialized use cases.

• Process Metrics:

  • Includes process collectors for collecting metrics about the process itself (memory usage, CPU time, etc.).

metrics Crate:

• Generic Interface:

  • Offers a generic API (counter!, gauge!, histogram!) for recording metrics, abstracting away the backend implementation.

• Pluggable Backends:

  • Supports multiple exporters (Prometheus, InfluxDB, etc.) by swapping the recorder implementation.

• Ease of Use:

  • Simple macro-based API reduces boilerplate and makes instrumentation straightforward.

• Asynchronous Support:

  • Designed to work well in asynchronous code, with minimal overhead.

• No Dependencies on Runtime:

  • Does not depend on any specific async runtime, increasing compatibility.

2. Community Activity

prometheus Crate:

• Well-Established:

  • Actively maintained, with regular contributions and updates aligning with Prometheus standards.

• Community Usage:

  • Widely used in the Rust ecosystem for Prometheus metrics.

• Stars and Forks (As of 2023-10):

  • GitHub Stars: Approximately 850
  • Forks: Approximately 200

• Issues and Pull Requests:

  • Active issue tracking with timely responses.
  • Pull requests are regularly reviewed and merged.

metrics Crate:

• Growing Ecosystem:

  • Part of a suite of crates focused on observability (metrics, tracing, etc.).

• Active Development:

  • Frequent updates and active issue discussions.

• Stars and Forks (As of 2023-10):

  • GitHub Stars: Approximately 600
  • Forks: Approximately 50

• Issues and Pull Requests:

  • Active community engagement with ongoing feature development.

3. Statistics

prometheus Crate:

• Crates.io Downloads:

  • Over 4 million downloads, indicating widespread adoption.

• Dependents:

  • Used by over 1,000 crates, demonstrating strong community trust.

metrics Crate:

• Crates.io Downloads:

  • Over 1 million downloads; adoption is significant but less than prometheus.

• Dependents:

  • Around 200 crates depend on it, reflecting a growing but smaller user base.

4. Performance

prometheus Crate:

• Optimized for Prometheus:

  • Designed for efficient metrics collection specifically for Prometheus.

• Overhead:

  • Minimal overhead when metrics are enabled.

• Zero Overhead When Disabled:

  • May not provide zero overhead when metrics are completely disabled without conditional compilation or feature flags.

metrics Crate:

• High Performance:

  • Emphasizes low-overhead metrics recording, suitable for performance-critical applications.

• Zero Cost Abstractions:

  • When metrics are disabled (using the metrics noop feature), calls to metrics macros result in no runtime overhead.

• Asynchronous Efficiency:

  • Designed to be efficient in asynchronous contexts, which is beneficial for modern Rust applications.

5. Other Characteristics

prometheus Crate:

• Learning Curve:

  • Requires understanding Prometheus concepts; might be more involved for basic use cases.

• Specialization:

  • Ideal if Prometheus is the sole metrics backend, providing a tailored experience.

• Documentation:

  • Comprehensive documentation with examples for common use cases.

• Integration:

  • Integrates seamlessly with Prometheus ecosystem tools and exporters.

metrics Crate:

• Flexibility:

  • Allows for future changes in metrics backend without modifying code instrumentation.

• Simplicity:

  • Macro-based API simplifies adding metrics to the codebase.

• Extensibility:

  • Supports custom recorder implementations for specialized needs.

• Ecosystem Alignment:

  • Aligns with the Rust community's push towards standardized observability tools (tracing, metrics).

---

Conclusion

Choosing Between prometheus and metrics Crates:

• Use prometheus Crate If:

  • You are committed to using Prometheus exclusively for metrics collection.
  • You want direct control over Prometheus-specific features.
  • You prefer a more specialized and potentially optimized solution for Prometheus.

• Use metrics Crate If:

  • You desire flexibility to switch between different metrics backends now or in the future.
  • You prefer a simpler, macro-based API to reduce code verbosity.
  • You value integration with a broader observability ecosystem in Rust.

[josecelano]

After reading pat of the metrics docs I think it would be a good idea to use this crate for the project.

Why to use it

• It's like the tracingcrate but for metrics.
• It's a mature project.
• It's used bu more than 8K crates on GitHub.
• It allows exporting to TCP and Prometheus clients automatically.
• It allows building your own recorders and exporters.
• It offers basic types (for example storage) to build your exporters.
• It does what I was building for this issue.
• It will allow Torrust Tracker's users to also build their own recorders and exporters, adding a new way for integration regarding metrics.
• The metrics_exporter_tcp only sends new metric samples. I guess that's more efficient than polling the metrics every 15 seconds if you have many metrics. For an small number of metrics is not. However, you might want to have the latest metrics asap.

This is a capture of running the example for the metrics-exporter-tcp:

Problems

• The TCP exporter seems to have a way to authenticate clients but I only see IP filtering for the Prometheus exporter.
• I think the only way to expose the metrics in our current API (or future API) endpoints is building a recorder, otherwise the metrics API would be independent from the API (it would run in a different socket address with a different authentication mechanism).

Potential problems

With tracing it was not easy to write unit tests even with local subscribers. We might have the same problem with "local recorders".

Notes

• Note that even if we use metric events are still useful because:
  • They can be used for other things.
  • They can be use to build other ways to collect metrics not even related to metrics crate or Prometheus, like writing directly into a database.
  • They decouple metrics (and other future secondary tasks) from the main logic. So the main tracker logic remains simple.

Conclusion

I would migrate to metrics or at least I would build a PoC. In the first iteration we could:

• Use the metrics macros to emit the events.
• Build a custom recorder or/and exporter that updates the current repository, so we can continue using the output format we are planning to use in the stats overhaul epic.

Given:

• We would continue using the already-defined repositories and
• We are not emitting many events

I would continue with the current plan and I would add a new issue for the PoC if this migration is a priority.

Questions

I think the most important question is:

Do we want to integrate metrics in the tracker APIs (current REST API and future GraphQL API)?

or are we open to use independent specific APIs for metrics?

or do we want to provide both alternatives?

I think metrics are very important for a tracker, they are not only used only by developers, so I think they should be part of the public API for third-party integration. However, they are also very complex and there are standards for them. Performance is also critical because they produce a huge amount of data.

cc @da2ce7