Implement health check for the tracker application

[josecelano]

Implement a HEALTHCHECK for the whole application. We are doing the same for the index.

The goal is to add a line similar to this:

HEALTHCHECK --interval=5s --timeout=5s --start-period=3s --retries=3 \  
  CMD    /usr/bin/health_check --http http://localhost:${API_PORT}/health_check \
      && /usr/bin/health_check --http http://localhost:${HTTP_PORT}/health_check \
      && /usr/bin/health_check --udp udp://localhost:${UDP_PORT} \
      || exit 1

So when you run the tracker in a container, it can show its status (healthy or not).

For the HTTP tracker and Tracker API, we can add health_check endpoints like in the Index.

For the UDP Tracker, maybe we can try to connect.

[josecelano]

Today @da2ce7 and I agree on try this approach:

• We add a high-level Management API.
• That API uses a new socket.
• For the time being, it only has one endpoint: GET HTTP://localhost:1313/health_check.
• The health check handler has to check if the other services are healthy.

Proposed solution 1: one-shot channels

The proposed implementation is to use channels between the main Management API and the other three services: API, HTTP Tracker and UDP Tracker. We can modify the start_job functions to accept a new parameter which is the channel sender to send a message:

use std::sync::mpsc;

let (tx_tracker_api, rx_tracker_api) = mpsc::channel();
let (tx_http_tracker, rx_http_tracker) = mpsc::channel();
let (tx_udp_tracker, rx_udp_tracker) = mpsc::channel();

When we start the service we send back to the main app a message and we change the state so that the Management API can use that state. With this solution, this would be checked only once. If the service crashes the health checkpoint is going to return always a healthy state.

We can send the message from the service, not from the launcher, but in that case, the service must know how to communicate with the high-level service. And that is something I do not like. We should be able to move the UDP to its own package in the future.

Proposed solution 2: Rest API (polling)

What I did for the Index was this:

There are 2 services:

• The API
• A runner (importer executed every 2 minutes)

The importer process sends a heartbeat periodically to the importer API to inform it's still alive. Both components are inside the same package/module. The high-level service polls both low-level services, so neither the API nor the importer know the heath checker.

We can do the same here. Adding an endpoint for health checks to the three services would allow us to poll instead of pushing from those services. I think Rest API are a less couple mechanism in comparison with channels if we want to create separate packages for the UDP Tracker, HTTP tracker and API.

In summary, service should provide a way to monitor them without coupling them to the processes that are monitoring. We could implement a "push" approach, where the services send a message periodically to the monitor but I would use anyway an API instead of shared memory between processes. With shared memory, we will never be able to deploy those services independently. And there could be some scenarios where we want to do that. Probably API requests are less than UDP announce requests.

@da2ce7 what do you think?