Error updating the tracker to the latest docker image in the live demo

[josecelano]

After updating the live demo with the latest docker image the container is unhealthy:

5e1d552e3fca   torrust/tracker:develop     "/usr/local/bin/entr…"   26 minutes ago   Up 26 minutes (unhealthy)   0.0.0.0:1212->1212/tcp, :::1212->1212/tcp, 0.0.0.0:7070->7070/tcp, :::7070->7070/tcp, 1313/tcp, 0.0.0.0:6969->6969/udp, :::6969->6969/udp   tracker

And this are the logs:

Loading configuration file: `/etc/torrust/tracker/tracker.toml` ...
2024-01-04T13:14:14.555359324+00:00 [torrust_tracker::bootstrap::logging][INFO] logging initialized.
2024-01-04T13:14:14.559268954+00:00 [torrust_tracker::servers::udp::server][INFO] Running UDP Tracker on Socket: 0.0.0.0:6969
2024-01-04T13:14:14.559910849+00:00 [torrust_tracker::bootstrap::jobs][INFO] tls not enabled
thread 'tokio-runtime-worker' panicked at /build/src/src/servers/signals.rs:49:38:
Failed to install stop signal.: RecvError(())
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2024-01-04T13:14:14.562318295+00:00 [torrust_tracker::bootstrap::jobs][INFO] tls not enabled
thread 'tokio-runtime-worker' panicked at src/servers/signals.rs:49:38:
Failed to install stop signal.: RecvError(())
thread 'tokio-runtime-worker' panicked at src/servers/signals.rs:49:38:
Failed to install stop signal.: RecvError(())
2024-01-04T13:14:14.563936493+00:00 [torrust_tracker::bootstrap::jobs::health_check_api][INFO] Starting Health Check API server: http://127.0.0.1:1313
2024-01-04T13:14:14.564108924+00:00 [torrust_tracker::bootstrap::jobs::health_check_api][INFO] Torrust Health Check API server started on socket: 127.0.0.1:1313
2024-01-04T13:24:14.565271668+00:00 [torrust_tracker::bootstrap::jobs::torrent_cleanup][INFO] Cleaning up torrents..
2024-01-04T13:24:14.565336880+00:00 [torrust_tracker::bootstrap::jobs::torrent_cleanup][INFO] Cleaned up torrents in: 0ms
2024-01-04T13:34:14.565066347+00:00 [torrust_tracker::bootstrap::jobs::torrent_cleanup][INFO] Cleaning up torrents..
2024-01-04T13:34:14.565135670+00:00 [torrust_tracker::bootstrap::jobs::torrent_cleanup][INFO] Cleaned up torrents in: 0ms

I'm trying to find out what happened.

cc @da2ce7

UPDATE!

I'm going to split this issue into two subtasks:

Subtasks

• Error starting tracker API: Failed to install stop signal.: RecvError(()) #588
• Error starting HTTP tracker: Failed to install stop signal: channel closed #592
• Error starting UDP tracker: Failed to install stop signal: channel closed #591
• Fix zombie healtcheck threads #597

[josecelano]

I have the same problem running the tracker locally:

Loading default configuration file: `./share/default/config/tracker.development.sqlite3.toml` ...
2024-01-04T13:48:11.596302278+00:00 [torrust_tracker::bootstrap::logging][INFO] logging initialized.
2024-01-04T13:48:11.596999183+00:00 [torrust_tracker::bootstrap::jobs::http_tracker][INFO] Note: Not loading Http Tracker Service, Not Enabled in Configuration.
2024-01-04T13:48:11.597005853+00:00 [torrust_tracker::bootstrap::jobs][INFO] tls not enabled
thread 'tokio-runtime-worker' panicked at src/servers/signals.rs:49:38:
Failed to install stop signal.: RecvError(())
2024-01-04T13:48:11.597198191+00:00 [torrust_tracker::bootstrap::jobs::health_check_api][INFO] Starting Health Check API server: http://127.0.0.1:1313
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2024-01-04T13:48:11.597253671+00:00 [torrust_tracker::bootstrap::jobs::health_check_api][INFO] Torrust Health Check API server started on socket: 127.0.0.1:1313

[josecelano]

And it seems the server it's using 100% of memory and CPU. The demo isn't working

[josecelano]

Hi @da2ce7 it seems spawned threads for running health checks are being run forever (in the tracker). Maybe that's the reason the system is using 100% CPU and memory.

live-demo-console-log.txt

[josecelano]

Hi @da2ce7 regarding the panic Failed to install stop signal.: RecvError(()):

I'm debugging with only the API enabled. The API starts successfully but the shutdown channel is closed immediately. I think the problem is the channel is dropped after starting the API.
The lifetime for that channel should be the same as the API task.

    pub async fn start(self, tracker: Arc<Tracker>) -> Result<ApiServer<Running>, Error> {
        let (tx_start, rx_start) = tokio::sync::oneshot::channel::<Started>();
        let (tx_halt, rx_halt) = tokio::sync::oneshot::channel::<Halted>();

        let launcher = self.state.launcher;

        let task = tokio::spawn(async move {
            launcher.start(tracker, tx_start, rx_halt).await;
            launcher
        });

        Ok(ApiServer {
            state: Running {
                binding: rx_start.await.expect("unable to start service").address,
                halt_task: tx_halt,
                task,
            },
        })
    }

Maybe we need to move that channel one level up.

On the other hand, it seems pub struct ApiServerJobStarted(); is not used anymore.

[josecelano]

Hi @da2ce7 regarding the panic Failed to install stop signal.: RecvError(()):

I'm debugging with only the API enabled. The API starts successfully but the shutdown channel is closed immediately. I think the problem is the channel is dropped after starting the API. The lifetime for that channel should be the same as the API task.

    pub async fn start(self, tracker: Arc<Tracker>) -> Result<ApiServer<Running>, Error> {
        let (tx_start, rx_start) = tokio::sync::oneshot::channel::<Started>();
        let (tx_halt, rx_halt) = tokio::sync::oneshot::channel::<Halted>();

        let launcher = self.state.launcher;

        let task = tokio::spawn(async move {
            launcher.start(tracker, tx_start, rx_halt).await;
            launcher
        });

        Ok(ApiServer {
            state: Running {
                binding: rx_start.await.expect("unable to start service").address,
                halt_task: tx_halt,
                task,
            },
        })
    }

Maybe we need to move that channel one level up.

On the other hand, it seems pub struct ApiServerJobStarted(); is not used anymore.

I've just realized the sender is included in the ApiServer (tx_halt) so it should not be droped.

[josecelano]

Notice that this issue has been split in subtasks.

[josecelano]

Hi @da2ce7 regarding the panic Failed to install stop signal.: RecvError(()):
I'm debugging with only the API enabled. The API starts successfully but the shutdown channel is closed immediately. I think the problem is the channel is dropped after starting the API. The lifetime for that channel should be the same as the API task.

    pub async fn start(self, tracker: Arc<Tracker>) -> Result<ApiServer<Running>, Error> {
        let (tx_start, rx_start) = tokio::sync::oneshot::channel::<Started>();
        let (tx_halt, rx_halt) = tokio::sync::oneshot::channel::<Halted>();

        let launcher = self.state.launcher;

        let task = tokio::spawn(async move {
            launcher.start(tracker, tx_start, rx_halt).await;
            launcher
        });

        Ok(ApiServer {
            state: Running {
                binding: rx_start.await.expect("unable to start service").address,
                halt_task: tx_halt,
                task,
            },
        })
    }

Maybe we need to move that channel one level up.
On the other hand, it seems pub struct ApiServerJobStarted(); is not used anymore.

I've just realized the sender is included in the ApiServer (tx_halt) so it should not be droped.

In the end, It seems it was dropped. See #588

Just adding an assert using that part (tx_halt) of the ApiServer fixed the problem.

[josecelano]

Regarding healthcheck zombie threads, they could be produced because we don't have a timeout for requesting the healthcheck endpoint as we discussed this morning.

cc @da2ce7