[Question] Long response times and UptimeKuma ping issues

[Jacob8ch]

Pre-work

• I have read the docs.
• I have searched open and closed issues.
• I agree to follow this project's Code of Conduct.

Description

I have an issue where ping times are in excess of 1000ms. I had UptimeKuma to check if the service is alive. Recently, there was a steady climb in response times to which there was a peak response of 17000ms before UptimeKuma registered it offline. I have also noticed that load times for the web UI takes a significant amount of time compared to other services running in docker.

As seen in this picture, the CPU usage is not above a baseline level, maintaining ~0.7% usage. The CPU is not throttling or causing this issue. All other stats show similar results.

---

Here is the UptimeKuma graph which shows the high response times over the past 24 hours. The yellow means there was less than 3 pings that had ping of more than 45 seconds, red means there was more than 3 pings over 45 seconds.

---

Here are the ping results over the past 7 days, showing an exponential upward trend in ping times.

I would like to note that the system has lost performance and this is likely the cause for the high response times. Nothing but Docker and Speedtest Tracker are running on the LXC. There are about 250-300 speedtests in the database.

---

I am running this in a Proxmox Debian 12 LXC with docker installed. This is fairly standard when it comes to running Docker processes in an LXC from what I can tell.

Here's the specs of the LXC:
2 CPU cores (8th gen i7)
2GB Ram
2GB Swap
32GB storage SATA SSD (1.61GB in use)

Expected Behavior

The response time would be lowered to an absolute max of 250ms. Most services running under the same conditions run with single digit response times, but there are single spikes every now and then.

Steps to Reproduce

I don't think this is something only I am experiencing running this in an LXC. So I think just a regular install would work. If the response time is not what I am experiencing, try running in an LXC with the same specs listed above.

1. Run Speedtest Tracker
2. Observe pings and response times (about two to three weeks of time has elapsed with a speedtest taken every hour).

Deployment Environment

Docker Compose

Application Information

php not installed? Runs in a docker environment.

What browser(s) are you seeing the problem on?

Brave, Chrome, Insomnia HTTP Requests, UptimeKuma Pings

Logs

[svenvg93]

Just for my understanding about the problem. What are you pinging in these tests? Is the response time from the healthcheck endpoint on /api/healthcheck.

[Jacob8ch]

This data shows me pinging the web GUI at the / endpoint, not the health endpoint. However, it appears that the endpoint is giving the same behavior.

> Here is an insomnia ping to the healthcheck endpoint. It shows 23.4 seconds before the server responds.

I am investigating to see if it is a network configuration issue, however I am not seeing that it is the case, as I can ping the server just fine with <1ms of latency

This appears to me to be an issue with the web server, but I could be wrong.

[alexjustesen]

Let me know if you find anything as you dig deeper, I'm not seeing the same issues when pinging the server vs. the application.

[Jacob8ch]

After some investigation, it appears that the entire LXC container is running slow. I'm still checking to see if it is my issue or yours. I have done reboots of the LXC and the node associated with it. The issue is still present. There are no random large files or anything. The LXC has plenty of resources available, so nothing is being bottlenecked. Nothing is even being used. from this assumption I can only think it's something with the web server not responding quickly. I can no longer connect to it via the web interface on a browser to see the data. I believe this issue occurred running v1.8.0. I am currently updating the docker compose file, and will update you on the results.

[alexjustesen]

it appears that the entire LXC container is running slow.

In the past I've seen instances where LXC containers have had poor performance. Could be unrelated but this user found a fix for their issue: #1672

[Jacob8ch]

I think that issue was because the computer disabled his PCIe NIC due to ASPM. I have disabled it anyway to see if it fixed the issue, but it did not work. This is definitely something with the service, as the speed of the LXC has returned, but the pings have not gone to a normal level.

Here is a screenshot from the LXC's console of a wget request to the healthcheck endpoint. The ping took 18 seconds despite it being a process on localhost.

If you have any ideas, let me know. I'm out of ideas. It will sit and wait for a 200 code, and then load the contents of the data immediately. It is a little strange to me, it isn't buffering like a normal slow website would. I am still unable to connect to it via a browser. The problem seems intermittent as well, as randomly I will get a 2 second health check, then I will get a 20 second one. I am stumped at the moment.

Edit: More times than not, it will not be a short ping. 98% of the time it would be 20+ seconds.

[svenvg93]

Do you know the last version where the API response times were still fast? We haven't changed anything to the API in a long time that might cause this delay.
I’m wondering if the recent LSIO changes to the Nginx configuration might be causing the slowdown. Could you try v1.7.1-ls116 to see if that version gives you better response times?

[Jacob8ch]

I don't know what version the API times were fast on. I've only ever ran 1.8.0 and it slowly got worse as time progressed. I just ran a Docker compose pull, but the issue still persists.

[alexjustesen]

@Jacob8ch would you be willing to test a different image?

[Jacob8ch]

Sure, I'd be willing to do that.

[alexjustesen]

Sure, I'd be willing to do that.

I've been testing a new image which is plug-n-play with the LSIO one to use it:

1. Change your image to ghcr.io/alexjustesen/docker-speedtest-tracker:latest (this will pull v1.11.2 as of this morning)
2. The image exposes 8080 (http) and 8443 (https) so update your port mappings accordingly.

So far @svenvg93 and I have been seeing pretty significant improvements in application performance across the board, we're seeing API response times comparing the two images from ~160ms (LSIO) to ~30ms (mine).

[github-actions[bot]]

👋 This issue has been automatically marked as stale due to inactivity. If this issue is still relevant, please comment to keep it open. Without activity, it will be closed in 7 days.

[Jacob8ch]

After running this, my problem is still present. I have reinstalled it on a new LXC and this issue is starting to come back again on the new container. Perhaps there is some background process, memory leak, or cache that isn't cleared that is slowing the machine down?

[alexjustesen]

Generally speaking I haven't supported LXC and there has been a history of issues with performance problems. I'm converting this to a discussion so the community can provide additional support/recommendations.