[Question] Multiple Telegram notifications for a single scheduled speedtest (again)

[aferende]

Welcome!

• I have read the documentation and my problem was not listed in the help section.
• I have searched open and closed issues and my problem was not mentioned before.
• I have verified I am using the latest version available. You can check the latest release here.
• I agree to follow this project's Code of Conduct.

What did you do?

Related to ticket: #2615 (comment)

Expected Behavior

Just one notification

Steps to Reproduce

the same as previous ticket #2615 (comment)

Deployment Environment

Docker Compose

What is your environment & configuration?

the same as previous ticket #2615 (comment)

Application Information

Related to ticket: https://github.com/alexjustesen/speedtest-tracker/issues/2615#issue-3770250256

Hi, this morning I received three Telegram notifications again.
The version I have installed is v1.13.5 (which should be the latest available).
Can you confirm that this version includes the timeout increase?
(I think so because the timestamps are 30 seconds apart).
These are the speedtest tracker logs:
2026-01-19T06:04:27.355833595Z [2026-01-19 06:04:27] production.ERROR: Apprise notification failed {"channel":"tgram://2667XXXXXX:XXXXXXXXXXXXXXXXXXXmYebTI/5932912?preview=yes","instance":"http://192.168.100.105:8888/notify","message":"cURL error 28: Operation timed out after 30002 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://192.168.100.105:8888/notify","exception":"Illuminate\\Http\\Client\\ConnectionException"} 
2026-01-19T06:05:05.073442845Z [2026-01-19 06:05:05] production.ERROR: Apprise notification failed {"channel":"tgram://2667XXXXXX:XXXXXXXXXXXXXXXXXXXmYebTI/5932912?preview=yes","instance":"http://192.168.100.105:8888/notify","message":"cURL error 28: Operation timed out after 30006 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://192.168.100.105:8888/notify","exception":"Illuminate\\Http\\Client\\ConnectionException"} 
2026-01-19T06:05:43.405986017Z [2026-01-19 06:05:43] production.ERROR: Apprise notification failed {"channel":"tgram://2667XXXXXX:XXXXXXXXXXXXXXXXXXXmYebTI/5932912?preview=yes","instance":"http://192.168.100.105:8888/notify","message":"cURL error 28: Operation timed out after 30002 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://192.168.100.105:8888/notify","exception":"Illuminate\\Http\\Client\\ConnectionException"} 


These are the apprise logs:
2026-01-18T06:00:42.463576626Z 2026/01/18 06:00:42 [info] 34#34: *1 client 172.17.0.1 closed keepalive connection
2026-01-19T06:04:26.951537542Z 2026/01/19 06:04:26 [info] 36#36: *3 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: 172.17.0.1, server: , request: "POST /notify HTTP/1.1", upstream: "http://unix:/tmp/apprise/gunicorn.sock:/notify", host: "192.168.100.105:8888"
2026-01-19T06:04:27.051977682Z 172.17.0.1 - - [19/Jan/2026:06:04:26 +0000] "POST /notify HTTP/1.1" 499 0 "-" "GuzzleHttp/7"
2026-01-19T06:04:44.842594368Z 2026-01-19 00:04:44,750 [INFO] apprise: Sent Telegram notification.
2026-01-19T06:04:44.843122082Z 2026-01-19 00:04:44,801 [INFO] django: NOTIFY -  - Delivered Stateless Notification(s)
2026-01-19T06:05:04.945523393Z 2026/01/19 06:05:04 [info] 37#37: *5 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: 172.17.0.1, server: , request: "POST /notify HTTP/1.1", upstream: "http://unix:/tmp/apprise/gunicorn.sock:/notify", host: "192.168.100.105:8888"
2026-01-19T06:05:04.946021598Z 172.17.0.1 - - [19/Jan/2026:06:05:04 +0000] "POST /notify HTTP/1.1" 499 0 "-" "GuzzleHttp/7"
2026-01-19T06:05:11.674669300Z 2026-01-19 00:05:11,673 [INFO] apprise: Sent Telegram notification.
2026-01-19T06:05:11.678709207Z 2026-01-19 00:05:11,678 [INFO] django: NOTIFY -  - Delivered Stateless Notification(s)
2026-01-19T06:05:43.405525473Z 2026/01/19 06:05:43 [info] 37#37: *7 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: 172.17.0.1, server: , request: "POST /notify HTTP/1.1", upstream: "http://unix:/tmp/apprise/gunicorn.sock:/notify", host: "192.168.100.105:8888"
2026-01-19T06:05:43.523878947Z 172.17.0.1 - - [19/Jan/2026:06:05:43 +0000] "POST /notify HTTP/1.1" 499 0 "-" "GuzzleHttp/7"
2026-01-19T06:05:57.937263899Z 2026-01-19 00:05:57,792 [INFO] apprise: Sent Telegram notification.
2026-01-19T06:05:57.954779489Z 2026-01-19 00:05:57,847 [INFO] django: NOTIFY -  - Delivered Stateless Notification(s)



In the apprise logs, I notice this strange message:

epoll_wait() reported that client prematurely closed the connection

What do you think?

What browser(s) are you seeing the problem on?

Chrome

Logs

These are the speedtest tracker logs:
2026-01-19T06:04:27.355833595Z [2026-01-19 06:04:27] production.ERROR: Apprise notification failed {"channel":"tgram://2667XXXXXX:XXXXXXXXXXXXXXXXXXXmYebTI/5932912?preview=yes","instance":"http://192.168.100.105:8888/notify","message":"cURL error 28: Operation timed out after 30002 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://192.168.100.105:8888/notify","exception":"Illuminate\\Http\\Client\\ConnectionException"} 
2026-01-19T06:05:05.073442845Z [2026-01-19 06:05:05] production.ERROR: Apprise notification failed {"channel":"tgram://2667XXXXXX:XXXXXXXXXXXXXXXXXXXmYebTI/5932912?preview=yes","instance":"http://192.168.100.105:8888/notify","message":"cURL error 28: Operation timed out after 30006 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://192.168.100.105:8888/notify","exception":"Illuminate\\Http\\Client\\ConnectionException"} 
2026-01-19T06:05:43.405986017Z [2026-01-19 06:05:43] production.ERROR: Apprise notification failed {"channel":"tgram://2667XXXXXX:XXXXXXXXXXXXXXXXXXXmYebTI/5932912?preview=yes","instance":"http://192.168.100.105:8888/notify","message":"cURL error 28: Operation timed out after 30002 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://192.168.100.105:8888/notify","exception":"Illuminate\\Http\\Client\\ConnectionException"} 


These are the apprise logs:
2026-01-18T06:00:42.463576626Z 2026/01/18 06:00:42 [info] 34#34: *1 client 172.17.0.1 closed keepalive connection
2026-01-19T06:04:26.951537542Z 2026/01/19 06:04:26 [info] 36#36: *3 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: 172.17.0.1, server: , request: "POST /notify HTTP/1.1", upstream: "http://unix:/tmp/apprise/gunicorn.sock:/notify", host: "192.168.100.105:8888"
2026-01-19T06:04:27.051977682Z 172.17.0.1 - - [19/Jan/2026:06:04:26 +0000] "POST /notify HTTP/1.1" 499 0 "-" "GuzzleHttp/7"
2026-01-19T06:04:44.842594368Z 2026-01-19 00:04:44,750 [INFO] apprise: Sent Telegram notification.
2026-01-19T06:04:44.843122082Z 2026-01-19 00:04:44,801 [INFO] django: NOTIFY -  - Delivered Stateless Notification(s)
2026-01-19T06:05:04.945523393Z 2026/01/19 06:05:04 [info] 37#37: *5 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: 172.17.0.1, server: , request: "POST /notify HTTP/1.1", upstream: "http://unix:/tmp/apprise/gunicorn.sock:/notify", host: "192.168.100.105:8888"
2026-01-19T06:05:04.946021598Z 172.17.0.1 - - [19/Jan/2026:06:05:04 +0000] "POST /notify HTTP/1.1" 499 0 "-" "GuzzleHttp/7"
2026-01-19T06:05:11.674669300Z 2026-01-19 00:05:11,673 [INFO] apprise: Sent Telegram notification.
2026-01-19T06:05:11.678709207Z 2026-01-19 00:05:11,678 [INFO] django: NOTIFY -  - Delivered Stateless Notification(s)
2026-01-19T06:05:43.405525473Z 2026/01/19 06:05:43 [info] 37#37: *7 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: 172.17.0.1, server: , request: "POST /notify HTTP/1.1", upstream: "http://unix:/tmp/apprise/gunicorn.sock:/notify", host: "192.168.100.105:8888"
2026-01-19T06:05:43.523878947Z 172.17.0.1 - - [19/Jan/2026:06:05:43 +0000] "POST /notify HTTP/1.1" 499 0 "-" "GuzzleHttp/7"
2026-01-19T06:05:57.937263899Z 2026-01-19 00:05:57,792 [INFO] apprise: Sent Telegram notification.
2026-01-19T06:05:57.954779489Z 2026-01-19 00:05:57,847 [INFO] django: NOTIFY -  - Delivered Stateless Notification(s)

[svenvg93]

This behaviour is expected.
After increasing the notification timeout to 30 seconds, the worker will wait up to 30 seconds for Apprise to return an HTTP response.

If Apprise does not respond within this window the request is marked as failed and the worker will automatically retry up to 3 times.

Since the delay occurs entirely on the Apprise side, there is nothing we can do to compensate for a slow Apprise response without increasing the timeout further. Previously, your Apprise instance needed around 15 seconds to process a message; it is now taking more than 30 seconds, which strongly suggests an issue with the system or container running Apprise.

[aferende]

This log suggests that Speedtest Tracker closed the connection prematurely, causing a false apprehension timeout.

epoll_wait() reported that client prematurely closed connection

Am I wrong?

[svenvg93]

No, that is correct but it depends on which side you’re looking at 😉

From Speedtest Tracker’s perspective, it is expected to close the connection after 30 seconds if no response is received from Apprise.
From Apprise’s perspective, the connection is considered prematurely closed because it has not yet finished processing and sending the needed http messages.

This is the exact same issue and logging pattern as in #2615; the only difference is that this time the processing exceeds the increased 30-second timeout.

From the logs you can see that we close it 30 seconds after the POST. In the meantime Apprise Send the notification but we never get the answer after that.

2026-01-19T06:05:04.946021598Z 172.17.0.1 - - [19/Jan/2026:06:05:04 +0000] "POST /notify HTTP/1.1" 499 0 "-" "GuzzleHttp/7"

2026-01-19T06:05:11.674669300Z 2026-01-19 00:05:11,673 [INFO] apprise: Sent Telegram notification.
2026-01-19T06:05:11.678709207Z 2026-01-19 00:05:11,678 [INFO] django: NOTIFY -  - Delivered Stateless Notification(s)

2026-01-19T06:05:43.405525473Z 2026/01/19 06:05:43 [info] 37#37: *7 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: 172.17.0.1, server: , request: "POST /notify HTTP/1.1", upstream: "http://unix:/tmp/apprise/gunicorn.sock:/notify", host: "192.168.100.105:8888"

As Apprise says the connection is closed prematurely it means it did not send the reply and closed the connection as it should be.

Unfortunately, there is no alternative way for us to handle this, as Apprise only sends an HTTP response after it has completely finished processing the notification on its end.

In our opinion, 30 seconds is a reasonable amount of time to wait for sending a simple notification. If Apprise does not respond within this window, the request is marked as failed and the worker will automatically retry up to 3 times. Since Apprise continues processing even after the connection is closed, this results in duplicate notifications.

[aferende]

Okay, thanks for the explanation.
I'm trying to understand what's going on. It seems unlikely that three calls in a row would take more than 30 seconds to answer (I haven't had a load for days).
Anyway, thanks.

[svenvg93]

Depending on whether the issue occurs consistently, you can try sending test notifications directly through Apprise yourself from any Linux machine using curl:
https://github.com/caronc/apprise-api?tab=readme-ov-file#stateless-solution

At the moment, I don’t see anything we are doing incorrectly or could do differently. Unfortunately, Apprise does not provide an intermediate response when a message is accepted. It only responds once the notification has been sent or if an error occurs:
https://github.com/caronc/apprise-api?tab=readme-ov-file#api-response-codes.

I will look into adding more debug logging on our side to make the process more visible in situations like this.

[aferende]

I've already tried curl, apprise test, and running tests from speedtest tracker.
Replies and sending messages are always immediate, so I don't understand how it can respond with a 30-second timeout three times in a row. It doesn't seem plausible to me.

[svenvg93]

Does this happen consistently, or was it a one-off incident? I noticed that in both of your reports, it occurs around 06:00. Could there be something else running on the same system at that time—another scheduled task, a backup, or similar—that might be contributing?

I will try to reproduce the issue to see what is possible. We will need to close the connection at some point when there is no response from Apprise, so I don’t believe there’s anything more we can do on our side to handle it.