issue2551147 - Enable compression of http responses in roundup.

gzip, (brotli/zstd with optional packages) on the fly
compression/content-encoding enabled by default. Can serve
pre-compressed static assets as well if the client can accept it.

Docs updated.

Also added example nginx config to installation.txt. The config allows
nginx to compress data on the fly. If the config is used, dynamic
compression in roundup can be disabled.

Dedicating this checkin to my father Paul Hector Rouillard 1930-2021.
I did much of the development in this changeset while sitting with him
as he slept/transitioned. Without his encouragement and example, my
desire to learn would not be what it is and I wouldn't be half the
person I am.

Author: rouilj

.travis.yml

@@ -120,6 +120,7 @@ install:
   - if [[ $TRAVIS_PYTHON_VERSION != "3.4"* ]]; then pip install mistune; fi
   - if [[ $TRAVIS_PYTHON_VERSION != "3.4"* && $TRAVIS_PYTHON_VERSION != "2."* ]]; then pip install Markdown; fi
   - pip install markdown2
+  - pip install brotli zstd
 
 before_script:
   # set up mysql database

CHANGES.txt

@@ -11,6 +11,18 @@ Python v2.5 and v2.6.  Starting with the v1.6 releases of Roundup
 v2.7.2 or later are required to run newer releases of Roundup.  From v2.0
 onwards Python 3.4 and later are also supported.
 
+20xx-yy-zz 2.2.0
+
+Fixed:
+
+Features:
+
+- issue2551147 - Enable compression of http responses in roundup.
+  Allow roundup to return gzip, (br or zstd with added modules)
+  Content-Encoded replies. Compression could be done in upstream
+  proxies/wsgi server but this allows it to occur natively. (John
+  Rouillard)
+
 2021-07-13 2.1.0
 
 Fixed:

doc/admin_guide.txt

@@ -191,6 +191,84 @@ overwrite an existing file) from the roundup-server command line use::
  roundup_server -p 8017  -u roundup --save-config  demo=/trackers/demo \
     sysadmin=/trackers/sysadmin
 
+Configuring Compression
+=======================
+
+Roundup will compress HTTP responses to clients on the fly. Dynamic,
+on the fly, compression is enabled by default, to disable it set::
+
+    [web]
+    ...
+    dynamic_compression = No
+
+in the tracker's ``config.ini``. You should disable compression if
+your proxy (e.g. nginx or apache) or wsgi server (uwsgi) is configured
+to compress responses on the fly. The python standard library includes
+gzip support. For brotli or zstd you will need to install packges. See
+the `installation documentation`_ for details.
+
+Some assets will not be compressed on the fly. Assets with mime types
+of "image/png" or "image/jpeg" will not be compressed. You
+can add mime types to the list by using ``interfaces.py`` as discussed
+in the `customisation documentation`_. As an example adding::
+
+  from roundup.cgi.client import Client
+
+  Client.precompressed_mime_types.append('application/zip`)
+
+to ``interfaces.py`` will prevent zip files from being compressed.
+
+Any content less than 100 bytes in size will not be compressed (e.g
+errors messages, short json responses).
+
+Zstd will be used if the client can understand it, followed by brotli
+then gzip encoding. Currently the preference order is hard coded into
+the server and not parsed using ``q`` values from the client's
+Accept-Encoding header. This is an area for improvement.
+
+In addition to dynamic compression, static files/assets accessed using
+``@@file`` can be pre-compressed. This reduces CPU load on the server
+and reduces the time required to respond to the client. By default
+searching for pre-compressed files is disabled. To enable it set::
+
+    [web]
+    ...
+    use_precompressed_files = Yes
+
+in the tracker's ``config.ini`` file. Then you can create a
+precompressed file and it will be served if the client is able to
+accept it. For a file ``.../@@file/library.js`` you can create::
+
+    tracker_home/html/library.js.gzip
+    tracker_home/html/library.js.br
+    tracker_home/html/library.js.zstd
+
+which should be created by using (respectively)::
+
+      gzip --keep --suffix .gzip library.js
+      brotli library.js
+      zstd library.js && mv library.js.zst library.js.zstd
+
+see the man pages for options that control compression level. Note
+that some levels require additional memory on the client side, so you
+may not always want to use the highest compression available.
+
+A pre-compressed file will not be used if its modified date is earlier
+than the uncompressed file. For example, if ``library.js.gzip`` is
+older (has earlier modification date) than ``library.js``,
+``library.js.gzip`` will be ignored. ``library.js`` will be
+served instead.  ``library.js`` will be dynamically compressed on the
+fly and a warning message will be logged.
+
+Precompressed files override dynamic compression. For example, assume
+the client can accept brotli and gzip. If there are no precompressed
+files, the data will be compressed dynamically (on the fly) using
+brotli. If there is a precompressed gzip file present the client will
+get the gzip version and not a brotli compressed version. This
+mechanism allows the admin to allow use of brotli and zstd for
+dynamic content, but not for static content.
+
+
 Users and Security
 ==================
 
@@ -525,3 +603,4 @@ Run ``roundup-admin help commands`` for a complete list of subcommands.
 
 .. _`customisation documentation`: customizing.html
 .. _`upgrading documentation`: upgrading.html
+.. _`installation documentation`: installation.html

doc/installation.txt

@@ -129,6 +129,14 @@ markdown, markdown2 or mistune
   To use markdown rendering you need to have the markdown_, markdown2_
   or mistune_ (v0.8.4 tested) package installed.
 
+zstd, brotli
+  To have roundup compress the returned data using one of these
+  algorithms, you can install one or more of zstd_ or brotli_.
+  Roundup's responses can always be compressed with gzip from the
+  Python standard library. Also nginx and various wsgi server can
+  compress the response from roundup as they transmit/proxy it to the
+  client.
+
 Windows Service
   You can run Roundup as a Windows service if pywin32_ is installed.
   Otherwise it must be started manually.
@@ -386,7 +394,8 @@ There are multiple web interfaces to choose from:
 4. `Zope product - ZRoundup`_
 5. `Apache HTTP Server with mod_wsgi`_
 6. `Apache HTTP Server with mod_python`_  (deprecated)
-7. `WSGI Variations`_
+7. `Nginx HTTP Server`_
+8. `WSGI Variations`_
 
 You may need to give the web server user permission to access the tracker home
 - see the `UNIX environment steps`_ for information. You may also need to
@@ -818,6 +827,120 @@ The URL for accessing these trackers then become:
 Note that in order to use https connections you must set up Apache for secure
 serving with SSL.
 
+Nginx HTTP Server
+~~~~~~~~~~~~~~~~~
+
+This configuration uses gunicorn to run roundup behind an Nginx proxy.
+The proxy also compresses the data using gzip. The url for the tracker
+in config.ini should be ``https://tracker.example.org``.
+
+  .. code:: 
+
+    user nginx;
+    worker_processes auto;
+    worker_rlimit_nofile 10000;
+
+    error_log /var/log/nginx/global-error.log;
+    pid /var/run/nginx.pid;
+
+    events {
+	worker_connections 1024;
+    }
+
+    upstream tracker-tracker {
+      # gunicorn uses this socket for communication
+      server unix:/var/run/roundup/tracker.sock fail_timeout=0;
+    }
+
+    http {
+      include /etc/nginx/mime.types;
+      default_type application/octet-stream;
+
+      log_format main '$remote_addr - $remote_user [$time_local] "$request" '
+		      '$status $body_bytes_sent "$http_referer" '
+		      '"$http_user_agent" "$http_x_forwarded_for"';
+
+      access_log /var/log/nginx/global-access.log main;
+      sendfile on;
+      tcp_nopush on;
+      tcp_nodelay on;
+      server_tokens off;
+
+      gzip_http_version 1.1;
+      gzip_proxied      any;
+      gzip_min_length   500;
+      # default comp_level is 1
+      gzip_comp_level   6;
+      gzip_disable      msie6
+      gzip_types        text/plain text/css
+			text/xml application/xml
+			text/javascript application/javascript
+			text/json application/json;
+      # upstream proxies need to match Accept-Encoding as
+      # part of their cache check
+      gzip_vary         on
+
+      server {
+	listen 80;
+	server_name tracker.example.org;
+
+	location /.well-known/acme-challenge/ {
+	    alias /etc/lego/.well-known/acme-challenge/;
+	    try_files $uri =404;
+	}
+
+	location / {
+	  return 301 https://$http_host$request_uri;
+	}
+      }
+
+      server {
+	listen 443 ssl;
+	server_name tracker.example.org;
+	include mime.types;
+
+	# By default use the snakeoil certificate...
+	# change this if you are using a real SSL cert
+	ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
+	ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;
+
+	# These are useful for @@files where roundup is bypassed.
+	# but can be set by roundup as well. See:
+        #    https://wiki.roundup-tracker.org/AddingContentSecurityPolicy
+        # which also sets other security headers.
+	add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
+	add_header X-Frame-Options "sameorigin";
+	add_header X-Xss-Protection "1; mode=block";
+	add_header X-Content-Type-Options "nosniff";
+	add_header X-Permitted-Cross-Domain-Policies "none";
+
+	error_log /var/log/nginx/roundup-tracker.error.log;
+	access_log /var/log/nginx/roundup-tracker.access.log
+
+	root /home/roundup/trackers/tracker/;
+
+	# have nginx return files from @@file directly rather than
+	# going though roundup
+	location /@@file/ {
+	  rewrite ^/@@file/(.*) /html/$1 break;
+	  # note that you can not use cache control (see customizing doc)
+          # in roundup to set the expires headers since we are
+	  # bypassing roundup. Consider using a map or different
+	  # location stanzas to vary the expiration times.  
+	  expires 1h;
+	}
+
+	location / {
+	  # must define tracker-tracker in upstream stanza
+	  proxy_pass http://tracker-tracker/;
+	  proxy_set_header Host $host;
+	  proxy_set_header X-Real-IP $remote_addr;
+	  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+	}
+      }
+    }
+
+
 WSGI Variations
 ~~~~~~~~~~~~~~~
 
@@ -957,6 +1080,9 @@ Assuming the proxy forwards /tracker, run gunicorn as::
 
 this runs roundup at port 8917 on the loopback interface. You should
 configure the reverse proxy to talk to 127.0.0.1 at port 8917.
+If you want you can use a unix domain socket instead. Example:
+``--bind unix:///var/run/roundup/tracker.sock`` would be used for the
+nginx configuration below.
 
 .. index:: pair: web interface; uWSGI
    single: wsgi; uWSGI
@@ -1438,6 +1564,7 @@ there are no errors. If there are errors, please let us know!
 .. _`adding MySQL users`:
     https://dev.mysql.com/doc/refman/8.0/en/creating-accounts.html
 .. _apache: https://httpd.apache.org/
+.. _brotli: https://pypi.org/project/Brotli/
 .. _docutils: https://pypi.org/project/docutils/
 .. _flup: https://pypi.org/project/flup/
 .. _gpg: https://www.gnupg.org/software/gpgme/index.html
@@ -1457,3 +1584,4 @@ there are no errors. If there are errors, please let us know!
 .. _pywin32: https://pypi.org/project/pywin32/
 .. _Whoosh: https://whoosh.readthedocs.org/en/latest
 .. _Xapian: https://xapian.org/
+.. _zstd: https://pypi.org/project/zstd/

doc/upgrading.txt

@@ -26,6 +26,25 @@ Contents:
 
 .. index:: Upgrading; 2.0.0 to 2.1.0
 
+Migrating from 2.1.0 to 2.x.y
+=============================
+
+Check Compression Settings
+--------------------------
+
+Read the `administration guide`_ section on 'Configuring Compression'.
+
+Upgrade tracker's config.ini file. Use::
+
+  roundup-admin -i /path/to/tracker updateconfig newconfig.ini
+
+to generate a new ini file preserving all your settings. You can then
+merge any local comments from the tracker's ``config.ini`` into
+``newconfig.ini``. Compare the old and new files and configure new
+compression settings as you want. Then replace ``config.ini`` with the
+``newconfig.ini`` file.
+
+
 Migrating from 2.0.0 to 2.1.0
 =============================