Summary: Support selecion session/otk data store. Add redis as data store.

Allow admin to select the backend data store. Compatibility matrix:

  main\/ session>| anydbm | sqlite | redis | mysql | postgresql |
  anydbm         |    D   |        |   X   |       |            |
  sqlite         |    X   |    D   |   X   |       |            |
  mysql          |        |        |       |   D   |            |
  postgresql     |        |        |       |       |      D     |
  --------------------------------------------------------------+
        D - default if unconfigured,   X - compatible choice

DETAILS

roundup/configuration.py:
  add config.ini section sessiondb with settings: backend and redis_url.

CHANGES.txt, doc/admin_guide.txt, doc/installation.txt, doc/upgrading.txt:
  doc on config of session db and redis. Plus some other fixes:

   admin - clarified why we do not drop __words and __testids
      table in native-fts conversion. TYpo fix.

   upgrading - doc how you can keep using anydbm for session data with
               sqlite. Fix dupe sentence in an upgrading config.ini
	       section.

roundup/backends/back_anydbm.py, roundup/backends/back_sqlite.py:
  code to support redis, redis/anydbm backends respectively.

roundup/backends/sessions_redis.py
  new storage backend for redis.

roundup/rest.py, roundup/cgi/actions.py, roundup/cgi/templating.py
  redis uses a different way of calculating lifetime/timestamp.
  Since expiration of an item occurred if its timestamp was more
  than 1 week old, code would calculate:
    now - 1 week + lifetime.
  But this results in faster expiration in redis if used for
   lifetime/timestamp.
  Convert code to use the lifetime() method in BasicDatabase
  that generates the right timestamp for each backend.

test/session_common.py:
   added tests for more cases, get without default, getall non-existing
        key etc. timestamp test changed to use new self.get_ts which is
        overridden in other tests. Test that datatypes survive storage.

test/test_redis_session.py:
  test redis session store with sqlite and anydbm primary databases

test/test_anydbm.py, test/test_sqlite.py
  add test to make sure the databases are properly set up

  sqlite - add test cases where anydbm is used as datastore
  anydbm - remove updateTimestamp override add get_ts().

test/test_config.py
   tests on redis_url and compatibility on choice of sessiondb backend

.travis.yml:
add redis db and redis-py

Author: rouilj

.travis.yml

@@ -31,6 +31,7 @@ python:
 services:
   - mysql
   - postgresql
+  - redis-server
 
 jobs:
     allow_failures:  # releases not ready for prime time yet.
@@ -113,6 +114,7 @@ install:
   - if [[ $TRAVIS_PYTHON_VERSION == "3.4"* ]]; then  pip install mysqlclient==1.3.14; fi
   - if [[ $TRAVIS_PYTHON_VERSION != "3.4"* ]]; then pip install mysqlclient; fi
   - pip install psycopg2
+  - pip install redis
   - pip install gpg pytz whoosh pyjwt requests
   - pip install jinja2
   - pip install pytest-cov codecov

CHANGES.txt

@@ -35,6 +35,8 @@ Features:
 
 - Dockerfile build allows adding additional python packages via
   pip, setting UID tracker is run under. (John Rouillard)
+- issue2551140 - Added redis as a session and otk database for use
+  with anydbm and sqlite primary databases. (John Rouillard)
 
 2022-07-13 2.2.0
 

doc/admin_guide.txt

@@ -443,8 +443,94 @@ the following SQL commands::
    delete from __words;
    delete from __textids;
 
-Note this deletes data from the tables and does *not* delete the
-table.
+Note this deletes data from the tables and does *not* delete
+the table. This allows you to revert to Roundup's native
+full text indexing on SQLite or Postgres. If you were to
+delete the tables, Roundup will not recreate the
+tables. Under PostgreSQL, you can use the ``truncate
+<tablename>`` command if you wish.
+
+Configuring Session Databases
+=============================
+
+The session and OTK (one time key) databases
+store information about the operation of Roundup.
+This ephemeral data:
+
+* web login session keys,
+* CSRF tokens,
+* email password recovery one time keys,
+* rate limiting data,
+* ...
+
+can be a performance bottleneck. It usually happens with
+anydbm or SQLite backends. PostgreSQL and MySQL are
+sufficiently powerful that they can handle the higher
+transaction rates.
+
+If you are using sqlite, you can choose to use the anydbm
+database for session data. By default it will use additional
+sqlite databases for storing the session and otk data.
+
+The following table shows which primary databases support
+different session database backends::
+
+
+  main\/ session>| anydbm | sqlite | redis | mysql | postgresql |
+  anydbm         |    D   |        |   X   |       |            |
+  sqlite         |    X   |    D   |   X   |       |            |
+  mysql          |        |        |       |   D   |            |
+  postgresql     |        |        |       |       |      D     |
+  --------------------------------------------------------------+
+        D - default if unconfigured,   X - compatible choice
+
+The ``backend`` setting is in the tracker's ``config.ini``
+file under the ``sessiondb`` section.
+
+Using Redis for Session Databases
+---------------------------------
+
+Redis is an in memory key/value data structure store.
+
+To use redis you must be using Python 3.6 or newer.  You
+need to install the redis-py_ module from pypi. Then install
+Redis using your package manager or by downloading it from
+the Redis_ website.
+
+You need to secure your redis instance. The data that
+Roundup stores includes session cookies and other
+authentication tokens. At minimum you should require a
+password to connect to your redis database. Set
+``requirepass`` in ``redis.conf``. Then change the
+``redis_url`` in ``config.ini`` to use the password.
+
+
+For example::
+
+   redis://:mypassword@localhost:7200/10
+
+will connect to the redis instance running on localhost at
+port 7200 using the password ``mypassword`` to open database
+10. The ``redis_url`` setting can load a file to better
+secure the url. If you are using redis 6.0 or newer, you can
+specify a username/password and access control lists to
+improv the security of your data. Another good alternative
+is to talk to redis using a Unix domain socket.
+
+If you are connecting to redis across the network rather
+than on localhost, you should configure ssl/tls and use the
+``rediss`` scheme in the url along with the query
+parameters::
+
+	ssl_cert_reqs=required&ssl_ca_certs=/path/to/custom/ca-cert
+
+where you specify the file that can be used to validate the
+SSL certificate. `Securing Redis`_ has more details.
+
+.. _Redis: https://redis.io
+.. _redis-py: https://pypi.org/project/redis/
+.. _Securing Redis: https://redis.io/docs/manual/security/
+
 
 Users and Security
 ==================
@@ -579,7 +665,7 @@ Steps you may take:
     python setup.py install
 
 6. Follow the steps in the `upgrading documentation`_ for all the
-   version between your original version and th new version.
+   versions between your original version and the new version.
 
    Usually you should run `roundup_admin -i <tracker_home> migrate`
    on your tracker(s) before you allow users to start accessing the tracker.

doc/installation.txt

@@ -137,10 +137,20 @@ zstd, brotli
   compress the response from roundup as they transmit/proxy it to the
   client.
 
+redis
+  Storing ephemeral data: session keys, CSRF tokens etc. can be 
+  performance bottleneck. You can choose to deploy a Redis_ database
+  using the redis-py_ pypi package. See the section on
+  `Using Redis for Session Databases`_ in the `administration
+  guide`_ for details.
+
 Windows Service
   You can run Roundup as a Windows service if pywin32_ is installed.
   Otherwise it must be started manually.
 
+.. _Using Redis for Session Databases:
+   admin_guide.html#using-redis-for-session-databases
+
 Getting Roundup
 ===============
 
@@ -1770,6 +1780,8 @@ there are no errors. If there are errors, please let us know!
 .. _pysqlite: https://pysqlite.org/
 .. _pytz: https://pypi.org/project/pytz/
 .. _pywin32: https://pypi.org/project/pywin32/
+.. _Redis: https://redis.io
+.. _redis-py: https://pypi.org/project/redis/
 .. _Whoosh: https://whoosh.readthedocs.org/en/latest
 .. _Xapian: https://xapian.org/
 .. _zstd: https://pypi.org/project/zstd/

doc/upgrading.txt

@@ -35,6 +35,18 @@ Contents:
 Migrating from 2.2.0 to 2.3.0
 =============================
 
+Update your ``config.ini`` (required)
+-------------------------------------
+
+Upgrade tracker's config.ini file. Use::
+
+  roundup-admin -i /path/to/tracker updateconfig newconfig.ini
+
+to generate a new ini file preserving all your settings.
+You can then merge any local comments from the tracker's
+``config.ini`` to ``newconfig.ini`` and replace
+``config.ini`` with ``newconfig.ini``.
+
 Rdbms version change from 7 to 8 (required)
 -------------------------------------------
 
@@ -67,8 +79,8 @@ RDBMS backend) or "No migration action required" (if you
 have run it, or have used another interface to the tracker,
 or are using anydbm).
 
-Session/OTK data storage for SQLite backend changed
----------------------------------------------------
+Session/OTK data storage for SQLite backend changed (required)
+--------------------------------------------------------------
 
 Roundup stores a lot of ephemeral data:
 
@@ -83,17 +95,35 @@ is stored in a SQLite db. Using both dbm and sqlite style
 files is surprising and due to how we lock dbm files can be
 a performance issue.
 
-In this release two sqlite databases called ``db-otk`` and
-``db-session`` replace the dbm databases. Once you make the
-change the old ``otks`` and ``sessions`` dbm databases can
-be removed.
+However you can continue to use the dbm files by setting the
+``backend`` option in the ``[sessiondb]`` section of
+``config.ini`` to ``anydbm``.
+
+If you do not change the setting, two sqlite databases
+called ``db-otk`` and ``db-session`` replace the dbm
+databases. Once you make the change the old ``otks`` and
+``sessions`` dbm databases can be removed.
 
 Note this replacement will require users to log in again and
-refresh web pages to save data. It is best is people save
+refresh web pages to save data. It is best if people save
 all their changes and log out of Roundup before the upgrade
 is done to minimize confusion. Because the data is
 ephemeral, there is no plan to migrate this data to the new
-SQLite databases.
+SQLite databases. If you want to keep using the data set the
+``sessiondb`` ``backend`` option as described above.
+
+Session/OTK data storage using Redis (optional)
+-----------------------------------------------
+
+If you are running Roundup with Python 3.6 or newer, you can
+store your ephemeral data in a Redis database. This provides
+significantly better performance for ephemeral data than 
+SQLite or dbm files. See the section `Using Redis for Session
+Databases`_ in the `administration guide`_
+
+
+.. _Using Redis for Session Databases:
+    admin_guide.html#using-redis-for-session-databases
 
 .. index:: Upgrading; 2.1.0 to 2.2.0
 
@@ -107,11 +137,10 @@ Upgrade tracker's config.ini file. Use::
 
   roundup-admin -i /path/to/tracker updateconfig newconfig.ini
 
-to generate a new ini file preserving all your settings. You
-can then merge any local comments from the tracker's
-``config.ini`` into ``newconfig.ini``. You can then merge
-comments from ``config.ini`` to ``newconfig.ini`` and
-replace ``config.ini`` with ``newconfig.ini``.
+to generate a new ini file preserving all your settings.
+You can then merge any local comments from the tracker's
+``config.ini`` to ``newconfig.ini`` and replace
+``config.ini`` with ``newconfig.ini``.
 
 Rdbms version change from 6 to 7 (required)
 -------------------------------------------

roundup/backends/back_anydbm.py

@@ -33,7 +33,12 @@
 from roundup.i18n import _
 
 from roundup.backends.blobfiles import FileStorage
-from roundup.backends.sessions_dbm import Sessions, OneTimeKeys
+from roundup.backends import sessions_dbm
+
+try:
+    from roundup.backends import sessions_redis
+except ModuleNotFoundError:
+    sessions_redis = None
 
 from roundup.backends.indexer_common import get_indexer
 
@@ -133,12 +138,26 @@ def refresh_database(self):
 
     def getSessionManager(self):
         if not self.Session:
-            self.Session = Sessions(self)
+            if self.config.SESSIONDB_BACKEND == "redis":
+                if sessions_redis is None:
+                    self.Session = sessions_dbm.Sessions(self)
+                    raise ValueError("[redis] session is set, but "
+                                     "redis is not found")
+                self.Session = sessions_redis.Sessions(self)
+            else:
+                self.Session = sessions_dbm.Sessions(self)
         return self.Session
 
     def getOTKManager(self):
         if not self.Otk:
-            self.Otk = OneTimeKeys(self)
+            if self.config.SESSIONDB_BACKEND == "redis":
+                if sessions_redis is None:
+                    self.Session = sessions_dbm.OneTimeKeys(self)
+                    raise ValueError("[redis] session is set, but "
+                                     "redis is not found")
+                self.Otk = sessions_redis.OneTimeKeys(self)
+            else:
+                self.Otk = sessions_dbm.OneTimeKeys(self)
         return self.Otk
 
     def reindex(self, classname=None, show_progress=False):

roundup/backends/back_sqlite.py

@@ -12,7 +12,14 @@
 
 from roundup import hyperdb, date, password
 from roundup.backends import rdbms_common
-from roundup.backends.sessions_sqlite import Sessions, OneTimeKeys
+from roundup.backends import sessions_sqlite
+from roundup.backends import sessions_dbm
+
+try:
+    from roundup.backends import sessions_redis
+except ModuleNotFoundError:
+    sessions_redis = None
+
 from roundup.anypy.strings import uany2s
 
 sqlite_version = None
@@ -105,12 +112,30 @@ class Database(rdbms_common.Database):
     # connections to the same database which SQLite doesn't support
     def getSessionManager(self):
         if not self.Session:
-            self.Session = Sessions(self)
+            if self.config.SESSIONDB_BACKEND == "redis":
+                if sessions_redis is None:
+                    self.Session = sessions_sqlite.Sessions(self)
+                    raise ValueError("[redis] session is set, but "
+                                     "redis module is not found")
+                self.Session = sessions_redis.Sessions(self)
+            elif self.config.SESSIONDB_BACKEND == "anydbm":
+                self.Session = sessions_dbm.Sessions(self)
+            else:
+                self.Session = sessions_sqlite.Sessions(self)
         return self.Session
 
     def getOTKManager(self):
         if not self.Otk:
-            self.Otk = OneTimeKeys(self)
+            if self.config.SESSIONDB_BACKEND == "redis":
+                if sessions_redis is None:
+                    self.Session = sessions_sqlite.OneTimeKeys(self)
+                    raise ValueError("[redis] session is set, but "
+                                     "redis is not found")
+                self.Otk = sessions_redis.OneTimeKeys(self)
+            elif self.config.SESSIONDB_BACKEND == "anydbm":
+                self.Otk = sessions_dbm.OneTimeKeys(self)
+            else:
+                self.Otk = sessions_sqlite.OneTimeKeys(self)
         return self.Otk
 
     def sqlite_busy_handler(self, data, table, count):