feat: add support for using dictConfig to configure logging.

Basic logging config (one level and one output file non-rotating) was
always possible from config.ini. However the LOGGING_CONFIG setting
could be used to load an ini fileConfig style file to set various
channels (e.g. roundup.hyperdb) (also called qualname or tags) with
their own logging level, destination (rotating file, socket,
/dev/null) and log format.

This is now a deprecated method in newer logging modules. The
dictConfig format is preferred and allows disabiling other loggers as
well as invoking new loggers in local code. This commit adds support
for it reading the dict from a .json file. It also implements a
comment convention so you can document the dictConfig.

configuration.py:
 new code

test_config.py:
 test added for the new code.

admin_guide.txt, upgrading.txt CHANGES.txt:
  docs added upgrading references the section in admin_guid.

Author: rouilj

CHANGES.txt

@@ -32,6 +32,8 @@ Features:
 - add support for authorized changes. User can be prompted to enter
   their password to authorize a change. If the user's password is
   properly entered, the change is committed. (John Rouillard)
+- add support for dictConfig style logging configuration. Ini/File
+  style configs will still be supported. (John Rouillard)
 
 2025-07-13 2.5.0
 

doc/admin_guide.txt

@@ -47,31 +47,284 @@ There's two "installations" that we talk about when using Roundup:
    in the tracker's config.ini.
 
 
-Configuring Roundup's Logging of Messages For Sysadmins
-=======================================================
+Configuring Roundup Message Logging
+===================================
 
-You may configure where Roundup logs messages in your tracker's config.ini
-file. Roundup will use the standard Python (2.3+) logging implementation.
+You can control how Roundup logs messages using your tracker's
+config.ini file. Roundup uses the standard Python (2.3+) logging
+implementation.  The config file and ``roundup-server`` provide very
+basic control over logging.
 
-Configuration for standard "logging" module:
- - tracker configuration file specifies the location of a logging
-   configration file as ``logging`` -> ``config``
- - ``roundup-server`` specifies the location of a logging configuration
-   file on the command line
 Configuration for "BasicLogging" implementation:
  - tracker configuration file specifies the location of a log file
    ``logging`` -> ``filename``
  - tracker configuration file specifies the level to log to as
    ``logging`` -> ``level``
+ - tracker configuration file lets you disable other loggers
+   (e.g. when running under a wsgi framework) with
+   ``logging`` -> ``disable_loggers``.
  - ``roundup-server`` specifies the location of a log file on the command
    line
- - ``roundup-server`` specifies the level to log to on the command line
+ - ``roundup-server`` enable  using the standard python logger with
+    the tag/channel ``roundup.http`` on the command line
+
+By supplying a standard log config file in ini or json (dictionary)
+format, you get more control over the logs. You can set different
+levels for logs (e.g. roundup.hyperdb can be set to WARNING while
+other Roundup log channels are set to INFO and roundup.mailgw logs at
+DEBUG level). You can also send the logs for roundup.mailgw to syslog,
+and other roundup logs go to an automatically rotating log file, or
+are submitted to your log aggregator over https.
 
-(``roundup-mailgw`` always logs to the tracker's log file)
+Configuration for standard "logging" module:
+ - tracker configuration file specifies the location of a logging
+   configuration file as ``logging`` -> ``config``.
 
 In both cases, if no logfile is specified then logging will simply be sent
 to sys.stderr with only logging of ERROR messages.
 
+Standard Logging Setup
+----------------------
+
+You can specify your log configs in one of two formats:
+
+  * `fileConfig format
+    <https://docs.python.org/3/library/logging.config.html#logging.config.fileConfig>`_
+    in ini style
+  * `dictConfig format
+    <https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig>`_
+    using json with comment support
+
+The dictConfig allows more control over configuration including
+loading your own log handlers and disabling existing handlers. If you
+use the fileConfig format, the ``logging`` -> ``disable_loggers`` flag
+in the tracker's config is used to enable/disable pre-existing loggers
+as there is no way to do this in the logging config file.
+
+.. _`dictLogConfig`:
+
+dictConfig Based Logging Config
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+dictConfigs are specified in JSON format with support for comments.
+The file name in the tracker's config for the ``logging`` -> ``config``
+setting must end with ``.json`` to choose the correct processing.
+
+Comments have to be in one of two forms:
+
+1. A ``#`` with preceding white space is considered a comment and is
+   stripped from the file before being passed to the json parser. This
+   is a "block comment".
+
+2. A ``#`` preceded by at least three
+   white space characters is stripped from the end of the line before
+   begin passed to the json parser. This is an "inline comment".
+
+Other than this the file is a standard json file that matches the
+`Configuration dictionary schema
+<https://docs.python.org/3/library/logging.config.html#configuration-dictionary-schema>`_
+defined in the Python documentation.
+
+
+Example dictConfig Logging Config
+.................................
+
+Note that this file is not actually JSON format as it include comments.
+So you can not use tools that expect JSON (linters, formatters) to
+work with it.
+
+The config below works with the `Waitress wsgi server
+<https://github.com/Pylons/waitress>`_ configured to use the
+roundup.wsgi channel. It also controls the `TransLogger middleware
+<https://github.com/pasteorg/paste>`_ configured to use
+roundup.wsgi.translogger, to produce httpd style combined logs. The
+log file is specified relative to the current working directory not
+the tracker home. The tracker home is the subdirectory demo under the
+current working directory. The commented config is::
+
+  {
+    "version": 1,   # only supported version
+    "disable_existing_loggers": false,      # keep the wsgi loggers
+
+    "formatters": {
+      # standard format for Roundup messages
+      "standard": {
+        "format": "%(asctime)s %(levelname)s %(name)s:%(module)s %(msg)s"
+      },
+      # used for waitress wsgi server to produce httpd style logs
+      "http": {
+	"format": "%(message)s"
+      }
+    },
+    "handlers": {
+      # create an access.log style http log file
+      "access": {
+	"level": "INFO",
+	"formatter": "http",
+	"class": "logging.FileHandler",
+	"filename": "demo/access.log"
+      },
+      # logging for roundup.* loggers
+      "roundup": {
+	"level": "DEBUG",
+	"formatter": "standard",
+	"class": "logging.FileHandler",
+	"filename": "demo/roundup.log"
+      },
+      # print to stdout - fall through for other logging
+      "default": {
+	"level": "DEBUG",
+	"formatter": "standard",
+	"class": "logging.StreamHandler",
+	"stream": "ext://sys.stdout"
+      }
+  },
+    "loggers": {
+      "": {
+	"handlers": [
+	  "default"
+	],
+	"level": "DEBUG",
+	"propagate": false
+      },
+      # used by roundup.* loggers
+      "roundup": {
+	"handlers": [
+	  "roundup"
+	],
+	"level": "DEBUG",
+	"propagate": false   # note pytest testing with caplog requires
+			     # this to be true
+      },
+      "roundup.hyperdb": {
+	"handlers": [
+	  "roundup"
+	],
+	"level": "INFO",    # can be a little noisy use INFO for production
+	"propagate": false
+      },
+     "roundup.wsgi": {    # using the waitress framework
+	"handlers": [
+	  "roundup"
+	],
+	"level": "DEBUG",
+	"propagate": false
+      },
+     "roundup.wsgi.translogger": {   # httpd style logging
+	"handlers": [
+	  "access"
+	],
+	"level": "DEBUG",
+	"propagate": false
+      },
+     "root": {
+	"handlers": [
+	  "default"
+	],
+	"level": "DEBUG",
+	"propagate": false
+      }
+    }
+  }
+
+fileConfig Based Logging Config
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The file config is an older and more limited method of configuring
+logging. It is described by the `Configuration file format
+<https://docs.python.org/3/library/logging.config.html#configuration-file-format>`_
+in the Python documentation. The file name in the tracker's config for
+the ``logging`` -> ``config`` setting must end with ``.ini`` to choose
+the correct processing.
+
+Example fileConfig LoggingConfig
+................................
+
+This is an example .ini used with roundup-server configured to use
+``roundup.http`` channel. It also includes some custom logging
+qualnames/tags/channels for logging schema/permission detector and
+extension output::
+
+  [loggers]
+  #other keys: roundup.hyperdb.backend
+  keys=root,roundup,roundup.http,roundup.hyperdb,actions,schema,extension,detector
+
+  [logger_root]
+  #also for root where channlel is not set (NOTSET) aka all
+  level=DEBUG
+  handlers=rotate
+
+  [logger_roundup]
+  # logger for all roundup.* not otherwise configured
+  level=DEBUG
+  handlers=rotate
+  qualname=roundup
+  propagate=0
+
+  [logger_roundup.http]
+  level=INFO
+  handlers=rotate_weblog
+  qualname=roundup.http
+  propagate=0
+
+  [logger_roundup.hyperdb]
+  level=WARNING
+  handlers=rotate
+  qualname=roundup.hyperdb
+  propagate=0
+
+  [logger_actions]
+  level=INFO
+  handlers=rotate
+  qualname=actions
+  propagate=0
+
+  [logger_detector]
+  level=INFO
+  handlers=rotate
+  qualname=detector
+  propagate=0
+
+  [logger_schema]
+  level=DEBUG
+  handlers=rotate
+  qualname=schema
+  propagate=0
+
+  [logger_extension]
+  level=INFO
+  handlers=rotate
+  qualname=extension
+  propagate=0
+
+  [handlers]
+  keys=basic,rotate,rotate_weblog
+
+  [handler_basic]
+  class=StreamHandler
+  args=(sys.stderr,)
+  formatter=basic
+
+  [handler_rotate]
+  class=logging.handlers.RotatingFileHandler
+  args=('roundup.log','a', 5120000, 2)
+  formatter=basic
+
+  [handler_rotate_weblog]
+  class=logging.handlers.RotatingFileHandler
+  args=('httpd.log','a', 1024000, 2)
+  formatter=plain
+
+  [formatters]
+  keys=basic,plain
+
+  [formatter_basic]
+  format=%(asctime)s %(process)d %(name)s:%(module)s.%(funcName)s,%(levelname)s: %(message)s
+  datefmt=%Y-%m-%d %H:%M:%S
+
+  [formatter_plain]
+  format=%(process)d %(message)s
+
 
 Configuring roundup-server
 ==========================

doc/upgrading.txt

@@ -133,6 +133,21 @@ date, text etc.) do not need JavaScript to work.
 
 See :ref:`Confirming the User` in the reference manual for details.
 
+Support for dictConfig Logging Configuration (optional)
+-------------------------------------------------------
+
+Roundup's basic log configuration via config.ini has always had the
+ability to use an ini style logging configuration to set levels per
+log channel, control output file rotation etc.
+
+With Roundup 2.6 you can use a JSON like file to configure logging
+using `dictConfig
+<https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig>`_. The
+JSON file format as been enhanced to support comments that are
+stripped before being processed by the logging system.
+
+You can read about the details in the :ref:`admin manual <dictLogConfig>`.
+
 .. index:: Upgrading; 2.4.0 to 2.5.0
 
 Migrating from 2.4.0 to 2.5.0