Merge

Author: schlatterbeck

.github/workflows/ci-test.yml

@@ -3,7 +3,8 @@
 # reference docs:
 # https://blog.deepjyoti30.dev/tests-github-python
 # https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
-#
+# https://github.com/pypa/twine/blob/main/.github/workflows/main.yml
+
 name: roundup-ci
 
 on: 
@@ -44,30 +45,36 @@ jobs:
       max-parallel: 4
       matrix:
         # Run in all these versions of Python
-        python-version: [ "2.7", "3.10", "3.9", "3.8", "3.6", "3.11-dev" ]
+        python-version:
+          - "2.7"
+          - "3.10"
+          - "3.9"
+          - "3.8"
+          - "3.7"
+          - "3.11"
 
         # use for multiple os or ubuntu versions
         #os: [ubuntu-latest, macos-latest, windows-latest]
-        os: [ubuntu-latest, ubuntu-22.04]
+        # ubuntu latest 22.04 12/2022
+        os: [ubuntu-latest, ubuntu-20.04]
 
         # if the ones above fail. fail the build
         experimental: [false]
 
         include:
-           # example: if 3.12 fails the jobs still succeeds
-        #   - python-version: 3.12
-        #     experimental: [true]
-           # version 2.7 not available on unbuntu-22.04 github
-        #    - python-version: 2.7
-        #      os: ubuntu-22.04
-        #      experimental: true
-            - python-version: 3.11-dev
+            # example: if 3.12 fails the jobs still succeeds
+            - python-version: 3.12-dev
               os: ubuntu-22.04
               experimental: [true]
+            - python-version: 3.11-dev
+              os: ubuntu-20.04
+            # 3.6 not available on new 22.04 runners, so run on 20.04 ubuntu
+            - python-version: 3.6
+              os: ubuntu-20.04
 
         exclude:
-            # skip all python versions on 22.04 except explicitly included
-            - os: ubuntu-22.04
+            # skip all python versions on 20.04 except explicitly included
+            - os: ubuntu-20.04
 
     env:
       # get colorized pytest output even without a controlling tty

CHANGES.txt

@@ -50,6 +50,11 @@ Fixed:
   more than one issue with a matching parent message, fall back to
   subject matching. See upgrading.txt for details. (John Rouillard)
 - issue2551195 - port scripts from optparse to argparse (Ralf Schlatterbeck)
+- issue2551246 - mitigation, document how -u doesn't work for
+  roundup-admin. (John Rouillard)
+- Document better that files in the template or static_files
+  directories accessed via @@file are available to any user with the
+  url.
 
 Features:
 
@@ -182,6 +187,9 @@ Fixed:
   if the user doesn't have edit permissions. (John Rouillard)
 - issue2551216 - create new mysql databases using COLLATE
   utf8_general_ci to prevent crashes in test suite. (John Rouillard)
+- issue2551146 - fix issues with strings that have multiple %s
+  substutions that were not labeled making i18n difficult/impossible.
+  (John Rouillard)
 
 Features:
 

doc/admin_guide.txt

@@ -878,6 +878,22 @@ A brief (incomplete) summary is::
 
 Run ``roundup-admin help commands`` for a complete list of subcommands.
 
+One thing to note, The ``-u user`` setting does not currently operate
+like a user logging in via the web. The user running roundup-admin
+must have read access to the tracker home directory. As a result the
+user has access to the files and the database info contained in
+config.ini.
+
+Using ``-u user`` sets the actor/user parameter in the
+journal. Changes that are made are attributed to that
+user. The password is ignored if provided. Any existing
+username has full access to the data just like the admin
+user. This is an area for further development so that
+roundup-admin could be used with sudo to provide secure
+command line access to a tracker.
+
+In general you should forget that there is a -u parameter.
+
 .. _`customisation documentation`: customizing.html
 .. _`upgrading documentation`: upgrading.html
 .. _`installation documentation`: installation.html

doc/customizing.txt

@@ -2309,6 +2309,16 @@ Serving static content
 See the previous section `determining web context`_ where it describes
 ``@@file`` paths.
 
+These files are served without any permission checks. Any user on the
+internet with the url can download the file.
+
+This is rarely an issue since the html templates are just source code
+and much of it can be found in the Roundup repository. Other
+decoration (logos, stylesheets) are similarly not security sensitive.
+You can use the static_files setting in config.ini to eliminate
+access to the templates directory if desired.
+
+If a file resolves to a symbolic link, it is not served.
 
 Performing actions in web requests
 ----------------------------------

doc/developers.txt

@@ -4,73 +4,120 @@ Developing Roundup
 
 .. note::
    The intended audience of this document is the developers of the core
-   Roundup code. If you just wish to alter some behaviour of your Roundup
-   installation, see `customising roundup`_.
+   Roundup code. If you just wish to alter some behavior of your Roundup
+   installation, see `Customising Roundup`_.
 
 Contents
 
 .. contents::
    :local:
 
-If you are looking for info on managing the roundup-tracker.org
-infrastructure, that information has migrated to website/README.txt
-in the roundup repo.
 
 Getting Started
 ---------------
 
-Anyone wishing to help in the development of Roundup must read `Roundup's
-Design Document`_ and the `implementation notes`_.
+If you are looking for a good first issue, search for `StarterTicket
+on https://issues.Roundup-tracker.org`_. These include issues where
+Python development, Documentation or Web design skills are useful.
+
+You can continue the conversation using the issue or join the
+Roundup-devel list to get assistance and verify your planned changes.
 
 All development is coordinated through two resources:
 
-- roundup-devel mailing list at
-  https://sourceforge.net/projects/roundup/lists/roundup-devel
+- Roundup-devel mailing list at
+  https://sourceforge.net/projects/Roundup/lists/Roundup-devel
 - The issue tracker running at
-  https://issues.roundup-tracker.org/
+  https://issues.Roundup-tracker.org/
 
+In addition, the Roundup IRC channel on irc.oftc.net can be accessed
+via the web interface shown on the Contact page. The channel is logged
+and the web sites for the logs are shown in the channel topic. You can
+ask questions and use it to coordinate work discussed using the
+resources above.
 
-Small Changes
--------------
+Anyone wishing to help in the development of the Roundup Python core
+may find `Roundup's Design Document`_ and the `implementation notes`_
+helpful.
 
-Most small changes can be submitted as patches through the
-`issue tracker`_ or sent to `roundup-devel mailing list`_.
+People working on Documentation or designing changes to the Web
+interface don't need to get into the implementation internals.
 
 
-Source Repository Access
-------------------------
+Small Changes
+-------------
 
-See https://www.roundup-tracker.org/code.html.
-For all other questions ask on the development mailinglist.
+Most small changes can be submitted as patches through the
+`issue tracker`_ or sent to `Roundup-devel mailing list`_.
 
 
 Project Rules
 -------------
 
-Mostly the project follows Guido's Style (though naming tends to be a little
-relaxed sometimes). In short:
+Be polite to others. There is no place for ad hominem attacks.
+
+Mostly the project follows Guido's Python Style (though naming tends
+to be a little relaxed sometimes). In short:
 
 - 80 column width code
 - 4-space indentations
-- All modules must have an Id line near the top
 
 Other project rules:
 
-- New functionality must be documented, even briefly (so at least we know
-  where there's missing documentation) and changes to tracker configuration
-  must be logged in the upgrading document.
-- subscribe to roundup-checkins to receive checkin notifications from the
-  other developers with write access to the source-code repository.
-- discuss any changes with the other developers on roundup-dev. If nothing
-  else, this makes sure there's no rude shocks
-- write unit tests for changes you make (where possible), and ensure that
-  all unit tests run before committing changes
-- run pychecker over changed code
+- new functionality must be documented, even briefly (so at
+  least we know where there's missing documentation) and
+  changes to tracker configuration must be logged in the
+  upgrading document.
+- discuss any changes with the other developers on
+  Roundup-dev. If nothing else, this makes sure there's no
+  rude shocks.
+- write unit tests for changes you make (where possible),
+  and ensure that all unit tests run before committing
+  changes.
+- run flake8_ or pylint_ over changed code.
+- if you have direct commit access to the repository,
+  subscribe to Roundup-checkins to receive checkin
+  notifications from the other developers with write access
+  to the source-code repository.
+
+The goal is to have no flake8 issues. Current code does include long
+lines and use of mutable objects in function signatures. Some third
+party code (e.g. ZPT) vendored into the codebase has more issues.
 
 The administrators of the project reserve the right to boot developers who
 consistently check in code which is either broken or takes the codebase in
 directions that have not been agreed to.
 
+Source Repository Access
+------------------------
+
+Roundup is developed using the `Mercurial distributed version control
+system (DVCS)`_ [1]_. It is `hosted at Sourceforge`_. See
+https://www.Roundup-tracker.org/code.html for details.
+For all other questions ask on the development mailing list.
+
+Other Resources - CI, Code Coverage 
+-----------------------------------
+
+Roundup has a `copy of the mercurial repository on GitHub`_. It is
+updated manually after every few commits to the Mercurial
+repository. Updates trigger the CI pipeline which happens on two
+services:
+
+  1. `GitHub Actions`_. It runs Docker container scans using Anchore as
+     well as security scans for dependencies using CodeQL. Also it
+     runs the test suite on multiple versions of Python.
+  2. `TravisCI`_ is also used to run CI. It runs the test suite on
+     multiple Python versions. It also provides alpha and development
+     Python releases faster than GitHub.
+
+GitHub actions upload coverage statistics to both `CodeCov`_ and
+`Coveralls`_. TravisCI only uploads to CodeCov.
+
+We run our own issue tracker so we can dogfood the code. So we do not
+use GitHub issues. Pull requests are grudgingly accepted. They have to
+be exported and applied to the Mercurial repository. This is time
+consuming so patches attached to the issue are preferred.
 
 Debugging Aids
 --------------
@@ -126,7 +173,7 @@ GNU gettext package
 
 This chapter is full of references to GNU `gettext package`_.
 GNU gettext is a "must have" for nearly all steps of internationalizing
-any program, and it's manual is definetely a recommended reading
+any program, and it's manual is definitely a recommended reading
 for people involved in `I18N`_.
 
 There are GNU gettext ports to all major OS platforms.
@@ -269,7 +316,7 @@ Template markup examples:
   will translate the caption (and return value) for the "wink" button.
 
 * explicit msgids.  Sometimes it may be useful to specify msgid
-  for the element translation explicitely, like this::
+  for the element translation explicitly, like this::
 
     <span i18n:translate="know what i mean?">this text is ignored</span>
 
@@ -400,6 +447,11 @@ Translatable Messages`_.)
 At run time, Roundup automatically compiles message catalogs whenever
 `PO`_ file is changed.
 
+.. [1] Roundup is written in Python and we believe in using tools in
+   the Python ecosystem whenever possible.
+
+
+
 .. _`Customising Roundup`: customizing.html
 .. _`Roundup's Design Document`: spec.html
 .. _`implementation notes`: implementation.html
@@ -409,33 +461,47 @@ At run time, Roundup automatically compiles message catalogs whenever
 
 .. _alexander smishlajev:
 .. _als: https://sourceforge.net/u/a1s/profile/
+.. _CodeCov: https://app.codecov.io/gh/roundup-tracker/roundup
+.. _copy of the mercurial repository on GitHub:
+    https://github.com/roundup-tracker/roundup
+.. _Coveralls: https://coveralls.io/github/roundup-tracker/roundup
 .. _cygwin: https://www.cygwin.com/
 .. _emacs: https://www.gnu.org/software/emacs/
-.. _gettext package: http://www.gnu.org/savannah-checkouts/gnu/gettext/manual/gettext.html
+.. _flake8: https://flake8.pycqa.org/en/latest/
+.. _gettext package: https://www.gnu.org/savannah-checkouts/gnu/gettext/manual/gettext.html
 .. _gettext module: https://docs.python.org/2/library/gettext.html
-.. _GNU: http://www.gnu.org/
-.. _GNU mirror sites: http://www.gnu.org/prep/ftp.html
+.. _GitHub Actions: https://github.com/roundup-tracker/roundup/actions
+.. _GNU: https://www.gnu.org/
+.. _GNU mirror sites: https://www.gnu.org/prep/ftp.html
+.. _hosted at sourceforge:
+   https://sourceforge.net/p/roundup/code/ci/default/tree/
 .. _issue tracker: https://issues.roundup-tracker.org/
 .. _Lokalize: https://apps.kde.org/lokalize/
 .. _KDE: https://kde.org/
 .. _linux: https://www.linux.org/
+.. _Mercurial distributed version control system (DVCS):
+   https://www.mercurial-scm.org/
 .. _Plural Forms:
-    http://www.gnu.org/savannah-checkouts/gnu/gettext/manual/gettext.html
+   https://www.gnu.org/savannah-checkouts/gnu/gettext/manual/gettext.html
 .. _po filetype plugin:
     https://vim.sourceforge.io/scripts/script.php?script_id=695
 .. _PO utilities: https://github.com/pinard/po-utils
 .. _poEdit: https://poedit.net/
+.. _pylint: https://pylint.pycqa.org/en/latest/
 .. _Roundup Source:
 .. _Roundup source distribution:
 .. _Roundup binary distribution:
     https://sourceforge.net/projects/roundup/files/
 .. _roundup-devel mailing list:
    https://sourceforge.net/projects/roundup/lists/roundup-devel
+.. _StarterTicket on https://issues.roundup-tracker.org:
+   https://issues.roundup-tracker.org/issue?@columns=title,id,activity,status&@sort=activity&@filter=status,keywords&status=-1,1,2&keywords=15&@dispname=Starter%20tickets
 .. _TAL:
 .. _Template Attribute Language:
    https://pagetemplates.readthedocs.io/en/latest/history/TALSpecification14.html
 .. _TALES:
 .. _Template Attribute Language Expression Syntax:
    https://pagetemplates.readthedocs.io/en/latest/history/TALESSpecification13.html
+.. _TravisCI: https://app.travis-ci.com/github/roundup-tracker/roundup
 .. _vim: https://www.vim.org/
 .. _ZPTInternationalization: http://grok.zope.org/documentation/how-to/how-to-internationalize-your-application/view

doc/index.txt

@@ -17,23 +17,25 @@ Contents
    features
    installation
    upgrading
+   security
    FAQ
    user_guide
    customizing
    admin_guide
+   security
    debugging
    xmlrpc
    rest
-   overview
+   tracker_templates
+   glossary
+   acknowledgements
+   license
+   Design Overview <overview>
    Design (original) <design>
    developers
-   tracker_templates
    Notes about the MySQL Database backend <mysql>
    Notes about the PostgreSQL Database backend <postgresql>
-   glossary
-   acknowledgements
    Richard Jones implementation notes <implementation>
-   license
 
 See: https://wiki.roundup-tracker.org/ReleaseErrata for fixes to
 documentation.