mailgw subject parsing has configurable levels of strictness

nosy messages may be sent individually to all recipients

Author: Richard Jones

CHANGES.txt

@@ -1,14 +1,16 @@
 This file contains the changes to the Roundup system over time. The entries
 are given with the most recent entry first.
 
-????-??-?? 0.9.0
+2005-10-07 0.9.0b1
 Feature:
 - added "imapServer.py" script (sf patch 934567)
 - added date selection popup windows (thanks Marcus Priesch)
 - added Xapian indexer; replaces standard indexers if Xapian is available
+- mailgw subject parsing has configurable levels of strictness
+- nosy messages may be sent individually to all recipients
 
 
-2005-??-?? 0.8.5
+2005-10-07 0.8.5
 Feature:
 - Argentinian Spanish translation by Ramiro Morales
 

doc/customizing.txt

@@ -2,7 +2,7 @@
 Customising Roundup
 ===================
 
-:Version: $Revision: 1.184 $
+:Version: $Revision: 1.185 $
 
 .. This document borrows from the ZopeBook section on ZPT. The original is at:
    http://www.zope.org/Documentation/Books/ZopeBook/current/ZPT.stx
@@ -255,6 +255,31 @@ Section **mailgw**
   Default class to use in the mailgw if one isn't supplied in email subjects.
   To disable, leave the value blank.
 
+ subject_prefix_parsing -- ``strict``
+  Controls the parsing of the [prefix] on subject lines in incoming emails.
+  ``strict`` will return an error to the sender if the [prefix] is not
+  recognised. ``loose`` will attempt to parse the [prefix] but just
+  pass it through as part of the issue title if not recognised. ``none``
+  will always pass any [prefix] through as part of the issue title.
+
+ subject_suffix_parsing -- ``strict``
+  Controls the parsing of the [suffix] on subject lines in incoming emails.
+  ``strict`` will return an error to the sender if the [suffix] is not
+  recognised. ``loose`` will attempt to parse the [suffix] but just
+  pass it through as part of the issue title if not recognised. ``none``
+  will always pass any [suffix] through as part of the issue title.
+
+ subject_suffix_delimiters -- ``[]``
+  Defines the brackets used for delimiting the commands suffix in a subject
+  line.
+
+ subject_content_match -- ``always``
+  Controls matching of the incoming email subject line against issue titles
+  in the case where there is no designator [prefix]. ``never`` turns off
+  matching. ``creation + interval`` or ``activity + interval`` will match
+  an issue for the interval after the issue's creation or last activity.
+  The interval is a standard Roundup interval.
+
 Section **nosy**
  Nosy messages sending
 
@@ -280,6 +305,11 @@ Section **nosy**
   on followups too. If ``no``, they're never added to the nosy.
   Allowed values: ``yes``, ``no``, ``new``
 
+ email_sending -- ``single``
+  Controls the email sending from the nosy reactor. If ``multiple`` then
+  a separate email is sent to each recipient. If ``single`` then a single
+  email is sent with each recipient as a CC address.
+
 You may generate a new default config file using the ``roundup-admin
 genconfig`` command.
 

doc/installation.txt

@@ -56,7 +56,7 @@ An RDBMS
 Xapian full-text indexer
   The Xapian_ full-text indexer is also supported and will be used by
   default if it is available. This is strongly recommended if you are
-  anticipating a large number of issues (> 1000).
+  anticipating a large number of issues (> 5000).
 
   You may install Xapian at any time, even after a tracker has been
   installed and used. You will need to run the "roundup-admin reindex"
@@ -246,6 +246,10 @@ create your tracker on.
 Choosing Your Backend
 ---------------------
 
+.. note::
+
+   The tsearch2 backend is experimental and should not be selected.
+
 The actual storage of Roundup tracker information is handled by backends.
 There's several to choose from, each with benefits and limitations:
 

roundup/configuration.py

@@ -1,6 +1,6 @@
 # Roundup Issue Tracker configuration support
 #
-# $Id: configuration.py,v 1.26 2005-06-24 05:22:03 richard Exp $
+# $Id: configuration.py,v 1.27 2005-10-07 04:42:13 richard Exp $
 #
 __docformat__ = "restructuredtext"
 
@@ -569,6 +569,33 @@ class NullableFilePathOption(NullableOption, FilePathOption):
             "Default locale name for the tracker mail gateway.\n"
             "If this option is not set, mail gateway will use\n"
             "the language of the tracker instance."),
+        (Option, "subject_prefix_parsing", "strict",
+            "Controls the parsing of the [prefix] on subject\n"
+            "lines in incoming emails. \"strict\" will return an\n"
+            "error to the sender if the [prefix] is not recognised.\n"
+            "\"loose\" will attempt to parse the [prefix] but just\n"
+            "pass it through as part of the issue title if not\n"
+            "recognised. \"none\" will always pass any [prefix]\n"
+            "through as part of the issue title."),
+        (Option, "subject_suffix_parsing", "strict",
+            "Controls the parsing of the [suffix] on subject\n"
+            "lines in incoming emails. \"strict\" will return an\n"
+            "error to the sender if the [suffix] is not recognised.\n"
+            "\"loose\" will attempt to parse the [suffix] but just\n"
+            "pass it through as part of the issue title if not\n"
+            "recognised. \"none\" will always pass any [suffix]\n"
+            "through as part of the issue title."),
+        (Option, "subject_suffix_delimiters", "[]",
+            "Defines the brackets used for delimiting the commands\n"
+            "suffix in a subject line."),
+        (Option, "subject_content_match", "always",
+            "Controls matching of the incoming email subject line\n"
+            "against issue titles in the case where there is no\n"
+            "designator [prefix]. \"never\" turns off matching.\n"
+            "\"creation + interval\" or \"activity + interval\"\n"
+            "will match an issue for the interval after the issue's\n"
+            "creation or last activity. The interval is a standard\n"
+            "Roundup interval."),
     ), "Roundup Mail Gateway options"),
     ("nosy", (
         (RunDetectorOption, "messages_to_author", "no",
@@ -592,6 +619,11 @@ class NullableFilePathOption(NullableOption, FilePathOption):
             "If 'yes', then the recipients will be added on followups too.\n"
             "If 'no', they're never added to the nosy.\n",
             ["ADD_RECIPIENTS_TO_NOSY"]),
+        (Option, "email_sending", "single",
+            "Controls the email sending from the nosy reactor. If\n"
+            "\"multiple\" then a separate email is sent to each\n"
+            "recipient. If \"single\" then a single email is sent with\n"
+            "each recipient as a CC address."),
     ), "Nosy messages sending"),
 )
 

roundup/mailgw.py

@@ -72,7 +72,7 @@ class node. Any parts of other types are each stored in separate files
 an exception, the original message is bounced back to the sender with the
 explanatory message given in the exception.
 
-$Id: mailgw.py,v 1.167 2005-09-28 05:48:23 richard Exp $
+$Id: mailgw.py,v 1.168 2005-10-07 04:42:13 richard Exp $
 """
 __docformat__ = 'restructuredtext'
 
@@ -297,19 +297,6 @@ def as_attachment(self):
 
 class MailGW:
 
-    # Matches subjects like:
-    # Re: "[issue1234] title of issue [status=resolved]"
-    subject_re = re.compile(r'''
-        (?P<refwd>\s*\W?\s*(fw|fwd|re|aw)\W\s*)*\s*   # Re:
-        (?P<quote>")?                                 # Leading "
-        (\[(?P<classname>[^\d\s]+)                    # [issue..
-           (?P<nodeid>\d+)?                           # ..1234]
-         \])?\s*
-        (?P<title>[^[]+)?                             # issue title
-        "?                                            # Trailing "
-        (\[(?P<args>.+?)\])?                          # [prop=value]
-        ''', re.IGNORECASE|re.VERBOSE)
-
     def __init__(self, instance, db, arguments=()):
         self.instance = instance
         self.db = db
@@ -611,6 +598,23 @@ def handle_message(self, message):
         # make local variable for easier access
         config = self.instance.config
 
+        # determine the sender's address
+        from_list = message.getaddrlist('resent-from')
+        if not from_list:
+            from_list = message.getaddrlist('from')
+
+        # check for registration OTK
+        # or fallback on the default class
+        otk_re = re.compile('-- key (?P<otk>[a-zA-Z0-9]{32})')
+        otk = otk_re.search(subject)
+        if otk:
+            self.db.confirm_registration(otk.group('otk'))
+            subject = 'Your registration to %s is complete' % \
+                      config['TRACKER_NAME']
+            sendto = [from_list[0][1]]
+            self.mailer.standard_message(sendto, subject, '')
+            return
+
         # XXX Don't enable. This doesn't work yet.
 #  "[^A-z.]tracker\+(?P<classname>[^\d\s]+)(?P<nodeid>\d+)\@some.dom.ain[^A-z.]"
         # handle delivery to addresses like:[email protected]
@@ -627,37 +631,45 @@ def handle_message(self, message):
 #                    nodeid = issue.group('nodeid')
 #                    break
 
-        # determine the sender's address
-        from_list = message.getaddrlist('resent-from')
-        if not from_list:
-            from_list = message.getaddrlist('from')
+        # Matches subjects like:
+        # Re: "[issue1234] title of issue [status=resolved]"
+        open, close = config['MAILGW_SUBJECT_SUFFIX_DELIMITERS']
+        delim_open = re.escape(open)
+        delim_close = re.escape(close)
+        subject_re = re.compile(r'''
+        (?P<refwd>\s*\W?\s*(fw|fwd|re|aw)\W\s*)*\s*   # Re:
+        (?P<quote>")?                                 # Leading "
+        (\[(?P<classname>[^\d\s]+)                    # [issue..
+           (?P<nodeid>\d+)?                           # ..1234]
+         \])?\s*
+        (?P<title>[^%s]+)?                             # issue title
+        "?                                            # Trailing "
+        (?P<argswhole>%s(?P<args>.+?)%s)?             # [prop=value]
+        '''%(delim_open, delim_open, delim_close),
+        re.IGNORECASE|re.VERBOSE)
+
+        # figure subject line parsing modes
+        pfxmode = config['MAILGW_SUBJECT_PREFIX_PARSING']
+        sfxmode = config['MAILGW_SUBJECT_SUFFIX_PARSING']
 
         # check for well-formed subject line
-        m = self.subject_re.match(subject)
+        m = subject_re.match(subject)
         if m:
             # get the classname
-            classname = m.group('classname')
+            if pfxmode == 'none':
+                classname = None
+            else:
+                classname = m.group('classname')
             if classname is None:
-                # no classname, check if this a registration confirmation email
-                # or fallback on the default class
-                otk_re = re.compile('-- key (?P<otk>[a-zA-Z0-9]{32})')
-                otk = otk_re.search(m.group('title'))
-                if otk:
-                    self.db.confirm_registration(otk.group('otk'))
-                    subject = 'Your registration to %s is complete' % \
-                              config['TRACKER_NAME']
-                    sendto = [from_list[0][1]]
-                    self.mailer.standard_message(sendto, subject, '')
-                    return
-                elif self.default_class:
+                if self.default_class:
                     classname = self.default_class
                 else:
                     classname = config['MAILGW_DEFAULT_CLASS']
                     if not classname:
                         # fail
                         m = None
 
-        if not m:
+        if not m and pfxmode == 'strict':
             raise MailUsageError, """
 The message you sent to roundup did not contain a properly formed subject
 line. The subject must contain a class name or designator to indicate the
@@ -672,10 +684,22 @@ def handle_message(self, message):
 Subject was: '%s'
 """%subject
 
-        # get the class
-        try:
-            cl = self.db.getclass(classname)
-        except KeyError:
+        # try to get the class specified - if "loose" then fall back on the
+        # default
+        attempts = [classname]
+        if pfxmode == 'loose':
+            if self.default_class:
+                attempts.append(self.default_class)
+            else:
+                attempts.append(config['MAILGW_DEFAULT_CLASS'])
+        cl = None
+        for trycl in attempts:
+            try:
+                cl = self.db.getclass(classname)
+                break
+            except KeyError:
+                pass
+        if not cl:
             raise MailUsageError, '''
 The class name you identified in the subject line ("%s") does not exist in the
 database.
@@ -685,7 +709,10 @@ def handle_message(self, message):
 '''%(classname, ', '.join(self.db.getclasses()), subject)
 
         # get the optional nodeid
-        nodeid = m.group('nodeid')
+        if pfxmode == 'none':
+            nodeid = None
+        else:
+            nodeid = m.group('nodeid')
 
         # title is optional too
         title = m.group('title')
@@ -703,7 +730,7 @@ def handle_message(self, message):
         if nodeid is None and not title:
             raise MailUsageError, '''
 I cannot match your message to a node in the database - you need to either
-supply a full node identifier (with number, eg "[issue123]" or keep the
+supply a full designator (with number, eg "[issue123]" or keep the
 previous subject title intact so I can match that.
 
 Subject was: "%s"
@@ -713,20 +740,36 @@ def handle_message(self, message):
         # maybe someone's responded to the initial mail that created an
         # entry. Try to find the matching nodes with the same title, and
         # use the _last_ one matched (since that'll _usually_ be the most
-        # recent...)
-        if nodeid is None and m.group('refwd'):
+        # recent...). The subject_content_match config may specify an
+        # additional restriction based on the matched node's creation or
+        # activity.
+        tmatch_mode = config['MAILGW_SUBJECT_CONTENT_MATCH']
+        if tmatch_mode != 'never' and nodeid is None and m.group('refwd'):
             l = cl.stringFind(title=title)
-            if l:
-                nodeid = l[-1]
+            limit = None
+            if (tmatch_mode.startswith('creation') or
+                    tmatch_mode.startswith('activity')):
+                limit, interval = tmatch_mode.split(' ', 1)
+                threshold = date.Date('.') - date.Interval(interval)
+            for id in l:
+                if limit:
+                    if threshold < cl.get(id, limit):
+                        nodeid = id
+                else:
+                    nodeid = id
 
         # if a nodeid was specified, make sure it's valid
         if nodeid is not None and not cl.hasnode(nodeid):
-            raise MailUsageError, '''
+            if pfxmode == 'strict':
+                raise MailUsageError, '''
 The node specified by the designator in the subject of your message ("%s")
 does not exist.
 
 Subject was: "%s"
 '''%(nodeid, subject)
+            else:
+                title = subject
+                nodeid = None
 
         # Handle the arguments specified by the email gateway command line.
         # We do this by looping over the list of self.arguments looking for
@@ -844,17 +887,24 @@ def handle_message(self, message):
         properties = cl.getprops()
         props = {}
         args = m.group('args')
+        argswhole = m.group('argswhole')
         if args:
-            errors, props = setPropArrayFromString(self, cl, args, nodeid)
-            # handle any errors parsing the argument list
-            if errors:
-                errors = '\n- '.join(map(str, errors))
-                raise MailUsageError, '''
+            if sfxmode == 'none':
+                title += ' ' + argswhole
+            else:
+                errors, props = setPropArrayFromString(self, cl, args, nodeid)
+                # handle any errors parsing the argument list
+                if errors:
+                    if sfxmode == 'strict':
+                        errors = '\n- '.join(map(str, errors))
+                        raise MailUsageError, '''
 There were problems handling your subject line argument list:
 - %s
 
 Subject was: "%s"
 '''%(errors, subject)
+                    else:
+                        title += ' ' + argswhole
 
 
         # set the issue title to the subject