Merge branch issue2551008 into default.

Author: ezio-melotti

CHANGES.txt

@@ -150,6 +150,8 @@ Fixed:
   file: config_ini.ini for required config settings that are merged
   into the default values producing an up to date config.ini on
   install.
+- issue2551008: fix incorrect encoding handling in mailgw.py
+  (Ezio Melotti, John Rouillard)
 
 2018-07-13 1.6.0
 
@@ -380,9 +382,9 @@ Features:
 
 Fixed:
 
-- issue1615201: Optionally restore the original (version 0.6) mailgw 
-  behaviour of ignoring a Resent-From:-header and using the real 
-  From-header instead: new configuration option EMAIL_KEEP_REAL_FROM 
+- issue1615201: Optionally restore the original (version 0.6) mailgw
+  behaviour of ignoring a Resent-From:-header and using the real
+  From-header instead: new configuration option EMAIL_KEEP_REAL_FROM
   (Peter Funk aka Pefu).
 - issue2550717: Changed a couple of residual email references into
   E-Mail in German translation (John Rouillard)
@@ -395,7 +397,7 @@ Fixed:
   multilink values (in classic template). Applied patch so it
   now errors the same way as an update does. (applied by John Rouillard)
 - issue2550757: one bug raised by issue fixed. Patch created by
-  W. Trevor King (wking) for documentation of mailgw applied by 
+  W. Trevor King (wking) for documentation of mailgw applied by
   John Rouillard.
 - Fix processing of additional arguments to cgi method 'menu': This
   would not work if more than one additional argument is used.
@@ -643,8 +645,8 @@ Fixed:
   may see removed properties. By default only role Admin is allowed to
   see these.
 - Fix issue2550955: Roundup commits although a Reject exception is raised
-  Fix the problem that changes are committed to the database (due to 
-  commits to otk handling) even when a Reject exception occurs. The fix 
+  Fix the problem that changes are committed to the database (due to
+  commits to otk handling) even when a Reject exception occurs. The fix
   implements separate database connections for otk/session handling and
   normal database operation.
 - Allow empty content property for file and message via xmlrpc
@@ -717,11 +719,11 @@ Features:
 
 Fixed:
 
-- issue2550869 Duplicate mail headers (Reply-To, Message-ID, In-Reply-To) 
+- issue2550869 Duplicate mail headers (Reply-To, Message-ID, In-Reply-To)
   when sending out email. Reported with first fix by Mathias Behrle.
   (Bernhard Reiter)
-- issue2550830 An empty LinkHTMLProperty cannot be compared successfully. 
-  Improves the query editing page. Reported and fixed by R David Murray 
+- issue2550830 An empty LinkHTMLProperty cannot be compared successfully.
+  Improves the query editing page. Reported and fixed by R David Murray
   (Bernhard Reiter).
 - Fix Release-date of 1.5.0 in this file (thanks to Bernhard for
   discovery) (Ralf Schlatterbeck)
@@ -815,18 +817,18 @@ Fixed:
   always work. (Ralf Schlatterbeck)
 - issue2550881 demo.py: Add pointer how to access demo from remote host.
   Suggested by Karl-Philipp Richter. (Bernhard Reiter)
-- issue2550884 roundup-mailgw --help text improved to explain the allowed 
+- issue2550884 roundup-mailgw --help text improved to explain the allowed
   parameters better. Suggested by by Karl-Philipp Richter. (Bernhard Reiter)
 - Fix form-parsing: If multiple new items are added to a multilink
   property, the old version would create the new items but only link
   one. (Ralf Schlatterbeck)
-- issue2550892 (translation error of priority in locale de) Thanks 
+- issue2550892 (translation error of priority in locale de) Thanks
   Martin Thomas Swaton for reporting. (Bernhard Reiter)
 - Help-Window now gets focus, this prevents the case that help doesn't
   work because an old help-window is below the main window.
   (Ralf Schlatterbeck)
-- issue2550811 20% fix: jinja2 template engine now has an example 
-  how to use non-ascii unicode contents with a custom filter ('| u'). 
+- issue2550811 20% fix: jinja2 template engine now has an example
+  how to use non-ascii unicode contents with a custom filter ('| u').
   See updates on http://www.roundup-tracker.org/cgi-bin/moin.cgi/Jinja2
   (Bernhard Reiter)
 
@@ -867,7 +869,7 @@ Features:
 - Experimental support for Jinja2, try 'jinja2' for template_engine
   in config (anatoly techtonik)
 - A new jinja2 template based on Classic schema and using Twitter
-  bootstrap for responsive behaviour.  Run as - 
+  bootstrap for responsive behaviour.  Run as -
   python demo.py -t jinja2 nuke (Pradip P Caulagi)
 - roundup_admin.py and other scripts can now be run directly from the
   sources dir as roundup\scripts\roundup_admin.py (anatoly techtonik)
@@ -887,7 +889,7 @@ Fixed:
 
 - issue2550789: add documentation on how to initialise a tracker
   without exposing the admin password.
-- issue2550805: Postgres should search title attribute case insensitive 
+- issue2550805: Postgres should search title attribute case insensitive
   like sqlite. Reported and fixed by Tom Ekberg. (Bernhard Reiter)
 - Removed some old left over "rlog" references in documentation and code.
   Makes the debugging.txt advise for the database unit tests work again.
@@ -897,7 +899,7 @@ Fixed:
 - Make roundup play nice with setup tools (for using with virtualenv)
   (Pradip Caulagi)
 - [minor] Template responsive: make demo.py work out of the box with it,
-  by setting the static_files config.ini setting to "static". 
+  by setting the static_files config.ini setting to "static".
   Footer: link fixed and hardcoded last modified date removed. (Bernhard Reiter)
 - demo.py print location of tracker home and fully erase its directory
   when nuking (anatoly techtonik)
@@ -963,7 +965,7 @@ Fixed:
   least python 2.5, thanks to John P. Rouillard for discovering this.
   (committed by Ralf Schlatterbeck)
 - Fix version_check.py to require at least python 2.5 (anatoly techtonik)
-- Fixing the download button re-activating the cheeseshop plugin in the 
+- Fixing the download button re-activating the cheeseshop plugin in the
   sphinx config. Thanks to Richard for the hint. (Bernhard Reiter)
 - issue2550783 devel template's schema.py permissions referenced the
   organization property for the user, but the property is called

roundup/anypy/email_.py

@@ -2,6 +2,7 @@
 import binascii
 import email
 from email import quoprimime, base64mime
+from email import charset as _charset
 
 if str == bytes:
     message_from_bytes = email.message_from_string

roundup/mailgw.py

@@ -202,8 +202,19 @@ class RoundupMessage(email.message.Message):
     def _decode_header(self, hdr):
         parts = []
         for part, encoding in decode_header(hdr):
-            if encoding:
-                part = part.decode(encoding)
+            # decode_header might return either bytes or unicode,
+            # see https://bugs.python.org/issue21492
+            # If part is bytes, try to decode it with the specified
+            # encoding if it's provided, otherwise try utf-8 and
+            # fallback on iso-8859-1 if that fails.
+            if isinstance(part, bytes):
+                if encoding:
+                    part = part.decode(encoding)
+                else:
+                    try:
+                        part = part.decode('utf-8')
+                    except UnicodeDecodeError:
+                        part = part.decode('iso-8859-1')
             # RFC 2047 specifies that between encoded parts spaces are
             # swallowed while at the borders from encoded to non-encoded
             # or vice-versa we must preserve a space. Multiple adjacent
@@ -530,7 +541,7 @@ def check_subject(self):
     def parse_subject(self):
         ''' Matches subjects like:
         Re: "[issue1234] title of issue [status=resolved]"
-        
+
         Each part of the subject is matched, stored, then removed from the
         start of the subject string as needed. The stored values are then
         returned
@@ -603,11 +614,11 @@ def parse_subject(self):
         if self.matches['quote'] and not self.matches['argswhole'] \
            and self.matches['title'].endswith('"'):
             self.matches['title'] = self.matches['title'][:-1]
-        
+
     def rego_confirm(self):
         ''' Check for registration OTK and confirm the registration if found
         '''
-        
+
         if self.config['EMAIL_REGISTRATION_CONFIRMATION']:
             otk_re = re.compile('-- key (?P<otk>[a-zA-Z0-9]{32})')
             otk = otk_re.search(self.matches['title'] or '')
@@ -693,14 +704,14 @@ def get_classname(self):
 """) % locals())
         # get the class properties
         self.properties = self.cl.getprops()
-        
+
 
     def get_nodeid(self):
         ''' Determine the nodeid from the message and return it if found
         '''
         title = self.matches['title']
         subject = self.subject
-        
+
         if self.pfxmode == 'none':
             nodeid = None
         else:
@@ -873,10 +884,10 @@ def get_props(self):
         ''' Generate all the props for the new/updated node and return them
         '''
         subject = self.subject
-        
+
         # get the commandline arguments for issues
         issue_props = self.mailgw.get_class_arguments('issue', self.classname)
-        
+
         #
         # handle the subject argument list
         #
@@ -885,11 +896,11 @@ def get_props(self):
         args = self.matches['args']
         argswhole = self.matches['argswhole']
         title = self.matches['title']
-        
-        # Reform the title 
+
+        # Reform the title
         if self.matches['nodeid'] and self.nodeid is None:
             title = subject
-        
+
         if args:
             if self.sfxmode == 'none':
                 title += ' ' + argswhole
@@ -986,7 +997,7 @@ def pgp_role():
                         # something failed with the message decryption/sig
                         # chain. Pass the error up.
                         raise
-                # store the decrypted message      
+                # store the decrypted message
                 self.message = message
             elif pgp_role():
                 raise MailUsageError(_("""
@@ -1052,7 +1063,7 @@ def create_msg(self):
             return
         msg_props = self.mailgw.get_class_arguments('msg')
         self.msg_props.update (msg_props)
-        
+
         # Get the message ids
         inreplyto = self.message.get_header('in-reply-to') or ''
         messageid = self.message.get_header('message-id')
@@ -1061,7 +1072,7 @@ def create_msg(self):
             messageid = "<%s.%s.%s%s@%s>"%(time.time(),
                 b2s(base64.b32encode(random_.token_bytes(10))),
                 self.classname, self.nodeid, self.config['MAIL_DOMAIN'])
-        
+
         if self.content is None:
             raise MailUsageError(_("""
 Roundup requires the submission to be plain text. The message parser could
@@ -1103,7 +1114,7 @@ def create_msg(self):
                 self.props['messages'] = [message_id]
 
     def create_node(self):
-        ''' Create/update a node using self.props 
+        ''' Create/update a node using self.props
         '''
         classname = self.classname
         try:
@@ -1583,7 +1594,7 @@ def get_class_arguments(self, class_type, classname=None):
 
         classname = classname or class_type
         cls_lookup = { 'issue' : classname }
-        
+
         # Allow other issue-type classes -- take the real classname from
         # previous parsing-steps of the message:
         clsname = cls_lookup.get (class_type, class_type)
@@ -1598,21 +1609,21 @@ def get_class_arguments(self, class_type, classname=None):
 %(mailadmin)s and have them fix the incorrect class specified as:
   %(clsname)s
 """) % locals())
-        
+
         if self.arguments:
             # The default type on the commandline is msg
             if class_type == 'msg':
                 current_type = class_type
             else:
                 current_type = None
-            
+
             # Handle the arguments specified by the email gateway command line.
             # We do this by looping over the list of self.arguments looking for
             # a -C to match the class we want, then use the -S setting string.
             for option, propstring in self.arguments:
                 if option in ( '-C', '--class'):
                     current_type = propstring.strip()
-                    
+
                     if current_type != class_type:
                         current_type = None
 

test/test_mailgw_roundupmessage.py

@@ -75,6 +75,34 @@ class HeaderRoundupMessageTests(TestCase):
         This is a test submission of a new issue.
     """)
 
+    # From line has a null/empty encoding spec
+    # to trigger failure in mailgw.py:RoundupMessage::_decode_header
+    bad_msg_utf8 = message_from_string("""
+        Content-Type: text/plain;
+            charset="iso-8859-1"
+        From: =??b?SOKCrGxsbw=====?= <[email protected]>
+        To: Issue Tracker <[email protected]>
+        Cc: =?utf8?b?SOKCrGxsbw==?= <[email protected]>,
+            Some User <[email protected]>
+        Message-Id: <dummy_test_message_id>
+        Subject: [issue] Testing...
+
+        This is a test submission of a new issue.
+    """)
+
+    bad_msg_iso_8859_1 = message_from_string("""
+        Content-Type: text/plain;
+            charset="iso-8859-1"
+        From: =??q?\x80SOKCrGxsbw=====?= <[email protected]>
+        To: Issue Tracker <[email protected]>
+        Cc: =?utf8?b?SOKCrGxsbw==?= <[email protected]>,
+            Some User <[email protected]>
+        Message-Id: <dummy_test_message_id>
+        Subject: [issue] Testing...
+
+        This is a test submission of a new issue.
+    """)
+
     def test_get_plain_header(self):
         self.assertEqual(
             self.msg.get_header('to'),
@@ -85,6 +113,21 @@ def test_get_encoded_header(self):
             self.msg.get_header('from'),
             'H€llo <[email protected]>')
 
+        # issue2551008 null encoding causes crash.
+        self.assertEqual(
+            self.bad_msg_utf8.get_header('from'),
+            'H€llo <[email protected]>')
+
+        # the decoded value is not what the user wanted,
+        # but they should have created a valid header
+        # if they wanted the right outcome...
+        self.assertIn(
+            self.bad_msg_iso_8859_1.get_header('from'),
+            (
+                '\xc2\x80SOKCrGxsbw===== <[email protected]>', # python 2
+                '\x80SOKCrGxsbw===== <[email protected]>'      # python 3
+            ))
+
     def test_get_address_list(self):
         self.assertEqual(self.msg.get_address_list('cc'), [
             ('H€llo', '[email protected]'),