Fix breakage caused by transaction changes.

Sorry for the huge checkin message - I was only intending to implement
[SF#496356] but I found a number of places where things had been
broken by transactions:

 . modified ROUNDUPDBSENDMAILDEBUG to be SENDMAILDEBUG and hold a filename
   for _all_ roundup-generated smtp messages to be sent to.
 . the transaction cache had broken the roundupdb.Class set() reactors
 . newly-created author users in the mailgw weren't being committed to the db

Stuff that made it into CHANGES.txt (ie. the stuff I was actually working
on when I found that stuff :):
 . [SF#496356] Use threading in messages
 . detectors were being registered multiple times
 . added tests for mailgw
 . much better attaching of erroneous messages in the mail gateway

Author: Richard Jones

CHANGES.txt

@@ -1,7 +1,7 @@
 This file contains the changes to the Roundup system over time. The entries
 are given with the most recent entry first.
 
-2001-12-?? - 0.3.1b1
+2001-12-?? - 0.4.0b1
 Feature:
  . Added INSTANCE_NAME to configuration - used in web and email to identify
    the instance.
@@ -25,6 +25,11 @@ Feature:
  . Added a Zope frontend for roundup.
  . Centralised the python version check code, bumped version to 2.1.1 (really
    needs to be 2.1.2, but that isn't released yet :)
+ . much better attaching of erroneous messages in the mail gateway
+ . #496356 ] Use threading in messages
+   This adds the tracking of messages by message-id and allows threading
+   using in-reply-to. Most e-mail clients support threading using this
+   feature, and we hope to add support for it to the web gateway.
 
 Fixed:
  . Lots of bugs, thanks Roch� and others on the devel mailing list!
@@ -47,7 +52,12 @@ Fixed:
  . envelope-from is now set to the roundup-admin and not roundup itself so
    delivery reports aren't sent to roundup (thanks Patrick Ohly)
  . #495400 ] entering blanks
+   Values with spaces are now accepted in roundup-admin - check the long help
+   for details.
  . #496360 ] table width does not work
+ . detectors were being registered multiple times
+ . added tests for mailgw
+
 
 2001-11-23 - 0.3.0 
 Feature:

MIGRATION.txt

@@ -1,13 +1,54 @@
 Migrating to newer versions of Roundup
 ======================================
 
+Please read each section carefully and edit your instance home files
+accordingly.
 
-Migrating from 0.2.x to 0.3.x
+This file contains information for users upgrading from:
+  0.3.x -> 0.4.x
+  0.2.x -> 0.3.x
+
+
+Migrating from 0.3.x to 0.4.x
 =============================
 
-Please read each section carefully and edit your instance home files
-accordingly.
+Message-ID and In-Reply-To addition
+-----------------------------------
+0.4.0 adds the tracking of messages by message-id and allows threading
+using in-reply-to. Most e-mail clients support threading using this
+feature, and we hope to add support for it to the web gateway. If you
+have not edited the dbinit.py file in your instance home directory, you may
+simply copy the new dbinit.py file from the core code. If you used the
+classic schema, the interfaces file is in:
 
+ <roundup source>/roundup/templates/classic/dbinit.py
+
+If you used the extended schema, the file is in:
+
+ <roundup source>/roundup/templates/extended/dbinit.pybinit.py needs updating from the original. 
+
+If you have modified your dbinit.py file, you may use encoded passwords:
+
+ 1. Edit the dbinit.py file in your instance home directory. Find the lines
+ which define the msg class:
+
+    msg = FileClass(db, "msg",
+                    author=Link("user"), recipients=Multilink("user"),
+                    date=Date(),         summary=String(),
+                    files=Multilink("file"))
+
+ and add the messageid and inreplyto properties like so:
+
+    msg = FileClass(db, "msg",
+                    author=Link("user"), recipients=Multilink("user"),
+                    date=Date(),         summary=String(),
+                    files=Multilink("file"),
+                    messageid=String(),  inreplyto=String())
+
+
+
+Migrating from 0.2.x to 0.3.x
+=============================
 
 Cookie Authentication changes
 -----------------------------

roundup/backends/back_anydbm.py

@@ -15,7 +15,7 @@
 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
 # 
-#$Id: back_anydbm.py,v 1.20 2001-12-18 15:30:34 rochecompaan Exp $
+#$Id: back_anydbm.py,v 1.21 2002-01-02 02:31:38 richard Exp $
 '''
 This module defines a backend that saves the hyperdatabase in a database
 chosen by anydbm. It is guaranteed to always be available in python
@@ -187,23 +187,25 @@ def savenode(self, classname, nodeid, node):
             print 'savenode', (self, classname, nodeid, node)
         self.transactions.append((self._doSaveNode, (classname, nodeid, node)))
 
-    def getnode(self, classname, nodeid, db=None):
+    def getnode(self, classname, nodeid, db=None, cache=1):
         ''' get a node from the database
         '''
         if DEBUG:
             print 'getnode', (self, classname, nodeid, cldb)
-        # try the cache
-        cache = self.cache.setdefault(classname, {})
-        if cache.has_key(nodeid):
-            return cache[nodeid]
+        if cache:
+            # try the cache
+            cache = self.cache.setdefault(classname, {})
+            if cache.has_key(nodeid):
+                return cache[nodeid]
 
         # get from the database and save in the cache
         if db is None:
             db = self.getclassdb(classname)
         if not db.has_key(nodeid):
-            raise IndexError, nodeid
+            raise IndexError, "no such %s %s"%(classname, nodeid)
         res = marshal.loads(db[nodeid])
-        cache[nodeid] = res
+        if cache:
+            cache[nodeid] = res
         return res
 
     def hasnode(self, classname, nodeid, db=None):
@@ -402,6 +404,15 @@ def rollback(self):
 
 #
 #$Log: not supported by cvs2svn $
+#Revision 1.20  2001/12/18 15:30:34  rochecompaan
+#Fixed bugs:
+# .  Fixed file creation and retrieval in same transaction in anydbm
+#    backend
+# .  Cgi interface now renders new issue after issue creation
+# .  Could not set issue status to resolved through cgi interface
+# .  Mail gateway was changing status back to 'chatting' if status was
+#    omitted as an argument
+#
 #Revision 1.19  2001/12/17 03:52:48  richard
 #Implemented file store rollback. As a bonus, the hyperdb is now capable of
 #storing more than one file per node - if a property name is supplied,

roundup/cgi_client.py

@@ -15,14 +15,14 @@
 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
 # 
-# $Id: cgi_client.py,v 1.87 2001-12-23 23:18:49 richard Exp $
+# $Id: cgi_client.py,v 1.88 2002-01-02 02:31:38 richard Exp $
 
 __doc__ = """
 WWW request handler (also used in the stand-alone server).
 """
 
 import os, cgi, pprint, StringIO, urlparse, re, traceback, mimetypes
-import binascii, Cookie, time
+import binascii, Cookie, time, random
 
 import roundupdb, htmltemplate, date, hyperdb, password
 from roundup.i18n import _
@@ -456,11 +456,16 @@ def _handle_message(self):
             # don't generate a useless message
             return None, files
 
+        # handle the messageid
+        # TODO: handle inreplyto
+        messageid = "%s.%s.%s%s-%s"%(time.time(), random.random(),
+            classname, nodeid, self.MAIL_DOMAIN)
+
         # now create the message, attaching the files
         content = '\n'.join(m)
         message_id = self.db.msg.create(author=self.getuid(),
             recipients=[], date=date.Date('.'), summary=summary,
-            content=content, files=files)
+            content=content, files=files, messageid=messageid)
 
         # update the messages property
         return message_id, files
@@ -1163,6 +1168,10 @@ def parsePropsFromForm(db, cl, form, nodeid=0):
 
 #
 # $Log: not supported by cvs2svn $
+# Revision 1.87  2001/12/23 23:18:49  richard
+# We already had an admin-specific section of the web heading, no need to add
+# another one :)
+#
 # Revision 1.86  2001/12/20 15:43:01  rochecompaan
 # Features added:
 #  .  Multilink properties are now displayed as comma separated values in

roundup/hyperdb.py

@@ -15,7 +15,7 @@
 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
 # 
-# $Id: hyperdb.py,v 1.43 2001-12-20 06:13:24 rochecompaan Exp $
+# $Id: hyperdb.py,v 1.44 2002-01-02 02:31:38 richard Exp $
 
 __doc__ = """
 Hyperdatabase implementation, especially field types.
@@ -240,18 +240,23 @@ def create(self, **propvalues):
         self.db.addjournal(self.classname, newid, 'create', propvalues)
         return newid
 
-    def get(self, nodeid, propname, default=_marker):
+    def get(self, nodeid, propname, default=_marker, cache=1):
         """Get the value of a property on an existing node of this class.
 
         'nodeid' must be the id of an existing node of this class or an
         IndexError is raised.  'propname' must be the name of a property
         of this class or a KeyError is raised.
+
+        'cache' indicates whether the transaction cache should be queried
+        for the node. If the node has been modified and you need to
+        determine what its values prior to modification are, you need to
+        set cache=0.
         """
         if propname == 'id':
             return nodeid
 
         # get the node's dict
-        d = self.db.getnode(self.classname, nodeid)
+        d = self.db.getnode(self.classname, nodeid, cache=cache)
         if not d.has_key(propname) and default is not _marker:
             return default
 
@@ -271,10 +276,18 @@ def get(self, nodeid, propname, default=_marker):
         return d[propname]
 
     # XXX not in spec
-    def getnode(self, nodeid):
-        ''' Return a convenience wrapper for the node
+    def getnode(self, nodeid, cache=1):
+        ''' Return a convenience wrapper for the node.
+
+        'nodeid' must be the id of an existing node of this class or an
+        IndexError is raised.
+
+        'cache' indicates whether the transaction cache should be queried
+        for the node. If the node has been modified and you need to
+        determine what its values prior to modification are, you need to
+        set cache=0.
         '''
-        return Node(self, nodeid)
+        return Node(self, nodeid, cache=cache)
 
     def set(self, nodeid, **propvalues):
         """Modify a property on an existing node of this class.
@@ -824,36 +837,37 @@ def addprop(self, **properties):
 class Node:
     ''' A convenience wrapper for the given node
     '''
-    def __init__(self, cl, nodeid):
+    def __init__(self, cl, nodeid, cache=1):
         self.__dict__['cl'] = cl
         self.__dict__['nodeid'] = nodeid
+        self.cache = cache
     def keys(self, protected=1):
         return self.cl.getprops(protected=protected).keys()
     def values(self, protected=1):
         l = []
         for name in self.cl.getprops(protected=protected).keys():
-            l.append(self.cl.get(self.nodeid, name))
+            l.append(self.cl.get(self.nodeid, name, cache=self.cache))
         return l
     def items(self, protected=1):
         l = []
         for name in self.cl.getprops(protected=protected).keys():
-            l.append((name, self.cl.get(self.nodeid, name)))
+            l.append((name, self.cl.get(self.nodeid, name, cache=self.cache)))
         return l
     def has_key(self, name):
         return self.cl.getprops().has_key(name)
     def __getattr__(self, name):
         if self.__dict__.has_key(name):
             return self.__dict__[name]
         try:
-            return self.cl.get(self.nodeid, name)
+            return self.cl.get(self.nodeid, name, cache=self.cache)
         except KeyError, value:
             # we trap this but re-raise it as AttributeError - all other
             # exceptions should pass through untrapped
             pass
         # nope, no such attribute
         raise AttributeError, str(value)
     def __getitem__(self, name):
-        return self.cl.get(self.nodeid, name)
+        return self.cl.get(self.nodeid, name, cache=self.cache)
     def __setattr__(self, name, value):
         try:
             return self.cl.set(self.nodeid, **{name: value})
@@ -875,6 +889,16 @@ def Choice(name, *options):
 
 #
 # $Log: not supported by cvs2svn $
+# Revision 1.43  2001/12/20 06:13:24  rochecompaan
+# Bugs fixed:
+#   . Exception handling in hyperdb for strings-that-look-like numbers got
+#     lost somewhere
+#   . Internet Explorer submits full path for filename - we now strip away
+#     the path
+# Features added:
+#   . Link and multilink properties are now displayed sorted in the cgi
+#     interface
+#
 # Revision 1.42  2001/12/16 10:53:37  richard
 # take a copy of the node dict so that the subsequent set
 # operation doesn't modify the oldvalues structure