Added some words to the installation doc about choosing backends.

Added hyperdb Class.filter unit tests - gadfly currently fails
substring searching, but I knew it would :(

Lots of fixes to the RDBMS backend - it works a treat now!

A couple of other cleanups in CGI land...

Author: Richard Jones

doc/installation.txt

@@ -2,7 +2,7 @@
 Installing Roundup
 ==================
 
-:Version: $Revision: 1.23 $
+:Version: $Revision: 1.24 $
 
 .. contents::
 
@@ -35,20 +35,40 @@ holds issues which have priorities and statuses. Each issue may also have a
 set of messages which are disseminated to the issue's list of nosy users.
 
 
-Extended Template
------------------
-
-The extended template adds additional information to issues: product,
-platform, version, targetversion and supportcall.
-There is an additional class for
-handling support calls, which includes a time log, customername, rate and
-source.
-
-The priorty class has different default entries too: "fatal-bug", "bug",
-"usability" and "feature".
-
-Users of this template will want to change the contents of the product
-class as soon as the tracker is created.
+Backends
+--------
+
+The actual storage of Roundup tracker information is handled by backends.
+There's several to choose from, each with benefits and limitations:
+
+**anydbm**
+  This backend is guaranteed to work on any system that Python runs on. It
+  will generally choose the best *dbm backend that is available on your system
+  (from the list dbhash, gdbm, dbm, dumbdbm). It is the least scaleable of all
+  backends, but performs well enough for a smallish tracker (a couple of
+  thousand issues, under fifty users, ...).
+**bsddb**
+  This effectively the same as anydbm, but uses the bsddb backend. This allows
+  it to gain some performance and scaling benefits.
+**bsddb3**
+  Again, this effectively the same as anydbm, but uses the bsddb3 backend.
+  This allows it to gain some performance and scaling benefits.
+**sqlite**
+  This uses the SQLite embedded RDBMS to provide a fast, scaleable backend.
+  There are no limitations.
+**gadfly**
+  This is a proof-of-concept relational database backend, not really intended
+  for actual production use, although it can be. It uses the Gadfly RDBMS
+  to store data. It is unable to perform string searches due to gadfly not
+  having a LIKE operation. It should scale well, assuming a client/server
+  setup is used.
+**metakit**
+  This backend is implemented over the metakit storage system, using Mk4Py as
+  the interface. It scales much better than the *dbm backends, but has some
+  missing features:
+
+  - you may not unset properties once they are set
+  - journal retrieval is not implemented 
 
 
 Prerequisites

roundup/backends/back_anydbm.py

@@ -15,7 +15,7 @@
 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
 # 
-#$Id: back_anydbm.py,v 1.80 2002-09-17 23:59:59 richard Exp $
+#$Id: back_anydbm.py,v 1.81 2002-09-19 02:37:41 richard Exp $
 '''
 This module defines a backend that saves the hyperdatabase in a database
 chosen by anydbm. It is guaranteed to always be available in python
@@ -1499,6 +1499,10 @@ def filter(self, search_matches, filterspec, sort, group,
             "sort" and "group" are (dir, prop) where dir is '+', '-' or None
                                and prop is a prop name or None
             "search_matches" is {nodeid: marker}
+
+            The filter must match all properties specificed - but if the
+            property value to match is a list, any one of the values in the
+            list may match for that property to match.
         '''
         cn = self.classname
 

roundup/backends/back_gadfly.py

@@ -1,4 +1,4 @@
-# $Id: back_gadfly.py,v 1.22 2002-09-18 05:07:47 richard Exp $
+# $Id: back_gadfly.py,v 1.23 2002-09-19 02:37:41 richard Exp $
 __doc__ = '''
 About Gadfly
 ============
@@ -136,3 +136,120 @@ def load_journal(self, cursor, classname, cols, nodeid):
             res.append((nodeid, date.Date(date_stamp), user, action, params))
         return res
 
+class GadflyClass:
+    def filter(self, search_matches, filterspec, sort, group):
+        ''' Gadfly doesn't have a LIKE predicate :(
+        '''
+        cn = self.classname
+
+        # figure the WHERE clause from the filterspec
+        props = self.getprops()
+        frum = ['_'+cn]
+        where = []
+        args = []
+        a = self.db.arg
+        for k, v in filterspec.items():
+            propclass = props[k]
+            if isinstance(propclass, Multilink):
+                tn = '%s_%s'%(cn, k)
+                frum.append(tn)
+                if isinstance(v, type([])):
+                    s = ','.join([self.arg for x in v])
+                    where.append('id=%s.nodeid and %s.linkid in (%s)'%(tn,tn,s))
+                    args = args + v
+                else:
+                    where.append('id=%s.nodeid and %s.linkid = %s'%(tn, tn, a))
+                    args.append(v)
+            else:
+                if isinstance(v, type([])):
+                    s = ','.join([a for x in v])
+                    where.append('_%s in (%s)'%(k, s))
+                    args = args + v
+                else:
+                    where.append('_%s=%s'%(k, a))
+                    args.append(v)
+
+        # add results of full text search
+        if search_matches is not None:
+            v = search_matches.keys()
+            s = ','.join([a for x in v])
+            where.append('id in (%s)'%s)
+            args = args + v
+
+        # figure the order by clause
+        orderby = []
+        ordercols = []
+        if sort[0] is not None and sort[1] is not None:
+            direction, colname = sort
+            if direction != '-':
+                if colname == 'activity':
+                    orderby.append('activity')
+                    ordercols.append('max(%s__journal.date) as activity'%cn)
+                    frum.append('%s__journal'%cn)
+                    where.append('%s__journal.nodeid = _%s.id'%(cn, cn))
+                elif colname == 'id':
+                    orderby.append(colname)
+                    ordercols.append(colname)
+                else:
+                    orderby.append('_'+colname)
+                    ordercols.append('_'+colname)
+            else:
+                if colname == 'activity':
+                    orderby.append('activity desc')
+                    ordercols.append('max(%s__journal.date) as activity'%cn)
+                    frum.append('%s__journal'%cn)
+                    where.append('%s__journal.nodeid = _%s.id'%(cn, cn))
+                elif colname == 'id':
+                    orderby.append(colname+' desc')
+                    ordercols.append(colname)
+                else:
+                    orderby.append('_'+colname+' desc')
+                    ordercols.append('_'+colname)
+
+        # figure the group by clause
+        groupby = []
+        groupcols = []
+        if group[0] is not None and group[1] is not None:
+            if group[0] != '-':
+                groupby.append('_'+group[1])
+                groupcols.append('_'+group[1])
+            else:
+                groupby.append('_'+group[1]+' desc')
+                groupcols.append('_'+group[1])
+
+        # construct the SQL
+        frum = ','.join(frum)
+        where = ' and '.join(where)
+        cols = []
+        if orderby:
+            cols = cols + ordercols
+            order = ' order by %s'%(','.join(orderby))
+        else:
+            order = ''
+        if 0: #groupby:
+            cols = cols + groupcols
+            group = ' group by %s'%(','.join(groupby))
+        else:
+            group = ''
+        if 'id' not in cols:
+            cols.append('id')
+        cols = ','.join(cols)
+        sql = 'select %s from %s where %s%s%s'%(cols, frum, where, order,
+            group)
+        args = tuple(args)
+        if __debug__:
+            print >>hyperdb.DEBUG, 'filter', (self, sql, args)
+        cursor = self.db.conn.cursor()
+        cursor.execute(sql, args)
+        l = cursor.fetchall()
+
+        # return the IDs
+        return [row[0] for row in l]
+
+class Class(GadflyClass, Class):
+    pass
+class IssueClass(GadflyClass, IssueClass):
+    pass
+class FileClass(GadflyClass, FileClass):
+    pass
+

roundup/backends/back_sqlite.py

@@ -1,4 +1,4 @@
-# $Id: back_sqlite.py,v 1.2 2002-09-18 07:04:37 richard Exp $
+# $Id: back_sqlite.py,v 1.3 2002-09-19 02:37:41 richard Exp $
 __doc__ = '''
 See https://pysqlite.sourceforge.net/ for pysqlite info
 '''
@@ -128,80 +128,3 @@ def unserialise(self, classname, node):
                 d[k] = v
         return d
 
-class Class(Class):
-    _marker = []
-    def get(self, nodeid, propname, default=_marker, cache=1):
-        '''Get the value of a property on an existing node of this class.
-
-        'nodeid' must be the id of an existing node of this class or an
-        IndexError is raised.  'propname' must be the name of a property
-        of this class or a KeyError is raised.
-
-        'cache' indicates whether the transaction cache should be queried
-        for the node. If the node has been modified and you need to
-        determine what its values prior to modification are, you need to
-        set cache=0.
-        '''
-        if propname == 'id':
-            return nodeid
-
-        if propname == 'creation':
-            if not self.do_journal:
-                raise ValueError, 'Journalling is disabled for this class'
-            journal = self.db.getjournal(self.classname, nodeid)
-            if journal:
-                return self.db.getjournal(self.classname, nodeid)[0][1]
-            else:
-                # on the strange chance that there's no journal
-                return date.Date()
-        if propname == 'activity':
-            if not self.do_journal:
-                raise ValueError, 'Journalling is disabled for this class'
-            journal = self.db.getjournal(self.classname, nodeid)
-            if journal:
-                return self.db.getjournal(self.classname, nodeid)[-1][1]
-            else:
-                # on the strange chance that there's no journal
-                return date.Date()
-        if propname == 'creator':
-            if not self.do_journal:
-                raise ValueError, 'Journalling is disabled for this class'
-            journal = self.db.getjournal(self.classname, nodeid)
-            if journal:
-                name = self.db.getjournal(self.classname, nodeid)[0][2]
-            else:
-                return None
-            try:
-                return self.db.user.lookup(name)
-            except KeyError:
-                # the journaltag user doesn't exist any more
-                return None
-
-        # get the property (raises KeyErorr if invalid)
-        prop = self.properties[propname]
-
-        # get the node's dict
-        d = self.db.getnode(self.classname, nodeid) #, cache=cache)
-
-        if not d.has_key(propname):
-            if default is self._marker:
-                if isinstance(prop, Multilink):
-                    return []
-                else:
-                    return None
-            else:
-                return default
-
-        # special handling for some types
-        if isinstance(prop, Multilink):
-            # don't pass our list to other code
-            return d[propname][:]
-        elif d[propname] is None:
-            # always return None right now, no conversion
-            return None
-        elif isinstance(prop, Boolean) or isinstance(prop, Number):
-            # turn Booleans and Numbers into integers
-            return int(d[propname])
-
-        return d[propname]
-