support sqlite3

Author: Richard Jones

CHANGES.txt

@@ -3,6 +3,7 @@ are given with the most recent entry first.
 
 2006-??-?? 1.1.3
 Feature:
+- supports Python 2.5, including the sqlite3 module
 - full timezone support (sf patch 1465296)
 - handle connection loss when responding to web requests
 - match incoming mail In-Reply-To against existing messages when no issue
@@ -23,6 +24,7 @@ Feature:
 - added filter_sql to SQL backends which takes an arbitrary SQL statement
   and returns a list of item ids
 
+
 Fixed:
 - Verbose option for import and export (sf bug 1505645)
 - -c option for roundup-mailgw won't accept parameter (sf bug 1505649)

doc/announcement.txt

@@ -1,29 +1,29 @@
-I'm proud to release version 1.1.2 of Roundup.
+I'm proud to release version 1.1.3 of Roundup.
 
 Feature:
-
-- server-ctl script uses server configuration file (sf bug 1443805)
-
-Fixed:
-
-- indexing may be turned off for FileClass "content" now
-  ("content" and "type" properties are now automatically included in the
-  FileClass schema where previously the "content" property was faked and
-  "type" was optional)
-- reduced frequency of session timestamp update
-- progress display in roundup-admin reindex
-- bug in menu() permission filter (sf bug 1444440)
-- verbose output during import is optional now (sf bug 1475624)
-- escape *all* uses of "schema" in mysql backend (sf bug 1472120)
-- responses to user rego email (sf bug 1470254)
-- dangling connections in session handling (sf bug 1463359)
-- classhelp popup pagination forgot about "type" (sf bug 1465836)
-- umask is now configurable (with the same 0002 default)
-- sorting of entries in classhelp popup (sf bug 1449000)
-- allow single digit seconds in date spec (sf bug 1447141)
-- prevent generation of new single-digit seconds dates (sf bug 1429390)
-- implement close() on all indexers (sf bug 1242477)
-
+- supports Python 2.5, including the sqlite3 module
+- full timezone support (sf patch 1465296)
+- handle connection loss when responding to web requests
+- match incoming mail In-Reply-To against existing messages when no issue
+  id is specified in the Subject
+- added StringHTMLProperty wrapped() method to wrap long lines in issue
+  display
+- include the popcal in Date field editing and search fields by default
+- @required in forms may now specify properties of linked items (sf patch
+  1507093)
+- update for latest version of pysqlite (sf bug 1487098; patch 1534227)
+- update for latest version of psycopg2 (sf patch 1429391)
+- new "exporttables" command in roundup-admin (sf bug 1533791)
+- roundup-admin "export" may specify classes to exclude (sf bug 1533791)
+- sorting and grouping by multiple properties is now supported by the
+  backends *and* the classic template.
+- sorting, grouping, and searching by transitive properties (e.g.,
+  messages.author.supervisor) is now supported in all backends
+- added filter_sql to SQL backends which takes an arbitrary SQL statement
+  and returns a list of item ids
+
+There was also a lot of bugfixes - see the bundled CHANGES.txt file for the
+list.
 
 If you're upgrading from an older version of Roundup you *must* follow
 the "Software Upgrade" guidelines given in the maintenance documentation.

doc/customizing.txt

@@ -2,7 +2,7 @@
 Customising Roundup
 ===================
 
-:Version: $Revision: 1.208 $
+:Version: $Revision: 1.209 $
 
 .. This document borrows from the ZopeBook section on ZPT. The original is at:
    http://www.zope.org/Documentation/Books/ZopeBook/current/ZPT.stx
@@ -1869,8 +1869,14 @@ filter      lists of items from this class, filtered and sorted. Two
                where dir is '+', '-' or None
                and prop is a prop name or None.
 
-            eg. ``issue.filter(filterspec={"priority": "1"},
-            sort=[('activity', '+')])``
+               The propname in filterspec and prop in a sort/group spec
+               may be transitive, i.e., it may contain properties of
+               the form link.link.link.name.
+
+            eg. All issues with a priority of "1" with messages added in
+            the last week, sorted by activity date:
+            ``issue.filter(filterspec={"priority": "1",
+            'messages.creation' : '.-1w;'}, sort=[('activity', '+')])``
 
 filter_sql  **Only in SQL backends**
 
@@ -2199,7 +2205,8 @@ show        a convenience access to columns - request/show/colname will
 sort        index sort columns [(direction, column name)]
 group       index grouping properties [(direction, column name)]
 filter      properties to filter the index on
-filterspec  values to filter the index on
+filterspec  values to filter the index on (property=value, eg
+            ``priority=1`` or ``messages.author=42``
 search_text text to perform a full-text search on for an index
 =========== ============================================================
 

doc/design.txt

@@ -490,11 +490,11 @@ Here is the interface provided by the hyperdatabase::
             1. String properties must match all elements in the list, and
             2. Other properties must match any of the elements in the list.
 
-            Note that now the propname in filterspec and prop in a 
-            sort/group spec may be transitive, i.e., it may contain
-            properties of the form link.link.link.name, e.g. you can search
-            for all issues where a message was added by a certain user in
-            the last week with a filterspec of
+            The propname in filterspec and prop in a sort/group spec may be
+            transitive, i.e., it may contain properties of the form
+            link.link.link.name, e.g. you can search for all issues where
+            a message was added by a certain user in the last week with a
+            filterspec of
             {'messages.author' : '42', 'messages.creation' : '.-1w;'}
             """
 

roundup/backends/__init__.py

@@ -15,7 +15,7 @@
 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
 #
-# $Id: __init__.py,v 1.36 2006-01-25 03:24:09 richard Exp $
+# $Id: __init__.py,v 1.37 2006-10-04 01:12:00 richard Exp $
 
 '''Container for the hyperdb storage backend implementations.
 '''
@@ -28,9 +28,10 @@
 # module name, have_backend quietly returns False.
 # Otherwise the error is reraised.
 _modules = {
-    'mysql': 'MySQLdb',
-    'postgresql': 'psycopg',
-    'tsearch2': 'psycopg',
+    'mysql': ('MySQLdb',),
+    'postgresql': ('psycopg',),
+    'tsearch2': ('psycopg',),
+    'sqlite': ('pysqlite', 'pysqlite2'),
 }
 
 def get_backend(name):
@@ -65,10 +66,10 @@ def have_backend(name):
         get_backend(name)
         return 1
     except ImportError, e:
-        global _modules
-        if not str(e).startswith('No module named %s'
-                % _modules.get(name, name)):
-            raise
+        for name in _modules.get(name, (name,)):
+            if str(e).startswith('No module named %s'%name):
+                return 0
+        raise
     return 0
 
 def list_backends():

roundup/backends/back_sqlite.py

@@ -1,4 +1,4 @@
-# $Id: back_sqlite.py,v 1.46 2006-08-30 09:05:30 schlatterbeck Exp $
+# $Id: back_sqlite.py,v 1.47 2006-10-04 01:12:00 richard Exp $
 '''Implements a backend for SQLite.
 
 See https://pysqlite.sourceforge.net/ for pysqlite info
@@ -13,10 +13,15 @@
 
 from roundup import hyperdb, date, password
 from roundup.backends import rdbms_common
+is_sqlite3 = False
 try:
     import sqlite
 except ImportError:
-    from pysqlite2 import dbapi2 as sqlite
+    try:
+        from pysqlite2 import dbapi2 as sqlite
+    except ImportError:
+        import sqlite3 as sqlite
+        is_sqlite3 = True
 
 def db_exists(config):
     return os.path.exists(os.path.join(config.DATABASE, 'db'))
@@ -26,7 +31,10 @@ def db_nuke(config):
 
 class Database(rdbms_common.Database):
     # char to use for positional arguments
-    arg = '%s'
+    if is_sqlite3:
+        arg = '?'
+    else:
+        arg = '%s'
 
     # used by some code to switch styles of query
     implements_intersect = 1
@@ -88,10 +96,14 @@ def sql_open_connection(self):
 
         db = os.path.join(self.config.DATABASE, 'db')
         logging.getLogger('hyperdb').info('open database %r'%db)
-        conn = sqlite.connect(db=db)
         # set a 30 second timeout (extraordinarily generous) for handling
         # locked database
-        conn.db.sqlite_busy_handler(self.sqlite_busy_handler)
+        if is_sqlite3:
+            conn = sqlite.connect(db, 30)
+            conn.row_factory = sqlite.Row
+        else:
+            conn = sqlite.connect(db=db)
+            conn.db.sqlite_busy_handler(self.sqlite_busy_handler)
         cursor = conn.cursor()
         return (conn, cursor)
 
@@ -252,11 +264,21 @@ def update_class(self, spec, old_spec, force=0, adding_v2=0):
                     # generate the new value for the Interval int column
                     if name.endswith('_int__'):
                         name = name[2:-6]
-                        if entry.has_key(name):
+                        if is_sqlite3:
+                            try:
+                                v = hyperdb.Interval(entry[name]).as_seconds()
+                            except IndexError:
+                                v = None
+                        elif entry.has_key(name):
                             v = hyperdb.Interval(entry[name]).as_seconds()
                         else:
                             v = None
-                    elif entry.has_key(name):
+                    elif is_sqlite3:
+                        try:
+                            v = entry[name]
+                        except IndexError:
+                            v = None
+                    elif (not is_sqlite3 and entry.has_key(name)):
                         v = entry[name]
                     else:
                         v = None
@@ -342,10 +364,19 @@ def clear(self):
 
     def create_class(self, spec):
         rdbms_common.Database.create_class(self, spec)
-        sql = 'insert into ids (name, num) values (%s, %s)'
+        sql = 'insert into ids (name, num) values (%s, %s)'%(self.arg, self.arg)
         vals = (spec.classname, 1)
         self.sql(sql, vals)
 
+    if is_sqlite3:
+        def load_journal(self, classname, cols, nodeid):
+            '''We need to turn the sqlite3.Row into a tuple so it can be
+            unpacked'''
+            l = rdbms_common.Database.load_journal(self,
+                classname, cols, nodeid)
+            cols = range(5)
+            return [[row[col] for col in cols] for row in l]
+
 class sqliteClass:
     def filter(self, search_matches, filterspec, sort=(None,None),
             group=(None,None)):

roundup/backends/indexer_common.py

@@ -1,4 +1,4 @@
-#$Id: indexer_common.py,v 1.6 2006-04-27 05:48:26 richard Exp $
+#$Id: indexer_common.py,v 1.7 2006-10-04 01:12:00 richard Exp $
 import re, sets
 
 from roundup import hyperdb
@@ -49,12 +49,17 @@ def search(self, search_terms, klass, ignore={}):
         propspec = {}     # used to do the klass.find
         for propname in designator_propname.values():
             propspec[propname] = {}   # used as a set (value doesn't matter)
-        for classname, nodeid, property in hits:
+
+        # don't unpack hits entries as sqlite3's Row can't be unpacked :(
+        for entry in hits:
             # skip this result if we don't care about this class/property
+            classname = entry[0]
+            property = entry[2]
             if ignore.has_key((classname, property)):
                 continue
 
             # if it's a property on klass, it's easy
+            nodeid = entry[1]
             if classname == klass.classname:
                 if not nodeids.has_key(nodeid):
                     nodeids[nodeid] = {}

roundup/backends/indexer_rdbms.py

@@ -1,4 +1,4 @@
-#$Id: indexer_rdbms.py,v 1.14 2006-05-06 16:19:27 a1s Exp $
+#$Id: indexer_rdbms.py,v 1.15 2006-10-04 01:12:00 richard Exp $
 ''' This implements the full-text indexer over two RDBMS tables. The first
 is a mapping of words to occurance IDs. The second maps the IDs to (Class,
 propname, itemid) instances.
@@ -95,7 +95,7 @@ def find(self, wordlist):
             a = ','.join([self.db.arg] * len(r))
             sql = 'select _class, _itemid, _prop from __textids '\
                 'where _textid in (%s)'%a
-            self.db.cursor.execute(sql, tuple([int(id) for (id,) in r]))
+            self.db.cursor.execute(sql, tuple([int(row[0]) for row in r]))
 
         else:
             # A more complex version for MySQL since it doesn't implement INTERSECT

roundup/backends/rdbms_common.py

@@ -15,7 +15,7 @@
 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
 #
-#$Id: rdbms_common.py,v 1.181 2006-08-30 09:28:26 schlatterbeck Exp $
+#$Id: rdbms_common.py,v 1.182 2006-10-04 01:12:00 richard Exp $
 ''' Relational database (SQL) backend common code.
 
 Basics:
@@ -191,7 +191,7 @@ def save_dbschema(self):
         '''
         s = repr(self.database_schema)
         self.sql('delete from schema')
-        self.sql('insert into schema values (%s)', (s,))
+        self.sql('insert into schema values (%s)'%self.arg, (s,))
 
     def post_init(self):
         ''' Called once the schema initialisation has finished.