More transitive-property support.

- Implemented transitive properties in sort and group specs. Sort/group
  specs can now be lists of specs.
- All regression tests except for one metakit backend test related to
  metakit having no representation of NULL pass
- Fixed more PEP 8 whitespace peeves (and probably introduced some new
  ones :-)
- Moved Proptree from support.py to hyperdb.py due to circular import
- Moved some proptree-specific methods from Class to Proptree
- Added a test for sorting by ids -> should be numeric sort (which now
  really works for all backends)
- Added "required" attribute to all property classes in hyperdb (e.g.,
  String, Link,...), see Feature Requests [SF#539081]
  -> factored common stuff to _Type. Note that I also converted to a
  new-style class when I was at it. Bad: The repr changes for new-style
  classes which made some SQL backends break (!) because the repr of
  Multilink is used in the schema storage. Fixed the repr to be
  independent of the class type.
- Added get_required_props to Class. Todo: should also automagically
  make the key property required...
- Add a sort_repr method to property classes. This defines the
  sort-order. Individual backends may use diffent routines if the
  outcome is the same. This one has a special case for id properties to
  make the sorting numeric. Using these methods isn't mandatory in
  backends as long as the sort-order is correct.
- Multilink sorting takes orderprop into account. It used to sort by
  ids. You can restore the old behaviour by specifying id as the
  orderprop of the Multilink if you really need that.
- If somebody specified a Link or Multilink as orderprop, we sort by
  labelprop of that class -- not transitively by orderprop. I've
  resited the tempation to implement recursive orderprop here: There
  could even be loops if several classes specify a Link or Multilink as
  the orderprop...
- Fixed a bug in Metakit-Backend: When sorting by Links, the backend
  would do a natural join to the Link class. It would rename the "id"
  attribute before joining but *not* all the other attributes of the
  joined class.  So in one test-case we had a name-clash with
  priority.name and status.name when sorting *and* grouping by these
  attributes. Depending on the order of joining this would produce a
  name-clash with broken sort-results (and broken display if the
  original class has an attribute that clashes). I'm now doing the
  sorting of Links in the generic filter method for the metakit backend.
  I've left the dead code in the metakit-backend since correctly
  implementing this in the backend will probably be more efficient.
- updated doc/design.html with the new docstring of filter.

Author: Ralf Schlatterbeck

CHANGES.txt

@@ -16,6 +16,10 @@ Feature:
 - update for latest version of psycopg2 (sf patch 1429391)
 - new "exporttables" command in roundup-admin (sf bug 1533791)
 - roundup-admin "export" may specify classes to exclude (sf bug 1533791)
+- sorting and grouping by multiple properties is now supported by the
+  backends
+- sorting, grouping, and searching by transitive properties (e.g.,
+  messages.author.supervisor) is now supported in all backends
 
 Fixed:
 - Verbose option for import and export (sf bug 1505645)
@@ -36,7 +40,13 @@ Fixed:
 - fix help message for roundup-admin install (sf bug 1494990)
 - removed traceback with OTK is used multiple times (sf bug 1240539)
 - metakit backend was indexing FileClass content even when asked not to
-
+- anydbm backend will finally sort numerically by ID
+- problem with string sorting in anydbm backend fixed: If a string was
+  fully numeric it was sorted as a number
+- Multilink-sorting now sorts by orderprop not by ID and works for all
+  backends
+- Bug with name-collisions in sorted classes when sorting by Link
+  properties in metakit backend fixed
 
 2006-04-27 1.1.2
 Feature:

doc/design.txt

@@ -471,9 +471,31 @@ Here is the interface provided by the hyperdatabase::
             """
 
         def filter(self, search_matches, filterspec, sort, group):
-            """ Return a list of the ids of the active items in this
-            class that match the 'filter' spec, sorted by the group spec
-            and then the sort spec.
+            """Return a list of the ids of the active nodes in this class that
+            match the 'filter' spec, sorted by the group spec and then the
+            sort spec.
+
+            "filterspec" is {propname: value(s)}
+
+            "sort" and "group" are [(dir, prop), ...] where dir is '+', '-'
+            or None and prop is a prop name or None. Note that for
+            backward-compatibility reasons a single (dir, prop) tuple is
+            also allowed.
+
+            "search_matches" is {nodeid: marker}
+
+            The filter must match all properties specificed. If the property
+            value to match is a list:
+
+            1. String properties must match all elements in the list, and
+            2. Other properties must match any of the elements in the list.
+
+            Note that now the propname in filterspec and prop in a 
+            sort/group spec may be transitive, i.e., it may contain
+            properties of the form link.link.link.name, e.g. you can search
+            for all issues where a message was added by a certain user in
+            the last week with a filterspec of
+            {'messages.author' : '42', 'messages.creation' : '.-1w;'}
             """
 
         def list(self):

roundup/backends/back_anydbm.py

@@ -15,7 +15,7 @@
 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
 #
-#$Id: back_anydbm.py,v 1.200 2006-07-08 18:28:18 schlatterbeck Exp $
+#$Id: back_anydbm.py,v 1.201 2006-08-21 12:19:48 schlatterbeck Exp $
 '''This module defines a backend that saves the hyperdatabase in a
 database chosen by anydbm. It is guaranteed to always be available in python
 versions >2.1.1 (the dumbdbm fallback in 2.1.1 and earlier has several
@@ -36,6 +36,7 @@
 import whichdb, os, marshal, re, weakref, string, copy, time, shutil, logging
 
 from roundup import hyperdb, date, password, roundupdb, security, support
+from roundup.support import reversed
 from roundup.backends import locking
 from roundup.i18n import _
 
@@ -1531,8 +1532,8 @@ def getnodeids(self, db=None, retired=None):
                 db.close()
         return res
 
-    def _filter(self, search_matches, filterspec, sort=(None,None),
-            group=(None,None), num_re = re.compile('^\d+$')):
+    def _filter(self, search_matches, filterspec, proptree,
+            num_re = re.compile('^\d+$')):
         """Return a list of the ids of the active nodes in this class that
         match the 'filter' spec, sorted by the group spec and then the
         sort spec.
@@ -1706,18 +1707,21 @@ def _filter(self, search_matches, filterspec, sort=(None,None),
                         k.append(v)
                 matches = k
 
-            # always sort by id if no other sort is specified
-            if sort == (None, None):
-                sort = ('+', 'id')
-
-            # add sorting information to the match entries
-            directions = []
+            # add sorting information to the proptree
             JPROPS = {'actor':1, 'activity':1, 'creator':1, 'creation':1}
-            for dir, prop in sort, group:
-                if dir is None or prop is None:
-                    continue
-                directions.append(dir)
+            children = []
+            if proptree:
+                children = proptree.sortable_children()
+            for pt in children:
+                dir = pt.sort_direction
+                prop = pt.name
+                assert (dir and prop)
                 propclass = props[prop]
+                pt.sort_ids = []
+                is_pointer = isinstance(propclass,(hyperdb.Link,
+                    hyperdb.Multilink))
+                if not is_pointer:
+                    pt.sort_result = []
                 try:
                     # cache the opened link class db, if needed.
                     lcldb = None
@@ -1738,33 +1742,42 @@ def _filter(self, search_matches, filterspec, sort=(None,None),
                             else:
                                 # the node doesn't have a value for this
                                 # property
-                                if isinstance(propclass, hyperdb.Multilink): v = []
-                                else: v = None
-                                entry.insert(0, v)
+                                v = None
+                                if isinstance(propclass, hyperdb.Multilink):
+                                    v = []
+                                if prop == 'id':
+                                    v = int (itemid)
+                                pt.sort_ids.append(v)
+                                if not is_pointer:
+                                    pt.sort_result.append(v)
                                 continue
 
                         # missing (None) values are always sorted first
                         if v is None:
-                            entry.insert(0, v)
+                            pt.sort_ids.append(v)
+                            if not is_pointer:
+                                pt.sort_result.append(v)
                             continue
 
-                        if isinstance(propclass, hyperdb.String):
-                            # it might be a string that's really an integer
-                            try: tv = int(v)
-                            except: v = v.lower()
-                            else: v = tv
-                        elif isinstance(propclass, hyperdb.Link):
+                        if isinstance(propclass, hyperdb.Link):
                             lcn = propclass.classname
                             link = self.db.classes[lcn]
                             key = link.orderprop()
+                            child = pt.propdict[key]
                             if key!='id':
                                 if not lcache.has_key(v):
                                     # open the link class db if it's not already
                                     if lcldb is None:
                                         lcldb = self.db.getclassdb(lcn)
                                     lcache[v] = self.db.getnode(lcn, v, lcldb)
-                                v = lcache[v][key]
-                        entry.insert(0, v)
+                                r = lcache[v][key]
+                                child.propdict[key].sort_ids.append(r)
+                            else:
+                                child.propdict[key].sort_ids.append(v)
+                        pt.sort_ids.append(v)
+                        if not is_pointer:
+                            r = propclass.sort_repr(pt.parent.cls, v, pt.name)
+                            pt.sort_result.append(r)
                 finally:
                     # if we opened the link class db, close it now
                     if lcldb is not None:
@@ -1773,29 +1786,6 @@ def _filter(self, search_matches, filterspec, sort=(None,None),
         finally:
             cldb.close()
 
-        # sort vals are inserted, but directions are appended, so reverse
-        directions.reverse()
-
-        if '-' in directions:
-            # one or more of the sort specs is in reverse order, so we have
-            # to use this icky function to sort
-            def sortfun(a, b, directions=directions, n=range(len(directions))):
-                for i in n:
-                    if not cmp(a[i], b[i]):
-                        continue
-                    if directions[i] == '+':
-                        # compare in the usual, ascending direction
-                        return cmp(a[i],b[i])
-                    else:
-                        # compare in the reverse, descending direction
-                        return cmp(b[i],a[i])
-                # for consistency, sort by the id if the items are equal
-                return cmp(a[-2], b[-2])
-            matches.sort(sortfun)
-        else:
-            # sorting is in the normal, ascending direction
-            matches.sort()
-
         # pull the id out of the individual entries
         matches = [entry[-2] for entry in matches]
         if __debug__:

roundup/backends/back_metakit.py

@@ -1,4 +1,4 @@
-# $Id: back_metakit.py,v 1.111 2006-08-18 01:36:58 richard Exp $
+# $Id: back_metakit.py,v 1.112 2006-08-21 12:19:48 schlatterbeck Exp $
 '''Metakit backend for Roundup, originally by Gordon McMillan.
 
 Known Current Bugs:
@@ -42,6 +42,7 @@
 BACKWARDS_COMPATIBLE = 1
 
 from roundup import hyperdb, date, password, roundupdb, security
+from roundup.support import reversed
 import logging
 import metakit
 from sessions_dbm import Sessions, OneTimeKeys
@@ -1155,8 +1156,7 @@ def addprop(self, **properties):
         self.db.commit()
     # ---- end of ping's spec
 
-    def _filter(self, search_matches, filterspec, sort=(None,None),
-            group=(None,None)):
+    def _filter(self, search_matches, filterspec, proptree):
         '''Return a list of the ids of the active nodes in this class that
         match the 'filter' spec, sorted by the group spec and then the
         sort spec
@@ -1326,37 +1326,58 @@ def ff(row, r=regexes):
             iv = v.filter(ff)
             v = v.remapwith(iv)
 
-        if sort or group:
-            sortspec = []
-            rev = []
-            for dir, propname in group, sort:
-                if propname is None: continue
-                isreversed = 0
-                if dir == '-':
-                    isreversed = 1
-                try:
-                    prop = getattr(v, propname)
-                except AttributeError:
+        # Handle all the sorting we can inside Metakit. If we encounter
+        # transitive attributes or a Multilink on the way, we sort by
+        # what we have so far and defer the rest to the outer sorting
+        # routine. We mark the attributes for which sorting has been
+        # done with sort_done. Of course the whole thing works only if
+        # we do it backwards.
+        sortspec = []
+        rev = []
+        sa = []
+        if proptree:
+            sa = reversed(proptree.sortattr)
+        for pt in sa:
+            if pt.parent != proptree:
+                break;
+            propname = pt.name
+            dir = pt.sort_direction
+            assert (dir and propname)
+            isreversed = 0
+            if dir == '-':
+                isreversed = 1
+            try:
+                prop = getattr(v, propname)
+            except AttributeError:
+                logging.getLogger("hyperdb").error(
+                    "MK has no property %s" % propname)
+                continue
+            propclass = self.ruprops.get(propname, None)
+            if propclass is None:
+                propclass = self.privateprops.get(propname, None)
+                if propclass is None:
                     logging.getLogger("hyperdb").error(
-                        "MK has no property %s" % propname)
+                        "Schema has no property %s" % propname)
                     continue
-                propclass = self.ruprops.get(propname, None)
-                if propclass is None:
-                    propclass = self.privateprops.get(propname, None)
-                    if propclass is None:
-                        logging.getLogger("hyperdb").error(
-                            "Schema has no property %s" % propname)
-                        continue
-                if isinstance(propclass, hyperdb.Link):
-                    linkclass = self.db.getclass(propclass.classname)
-                    lv = linkclass.getview()
-                    lv = lv.rename('id', propname)
-                    v = v.join(lv, prop, 1)
-                    prop = getattr(v, linkclass.orderprop())
-                if isreversed:
-                    rev.append(prop)
-                sortspec.append(prop)
-            v = v.sortrev(sortspec, rev)[:] #XXX Metakit bug
+            # Dead code: We dont't find Links here (in sortattr we would
+            # see the order property of the link, but this is not in the
+            # first level of the tree). The code is left in because one
+            # day we might want to properly implement this.  The code is
+            # broken because natural-joining to the Link-class can
+            # produce name-clashes wich result in broken sorting.
+            if isinstance(propclass, hyperdb.Link):
+                linkclass = self.db.getclass(propclass.classname)
+                lv = linkclass.getview()
+                lv = lv.rename('id', propname)
+                v = v.join(lv, prop, 1)
+                prop = getattr(v, linkclass.orderprop())
+            if isreversed:
+                rev.append(prop)
+            sortspec.append(prop)
+            pt.sort_done = True
+        sortspec.reverse()
+        rev.reverse()
+        v = v.sortrev(sortspec, rev)[:] #XXX Metakit bug
 
         rslt = []
         for row in v: