issue2550653: xapian search, stemming is not working

This is a partial fix for the issue. It does make stemming work
(so searching for silent will also return docs with silently in
them). However to do this we need to lowercase the text so the
porter stemmer will work. This means capitalization is not
preserved.

Tests in test/test_indexer for xapian backend all pass.

David Wolever (wolever) did the work.

Author: rouilj

CHANGES.txt

@@ -169,7 +169,14 @@ Fixed:
   the same exact nosy list. Fixed a missing reinitialization that has
   to occur every time though the loop in do_set. Manual tests work.
   (John Rouillard)
-  
+- issue2550653: xapian search, stemming is not working
+  This is a partial fix for the issue. It does make stemming work
+  (so searching for silent will also return docs with silently in
+  them). However to do this we need to lowercase the text so the
+  porter stemmer will work. This means capitalization is not
+  preserved. Fix done by David Wolever (wolever). Committed and doc
+  updates John Rouillard.
+
 2016-01-11: 1.5.1
 
 Pay attention:

doc/installation.txt

@@ -67,6 +67,11 @@ Xapian full-text indexer
 
   Roundup requires Xapian 1.0.0 or newer.
 
+  Note that capitalization is not preserved by the Xapian search.
+  This is required to make the porter stemmer work so that searching
+  for silent also returns documents with the word silently. Note that
+  the current stemming implementation is designed for English.
+
 Whoosh full-text indexer
   The Whoosh_ full-text indexer is also supported and will be used by
   default if it is available (and Xapian is not installed). This is

doc/upgrading.txt

@@ -65,6 +65,20 @@ setting this value in the [main] section of the tracker's
   # Possible values: xapian, whoosh, native (internal).
   indexer =
 
+Stemming improved in Xapian Indexer
+-----------------------------------
+
+Stemming allows a search for "silent" also match silently.  The Porter
+stemmer in Xapian works with lowercase English text. In this release we
+lowercase the documents as they are put into the indexer.
+
+This means capitalization is not preserved, but produces more hits by
+using the stemmer.
+
+You will need to do a roundup-admin reindex if you are using the
+Xapian full text indexer on your tracker.
+
+
 New config file option 'replyto_address' added
 ----------------------------------------------
 

roundup/backends/indexer_xapian.py

@@ -82,7 +82,7 @@ def add_text(self, identifier, text, mime_type='text/plain'):
             word = match.group(0)
             if self.is_stopword(word):
                 continue
-            term = stemmer(word)
+            term = stemmer(word.lower())
             doc.add_posting(term, match.start(0))
 
         database.replace_document(identifier, doc)
@@ -103,7 +103,7 @@ def find(self, wordlist):
         for term in [word.upper() for word in wordlist
                           if self.minlength <= len(word) <= self.maxlength]:
             if not self.is_stopword(term):
-                terms.append(stemmer(term))
+                terms.append(stemmer(term.lower()))
         query = xapian.Query(xapian.Query.OP_AND, terms)
 
         enquire.set_query(query)