Changes to Mailman to support SWISH-E searching

Last update: $Id: mailman.html,v 1.7 2008/07/04 22:31:07 jordan Exp jordan $

This document is still being worked on!

Here are a few changes I made to my Mailman instalation.

I'm presently using Mailman v2.1.11, and you should too. But if you're reading this and v2.1.11 isn't current, please tell me!

If you don't run Mailman already, install and learn about that first. Then install and learn about SWISH-E. Both of these have to be up and running before this will make any sense to you. I'm using Apache so if you're using some other web server, good luck.

I've made both of these changes to several Mailman installations, but the one you should look at is for a mailing list run by Doug Henwood called lbo-talk.

I put a SWISH-E search box into my Mailman templates. This involved changing the templates that Mailman uses to add a few things:

More fields to search on like author and date

SWISH-E uses HTML META tags as hints to the indexer for properties that you can then search on. We want some new fields to search on. This is a simplistic view of things, but there you have it. This was accomplished by adding some code to templates/en/article.html

  diff -r1.1 templates/en/article.html
  8a9,11
  >    <META NAME="author" CONTENT="%(author_html)s">
  >    <META NAME="datestr" CONTENT="%(datestr_html)s">
  >    <META NAME="unixdate" CONTENT="%(date)s">

You need both "datestr" and "unixdate" because in one case you want SWISH-E to display the (message) date and in the other you want an integer that you can compare for sorting by the date. You can add more here if you want. I couldn't think of any. Notice that I only did the English version of the templates. YMMV. See Figure 1. If you just want this for one list, you'll need to move them into ~mailman/lists/{your-list}/en/ and then restart mailman. In a generated archive entry, right under the "robots" META NAME, you should see something like this:

  <META NAME="author" CONTENT="Jordan Hayes">
  <META NAME="datestr" CONTENT="Sun Jan  2 13:44:55 PST 2005">
  <META NAME="unixdate" CONTENT="01104702295">

A SWISH-E search entry box to each message

SWISH-E includes a ("sample") CGI script called swish.cgi that needs one minor change; there's a bit of Perl code to figure out whether reversing the meaning of the sort field is ascending or descending. It's around line 1517 or so in my version. It looks like this:


  my $direction = $sortby eq 'swishrank' || $sortby eq 'unixdate'
     ? $q->param('reverse') ? 'asc' : 'desc'
     : $q->param('reverse') ? 'desc' : 'asc';

Just add the bold code to the first line, because dates are like 'rank' (a measure of how good the match was) too: you want the default to be to show the most recently posted message. You're almost done. Now you just need to add a form to templates that you want to be able to search from. Here's an example:


<B>Search %(listname)s Archives</B>
<form method="get" action="/%(listname)s/search/swish.cgi"
 enctype="application/x-www-form-urlencoded" class="form">
  <input maxlength="200" value="" size="32" type="text" name="query"/>
  <input value="Search!" type="submit" name="submit"/>
Limit search to:
  <input type="radio" name="metaname" value="swishdefault" checked/>Subject & Body
  <input type="radio" name="metaname" value="swishtitle"/>Subject
  <input type="radio" name="metaname" value="author"/>Author
<br>
Sort by:
  <select name="sort">
    <option selected value="unixdate">Date
    <option value="swishrank">Rank
    <option value="author">Author
    <option value="swishtitle">Subject
  </select>
  <input type="checkbox" name="reverse" value="on"/>Reverse Sort
</form>

For me, that's:

templates/en/archidxhead.html
templates/en/archidxfoot.html
templates/en/archtoc.html
templates/en/archtocnombox.html
templates/en/article.html

You'll need to rebuild your archives after you change the templates, like this:

    % cd ~mailman
    % bin/mailmanctl stop
    % bin/arch --wipe lbo-talk
    % bin/mailmanctl start

Your .swishcgi.conf file might look something like this:


{
    title           => 'Search LBO-Talk Archives',
    metanames       => [qw/swishdefault swishdocsize swishtitle author/],
    display_props   => [qw/swishdocsize author datestr/],
    sorts           => [qw/unixdate swishrank author swishtitle/],
    swish_index     => [qw( index.older index.ThisYear index.swish-e )],

    name_labels => {
        author              => 'Author',
        unixdate            => 'Date',
        swishdefault        => 'Subject & Body',
        swishtitle          => 'Subject',
        swishrank           => 'Rank',
        swishlastmodified   => 'Last Modified Date',
        swishdocpath        => 'Document Path',
        swishdocsize        => 'Document Size',
        all                 => 'All',              # group of metanames

        subject             => 'Message Subject',  # other examples
        name                => "Poster's Name",
        email               => "Poster's Email",
        sent                => 'Message Date',
    },
};

Your swish-e.conf file might look something like this:


IndexDir archives
IndexOnly .html
IndexReport 0
IndexContents HTML* .html
StoreDescription HTML* <PRE> 1000

PropertyNames author datestr
PropertyNamesDate unixdate
MetaNames author datestr swishtitle unixdate

FollowSymLinks yes

FileRules filename is author.html
FileRules filename is date.html
FileRules filename is index.html
FileRules filename is subject.html
FileRules filename is thread.html

ReplaceRules regex  !^archives/!http://mailman.lbo-talk.org/pipermail/lbo-talk/!

I wrote a simple RSS feed generator to use in conjunction with this setup. I think a lot of people have flamed out on getting RSS support into Mailman, but since I'm already creating the SWISH-E index, I can use it for the feed. Here's the source code ... it's only about 100 lines of code, so you should be able to tailor it to your MetaNames if you don't use the ones I do. You'll need to have the SWISH-E C library installed, and you'll want to grab libcgi. Other CGI-helper libraries should be simple to integrate. Put the resulting binary into your CGI directory. Let me know if you come up with something better. Watching this URL in your favorite reader should give you a full feed:

http://search.lbo-talk.org/search/feed

You can perform special searches for subset-feeds:

http://search.lbo-talk.org/search/feed?subject=kill
http://search.lbo-talk.org/search/feed?author=Doug
http://search.lbo-talk.org/search/feed?msg=Zizek

That last one looks in both Subject & Body like the SWISH-E search.

Last update: $Id: mailman.html,v 1.7 2008/07/04 22:31:07 jordan Exp jordan $