- CGI/admin.py

   The email address which forms a part of the various CGI data keys
   in the admin membership list is now urllib.quote()ed. This allows
   changing options for and unsubbing an address which contains a
   double-quote character.

- CGI/admindb.py
   Added additional test to not display "Database Updated ..." when
   coming from the login page.

- CGI/roster.py, HTMLFormatter.py
   Changed to show hidden members when authorization is site or list's
   admin or moterator password.  Patch 1587651.

- Defaults.py.in, Handlers/Cleanse_DKIM.py
   Added a new REMOVE_DKIM_HEADERS Defaults.py/mm_cfg.py setting (default
   = No) to control removing dkim/domainkey signatures from posts and
   mail to -owner.

- Handlers/Decorate.py, Handlers/Scrubber.py
   Changed to preserve format=flowed and delsp=yes in the Content-Type:
   of the body when adding header/footer and when scrubbing attachments
   and to remove trailing spaces from the header/footer lines so they
   won't be flowed. Bug 1495122.

   Fixed a scrubber issue where the i18n translated 'next part' separator
   can be garbled if the list charset is different from the message.

- Queue/Runner.py. Queue/Switchboard.py
   Now that we have .bak queue entries for recovery, it is no longer the
   case that an unparseable message is lost.  In this case, and in case
   of other exceptions when dequeueing, I added a preservation feature
   to move the .bak file to qfiles/shunt as a .psv file and write an
   appropriate log entry.  It is also possible for an attempt to shunt
   a message to fail.  One example that occurred in practice (bug 1656289)
   was caused by a huge message that threw a MemoryError in processing and
   then threw another MemoryError in the attempt to pickle the message for
   the shunt queue.  In this case as well, I log and attempt to preserve
   the original queue entry by renaming.

1 files changed, 27 insertions, 13 deletions

diff --git a/Mailman/Queue/Runner.py b/Mailman/Queue/Runner.py
index e229043e..1724f043 100644
--- a/Mailman/Queue/Runner.py
+++ b/Mailman/Queue/Runner.py
@@ -1,4 +1,4 @@
-# Copyright (C) 1998-2006 by the Free Software Foundation, Inc.
+# Copyright (C) 1998-2007 by the Free Software Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -98,16 +98,17 @@ class Runner:
                 # Ask the switchboard for the message and metadata objects
                 # associated with this filebase.
                 msg, msgdata = self._switchboard.dequeue(filebase)
-            except email.Errors.MessageParseError, e:
-                # It's possible to get here if the message was stored in the
-                # pickle in plain text, and the metadata had a _parsemsg key
-                # that was true, /and/ if the message had some bogosity in
-                # it.  It's almost always going to be spam or bounced spam.
-                # There's not much we can do (and we didn't even get the
-                # metadata, so just log the exception and continue.
+            except Exception, e:
+                # This used to just catch email.Errors.MessageParseError,
+                # but other problems can occur in message parsing, e.g.
+                # ValueError, and exceptions can occur in unpickling too.
+                # We don't want the runner to die, so we just log and skip
+                # this entry, but preserve it for analysis.
                 self._log(e)
-                syslog('error', 'Ignoring unparseable message: %s', filebase)
-                self._switchboard.finish(filebase)
+                syslog('error',
+                       'Skipping and preserving unparseable message: %s',
+                       filebase)
+                self._switchboard.finish(filebase, preserve=True)
                 continue
             try:
                 self._onefile(msg, msgdata)
@@ -122,9 +123,22 @@ class Runner:
                 self._log(e)
                 # Put a marker in the metadata for unshunting
                 msgdata['whichq'] = self._switchboard.whichq()
-                new_filebase = self._shunt.enqueue(msg, msgdata)
-                syslog('error', 'SHUNTING: %s', new_filebase)
-                self._switchboard.finish(filebase)
+                # It is possible that shunting can throw an exception, e.g. a
+                # permissions problem or a MemoryError due to a really large
+                # message.  Try to be graceful.
+                try:
+                    new_filebase = self._shunt.enqueue(msg, msgdata)
+                    syslog('error', 'SHUNTING: %s', new_filebase)
+                    self._switchboard.finish(filebase)
+                except Exception, e:
+                    # The message wasn't successfully shunted.  Log the
+                    # exception and try to preserve the original queue entry
+                    # for possible analysis.
+                    self._log(e)
+                    syslog('error',
+                           'SHUNTING FAILED, preserving original entry: %s',
+                           filebase)
+                    self._switchboard.finish(filebase, preserve=True)
             # Other work we want to do each time through the loop
             Utils.reap(self._kids, once=True)
             self._doperiodic()