Update Brad Knowles' mailman daily status report.

2 files changed, 141 insertions, 39 deletions

diff --git a/contrib/README.mmdsr b/contrib/README.mmdsr
index d7aa2543..5159171e 100644
--- a/contrib/README.mmdsr
+++ b/contrib/README.mmdsr
@@ -5,31 +5,41 @@ for mailman.  It was initially posted at
 <http://sourceforge.net/tracker/index.php?func=detail&aid=1123383&group_id=103&atid=300103>
 which see for possible patches and other enhancements.
 
-It is intended that there will be a 'cleaner', more complete readme file in
-the future.  In the interim, here are Brad's original comments.
-
-I quickly whacked together a Daily Status Report script for Mailman (using
-Bourne shell, not Python ;), and thought that other folks might be interested
-in seeing it.
-
-The basic concept is a program that gets fired off at 23:59 every night, and
-goes through a variety of log files looking for entries specific to that date,
-and indicating problems or certain types of activity that might be of interest
-to someone trying to administer the server.  It also does an "ls -la" of
-/usr/local/mailman/qfiles/*, so that you can see what is in the queue at the
-time of the running of the script.
-
-My concept was that this daily report would get e-mailed to the admin, or
-posted to a "reports" mailing list, where they could be archived and kept for
-future reference.
-
-The script does not (yet) do any statistics calculations, although it should
-be relatively easy to hack together some basic stats using awk, sort, etc....
-
-Anyway, I thought I'd share it and let folks take a look at it, and if anyone
-has any recommended improvements, we can incorporate those and share them back
-out with everyone.
-
-The code is written under a BSD-style license, so if you don't want to
-contribute any changes back to me, that's okay. Of course, I would prefer that
-you did, but I leave the choice up to you.
+Here goes the original mmdsr.readme by Brad ...
+
+========================================================================
+This is a basic Bourne shell script that I quickly hacked together for
+my own purposes, designed to be fired off at 23:59 every night, going
+through a variety of Mailman log files looking for entries specific
+to that date, summarizing the activities, and indicating problems or
+certain types of activity that might be of interest to someone trying
+to administer the server.
+
+It also does an "ls -la" of /usr/local/mailman/qfiles/*, so that you
+can see what is in the queue at the time of the execution of the script.
+
+This daily report will get e-mailed to the admin, or posted to a "reports"
+mailing list, where they can be archived and kept for future reference.
+If you don't define an address where the output e-mail should be sent,
+it will instead be printed to the standard output (thus allowing you to
+do something else with it).
+
+
+Once I'd gone through a few revisions of my own on this tool, I
+thought that I would release the code to the public and get comments
+and suggestions from others in the Mailman community.  This program is
+currently being used actively on the mail servers for python.org (where
+the mailman-users and other Mailman-related mailing lists are hosted),
+as well as many others.
+
+Note that this script needs to be configured once to know where standard
+commands are located, where log files are kept, etc... (see the top
+500 lines or so of the script), but after that you don't need to feed
+it any input, or capture the output to be sent anywhere.  This script
+takes care of all of that.  All you should need to do is to call this
+script from a cron job at 23:59 (local time) every night.
+
+
+When looking at this script, perhaps during configuration, please keep
+in mind that it is heavily commented at the top, and everything should
+hopefully be self-evident.
diff --git a/contrib/mmdsr b/contrib/mmdsr
index 77cd3c4c..3b6e49fd 100644
--- a/contrib/mmdsr
+++ b/contrib/mmdsr
@@ -112,6 +112,38 @@
 #		Adrian discovered that there was a log file format
 #		change between Mailman 2.1.5 and 2.1.6, which broke
 #		hourly statistics.
+#
+# 0.0.13	Update by Brad Knowles <brad@stop.mail-abuse.org>
+#		Updated on: Mon Dec 26 05:54:27 CET 2005
+#		Bugs found by Tom G. Christensen (tgc99):
+#		> ps output on solaris is full of whitespace but a further
+#		> echo get's rid of it.
+#		> The lines in the smtp log are sometimes broken up by a
+#		> newline (right before the msgid) which throws of the
+#		> summary. Piping it through sed first will rejoin the broken
+#		> lines.
+#		> Use $AWK instead of awk.
+#		Enhancements from Mark Sapiro:
+#		> The vette log summary lists posts held for moderation
+#		> individually under "Other Errors:". The following patch
+#		> (watch out for wrapped lines) summarizes them by list instead.
+#
+# 0.0.14	Update by Brad Knowles <brad@stop.mail-abuse.org>
+#		Updated on: Thu Dec 29 08:17:38 CET 2005
+#		Added code to check /usr/local/mailman/data for
+#		moderation hold queue
+#
+# 0.0.15	Update by Brad Knowles <brad@stop.mail-abuse.org>
+#		Updated on: Thu Jan 26 02:39:38 CET 2006
+#		Tweaked display of summary data in the "smtp-failure" log,
+#		and "other" category in the "vette" log, so as to reduce
+#		the spewage when things go wonky, mostly by removing
+#		unique-ifying data like message-id or sender address.
+#		Also tweaked moderation queue information, to tell us
+#		how many Python pickle files are in the data directory,
+#		as opposed to just doing a directory listing and skipping
+#		the files in the middle if there are too many -- now we
+#		know how many are being skipped.
 
 ###############################################################################
 # Set up locations of standard commands, directories, etc....
@@ -156,9 +188,19 @@ QUEUEDIR="/usr/local/mailman/qfiles"
 LOGDIR="/usr/local/mailman/logs"
 
 ###############################################################################
+# Mailman data directory
+###############################################################################
+DATADIR="/usr/local/mailman/data"
+
+###############################################################################
 # Maximum number of subdirectory entries to display in report
 ###############################################################################
-MAX_LINES=20
+MAX_QUEUE_LINES=20
+
+###############################################################################
+# Maximum number of moderation queue pickle files to display in report
+###############################################################################
+MAX_DATA_LINES=100
 
 ###############################################################################
 # Mailman Log files to check for errors.
@@ -182,8 +224,8 @@ SUBDIRS="archive bounces commands in news out retry shunt virgin"
 # Specify sender address for report.  Not used if recipient list is empty.
 ###############################################################################
 
-SENDER="put@your.address.here"
-RCPTS="put@the.address.of,your@recipients.here"
+SENDER="INSERT.YOUR.SENDER.ADDRESS@HERE"
+RCPTS="INSERT.YOUR.RECIPIENTS.ADDRESSES@HERE"
 
 ###############################################################################
 # If you run this program in cron at 23:59:00, you need to sleep sixty
@@ -203,7 +245,8 @@ SLEEPTIME=60
 #	there is an easier cross-platform way to do it, please let me know.
 ###############################################################################
 
-MYUID=`$PS -o user -p $$ | $TAIL -1`
+GRABUID=`$PS -o user -p $$ | $TAIL -1`
+MYUID=`echo $GRABUID`
 RUNAS="mailman"
 
 ###############################################################################
@@ -254,7 +297,7 @@ do
     $TOUCH $TMPLOG
     echo "Log file: $LOG" >> $TMP
     echo "==============================" >> $TMP
-    $GREP -si "^$DAY [0-9][0-9:]* $YEAR" $LOGDIR/$LOG >> $TMPLOG
+    $SED -e :a -e '$!N;s/\n //;ta' -e 'P;D' $LOGDIR/$LOG | $GREP -si "^$DAY [0-9][0-9:]* $YEAR" >> $TMPLOG
 
     if [ -f "$LOGDIR/${LOG}" ] ; then
 
@@ -264,7 +307,7 @@ do
 	    echo "Hourly Summary of Posts" >> $TMP
 	    echo "-----------------------" >> $TMP
 
-	    $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e 's/:.*$//' $TMPLOG | $UNIQ -c | $SORT -n +1 | awk '{ printf( "%8d  %02d:00-%02d:59\n", $1, $2, $2 ) }' >> $TMP
+	    $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e 's/:.*$//' $TMPLOG | $UNIQ -c | $SORT -n +1 | $AWK '{ printf( "%8d  %02d:00-%02d:59\n", $1, $2, $2 ) }' >> $TMP
 
 	    echo "" >> $TMP
 	    echo "Post Count by List" >> $TMP
@@ -295,7 +338,7 @@ do
 	    echo "" >> $TMP
 	    echo "Hourly Summary of Messages Sent" >> $TMP
 	    echo "-------------------------------" >> $TMP
-	    $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e 's/:.* for / /' -e 's/ recips,.*$//' $TMPLOG | awk '{ val=int($1); sum[val]+=$2 } END { for (i=0; i<24; i++) { printf "%8d %02d:00-%02d:59\n", sum[i], i, i } }' >> $TMP
+	    $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e 's/:.* for / /' -e 's/ recips,.*$//' $TMPLOG | $AWK '{ val=int($1); sum[val]+=$2 } END { for (i=0; i<24; i++) { printf "%8d %02d:00-%02d:59\n", sum[i], i, i } }' >> $TMP
 
 	else
 
@@ -416,9 +459,19 @@ do
 	    $GREP -i 'Posting to a moderated newsgroup' $TMPLOG | $AWK '{ print $6 }' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
 
 	    echo "" >> $TMP
+	    echo "Post to moderated list (by list):" >> $TMP
+	    echo "------------------------------" >> $TMP
+	    $GREP -i 'Post to moderated list' $TMPLOG | $AWK '{ print $6 }' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	    echo "" >> $TMP
 	    echo "Other Errors:" >> $TMP
 	    echo "------------------------------" >> $TMP
-	    $EGREP -vi '(Post by non-member|suspicious header|message approved|Discarded posting|bulk message discarded|junk message discarded|Message has implicit destination|Posting to a moderated newsgroup|Message discarded, msgid)' $TMPLOG | $SED 's/^.* ([0-9]*) //' | $SORT | $UNIQ -c | $SORT -nr >> $TMP 
+MYUID=`$PS -o user -p $$ | $TAIL -1`
+	    $EGREP -vi '(Post by non-member|suspicious header|message approved|Discarded posting|bulk message discarded|junk message discarded|Message has implicit destination|Posting to a moderated newsgroup|Post to moderated list|Message discarded, msgid)' $TMPLOG | $SED -e 's/^.* ([0-9]*) //' -e 's/, message-id=<[^> ]*>:/:/' | $SORT | $UNIQ -c | $SORT -nr >> $TMP 
+
+	elif [ "${LOG}" = "smtp-failure" ] ; then
+
+	    $SED 's/^.* ([0-9]*) //' $TMPLOG | $SED 's/delivery to [^@ ]*@[^@ ]* failed with code/delivery failed with code/g' | $SORT | $UNIQ -c | $SORT -nr >> $TMP 
 
 	else
 
@@ -446,7 +499,7 @@ do
     $RM -f $TMPLOG
     $TOUCH $TMPLOG
     $LS -la $DIR >> $TMPLOG
-    LINES=`wc -l < $TMPLOG`
+    LINES=`$WC -l < $TMPLOG`
     LINES=`expr $LINES - 3`
 
     echo "Subdirectory: $DIR" >> $TMP
@@ -454,12 +507,12 @@ do
     echo "Entries: $LINES" >> $TMP
     echo "------------------------------" >> $TMP
 
-    if [ $LINES -le $MAX_LINES ] ; then
+    if [ $LINES -le $MAX_QUEUE_LINES ] ; then
     	$CAT $TMPLOG >> $TMP
     else
     	$HEAD $TMPLOG >> $TMP
 	echo "" >> $TMP
-	echo "	More than $MAX_LINES total entries, skipping ..." >> $TMP
+	echo "	More than $MAX_QUEUE_LINES total entries, skipping ..." >> $TMP
 	echo "" >> $TMP
     	$TAIL $TMPLOG >> $TMP
     fi
@@ -467,6 +520,45 @@ do
 
 done
 
+echo "******************************" >> $TMP
+echo "Data Directory Contents" >> $TMP
+echo "	(Python pickle files only)" >> $TMP
+echo "******************************" >> $TMP
+
+cd $DATADIR
+
+$RM -f $TMPLOG
+$TOUCH $TMPLOG
+$LS -la | $GREP -i '\.pck$' >> $TMPLOG
+LINES=`$WC -l < $TMPLOG`
+
+echo "Python pickle files: $LINES" >> $TMP
+echo "==============================" >> $TMP
+echo "" >> $TMP
+
+if [ $LINES -le $MAX_DATA_LINES ] ; then
+    $CAT $TMPLOG >> $TMP
+else
+    $HEAD $TMPLOG >> $TMP
+    echo "" >> $TMP
+    echo "	More than $MAX_DATA_LINES total entries, skipping ..." >> $TMP
+    echo "" >> $TMP
+    $TAIL $TMPLOG >> $TMP
+fi
+echo "" >> $TMP
+
+echo "******************************" >> $TMP
+echo "Moderation Queue Summary" >> $TMP
+echo "******************************" >> $TMP
+echo "" >> $TMP
+
+$AWK '{ print $NF }' $TMPLOG | $GREP -i '^heldmsg-' | $SED -e 's/^heldmsg-//' -e 's/-[0-9]*\.pck$//' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+echo "" >> $TMP
+
+###############################################################################
+# Okay, we're done with all the code stuff, now handle the output
+###############################################################################
+
 if [ "${RCPTS}x" != "x" ] ; then
     $SENDMAIL -t -f$SENDER < $TMP
 else