Initial Mailman Daily Status Report script.

2 files changed, 509 insertions, 0 deletions

diff --git a/contrib/mmdsr b/contrib/mmdsr
new file mode 100644
index 00000000..77cd3c4c
--- /dev/null
+++ b/contrib/mmdsr
@@ -0,0 +1,476 @@
+#!/bin/sh
+###############################################################################
+# mmdsr -- Mailman Daily Status Report (cron job, run at 23:59)
+###############################################################################
+# Copyright (c) 2005, Brad Knowles
+# All rights reserved.
+# 
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 
+#     Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+# 
+#     Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer
+#     in the documentation and/or other materials provided with the
+#     distribution.
+# 
+#     The name of the author or other contributors may not be used
+#     to endorse or promote products derived from this software without
+#     specific prior written permission.
+# 
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+# FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+# COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
+# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+###############################################################################
+
+###############################################################################
+# Version history
+###############################################################################
+# 0.0.1		Initial creation by Brad Knowles <brad@stop.mail-abuse.org>
+#		Created on: Tue Feb 15 04:01:20 PST 2005
+#
+# 0.0.2         Update by Brad Knowles <brad@stop.mail-abuse.org>
+#               Updated on: Wed Feb 16 18:55:52 CET 2005
+#               Feedback from Tom G. Christensen (tgc99):
+#               > The current UID grab command doesn't work on Solaris 
+#               > (2.6 & 8 tested).      
+#               >
+#               > I'd recommend this instead:
+#               > ps -o user -p $$|tail -1
+#               >
+#               > This is tested and works on RH 6.2, RH 7.3, RHEL 2.1,
+#               > RHEL3, FC3, FreeBSD 4.9, Solaris 2.6, 8.
+#
+# 0.0.3         Update by Brad Knowles <brad@stop.mail-abuse.org>
+#               Updated on: Sun Mar 13 01:15:24 CET 2005
+#               Noted errors from grep when trying to search a nonexistant
+#               file.  Added "-s" option to compensate, and extra code to
+#		notify the admin if the log file doesn't exist.
+#
+# 0.0.4         Update by Brad Knowles <brad@stop.mail-abuse.org>
+#               Updated on: Sun Mar 13 01:58:37 CET 2005
+#               Eliminate some extra cruft from the "fromusenet" logs.
+#
+# 0.0.5         Update by Brad Knowles <brad@stop.mail-abuse.org>
+#               Updated on: Wed Apr 13 00:54:42 CEST 2005
+#		Eliminate a lot of extra cruft from the "error" & "vette" logs.
+#               Eliminate more cruft from the "fromusenet" logs.
+#
+# 0.0.6         Update by Brad Knowles <brad@stop.mail-abuse.org>
+#               Updated on: Thu Apr 14 03:07:08 CEST 2005
+#               Eliminate even more cruft from the "fromusenet" logs.
+#               Eliminate more cruft from the "vette" logs.
+#               Eliminate cruft from the "mischief" logs.
+#
+# 0.0.7         Update by Brad Knowles <brad@stop.mail-abuse.org>
+#               Updated on: Sun May  1 22:29:13 CEST 2005   [guess]
+#		The "error" log "no such list" errors didn't need to be
+#		quite so "compressed", open them up to be more readable.
+#		Also eliminate all use of "xargs", use "fmt -w 75" instead.
+#
+# 0.0.8         Update by Brad Knowles <brad@stop.mail-abuse.org>
+#               Updated on: Sun May  1 23:09:13 CEST 2005
+#		Clean up code for queue subdirectories to check.
+#
+# 0.0.9         Update by Brad Knowles <brad@stop.mail-abuse.org>
+#               Updated on: Wed May 11 01:23:40 CEST 2005
+#		Eliminate date/time stamps from the rest of the logs,
+#		then pipe through `sort | uniq -c | sort -nr`
+#		so that all possible duplicates are eliminated.
+#
+# 0.0.10	Update by Brad Knowles <brad@stop.mail-abuse.org>
+#		Updated on: Tue Jul 12 16:53:49 CDT 2005
+#		Add code to do summary of normal list activity
+#
+# 0.0.11	Update by Brad Knowles <brad@stop.mail-abuse.org>
+#		Updated on: Tue Sep  6 15:44:19 CEST 2005
+#		Tweaked display of Mailman/qfiles subdirectories
+#		so as to avoid displaying more than $MAX_LINES
+#		output, which really helps if you have thousands
+#		of bounces sitting around, etc....
+#		Also slightly tweaked display of fromusenet logs
+#		to split summary from errors.
+#
+# 0.0.12	Update by Brad Knowles <brad@stop.mail-abuse.org>
+#		Updated on: Thu Sep 22 22:37:35 CEST 2005
+#		Bugs found by Mark Sapiro and Adrian Wells.
+#		Mark suggested splitting the Mailman log directory
+#		from the rest of the Mailman source and queues,
+#		as well as making sure to clean up all temp files.
+#		Adrian discovered that there was a log file format
+#		change between Mailman 2.1.5 and 2.1.6, which broke
+#		hourly statistics.
+
+###############################################################################
+# Set up locations of standard commands, directories, etc....
+# You may need to modify these for your platform or installation.
+###############################################################################
+
+AWK="/usr/bin/awk"
+BASENAME="/usr/bin/basename"
+CAT="/bin/cat"
+DATE="/bin/date"
+EGREP="/bin/egrep"
+FMT="/usr/bin/fmt"
+GREP="/bin/grep"
+HEAD="/usr/bin/head"
+LS="/bin/ls"
+PS="/bin/ps"
+RM="/bin/rm"
+SED="/bin/sed"
+SENDMAIL="/usr/sbin/sendmail"
+SLEEP="/bin/sleep"
+SORT="/usr/bin/sort"
+TAIL="/usr/bin/tail"
+TOUCH="/bin/touch"
+TR="/usr/bin/tr"
+UNIQ="/usr/bin/uniq"
+XARGS="/usr/bin/xargs"
+WC="/usr/bin/wc"
+
+###############################################################################
+# Directory for temporary files
+###############################################################################
+TMPDIR="/tmp"
+
+###############################################################################
+# Mailman queue directory
+###############################################################################
+QUEUEDIR="/usr/local/mailman/qfiles"
+
+###############################################################################
+# Mailman log directory
+###############################################################################
+LOGDIR="/usr/local/mailman/logs"
+
+###############################################################################
+# Maximum number of subdirectory entries to display in report
+###############################################################################
+MAX_LINES=20
+
+###############################################################################
+# Mailman Log files to check for errors.
+# No need to specify path, only log file name.
+###############################################################################
+ERR_LOGS="error fromusenet locks mischief post qrunner smtp-failure vette"
+
+###############################################################################
+# Mailman Log files to summarize.
+# No need to specify path, only log file name.
+###############################################################################
+SUM_LOGS="fromusenet post smtp"
+
+###############################################################################
+# Mailman queue subdirectories to check.  No need to specify path, only name.
+###############################################################################
+SUBDIRS="archive bounces commands in news out retry shunt virgin"
+
+###############################################################################
+# Specify recipients for report.  If none, then simply print to "STDOUT".
+# Specify sender address for report.  Not used if recipient list is empty.
+###############################################################################
+
+SENDER="put@your.address.here"
+RCPTS="put@the.address.of,your@recipients.here"
+
+###############################################################################
+# If you run this program in cron at 23:59:00, you need to sleep sixty
+# seconds to make sure that you capture all the logs for the previous day.
+# Comment this line out to disable sleeping altogether.
+###############################################################################
+
+SLEEPTIME=60
+
+###############################################################################
+# What user id is Mailman installed under?
+# How do we determine what UID this program is running as?
+# NB:	There will probably be SysV vs. BSD semantic differences here.
+#	Be careful.  Make sure the command you specify actually works.
+#	The command string specified here may seem arcane, but should pull
+#	out the information we need, and throw away everything else.  If
+#	there is an easier cross-platform way to do it, please let me know.
+###############################################################################
+
+MYUID=`$PS -o user -p $$ | $TAIL -1`
+RUNAS="mailman"
+
+###############################################################################
+# No modifications below this line should be required.
+###############################################################################
+
+DAY=`$DATE '+%b %d'`
+YEAR=`$DATE '+%Y'`
+
+PROG=`$BASENAME $0`
+
+if [ "${MYUID}" != "${RUNAS}" -a "${MYUID}" != "root" ] ; then
+    echo "ERROR: You must be root or ${RUNAS} to run $PROG."
+    exit 1
+fi
+
+if [ "${SLEEPTIME}" != "" -a "${RCPTS}x" != "${SENDER}x" ] ; then
+    if [ $SLEEPTIME -gt 0 ] ; then
+	$SLEEP $SLEEPTIME
+    fi
+fi
+
+TMP="$TMPDIR/.$PROG.$$"
+TMPLOG="$TMP.log"
+$RM -f $TMP
+$TOUCH $TMP
+
+if [ "${RCPTS}x" != "x" ] ; then
+    echo "From: ${SENDER}" >> $TMP
+    echo "To: ${RCPTS}" >> $TMP
+    echo "Subject: Mailman Daily Status Report: $DAY $YEAR" >> $TMP
+    echo "" >> $TMP
+fi
+
+echo "		Mailman Daily Status Report: $DAY $YEAR" >> $TMP
+echo "" >> $TMP
+echo "" >> $TMP
+
+echo "******************************" >> $TMP
+echo "Log File Summary" >> $TMP
+echo "******************************" >> $TMP
+echo "" >> $TMP
+
+for LOG in $SUM_LOGS
+do
+
+    $RM -f $TMPLOG
+    $TOUCH $TMPLOG
+    echo "Log file: $LOG" >> $TMP
+    echo "==============================" >> $TMP
+    $GREP -si "^$DAY [0-9][0-9:]* $YEAR" $LOGDIR/$LOG >> $TMPLOG
+
+    if [ -f "$LOGDIR/${LOG}" ] ; then
+
+	if [ "${LOG}" = "post" ] ; then
+
+	    echo "" >> $TMP
+	    echo "Hourly Summary of Posts" >> $TMP
+	    echo "-----------------------" >> $TMP
+
+	    $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e 's/:.*$//' $TMPLOG | $UNIQ -c | $SORT -n +1 | awk '{ printf( "%8d  %02d:00-%02d:59\n", $1, $2, $2 ) }' >> $TMP
+
+	    echo "" >> $TMP
+	    echo "Post Count by List" >> $TMP
+	    echo "------------------" >> $TMP
+
+	    $SED -e 's/^.* post to //' -e 's/ .*$//' $TMPLOG | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	    echo "" >> $TMP
+	    echo "Post Count by Sender" >> $TMP
+	    echo "--------------------" >> $TMP
+
+	    $SED -e 's/^.* from //' -e 's/,.*$//' $TMPLOG | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	elif [ "${LOG}" = "fromusenet" ] ; then
+
+	    GRPS=`$EGREP -vi '(watermark:|nothing new| posted to |: \[[0-9\.]*\])' $TMPLOG | $GREP -i ' gating ' | $AWK '{ print $7 }' | $SORT -u | $FMT -w 75`
+
+	    for GRP in $GRPS
+	    do
+		echo "" >> $TMP
+		echo "$GRP Article #'s Gated:" >> $TMP
+		echo "------------------------------" >> $TMP
+		$EGREP -vi '(watermark:|nothing new| posted to |: \[[0-9\.]*\])' $TMPLOG | $GREP -i " gating " | $GREP -i " $GRP " | $SED -e 's/^.*\[//' -e 's/\]$//' -e 's/\.\./ /' | $TR ' ' '\n' |  $SORT -u | $FMT -w 75 >> $TMP
+	    done
+
+	elif [ "${LOG}" = "smtp" ] ; then
+
+	    echo "" >> $TMP
+	    echo "Hourly Summary of Messages Sent" >> $TMP
+	    echo "-------------------------------" >> $TMP
+	    $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e 's/:.* for / /' -e 's/ recips,.*$//' $TMPLOG | awk '{ val=int($1); sum[val]+=$2 } END { for (i=0; i<24; i++) { printf "%8d %02d:00-%02d:59\n", sum[i], i, i } }' >> $TMP
+
+	else
+
+	    $SED 's/^.* ([0-9]*) //' $TMPLOG | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	fi
+
+    else
+
+    	echo " ### Log file ${LOG} does not exist ### " >> $TMP
+
+    fi
+
+    echo "" >> $TMP
+
+done
+
+echo "******************************" >> $TMP
+echo "Log File Squawks" >> $TMP
+echo "******************************" >> $TMP
+echo "" >> $TMP
+
+for LOG in $ERR_LOGS
+do
+    $RM -f $TMPLOG
+    $TOUCH $TMPLOG
+    echo "Log file: $LOG" >> $TMP
+    echo "==============================" >> $TMP
+    $GREP -si "^$DAY [0-9][0-9:]* $YEAR" $LOGDIR/$LOG >> $TMPLOG
+
+    if [ -f "$LOGDIR/${LOG}" ] ; then
+
+	if [ "${LOG}" = "error" ] ; then
+
+	    echo "Uncaught Runner Exceptions:" >> $TMP
+	    echo "------------------------------" >> $TMP
+	    $GREP 'Uncaught runner exception' $TMPLOG | $SED 's/^.*exception: //' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	    echo "" >> $TMP
+	    echo "No Such List:" >> $TMP
+	    echo "------------------------------" >> $TMP
+	    $GREP 'No such list' $TMPLOG | $SED -e 's/^.* "//' -e 's/".*$//' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	    CNT=`$GREP -i 'shunting' $TMPLOG | $WC -l`
+	    if [ "${CNT}x" != "x" ] ; then
+		if [ ${CNT} -gt 0 ] ; then
+		    echo "" >> $TMP
+		    echo "Shunting Count: $CNT" >> $TMP
+		    echo "------------------------------" >> $TMP
+		fi
+	    fi
+
+	    echo "" >> $TMP
+	    echo "Other Errors:" >> $TMP
+	    echo "------------------------------" >> $TMP
+	    $EGREP -vi '(Uncaught runner exception|No such list|Ignoring unparseable message|Traceback|shunting)' $TMPLOG | $SED 's/^.* ([0-9]*) //' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	elif [ "${LOG}" = "fromusenet" ] ; then
+
+	    $EGREP -vi '(watermark:|nothing new| posted to |: \[[0-9\.]*\])' $TMPLOG | $GREP -vi " gating " | $SED 's/^.* ([0-9]*) //' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	elif [ "${LOG}" = "mischief" ] ; then
+
+	    echo "" >> $TMP
+	    echo "Login failure with private rosters (by user):" >> $TMP
+	    echo "------------------------------" >> $TMP
+	    $GREP -i 'Login failure with private rosters' $TMPLOG | $AWK '{ print $NF }' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	    echo "" >> $TMP
+	    echo "Unsub attempt of non-member (by user):" >> $TMP
+	    echo "------------------------------" >> $TMP
+	    $GREP -i 'Unsub attempt of non-member' $TMPLOG | $AWK '{ print $NF }' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	    echo "" >> $TMP
+	    echo "Reminder attempt of non-member (by user):" >> $TMP
+	    echo "------------------------------" >> $TMP
+	    $GREP -i 'Reminder attempt of non-member' $TMPLOG | $AWK '{ print $NF }' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	    echo "" >> $TMP
+	    echo "Other Messages:" >> $TMP
+	    echo "------------------------------" >> $TMP
+	    $EGREP -vi '(Login failure with private rosters|Unsub attempt of non-member|Reminder attempt of non-member)' $TMPLOG | $SED 's/^.* ([0-9]*) //' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	elif [ "${LOG}" = "post" ] ; then
+
+	    $GREP -vi 'success' $TMPLOG | $SED 's/^.* ([0-9]*) //' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	elif [ "${LOG}" = "vette" ] ; then
+
+	    echo "" >> $TMP
+	    echo "Message held -- Post by non-member (by list):" >> $TMP
+	    echo "------------------------------" >> $TMP
+	    $GREP 'Post by non-member' $TMPLOG | $AWK '{ print $6 }' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	    echo "" >> $TMP
+	    echo "Message held -- Suspicious header (by list):" >> $TMP
+	    echo "------------------------------" >> $TMP
+	    $GREP -i 'suspicious header' $TMPLOG | $AWK '{ print $6 }' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	    echo "" >> $TMP
+	    echo "Discarded posting (by list):" >> $TMP
+	    echo "------------------------------" >> $TMP
+	    $GREP -i 'Discarded posting' $TMPLOG | $AWK '{ print $6 }' | $SED 's/:$//' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	    echo "" >> $TMP
+	    echo "Bulk/Junk message discarded (by list):" >> $TMP
+	    echo "------------------------------" >> $TMP
+	    $EGREP -i '(bulk|junk) message discarded' $TMPLOG | $AWK '{ print $NF }' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	    echo "" >> $TMP
+	    echo "Implicit destination (by list):" >> $TMP
+	    echo "------------------------------" >> $TMP
+	    $GREP -i 'Message has implicit destination' $TMPLOG | $AWK '{ print $6 }' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	    echo "" >> $TMP
+	    echo "Post to moderated newsgroup (by list):" >> $TMP
+	    echo "------------------------------" >> $TMP
+	    $GREP -i 'Posting to a moderated newsgroup' $TMPLOG | $AWK '{ print $6 }' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	    echo "" >> $TMP
+	    echo "Other Errors:" >> $TMP
+	    echo "------------------------------" >> $TMP
+	    $EGREP -vi '(Post by non-member|suspicious header|message approved|Discarded posting|bulk message discarded|junk message discarded|Message has implicit destination|Posting to a moderated newsgroup|Message discarded, msgid)' $TMPLOG | $SED 's/^.* ([0-9]*) //' | $SORT | $UNIQ -c | $SORT -nr >> $TMP 
+
+	else
+
+	    $SED 's/^.* ([0-9]*) //' $TMPLOG | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+	fi
+    else
+
+    	echo " ### Log file ${LOG} does not exist ### " >> $TMP
+
+    fi
+    echo "" >> $TMP
+
+done
+
+echo "******************************" >> $TMP
+echo "Queue Directory Contents" >> $TMP
+echo "******************************" >> $TMP
+echo "" >> $TMP
+
+cd $QUEUEDIR
+for DIR in $SUBDIRS
+do
+
+    $RM -f $TMPLOG
+    $TOUCH $TMPLOG
+    $LS -la $DIR >> $TMPLOG
+    LINES=`wc -l < $TMPLOG`
+    LINES=`expr $LINES - 3`
+
+    echo "Subdirectory: $DIR" >> $TMP
+    echo "==============================" >> $TMP
+    echo "Entries: $LINES" >> $TMP
+    echo "------------------------------" >> $TMP
+
+    if [ $LINES -le $MAX_LINES ] ; then
+    	$CAT $TMPLOG >> $TMP
+    else
+    	$HEAD $TMPLOG >> $TMP
+	echo "" >> $TMP
+	echo "	More than $MAX_LINES total entries, skipping ..." >> $TMP
+	echo "" >> $TMP
+    	$TAIL $TMPLOG >> $TMP
+    fi
+    echo "" >> $TMP
+
+done
+
+if [ "${RCPTS}x" != "x" ] ; then
+    $SENDMAIL -t -f$SENDER < $TMP
+else
+    $CAT $TMP
+fi
+
+$RM $TMP $TMPLOG
diff --git a/contrib/mmdsr.readme b/contrib/mmdsr.readme
new file mode 100644
index 00000000..fa074704
--- /dev/null
+++ b/contrib/mmdsr.readme
@@ -0,0 +1,33 @@
+Daily Status Report script...
+
+The mmdsr script was created by Brad Knowles to produce a daily status report for mailman. It was initially posted at <http://sourceforge.net/tracker/index.php?func=detail&aid=1123383&group_id=103&atid=300103> which see for possible patches and other enhancements.
+
+It is intended that there will be a 'cleaner', more complete readme file in the future. In the interim, here are Brad's original comments.
+
+I quickly whacked together a Daily Status Report script for
+Mailman (using Bourne shell, not Python ;), and thought that other
+folks might be interested in seeing it.
+
+The basic concept is a program that gets fired off at 23:59 every
+night, and goes through a variety of log files looking for entries
+specific to that date, and indicating problems or certain types of
+activity that might be of interest to someone trying to administer
+the server. It also does an "ls -la" of /usr/local/mailman/qfiles/*,
+so that you can see what is in the queue at the time of the running
+of the script.
+
+My concept was that this daily report would get e-mailed to the
+admin, or posted to a "reports" mailing list, where they could be
+archived and kept for future reference.
+
+The script does not (yet) do any statistics calculations, although it
+should be relatively easy to hack together some basic stats using
+awk, sort, etc....
+
+Anyway, I thought I'd share it and let folks take a look at it, and if
+anyone has any recommended improvements, we can incorporate
+those and share them back out with everyone.
+
+The code is written under a BSD-style license, so if you don't want
+to contribute any changes back to me, that's okay. Of course, I
+would prefer that you did, but I leave the choice up to you.