jstott.me.uk: E-Mail Stats How-To

Introduction

I have had several requests to explain how I create my e-mail stats charts. This page describes how to set up *nix to record and generate e-mail statistics charts as used here. To get started, you need to make sure that you have exim, procmail and rrdtool all set up and working nicely.

The process involves maintaining a count of how many genuine and spam e-mails that are received. Procmail is responsible for filtering the e-mail and executing the appropriate script to increment the counters. A cron job runs periodically to generate the charts using rrdtool.

Exim Filter

The first thing to set up is an exim filter. The filter simply invokes procmail:

#   Exim filter   <<== do not edit or remove this line!

pipe "/usr/local/bin/procmail -t"
seen finish

The filter is saved in your home directory (such as /home/jms/) as .forward

.procmailrc

The second file needed is the .procmailrc file. This file describes how e-mail is to be filtered by procmail. My .procmailrc file looks like this:

:0
* ^X-UKC-SpamScore: sssssssssss
{
    :0 c
    |/home/cug/jms30/maillog/eat-spam

    :0
    Mail/spam
}

:0
* ^X-UKC-CSSpamCheck-Level: \*\*\*\*\*\*\*
{
    :0 c
    |/home/cug/jms30/maillog/eat-spam

    :0
    Mail/spam
}

:0
* ^X-UKC-CSSpamCheck-Flag: YES
{
    :0 c
    |/home/cug/jms30/maillog/eat-spam

    :0
    Mail/spam
}

:0
* ^X-UKC-SpamScore: ssssssssss
{
    :0 c
    |/home/cug/jms30/maillog/eat-spam

    :0
    Mail/spam
}

:0
* ^X-UKC-CSSpamCheck-Level: \*\*\*\*
{
    :0 c
    |/home/cug/jms30/maillog/eat-spam

    :0
    Mail/spam
}

:0
{
    :0 c
    |/home/cug/jms30/maillog/eat-inbox

    :0
    .mail
}

The first five sections match spam which is checked by my University. If it matches any of the given headers, it executes the /home/cug/jms30/maillog/eat-spam shell script (which increments the spam counter) and moves the e-mail to the Mail/spam directory. Anything that doesn't match as spam is moved to my inbox (the last section).

You can modify your .procmailrc file to match your requirements.

E-mail Counter Scripts

You will notice reference to two shell scripts in the .procmailrc file - eat-spam and eat-inbox. Each of these scripts simply executes another script called up-spam or up-inbox. eat-spam looks like this:

#! /bin/sh

cat > /dev/null
exec /home/cug/jms30/maillog/up-spam

and eat-inbox looks like this:

#! /bin/sh

cat > /dev/null
exec /home/cug/jms30/maillog/up-inbox

up-spam:

#! /bin/bash

MAILLOGPATH=/home/cug/jms30/maillog

printf '%d' $(($(cat $MAILLOGPATH/spam-counter) + 1)) > $MAILLOGPATH/spam-newcount
mv $MAILLOGPATH/spam-newcount $MAILLOGPATH/spam-counter

and up-inbox:

#! /bin/bash

MAILLOGPATH=/home/cug/jms30/maillog

printf '%d' $(($(cat $MAILLOGPATH/inbox-counter) + 1)) > $MAILLOGPATH/inbox-newcount
mv $MAILLOGPATH/inbox-newcount $MAILLOGPATH/inbox-counter

You will need to change MAILLOGPATH in both up-spam and up-inbox to point to the directory where your scripts are stored. I keep mine in a directory called maillog in my home directory.

You will also need two files called spam-counter and inbox-counter which actually keep the count of the number of spam and genuine e-mails. Both these files will initially contain:

Finally you need to make sure that you can read and write to the counters and that you can execute the scripts.

rrdtool Setup

rrdtool is a tool for handling round-robin databases. A round-robin database is ideal for our requirements here because we are only charting the previous day's e-mail.

First off, we need to create the database. The database needs to contain two data sources for the inbox and spam. Data is added every 15 minutes and an archive of 96 readings are stored. The command needed to create the database is:

rrdtool create data.rrd        \
    --step 900                 \
    DS:inbox:COUNTER:1800:U:U  \
    DS:spam:COUNTER:1800:U:U   \
    RRA:AVERAGE:0.5:1:96

Make sure that you don't overwrite any existing files - the database filename will be data.rrd.

Two scripts are periodically executed using a cron job. The first adds the counter data to the round-robin database. This script is called adddata:

#! /bin/bash

RRDTOOL=/usr/local/bin/rrdtool
MAILLOGPATH=/home/cug/jms30/maillog

for file in inbox spam; do
	cp $MAILLOGPATH/$file-counter $MAILLOGPATH/$file-curdata
#	printf "0" > $MAILLOGPATH/$file-counter
done

$RRDTOOL update $MAILLOGPATH/data.rrd \
	-t inbox:spam \
	N:$(cat $MAILLOGPATH/inbox-curdata):$(cat $MAILLOGPATH/spam-curdata)

for file in inbox spam; do
	rm $MAILLOGPATH/$file-curdata
done

You will need to change RRDTOOL to be the path to rrdtool and MAILLOGPATH to be the path to your e-mail stats scripts.

The script to generate the charts is called graphdata and looks like this:

#! /bin/bash

RRDTOOL=/usr/local/bin/rrdtool
MAILLOGPATH=/home/cug/jms30/maillog
GRAPHFILE=/home/cug/jms30/webpages/mailbox.png

cd $MAILLOGPATH

$RRDTOOL graph $GRAPHFILE \
--title="Daily mailbox traffic" \
	DEF:inbox=data.rrd:inbox:AVERAGE \
	CDEF:rinbox=inbox,900,\* \
	DEF:spam=data.rrd:spam:AVERAGE \
	CDEF:rspam=spam,900,\* \
	'AREA:rinbox#10c0d0:Inbox' \
	'GPRINT:rinbox:LAST:Current\: %8.0lf %s' \
	'GPRINT:rinbox:AVERAGE:Average\: %8.2lf %s' \
	'GPRINT:rinbox:MAX:Max\: %8.0lf %s\n' \
	'STACK:rspam#c04040:Spam' \
	'GPRINT:rspam:LAST: Current\: %8.0lf %s' \
	'GPRINT:rspam:AVERAGE:Average\: %8.2lf %s' \
	'GPRINT:rspam:MAX:Max\: %8.0lf %s' > /dev/null

Again, you will need to change RRDTOOL to be the path to rrdtool, MAILLOGPATH to be the path to your e-mail stats scripts and GRAPHFILE tp be the path to where the charts should be saved after they have been created.

Finally, you need a crontab to say when to execute the adddata and graphdata scripts:

0,15,30,45 * * * * /home/cug/jms30/maillog/adddata
1,16,31,46 * * * * /home/cug/jms30/maillog/graphdata

You can save your crontab in a file (say cron.tab) and use

crontab cron.tab

to update your crontab.

This will make sure that the adddata script is executed every 15 minutes and that the graphdata script is executed every 15 minutes but one minute after the adddata script.