NET-SNMP Linux disk IO collector

NET-SNMP Linux disk IO collector

The UCD-DISKIO-MIB isn’t supported in versions of net-snmp on most stable distros, but a few tools have add-on support for trending the data returned by this MIB.

You can recompile net-snmp, but I find that both boring and a maintenance headache; I don’t manually build anything that the vendor already supplies unless there’s a really good reason. Disk IO statistics are not a good enough reason to recompile.

The other option, the one I took, is to use the pass option of snmpd.conf to offload the collection of a MIBOID subtree to an external program. The API is pretty simple but there’s a dearth of examples on the interweb, so I present this one hopefully as an example for others to extend their own MIBs and also in the hope that others wanting to analyse disk performance find it useful.

It’s not terribly efficient, each call of the script looks at /proc/diskstats twice, there’s some tens of forks involved, and if you’re walking the subtree, that means you can be calling this script 6 times for every block device in the system. However, if you want efficiency, rather than trying to optimise this script, just rebuild net-snmp. My goal was to reduce deployment effort.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
#!/bin/sh

# /usr/local/sbin/snmp-diskio-collector    
# by Jamie Wilkinson <jamie@anchor.net.au>
# this code is in the public domain
# based on passtest from the net-snmp distribution examples
# WARNING there is shitloads of IO required to get this information :)

PLACE=".1.3.6.1.4.1.2021.13.15"
REQ="$2"

if [ "$1" = "-s" ]; then
  exit 0
fi

# the 'tail' of the oid, everything after $PLACE
oidtail=`echo $REQ | sed "s/^$PLACE//"`

# number of devices we can export
devcount=`wc -l /proc/diskstats | cut -f1 -d' '`

if [ "$1" = "-n" ]; then
    case x"$oidtail" in
        x|x.1|x.1.1|x.1.1.1)
            # bootstrap a snmpwalk to the first index and item
            RET=$PLACE.1.1.1.1
            ;;
        x.1.1.*.*)
            # item is the element of the diskIOEntry
            item=`echo $oidtail | sed 's/\.1\.1\.\([^\.]*\)\..*/\1/'`
            # index is our disk table entry
            index=`echo $oidtail | sed 's/\.1\.1\.[^\.]*\.\(.*\)/\1/'`
           
            if [ "$index" -eq $devcount ]; then
                # wrap to the next item, starting at index 1
                RET=$PLACE.1.1.`expr $item + 1`.1
            else
                if [ x"$item" = x7 ]; then
                    # bail because we aren't doing more than 6 items in the
                    # entry
                    exit 0
                fi
                RET=$PLACE.1.1.$item.`expr $index + 1`
            fi
            ;;
        *) exit 0;;
    esac
else
    case "$REQ" in
        $PLACE)    exit 0 ;;
        *)         RET=$REQ ;;
    esac
fi

echo "$RET"

oid=`echo $RET | sed "s/^$PLACE//"`
item=`echo $oid | sed "s/.*\.//"`

# see linux kernel Documentation/iostats.txt for format
case "$RET" in
    $PLACE.1) exit 0 ;; #diskIOTable
    $PLACE.1.1) exit 0 ;; #diskIOEntry
    $PLACE.1.1.1) exit 0;; #diskIOIndex
    $PLACE.1.1.1.*)
        # diskIOIndex
        echo "integer"
        echo $item
        exit 0
        ;;
    $PLACE.1.1.2.*)
        # diskIODevice
        echo "string"
        head -n $item /proc/diskstats | tail -n 1 | awk '{print $3}'
        exit 0
        ;;
    $PLACE.1.1.3.*)
        # diskIONRead
        echo "counter"
        head -n $item /proc/diskstats | tail -n 1 | awk '{print $6}'
        exit 0
        ;;
    $PLACE.1.1.4.*)
        # diskIONWritten
        echo "counter"
        head -n $item /proc/diskstats | tail -n 1 | awk '{print $10}'
        exit 0
        ;;
    $PLACE.1.1.5.*)
        # diskIOReads
        echo "counter"
        head -n $item /proc/diskstats | tail -n 1 | awk '{print $4}'
        exit 0
        ;;
    $PLACE.1.1.6.*)
        # diskIOWrites
        echo "counter"
        head -n $item /proc/diskstats | tail -n 1 | awk '{print $8}'
        exit 0
        ;;
    *) exit 0;;
esac

This script gets hooked into snmpd.conf with a line like the following:

pass .1.3.6.1.4.1.2021.13.15 /usr/local/sbin/snmp-diskio-collector

and then you can query the MIB with either

snmpwalk -m ALL -v1 -c public localhost diskIOTable

or

snmptable -m ALL -v1 -c public localhost diskIOTable

(but of course you wouldn’t use SNMPv1 or v2c if you were going over the internets, right? :-)

Now you can tie this into Cacti or your favourite NMS, such as described here.