spaceblog

NET-SNMP Linux disk IO collector, mark III

Ok, this might be the last one ;-)

As mentioned previously, I’ve been writing a script that net-snmp can pass off the UCD-DISKIO-MIB requests to, for those versions that don’t have it built in, and to save one from rebuilding net-snmp packages.

This time around, I’ve added support for 2.4 series kernels, which use /proc/partitions instead of /proc/diskstats to log the disk IO stats.

I’ve now got this running in production across a handful of machines, and it’s pretty neat – I am able to see the shape of disk traffic during a vacuum on one of our machines, alongside the CPU and memory usage, and load average graphs, for example.

As before, plop it into /usr/local/sbin/snmp-diskio-collector and add this line to your snmpd.conf:

pass .1.3.6.1.4.1.2021.13.15 /usr/local/sbin/snmp-diskio-collector

Here’s the script:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146

#!/bin/sh

# by Jamie Wilkinson <jamie@anchor.net.au>
# this code is in the public domain
# based on passtest from the net-snmp distribution examples
# WARNING there is shitloads of IO required to get this information :)

debug_flag=0

debug () {
    if [ $debug_flag -eq 1 ]; then
        echo $* >> /tmp/snmp-diskio-collector-debug.log
    fi
}

PLACE=".1.3.6.1.4.1.2021.13.15"
REQ="$2"

debug
debug "new run" 
debug "args $*" 
if [ "$1" = "-s" ]; then
  exit 0
fi

# the 'tail' of the oid, everything after $PLACE
oidtail=`echo $REQ | sed "s/^$PLACE//"`
debug "oidtail=$oidtail" 

# number of devices we can export
if [ -f /proc/diskstats ]; then
    devcount=`wc -l /proc/diskstats | sed 's/^ *//' | cut -f1 -d' '`
else
    devcount=`wc -l /proc/partitions | sed 's/^ *//' | cut -f1 -d' '`
    devcount=`expr $devcount - 2`
fi
debug "devcount=$devcount" 

item=`echo $oidtail | cut -f4 -d.`
index=`echo $oidtail | cut -f5 -d.`
debug "oidtail=$oidtail, item=$item, index=$index" 

if [ "$1" = "-n" ]; then
    if [ -z "$item" ]; then
        item=1
        index=1
    elif [ -z "$index" ]; then
        index=1
    else
        index=`expr $index + 1`
        if [ "$index" -gt "$devcount" ]; then
            index=1
            item=`expr $item + 1`
            if [ "$item" -gt 6 ]; then
                # break out of the loop
                exit 0;
            fi
        fi
    fi
    RET=$PLACE.1.1.$item.$index
else
    case "$REQ" in
        $PLACE) exit 0;;
        *) RET=$REQ ;;
    esac
fi

debug "after -n, item=$item, index=$index" 
debug "RET is now $RET" 

echo "$RET"

debug "oidtail=$oidtail, item=$item, index=$index" 

# awk uses this variable in the environment below
export index
# see linux kernel Documentation/iostats.txt for format
if [ -n "$index" ]; then
    case "$item" in
        1)
            # diskIOIndex
            debug "result: diskIOIndex $index" 
            echo "integer"
            echo $index
            exit 0
            ;;
        2)
            # diskIODevice
            debug "result: diskIODevice $index" 
            echo "string"
	    if [ -f /proc/diskstats ]; then
		awk 'FNR == ENVIRON["index"] { print $3 }' /proc/diskstats
	    else
		awk 'FNR == ENVIRON["index"] + 2 { print $4 }' /proc/partitions
	    fi
            exit 0
            ;;
        3)
            # diskIONRead
            debug "result: diskIONRead $index" 
            echo "counter"
	    if [ -f /proc/diskstats ]; then
		awk 'FNR == ENVIRON["index"] { print $6 }' /proc/diskstats
	    else
		awk 'FNR == ENVIRON["index"] + 2 { print $7 }' /proc/partitions
	    fi
            exit 0
            ;;
        4)
            # diskIONWritten
            debug "result: diskIONWritten $index" 
            echo "counter"
	    if [ -f /proc/diskstats ]; then
		awk 'FNR == ENVIRON["index"] { print $10 }' /proc/diskstats
	    else
		awk 'FNR == ENVIRON["index"] + 2 { print $11 }' /proc/partitions
	    fi
            exit 0
            ;;
        5)
            # diskIOReads
            debug "result: diskIOReads $index" 
            echo "counter"
	    if [ -f /proc/diskstats ]; then
		awk 'FNR == ENVIRON["index"] { print $4 }' /proc/diskstats
	    else
		awk 'FNR == ENVIRON["index"] + 2 { print $5 }' /proc/partitions
	    fi
            exit 0
            ;;
        6)
            # diskIOWrites
            debug "result: diskIOWrites $index" 
            echo "counter"
	    if [ -f /proc/diskstats ]; then
		awk 'FNR == ENVIRON["index"] { print $8 }' /proc/diskstats
	    else
		awk 'FNR == ENVIRON["index"] + 2 { print $9 }' /proc/partitions
	    fi
            exit 0
            ;;
        *) exit 0; #echo "string"; echo "debug... $RET $REQ"; exit 0 ;;
    esac
else
    exit 0
fi

NET-SNMP Linux disk IO collector, mark II

This is an improvement on a previous post on adding UCD-DISKIO-MIB support to older versions of net-snmp; this one actually works on 2.6 kernels (and thanks to Pete loses about a million forks and pipes).

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121

#!/bin/sh

# by Jamie Wilkinson <jamie@anchor.net.au>
# this code is in the public domain
# based on passtest from the net-snmp distribution examples
# WARNING there is shitloads of IO required to get this information :)

debug_flag=0

debug () {
    if [ $debug_flag -eq 1 ]; then
        echo $* >> /tmp/snmp-diskio-collector-debug.log
    fi
}

PLACE=".1.3.6.1.4.1.2021.13.15"
REQ="$2"

debug
debug "new run" 
debug "args $*" 
if [ "$1" = "-s" ]; then
  exit 0
fi

# the 'tail' of the oid, everything after $PLACE
oidtail=`echo $REQ | sed "s/^$PLACE//"`
debug "oidtail=$oidtail" 

# number of devices we can export
devcount=`wc -l /proc/diskstats | cut -f1 -d' '`
debug "devcount=$devcount" 

item=`echo $oidtail | cut -f4 -d.`
index=`echo $oidtail | cut -f5 -d.`
debug "oidtail=$oidtail, item=$item, index=$index" 

if [ "$1" = "-n" ]; then
    if [ -z "$item" ]; then
        item=1
        index=1
    elif [ -z "$index" ]; then
        index=1
    else
        index=`expr $index + 1`
        if [ "$index" -gt "$devcount" ]; then
            index=1
            item=`expr $item + 1`
            if [ "$item" -gt 6 ]; then
                # break out of the loop
                exit 0;
            fi
        fi
    fi
    RET=$PLACE.1.1.$item.$index
else
    case "$REQ" in
        $PLACE) exit 0;;
        *) RET=$REQ ;;
    esac
fi

debug "after -n, item=$item, index=$index" 
debug "RET is now $RET" 

echo "$RET"

debug "oidtail=$oidtail, item=$item, index=$index" 

# awk uses this variable in the environment below
export index
# see linux kernel Documentation/iostats.txt for format
if [ -n "$index" ]; then
    case "$item" in
        1)
            # diskIOIndex
            debug "result: diskIOIndex $index" 
            echo "integer"
            echo $index
            exit 0
            ;;
        2)
            # diskIODevice
            debug "result: diskIODevice $index" 
            echo "string"
            awk 'FNR == ENVIRON["index"] { print $3 }' /proc/diskstats
            exit 0
            ;;
        3)
            # diskIONRead
            debug "result: diskIONRead $index" 
            echo "counter"
            awk 'FNR == ENVIRON["index"] { print $6 }' /proc/diskstats
            exit 0
            ;;
        4)
            # diskIONWritten
            debug "result: diskIONWritten $index" 
            echo "counter"
            awk 'FNR == ENVIRON["index"] { print $10 }' /proc/diskstats
            exit 0
            ;;
        5)
            # diskIOReads
            debug "result: diskIOReads $index" 
            echo "counter"
            awk 'FNR == ENVIRON["index"] { print $4 }' /proc/diskstats
            exit 0
            ;;
        6)
            # diskIOWrites
            debug "result: diskIOWrites $index" 
            echo "counter"
            awk 'FNR == ENVIRON["index"] { print $8 }' /proc/diskstats
            exit 0
            ;;
        *) exit 0; #echo "string"; echo "debug... $RET $REQ"; exit 0 ;;
    esac
else
    exit 0
fi

NET-SNMP Linux disk IO collector

The UCD-DISKIO-MIB isn’t supported in versions of net-snmp on most stable distros, but a few tools have add-on support for trending the data returned by this MIB.

You can recompile net-snmp, but I find that both boring and a maintenance headache; I don’t manually build anything that the vendor already supplies unless there’s a really good reason. Disk IO statistics are not a good enough reason to recompile.

The other option, the one I took, is to use the pass option of snmpd.conf to offload the collection of a MIBOID subtree to an external program. The API is pretty simple but there’s a dearth of examples on the interweb, so I present this one hopefully as an example for others to extend their own MIBs and also in the hope that others wanting to analyse disk performance find it useful.

It’s not terribly efficient, each call of the script looks at /proc/diskstats twice, there’s some tens of forks involved, and if you’re walking the subtree, that means you can be calling this script 6 times for every block device in the system. However, if you want efficiency, rather than trying to optimise this script, just rebuild net-snmp. My goal was to reduce deployment effort.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102

#!/bin/sh

# /usr/local/sbin/snmp-diskio-collector    
# by Jamie Wilkinson <jamie@anchor.net.au>
# this code is in the public domain
# based on passtest from the net-snmp distribution examples
# WARNING there is shitloads of IO required to get this information :)

PLACE=".1.3.6.1.4.1.2021.13.15"
REQ="$2"

if [ "$1" = "-s" ]; then
  exit 0
fi

# the 'tail' of the oid, everything after $PLACE
oidtail=`echo $REQ | sed "s/^$PLACE//"`

# number of devices we can export
devcount=`wc -l /proc/diskstats | cut -f1 -d' '`

if [ "$1" = "-n" ]; then
    case x"$oidtail" in
        x|x.1|x.1.1|x.1.1.1)
            # bootstrap a snmpwalk to the first index and item
            RET=$PLACE.1.1.1.1
            ;;
        x.1.1.*.*)
            # item is the element of the diskIOEntry
            item=`echo $oidtail | sed 's/\.1\.1\.\([^\.]*\)\..*/\1/'`
            # index is our disk table entry
            index=`echo $oidtail | sed 's/\.1\.1\.[^\.]*\.\(.*\)/\1/'`
           
            if [ "$index" -eq $devcount ]; then
                # wrap to the next item, starting at index 1
                RET=$PLACE.1.1.`expr $item + 1`.1
            else
                if [ x"$item" = x7 ]; then
                    # bail because we aren't doing more than 6 items in the
                    # entry
                    exit 0
                fi
                RET=$PLACE.1.1.$item.`expr $index + 1`
            fi
            ;;
        *) exit 0;;
    esac
else
    case "$REQ" in
        $PLACE)    exit 0 ;;
        *)         RET=$REQ ;;
    esac
fi

echo "$RET"

oid=`echo $RET | sed "s/^$PLACE//"`
item=`echo $oid | sed "s/.*\.//"`

# see linux kernel Documentation/iostats.txt for format
case "$RET" in
    $PLACE.1) exit 0 ;; #diskIOTable
    $PLACE.1.1) exit 0 ;; #diskIOEntry
    $PLACE.1.1.1) exit 0;; #diskIOIndex
    $PLACE.1.1.1.*)
        # diskIOIndex
        echo "integer"
        echo $item
        exit 0
        ;;
    $PLACE.1.1.2.*)
        # diskIODevice
        echo "string"
        head -n $item /proc/diskstats | tail -n 1 | awk '{print $3}'
        exit 0
        ;;
    $PLACE.1.1.3.*)
        # diskIONRead
        echo "counter"
        head -n $item /proc/diskstats | tail -n 1 | awk '{print $6}'
        exit 0
        ;;
    $PLACE.1.1.4.*)
        # diskIONWritten
        echo "counter"
        head -n $item /proc/diskstats | tail -n 1 | awk '{print $10}'
        exit 0
        ;;
    $PLACE.1.1.5.*)
        # diskIOReads
        echo "counter"
        head -n $item /proc/diskstats | tail -n 1 | awk '{print $4}'
        exit 0
        ;;
    $PLACE.1.1.6.*)
        # diskIOWrites
        echo "counter"
        head -n $item /proc/diskstats | tail -n 1 | awk '{print $8}'
        exit 0
        ;;
    *) exit 0;;
esac

This script gets hooked into snmpd.conf with a line like the following:

pass .1.3.6.1.4.1.2021.13.15 /usr/local/sbin/snmp-diskio-collector

and then you can query the MIB with either

snmpwalk -m ALL -v1 -c public localhost diskIOTable

or

snmptable -m ALL -v1 -c public localhost diskIOTable

(but of course you wouldn’t use SNMPv1 or v2c if you were going over the internets, right? :-)

Now you can tie this into Cacti or your favourite NMS, such as described here.

cross-language bindings, marshalling, a thought experiment in evile

Last night, Statler and Waldorf talked about free software that they hated on the LA live broadcast, including a lovely piece of work Erik mentioned, called intelib.

Putting LISP-like syntax into C++ didn’t actually surprise me, maybe that’s more of an insight into my own dark and twisted past than anything, there’s plenty of cross-language stuff around; in fact as Matt went on a small tangent talking about wrapping APIs for the purposes of adding functionality to other languages, I realised I could concoct something evil enough to make them both vomit.

Conveniently, James Dumay later talked about Mono and .NET, and showed a few languages that are usable with the language/platform. Again, not surprising, I did some stuff with marshalling types and function calls between C and Haskell back at uni, and as long as you keep all the type information around (as .NET CIL does) it’s pretty easy. However, you don’t need .NET to marshall data and function calls between languages:

  • Boost.Python is “A C++ library which enables seamless interoperability between C++ and the Python programming language.” It lets you call Python methods and objects from C++ code as if they were native to the language.

  • PyPerl is “a module for integrating Perl in Python.” Again, it lets you use a Perl module’s API directly within your Python code.

  • The Inline:: series of Perl modules allow you to “write Perl subroutines in X”, for values of X such as assembler, awk, Basic, Befunge, C, Interp, Java, Lua, Octave, Parrot, Perl (wtf?), Python, Ruby, SLang, Tcl, and a whole lot of other shit that you’ll never want to use. So, you can call functions in language X from within Perl.

I think you can see where I’m going with this. It’s possible to write a piece of code in C++ that links Boost.Python to import PyPerl to use Inline::Lua to call functions in a binding around a C library. That’s just the tip of the iceberg ;-)

backporting OpenLDAP to old distros

It would be really nice if those projects that consider themselves to be major pieces of infrastructure made it a priority to support a build on stock vendor releases of distros older than the current release.

I’m trying to build OpenLDAP 2.3.20 on Red Hat 7.3. Yes, that sounds like a bad idea, and it is quite painful. This pain is mostly due to the versions of Berkeley DB that OpenLDAP wants to build against: Red Hat 7.3 shipped with BDB 3.3.11; it wants at least version 4, bud the configure script makes it quite clear that version 4.1 isn’t supported.

The LDBM backend can use older BDBs, i.e. 3.x, but OpenLDAP 2.2 deprecates the LDBM backend in favour of back-bdb. That concerns me for a 2.3 series build of OpenLDAP: is it still stable, or has it been left to rot? Of course the test suite doesn’t cover the LDBM backend…

I would really like to not have to build for this old OS version, but sadly it exists and it needs to be a syncrepl slave. In order to finally put this box to rest it needs to have OpenLDAP running, so that we can migrate the services off it with minimal outage. There are so many bugs running OpenLDAP on this box that I want to try this 2.3 series version on it, in the same configuration as every other machine on the network, but the more I work on it, the less it feels like I’m making any progress on it at all.

Fighting version dependencies on old distros just isn’t fun. If the developers had considered this, and at least said “OK, well, RHEL 2.1 is still supported by Red Hat, so let’s try to configure and build on that platform too” then this wouldn’t nearly be as painful as it is.

The Annodex developers, as well as Conrads’ other projects (sweep, etc) work quite hard to make sure they build with the versions of libraries on peoples year-old desktop machines; sure this means sometimes having extra code to cope with API differences, but a small amount of effort on the developers part makes a massive difference when you think about how much time the users will spend trying to get the software to build.

It’s the little things that make the difference between an OK project and an Awesome project.

Meanwhile, on the cutting edge of RHEL 3 and 4, and FC 4 and 5, latest OpenLDAP 2.3 looks to be a really promising piece of work, the amount of work that’s gone into it since 2.2.27 is impressive, and if it works nearly as well as it looks like it will from the ChangeLog, then that’ll finally put to rest a whole lot of problems we’ve had since deploying LDAP as our authentication database.

Fingers crossed I can get there without losing all my hair.