spaceblog

welcome to 1998

My PCMCIA wireless card was flaking out today, after I got my IrDA port working; I’d reconfigued the port in the BIOS to use IRQ 3. After some time I found that the wireless card was also choosing IRQ 3, thanks to the output from dmesg. Pushing the IrDA port to IRQ 4 fixed that little conflict. I’m still left wondering why in this day and age I’m editing IRQ settings on a laptop that’s barely a year old.

I started out trying to get photos off of my phone, which I’ve successfully done in the past but always forget how; today the phone and laptop just refused to connect. The cause of the problem was that the serial8250 driver was grabbing the IO ports and IRQ first, and preventing the nsc-ircc driver from getting it later in the boot process. I think the fix for that was to put nsc-ircc at the top of /etc/modules; at least, it’s working now.

As I was rolling on my “fix things that are pissing me off” rampage, I decided to crack at the “why the hell isn’t DMA working” one. The X40 has an ICH4-M chipset, handled by the piix driver. The driver was loading, but hdparm was always getting this:

# hdparm -d 1 /dev/hda
/dev/hda:
 setting using_dma to 1 (on)
 HDIO_SET_DMA failed: Operation not permitted
 using_dma    =  0 (off)

After spending way too long following threads full of erroneous opinion on the ubuntu forums, debian-user mailing list archives, and other useful tomes, I discovered this clue in dmesg:

ICH4: port 0x01f0 already claimed by ide0

which led me to this post, cluing me to the fact that the ide-generic driver was being greedy. Again, a driver conflict, but to fix this one, I had to put piix at the top of /etc/mkinitramfs/modules and then rebuild the initrd (running dpkg-reconfigure on your kernel package will take care of this). So, thanks Christophe, for your bug report that finally helped solve this one, and now holy shit my computer is fast.

Datarock programmed Rage tonight, too, which was an awesome soundtrack to fix things to ;-) They and I have such similar tastes in music.

NET-SNMP Linux disk IO collector, mark III

Ok, this might be the last one ;-)

As mentioned previously, I’ve been writing a script that net-snmp can pass off the UCD-DISKIO-MIB requests to, for those versions that don’t have it built in, and to save one from rebuilding net-snmp packages.

This time around, I’ve added support for 2.4 series kernels, which use /proc/partitions instead of /proc/diskstats to log the disk IO stats.

I’ve now got this running in production across a handful of machines, and it’s pretty neat – I am able to see the shape of disk traffic during a vacuum on one of our machines, alongside the CPU and memory usage, and load average graphs, for example.

As before, plop it into /usr/local/sbin/snmp-diskio-collector and add this line to your snmpd.conf:

pass .1.3.6.1.4.1.2021.13.15 /usr/local/sbin/snmp-diskio-collector

Here’s the script:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
#!/bin/sh

# by Jamie Wilkinson <jamie@anchor.net.au>
# this code is in the public domain
# based on passtest from the net-snmp distribution examples
# WARNING there is shitloads of IO required to get this information :)

debug_flag=0

debug () {
    if [ $debug_flag -eq 1 ]; then
        echo $* >> /tmp/snmp-diskio-collector-debug.log
    fi
}

PLACE=".1.3.6.1.4.1.2021.13.15"
REQ="$2"

debug
debug "new run" 
debug "args $*" 
if [ "$1" = "-s" ]; then
  exit 0
fi

# the 'tail' of the oid, everything after $PLACE
oidtail=`echo $REQ | sed "s/^$PLACE//"`
debug "oidtail=$oidtail" 

# number of devices we can export
if [ -f /proc/diskstats ]; then
    devcount=`wc -l /proc/diskstats | sed 's/^ *//' | cut -f1 -d' '`
else
    devcount=`wc -l /proc/partitions | sed 's/^ *//' | cut -f1 -d' '`
    devcount=`expr $devcount - 2`
fi
debug "devcount=$devcount" 

item=`echo $oidtail | cut -f4 -d.`
index=`echo $oidtail | cut -f5 -d.`
debug "oidtail=$oidtail, item=$item, index=$index" 

if [ "$1" = "-n" ]; then
    if [ -z "$item" ]; then
        item=1
        index=1
    elif [ -z "$index" ]; then
        index=1
    else
        index=`expr $index + 1`
        if [ "$index" -gt "$devcount" ]; then
            index=1
            item=`expr $item + 1`
            if [ "$item" -gt 6 ]; then
                # break out of the loop
                exit 0;
            fi
        fi
    fi
    RET=$PLACE.1.1.$item.$index
else
    case "$REQ" in
        $PLACE) exit 0;;
        *) RET=$REQ ;;
    esac
fi

debug "after -n, item=$item, index=$index" 
debug "RET is now $RET" 

echo "$RET"

debug "oidtail=$oidtail, item=$item, index=$index" 

# awk uses this variable in the environment below
export index
# see linux kernel Documentation/iostats.txt for format
if [ -n "$index" ]; then
    case "$item" in
        1)
            # diskIOIndex
            debug "result: diskIOIndex $index" 
            echo "integer"
            echo $index
            exit 0
            ;;
        2)
            # diskIODevice
            debug "result: diskIODevice $index" 
            echo "string"
	    if [ -f /proc/diskstats ]; then
		awk 'FNR == ENVIRON["index"] { print $3 }' /proc/diskstats
	    else
		awk 'FNR == ENVIRON["index"] + 2 { print $4 }' /proc/partitions
	    fi
            exit 0
            ;;
        3)
            # diskIONRead
            debug "result: diskIONRead $index" 
            echo "counter"
	    if [ -f /proc/diskstats ]; then
		awk 'FNR == ENVIRON["index"] { print $6 }' /proc/diskstats
	    else
		awk 'FNR == ENVIRON["index"] + 2 { print $7 }' /proc/partitions
	    fi
            exit 0
            ;;
        4)
            # diskIONWritten
            debug "result: diskIONWritten $index" 
            echo "counter"
	    if [ -f /proc/diskstats ]; then
		awk 'FNR == ENVIRON["index"] { print $10 }' /proc/diskstats
	    else
		awk 'FNR == ENVIRON["index"] + 2 { print $11 }' /proc/partitions
	    fi
            exit 0
            ;;
        5)
            # diskIOReads
            debug "result: diskIOReads $index" 
            echo "counter"
	    if [ -f /proc/diskstats ]; then
		awk 'FNR == ENVIRON["index"] { print $4 }' /proc/diskstats
	    else
		awk 'FNR == ENVIRON["index"] + 2 { print $5 }' /proc/partitions
	    fi
            exit 0
            ;;
        6)
            # diskIOWrites
            debug "result: diskIOWrites $index" 
            echo "counter"
	    if [ -f /proc/diskstats ]; then
		awk 'FNR == ENVIRON["index"] { print $8 }' /proc/diskstats
	    else
		awk 'FNR == ENVIRON["index"] + 2 { print $9 }' /proc/partitions
	    fi
            exit 0
            ;;
        *) exit 0; #echo "string"; echo "debug... $RET $REQ"; exit 0 ;;
    esac
else
    exit 0
fi

NET-SNMP Linux disk IO collector, mark II

This is an improvement on a previous post on adding UCD-DISKIO-MIB support to older versions of net-snmp; this one actually works on 2.6 kernels (and thanks to Pete loses about a million forks and pipes).

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
#!/bin/sh

# by Jamie Wilkinson <jamie@anchor.net.au>
# this code is in the public domain
# based on passtest from the net-snmp distribution examples
# WARNING there is shitloads of IO required to get this information :)

debug_flag=0

debug () {
    if [ $debug_flag -eq 1 ]; then
        echo $* >> /tmp/snmp-diskio-collector-debug.log
    fi
}

PLACE=".1.3.6.1.4.1.2021.13.15"
REQ="$2"

debug
debug "new run" 
debug "args $*" 
if [ "$1" = "-s" ]; then
  exit 0
fi

# the 'tail' of the oid, everything after $PLACE
oidtail=`echo $REQ | sed "s/^$PLACE//"`
debug "oidtail=$oidtail" 

# number of devices we can export
devcount=`wc -l /proc/diskstats | cut -f1 -d' '`
debug "devcount=$devcount" 

item=`echo $oidtail | cut -f4 -d.`
index=`echo $oidtail | cut -f5 -d.`
debug "oidtail=$oidtail, item=$item, index=$index" 

if [ "$1" = "-n" ]; then
    if [ -z "$item" ]; then
        item=1
        index=1
    elif [ -z "$index" ]; then
        index=1
    else
        index=`expr $index + 1`
        if [ "$index" -gt "$devcount" ]; then
            index=1
            item=`expr $item + 1`
            if [ "$item" -gt 6 ]; then
                # break out of the loop
                exit 0;
            fi
        fi
    fi
    RET=$PLACE.1.1.$item.$index
else
    case "$REQ" in
        $PLACE) exit 0;;
        *) RET=$REQ ;;
    esac
fi

debug "after -n, item=$item, index=$index" 
debug "RET is now $RET" 

echo "$RET"

debug "oidtail=$oidtail, item=$item, index=$index" 

# awk uses this variable in the environment below
export index
# see linux kernel Documentation/iostats.txt for format
if [ -n "$index" ]; then
    case "$item" in
        1)
            # diskIOIndex
            debug "result: diskIOIndex $index" 
            echo "integer"
            echo $index
            exit 0
            ;;
        2)
            # diskIODevice
            debug "result: diskIODevice $index" 
            echo "string"
            awk 'FNR == ENVIRON["index"] { print $3 }' /proc/diskstats
            exit 0
            ;;
        3)
            # diskIONRead
            debug "result: diskIONRead $index" 
            echo "counter"
            awk 'FNR == ENVIRON["index"] { print $6 }' /proc/diskstats
            exit 0
            ;;
        4)
            # diskIONWritten
            debug "result: diskIONWritten $index" 
            echo "counter"
            awk 'FNR == ENVIRON["index"] { print $10 }' /proc/diskstats
            exit 0
            ;;
        5)
            # diskIOReads
            debug "result: diskIOReads $index" 
            echo "counter"
            awk 'FNR == ENVIRON["index"] { print $4 }' /proc/diskstats
            exit 0
            ;;
        6)
            # diskIOWrites
            debug "result: diskIOWrites $index" 
            echo "counter"
            awk 'FNR == ENVIRON["index"] { print $8 }' /proc/diskstats
            exit 0
            ;;
        *) exit 0; #echo "string"; echo "debug... $RET $REQ"; exit 0 ;;
    esac
else
    exit 0
fi

NET-SNMP Linux disk IO collector

The UCD-DISKIO-MIB isn’t supported in versions of net-snmp on most stable distros, but a few tools have add-on support for trending the data returned by this MIB.

You can recompile net-snmp, but I find that both boring and a maintenance headache; I don’t manually build anything that the vendor already supplies unless there’s a really good reason. Disk IO statistics are not a good enough reason to recompile.

The other option, the one I took, is to use the pass option of snmpd.conf to offload the collection of a MIBOID subtree to an external program. The API is pretty simple but there’s a dearth of examples on the interweb, so I present this one hopefully as an example for others to extend their own MIBs and also in the hope that others wanting to analyse disk performance find it useful.

It’s not terribly efficient, each call of the script looks at /proc/diskstats twice, there’s some tens of forks involved, and if you’re walking the subtree, that means you can be calling this script 6 times for every block device in the system. However, if you want efficiency, rather than trying to optimise this script, just rebuild net-snmp. My goal was to reduce deployment effort.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
#!/bin/sh

# /usr/local/sbin/snmp-diskio-collector    
# by Jamie Wilkinson <jamie@anchor.net.au>
# this code is in the public domain
# based on passtest from the net-snmp distribution examples
# WARNING there is shitloads of IO required to get this information :)

PLACE=".1.3.6.1.4.1.2021.13.15"
REQ="$2"

if [ "$1" = "-s" ]; then
  exit 0
fi

# the 'tail' of the oid, everything after $PLACE
oidtail=`echo $REQ | sed "s/^$PLACE//"`

# number of devices we can export
devcount=`wc -l /proc/diskstats | cut -f1 -d' '`

if [ "$1" = "-n" ]; then
    case x"$oidtail" in
        x|x.1|x.1.1|x.1.1.1)
            # bootstrap a snmpwalk to the first index and item
            RET=$PLACE.1.1.1.1
            ;;
        x.1.1.*.*)
            # item is the element of the diskIOEntry
            item=`echo $oidtail | sed 's/\.1\.1\.\([^\.]*\)\..*/\1/'`
            # index is our disk table entry
            index=`echo $oidtail | sed 's/\.1\.1\.[^\.]*\.\(.*\)/\1/'`
           
            if [ "$index" -eq $devcount ]; then
                # wrap to the next item, starting at index 1
                RET=$PLACE.1.1.`expr $item + 1`.1
            else
                if [ x"$item" = x7 ]; then
                    # bail because we aren't doing more than 6 items in the
                    # entry
                    exit 0
                fi
                RET=$PLACE.1.1.$item.`expr $index + 1`
            fi
            ;;
        *) exit 0;;
    esac
else
    case "$REQ" in
        $PLACE)    exit 0 ;;
        *)         RET=$REQ ;;
    esac
fi

echo "$RET"

oid=`echo $RET | sed "s/^$PLACE//"`
item=`echo $oid | sed "s/.*\.//"`

# see linux kernel Documentation/iostats.txt for format
case "$RET" in
    $PLACE.1) exit 0 ;; #diskIOTable
    $PLACE.1.1) exit 0 ;; #diskIOEntry
    $PLACE.1.1.1) exit 0;; #diskIOIndex
    $PLACE.1.1.1.*)
        # diskIOIndex
        echo "integer"
        echo $item
        exit 0
        ;;
    $PLACE.1.1.2.*)
        # diskIODevice
        echo "string"
        head -n $item /proc/diskstats | tail -n 1 | awk '{print $3}'
        exit 0
        ;;
    $PLACE.1.1.3.*)
        # diskIONRead
        echo "counter"
        head -n $item /proc/diskstats | tail -n 1 | awk '{print $6}'
        exit 0
        ;;
    $PLACE.1.1.4.*)
        # diskIONWritten
        echo "counter"
        head -n $item /proc/diskstats | tail -n 1 | awk '{print $10}'
        exit 0
        ;;
    $PLACE.1.1.5.*)
        # diskIOReads
        echo "counter"
        head -n $item /proc/diskstats | tail -n 1 | awk '{print $4}'
        exit 0
        ;;
    $PLACE.1.1.6.*)
        # diskIOWrites
        echo "counter"
        head -n $item /proc/diskstats | tail -n 1 | awk '{print $8}'
        exit 0
        ;;
    *) exit 0;;
esac

This script gets hooked into snmpd.conf with a line like the following:

pass .1.3.6.1.4.1.2021.13.15 /usr/local/sbin/snmp-diskio-collector

and then you can query the MIB with either

snmpwalk -m ALL -v1 -c public localhost diskIOTable

or

snmptable -m ALL -v1 -c public localhost diskIOTable

(but of course you wouldn’t use SNMPv1 or v2c if you were going over the internets, right? :-)

Now you can tie this into Cacti or your favourite NMS, such as described here.

cross-language bindings, marshalling, a thought experiment in evile

Last night, Statler and Waldorf talked about free software that they hated on the LA live broadcast, including a lovely piece of work Erik mentioned, called intelib.

Putting LISP-like syntax into C++ didn’t actually surprise me, maybe that’s more of an insight into my own dark and twisted past than anything, there’s plenty of cross-language stuff around; in fact as Matt went on a small tangent talking about wrapping APIs for the purposes of adding functionality to other languages, I realised I could concoct something evil enough to make them both vomit.

Conveniently, James Dumay later talked about Mono and .NET, and showed a few languages that are usable with the language/platform. Again, not surprising, I did some stuff with marshalling types and function calls between C and Haskell back at uni, and as long as you keep all the type information around (as .NET CIL does) it’s pretty easy. However, you don’t need .NET to marshall data and function calls between languages:

  • Boost.Python is “A C++ library which enables seamless interoperability between C++ and the Python programming language.” It lets you call Python methods and objects from C++ code as if they were native to the language.

  • PyPerl is “a module for integrating Perl in Python.” Again, it lets you use a Perl module’s API directly within your Python code.

  • The Inline:: series of Perl modules allow you to “write Perl subroutines in X”, for values of X such as assembler, awk, Basic, Befunge, C, Interp, Java, Lua, Octave, Parrot, Perl (wtf?), Python, Ruby, SLang, Tcl, and a whole lot of other shit that you’ll never want to use. So, you can call functions in language X from within Perl.

I think you can see where I’m going with this. It’s possible to write a piece of code in C++ that links Boost.Python to import PyPerl to use Inline::Lua to call functions in a binding around a C library. That’s just the tip of the iceberg ;-)