spaceblog

Monitoring Cron Jobs

Wed, 24 Feb 2010 22:27:00 +1000

(This article comes from one I helped edit and publish inside work, so I can’t take any credit for the ideas expressed within, though I do vehemently and violently subscribe to the sentiment! Thanks to Alan Sundell for originally educating me.)

When you set (or don’t set at all) MAILTO in a crontab fragment, typically it’s because you want to be notified if your job fails – failure in this case if and only if the job only prints to stdout/stderr if there is an exceptional condition… However not all jobs print only on exceptional conditions, many use stderr for logging, and email is just not a great solution to this problem, especially at scale.

Problems with alerting by email from `cron`.

Why is it a bad idea to rely on cron mail?

We all get so much mail from cron that few are in the habit of reading it anymore.
If your server is broken, the mail submission agent may be broken too.
You may handle your cron mail, but you may be on holiday when it arrives.
crond can crash.
You will get mutiple emails for the same failure.
Your cron mail will get delivered to every one of your mailboxes, eating up storage.
You cannot suppress cron mail notifications.
Your cron mail has no concept of dependencies.
You will get notified of temporary failures when you only care about persistent ones.

If a cronjob running successfully is critical to operation, then it seems that what you really need is some kind of monitoring system that addresses all of these things, and can send alerts to some oncall rotation that determines who is responsible for handling alerts.

A potential solution

Here’s an idea that might help with that.

Direct the output of your job to some log file for debugging, in the event of persistent failure. Note the truncate:
```
MAILTO=""
*/1 * * * * root cronjob > /var/log/cronjob.log 2>&1
```
(If you decide to append, not overwrite the log each execution, then make sure you logrotate that file.)
At the end of cronjob, update a status file, like so:
```
scriptname=$(basename $0)
date +%s > /var/tmp/cronjob-last-success-timestamp
```
Ensure that your job exits on error before reaching the last line!
Collect the content of that file regularly with your monitoring system; scrape it with the nagios host agent, pump it into collectd, whatever you hip open source cats are using these days.
Configure your monitoring system to send a notification on the timestamp having not been updated in some time period.
```
if cronjob-last-success-timestamp
<
(time() - 30m)
then alert
```
Profit!

Now you only generate an alert if the cron job hasn’t succeeded in the last 30 minutes (a threshold you can adjust to match your monitoring scrape intervals and service SLAs), and with a sufficiently mature monitoring system you can now express dependencies, suppress the notification, and send it to an oncall rotation, and so on!

Most significantly, we have converted a system that always reported failure, into a system that is based around checking for success – a failsafe.

I, for one, welcome our new enhancement proposals

Mon, 04 Feb 2008 13:08:00 +1000

I just read with excitement the announcement of Debian Enhancement Proposals, something that I too have been contemplating in recent months (but due to my ghastly lack of commitment to the Debian community doubted my ability to drive it).

I work in company driven by engineering documents and designs, I like RFCs, and I like what Python has done with its PEPs. Debian’s adoption of this useful tool can only improve the community and the distribution.

Yay!

nsscache open source launch

Mon, 05 Nov 2007 21:54:00 +1000

Today we open sourced the project I’ve been working on for the last 9 months, nsscache.

It’s a glorified version of:

ldapsearch | awk > /etc/passwd

in that we in theory support more than just LDAP as a data source, and offer two types of database storage (nss_db using Berkeley DB, and plain text files).

If you’re having issues with your nss_ldap setup, then try it out :)

linux.conf.au 2007 programme choices

Fri, 12 Jan 2007 01:39:00 +1000

As the clock ticks approach single digit figures, the organising team are ramping up. Everything’s coming along smoothly in time for a kick-arse start on Monday.

One thing that makes me sad is that I’ll likely not be able to watch a lot of the talks – but if I do get a chance, I’d see these:

clustering tdb by Andrew Tridgell. When I first saw this proposal came in, I knew it was a good one. Someone should make an LDAP server that doesn’t suck, and use tdb to solve the multimaster replication problem in the storage layer. Oh wait – that’s what he’s doing already.
Puppet by Luke Kanies. Luke’s been working on this awesome next-generation systems configuration management tool for a few years now. He approached me, back in the day, to be a beta-tester – he and I were both hitting scaling problems with cfengine. I hope every sysadmin makes it to this talk!

There’s a few others to note: Theodore T’so always has an interesting talk; this year he’s giving two cool subjects a run. The tutorial on heartbeat 2 by Alan Robertson is sure to be full of good loadbalancing fu.

There’s lots of exciting things going on in the programme, so I hope to see you all next week!

we're #1!

Tue, 14 Nov 2006 19:31:00 +1000

Who’s the best linux conference in the world? WE ARE, BABY!

Turns out we’re only second hit for “linux conference”, but we’re still best!

linux.conf.au boned, fixed again

Tue, 14 Nov 2006 18:54:00 +1000

Yeah, I broke the registration process when I rolled out some bugfixes last night, but I’ve fixed them now! Continue in your merry scramble for tickets to the ROCKIN’EST FREE AND OPEN SORES GIG IN THE SOUTH.

(In unrelated news, can you think of a good reason why they shouldn’t go to lca2007?

lca2007 registrations now open!

Wed, 01 Nov 2006 20:34:00 +1000

Registrations to LCA 2007 are now open! Get in quick!

linux.conf.au 2007 programme first draft

Wed, 18 Oct 2006 07:41:00 +1000

Last night the Seven met, as they do, around a dark table, deep below the city in a room built by the Templar Knights; surrounded, as normal, by ancient iconography of power and knowledge. Their goal: a draft programme for LCA 2007.

We ranked the streams in order of popularity; nothing too scientific, based purely on presenter name and subject matter, what would likely draw the biggest crowds?

Then with that ordering, we’d go through the streams, picking off the top of all lists, and putting them in the biggest room. Repeat for the next one in the next talk slot, offsetting them so that no two venues would carry the same stream at the same time.

By the end of it, we had a nice patchwork quilt.

I don’t think we’re done with it, yet, but it’s a good start!

linux.conf.au 2007 proposal count through the roof!

Sun, 17 Sep 2006 09:59:00 +1000

The response to the CFP has been massive! Right now our reviewers are starting to read the torrent of submissions…

212 proposals for presentations/seminars
31 proposals for tutorials
17 proposals for miniconfs

Holy shit!

Reading through some of the titles and abstracts, theres a lot of really awesome stuff that people have been working on, I’m really excited about what the programme is going to look like!

Unfortunately, we’re going to have to reject a lot of these :( On the upside, it does mean that the stuff that does make it through is going to be so mindblowingly awesome that you can’t afford not to be at linux.conf.au 2007!

Rock!

woo, new command

Thu, 14 Sep 2006 10:51:00 +1000

Shoulder-surfing johnf the other night at the seven meeting, I saw him use

cd -

to return to a previous directory…

I’d been using pushd and popd when I remembered; tried out cd - just now and bam! productivity increased 300%! Now I won’t be opening new terminals just to keep a shell in past directories…

wrapping CGI applications in WSGI

Tue, 12 Sep 2006 12:30:00 +1000

We’ve got a large “legacy” body of code that is used by our staff to track most of our business, it’s a whole lot of Python CGI that uses some custom HTML and DB frameworky code; it’s pretty ugly and having become a convert to the cult of Pylons, WSGI, and SQLAlchemy, I really want to replace it.

Of course, anyone knows that one of the Things You Should Never Do is rewrite from scratch. Even in the same language.

It would be much easier to integrate the old app into a new Pylons app, have them running side-by-side, and slowly deprecate the old one as new interfaces are written. (This is still not a perfect idea, as demonstrated by the 4 year old TCL code that the current app was meant to replace still running in production ;-) As bugs in the old code are found, we can either beat our heads against brick walls or replace just that functionality with a sane data model, similar looking templates, and shiny new controller smarts, and no-one would be the wiser, except of course that for some reason the developers are no longer constantly grumpy and the webapp is running smoother and faster than before, and crashing less often…

It occurred to me yesterday the best way to get a legacy CGI app to run along with Pylons is to convert it to a WSGI application, and just mash it in at the bottom of the application stack, where Pylons would normally go when it 404s.

Here’s the result of some free time and caffeinated excitement this morning:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34


import imp
import sys
import StringIO

def application(environ, start_response):
 # trap the exit handler, we don't want scripts exiting randomly
 # we might want to do something with the return code later
 retcode = None
 def exit_handler(rc):
 retcode = rc

 sys.exit = exit_handler

 # trap the output buffer
 outbuf = StringIO.StringIO()
 sys.stdout = outbuf

 # catch stderr output in the parent's error stream
 sys.stderr = environ['wsgi.errors']

 # import the script
 script = environ['PATH_TRANSLATED']
 f = open(script, "rb")
 imp.load_module('__main__', f, script, ("py", "rb", imp.PY_SOURCE))
 f.close()

 # outbuf has a typical CGI response, headers separated by a double
 # newline, then content
 (header, content) = outbuf.getvalue().split('\n\n', 1)
 headers = [tuple(x.split(': ', 1)) for x in header.split('\n')]

 # return it wsgi style
 start_response('200 OK', headers)
 return [content]

Our CGI apps print out on stdout, as you’d expect, so we need to trap that, here done with a StringIO monkeypatched on the top of sys.stdout. We also need to hack sys.exit out of the way, so that the CGIs don’t quit before we’ve completed the WSGI protocol. (I think this might cause some bugs in the execution though, because now it’s not terminating execution of the module, but I haven’t found an example yet to bother worrying about it.)

I import the script, rather than using os.system, because it feels right. I use imp.load_module rather than import because we don’t know what the script is until runtime :)

The real trick comes from a tip I found here , whilst looking for how to run the imported module as __main__. Just imp.load_module and tell it that it’s __main__! Simple!

(The hardest part about this whole excercise was now fiddling with sys.path and the CWD to make sure the imported script was running with the right environment that the CGIs used to expect, this is all done in the CGI runner dispatch.cgi which I won’t copy here because it’s pretty trivial and well documented in the WSGI spec.)

more than a feeling

Sun, 10 Sep 2006 14:43:00 +1000

I woke up this morning, and the sun was gone Turned on some music to start my day..

For a while, I’ve wanted to be woken up by anything other than my clock radio, so last night I peeked at banshee to see if it had a remote control… turns out it does!

Hacked up this script, shins:

1
2
3
4
5
6
7


#!/bin/sh

DISPLAY=:0
XAUTHORITY=/home/jaq/.Xauthority
export DISPLAY XAUTHORITY
banshee --enqueue /media/usbdisk/music0/Albums/The\ Shins/Chutes\ Too\ Narrow/01\ -\ Kissing\ the\ Lipless.ogg
banshee --play

and set it to run at 9am:

dawn% at 9am
warning: commands will be executed using /bin/sh
at> sh shins
at> <EOT>
job 6 at Sun Sep 10 09:00:00 2006

and this morning I was woken to the soft sounds of The Shins, just as planned. Great start to the day!

pylons gotchas

Sun, 03 Sep 2006 12:42:00 +1000

Benno was over, hacking on the LCA 2007 website with me yesterday, and we hit two gotchas, both I knew about but when I explained them to him they sounded silly.

`c` considered harmful.

c is a request-local global object that you can attach objects to, which is useful as a way of passing data from the controller to the template code – when you’re calling a parameterised template you might not know at call time what the args the template wants are, but you can pass them all in on c. If you’re using some pattern like a mixin CRUD class for generalising common data operations, then the code that actually calls the template doesn’t know what the object is, but the template it’s calling does.

c has the magical property that it has overloaded __getattr__ to return an empty string if the attribute is not found. This is a mixed blessing; your templates can access an attribute that hasn’t been attached and it’ll mostly cope with it. (Problems happen when you try to access attributes of nonexistent attributes, and you get the confusing message ‘str has no attribute X’.)

However, this means you hide bugs; you’ve forgotten to attach the object you want to c and then your code runs fine; it’s the users who find the problem after deployment, not during development. Having a __getattr__ that throws exceptions means you find out about these problems a lot sooner.

I think both of these points show that c in general is a bad idea; you should make use of explicit args so that your template interface is clearly defined – I haven’t yet found a nice way of doing it that is as easy as or better than using c though.

Myghty expressions that evaluate to False return empty strings.

We had a simple construct like so:

1

 Count: <% len(c.review_collection) %>

which has the interesting property of evaluating to ’’ when c.review_collection is empty; len() returns 0 which is False.

This is pretty retarded; I suspect there’s a shortcut along the lines of:

1
2
3


 content = evaluate_fragment("len(c.review_collection)")
 if content:
 write(content)

when these inline blocks are rendered; the if block clearly will fail to trigger when the inline block evaluates to 0, False, [], or {}. I can’t think of a case where this is a good thing.

The workaround is to wrap the len() call in str(), so that the fragment doesn’t evaluate to false.

1

 <% str(len(c.review_collection)) %>

More gotchas as they come to hand.

LCA 2007 proposals so far

Thu, 31 Aug 2006 13:40:00 +1000

I’m having a browse of the submissions to LCA that we’ve got so far, and there’s some cool stuff in there!

There’s still a little over 2 weeks for you to get your proposals in, so don’t hold back! I’m sure you all have something very exciting you want to talk about at the conference!

The more submissions we get, the rockin’er the conference will be!

Dad, I dug another hole...

Tue, 15 Aug 2006 13:53:00 +1000

I wrote another mock object, this time replacing urlopen from urllib2.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50


import urllib2
import StringIO
import unittest

class Dummy_urllib2(object):

 def install(cls):
 urllib2.urlopen = Dummy_urllib2.urlopen

 install = classmethod(install)

 def urlopen(self, url, data=None):
 self.url = url
 self.data = data

 response = StringIO.StringIO("foo")

 def geturl():
 return url

 response.geturl = geturl

 def info():
 return {}

 response.info = info

 return response

 urlopen = classmethod(urlopen)


class TestDummy_urllib2(unittest.TestCase):
 def test_install(self):
 Dummy_urllib2.install()

 url = 'http://notfound.example.org'

 try:
 r = urllib2.urlopen(url)
 except urllib2.URLError, e:
 self.fail("URLError raised, Dummy_urllib2 not installed or failed: %s" % e)

 self.assertEqual(url, Dummy_urllib2.url)
 self.assertEqual(url, r.geturl())
 self.assertEqual(None, Dummy_urllib2.data)
 self.assertEqual("foo", r.read())

if __name__ == '__main__':
 unittest.main()

This time it comes with it’s own test suite. How meta!

linux.conf.au 2007 CFP updates

Mon, 14 Aug 2006 17:22:00 +1000

I’m exhausted; a weekend of hacking and deployment has left me a bit frazzled.

But it’s OK! linux.conf.au 2007’s website has had a facelift, thanks to Andy Fitzsimmons for the CSS tweaks, and our CFP submission process is a lot better.

If you haven’t yet put in a proposal for a talk, a miniconf, a tutorial, then now’s the time! The CFP is still open!

a sudoku solver in Erlang

Mon, 31 Jul 2006 12:36:00 +1000

Matt will be pleased to hear that despite my claims that I’d have a productive weekend doing important things for LCA 2007 organisation, I spent the weekend programming.

Some bastard gave a great talk on Erlang at SLUG last Friday, and I really wanted to cut my teeth on it. So, hungover, I spent most of Saturday and a little bit of Sunday morning playing with the getting started guide and writing a Sudoku solver.

If you’re curious, you can pull it down from my bzr repository:

http://repo.spacepants.org/sudoku/sudoku.dev

mock LDAP server object

Wed, 26 Jul 2006 19:04:00 +1000

Today’s big achievement was an LDAP mock object, similar to the SMTP mock object found in paste.fixture. I was refactoring the sign in and sign out code of an in-house application that uses LDAP as an authentication store, and I needed to test that the logic of the controllers was correct. So, referring back to Paste’s lovely fixture module, I came up with the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


import ldap

class Dummy_ldap(object):

 def __init__(self, server):
 print "dummy ldap init"
 self.server = server

 def install(cls):
 ldap.initialize = cls

 install = classmethod(install)

 def simple_bind_s(self, dn, passwd):
 return True

 def search(self, base, scope, search_filter):
 self.base = base
 self.filter = search_filter
 self.results = [(ldap.RES_SEARCH_ENTRY, [('dn', "cn=test,%s" % self.base),
 ('mail', ['test'])]),
 (ldap.RES_SEARCH_RESULT, []),
 ]
 self.counter = 0
 return 1

 def result(self, rid, number):
 r = self.results[self.counter]
 self.counter += 1
 return r

A bit hackish, yes, but I’m not trying to reimplement LDAP here, I just want to trap the calls I use. Obviously a little bit of work is needed to, say, disallow the bind, or throw an exception, but these are trivial extensions.

Just as in Dummy_smtplib, you call the classmethod install to set it up (i.e. monkeypatch ldap.initialize) and you get to trap how it behaves.

1
2
3
4
5
6


class TestAccountController(ControllerTest):

 def test_signin_signout(self):
 Dummy_ldap.install()

 # ... do stuff

Simple!

paste.fixture's Dummy_smtplib

Thu, 20 Jul 2006 21:21:00 +1000

I’m working on two webapps that need to send email to users, and in both, the user is expected to click on a URL to confirm their registration. A common enough idiom for website accounts.

As a disciple of the cult of test-driven development, I want to be able to make a test that generates that email, inspects the contents for the URL, visits that generated URL, and then checks that the registration is completed.

Michael K, who previously suggested other abuses of Python’s dynamic nature, reminded me a few months ago that you could monkeypatch an imported library with your own, and it’d be preserved throughout the “address space” of the running Python program.

I’d not really played with it, and had been putting off writing a test for email because I didn’t really understand what I wanted. The other night at DebSIG, I asked (complained?) again about it, and he said “Hey, paste.fixture already does it!” I’d said I knew, but it had no documentation, and no-one on the paste users list had responded to my request for examples. He gave me a curt “Read the fucking source” response (in a much nicer way of course) and I thought, he’s right! Back in the day I used to read library source code in order to work out poorly documented APIs, why now do I rely on clear documentation so much? I should just dig in and write some example code to test it out, and JFDI.

Enough of the backstory.

Here’s a quick guide to setting up an email sender test, using paste.fixture around your Pylons application.

Firstly, in your controller, you have something that sends email, like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


import smtplib

from app.lib.base import BaseController, m

class FooController(BaseController):
 def index(self):
 """Do something and send email while you're at it"""
 s = smtplib.SMTP("localhost")

 s.sendmail("from@example.org",
 "to@example.com",
 """
Hey, this is a message

http://localhost/foo/activate/37
"""
 s.quit()

 m.subexec("foo/index.myt")

 def activate(self, id):
 m.write("awesome, activating %s" % id)

Pretty basic, if we visit /foo/ then we send an email, and then if we visit /foo/activate/N we inform the visitor of the activation.

The test is pretty simple too:

1
2
3
4
5
6
7


import re
import unittest

from paste.fixture import Dummy_smtplib

def TestFooController(unittest.TestCase):
 def test_foo_activation(self):

You just call the classmethod install on Dummy_smtplib to set it up, which does some magic behind the scenes (really it just replaces smtplib.SMTP with itself)

8

 Dummy_smtplib.install()

then run through the process you want to test

10
11


 # get the start page
 res = self.app.get('/foo')

and now we check that the message was sent, and its contents

13
14
15
16
17
18
19


 self.failIfEqual(None, Dummy_smtplib.existing, "no message sent")

 match = re.match(r'^.*/foo/activate/([^ ]+)', Dummy_smtplib.existing.message)
 self.failIfEqual(None, match)

 # visit the URL
 res = self.app.get('/foo/activate/%s' % match.group(1))

and finally, test the result of the activation.

20
21


 res.mustcontain('awesome')
 res.mustcontain(match.group(1))

You’ve also got to clean up, reset the dummy SMTP library for next time (you’ll get an exception thrown if you don’t, to remind you).

20

 Dummy_smtplib.existing.reset()

If you do any database stuff, then the times to check the status of the data model are just before the actuvation URL is visited, and again afterwards. I keep the model empty for each test, so I can pull out all the records and make sure that there’s only one afterwards, and that it has the right attributes before and after.

Pretty easy stuff.

lca2007 CFP open!

Tue, 18 Jul 2006 19:49:00 +1000

I was on a jet from Austin to London when it happened, but busy the hours beforehand in a flat in Austin nursing a hangover putting the final touches on the website. So, though I missed the chance to announce at the time, I’ll take this opportunity to blog about it now!

The LCA 2007 CFP is open, so submit your proposal for a talk, miniconf, etc now! (Or in reality, start writing your proposal now, so that you can submit it before the CFP closes ;-)

We’re looking forward to your submissions!

pylons, paste, and wsgi

Mon, 17 Jul 2006 14:20:00 +1000

I scribbled this down on our whiteboard last Friday, trying to explain how Pylons and Paste fit together. Prevously jdub and Lindsay had asked me similar questions. Until Friday, I wasn’t even sure myself.

The first thing to note is that Paste is not a framework or single library, it’s a collection of components that by themselves don’t do a lot, but with their powers combined form a set of useful and sometimes essential tools for building a web application in Python.

Paste implements an interface known as WSGI, aka the Web Server Gateway Interface. It’s defined in PEP 333. Basically WSGI describes a Chain of Command design pattern; each piece of a WSGI application takes a request, and either acts on that request or passes it along the chain. The interface described by WSGI means you can plug WSGI apps (or as Pylons calls them, /middleware/) together in any order as you like.

Why is this useful? Well, it means you can take an off-the-shelf authentication handler to cope with 403 and 401 responses and take care of logins. One would only need to say “this is how you authenticate someone” and “this is how you ask the user for their password.” Other things are possible; Pylons ships with an ultra-sexy 500 handler that puts you in a DHTML debugger, complete with traceback and Python interpreter. (Of course such a tool is a giant security hole so it is easily turned off in production environments.)

So, that’s Paste. There’s a few special cases in there, though: PasteScript and PasteDeploy. They’re special in that they tend to be at the bottom of the stack – they’re specifically for launching WSGI applications, configuration of the application (e.g. authenticatoin details alluded to above) and connecting to the application (e.g. direct HTTP, FastCGI, and other connectors). I suspect that my diagram above doesn’t lend itself well to describing how PasteScript and PasteDeploy really work; it’s still a bit of dark magic to me. I hope someone else would be able to build on this article with their own that rebuts the errors and clears the grey areas.

In a Pylons app, you tend not to notice Paste, except when deploying (because you tend to run the command paster serve to launch a development environment). Pylons itself is mostly just glue. It’s a thin veil of a framework over the top of some very powerful supporting libraries but presents them in a convenient and well defined way.

When you create a Pylons app, you get your paste middleware built for you, and then the entry point for your app is created as a WSGI application too. So it sits on top of the stack, taking in requests, and sending out responses. Your app can define its own middleware, too, so you have a lot of control over what happens between your app and the browser.

The main components of a Pylons app are:

A route mapper, by default Routes. The route mapper takes in URLs from the request passed into the app, and maps that URL to a controller object and method call. (If you’ve used RoR then you probably are familiar with this already.)
A templating engine, by default Myghty. The templating engine generates the view presented to the browser.
A data model. Pylons doesn’t prefer any method of data model, it just makes available a model module within which you can define your own data model. I use SQLAlchemy as an ORM because it is very powerful and is nicely suited to working with existing schemas. It works as an MVC between the data model presented to the application and the database schema itself.

Pylons lets you swap out any of these components with your own, if you desire. I find Routes and Myghty to be powerful and flexible and friendly enough that there’s no reason to want anything else.

Your controller objects, like any MVC pattern, coordinate between the model and the view. An action performed on a controller retrieves some data from the model, possibly altering it, and renders that data using the template engine.

There are other parts, other libraries that you’ll see in a Pylons app, that aren’t represented here. WebHelpers is a library of convenience functions used in the template engine, for generating common HTML and JavaScript. paste.fixture is a web app test framework that takes advantage of the common interface of WSGI to allow one to test their application without requiring a full web server and socket handling. FormEncode handles form validation, useful from within a controller object. These are but to name a few.

Unfortunately there is a sore need for overviews like this one in the Paste and Pylons community; as stated earlier I didn’t fully understand the relationships myself until I came up with this diagram. Hopefully then, dear reader, you have a better insight into how this collection of names fit together, and can avoid the steep learning curve :-)

TDD promotes good health

Thu, 01 Jun 2006 16:01:00 +1000

There’s an important advantage to Test Driven Development that I don’t think was covered on list or by Rob at his talk.

By having a test suite, you can code after a heavy liquid lunch and be sure that you’re not decreasing the quality of existing code. It makes it easier to focus on a specific task and write code to solve that problem. Having something do all the work for you when testing is a massive bonus, because obviulsy the side-effects of a pub lunch are that you are easily distracted and lack the willingness to focus on the task at hand. Test suites lower the barrier of entry to getting work done.

Who’da thought that best practices would also be best for drinking practice?

semiautomatic test generation

Mon, 29 May 2006 19:30:00 +1000

Whilst developing two webapps, one at work and one for the los palmas seven, I found that when writing the tests for the data model (hell yes I use test driven development, it’s very productive as it defines short and medium term goals and keeps you in focus) I was writing the same code over and over, but was very reliant on the structure of the data model I was testing.

Having done lots of currying in functional programming back in uni, I could see there was an obvious pattern, and thus it needed refactoring to reduce the amount of code I was writing and in doing so reduce the probability of error.

So I pulled out a set of functoins from one test, and made it a base class, and refactored it to allow the child class to specify the data; but due to the way python’s unittest and the test runner nose work, I had to manually write a function in each child class to call the worker functoin in the parent.

Obviously requiring the user to type more meant that there was a risk of error, and it was also tedious (read: very boring) to have to do this every time. Not knowing enough about python internals, I posted to the new slug coders list, christening it with the first development related post, and got back a good suggestion, but I didn’t try it out. Michael K suggested at the slug meeting last friday that I check out metaclasses if his previous suggestion didn’t work.

Saturday I jumped into the deep end with metaclasses, and it turned out they were easier than I thought, but having done so now I wouldn’t recommend them as a hammer to bash in all those screw and rivet problems. They are very cool though.

So I ended up with a base class that had a metaclass that would monkey patch the child class with the correct method names just as it was being inspected, and had a small test program written that demonstrated my idea was sound.

However, applying this method to the actual tests running under nose didn’t work. Lots of debugging printfs later I eventually traced this to a peculiarity in the way nose decides on what tests to run: to prevent test methods being run twice, it makes sure that when running a test, the module that it is defined in is the same as the module currently being tested, i.e. it makes sure that __module__ matches in both the callable and the current test case.

Now, when you define your methods in a parent, the method’s module is that of the parent, so a structure like the following:

tests/module/
tests/module/__init__.py
tests/module/test_something.py

sets __module__ in __init__.py to tests.module and in test_something.py to tests.module.test_something. Running a method from tests.module.test_something that’s defined in tests.module, nose says no.

Enter Benno and his knowledge of python internals. An impromptu 7 hackfest on Sunday at jdub’s house let me show him what I was trying to do, where he suggested using python’s new package, and the im_func attributes on callables, to build a workaround for nose’s features, which is much better than what I was thinking of doing to solve the problem (something about injecting docstrings into the child class and then evaling them in the child’s context).

Some quick hacks later, we had a test program that showed it’d work, and patched up the base test class. Appended for your enjoyment, a base test class that one can inherit from to automatically generate common tests for tables when using SQLAlchemy.

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224


from unittest import TestCase

import sqlalchemy

import zookeepr.models as model


class TestBase(TestCase):
 def assertRaisesAny(self, callable_obj, *args, **kwargs):
 """Assert that the ``callable_obj`` raises any exception."""
 try:
 callable_obj(*args, **kwargs)
 except:
 pass
 else:
 self.fail("callable %s failed to raise an exception" % callable_obj)


def monkeypatch(cls, test_name, func_name):
 """Create a method on a class with a different name.

 This method patches ``cls`` with a method called ``test_name``, which
 is bound to the actual callable ``func_name``.

 In order to make sure test cases get run in children of the assortment
 of test base classes in this module, we do not name the worker methods
 with the prefix 'test_'. Instead they are named otherwise, and we
 alias them in the metaclass of the test class.

 However, due to the behaviour of ``nose`` to not run tests that are
 defined outside of the module of the current test class being run, we
 need to create these test aliases with the model of the child class,
 rather than simply calling ``setattr``.

 (Curious readers can study ``node.selector``, in particular the
 ``wantMethod``, ``callableInTests``, and ``anytests`` methods (as of
 this writing).)

 You can't set __module__ directly because it's a r/o attribute, so we
 call ``new.function`` to create a new function with the same code as
 the original. The __module__ attribute is set by the new.function
 method from the globals dict that it is passed, so here we make a
 shallow copy of the original and override the __name__ attribute to
 point to the module of the class we're actually testing.

 By this stage, you may think that this is crack. You're right.
 But at least I don't have to repeat the same code over and
 over in the actual tests ;-)
 """
 # get the code
 code = getattr(cls, func_name).im_func.func_code
 # get the function globals so we can overwrite the module
 g = getattr(cls, func_name).im_func.func_globals.copy()
 g['__name__'] = cls.__module__
 # create a new function with:
 # the code of the original function,
 # our patched globals,
 # and the new name of the function
 setattr(cls, test_name, new.function(code, g, test_name))


class TableTestGenerator(type):
 """Monkeypatching metaclass for table schema test classes.

 This metaclass does some funky class method rewriting to generate
 test methods so that one doesn't actually need to do any work to get
 table tests written. How awesome is that for TDD? :-)
 """
 def __init__(mcs, name, bases, classdict):
 type.__init__(mcs, name, bases, classdict)
 if 'table' in classdict:
 monkeypatch(mcs, 'test_insert', 'insert')

 for k in ['not_nullable', 'unique']:
 if k + 's' in classdict:
 monkeypatch(mcs, 'test_' + k, k)


class TableTest(TestBase):
 """Base class for testing the database schema.

 Derived classes should set the following attributes:

 ``table`` is a string containing the name of the table being tested,
 scoped relative to the module ``zookeepr.models``.

 ``samples`` is a list of dictionaries of columns and their values to use
 when inserting a row into the table.

 ``not_nullables`` is a list of column names that must not be undefined
 in the table.

 ``uniques`` is a list of column names that must uniquely identify
 the object.

 An example using this base class:

 class TestSomeTable(TestTable):
 table = 'module.SomeTable'
 samples = [dict(name='testguy', email_address='test@example.org')]
 not_nullables = ['name']
 uniques = ['name', 'email_address']
 """
 __metaclass__ = TableTestGenerator

 def get_table(self):
 """Return the table, coping with scoping.

 Set the ``table`` class variable to the name of the table variable
 relative to anchor.model.
 """
 module = model
 # cope with classes in sub-models
 for submodule in self.table.split('.'):
 module = getattr(module, submodule)
 return module

 def check_empty_table(self):
 """Check that the database was left empty after the test"""
 query = sqlalchemy.select([sqlalchemy.func.count(self.get_table().c.id)])
 result = query.execute()
 self.assertEqual(0, result.fetchone()[0])

 def insert(self):
 """Test insertion of sample data

 Insert a row into the table, check that it was
 inserted into the database, and then delete it.

 Set the attributes for this model object in the ``attrs`` class
 variable.
 """

 self.failIf(len(self.samples) < 1, "not enough sample data, stranger")

 for sample in self.samples:
 print "testing insert of s %s" % sample
 query = self.get_table().insert()
 print query
 query.execute(sample)

 for key in sample.keys():
 col = getattr(self.get_table().c, key)
 query = sqlalchemy.select([col])
 print "query", query
 result = query.execute()
 print result
 row = result.fetchone()
 print "row", row
 self.assertEqual(sample[key], row[0])

 self.get_table().delete().execute()

 # do this again to make sure the test data is all able to go into
 # the db, so that we know it's good to do uniqueness tests, for example
 for sample in self.samples:
 query = self.get_table().insert()
 query.execute(sample)

 # get the count of rows
 query = sqlalchemy.select([sqlalchemy.func.count(self.get_table().c.id)])
 result = query.execute()
 # check that it's the same length as the sample data
 self.assertEqual(len(self.samples), result.fetchone()[0])

 # ok, delete it
 self.get_table().delete().execute()

 self.check_empty_table()

 def not_nullable(self):
 """Check that certain columns of a table are not nullable.

 Specify the ``not_nullables`` class variable with a list of column names
 that must not be null, and this method will insert into the table rows
 with each set to null and test for an exception from the database layer.
 """

 self.failIf(len(self.samples) < 1, "not enough sample data, stranger")

 for col in self.not_nullables:
 print "testing that %s is not nullable" % col

 # construct an attribute dictionary without the 'not null' attribute
 coldata = {}
 coldata.update(self.samples[0])
 coldata[col] = None

 # create the model object
 print coldata

 query = self.get_table().insert()
 self.assertRaisesAny(query.execute, coldata)

 self.get_table().delete().execute()

 self.check_empty_table()

 def unique(self):
 """Check that certain attributes of a model object are unique.

 Specify the ``uniques`` class variable with a list of attributes
 that must be unique, and this method will create two copies of the
 model object with that attribute the same and test for an exception
 from the database layer.
 """

 self.failIf(len(self.samples) < 2, "not enough sample data, stranger")

 for col in self.uniques:

 self.get_table().insert().execute(self.samples[0])

 attr = {}
 attr.update(self.samples[1])

 attr[col] = self.samples[0][col]

 query = self.get_table().insert()
 self.assertRaisesAny(query.execute, attr)

 self.get_table().delete().execute()

 self.check_empty_table()

Feel free to use this in your own code, I’m placing it in the public domain.

cityrail ticket cubohemioctahedron

Mon, 15 May 2006 10:55:00 +1000

A while ago, as many of you would, I saw the Metrocard tricontahedron linked from the Makezine blog.

This weekend, whilst hacking away, I took a break and was distracted by the large pile of used tickets that have built up on my desk. I found the above site again, and spend about 15 minutes folding some tickets until I emerged to the loungeroom to show off the results.

(flickr)

So what it this shape actually called? I googled ’tricontrahedron' (because I can’t read) and then ’tricontahedron’, and discovered that this name is derived from the prefix tri-, the modifier conta- (which refers to a group of ten), and the suffix -hedron, meaning “faces”, so we have a 30-faced polyhedra. True so far, but actually googling or searching Mathworld for tricontahedron wasn’t showing me any graphical evidence that this was the right name. In fact, the closest by name, the rhombic tricontahedron, looks nothing like this shape.

But finally I stumbled on a set of excellent polyhedra sites:

Anthony Thyssen’s Studies into Polyhedra
V. Bulatov’s Polyhedra Collection
George Hart’s Encyclopaedia of Polyhedra

Thanks to FreeWRL I managed to finally find the cubohemioctahedron, which although the image on that linked page doesn’t look terribly much like our shape, upon loading the VRML for the polyhedron I was able to convince myself that this is the one.

welcome to 1998

Sun, 14 May 2006 00:47:00 +1000

My PCMCIA wireless card was flaking out today, after I got my IrDA port working; I’d reconfigued the port in the BIOS to use IRQ 3. After some time I found that the wireless card was also choosing IRQ 3, thanks to the output from dmesg. Pushing the IrDA port to IRQ 4 fixed that little conflict. I’m still left wondering why in this day and age I’m editing IRQ settings on a laptop that’s barely a year old.

I started out trying to get photos off of my phone, which I’ve successfully done in the past but always forget how; today the phone and laptop just refused to connect. The cause of the problem was that the serial8250 driver was grabbing the IO ports and IRQ first, and preventing the nsc-ircc driver from getting it later in the boot process. I think the fix for that was to put nsc-ircc at the top of /etc/modules; at least, it’s working now.

As I was rolling on my “fix things that are pissing me off” rampage, I decided to crack at the “why the hell isn’t DMA working” one. The X40 has an ICH4-M chipset, handled by the piix driver. The driver was loading, but hdparm was always getting this:

# hdparm -d 1 /dev/hda
/dev/hda:
setting using_dma to 1 (on)
HDIO_SET_DMA failed: Operation not permitted
using_dma = 0 (off)

After spending way too long following threads full of erroneous opinion on the ubuntu forums, debian-user mailing list archives, and other useful tomes, I discovered this clue in dmesg:

ICH4: port 0x01f0 already claimed by ide0

which led me to this post, cluing me to the fact that the ide-generic driver was being greedy. Again, a driver conflict, but to fix this one, I had to put piix at the top of /etc/mkinitramfs/modules and then rebuild the initrd (running dpkg-reconfigure on your kernel package will take care of this). So, thanks Christophe, for your bug report that finally helped solve this one, and now holy shit my computer is fast.

Datarock programmed Rage tonight, too, which was an awesome soundtrack to fix things to ;-) They and I have such similar tastes in music.

NET-SNMP Linux disk IO collector, mark III

Fri, 12 May 2006 17:02:00 +1000

Ok, this might be the last one ;-)

As mentioned previously, I’ve been writing a script that net-snmp can pass off the UCD-DISKIO-MIB requests to, for those versions that don’t have it built in, and to save one from rebuilding net-snmp packages.

This time around, I’ve added support for 2.4 series kernels, which use /proc/partitions instead of /proc/diskstats to log the disk IO stats.

I’ve now got this running in production across a handful of machines, and it’s pretty neat – I am able to see the shape of disk traffic during a vacuum on one of our machines, alongside the CPU and memory usage, and load average graphs, for example.

As before, plop it into /usr/local/sbin/snmp-diskio-collector and add this line to your snmpd.conf:

pass .1.3.6.1.4.1.2021.13.15 /usr/local/sbin/snmp-diskio-collector

Here’s the script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146


#!/bin/sh

# by Jamie Wilkinson <jamie@anchor.net.au>
# this code is in the public domain
# based on passtest from the net-snmp distribution examples
# WARNING there is shitloads of IO required to get this information :)

debug_flag=0

debug () {
 if [ $debug_flag -eq 1 ]; then
 echo $* >> /tmp/snmp-diskio-collector-debug.log
 fi
}

PLACE=".1.3.6.1.4.1.2021.13.15"
REQ="$2"

debug
debug "new run"
debug "args $*"
if [ "$1" = "-s" ]; then
 exit 0
fi

# the 'tail' of the oid, everything after $PLACE
oidtail=`echo $REQ | sed "s/^$PLACE//"`
debug "oidtail=$oidtail"

# number of devices we can export
if [ -f /proc/diskstats ]; then
 devcount=`wc -l /proc/diskstats | sed 's/^ *//' | cut -f1 -d' '`
else
 devcount=`wc -l /proc/partitions | sed 's/^ *//' | cut -f1 -d' '`
 devcount=`expr $devcount - 2`
fi
debug "devcount=$devcount"

item=`echo $oidtail | cut -f4 -d.`
index=`echo $oidtail | cut -f5 -d.`
debug "oidtail=$oidtail, item=$item, index=$index"

if [ "$1" = "-n" ]; then
 if [ -z "$item" ]; then
 item=1
 index=1
 elif [ -z "$index" ]; then
 index=1
 else
 index=`expr $index + 1`
 if [ "$index" -gt "$devcount" ]; then
 index=1
 item=`expr $item + 1`
 if [ "$item" -gt 6 ]; then
 # break out of the loop
 exit 0;
 fi
 fi
 fi
 RET=$PLACE.1.1.$item.$index
else
 case "$REQ" in
 $PLACE) exit 0;;
 *) RET=$REQ ;;
 esac
fi

debug "after -n, item=$item, index=$index"
debug "RET is now $RET"

echo "$RET"

debug "oidtail=$oidtail, item=$item, index=$index"

# awk uses this variable in the environment below
export index
# see linux kernel Documentation/iostats.txt for format
if [ -n "$index" ]; then
 case "$item" in
 1)
 # diskIOIndex
 debug "result: diskIOIndex $index"
 echo "integer"
 echo $index
 exit 0
 ;;
 2)
 # diskIODevice
 debug "result: diskIODevice $index"
 echo "string"
 if [ -f /proc/diskstats ]; then
 awk 'FNR == ENVIRON["index"] { print $3 }' /proc/diskstats
 else
 awk 'FNR == ENVIRON["index"] + 2 { print $4 }' /proc/partitions
 fi
 exit 0
 ;;
 3)
 # diskIONRead
 debug "result: diskIONRead $index"
 echo "counter"
 if [ -f /proc/diskstats ]; then
 awk 'FNR == ENVIRON["index"] { print $6 }' /proc/diskstats
 else
 awk 'FNR == ENVIRON["index"] + 2 { print $7 }' /proc/partitions
 fi
 exit 0
 ;;
 4)
 # diskIONWritten
 debug "result: diskIONWritten $index"
 echo "counter"
 if [ -f /proc/diskstats ]; then
 awk 'FNR == ENVIRON["index"] { print $10 }' /proc/diskstats
 else
 awk 'FNR == ENVIRON["index"] + 2 { print $11 }' /proc/partitions
 fi
 exit 0
 ;;
 5)
 # diskIOReads
 debug "result: diskIOReads $index"
 echo "counter"
 if [ -f /proc/diskstats ]; then
 awk 'FNR == ENVIRON["index"] { print $4 }' /proc/diskstats
 else
 awk 'FNR == ENVIRON["index"] + 2 { print $5 }' /proc/partitions
 fi
 exit 0
 ;;
 6)
 # diskIOWrites
 debug "result: diskIOWrites $index"
 echo "counter"
 if [ -f /proc/diskstats ]; then
 awk 'FNR == ENVIRON["index"] { print $8 }' /proc/diskstats
 else
 awk 'FNR == ENVIRON["index"] + 2 { print $9 }' /proc/partitions
 fi
 exit 0
 ;;
 *) exit 0; #echo "string"; echo "debug... $RET $REQ"; exit 0 ;;
 esac
else
 exit 0
fi

NET-SNMP Linux disk IO collector, mark II

Fri, 12 May 2006 14:28:00 +1000

This is an improvement on a previous post on adding UCD-DISKIO-MIB support to older versions of net-snmp; this one actually works on 2.6 kernels (and thanks to Pete loses about a million forks and pipes).

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121


#!/bin/sh

# by Jamie Wilkinson <jamie@anchor.net.au>
# this code is in the public domain
# based on passtest from the net-snmp distribution examples
# WARNING there is shitloads of IO required to get this information :)

debug_flag=0

debug () {
 if [ $debug_flag -eq 1 ]; then
 echo $* >> /tmp/snmp-diskio-collector-debug.log
 fi
}

PLACE=".1.3.6.1.4.1.2021.13.15"
REQ="$2"

debug
debug "new run"
debug "args $*"
if [ "$1" = "-s" ]; then
 exit 0
fi

# the 'tail' of the oid, everything after $PLACE
oidtail=`echo $REQ | sed "s/^$PLACE//"`
debug "oidtail=$oidtail"

# number of devices we can export
devcount=`wc -l /proc/diskstats | cut -f1 -d' '`
debug "devcount=$devcount"

item=`echo $oidtail | cut -f4 -d.`
index=`echo $oidtail | cut -f5 -d.`
debug "oidtail=$oidtail, item=$item, index=$index"

if [ "$1" = "-n" ]; then
 if [ -z "$item" ]; then
 item=1
 index=1
 elif [ -z "$index" ]; then
 index=1
 else
 index=`expr $index + 1`
 if [ "$index" -gt "$devcount" ]; then
 index=1
 item=`expr $item + 1`
 if [ "$item" -gt 6 ]; then
 # break out of the loop
 exit 0;
 fi
 fi
 fi
 RET=$PLACE.1.1.$item.$index
else
 case "$REQ" in
 $PLACE) exit 0;;
 *) RET=$REQ ;;
 esac
fi

debug "after -n, item=$item, index=$index"
debug "RET is now $RET"

echo "$RET"

debug "oidtail=$oidtail, item=$item, index=$index"

# awk uses this variable in the environment below
export index
# see linux kernel Documentation/iostats.txt for format
if [ -n "$index" ]; then
 case "$item" in
 1)
 # diskIOIndex
 debug "result: diskIOIndex $index"
 echo "integer"
 echo $index
 exit 0
 ;;
 2)
 # diskIODevice
 debug "result: diskIODevice $index"
 echo "string"
 awk 'FNR == ENVIRON["index"] { print $3 }' /proc/diskstats
 exit 0
 ;;
 3)
 # diskIONRead
 debug "result: diskIONRead $index"
 echo "counter"
 awk 'FNR == ENVIRON["index"] { print $6 }' /proc/diskstats
 exit 0
 ;;
 4)
 # diskIONWritten
 debug "result: diskIONWritten $index"
 echo "counter"
 awk 'FNR == ENVIRON["index"] { print $10 }' /proc/diskstats
 exit 0
 ;;
 5)
 # diskIOReads
 debug "result: diskIOReads $index"
 echo "counter"
 awk 'FNR == ENVIRON["index"] { print $4 }' /proc/diskstats
 exit 0
 ;;
 6)
 # diskIOWrites
 debug "result: diskIOWrites $index"
 echo "counter"
 awk 'FNR == ENVIRON["index"] { print $8 }' /proc/diskstats
 exit 0
 ;;
 *) exit 0; #echo "string"; echo "debug... $RET $REQ"; exit 0 ;;
 esac
else
 exit 0
fi

NET-SNMP Linux disk IO collector

Wed, 10 May 2006 15:35:00 +1000

The UCD-DISKIO-MIB isn’t supported in versions of net-snmp on most stable distros, but a few tools have add-on support for trending the data returned by this MIB.

You can recompile net-snmp, but I find that both boring and a maintenance headache; I don’t manually build anything that the vendor already supplies unless there’s a really good reason. Disk IO statistics are not a good enough reason to recompile.

The other option, the one I took, is to use the pass option of snmpd.conf to offload the collection of a MIBOID subtree to an external program. The API is pretty simple but there’s a dearth of examples on the interweb, so I present this one hopefully as an example for others to extend their own MIBs and also in the hope that others wanting to analyse disk performance find it useful.

It’s not terribly efficient, each call of the script looks at /proc/diskstats twice, there’s some tens of forks involved, and if you’re walking the subtree, that means you can be calling this script 6 times for every block device in the system. However, if you want efficiency, rather than trying to optimise this script, just rebuild net-snmp. My goal was to reduce deployment effort.

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102


#!/bin/sh

# /usr/local/sbin/snmp-diskio-collector 
# by Jamie Wilkinson <jamie@anchor.net.au>
# this code is in the public domain
# based on passtest from the net-snmp distribution examples
# WARNING there is shitloads of IO required to get this information :)

PLACE=".1.3.6.1.4.1.2021.13.15"
REQ="$2"

if [ "$1" = "-s" ]; then
 exit 0
fi

# the 'tail' of the oid, everything after $PLACE
oidtail=`echo $REQ | sed "s/^$PLACE//"`

# number of devices we can export
devcount=`wc -l /proc/diskstats | cut -f1 -d' '`

if [ "$1" = "-n" ]; then
 case x"$oidtail" in
 x|x.1|x.1.1|x.1.1.1)
 # bootstrap a snmpwalk to the first index and item
 RET=$PLACE.1.1.1.1
 ;;
 x.1.1.*.*)
 # item is the element of the diskIOEntry
 item=`echo $oidtail | sed 's/\.1\.1\.\([^\.]*\)\..*/\1/'`
 # index is our disk table entry
 index=`echo $oidtail | sed 's/\.1\.1\.[^\.]*\.\(.*\)/\1/'`

 if [ "$index" -eq $devcount ]; then
 # wrap to the next item, starting at index 1
 RET=$PLACE.1.1.`expr $item + 1`.1
 else
 if [ x"$item" = x7 ]; then
 # bail because we aren't doing more than 6 items in the
 # entry
 exit 0
 fi
 RET=$PLACE.1.1.$item.`expr $index + 1`
 fi
 ;;
 *) exit 0;;
 esac
else
 case "$REQ" in
 $PLACE) exit 0 ;;
 *) RET=$REQ ;;
 esac
fi

echo "$RET"

oid=`echo $RET | sed "s/^$PLACE//"`
item=`echo $oid | sed "s/.*\.//"`

# see linux kernel Documentation/iostats.txt for format
case "$RET" in
 $PLACE.1) exit 0 ;; #diskIOTable
 $PLACE.1.1) exit 0 ;; #diskIOEntry
 $PLACE.1.1.1) exit 0;; #diskIOIndex
 $PLACE.1.1.1.*)
 # diskIOIndex
 echo "integer"
 echo $item
 exit 0
 ;;
 $PLACE.1.1.2.*)
 # diskIODevice
 echo "string"
 head -n $item /proc/diskstats | tail -n 1 | awk '{print $3}'
 exit 0
 ;;
 $PLACE.1.1.3.*)
 # diskIONRead
 echo "counter"
 head -n $item /proc/diskstats | tail -n 1 | awk '{print $6}'
 exit 0
 ;;
 $PLACE.1.1.4.*)
 # diskIONWritten
 echo "counter"
 head -n $item /proc/diskstats | tail -n 1 | awk '{print $10}'
 exit 0
 ;;
 $PLACE.1.1.5.*)
 # diskIOReads
 echo "counter"
 head -n $item /proc/diskstats | tail -n 1 | awk '{print $4}'
 exit 0
 ;;
 $PLACE.1.1.6.*)
 # diskIOWrites
 echo "counter"
 head -n $item /proc/diskstats | tail -n 1 | awk '{print $8}'
 exit 0
 ;;
 *) exit 0;;
esac

This script gets hooked into snmpd.conf with a line like the following:

pass .1.3.6.1.4.1.2021.13.15 /usr/local/sbin/snmp-diskio-collector

and then you can query the MIB with either

snmpwalk -m ALL -v1 -c public localhost diskIOTable

snmptable -m ALL -v1 -c public localhost diskIOTable

(but of course you wouldn’t use SNMPv1 or v2c if you were going over the internets, right? :-)

Now you can tie this into Cacti or your favourite NMS, such as described here.

cross-language bindings, marshalling, a thought experiment in evile

Sat, 29 Apr 2006 14:02:00 +1000

Last night, Statler and Waldorf talked about free software that they hated on the LA live broadcast, including a lovely piece of work Erik mentioned, called intelib.

Putting LISP-like syntax into C++ didn’t actually surprise me, maybe that’s more of an insight into my own dark and twisted past than anything, there’s plenty of cross-language stuff around; in fact as Matt went on a small tangent talking about wrapping APIs for the purposes of adding functionality to other languages, I realised I could concoct something evil enough to make them both vomit.

Conveniently, James Dumay later talked about Mono and .NET, and showed a few languages that are usable with the language/platform. Again, not surprising, I did some stuff with marshalling types and function calls between C and Haskell back at uni, and as long as you keep all the type information around (as .NET CIL does) it’s pretty easy. However, you don’t need .NET to marshall data and function calls between languages:

Boost.Python is “A C++ library which enables seamless interoperability between C++ and the Python programming language.” It lets you call Python methods and objects from C++ code as if they were native to the language.
PyPerl is “a module for integrating Perl in Python.” Again, it lets you use a Perl module’s API directly within your Python code.
The Inline:: series of Perl modules allow you to “write Perl subroutines in X”, for values of X such as assembler, awk, Basic, Befunge, C, Interp, Java, Lua, Octave, Parrot, Perl (wtf?), Python, Ruby, SLang, Tcl, and a whole lot of other shit that you’ll never want to use. So, you can call functions in language X from within Perl.

I think you can see where I’m going with this. It’s possible to write a piece of code in C++ that links Boost.Python to import PyPerl to use Inline::Lua to call functions in a binding around a C library. That’s just the tip of the iceberg ;-)

backporting OpenLDAP to old distros

Mon, 24 Apr 2006 11:06:00 +1000

It would be really nice if those projects that consider themselves to be major pieces of infrastructure made it a priority to support a build on stock vendor releases of distros older than the current release.

I’m trying to build OpenLDAP 2.3.20 on Red Hat 7.3. Yes, that sounds like a bad idea, and it is quite painful. This pain is mostly due to the versions of Berkeley DB that OpenLDAP wants to build against: Red Hat 7.3 shipped with BDB 3.3.11; it wants at least version 4, bud the configure script makes it quite clear that version 4.1 isn’t supported.

The LDBM backend can use older BDBs, i.e. 3.x, but OpenLDAP 2.2 deprecates the LDBM backend in favour of back-bdb. That concerns me for a 2.3 series build of OpenLDAP: is it still stable, or has it been left to rot? Of course the test suite doesn’t cover the LDBM backend…

I would really like to not have to build for this old OS version, but sadly it exists and it needs to be a syncrepl slave. In order to finally put this box to rest it needs to have OpenLDAP running, so that we can migrate the services off it with minimal outage. There are so many bugs running OpenLDAP on this box that I want to try this 2.3 series version on it, in the same configuration as every other machine on the network, but the more I work on it, the less it feels like I’m making any progress on it at all.

Fighting version dependencies on old distros just isn’t fun. If the developers had considered this, and at least said “OK, well, RHEL 2.1 is still supported by Red Hat, so let’s try to configure and build on that platform too” then this wouldn’t nearly be as painful as it is.

The Annodex developers, as well as Conrads’ other projects (sweep, etc) work quite hard to make sure they build with the versions of libraries on peoples year-old desktop machines; sure this means sometimes having extra code to cope with API differences, but a small amount of effort on the developers part makes a massive difference when you think about how much time the users will spend trying to get the software to build.

It’s the little things that make the difference between an OK project and an Awesome project.

Meanwhile, on the cutting edge of RHEL 3 and 4, and FC 4 and 5, latest OpenLDAP 2.3 looks to be a really promising piece of work, the amount of work that’s gone into it since 2.2.27 is impressive, and if it works nearly as well as it looks like it will from the ChangeLog, then that’ll finally put to rest a whole lot of problems we’ve had since deploying LDAP as our authentication database.

Fingers crossed I can get there without losing all my hair.