Category Archives: Linux

Tales of my climb up the Linux learning curve, on both desktop and server.

Introducing eximunit

For a few years now, I’ve run a hosting co-operative with a few friends. Although the cost savings versus all renting VMs individually are probably marginal at best these days, one of the nice things about it is the chance to run things like our incoming MX on one machine only, instead of all having to run our own anti-spam and other measures. The incoming mail is handled by Exim, and each user of our system can add domains for which mail is processed. They get to toggle SMTP-time rejection of spam and viruses, and specify the final destination machine for incoming mail to their domain.

This has all been working well for overĀ  two years, but occasionally something has to change: a few months ago, we got rid of sender verify callouts, now widely considered abusive by SMTP server admins, and more recently we added support for tagging messages with headers to say if they passed or failed DKIM verification. And every time I make such a change, I worry that I might have inadvertently broken something. This server handles mail for 30 domains and 8 people, some of who rely on it to run businesses! Panic!

I usually end up reassuring myself by doing some ad-hoc testing by hand after reconfiguring the server. At the most basic level, whatever your SMTP server is, you can use netcat to have a conversation with it on port 25:

d@s:~$ nc localhost 25
220 mailserver.splice.org.uk ESMTP Exim 4.71 Sat, 17 Mar 2012 09:51:20 +0000
HELO localhost
250 mailserver.splice.org.uk Hello localhost [127.0.0.1]
MAIL FROM: <>
250 OK
RCPT TO: someaddress@dnorth.net
550-Callout verification failed:
550 550 Unrouteable address
QUIT
221 mailserver.splice.org.uk closing connection

And there, I’ve just convinced myself that one of our features is still working: the mailserver should call forward to the final destination for mail to @dnorth.net addresses to check the local part (‘someaddress’ in this case) is valid, and reject the message up-front if it’s not.

Exim also has a load of other toys you can take advantage of: say I want to check how mail to webmaster@dnorth.net is routed:

d@s:~$ exim4 -bt webmaster@dnorth.net
R: hubbed_hosts for dnorth.net
webmaster@dnorth.net
 router = hubbed_hosts, transport = remote_smtp
 host mx-internal.dnorth.net [192.0.2.100]

(IP addresses changed for example purposes, obviously)

And finally, there’s debug mode: you can run

exim4 -bhc <ip address>

to run a complete ‘fake’ SMTP session as though you were connecting from the given IP address. You can send messages, but they won’t actually go through, and exim prints a lot of debug output to give you a clue as to its inner workings as it decides how to route the message.

This is all very well, but a quick brainstorming session gives a list of over 30 things I might want to check about my mailserver:

  • Basic check that mail is accepted to our domains
  • Only existent addresses on our domains should have mail accepted
  • Domains with SMTP-time spam rejection on should have spam rejected
  • Same for viruses
  • Same for greylisting

Testing all these by hand isn’t going to fly, so what tools can we find for automating it? A bit of Googling turns up swaks, which looks quite handy, but suffers from two drawbacks for me: first, it’s a bit low-level, and a collection of scripts calling it will be a bit difficult to read and maintain for testing all 30 of my assertions. Second, it really sends the e-mails in the success case, and I don’t want my users to get test messages or have to set up aliases for receiving and discarding them. swaks will definitely become my tool of choice for ad-hoc testing in future, but meanwhile…

The other promising Google result is Test::MTA::Exim4, which is a Perl wrapper for testing an exim config file. However, a few problems: (1) it’s Perl, and I Don’t Do Perl. (2), it’s limited to testing the routing of addresses, so it’s not going to cut it for checking spam rejection etc.

Having at least pretended not to be suffering from NIH syndrome, let’s spec out a fantasy system for doing what I want: I would like to be able to write some nice high-level tests in my favourite language, Python, which look a bit like this

class HubbedDomainTests(EximTestCase):
    """
    Tests for domains our server acts as the 'proxy MX' for, doing
    scanning etc before forwarding the mail to the destination machine
    """

    def testProxiedMailAccepted(self):
        """Proxied mail should be accepted"""
        session = self.newSession()
        session.mailFrom('dtnorthie@gmail.com').rcptTo('webmaster@dnorth.net').randomData()

    def testLocalPartsVerifiedWithDestinationMachine(self):
        """Local parts should be verified with the destination machine"""
        session = self.newSession()
        session.mailFrom('dtnorthie@gmail.com').assertRcptToRejected('doesnotexist@dnorth.net')

I could then run these in the usual manner for Python unit tests, and lastly, I want them backed by an exim4 -bhc session so that they’re as realistic as possible without actually sending messages.

This post is long enough already, so I’ll cut to the chase and say that I’ve made a start on writing it, and you can find out more at Bitbucket. In a follow-up post, I’ll talk about how it was done.

British Gas find a new way to annoy me…

As if they weren’t an inefficient enough organisation to deal with in other respects, today a division of British Gas asked me to send them a remittance advice to:

CardiffC&MFinance@centrica.com

Yes, that really is an ampersand in the local part. Sufficiently unusual that trying to send to it upset my default-configured installation of Exim:

rejected RCPT <CardiffC&MFinance@centrica.com>: restricted characters in address

You can tone down the (perfectly reasonable) check for these iffy characters by exempting centrica.com from it: edit /etc/exim4/conf.d/acl/30_exim4-config_check_rcpt and edit the domains line of the second ‘restricted characters in address’ ACL to read:

domains = !+local_domains : !centrica.com

India diary, day 5

In June/July 2010 I spent ten days travelling in Rajasthan, India with friends; this is my diary of the trip (full list of entries here).

I was quite ill today and spent it in the hotel feeling sorry for myself. Still; I had company – K also feeling pretty ropey. Happily we were both feeling a bit better by evening and resolved to press on in the morning for the third leg provided we felt no worse.

Slow outbound e-mail?

I’d noticed for years that sending outbound e-mail from Thunderbird to my server on port 587 took far longer than it should have – about six seconds of staring at the progress bar.

Today, I was finally bored enough to work out the cause: Exim is configured to perform ident checks, which take 5 seconds to time out. Since port 587 only accepts mail from authenticated users, we can disable the ident checks for it:

rfc1413_hosts = ${if eq{$interface_port}{25}{*}{! $sender_host_address}}

If you’re running Debian, change the above in /etc/exim4/conf.d/main/02_exim4-config_options.

A smartphone named Desire

After years of thinking about it, and even writing about it, I’ve finally bought myself a smartphone.

As I write, I’m coming to the end of my third week with my new HTC Desire S. And it hasn’t disappointed.

Carrying it around has proved less arduous than I imagined – although it weighs more than my clunky old Nokia 1100, it’s thinner and flatter, so it fits nicely in my shirt pocket. It gets quite warm when under heavy use (e.g. acting as a WiFi hotspot), but not unpleasantly so.

They’ve packed quite a lot into such a small case – in no particular order, we have FM radio, a 5 megapixel camera with LED flash which works quite well for basic snaps, a second front-facing camera for video calling (makes me look awful, but friends insist it’s accurate), GPS, a half-decent speaker, a headphone jack, a nice big touchscreen, and I’m told it also has a phone.

Typing on the on-screen keyboard has proven easier than I anticipated – even with my fat fingers, I can peck out a short e-mail with reasonable ease. The predictive/corrective text is actually surprisingly helpful here.

The built-in e-mail client is OK, if a little basic. It wouldn’t send outgoing mail via my Exim 4 server over TLS (a TLS packet with unexpected length was received), but I suspect that’s Debian’s fault for insisting that GNUTLS actually, er, works. I ditched the default client in favour of K9-Mail, which boasts PGP integration and is much more customizable. Apart from a few niggles with the UI, it does the job very well.

The browser works much as one would expect – even sites without a mobile option are surprisingly usable – the screen is a decent resolution for its size, and the usual gestures for zoom work well.

The music player worked straight out of the box when I copied some MP3s over, and the supplied headphones aren’t too shabby.

The ability to read bar codes and search online for the cheapest available version of the barcoded thing has proved endlessly amusing.

Irssi Connectbot deserves special mention for making IRC on the go dead easy.

One of the main reasons for opting for an Android phone were the tethering capabilities – with a couple of well-chosen taps, the phone can share its 3G internet connection by turning itself into a WiFi hotspot. This is one of the operations that makes it a bit warm, but it’s very handy if you’ve got a bigger computer with you, but no internet.

Battery life isn’t too bad given the capabilities of the device – it lasts a heavy day’s usage, and charges over USB from almost anything. They also throw in a standard wall-socket adapter too.

Android claims some IPv6 support, which I’ll report back on when my home IPv6 is raised from the dead. I’ve also got a few ideas for app development, so watch this space!

update-grub not working on your Debian Squeeze domU?

We‘ve been having a spot of bother with update-grub not working on our Debian Xen guests since upgrading them to squeeze.

The symptom: update-grub (as run after the installation of a new kernel package) fails because it’s ‘unable to find [a] GRUB drive for /dev/sda2 – check your device.map’. This happens using both grub-legacy and the new grub-pc package.

You’ll probably have run into this if your guests were created using xen-tools.

Skipping over the related bugs, the long and short of it seems to be:

  • The way Xen pokes /dev/sda1 and /dev/sda2 (which are probably LVMs on your dom0) into your guests without a corresponding block device confuses grub
  • grub used to be dumb enough to ignore the mysterious presence of ‘partitions’ with no block device and carry on anyway
  • The versions in Debian Squeeze are just clever enough to be dangerous and fall over in this situation

Happily, we’ve worked out the fix. Grub is hard-coded to do the right thing in this situation if your disk devices are called ‘/dev/xvd[a-z]’, so you need to:

  1. Fix all references to ‘/dev/sda2′ to read ‘/dev/xvda’ (and /dev/sda1 to /dev/xvdb). In practice this means grub’s menu.lst and possibly your /etc/fstab if that isn’t by UUID
  2. Edit the /etc/xen/guest-name file on the dom0 to rename the partitions (change the ‘root’ specifier and the lines in the ‘disk’ list)
  3. Destroy (sudo xm destroy) your guest. Note that you must destroy and recreate it; rebooting won’t give Xen a chance to rename the devices.
  4. Bring it up again with xm create -c config-file
  5. Profit.

When Sixxs met BytemarkDNS

Edit: Those of you coming across this post by googling for ‘Sixxs rDNS’ should note that Bytemark’s DNS service isn’t free unless you already have a server with them, but there are a number of free alternatives out there.

With IPv4 address space exhaustion practically upon us, I decided it was high time my house got IPv6. This is quite easy to do even if your ISP, like ours, doesn’t support it natively. Get a Sixxs tunnel, apply for a subnet, set up radvd on the Linux box behind your sofa (you do have one, right?) and there you are.

Even Windows XP can be trivially prodded into IPv6ing itself up, and my router seems not to mangle the radvd broadcasts so even wireless clients can have v6 if they support it.

What really impressed me, though, was what happened when I wanted to set up reverse DNS for my new Sixxs subnet. I’ve been assigned 2a01:348:1af::/48 and the Sixxs page tells you that you’ll need your own DNS server to host the necessary records. I don’t run my own DNS; I use Bytemark’s content DNS service.

I wasn’t expecting Bytemark to support adding rDNS records for random IPv6 /64s, however as it turns out, once you work out what the TinyDNS file needs to look like:

# IPv6 rDNS authority for Sixxs subnet 2a01:348:1af::/48

.f.a.1.0.8.4.3.0.1.0.a.2.ip6.arpa::a.ns.bytemark.co.uk
.f.a.1.0.8.4.3.0.1.0.a.2.ip6.arpa::b.ns.bytemark.co.uk
.f.a.1.0.8.4.3.0.1.0.a.2.ip6.arpa::c.ns.bytemark.co.uk

# Machines in the first /64 (home network)
6router.ipv6.dnorth.net:2a01034801af00000000000000000001:8640

… it Just Works:

$ host 2a01:348:1af::1
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.f.a.1.0.8.4.3.0.1.0.a.2.ip6.arpa domain name pointer router.ipv6.dnorth.net.

Nice one, Bytemark!

2011 geek wishlist

In no particular order…

  • That Debian Squeeze will release in Q1 2011
  • That I’ll discover a J2EE Servlet Container which isn’t totally horrible
  • That some kind of reconciliation between the JCP and Apache will occur
  • That Java 7 won’t be delayed forever
  • That Python 3 will get adopted faster than PHP 5 or Java 1.5 did
  • That we’ll finally see some major UK ISPs offering IPv6

Leave a comment if you can see anything I’ve missed, and we’ll check on them all at the end of the year…

Converting Word documents to PDF with OpenOffice and Python

The problem

A word document (plain old .doc, not 2007) should be received by e-mail, fed to a script, turned into a PDF and published on a website.

At my disposal

My server running Debian ‘Lenny’, which does not have a display of any kind.

How hard can it be?

Harder than it should have been, as ever. Here are my steps:

# aptitude install python-uno xvfb openoffice.org-java-common openoffice.org-writer unoconv

You’ll note the inclusion of Xvfb there, because it turns out that “headless” mode in OpenOffice isn’t really headless at all. Sigh. Also sigh some more at the broken dependencies of the unoconv Debian package.

Now we can write our script to do the actual conversion. Shame it took twice as long as it should have…

Fun with CurrentCost

Five years after the cool kids first started jumping on the bandwagon, I’ve got myself a CurrentCost CC128 (Southern Electric send them to some customers for free, it seems – e.g. my granddad who didn’t want it).

So, with the addition of an eight quid data cable and the Linux box running in my lounge, may I present my electricity usage graphs. Bear in mind that these are (at the time of writing) for a five-bedroom house in central Oxford.

The parser for the XML output of the device I’m using is this one – just swap “COM20″ for “/dev/ttyUSB0″ in their testrun script and fix it to ignore empty lines read from the serial port, and you’re in business. I then hackdapted this RRDTool tutorial to plot the graphs.