Requirements:

• Booking of different spaces by different users
• Ability to generate a report of billable hours outstanding for a given user
• Ability to send invoices and reminders to users and internal staff
• Integration with Google Calendar, as that’s how we publish hiring timetables on the website at the moment

At the moment, the most promising free option out there seems to be MRBS, but I’d have to add the Google Calendar and invoicing integration myself.

Does anyone have any bright ideas? Let me know if so.

Vauxhall do however lose a point for not making the display scroll if the text is too wide, but this gives rise to a fun in-car game guessing the end of the track titles.

The problem: you want to set up a web/mail/whatever server programmatically (e.g. as part of some tests). This server wants to bind to port 80/25/whatever. Your first problem is that it can’t bind to these because you’re not running your tests as root (or as an administrator on Windows).

The lazy approach at this point is to hard-code a port number over 1024, which you don’t have to be privileged to bind to. But this all falls to bits if you want to run the same test simultaneously on the same machine, or you need lots of different ports during the course of one test.

At this point, you can reach for the port 0 trick: on both Windows and Linux, if you bind a socket to port 0, the kernel will assign it a free port number somewhere above 1024. Truly well-written software (e.g. Jetty) will not only let you configure it to bind to port 0, but will make it easy to parse its logs to obtain the actual port number it got assigned. Less helpful software (Tomcat) will let you configure it to bind to port 0, but print 0 in all its logs, never the actual number. And the majority of software just won’t let you put 0 as a port number in its configuration.

At this point (subject to a slight race condition), you can grab some port numbers yourself and feed them to whatever you’re trying to configure:

def findFreePorts(howMany=1):
    """Return a list of n free port numbers on localhost"""
    results = []
    sockets = []
    for x in range(howMany):
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.bind(('localhost', 0))
        # work out what the actual port number it's bound to is
        addr, port = s.getsockname()
        results.append(port)
        sockets.append(s)

    for s in sockets:
        s.close()

    return results

The above is written in Python, but it translates trivially to any programming language which knows what a socket is.

You’ll note that the correct way to get, say, five free port numbers is to call the above method once with 5 as its argument. If you wrote a simple method which just returned one number, there would be nothing to stop it returning the same number each time you called it (the Linux kernel is rather more helpful, and will usually hand out a different number to each port 0 request, but I wouldn’t  rely on this).

I’ve done my best to fix all the historical links pointing to svn.dnorth.net so they go to the files now on my Bitbucket account, but do get in touch if I’ve missed one.

This has all been working well for over  two years, but occasionally something has to change: a few months ago, we got rid of sender verify callouts, now widely considered abusive by SMTP server admins, and more recently we added support for tagging messages with headers to say if they passed or failed DKIM verification. And every time I make such a change, I worry that I might have inadvertently broken something. This server handles mail for 30 domains and 8 people, some of who rely on it to run businesses! Panic!

I usually end up reassuring myself by doing some ad-hoc testing by hand after reconfiguring the server. At the most basic level, whatever your SMTP server is, you can use netcat to have a conversation with it on port 25:

d@s:~$ nc localhost 25
220 mailserver.splice.org.uk ESMTP Exim 4.71 Sat, 17 Mar 2012 09:51:20 +0000
HELO localhost
250 mailserver.splice.org.uk Hello localhost [127.0.0.1]
MAIL FROM: <>
250 OK
RCPT TO: someaddress@dnorth.net
550-Callout verification failed:
550 550 Unrouteable address
QUIT
221 mailserver.splice.org.uk closing connection

And there, I’ve just convinced myself that one of our features is still working: the mailserver should call forward to the final destination for mail to @dnorth.net addresses to check the local part (‘someaddress’ in this case) is valid, and reject the message up-front if it’s not.

Exim also has a load of other toys you can take advantage of: say I want to check how mail to webmaster@dnorth.net is routed:

d@s:~$ exim4 -bt webmaster@dnorth.net
R: hubbed_hosts for dnorth.net
webmaster@dnorth.net
 router = hubbed_hosts, transport = remote_smtp
 host mx-internal.dnorth.net [192.0.2.100]

(IP addresses changed for example purposes, obviously)

And finally, there’s debug mode: you can run

exim4 -bhc <ip address>

to run a complete ‘fake’ SMTP session as though you were connecting from the given IP address. You can send messages, but they won’t actually go through, and exim prints a lot of debug output to give you a clue as to its inner workings as it decides how to route the message.

This is all very well, but a quick brainstorming session gives a list of over 30 things I might want to check about my mailserver:

• Basic check that mail is accepted to our domains
• Only existent addresses on our domains should have mail accepted
• Domains with SMTP-time spam rejection on should have spam rejected
• Same for viruses
• Same for greylisting
• …

Testing all these by hand isn’t going to fly, so what tools can we find for automating it? A bit of Googling turns up swaks, which looks quite handy, but suffers from two drawbacks for me: first, it’s a bit low-level, and a collection of scripts calling it will be a bit difficult to read and maintain for testing all 30 of my assertions. Second, it really sends the e-mails in the success case, and I don’t want my users to get test messages or have to set up aliases for receiving and discarding them. swaks will definitely become my tool of choice for ad-hoc testing in future, but meanwhile…

The other promising Google result is Test::MTA::Exim4, which is a Perl wrapper for testing an exim config file. However, a few problems: (1) it’s Perl, and I Don’t Do Perl. (2), it’s limited to testing the routing of addresses, so it’s not going to cut it for checking spam rejection etc.

Having at least pretended not to be suffering from NIH syndrome, let’s spec out a fantasy system for doing what I want: I would like to be able to write some nice high-level tests in my favourite language, Python, which look a bit like this

class HubbedDomainTests(EximTestCase):
    """
    Tests for domains our server acts as the 'proxy MX' for, doing
    scanning etc before forwarding the mail to the destination machine
    """

    def testProxiedMailAccepted(self):
        """Proxied mail should be accepted"""
        session = self.newSession()
        session.mailFrom('dtnorthie@gmail.com').rcptTo('webmaster@dnorth.net').randomData()

    def testLocalPartsVerifiedWithDestinationMachine(self):
        """Local parts should be verified with the destination machine"""
        session = self.newSession()
        session.mailFrom('dtnorthie@gmail.com').assertRcptToRejected('doesnotexist@dnorth.net')

I could then run these in the usual manner for Python unit tests, and lastly, I want them backed by an exim4 -bhc session so that they’re as realistic as possible without actually sending messages.

This post is long enough already, so I’ll cut to the chase and say that I’ve made a start on writing it, and you can find out more at Bitbucket. In a follow-up post, I’ll talk about how it was done.

For the last couple of years, I’ve used scponly (this guide) to achieve roughly the right effect, but having an essentally unmaintained chroot on my box slowly collecting security vulnerabilities felt wrong. Surely it must be possible to provide secure FTP to users without using SSH at all, and without having to maintain a chroot?

Indeeed it is. ProFTPD is a well-recognised FTP server and has a handy SFTP module, and both are conveniently packaged for Debian:

# apt-get install proftpd-basic

The documentation is pretty good; it enabled me to arrive at the config below (suck it into the main one using an include for ease of maintenance) with just the one diversion to work out why my WinSCP wouldn’t talk to it (see the protocol switching line below). In WinSCP’s defence, I am using a pretty ancient version.

# Use SFTP with the same keys as SSH
# http://www.proftpd.org/docs/contrib/mod_sftp.html
SFTPEngine on
SFTPLog /var/log/proftpd/sftp.log

SFTPHostKey /etc/ssh/ssh_host_rsa_key
SFTPHostKey /etc/ssh/ssh_host_dsa_key

# Enable compression
SFTPCompression delayed

# Workaround for WinSCP bug: http://winscp.net/forum/viewtopic.php?t=8121
SFTPClientMatch ".*WinSCP.*" sftpProtocolVersion 4

# Allow the same number of authentication attempts as OpenSSH.
#
# It is recommended that you explicitly configure MaxLoginAttempts
# for your SSH2/SFTP instance to be higher than the normal
# MaxLoginAttempts value for FTP, as there are more ways to authenticate
# using SSH2.
MaxLoginAttempts 6

# Only allow specifically whitelisted users (members of the ftp group)
# http://www.proftpd.org/docs/howto/Limit.html
<Limit LOGIN>
 AllowGroup ftp
 DenyAll
</Limit>

Just make sure your users are in the ftp group, but not able to log in through SSHD.

Of course, with either solution you still have to worry about scripts and PHP executed by your user’s website being able to see the full filesystem of the machine, but mod_chroot and mod_suexec for Apache are both well documented and also Debian packaged.

So on Debian, you can apt-get install xinetd and use the config above to forward arbitrary ports (there’s also a ‘bind’ directive to specify the IP to bind to on the forwarding machine).

CardiffC&MFinance@centrica.com

Yes, that really is an ampersand in the local part. Sufficiently unusual that trying to send to it upset my default-configured installation of Exim:

rejected RCPT <CardiffC&MFinance@centrica.com>: restricted characters in address

You can tone down the (perfectly reasonable) check for these iffy characters by exempting centrica.com from it: edit /etc/exim4/conf.d/acl/30_exim4-config_check_rcpt and edit the domains line of the second ‘restricted characters in address’ ACL to read:

domains = !+local_domains : !centrica.com

Happily I think I’m now back to normal, and I have a few interesting things to write about, including some pretty graphs from the new church heating system, and some ideas for the Raspberry Pi when I finally get my hands on one. Oh, and last weekend, I bought a car, so I’m back on four wheels after an eleven month absence.

Watch this space!

* Don’t drink and cycle, kids. And don’t waste any sympathy on me.

This is something British Gas’s man clearly failed to do when fixing my grandparents’ central heating recently. I don’t know the full details of the problem, but I do know that their thermostat was broken, so he installed a new one.

Fair enough, but it turns out that the iron march of progress has changed a thermostat from a knob with some numbers on it to something ‘smart’:

And how does the new wireless thermostat make life worse for my nonagenarian grandparents? Let us count the ways:

• Because it’s wireless, it has batteries in it which need replacing every so often. This is achieved by opening a flimsy plastic door on the bottom of the unit which is fiddly to access once it’s wall mounted, then scrabbling on the carpet as the batteries fall to earth. It also means the thermostat will mysteriously stop working once every n months until someone younger sorts it out for them, since there’s no way they’ll hear a low-battery beep or spot an indicator on the screen.
• Since it’s superglued to the wall just inches away from the hot water tank it controls, the only advantage of wirelessness is to save the drilling of one hole and the running of a six-inch bit of cabling – and even these could presumably have been avoided by replacing the original thermostat instead of leaving it screwed to the wall but not doing anything.
• Instead of reading the numbers round a knob, you see them on an LCD display which is not backlit and not very big, thus making it perfect for people with poor eyesight to see in a not-very-well-lit hallway.
• Pressing the middle of it resets it to a pre-programmed ‘preset temperature’ (“ideal for the poorly sighted”, the manual claims with no sense of irony) – an unnecessary recipe for confusion if you knock the middle by mistake
• It doesn’t go ‘click’ as it passes the current room temperature like an electromechanical thermostat would, so you have to read the screen instead
• By default the display shows the current room temperature, meaning you can’t tell without adjusting the knob what temperature the thermostat is currently set at

Somewhat more subjectively, I think it’s more likely to malfunction than an electromechanical device with two moving parts, and presumably it has to fight for spectrum with all surrounding cordless phones, WiFi units and garage door openers – let’s hope the base station does something sensible in the face of losing contact with the unit.

Well done, lads. Another triumph of engineering.