Monthly Archives: July 2015

When my beeper goes off

As regular readers will know, I run a co-located server with a few friends. I’m also responsible for things like the WiFi network at church. Inevitably, these things run just fine for weeks, months or even years at a time – and then fail. The first I hear of it is usually from a would-be user, sometimes days later.

The world of open-source has a solution to this, and it’s called Nagios (actually, I think the cool kids have jumped ship to the more recent Icinga, but I haven’t played with that). It’s a monitoring system – the idea is that you install it somewhere, tell it what computers / devices / websites to monitor, then it can e-mail you or do something else to alert you when things fail or go offline.

The really nice thing about it is that it’s all pluggable – you can write your own checks as scripts, and also provide your own notification commands, for example to send you a text.

The only problem is, where to put it? Hosting it on the co-lo box I’m trying to monitor is a bit self-defeating. Fortunately this is a rather good use for that Raspberry Pi I had lying around – I found a convenient internet link to plug it into, removed all the GUI packages, and it’s running Nagios just fine. So far, without really even trying, I’ve got it monitoring 76 services across 29 hosts. Some of the checks are a bit random – e.g. a once-per-day check on whether any of my domain names have expired – but now it’s possible to buy ten-year renewals it’s nice to have an eye on this sort of thing, as who knows if one’s registrar will still have the right contact details to send reminders in a decade?*

So far it’s working a treat, texting me about problems so I can fix them before users notice, although if I add many more checks I suspect I’ll have found an excuse to invest in a Raspberry Pi 2 to keep up with the load.

If you’re doing it right, you’ll configure Nagios to know which routers and computers stand between it and the rest of the internet (and hence what it’s trying to monitor). This means it won’t generate a slew of alerts if its internet connection dies, just one. It also means you get a useful map of what you’re monitoring:

* I’m pretty sure Mythic will do, because they’re great, but it never hurts to have a backup in place.

Windows and SSDs

So, having had such good results with the SSD I put in my ageing desktop, I decided to get one for my ageing laptop too. I did the copying a bit differently – since the SSD was 50GB smaller than the spinning rust it was replacing, a straight dd wasn’t going to work. In any case, it’s a bit boring waiting hours for blank space to be copied across. There was only 30GB of actual data.

I connected both drives to another machine – I couldn’t get hold of a SATA caddy to have them both on the laptop – and again, booted to SystemRescueCD. This time, though, I used GParted to copy-paste the boot partition, made an empty NTFS “main” partition and used rsync to copy the contents of the old C: drive to the new. Much faster to just copy the data. Some quirk of ntfs-3g or rsync seems to have lost the “hidden” attribute from some files, but otherwise it all worked.

You’ll have realised from my mention of NTFS that this laptop runs Windows. The purists may call me a traitor to Linux, but having at least one Windows box around is useful for all sorts of things, and this is mine.

The final annoyance was that Windows refused to boot from the new drive:

A required device is inaccessible

It did however prompt me to boot from the Windows CD and hit repair, which worked. Given how rapidly it worked, I can only assume Microsoft write the UUID of the drive to the boot sector or some such, as a tedious anti-piracy measure.

The SSD magic definitely seems to have made this laptop usable again – previously it was slow to the point of being unusable and you could hear the disk crunching inside it. Hopefully the SSD should give battery life a slight boost and be more robust against being dropped or jolted too.

Since I’m now all SSD’d up, I’ve got my desktop running DBAN to erase the old mechanical hard disks I’ve replaced. Did I mention how handy having a network boot server is?

Down the rabbit hole: OAuth, service accounts and Google Apps

I have quite a few small programs which use Google’s APIs for one thing or another:

  • Updating pages on Google Sites automatically
  • Reading a Google Docs spreadsheet and sending SMS reminders for a rota inside it
  • Reading a Google Calendar

Until recently, reading a public Google Calendar didn’t require authentication – one could simply consume the XML from the calendar’s URL and work with it using the GData API. Google knocked that on the head at some stage. Shortly afterwards, I woke up to a slew of error e-mails marking the apocalypse.

The apocalypse is how I refer to the day when Google finally removed the ability for scripts like mine to authenticate by presenting a username and password. For sure, hard-coding these into a script is not good practice, but it worked well for many years, and was a lot simpler and better documented than the alternative…


Much has been written elsewhere about OAuth, but the main problem I had was that Google’s examples all seemed to centre around the idea of bouncing a user in their web browser to a prompt to authorize your use of their data. This is all very well for interactive web applications (and, indeed, much better than asking users to trust you with their password), but where does it leave my non-interactive scripts?

I eventually dug up the documentation on service accounts. The magic few lines of Python are these:

from oauth2client.client import SignedJwtAssertionCredentials
json_key = json.load(open('pkey.json'))
scope = ['']
credentials = SignedJwtAssertionCredentials(json_key['client_email'], json_key['private_key'], scope)

The JSON for the key can be obtained via the Google APIs developer console. Most Google APIs, and things built on top of them, can then take the credentials object, and authenticate as a service account. Apparently you can also do really clever things like impersonating arbitrary users in your Google Apps domain – for example, to send calendar invites as them – but that’s for another day.

Solid State Disks

For my less geeky readers, a Solid State Disk (SSD) is similar to the mechanical “hard disk” which has traditionally been the storage in most PCs and laptops. However, an SSD has no moving parts and works entirely off memory chips – a bit like a USB memory stick. The big advantage of them is they’re a lot faster to read data from than a mechanical hard disk.

SSDs have been standard issue at work for a while now, but I hadn’t yet had occasion to buy one myself. I was pleasantly surprised by how much of a price collapse had taken place – I picked up a 250GB SSD from Ebuyer for £64. The use-case was my home desktop PC, which was glacially slow at resuming from hibernate, and struggled to run Windows under VirtualBox. A Core2 Duo should have no problem with this from the CPU side, but it felt like I/O performance was the problem.

I was surprised by just how small and light the SSD was – of course, with no moving parts or motors to spin the platters, it’s about half the size and a tenth the weight of the equivalent mechanical disk.

Out with the old, in with the new

Out with the old, in with the new

Since the SSD was the same size (250 marketing gigabytes, which is to say 232.9 actual gigabytes) as the “spinning rust” it was replacing, I simply connected it, booted into SystemRescueCD from my network boot server, and used dd to copy the block device of the old disk over onto the new one. It looked like it was going to take about four hours, so I issued a shutdown for five hours hence and went out for the day.

Having now had a chance to try it out, I’m properly impressed – the machine boots and resumes much faster and Windows under VirtualBox is now snappy enough to be usable. At that sort of price, I shall have to see about getting one for my laptop too.

Update: some good discussion on social media. The downsides of SSDs are pointed out, e.g. limited write capacity (if you were to write to this one continuously for 2.5 days, you’d wear it out), and that they aren’t suitable for archiving as power-off data retention can be limited to months. None of this matters for my use-case, but enterprise-grade SSDs with enterprise-grade price tags also exist to try and solve at least the write lifetime issue.

Hive active heating

I had the horrible old boiler in my flat replaced recently, with British Gas doing the work. As part of it, they installed a Hive smart thermostat. This has a few blingsome features, but the main reason I wanted it was to gain remote control of my heating from an app on my phone. Living alone and having irregular comings and goings, this ability could be quite a nice money-saver (or, a means to come home to a warm flat rather than a cold one). I can’t say much about it yet, having had it installed in the hottest week of the year with no prospect of needing central heating until October.

However, a few bits and pieces do come to mind:

  • My version of the Hive “hub” (the bit that plugs into your router to do the internet access) is powered off a mains cable which ends in a USB plug – although they supplied it with a normal UK plug on the end of that, it seems quite happy to take its power from the USB port on the back of my router, which saves taking up a socket.
  • They clearly expect you to configure via the Android or iOS app, not the website. For example, I can’t find a button on the website to copy a day’s schedule from one day to the next, whereas the app has that (as does even the nastiest digital heating timer from the 90s).
  • The ability to specify different temperatures for different time periods is kinda neat.
  • The alerts are a bit dumb. I’ve had several e-mails this week warning me that my flat is exceeding 25°C. However, given that it’s the hottest week of the year and the heating hasn’t been on for a month, this is not terribly useful. What really grates about this is that their app has an icon showing the weather and external temperature – but they’re obviously not making much use of this information.
  • The geofencing feature is a disappointment. I want it to turn the heating on and off, not just alert me, since I’m a one-person household and usually on my bike and unable to respond to the alerts when travelling. However, despite the lack of an official API, they do have control by text message. So perhaps when winter sets in I’ll write an Android app to fill in the missing features. Watch this space…

Thunderbird Autoconfiguration

If you’re a diehard like me who still runs their own e-mail server – perhaps for a few friends and family as well as yourself – you might find Thunderbird’s autoconfiguration useful. Even if you only set up Thunderbird on a new machine once a year, you’ll be into a net saving of time the second year. And it’s especially useful if you have to talk a less technically-minded user through the setup, because all they need is their e-mail address and password.

You simply write an XML file in this format, and expose it at Thunderbird should then proudly announce “Configuration found at ISP”. Sorted.