Category: SysAdmin


SysAdmin stuff

It’s amazing how many fewer afternoons I seem to spend hacking around on my servers these days. Perhaps I got a life; I certainly got a full-time job. I have however sorted a few long-standing bits and pieces out today…

dnorth.net is now available over IPv6

As are its various satelite sites and www.saintcolumbas.org. Sorry, no, there is no bouncing logo to reward those of you viewing them via such.

A backup system that doesn’t Totally Suck

I’ve finally retired my creaking “run a shell script to rsync them onto my laptop when I remember (i.e. every six months)” manaul backup system in favour of an encrypted LVM partition on my home server, and rdiff-backup to make nice incremental backups of everything on a nightly basis. The instructions on how to do it are all out there on the interweb, and it’s not too difficult, fortunately. I’m a bit disappointed that backupninja doesn’t support remote rdiff-backup, but I guess I should submit a patch if it bothers me that much…meanwhile, my wrapper script seems to work just fine.

mod_wsgi delivers on the promise

It’s been over a year since I deployed Django in production, and I wasn’t looking forward to it. Last time, I had a lot of trouble with mod_python, sessions and decimal objects refuising to pickle.

Thankfully, all this really seems to have grown up in the last year – mod_wsgi is now the recommended way of deploying Django in production, and following the mod_wsgi django instructions, I was in business in 20 minutes. No fuss, no mess, no drama, and best of all, using daemon mode, no noticeable performance hit when serving static files and PHP off the same Apache installation. The ability to run the django project as its own unprivileged user when using daemon mode is also real handy.

2010 will be a bad year for IPv4

2010 will be a bad year for IPv4 – this is exactly why I designated native IPv6 support as fundamental, not merely a “nice-to-have”, when setting up Splice last July. Hats of to Bytemark for supplying IPv6 with their hosting. I’m sure the fact that it’s excluded from their SLA is something that won’t be the case in eighteen months’ time, and meanwhile, I and the rest of the crew are using it quite happily.

Footnote: those of you reaching for the comments button to sarcastically remark that this website appears to not be available over IPv6 will be pleased to hear that I intend to fix this in the immediate future.

A narrow escape

Let’s be honest about this. Looking back, I should have known better. Nevertheless, as I describe the problem that ate far too many hours of my weekend, judge for yourself whether I was entirely to blame for How It Went…

The problem

A friend of mine has a laptop. It’s about five years old, and it runs Windows XP. This means, inevitably, that it’s a mess. My personal metric of measuring how rodgered a machine is by the number of icons in its system tray gave it a ten, and that’s pretty nasty. Nevertheless, until last Friday, there was nothing wrong with it that an uninstall fest followed by a defrag wouldn’t have fixed.

Enter the Internet Man

Last Friday, a chap called round to set up some broadband for my friend, who’d previously been on dial-up. I’m not going to name the ISP concerned, since it gives me more freedom to say nasty things about them, but suffice to say that they’re big enough that they really should have done better.

Their engineer fixed the simple(ish) problem with wiring which was stopping the broadband from working, and what he should have done at that point was connect the laptop to the router (less than six inches away on the same desk) using the supplied ethernet cable, and left. What he actually did was shove the supplied CD in the drive, which helpfully installed a few hundred megabytes of crapware onto the machine, then hooked it up to the broadband via wireless. Then he left.

Despite the above totally unnecessary shoving of stuff onto it, the laptop struggled manfully on (system tray count now up to 12) and seemed superficially fine.

Enter Dragon Naturally Speaking

My friend makes extensive use of Dragon NaturallySpeaking, a voice-recognition product which seems to knock the socks off everything else on the market when it comes to actually coping with different accents. And it was here that the problem first manifested itself: trying to use Dragon to voice-control Internet Explorer caused it to crash with an error message along the lines of the one described in this Microsoft knowledgebase article.

Enter the sucker, stage left

At this point, I fetched up, and agreed to see if I could fix the problem. Although the above KB article looked ideal, being the first hit on Google when I exercised the too-useful-to-be-documented Windows feature of “Ctrl-C copies the text of the active dialog box to the clipboard”, the hotfix it supplies claimed to be already present in service pack 3 of Windows XP. Just about the only virtue of this laptop was that it was fully up to date on patches and service packs, so what now?

Don’t press that button

Being rather short on ideas at this point, I decided to fire up the nearest thing broken Win32 boxes have to a magic bullet, namely System Restore. The machine refused to roll back to any of the restore points at first, but restarting into safe mode fixed that, and it was soon rolled back to the Friday, at a time before the problem occured.

Unfortunately, Dragon now seemed completely broken, giving the error message described in this support article. And no, of course there were no backups, I hadn’t taken one before I started, and my friend is no different to most of the non-geeky people I know in not backing up, except for dragging his holiday snaps onto CD once every six months.

To their credit (and they’re about the only players in this story to be awarded any), Nuance’s suggestion in the article of how to manually restore the user files for Dragon did work, after I rolled back the fatal system restore [or rather, didn't, because it didn't seem to have made a pre-restore restore point. Fortunately picking one from the Saturday seemed to work].

So, after two hours feeling my friend’s anxiety at the thought of having to retrain the speech recog from scratch – not a pleasant accompinement to the sick, swoopy feeling we get when we know we’ve just permanently erased some irreplaceable data – we were back at square one with the original problem. One last shot in  the dark, disabling the ISP’s nasty extensions to IE, seemed to fix the issue.

So who do we blame here? Laptop vendors, for selling machines so laden with rubbish before they even leave the factory that DLL hell seems assured the minute they meet with real life usage? Or, we could blame the idiots who seem to write the nasty unsigned drivers for most hardware on the market. We can definitely blame the ISP’s engineer for installing the crapware, but perhaps he’d been trained to, and anyway, why do ISPs think we need a CD full of crap to supplement the TCP/IP standard that’s been around for several decades? Is it really asking too much of Johnny User to plug in a cable or enter some Wifi passwords in to the applet that’s sodding well supplied with Windows, thus making the poorly writtten replacements from laptop manufacturers and ISPs alike completely superfluous? We could also blame Microsoft for making system restore not clever enough to cope with software like Dragon. Or possibly blame Nuance for not fixing or documenting what has apparently been a known incompatability for several versions of Dragon*.

Certainly, we can blame me. I clearly need to have “I will not agree to even slightly ‘fix’ someone else’s computer without taking a full disk-image of it first” tattooed across my forehead. I also clearly need to reimmerse myself in the happy world of properly written software which I’m lucky enough to earn a living in and try to forget the horrors of the last 48 hours.

* The most useful reference I could find on Google was this page. The Nuance KB doesn’t mention system restore. Then again, perhaps their customers simply don’t know this is the cause of the issue, or don’t get round to reporting it. It’s not like I have.

On leaving Slicehost

Just over 18 months ago, I signed up for a 256slice from Slicehost to host this website, my email, etc. Later this week, I shall be shutting the machine down and cancelling my account with them.

For the record, this has nothing to do with the level of service I’ve received from them – I’ve always found their support team quick to respond and helpful, and their articles site and wiki are both very handy.

However, a combination of fluctuations of the pound against the dollar, a surge in demand for RAM by my applications and sites, and my getting increasingly fed up with transatlantic ping times of 130ms meant the machine was becoming unfit for purpose and overloaded.

Thus, I have now found a reasonably cheap way to bring “my stuff” home to hosting in the UK. About which, I will be writing more shortly!

So goodbye, Slicehost, and keep up the good work. I’d certainly recommend you to anyone who lives in America and needs a VM.

Windows XP loses its crown

And Ubuntu gets seven out of ten

Last week, it was clear my parents’ ageing XP box needed some attention – in fact, let’s be honest, it had been crying out for a zap-and-reload since last year.

This time, though, I was determined. With my sister safely moved off onto her new laptop, there should be no reason not to migrate the box to Ubuntu. The only things in use on it were Firefox, Thunderbird and Openoffice…so how hard could it be?

Five days on, I’m in a position to report that the answer is “harder than it should have been” – here’s what happened:

Problem 1: Audio CD autoplay

Dad wanted the machine to start playing his CDs automatically when he put them in, much like Windows does. Seemed perfectly reasonable to me, as I blithely assured him it’d be a matter of a few mouse clicks.

It wasn’t.

Much Googling and swearing later, this hack (fifth comment down) seems to do the necessary, though I had to bump the sleep interval to six seconds.

Problem 2: No matter how close KPatience comes, it’s not “real FreeCell”

…largely because, despite generating the same deals from the same numbers as the Windows version, it counts every loss of the same game number as a loss in the stats, whereas the Microsoft one counts them as one loss.

Fortunately, freecell.exe runs fine under wine, so copy it across and save yourself the bother.

In conclusion

Other than the above minor niggles, all seems well, and hopefully the machine should now do a few more years’ trouble-free duty.

So, as the title of this post says, Windows XP has now been deposed as my OS of choice for “real people”. Whoever would’a thought it?

Update, 18 July 2009

I forgot to say that probably the most impressive feature of the switch was that Ubuntu flawlessly detected and printed a test page on our trusty old Epson Stylus C62. That’s the sort of slickness we like to see.

Squeezing a quart of apps into a pint-pot server

Quick tip for anyone like me who’s foolish enough to cram Exim4, Spamassasin, Apache with WordPress running on it, and a pile of other stuff on to a server with 256MB of RAM: Install one of the WordPress caching plugins. Otherwise, your server is liable to start swapping frantically if you get a dozen or so hits on one of the WordPress-powered sites.

Indeed, it was possible to nuke jasper just by mashing the F5 key whilst sitting on the front page of this site.

Let’s be honest, shall we: if you’re reading this, you’re probably the sort of person who’s been where I was last week: on the recieving end of a phone call from a friend/relative/friend of a friend/friend’s relative, enquiring if you could help them with a “computer problem”…

In this case, it all seemed pretty simple: the chap on the other end of the phone was rightly concerned that his second-hand Windows XP box had come without any antivirus software installed.

“No problem”, I blithely replied, “just install the free edition of AVG“.

And therein began the difficulty: no matter how carefully I spelled it out, the chap insisted that visiting the URL for AVG’s site resulted in a 404 error. Other sites were visible, but not that one. Emailing the link produced the same result, and so, thoroughly puzzled, I agreed to a site visit the next day.

Sure enough, from both IE and Firefox, the machine would not display pages from avg.com. Further experimentation revealed that most of the other antivirus vendors’ sites were similarly blocked, along with microsoft.com.

This rang faint bells with me – viruses have been blocking such sites for years – and full of confidence that I was going to be able to pull off a quick fix and look like a genius, I pulled up the C:\Windows\System32\Drivers\etc\hosts file, expecting to find a bunch of domains being redirected to 127.0.0.1. What I actually found was nothing which shouldn’t have been there.

So then I broke out the usual tools: Spybot, Rootkit Revealer, a manual installation of AVG via disk from another machine, HijackThis … several hours of the machine wrongly being pronounced clean later, I remembered reading that Microsoft’s OncCare sometimes managed to disinfect machines other products didn’t.

I downloaded the free trial (via another machine as the Microsoft site was being blocked) and fired it up. Its first quick scan found nothing, but it also recommended installing Internet Explorer 7. I think it must have been the running of the Microsoft malicious software removal tool during the IE install which finally disinfected the machine to the point where a full OneCare scan found and removed the last of what it proclaimed to be the Conficker worm.

More Windows-savvy readers might well be saying at this point that I should have recognised the Conficker symptoms earlier. The truth is, though, that after nearly three years using Linux on my machines, it’s starting to grate to have to find my way around Windows and fix knackered installs thereof, and I’ve been cheerfully avoiding articles about the latest threats in the Windows world, smug in the knowledge that it’s not my problem any more. .. until the phone rings.

It turns out that the aspect of the infection which had me confused the most – the blocking of the AV sites – is achieved by hooking into the relevant windows API calls and tampering with their behaviour. The question has to be asked, though – why the hell is it possible for code running as a limited user – or, come to that, any user – to intercept API calls like this?

Asside from the still-woeful state of its security, Windows really does feel prehistoric to me now – no central mechanism for installing and updating software, leading to every app having its own updater (launched at startup for maximum slowdown of the machine, of course), no decent command-line, and all the nonsense of antivirus, antispyware and anti-everything-else.

It must be ten years since I started getting the phone calls from people with broken machines, and yet it’s still happening. I saw the future of home computing last week, and it was a broken Windows box. Time to think the unthinkable and start installing Ubuntu as the standard remedy?

VPNC mysteriously broken in Intrepid

Edit: As Michael points out in the comments, Oxford users can avoid the mess below by simply disabling hybrid mode.

So there I am, firing up the Advent to read my email between lectures when…

$ sudo vpn-connect oxford
vpnc-connect was built without openssl: Can't do hybrid or cert mode.

Gah? The VPN client has always worked in the past, but following the upgrade to Ubuntu Intrepid, it doesn’t. A quick search lead to a solution which can be summarised as:

$ cd /tmp
$ apt-get source vpnc
$ cd vpnc-0.5.1r275
$ $EDITOR Makefile

Uncomment the lines beginning with OPENSSL
Read this and despair for the future of open source software

$ sudo apt-get build-dep vpnc
$ dpkg-buildpkg

Wonder why it all collapses in a heap
Search and find this.

$ apt-get install libssl-dev
$ dpkg-buildpkg
$ cd ..
$ sudo dpkg -i vpnc_0.5.1r275-1ubuntu1_i386.deb

Wonder why this has changed between Hardy and Intrepid
Seriously consider going back to Windows on this machine.

Sigh.

Tightening up BIND9

I’ve been aware for some time that my DNS isn’t quite as securely configured as I’d like. http://crashrecovery.org/named/ looks pretty good, but the two main issues bugging me were:

  1. Anyone could do a ‘dig @ns.dnorth.net dnorth.net AXFR’ to retrieve a listing of all my DNS records – not great from a security point of view. This is a capability that should only be turned on for secondary DNS servers which need to fetch from the master.
  2. The server would perform arbitrary lookups [for any domain] on request. This means it’s operating in ‘recursive mode’, which is a Bad Thing for various reasons.

The solutions were:

  1. Add “allow-transfer { “slaves”; };” (without the double quotes) to the section of the configuration beginning “zone ‘dnorth.net’”. Then add a section defining the “slaves” access control list to be the local server, plus the secondaries: “acl slaves { 127.0.0.1; 123.45.67.89; }” replacing 123.45.67.89 by the IP address(es) of your secondary nameserver(s).
  2. Add “recursion: no;” to the “options” section of the configuration.

Then restart the BIND9 service – on Debian, this is “/etc/init.d/bind9 restart”.

Health warning: Don’t do (2) above if you rely on your server to do its own DNS resolution – follow the crashrecovery tutorial above instead.

Powered by WordPress | Theme: Motion by 85ideas.

Bad Behavior has blocked 61 access attempts in the last 7 days.