Category Archives: SysAdmin

How hard can it be?

Sometimes, I wonder if it’s me getting old or large corporations failing to shut up and take my money. Or in this case, somebody else’s money.

Although this is my last year “doing the money” for St Columba’s, I’m still one of an elite handful of people who “work in IT” and thus do all the geeky stuff. And our new minister moving into the manse should have been a chance to enjoy spending the church’s money on a stack of equipment (laptop, mobile phone, etc.) and sorting out an internet connection.

Oh dear, sorting out an internet connection. Here we go again. The default option was Virgin Media: after all, we already have one site with them, so no need for a tedious credit check and a load of faff, just call their sales team (very efficient) and get given a date for a site survey. That gets done, though annoyingly the team they send takes about five minutes to say “we’ll need to run a duct, somebody else does that” and buzz off.

They then fail totally to turn up to do the digging, blame it on “our landlords” (a likely story since we own the house outright – though it’s possible some tedious neighbour complained to the management company about digging up a shared driveway). They claimed it could be sorted, I decided I wanted it fixed before I turned 40, and moved on.

OK then, let’s use the BT phone line already in place. Zen are a supplier I know and trust in situations like this (i.e.: not interested in switching every 18 months to get a good deal, just want good service at a reasonable and stable price). Unfortunately, entering the post code on their site results in a blank grey page. And their social media team seem completely incapable of getting that sorted.

Finally, third time lucky, IDNet got the business, and (aside from some questionable non-default setting choices on their router), seem to have done everything right. I was particularly pleased that their team e-mailed me after I put in an order based on postcode, saying “there’s a stopped line in the property – want us to re-start it?” and giving me the number to confirm using 17070. It’s been up and running for 24 hours so far, so let’s see how it goes…

Bytemark bought by iomart

And so, after sixteen years, Bytemark has been bought out. In common with rather a lot of other customers (if Twitter is anything to go by), I was a bit saddened to hear about this via The Register rather than an announcement. I don’t blame the owners in the slightest – they have every right to cash in on their hard work after sixteen years. And whilst the construction of their own data centre undoubtedly gave them a cost edge over the long term, it no doubt needed to be paid for first.

Unlike other customers, I’m not going to idealise the company’s previous state – my nine year happy relationship with them has been based on our dedicated server Just Working for the most part, and me never needing to contact support. On the rare occasions when I have, it’s been a mixed experience. Such as the time when my query about adding a .mx domain name to their DNS service got a keyword based response: “your MX records look OK to me”. Or even better, the occasion when they managed to e-mail me another customer’s control panel password by accident.

I won’t be making any sudden moves, and if the founders are to be believed, neither will Bytemark.

I hope USB-C catches on

A small connector for a lot of power and data

New laptop, new docking station. But not the proprietary plastic tray of olden times; we’ve been trying out these new Anker USB-C mini-docks at work. I’m really impressed by mine: one cable to plug in when I get to my desk, and I have power and networking, an external monitor and mouse, and an SD card reader. Oh, and a USB port left over which charges my phone even when the laptop isn’t on the dock.

It does get properly toasty when the full set of ports are in use, but that’s not entirely a surprise.

The other nice thing about these are that they can be used with recent Macbooks as well as PC laptops, and if the interface lasts, they’ll be re-usable for the next generation of laptops too rather than tied to one model.

Lenovo T470s

Do your keys glow in the dark? Mine do.

I’ve just been issued with one of these as my new work laptop. So far, I have to say I’m impressed – it solves the poor screen resolution of the X230 I had for many years before it, weighs impressively little, doesn’t get hot during light to medium use; generally works.

Its feet don’t grip the desk well enough – they seem to be felt rather than rubber – but I’m sure I can MacGyver my way round that…

More to follow on how I find using Windows as my primary OS for the first time in five years!

SSH rate limiting vs check_by_ssh – oops

Now that I’ve once again got reliable notifications from my monitoring system, being woken up by them at 6.30AM on my first day off for the Christmas break needed fixing. My Nagios has sent out lots of “spurious” alerts for some time, and my hope was that getting woken up by them would motivate me to fix it. I’m pleased to say that it has!

In addition to checking various machines respond to ping, my system also logs into them using check_by_ssh and checks various things by running commands locally. Sometimes the entire block of SSH-based checks for a server would flip over to CRITICAL with “connection timed out”, even though the machine remained up and running. There was no evidence of a high load average to explain the timeouts, and a bit of checking with netcat revealed that connections to port 22 on the machine in question really did time out from the monitoring machine (but not from anywhere else).

At this point, I had a lightbulb moment and remembered that our firewalls automatically block SSH connections from any IP address which attempts more than 10 in a 60 second period. This crude rate limiting is one of many lines of defence against brute-force attacks, but of course, some hosts have more than 10 checks run over SSH. And the way Nagios runs means that quite often, it hits the rate limit, then continues to do so as it re-tries the checks one after another. The backoff it performs doesn’t help, because it backs off the retry interval in lock-step for all 10 checks.

Having added an exemption to the firewall rate limiting for our monitoring server’s IPs, all is now well in Nagios, and hopefully the only rude awakenings from the alerts will now be genuine outages.

(And it only took me five years to find time to get to the bottom of this intermittent problem!)

Pushover (Android push notifications)

I’ve been looking for a new means of getting alerts from my Nagios (server monitoring) for some time. Previously, I used SMS notifications from Nagios, but at upwards of 5p per credit, given the way a flakey link can result in 10 or 20 notifications in a short space of time, this was starting to get quite expensive. It also doesn’t fit very well with the fact that SMS can cost more to receive abroad and, generally, I spend more time in places with a reliable WiFi connection that with a good mobile signal – especially when visiting family at Christmas.

What would seem appropriate here, then, is some kind of push notification service working over data rather than SMS. Notify My Android doesn’t seem as well regarded these days, but Pushover looks really good. I particularly like that you’re provided with an e-mail address which you can just plug in to things like Nagios to have e-mails turn up as push notifications, without having to script their API.

I’m 24 hours into their 7 day trial, and so far so good. Their app is nicely featured including mute/unmute of sound and vibrate notifications, and quiet hours.

Porting a mobile number to AAISP

Just look at my custom network name

St Columba’s has had the same 07 mobile number for a long time, and consequently it’s printed on various signage and readily available online. Traditionally, it’s been assigned to a pay as you go SIM card in a creaky old Nokia phone carried by our facilities manager. However, this approach has its limitations: when they’re on holiday, they have to physically pass the handset to someone else to handle the calls, and it leaves the phone number (arguably an important asset in itself) attached to a pay as you go SIM which might have an employee’s name on it rather than the church’s.

Issuing a new smartphone seemed like the opportune moment to sort this out. I did some digging into business mobile contracts – having the number on one of those would sort the ownership – but none of them seemed to feature call divert which could be configured without having possession of the phone. So I turned to a service I’ve written about here before. In addition to being able to buy 07 numbers on VOIP, AAISP will let you port in existing numbers using a PAC code. Coupled with a Sip2Sim SIM card in the phone, this neatly de-couples the number from where the calls get sent to and gives us the divert via a web page and all the other bells and whistles.

The ordering process was nice and simple and fully automated – I punched the PAC into the order form at A&A, and the number moved over within a couple of working days (it got briefly stuck in the middle, but A&A fixed it out of hours when I asked on IRC – try getting that level of service from a big company). Billing is all via direct debit, and no messing around with paper invoices, which suits us fine.

As the icing on the cake, I calculate that for the volume of calls and data usage of this particular device, it will work out cheaper than any of the mainstream corporate mobile deals I saw. I’ll report back on that.

Virgin Media

We got a Virgin Media (business) line in the church office recently, and although it was quite unstable for the first few weeks, it has now settled down nicely. I can run my monitoring on the end of it without getting lots of false alarms:

vmedia

Colour me impressed, especially for what we’re paying.

URC e-mail fail

I don’t know exactly who my church has outsourced @urc.org.uk e-mail to, but whoever it is clearly hasn’t read RFC822 (“SMTP systems are expected to make every reasonable effort to accept mail directed to Postmaster from any other system on the Internet.”)…

fail

Rather difficult to complain about the original broken address I found…

Update: They did get back to me via Facebook eventually. Still not had an answer on why postmaster@ doesn’t work, though.