Category Archives: SysAdmin

Lenovo T470s

Do your keys glow in the dark? Mine do.

I’ve just been issued with one of these as my new work laptop. So far, I have to say I’m impressed – it solves the poor screen resolution of the X230 I had for many years before it, weighs impressively little, doesn’t get hot during light to medium use; generally works.

Its feet don’t grip the desk well enough – they seem to be felt rather than rubber – but I’m sure I can MacGyver my way round that…

More to follow on how I find using Windows as my primary OS for the first time in five years!

SSH rate limiting vs check_by_ssh – oops

Now that I’ve once again got reliable notifications from my monitoring system, being woken up by them at 6.30AM on my first day off for the Christmas break needed fixing. My Nagios has sent out lots of “spurious” alerts for some time, and my hope was that getting woken up by them would motivate me to fix it. I’m pleased to say that it has!

In addition to checking various machines respond to ping, my system also logs into them using check_by_ssh and checks various things by running commands locally. Sometimes the entire block of SSH-based checks for a server would flip over to CRITICAL with “connection timed out”, even though the machine remained up and running. There was no evidence of a high load average to explain the timeouts, and a bit of checking with netcat revealed that connections to port 22 on the machine in question really did time out from the monitoring machine (but not from anywhere else).

At this point, I had a lightbulb moment and remembered that our firewalls automatically block SSH connections from any IP address which attempts more than 10 in a 60 second period. This crude rate limiting is one of many lines of defence against brute-force attacks, but of course, some hosts have more than 10 checks run over SSH. And the way Nagios runs means that quite often, it hits the rate limit, then continues to do so as it re-tries the checks one after another. The backoff it performs doesn’t help, because it backs off the retry interval in lock-step for all 10 checks.

Having added an exemption to the firewall rate limiting for our monitoring server’s IPs, all is now well in Nagios, and hopefully the only rude awakenings from the alerts will now be genuine outages.

(And it only took me five years to find time to get to the bottom of this intermittent problem!)

Pushover (Android push notifications)

I’ve been looking for a new means of getting alerts from my Nagios (server monitoring) for some time. Previously, I used SMS notifications from Nagios, but at upwards of 5p per credit, given the way a flakey link can result in 10 or 20 notifications in a short space of time, this was starting to get quite expensive. It also doesn’t fit very well with the fact that SMS can cost more to receive abroad and, generally, I spend more time in places with a reliable WiFi connection that with a good mobile signal – especially when visiting family at Christmas.

What would seem appropriate here, then, is some kind of push notification service working over data rather than SMS. Notify My Android doesn’t seem as well regarded these days, but Pushover looks really good. I particularly like that you’re provided with an e-mail address which you can just plug in to things like Nagios to have e-mails turn up as push notifications, without having to script their API.

I’m 24 hours into their 7 day trial, and so far so good. Their app is nicely featured including mute/unmute of sound and vibrate notifications, and quiet hours.

Porting a mobile number to AAISP

Just look at my custom network name

St Columba’s has had the same 07 mobile number for a long time, and consequently it’s printed on various signage and readily available online. Traditionally, it’s been assigned to a pay as you go SIM card in a creaky old Nokia phone carried by our facilities manager. However, this approach has its limitations: when they’re on holiday, they have to physically pass the handset to someone else to handle the calls, and it leaves the phone number (arguably an important asset in itself) attached to a pay as you go SIM which might have an employee’s name on it rather than the church’s.

Issuing a new smartphone seemed like the opportune moment to sort this out. I did some digging into business mobile contracts – having the number on one of those would sort the ownership – but none of them seemed to feature call divert which could be configured without having possession of the phone. So I turned to a service I’ve written about here before. In addition to being able to buy 07 numbers on VOIP, AAISP will let you port in existing numbers using a PAC code. Coupled with a Sip2Sim SIM card in the phone, this neatly de-couples the number from where the calls get sent to and gives us the divert via a web page and all the other bells and whistles.

The ordering process was nice and simple and fully automated – I punched the PAC into the order form at A&A, and the number moved over within a couple of working days (it got briefly stuck in the middle, but A&A fixed it out of hours when I asked on IRC – try getting that level of service from a big company). Billing is all via direct debit, and no messing around with paper invoices, which suits us fine.

As the icing on the cake, I calculate that for the volume of calls and data usage of this particular device, it will work out cheaper than any of the mainstream corporate mobile deals I saw. I’ll report back on that.

Virgin Media

We got a Virgin Media (business) line in the church office recently, and although it was quite unstable for the first few weeks, it has now settled down nicely. I can run my monitoring on the end of it without getting lots of false alarms:

vmedia

Colour me impressed, especially for what we’re paying.

URC e-mail fail

I don’t know exactly who my church has outsourced @urc.org.uk e-mail to, but whoever it is clearly hasn’t read RFC822 (“SMTP systems are expected to make every reasonable effort to accept mail directed to Postmaster from any other system on the Internet.”)…

fail

Rather difficult to complain about the original broken address I found…

Update: They did get back to me via Facebook eventually. Still not had an answer on why postmaster@ doesn’t work, though.

Proxy DHCP

Somebody should have thought of this, and it turns out that somebody did.

A couple of years ago, I wrote about setting up a network boot server on my home LAN. I said this meant “taking the job of doing DHCP away from the router and doing it myself on a Linux box”.

Reader, I was wrong.

Setting up my new connection from BT today, I really didn’t want to keep running DHCP separately on a Raspberry Pi. What I wanted was for the Pi to just talk to clients which wanted to do network boot, and leave the BT router to do DHCP for ordinary devices.

This exists, and it’s called proxy DHCP. dnsmasq on the Pi can handle it; this tutorial is the one I got working. Sadly proxy DHCP doesn’t seem to honour options, so I had to take the path prefix (option 210) and manually prepend it to all the paths in my boot config.

This is all back up and working now, and hopefully should prove useful and less disruptive for a few years yet to come.

Now that i have native IPv6 and DHCP is back on the router, the Pi ceases to be a single point of failure, though it’s been rock solid for nearly three years now.

VirtualBox does network boot, of course

VirtualBox does network boot, of course

IPv6 at home from BT

It works – 10/10 on test-ipv6.com; can’t really say fairer than that. I did however enjoy the comedy reverse DNS:

$ curl http://files.dnorth.net/ipquick.php   (05/04 19:19)
2a00:23c4:e81:3100:fdbc:e60a:226e:9903
broadband.bt.com

Oddly enough, broadband.bt.com doesn’t resolve forwards to every IPv6 address on their network, so why bother claiming the reverse?

IPv6 only IMAP server

Because why not.

I have an IPv6 only virtual machine hosted at Mythic Beasts. Sadly, I don’t have much time to play with it these days, so I’m retiring it next month. Meanwhile, I’ve got something working which I thought ought to be possible but failed at the last time I tried…

I participate in the minority sport of hosting my own e-mail. That means I run an IMAP server which mail clients can view and search messages using. For myself, I have native IPv6 on my home internet connection, but when I’m out and about, my mobile phone is still IPv4 only, and most of my relatives who I host mail for are v4 only too. Mythic and others have blogged at length on hosting websites behind their IPv4-to-IPv6 reverse proxy, but can it handle IMAP?

$ nc -vv proxy.mythic-beasts.com 993                                 (01/04 17:06)
DNS fwd/rev mismatch: proxy.mythic-beasts.com != rproxy46-hex-a.mythic-beasts.com
DNS fwd/rev mismatch: proxy.mythic-beasts.com != rproxy46-sov-a.mythic-beasts.com
proxy.mythic-beasts.com [46.235.225.189] 993 (imaps) open

The fact that they’ve got it listening on the standard port for IMAP over SSL is encouraging. Let’s give it a go…

Prerequisites

In addition to your IPv6 only VM, you’ll need a domain and you’ll need to point some DNS. The examples below assume you’ve CNAMEd imap6.example.com to proxy.mythic-beasts.com.

Part 1: Install and configure Dovecot

Couldn’t be easier if we’re using Debian:

# apt-get install dovecot-imapd

Configuring Dovecot to properly store mail is beyond the scope of this article, so we’ll just do some minimal setup.

Part 2: SSL

It’s 2017, and SSL certificates are free. So let’s get some.

# apt-get install dehydrated apache2 dehydrated-apache2

The Apache server is only necessary because I’m taking the path of least resistance to getting SSL certificates from letsencrypt, which involves them verifying control over the domains using HTTP based challenges. You could set up Mythic’s DNS API to eliminate the need for this.

Once you’ve done this, edit /etc/dehydrated/domains.txt to include your domain (e.g. imap6.example.com) and add /etc/dehydrated/conf.d/config.sh containing CONTACT_EMAIL=”you@example.com”.

Having done this, you can run “dehydrated -c”. I’d recommend not running it as root, which means fixing the directory permissions when it falls over the first time. All that sorted, we should now have a directory /var/lib/dehydrated/certs/imap6.example.com/. That means we can go ahead and edit /etc/dovecot/conf.d/11-ssl.conf to read:

ssl = required
ssl_cert = </var/lib/dehydrated/certs/imap6.example.com/fullchain.pem
ssl_key = </var/lib/dehydrated/certs/imap6.example.com/privkey.pem

While you’re there, you should set port=0 on the non-SSL IMAP listener in 10-master.conf, to stop it listening on non-SSL ports.

All that sorted, we can restart Dovecot:

# service dovecot restart

And now we can jump over to a client machine and try it out:

$ ncat --ssl -v imap6.example.com 993
Ncat: Version 6.47 ( http://nmap.org/ncat )
Ncat: SSL connection to 2a00:1098:0:82:1000:3b:1:1:993.
Ncat: SHA-1 fingerprint: 2FB3 9166 A7B7 6552 6215 963C 43D0 824E A02F 9FB3
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN] Dovecot ready.

Assuming all is well, you’ll see the OK response from Dovecot, and when you hit Ctrl-C, you’ll get a line logged in /var/log/syslog on the server complaining about how your client hung up.

At this point, you should be able to try it with a real mail client, e.g. Thunderbird. And you’ll notice that everything in the garden is lovely, except that the logs show all connections coming from Mythic’s proxy, rather than their true source IPs.

Part 3: Proper IP addresses in the logs

Mythic’s proxy supports the PROXY protocol to enable this, but does Dovecot? As it turns out, they added support in 2.2.19, but Debian stable has 2.2.13. Backports to the rescue:

# apt-get install -t jessie-backports dovecot-imapd

Now we need to turn on PROXY support for imap6.example.com using the Mythic control panel. Having done that, I was very impressed with Dovecot – the logging in /var/log/syslog walked me through fixing each mistake I made, starting with needing to configure Dovecot for PROXY:

Edit /etc/dovecot/conf.d/10-master.conf and add haproxy = yes to the imaps listener. The defaults are sensibly secure: to avoid clients spoofing IP addresses, you must provide a whitelist of clients allowed to speak to listeners with haproxy support turned on. That means setting haproxy_trusted_networks in the same file. You can find the necessary IPv6 addresses to space-separate on this page.

Having done that and restarted Dovecot, I moved my laptop over to my guest wifi network (where there is no IPv6), and restarted Thunderbird…

$ grep dovecot /var/log/syslog
Apr  1 17:17:55 test-box dovecot: imap-login: Login: user=<david>, method=PLAIN, rip=192.0.2.1, lip=93.93.129.174, mpid=1234, TLS, session=<...>

Result! The IP quoted as “rip” (remote IP) is the IPv4 one I’m running Thunderbird on. Interestingly, Dovecot has chosen to log the IPv4 of Mythic’s proxy as the local IP (lip).