CloudFlare works

Every website should use HTTPS these days – EOC. There is no longer any financial reason not to.

Even if your site is using cheap and cheerful hosting which gives you horrible flashbacks to your first ventures onto the internet in 1999, simply flip the DNS over to CloudFlare and discover that they’ve got it all so well figured out that you can be up and running with HTTPS (on the free tier!) with ten minutes of setup plus 24 hours’ wait for the DNS to change

Of course, this does give you the chance to participate in their global outages, but let’s be fair – one of the recent two was not their fault, and they have been open about fixing the causes of the other.

I was really impressed overall, and I feel a lot happier about the walking club’s £20 per year cPanel hosting now it has the CloudFlare CDN standing between it and the nastiness of the public internet.

Result! Church website to follow shortly…

Innotech iComm – it actually has an API

I’ve mentioned before that we run a rather venerable HVAC controller at the church which drives our heating system.

Last weekend, I was bored enough to start digging in to the network protocol it speaks, but before I spent hours slaving over Wireshark, I took a last look around to make sure I hadn’t missed an “official” programmable interface.

Somewhat to my surprise, I had!

iCommX help screen

1999’s idea of the future

Leaving aside a bit of sniggering about how ActiveX hasn’t been “The Way of the Future” since I was half my current age, let’s try firing it up from a sensible programming language (Python)…

import win32com.client

def get_px(block_name, field_name):
    """Constructs the COM object for reading and writing a block
    Block names can be viewed in MAXMon; most of the field names are also visible there if you open (double click) the block"""
    px = win32com.client.Dispatch("icommx.pointx")
    px.ServerAddress = "localhost"
    px.DeviceAddress = "6:1" # connection number, device number
    px.PointAddress = "%s~%s" % (block_name, field_name)
    return px

def get_temps():
    for tzone in ["Outside Air", "Boiler Flow", "Utility Space", "Church Space", "Hall Space", "HWS Cylinder"]:
        px = get_px(tzone, "OUTPUT")
        print(tzone, px.Value)

get_temps()

There are various hoops to jump through to generate the necessary Python COM bindings, which you can Google for. Needless to say, all this COM stuff requires us to be on Windows. Saving this as icp.py, let’s give it a try:

C:\Users\dtn>python icomm-python\icp.py
Outside Air 27.03
Boiler Flow 24.23
Utility Space 20.67
Church Space 21.32
Hall Space 23.91
HWS Cylinder 54.75

Result! (Hot day in Oxford … the boiler is not running, yet the hot water is toasty thanks to our solar panels.) You can also, somewhat scarily, assign to the “Value” field for blocks which are inputs such as override buttons or temperature set points.

The API is a bit limited because there’s no way to enumerate the available blocks – you just have to know (using one of the companion apps like MaxMon) what they are. More disappointingly, I couldn’t find a way to write the schedule on the seven-day timer blocks. However, a bit of thought made me realise this doesn’t matter too much: I can still write to a boolean flag connected to the “override” pin on those blocks, and write an override value into each one – number of minutes until heating next goes on/off.

Long story short, we can finally wire our church heating directly into the bookings calendar and stop having to program it by hand! Of course, we will need a permanently stationed Windows box, but we’re going to need one anyway for some other projects. Exciting times…

Oxford Botley Road consultation response

This is my response to the Oxford City Council consultation on improving Botley Road for cyclists and busses…

Dear Sir/Madam,

I live in Abingdon and commute to work on Osney Mead, just off the Botley Road. As often as possible, I cycle into the city from the south and avoid the Botley Road altogether, but there are also occasions when I am a motorist on it. Additionally, I use it to cycle from the industrial estate into the city centre after 5pm on some evenings.

I note the following:

  • As a cyclist, I would much prefer on-carriageway cycle facilities along the entire length of Botley Road; ideally with physical barriers separating them from the traffic, like those used on Donnington Bridge. Off-carriageway facilities which share the same level as the pavement are rendered useless as pedestrians are routinely unaware of which half of the space they should confine themselves to. The proposed switch to cyclist priority across junctions is welcome, but in my view this is much better achieved by making cyclists first-class users of the road itself.
  • The raised walkways across side roads should be carefully reviewed from a drainage perspective, given Botley Road’s history of severe flooding.
  • The reduction of traffic exiting Ferry Hinksey Road to a single lane is undesirable as both a cyclist and a motorist. A majority of traffic turns left from Ferry Hinksey Road, particularly at the end of the working day, and the additional lane allows cyclists to safely overtake that traffic and turn right towards the city centre. Adding a new pedestrian refuge here seems excessive as the timing of the lights presently allows plenty of time for pedestrians to cross, and in any case there is a controlled crossing a short distance into Ferry Hinksey Road. Reducing traffic exiting the industrial estate to one lane will be more dangerous for cyclists as it will create increased congestion and force them to wait longer and breathe more fumes.

Although it is perhaps wider than the scope of this consultation, I note that the key stated goal of encouraging less car use on Botley Road is unlikely to be achieved without more drastic measures – for example, a congestion charge for non-resident travel inside the Oxford ring road. Alternatively, speaking in my motorist persona, I would be much more likely to use the Park and Ride (so far, I never have in ten years of working on Osney Mead) if it either (a) did not charge for both the parking and riding aspects of the journey or (b) had busses which served the industrial estates off Botley Road rather than requiring a ten minute walk through the rain in addition to the bus journey.

Yours faithfully,

David North

Replacing HallMaster

So, as explained in my last post, we needed a new invoicing system for the church hall. The back-of-a-beer-mat requirements were pretty simple…

  • One-off bookings can be invoiced by a manual process, but the regulars must be sent an invoice by e-mail automatically at the end of each month.
  • One invoice should contain all bookings for a customer in the period covered.
  • Must be possible to see an aggregated summary of which customers owe us how much money
  • Must be easy to track repeating series of bookings and make ad-hoc tweaks

There did not seem to be anything out there to buy (at least, not for sensible money), so I sat down over a couple of weekends and wrote it.

Naturally, I used other systems as much as possible. Back in the day, we used Google Calendar to track the actual bookings, which is a well known interface and worked nicely. Unlike HallMaster, it doesn’t automatically prevent clashing bookings – but we only have one human entering bookings, and this hasn’t been a problem in six months of operation. It also doesn’t guarantee that every calendar event is associated with a known customer – but since most of our invoices go out monthly, we solved that with a weekly nagging e-mail to the staff in question about any calendar events which can’t be matched to a customer or room.

So, with a Google Calendar full of bookings in place, I turned to Django for the next bit. Thanks to the automatic form generation (and a bit of customisation of the automatic admin interface), writing a web app to keep track of customers and their billing contact details was super-quick. Each customer also has a list of “event names” which are the titles under which their bookings appears in Google Calendar. This also allows the use of e.g. “Private Event 42” for customers who don’t want their name visible in the public calendar.

And now to the beating heart of the system: automatic invoicing.

This has to be customer-centric, so the first thing I implemented was a method on the Customer model which could generate an invoice for that customer’s bookings in a given time period. Looking things up in Google Calendar is super-easy: they have a nice, well-documented API including example code which can be copied and pasted for a fast start. Their API also has the crucial feature of forcing the “expansion” of repeating events, i.e. your code doesn’t have to understand repeats; it just gets an event for every repeat.

From there, looking up each room’s hourly cost according to whatever tariff the customer is on is an easy modelling and maths problem. Thankfully we’re not required to charge VAT owing to our size and charitable status, so that complication doesn’t have to be factored in.

The most complicated part turned out to be generating invoices in an acceptable format. My first thought was plain text e-mails, but getting a big table of bookings laid out nicely in one is a challenge, and probably impossible given that many e-mail clients don’t use a monospace font. What’s more, it would look rubbish on a mobile phone, which is where many e-mails get read these days.

Generating PDFs from Python doesn’t seem to be a super-easy solved problem, so in the end, I went for generating invoices as HTML (using a normal Django template) and then turning this in to PDF using wkhtmltopdf (which has some decent Python bindings). This does require the right fonts to be installed on the target system (ttf-mscorefonts-installer on Debian) to avoid invoices coming out in a blocky default font, but other than that blip it seems to be 100% reliable, and doesn’t require an X server.

We send these out attached to a plain text e-mail, which just says how much you owe and refers you to the attachment for details.

Naturally, the invoicing logic is covered by a wide set of automated tests. We do have the odd hirer whose bookings cross multiple days (i.e. span midnight), so I tested that in particular and also the possibility of bookings which span the clocks going forward or backward. Django makes this fairly easy to cope with: store all the times in UTC, which is also what the Google Calendar will give them to you in. Then all you need to do is format them for display to the user in local time (which covers BST when needed).

Automatically issued invoices (recipients redacted)

With that logic in place, it was just a quick custom management command to loop over all customers and invoice them if they are a regular and it’s the first of the month. Set that on a cron job for early morning on the first of the month, sit back, and relax.

Lessons learned

Make it easy to look under the hood. The Django admin interface helps with this, as does the facility I built in to CC all outgoing e-mails to extra recipients. I use this to stick them all in folders under my own inbox so I can refer back to them.

Ensure a failure to invoice one customer doesn’t stop all the others being invoiced. Easy enough with a try/catch, and the customer-centric design makes it easy to issue that one customer’s invoice later with a manual invocation from “manage.py shell” once the problem is fixed. This works well and we’ve had 100% successful invoicing runs for the last five months (out of six since going live).

Make it easy for people to get the payment reference right when paying you via BACS. Our system assigns a customer number to each hirer, and tells them to put just that as their payment reference. There’s no need for an invoice number or dates, and the reference can be the same for all payments, which helps with some online banking systems that make it painful to set a fresh reference for each payment to the same recipient.

  • This works pretty well, and nearly all customers manage to do the right thing. Some insist on quoting an invoice number (which is managable, we can just de-reference that back to the customer), and the odd one or two continue to make stuff up. I have an “upload bank statement” view which allows me to manually assign incoming payments where we haven’t detected the customer automatically. At the moment this is rare enough that I’m letting it slide, but if we ever get an automatic feed in of this information (Monzo and Starling, where are those charity accounts?), we will start cracking down and insisting on the right reference.

Don’t start your invoice numbers with the letter I, people will mis-read it as the number 1. We solved that by switching to “V” instead.

The end result is less than 2,000 lines of code which I’m continuing to evolve and tweak, but overall, this has solved it. We can see exactly who owes us money, split between the one-ofs and the regulars, and being reliably sent a bill on time is helping the vast majority of them to pay up promptly with the right reference. We’re approaching the mid-point of the year and well on track to get back to previously attained levels of hiring income.

The authors of HallMaster might well argue that I haven’t replaced their entire product with two weekends and not much code – and they’d be right, my system is designed to do exactly what we need and not service multiple customers – but I can’t help but feel it’s high time they offered fully automatic invoicing to really save their users some time. It’s not difficult and it makes things feel so much more professional.

Invoicing for the church hall

I did a long stint as treasurer of St Columba’s URC (finally stepping down last year). One of our challenges, which I suspect we have in common with a lot of other churches and community groups, was billing for letting out our premises.

Being as we are, bang in the middle of Oxford, and with a variety of different rooms available, we do our best to give something back to the community by renting them out to other charities and not for profits for a reasonable rate … and of course to profit-making enterprises for a commercially appropriate rate.

In the beginning (pre-2009), all of the invoicing and tracking of who owed what was done by hand, with spreadsheets being e-mailed around. This was very labour intensive, and for a number of years after that, we scaled up our lettings by moving to a largely honour-based system which tracked bookings in a Google Calendar and required people to proactively pay us on time via bank transfer.

This sort-of worked for quite a while, but of course many organisations, especially voluntary ones, do not cope well at paying people money if they haven’t been sent an invoice. We had a few embarrassing situations where we invoiced several years of historical debt and caused problems for the people on the other end of it. The bills were perfectly legit, but they had of course spent the money on other things in the interim.

A few years back, things really went downhill for us, and we lost quite a few customers, partly because of our sketchy billing process. Something had to change.

Naturally, I assumed this would be a solved problem – the business of letting a few rooms out with a calendar and invoicing according to a couple of different tariffs is pretty standard and universal. And on first inspection, HallMaster looked to be just the ticket:

  • Aimed at the voluntary sector, like us
  • Reasonably priced
  • Includes invoicing facilities

We ran with it for about eighteen months, but in the end it didn’t work out as hoped, for a few reasons:

  1. Invoicing is not automatic. This was one of the biggest disappointments to me. Most of our customers are “regulars” who book time each month and should get a bill at the end of the month with 30 day terms, covering all their usage that month. Unfortunately, HallMaster only supports issuing invoices as a manual process.
  2. Invoicing is booking-centric, not customer-centric. In HallMaster, you can only issue an invoice for a given booking, not a given customer. One booking can be a repeating series of events, which covers a lot of cases, but unfortunately some of our biggest customers have a couple of repeating series plus lots of smaller ad-hoc bookings. Getting all these onto one “booking” in HallMaster is tedious and error-prone, but if you don’t do it that way, you have to invoice them all separately.
  3. Who owes what is invoice, not customer-centric. Partly because of the above, the view which shows you who owes money lists invoices, not customers. This also means there is no facility (automated or manual) to send reminders to people who haven’t paid.
  4. Interface feels clunky. Somewhat subjective, but the manner in which the Javascript-heavy interface works is a bit iffy. Sometimes it would fail in certain browsers (e.g. Chrome), and sometimes long (tens of seconds) pauses with no progress spinner would occur. Sometimes it would get stuck if you asked for lots of results on one page.
  5. Took them a while to implement SSL. I complained when we first evaluated it that there was no https support (i.e. login details and personal data were being sent unencrypted over the internet to and from their site), and they didn’t take it as seriously as I felt they should (GDPR et al being a concern). They did implement it eventually, but it should have been there from the start.
  6. Confused the customers. It was difficult to remove HallMaster links from the e-mails it sent out, and some customers got confused by them. Customers were supposed to be able to sign up for free to view their invoice and payment history, but sometimes ended up at pages trying to sell HallMaster to venues, which confused them by talking about costs.
  7. Questionable e-mail behaviour. Their system attempted to send invoice e-mails “from” my e-mail address, e.g. invoices@saintcolumbas.org. Unfortunately, while this may have worked just fine in the 90s, these days the advent of SPF and DKIM mean that many spam filtering systems take a dim view of e-mails claiming a from address which the originating system is not authorized to send mail from. To be fair, they did “fix” this for me by setting all e-mails from our account to come from noreply@hallmaster.co.uk, but the ability to set a proper reply-to address would have been much nicer.
  8. No bank integration. OK, so given that Open Banking had barely been invented when we first tried this, a direct feed would have been asking rather a lot. But a simple option to upload a CSV bank statement and assign incoming payments to customers could have been a time saver over entering every payment by hand.

To be fair to HallMaster, I suspect it would work well for smaller venues or those with simpler usage patterns (and those still accepting cheques), but it didn’t hit the spot for us. In particular, it required too much staff/volunteer time to operate.

So what to do? See my next post for what we did.

On tour

I read a lot of crime fiction, and over the last few years, I’ve been working my way around, visiting cities where the books are set.

Naturally, Oxford (Inspector Morse) is a city I’m very familiar with, but I’ve also notched up Edinburgh (Rebus) in February 2017, and tomorrow I’m off for a weekend in Brighton (Roy Grace).

That just leaves Aberdeen (Logan MacRae), Los Angeles (Harry Bosch, Numb3rs), and New York (Castle, Law And Order) to go!

OK, so I drifted from books into TV at the end there, but it’s a hobby.

WiFi calling on the OnePlus 5t with Three UK

Over the years, I’ve often wondered why my mobile phone doesn’t do this “WiFi calling” thing. Being at a conference and not being able to make a call from the underground parts of the Barbican Centre (didn’t think of that in the 1970s, did they) finally prompted me to get it fixed.

A big shout out to this chap, who has figured out the right incantations to enable this with a few well-hidden settings and a reboot. I can confirm that in my case, the desired icon appears:

And it works! Where previously I got bleating about “mobile network not available”, I was able to make a five minute call which sounded clear as a bell. Calls don’t stay up when moving between mobile and WiFi (even with ‘WiFi preferred’ turned on), but it’s still a big leap forward over not being able to call at all – these days, there are lots of locations like remote holiday cottages where WiFi is no problem, but no mobile signal is to be found.

I’m not sure which of OnePlus or Three to blame for this not Just Working by default, but at least it is possible to kick it into life in a few minutes.

Update, 2019-06-22 – I ended up turning off “WiFi preferred” as it meant that moving from my desk to the kitchen at work to take calls without disturbing people would cause the audio to break up – probably because that journey puts me at the other end of the building to the WiFi access point which serves my desk. Hopefully increased adoption of Unifi kit at work will allow seamless roaming in future.

Android-controlled sound mixer for the church

Five years ago (!), we upgraded the AV systems at the church. One of the key changes made back then was to place the cabinet with the equipment in it at the back of the church, rather than the front – meaning it became possible to discreetly tweak the level on the microphones during a service.

However, that meant someone sat next to the cabinet, bending down to fiddle with the sliders on the preamp/mixer:

This is pretty standard kit and should be familiar to anyone who has worked with the microphones in a basic set-up like ours. Reading from left to right, you have a bunch of channels, each with a volume slider, and each plugged in to a microphone. In our case, that’d be two static, hard-wired microphones (“Pulpit” and “Lectern”) and a bunch of radio microphones (Lapel 1-3 and HandHeld). It also has a 3.5mm lead for plugging in MP3 players or phones, and a CD player hooked up too.

This is all very well, and a quantum leap over the 1960s technology it replaced, but it was really starting to grate to have to sit at the back and faff with the sliders to adjust during a service – and lowering a too-high mic which started to feed back when someone walked around with it required a dash to the back and unlocking the cabinet, while everyone enjoyed the ear-splitting whistling sound.

Someone suggested that it ought to be possible to replace the unit above with one which could be controlled from an app on a tablet or phone – thus allowing the sucker in charge of sound (that’d be me) to sit in a chair at the front with the rest of the congregation, and tweak as required.

A bit of googling led me to the Behringer X-Air XR16, which does exactly this sort of job. We bagged ours from Bax Music for £254, which seemed pretty reasonable when you consider it should be able to replace the Inter-M unit pictured above which still retails for nearly twice as much.

I mounted it in our rack inside the cabinet – the rubber sides unscrew to leave room to attach the supplied rack-mount “ears”. Since there wouldn’t be room to close the rack door if all the cables went in the front of this unit, I mounted it “backwards” with a blank metal plate facing outwards. But why not? There are no controls to be fiddled with on the unit itself; it’s all about the app.

Plugging everything in was straightforward, though I did need a couple of extra cables – a 3.5mm to 6.3mm lead to take over the input from phones/MP3 players, and an extra lead to patch in our hearing loop amplifier to one of the auxilliary outputs. These were easy to find online.

Sizing

A couple of things which weren’t apparent to me before ordering the mixer, but worked out for the best:

  • It’s less than the full width of a standard 19″ rack, but the supplied mounts fill that space
    • This also means there is plenty of room for the standard “kettle lead” power to go into the side-mounted port

So, we need a tablet

I ordered a Samsung Galaxy Tab A (10.1, 32GB, Wi-Fi) as the Android tablet to run the control app. Not much to say about that – it works, it was reasonably priced, it feels quite high quality for what it cost.

In order to set the mixer up, you have to power it on, then connect it to your network. You can plug in an ethernet cable if you happen to have one within reach. Alternatively, slide the switch on the unit to the right hand positon, “access point”, then connect your table to the WiFi network it creates. You need to have installed the free X-Air app before doing that.

Long-term, I’d recommend either using an ethernet cable if one will reach, or setting it to wifi client mode. When acting as an access point, it secures the network with WEP, which is an old and broken standard, so not much security at all. It can however join a pre-existing network with modern WPA security. If you do that, though, make sure you are happy that anyone on the WiFi with the correct app will be able to control the mixer.

App first impressions

X-Air seems reasonably nice. There’s quite a lot of capability in there, so budget for some time experimenting. Although you definitely need a decent sized tablet to drive it with confidence, the interface is as usuable as the space allows if you fire it up on a mobile phone:

Labels/naming

You can tap on the “CH01” labels in the app and then tap “CH01” again to reach a screen where you get to set the labels for all the channels. At this point, you either need to have been clever enough to transfer your microphone leads in a known order from your old kit to new, or work it out by trial and error (and move the leads around when they’re not in the order you want).

Phantom power

Phantom power is a fancy name for the preamp/mixer supplying electricial current to power certain microphones as well as receiving signals back from them. In our case, this applies to the static, hard-wired microphones. On the old amp, sending power down some of the microphone ports was controlled by physical switches on the back of the unit:

In the new world, this is of course part of the app. You get to it by tapping on the channel name and then pressing and holding the “48V” button. It’s a nice touch that the app tries to make it hard to turn this on by accident, as sending phantom power down the line to kit which isn’t expecting it might cause damage.

Gain

I found that even with the volume sliders for them all the way up, none of my microphone inputs were “loud enough”. Don’t try to compensate for this by turning the master output volume up too high, or turning your amp driving your speakers up too high – you’ll get a lot of annoying background hiss.

After some trial and error, I found that the “gain” setting was the one to fix this. It’s a control on the screen accessed by tapping the name of the channel once. In the case of microphone inputs, the gain control cranks up the input from the microphone before it goes through the preamp, which is particularly useful for radio microphones which people can’t or won’t clip close enough to their throat (these things are, like so much in the world, designed to be easy for men to use).

First trial

Our first service on the new system went well – certainly well enough for me to list the old preamp on eBay. Highly recommended! Drop me an e-mail if you want to know more.

The ability to have the system record directly onto an inserted USB stick is not one I’ve explored, but could be really useful for the future.

Afterword

Racking up and cabling in the new kit did cause me to slightly revise my good opinion of Expression Media, the original installers in our case. If I wanted eight UK power sockets available to my rack, I’d have tried a bit harder to find an eight-way lead/PDU rather than daisy-chaining two four-way extensions and using a two-way adapter to make up the eighth!

Business travel

Business travel is tedious at the best of times. But last night (and the early hours of this morning) were so tedious I thought I’d vent some steam here…

Because of the last minute nature of our trip to Italy, we ended up doing a silly pair of flights to get home at the end of Friday (Florence to Rome, Rome to Heathrow). An hour and 20 minutes on the ground in Rome should have allowed time for a nice dinner, albeit not a very leisurely one. But of course the first leg was 45 minutes late (thanks, Alitalia), so by the time we’d marched a mile from one part of the airport to another, we had to go straight into the queue for the final leg.

The final leg (Alitalia again) featured no food or drink apart from a packet of nibbles and some orange juice – making Ryanair’s offer to sell you an overpriced sandwich look like the stuff dreams are made of – and was aboard a creaky old aircraft whose lights flickered in an alarming manner when we landed. The staff were rude beyond belief (not a shred of help or sympathy in letting me get past the trolley when I wanted to go to the toilet, I just had to wait half an hour for them to finish), it was too hot, and generally deeply unpleasant. I’ll never be flying with them again.

Meanwhile back on the ground, the Brits were anxious to prove that we too can fail utterly at customer service: rather than sod about with two busses at what was by then 11.15PM, I thought I’d get a cab to take me the few miles from Heathrow’s Terminal 4 to the long stay parking at Terminal 5 where I’d left my car. The driver chatted to the chap in the booth and cheerfully informed me “we don’t do that”. Presumably because it’s not lucrative enough. Dear black cab drivers: This sort of behaviour is what makes me look forward to the day when Uber bankrupts the whole sodding lot of you.

Fortunately, a bus did turn up shortly afterwards, and the driver of the last T5 long-stay parking bus of the evening (which I had to myself!) kindly dropped me off right next to my car. After that, it was just some night closures on the M4 (should have taken the M40, but at least the road workers had the decency to actually be working behind all the cones, even after midnight on a Saturday) and home sweet home was finally reached behind the wheel of the mighty GTi.

At least Brexit didn’t take place while we were in the air, though, which was the original plan…