Network Update Errors Can Cost Lives

When you are updating a firewall, mistakes can make it impossible for your systems to talk to the outside world. If you work at a telephone/telecom company, you can disrupt phone service for a lot of people. From The Register: Ten mistakes marred firewall upgrade at Australian telco, contributing to two deaths

As The Register reported at the time, Australia’s equivalent of the USA’s 911 and the UK’s 999 and 112 emergency contact number is 000 – Triple Zero – and local law requires all telcos to route emergency calls to that number. For 14 hours on September 18, Optus could not route some customers’ calls to 000 and was unaware of any problems on its network. The company eventually learned of the situation from customers who complained to its call center.

During the 14 hours of the outage, 455 calls to Triple Zero didn't go through. Two of the callers died.

Click thru for the details, and there are a lot of them. How the change was made without review. Why people noticed a problem but didn't escalate it. The fact that the call center never expected to be the front line on Triple Zero problems, and they didn't know what to do. And other issues.

Calling 911 in the US, 999 in the UK or 000 in Australia is a fine thing to do, but as we have seen all too often, it only takes a missed bit in a configuration update, or a guy with a backhoe, to make a huge hole in those services. You should figure out what your Plan B is. These are complex systems built, run, and maintained by flawed humans. They won't work perfectly all the time. Sometimes, they won't work at all.

I won't compare the internet, and the technology that we've built on top of it, to a house of cards, but it isn't far off. The recent, and what seem to be escalating, outages make that point for me.