12BoC Ah, Christmas dos. Remember them? A chance to humiliate oneself in front of colleagues and make the odd drunken confession or two*. Welcome to the tenth whoopsie of our Twelve Borks of Christmas (12BoC) series*
Register reader “James” (a pseudonym generated by Santa’s Little Regomiser) recalls one that was blurted out over past Christmas lunches of a very well-known UK broadband provider concerning a cable modem whoopsie.
A call came through from the unfortunates staffing the first-line support: a customer was having some odd problems with their broadband, but nothing obvious was causing it. Ever service-focused, James called the customer to try some higher-level diagnostics.
“During the call,” he explained, “the usual troubleshooting process was followed.”
That all sounds very impressive until: “It was decided that following two customer reboots a remote reboot was in order to try to force the UBR [Universal Broadband Router] to reacquire the Cable Modem and thus reach a decision point of send tech or problem solved.”
The UBR is usually found lurking at the hub of a cable company. It is capable of serving thousands of cable modems. It can also send commands to a connected modem using the MAC address of the device.
In this instance, having the user turn their modem off and on again a couple of times hadn’t worked so the gang was to going to try and bounce the thing remotely. If that failed, then a new modem would be shipped.
“In order to boot the device off the UBR, the command ‘reboot cable modem [MAC address]’ is used,” explained James.
Unfortunately, a case of fumble fingers meant that the return key was hit just before the MAC address was entered.
“And so the UBR complied with the (unspecific) command as it was given, and proceeded to boot off all cable modems and set top boxes attached to that UBR.
“Some 7,000 devices.
“Essentially an entire metropolitan neighbourhood.”
The UBR immediately slowed to a crawl as the logs were flooded by DHCP requests from the thousands of devices that had suddenly rebooted at the same time.
What to do?
The customer was advised that “some offline investigation was required” and promised a callback in 30 minutes. The engineer responsible then logged off his phone, rolled a cigarette and strolled to smoking area. He returned, via the bathroom, some 20 minutes later.
Having had a while to recover, the UBR was now working. The customer’s modem was also back online and passing traffic bi-directionally.
Sure, there were red lines on the alarms board and chatter about “what must surely have been a brief outage on a UBR”, but nothing to explain what. There were, after all, no line-down alarms or other errors. No tell-tale signs of the typing error that had just happened.
“A brief call to the customer concluded all was fine,” said James, “and so the [ticket] was closed ‘remote reboot resolved, unknown cause’.”
He concluded: “And no one knew until the following Christmas do, which turned into a confessional amongst the engineers.” ®
A little ditty we are calling: The 12 Borks of Christmas
* ♬ On the tenth day of Christmas, the bork gods sent to me:
Time to modem down,