Tech

Never close the Automation Controller in a TV playout centre

Who, Me? A story from the world of television in this week’s edition of Who, Me? as a weather presenter sweats while panic reigns supreme in the backroom.

Our tale, from “Mike” (not his name), dates back to the early part of this century, when Windows NT 4 and 2000 ruled the roost and processes were a little less slick than today.

Mike worked for a television playout centre, where channels are processed before going out to transmitters or satellites. While there was a good deal of automation, some tasks remained resolutely manual, including the collection of transmission logs.

Our hero was more involved in the IT side of things, although there were also broadcast engineers on duty on the night in question. It was the late shift, and Mike was in the control room as the late evening news was going out. All seemed to be going swimmingly, as it always did.

A bit of background: all the PCs with transmission-critical functions were hooked up to a KVM-type system. Anyone on duty could therefore access the system and tweak things as needed. “The system worked on trust,” explained Mike, “and it was assumed that everyone knew what they were doing prior to gaining access to the back-end systems.”

There’s a notorious phrase about “assume”, but it escapes us for the moment…

Unbeknownst to Mike and the team, a new production assistant (PA) had been shown how to collect the transmission logs from the channels they were responsible for. A simple task, but one the PA had only done once before.

At 22:15 the PA started running through the task. However, the newly minted employee seemed to be having difficulty finding the right PCs on the KVM switch.

“We in engineering/control knew this because we were following their moves on the KVM,” Mike said.

“Instead of asking where the PC with the transmission logs was, they kept on moving on through every PC on the KVM shutting down running applications or minimising the screen of various applications – unassumingly until they found the right application.”

While watching the incompetence of others is rarely anything other than amusing, the issue needed dealing with, and the team was in the process of contacting the transmission suite to offer constructive assistance (or simply ask “what the heck is going on?”) when…

“To our horror, we saw the KVM screen move to the Automation Controller PC – and the mouse move to the close button.”

Time slowed down. Swearing started. Vocal octaves went up a notch. Mike glanced at the output screen and saw that the presenter of the weather report had just that second come to the end of the segment.

“To the uninitiated, I should explain what the Automation Controller does… it runs the output of the TV according to a schedule. It does all the donkey work of cutting between different input and output sources automatically – without need for a vision mixer desk.

“This particular Automation Controller controlled more than just one channel, though – it controlled about 10 channels and was temperamental at best!”

And now it had been abruptly brought low by a PA hunting for transmission logs.

The weather presenter’s smile became a grimace as realisation dawned that they remained on air. Millions of viewers were treated to the rictus grin and waxen visage as Mike and team sprinted to the transmission playout suite, where a full-blown row had kicked off.

The producer, transmission controllers and voice-over artists were shouting at each other over what to do while the weather presenter squirmed. “Cut to the next programme!” yelled Mike, only to be told that wasn’t an option “because it didn’t look good.”

And so the verbal ping-pong went on while the weather presenter remained in place, smile becoming ever more desperate. It was a long, long 75 seconds before the channel breakdown slide was finally shown and the voice-over artist apologised for the loss of service. Eventually nerves were calmed and the channel went back on air.

“Did I mention that the automation controller was controlling 10 other channels?” said Mike.

Oh dear. The team raced back to the control room to free the rest of the channels from the borked transmission controller PC. “The automation software had to be started up – just right – otherwise it had a habit of taking channels off-air,” he said. A very careful and controlled restart was needed…

“We earned our stripes that night – because other than the breakdown on one channel, we managed to keep the other nine going… without interruption.”

Readers will be unsurprised to know that the concept of passwords were swiftly reintroduced, as well as a red background on the Automation Controller PC with the words “DO NOT CLOSE DOWN THIS APPLICATION” in case users were in any doubt about what not to do.

Still, at least there was some material for that year’s Christmas Tape. The fate of the PA involved is, sadly, lost in the mists of time (or Mike’s memory, at least).

We’ve all felt our stomachs descend downwards as time slows before a cock-up. Perhaps you pressed the wrong button, or perhaps you were on the receiving end. Tell your story with an email to Who, Me? ®


Source link

Related Articles

Back to top button