Scott Detrow does the best he can with what he has, aided by WESUN editor D. Parvaz. (Susan Davis/NPR)

It was a routine Sunday morning in Studio 31 at NPR’s headquarters in Washington, D.C. Weekend Edition producers were loading files for broadcast and making last-minute music button selections, while engineers auditioned audio that was about to go out to hundreds of stations across the country.

Suddenly, less than an hour before air time on June 13, the audio became inaccessible because of a technical failure. Communications channels that NPR staff use to contact each other and member stations were also disrupted.

“Over the years we have conquered a variety of technical challenges during live broadcast,” recalled executive producer Sarah Oliver, who has been at Weekend Edition since 1997. “But that morning was like nothing imaginable.”

Eric Marrapodi, who runs NPR’s live special programming, was taking his son to an end-of-year baseball party when he saw a message about the meltdown and rushed to the studio.

“When I came in, it was pure triage,” Marrapodi said. “Who can we get on the air and what can they say?”

It was the worst broadcast crisis anyone at Weekend Edition could remember, at least since a Saturday in the early 1990s when, according to veteran show producer Ned Wharton, “someone sliced through all the reel-to-reel tapes,” including one with “a profile of Captain Kangaroo that someone worked really hard on!”

That was an accident: Someone was “salvaging” tapes — cutting them with a razor blade to free up empty reels. And yet the broadcast went forward because producers were able to raid the shelves for pieces lined up for other shows.

What made this latest crisis worse is that content nowadays is stored digitally.

But because of quick thinking, improvisation and doing what NPR journalists do so well — live hits — the show went on the air and covered the news of the day.

There were a few silences and other hiccups spread throughout Weekend Edition Sunday’s six hours of broadcasts. “But if you didn’t notice them, you wouldn’t have known there was a problem,” said Sharahn Thomas, NPR’s vice president for content operations.

Indeed, most listeners didn’t notice anything amiss — or if they did, they didn’t email NPR about the broadcast.

I was the National Desk editor on that extraordinary day, and after speaking with a number of people involved, I’ve come up with six takeaways on how best to respond to technical failures — especially now that so much of what gets broadcast is stored and manipulated digitally, making the potential for calamity greater than ever.

1. Think on your feet

Fortunately, WESUN had planned to start the show with live two-ways about President Biden’s trip to Europe and Israel’s new government.

So as the show’s soothing jingle played and the pre-recorded billboard was broadcast from the technical director’s computer (since the studio player software was offline) at 8 a.m. ET, the production team knew they had about 20 minutes before facing the abyss.

Deputy managing editor Jim Kane put out an appeal for correspondents to call in and talk to host Scott Detrow, who was filling in that Sunday for Lulu Garcia-Navarro, about newsy topics they’ve been covering.

Kane also activated an emergency conference line that would end up staying open all day, allowing editors throughout the organization keep up with developments.

But as reporters started phoning in, they couldn’t get scripts to the host because the email and internal communications systems were down. So Weekend Edition editor D. Parvaz sat next to Detrow in the studio, taking questions that were texted to her and turning them into interview scripts. Marrapodi was doing the same outside, taking notes on a computer, or sometimes with just pen and paper, snapping photos of them with a cell phone and texting them to the host.

Eric Marrapodi took notes on the fly and texted pictures of them to host Scott Detrow.

“It took the problem-solving skills of everyone onsite in those first minutes to keep pushing through and get the show on the air,” Oliver said.

Still, there were moments when Detrow had to come up with questions on the fly, and he did so with equanimity and sense of humor as he spoke to reporters who had dropped everything to lend a hand.

“Up next in our pickup basketball game of news this morning is Joe Palca, science correspondent,” Detrow ad-libbed, as they began a conversation on the global COVID-19 picture. “I don’t know what your original plans were for this Sunday, but I’m glad to have you on now.”

As attention began to shift toward Weekend All Things Considered (WATC), which airs at 5 p.m. ET, editors put a rundown in a Google document and shared it with stations.

From the very beginning, the outage posed a huge challenge for NPR Newscasts, which not only delivers top-of-the-hour news, but also twice-hourly headlines during WESUN. There was no tape, so anchors had to work with all-copy scripts written without access to wire services.

“We were quite literally in a technological foxhole together on 6/13,” Newscast executive producer Robert Garcia wrote in an email to staff later that week, praising producers who devised a way to save audio into Slack channels so that it could be played on air. “We didn’t miss a single cast. We didn’t miss a single post.”

2. Say “no, thank you”

When the call went out for live hits, so many NPR journalists responded that it overwhelmed the studio control room.

“The phone lines were full of reporters calling in,” said Oliver. “It was a good problem to have.”

Technical director Stu Rushfield recalls how “those constant phone calls made it nearly impossible to hear what we were actually doing on the air in the moment.”

For Marrapodi, it was comparable to a major breaking news situation.

“It’s tough because in the beginning you’re begging for anything,” he said. “But then you hit a point when everyone has mobilized and suddenly you’ve gone from having nothing to having way too much.”

Which means staying calm, making snap judgment calls and saying “no” to well-meaning individuals you might have jumped for joy to hear from minutes before.

The board for the show WESUN originally had planned.

“Indecision is the wrong decision in breaking news,” Marrapodi said, adding that he was influenced by the book, Extreme Ownership: How U.S. Navy SEALs Lead and Win, written by two former SEALs, on leadership in the heat of battle.

3. Train for it

Marrapodi, who coordinates live coverage of major planned and unplanned events, recommends all live shows periodically have a training exercise simulating a fire or server outage so that staff become familiar with emergency response procedures.

Thomas, who oversees newsroom operations at NPR, also emphasized training, but warned against expecting the same thing to happen again.

“You just never know where the next catastrophe is going to come from,” she said.

She recommends having a plan detailing who’s responsible for what, where backups live and how to get people into the studio.

And printing it out.

“What doesn’t fail is what’s on paper,” she said. “It may seem very rudimentary, but that’s what will save you.”

4. Your audience will forgive, up to a point

Kelsey Page of Audience Relations recalls a few emails from online listeners with questions like: “Where is the audio for this morning’s Weekend Edition (June 13)?” But she didn’t hear from anyone noticing changes in the audio or content.

That may be largely because a broadcast tech was able to track down the file containing the Sunday Puzzle, a fixture of the show since 1987.

Working off the estimated length and record date, Daniel Shukhin in Audio Engineering located it in a networked drive where some mixes are saved.

“While it wasn’t the final version, it was close enough,” Shukhin said, to be played on air.

Probably the main reason listeners didn’t notice anything out of the ordinary about Weekend Edition that Sunday, said producer Samantha Balaban, “was because the puzzle aired.”

5. Rethink storage and backup

As the day wore on, engineers were able to locate more files, allowing producers to remix other segments — to the point where WATC was able to broadcast its originally planned show — even though it had also prepared an alternative version that would have comprised all live hits, just in case. Morning Edition did the same for its Monday show.

So is the takeaway to make backups of everything slated to go to air? And, if so, on what — USB drives? SD cards? Reel-to-reel tapes like the ones that saved Weekend Edition Saturday back in the day?

Thomas says the medium isn’t as important as having a plan for backing up files systematically and sticking with it, even though your mind may be on more immediate demands.

“We get caught up in the day-to-day and have news cycles that don’t allow us much time for introspection,” she said, “but [a plan is] what will get you through.”

6. Balance transparency and security

Journalists prize transparency, and Detrow’s candid comments on air about the technical problems that day were straight from the playbook of great hosting.

Detrow also tweeted about the problems about 10 minutes into the show, leading some reporters to reply in the thread with offers to help.

Later in the day Rushfield, the audio engineer, tweeted a dramatic thread about it.

There are some concerns that giving too much detail about technical issues can reveal vulnerabilities that bad actors can take advantage of.

Rushfield said he believes his thread “engendered great goodwill among our listeners and stations, who celebrated the teamwork it took to keep the train on the tracks. And it almost certainly led to some kind of bump in listener donations to their member stations, if the responses to the thread are true.”

Marrapodi said it can be exciting for audiences to know that journalists are facing unforeseen problems.

“On the air, when things are falling apart, you should be honest with the audience and let them know you’re still in control,” he said. “But I don’t think it’s helpful to spell out what the trouble is.”

It’s a question of TMI. The larger point is that what in the analog age may have been unquestioned good practice, may now need to be balanced with the risks inherent in digital newsrooms.

Many thanks to Sarah Oliver, Ned Wharton, Samantha Balaban, D. Parvaz, Daniel Shukhin, Stu Rushfield, Robert Garcia, Eric Marrapodi, Jim Kane, Sharahn Thomas and Kelsey Page