C4 & C5 Down

[Blumf]

Odd one, C4 is stuck with the production card of the recently shown F1 qualifying highlights, and C5 is showing a problem card.

The F1 show missed out on the ad breaks too, which meant ended 10 mins early.

Something's buggered up with the distribution

https://twitter.com/Channel4/status/1441829845558833166

Channel 4 and More 4 are currently off air due to a technical problem. We are working to restore them as quickly as possible

[Blumf]

Seeing mention of a fire.

[Blumf]

https://twitter.com/nmsonline/status/1441831633871646720

TV channels Channel 4, Channel 5, More4 and 4Music are currently off air after Red Bee Media, who play out the channels and adverts, is reportedly evacuating its West London White City transmission centre.

[steveh]

Still a load of problems from this including channels owned by C4 still down and lots of minor issues across several channels from the same playout centre.

Reports say that the fire suppression system destroyed a load of hard drives in a server room which it is taking a lot of time to recover from, with added complications of moving things back from the temporary and failover systems.

[Sonny_Jim]

Odd, most 'important' places run halon systems, although I can't remember what the legality of those are in the UK as removing all the available oxygen in a room tends to make people dead.

[steveh]

Halon was banned by the EU in 2003 and I think generally replaced by CO2 systems. Apparently the sound levels and frequencies of gas discharge systems can wreck hard drives. Big data centres have systems designed to minimise such effects but if it was just a space in a normal office block then maybe it was using an inappropriate system.

[Glebe]

Here's what really happened:

[Blumf]

Surprisingly naff disaster recovery setup.

Thought a major national broadcaster would have a live mirror site ready to switch to.

[jobotic]

But it's Bake Off night!

What about the nation's morale? This is worse than covid!!

[steveh]

Channel 4 apparently have a deal which includes full failover to another site but for some reason it didn't happen or isn't yet in place. BBC, UKTV and Channel 5 who are at the same centre all did, though with some limitations for Channel 5. There's a bunch of other minor channels there too, some just using it for inserting ads like BT Sport and others doing full automated playout. Centralising so many broadcasters at one location seems kinda risky even if they do have backup sites.

[Ambient Sheep]

This Twitter account (linking to some very informative articles of theirs) has been doing a sterling job of logging the day-by-day details of the ongoing breakdown:

https://twitter.com/cleanfeed_ttvr

And yeah, the Beeb were affected too, but instantly switched over to Salford.

Normally BBC1 & BBC2 are broadcast from Red Bee in White City, and BBC4, CBBC & CBeebies from Media City in Salford, but both sites are set up to broadcast each others' channels at the pull of a big red (virtual) lever. 

Indeed, before CoViD, my understanding is that they would swap for a week every now and again (BBC1 & BBC2 from Salford; BBC4, CBBC & CBeebies from White City) just to prove the robustness of the system.[nb]Not entirely sure why CoViD put a stop to that, maybe it's because the live announcers would have to change sites?[/nb]

And indeed, robust it proved to be, the only noticeable difference being the lack of a live announcer to introduce Strictly etc. (pre-recorded "This is BBC1" announcements being used instead), because the live announcer was busy standing in the White City car park assembly point.

The fact that Channel 4 (and to a lesser extent, Channel 5), are suffering huge problems three days later is really quite incredible.  Whoever was responsible for their Disaster Recovery plans won't be getting their Christmas bonus...

[JesusAndYourBush]

I was going to watch Zoey's Extraordinary Playlist on E4+1 last night but E4+1 was off air.  E4 was still on though so I don't know why they're incapable of doing the +1 thing. Had to source a copy online instead.

[Captain Z]

I've been watching 4Music for the Taskmaster repeats and that's all over the place, seemed to be working a couple of days ago but no adverts, but for the last two nights has been showing 'The Box' music channel instead.

[JaDanketies]

C5  was fucked for the seeming entire duration of Milkshake this morning. We watched Peppa Pig on the other channel instead. Literally lolled at the episode The Noisy Night.

Fireman Sam or whatever was squeezed into that tiny box, and the whole of the rest of the screen was taken up by that promo for the Milkshake Magazine. I thought it'd disappear after a minute but it was there permanently.

(picture is from Twitter and is not my TV)

[Sonny_Jim]

I'm awaiting the inevitable Register article about it.  Should be a good one, as it's mindboggling that they'd go TITSUP[nb]Total Inability To Support Usual Performance[/nb] for so long.

[buzby]

Odd, most 'important' places run halon systems, although I can't remember what the legality of those are in the UK as removing all the available oxygen in a room tends to make people dead.

Halon was banned by the EU in 2003 and I think generally replaced by CO2 systems. Apparently the sound levels and frequencies of gas discharge systems can wreck hard drives. Big data centres have systems designed to minimise such effects but if it was just a space in a normal office block then maybe it was using an inappropriate system.

CO2 isn't that widely used to replace Halon in datacentre fire suppresion systems. We have Fike automatic discharge systems in our datacentre and model hall which use HFC-227ea/heptafluoropropane gas (also known under the proprietary name FM-200), which is the generally-accepted replacement for Halon 1301 in gas discharge systems. The big difference between Halon, CO2 and HFC-227ea is that the former two gases displace oxygen to suppress the fire. HFC-227ea does not, so we didn't need to have emergency respirators put in.

Gas discharge systems that use inert gas like CO2 or Argon/Nitrogen mix need to have very large tanks to displace all the oxygen in the environment, and they need to discharge at high pressure/high volume which is where the ensuing sound level damage to electronic systems becomes an issue and is why they aren't a popular choice for datacentres. HFC-227ea only requires an atmospheric concentration of 6-9% to suppress fires, so you can have much smaller tanks and discharge at lower pressures.

[Pinball]

Given the CO2 shortage, will we have to choose between stunning animals and watching shite TV? Watch C4 and save a turkey? But which turkey?

[Ambient Sheep]

I've been watching 4Music for the Taskmaster repeats and that's all over the place, seemed to be working a couple of days ago but no adverts, but for the last two nights has been showing 'The Box' music channel instead.

Yeah, as documented here
https://cleanfeed.thetvroom.com/9693/preswatch/fallout-from-red-bee-broadcast-centre-incident-day-4/ (and their previous updates) via the Twitter link above.

I was going to watch Zoey's Extraordinary Playlist on E4+1 last night but E4+1 was off air.  E4 was still on though so I don't know why they're incapable of doing the +1 thing. Had to source a copy online instead.

Yeah, that's possibly the weirdest one.  Presumably their DR site doesn't have the ability to support it... seems mad though.


Thanks to buzby for the fire-suppresion info, most interesting.

[steveh]

There's regional variations of channels so they can do different ads which has meant in some places channels are working fine but in others they aren't. The +1s may even run different ads. E4+1 seemed to be okay in London but saw reports that elsewhere it was off.

[confettiinmyhair]

E4 played the day befores episode of trashy reality favourite Married at First Sight instead of the finale. Imagine BBC playing the penultimate episode of Line of Duty instead of the final episode.

[seepage]

Kept getting the "this content isn't available right now" error after each advert break on All 4 last night [trying to watch last week's Taskmaster] which I haven't had in ages. Entailed having to start the program from the beginning each time and fast forwarding.

[Captain Z]

Two advert breaks fucked up and froze during Taskmaster on Ch4+1 last night.

[Attila]

Here's what really happened:

What am I supposed to make out of this?

[Attila]

.

[Attila]

gah -- meant to edit not quote

[JesusAndYourBush]

E4+1 seemed to be okay in London but saw reports that elsewhere it was off.

When it was off for me (on monday, tuesday,wednesday, it's back now, I just checked) it was displaying a purple screen with the words (something like) "E4 off air" which was odd because E4 was fine.  On Monday after discovering E4+1 off air I chenged channel to 4 Music and instead of The IT Crowd it was showing music videos (Little Mix was the one I saw, I watched to the end of the song to see if it was filling ad breaks with music videos but then it went to another video.)

[pigamus]

gah -- meant to edit not quote

Attila is currently experiencing technical issues, we hope to restore him to full functionality as soon as possible

[Attila]

Attila is currently experiencing technical issues, we hope to restore him her to full functionality as soon as possible

_{I'm a she. :)}

[Uncle TechTip]

4Music were just relaying output from The Box, not sure if that's changed.

[Ambient Sheep]

https://twitter.com/bohaynowell/status/1444047796634542080