Main Menu

Tip jar

If you like CaB and wish to support it, you can use PayPal or KoFi. Thank you, and I hope you continue to enjoy the site - Neil.

Buy Me a Coffee at ko-fi.com

Support CaB

Recent

Welcome to Cook'd and Bomb'd. Please login or sign up.

March 28, 2024, 02:36:41 PM

Login with username, password and session length

C4 & C5 Down

Started by Blumf, September 25, 2021, 07:23:26 PM

Previous topic - Next topic

JesusAndYourBush

Just before The Last Leg started (during the end credits of Gogglebox) Channel 4 cut to a testcard for a few seconds.

EDIT: Ah, there's a photo of it further down that Twitter thread.

Sebastian Cobb

I'm surprised CH4 are essentially relying on outsourced playout so much.

When I worked for a certain national television company they were essentially rebuilding their entire MCR and playout facilities to do tapeless ad-insertion and playout from the same place for all regions. Essentially most of this time, excluding regional news opt-outs and very rare national programming a uk-wide network feed was sent up, and broadcast and the MAM jumped in for ad-insertion and local idents for all regions, then it was sent back down to BT tower to be distributed to freeview/sat/etc.

I think originally they had a full 'second base' dr plan scoped but that got canned by the suits and possibly a non-regional backup was planned instead.

The only time I saw disaster recovery in action was when the whole site lost power, due to a UPS swap out that went bad, they'd already primed BT tower to switch to borders so flipped to that when the shit hit the fan after a few minutes of darkness.

I was on-call for the vod system, so had nothing to do with any of that; our stuff was hosted in the cloud so all we lost were the live streams and the access to the CMS as although it was cloud-hosted it authenticated to a server in the building (windows single sign on is a fucking stupid idea, don't do it), the first I knew about it was a support guy from the encoder providers saying they'd lost network connectivity. Of course the power issue sent the voip phones wonky so I didn't know what was going on, but while the main channel was down there was no heat on me.

pigamus


Attila


Ambient Sheep

Yeah, this article mentions a "sonic wave" taking out the hard drives:

https://www.datacenterdynamics.com/en/news/uk-tv-broadcasts-disrupted-over-weekend-after-fire-suppression-system-triggered/

...but as it says:

QuoteWhen fire suppression systems are set off, they can cause a shock wave which can damage sensitive hard drives. This happened this year (July 2021) at betting site Tabcorp in Australia, and previously at ING bank in 2016, and at Glasgow City Council in 2015. However, this problem is well understood and should be receding by now, as suppression system makers now baffle their nozzles to prevent the shock wave.

Again, someone not getting their Xmas bonus.

Sonny_Jim

Nice to see Guile from Street Fighter II still working as a fire suppressant system for data centres.

Elderly Sumo Prophecy

Channel 5 now has this symbol in the upper right hand corner:


Is this related to the current situation, or has it always been there and I've never noticed it before?

Ambient Sheep

There was something I read a few days ago about various little symbols meaning the programme was being broadcast from the Disaster Recovery site, but then you'd think that would've been there for the last nine days if so.

Presumably it does indicate SOMETHING about the tramsmission chain, though.

Alberon

When I have been watching things on C4 since the apocalypse everything looks like the contrast has been whacked right up to maximum. What should be totally black ends up a light grey.

seepage

Last Friday's & yesterdays Four in a Bed not available on catch-up yet :(

Ambient Sheep

Quote from: Alberon on October 05, 2021, 09:44:03 AMWhen I have been watching things on C4 since the apocalypse everything looks like the contrast has been whacked right up to maximum. What should be totally black ends up a light grey.

Yeah, the black level is all wrong, apparently, according to those articles.  (And "brightness", not "contrast", to be pedantic.)

Weird fact, in 8-bit broadcast D1 video, black is 16 (0x10) and white is 235 (0xEB), to allow for digitising analogue signals overshooting the limits due to ringing.[nb]No idea why it's asymmetrical, rather than, say, 16-239 (0x10-0xEF).  If I ever knew why I've long forgotten.[/nb]  10-bit the same, only multiplied by 4.  I forget about HD now, but similar to 10-bit SD as I recall.  No idea about HDR as that was invented after I left the industry.

My guess is that there's some 0-255 v. 16-235 fuckery going on, well, the HD equivalent anyway.

Captain Z

4Music still hasnt recovered, at least not on freeview here. Some online/app TV listings are showing their usual schedule (Malcolm In The Middle, Taskmaster etc) but the channel is still showing The Box.

Alberon

Apparently ITV has crashed now. I can't be bothered to go check myself.

JamesTC

I doubt your average ITV viewer will be able to tell the difference.

JesusAndYourBush

Quote from: Captain Z on October 09, 2021, 08:31:54 PM
4Music still hasnt recovered, at least not on freeview here. Some online/app TV listings are showing their usual schedule (Malcolm In The Middle, Taskmaster etc) but the channel is still showing The Box.

At least on Freeview the guide tells you that they're showing videos because of the fault/disruption.

amateur

Channel 4 just borked again, froze during Countdown before cutting to an ad.

Blumf

Quote from: Alberon on October 11, 2021, 05:02:51 PM
Apparently ITV has crashed now. I can't be bothered to go check myself.

Something to do with C4 channels returning their bitrate to normal, apparently.

https://twitter.com/a516digital/status/1447657128135626752/photo/1

JesusAndYourBush

E4 was freezing last night, just for about 5-10 seconds then unfreezing, did it twice while I was watching.

mothman

For about the past couple of weeks,  whenever we switch over to Channel 4 the volume is much lower than other channels - we need to increase it by at least 25% to make it audible. But I don't see how the current issues could cause that..?

ElTwopo

Quote from: Ambient Sheep on October 02, 2021, 10:59:24 PM
...suppression system makers now baffle their nozzles...

I bet they do, the dirty bollocks

JesusAndYourBush

#50
Channels 33, 41, 49 & 51(the GREAT! Movies channels) are all scrolled up so the top of the picture (around a quarter) is at the bottom, with a black bar separating the 2 halves.  I waited until a commercial to see what happened and it was doing it then as well.

EDIT: It went back to normal a little after 9pm.

JesusAndYourBush

Quote from: Alberon on October 05, 2021, 09:44:03 AM
When I have been watching things on C4 since the apocalypse everything looks like the contrast has been whacked right up to maximum. What should be totally black ends up a light grey.

I'd not noticed that until today, but noticed this evening that on E4 and one of the other Ch4 -owned channels the brightness/contrast was weird.


Icehaven

Quote from: steveh on October 19, 2021, 10:49:17 AM
New Channel 4 statement on why they haven't yet got subtitles back: https://www.channel4.com/press/news/whats-happened-access-services-channel-4.

Strangely enough they've managed to get subtitles back on the adverts though, just not the shows.

Ambient Sheep

That's an interesting press release, thanks for that.  Implies that they're not going to go back with Red Bee once this is all done.  I wonder where this all-new facility they're building is?


Quote from: icehaven on October 19, 2021, 12:35:48 PMStrangely enough they've managed to get subtitles back on the adverts though, just not the shows.

Probably because the advertisers will be supplying them with the adverts, whereas, from the sounds of it, all the hard drives that had the subtitle files for their regular programming got nuked.  Question is, why wasn't there a back up set of the same files at the disaster recovery site?


thr0b

There was. The backup also failed.


steveh

The press release says that the system at the emergency backup failed. Saw something I think on Twitter that they and Channel 5 are having to re-source the subtitles for everything, which is a pretty major failing if true.

I didn't read it as them leaving Red Bee and that hasn't been reported elsewhere. There's a new Red Bee facility in Leeds I believe as part of Channel 4's move there. If that hasn't finished being commissioned then it would explain the reports of them not having a proper disaster recovery site at the time of the incident and having to manually play out programmes from another London facility instead.

buzby

Quote from: thr0b on October 19, 2021, 02:36:25 PM
There was. The backup also failed.
For that to happen the backup must have been a 'cold 'system. I wonder when the last time they tested it was.

That's why you have tape or offline backups, so you can rebuild the system from scratch if you have a double hardware failure. We still back everything up to LTO tape in our datacentre, which then get sent to a secure offsite facility.

Ambient Sheep

Cool, thanks for the info, steveh.