Author Topic: C4 & C5 Down  (Read 3137 times)

JesusAndYourBush

  • Earnest silky coconut shell
    • http://www.google.com
Re: C4 & C5 Down
« Reply #30 on: October 01, 2021, 11:15:00 PM »
Just before The Last Leg started (during the end credits of Gogglebox) Channel 4 cut to a testcard for a few seconds.

EDIT: Ah, there's a photo of it further down that Twitter thread.

Sebastian Cobb

  • bad opinion haver
Re: C4 & C5 Down
« Reply #31 on: October 02, 2021, 12:12:28 AM »
I'm surprised CH4 are essentially relying on outsourced playout so much.

When I worked for a certain national television company they were essentially rebuilding their entire MCR and playout facilities to do tapeless ad-insertion and playout from the same place for all regions. Essentially most of this time, excluding regional news opt-outs and very rare national programming a uk-wide network feed was sent up, and broadcast and the MAM jumped in for ad-insertion and local idents for all regions, then it was sent back down to BT tower to be distributed to freeview/sat/etc.

I think originally they had a full 'second base' dr plan scoped but that got canned by the suits and possibly a non-regional backup was planned instead.

The only time I saw disaster recovery in action was when the whole site lost power, due to a UPS swap out that went bad, they'd already primed BT tower to switch to borders so flipped to that when the shit hit the fan after a few minutes of darkness.

I was on-call for the vod system, so had nothing to do with any of that; our stuff was hosted in the cloud so all we lost were the live streams and the access to the CMS as although it was cloud-hosted it authenticated to a server in the building (windows single sign on is a fucking stupid idea, don't do it), the first I knew about it was a support guy from the encoder providers saying they'd lost network connectivity. Of course the power issue sent the voip phones wonky so I didn't know what was going on, but while the main channel was down there was no heat on me.

pigamus

  • Member
  • **
  • Sex, death, mayonnaise.
Re: C4 & C5 Down
« Reply #32 on: October 02, 2021, 02:39:09 AM »

Attila

  • gif made by hedgehog90
Re: C4 & C5 Down
« Reply #33 on: October 02, 2021, 10:08:19 AM »
Sorry about that

No worries at all x

Re: C4 & C5 Down
« Reply #34 on: October 02, 2021, 10:59:24 PM »
Yeah, this article mentions a "sonic wave" taking out the hard drives:

https://www.datacenterdynamics.com/en/news/uk-tv-broadcasts-disrupted-over-weekend-after-fire-suppression-system-triggered/

...but as it says:

Quote
When fire suppression systems are set off, they can cause a shock wave which can damage sensitive hard drives. This happened this year (July 2021) at betting site Tabcorp in Australia, and previously at ING bank in 2016, and at Glasgow City Council in 2015. However, this problem is well understood and should be receding by now, as suppression system makers now baffle their nozzles to prevent the shock wave.

Again, someone not getting their Xmas bonus.

Sonny_Jim

  • Kirkladn has a small penis
Re: C4 & C5 Down
« Reply #35 on: October 03, 2021, 03:30:27 AM »
Nice to see Guile from Street Fighter II still working as a fire suppressant system for data centres.

Elderly Sumo Prophecy

  • Your sleep paralysis demon
Re: C4 & C5 Down
« Reply #36 on: October 05, 2021, 01:22:05 AM »
Channel 5 now has this symbol in the upper right hand corner:


Is this related to the current situation, or has it always been there and I've never noticed it before?

Re: C4 & C5 Down
« Reply #37 on: October 05, 2021, 03:55:37 AM »
There was something I read a few days ago about various little symbols meaning the programme was being broadcast from the Disaster Recovery site, but then you'd think that would've been there for the last nine days if so.

Presumably it does indicate SOMETHING about the tramsmission chain, though.

Alberon

  • His heart is an empty fridge
Re: C4 & C5 Down
« Reply #38 on: October 05, 2021, 09:44:03 AM »
When I have been watching things on C4 since the apocalypse everything looks like the contrast has been whacked right up to maximum. What should be totally black ends up a light grey.

seepage

  • Undetectable in normal everyday use
Re: C4 & C5 Down
« Reply #39 on: October 05, 2021, 10:08:13 AM »
Last Friday's & yesterdays Four in a Bed not available on catch-up yet :(

Re: C4 & C5 Down
« Reply #40 on: October 05, 2021, 07:54:35 PM »
When I have been watching things on C4 since the apocalypse everything looks like the contrast has been whacked right up to maximum. What should be totally black ends up a light grey.

Yeah, the black level is all wrong, apparently, according to those articles.  (And "brightness", not "contrast", to be pedantic.)

Weird fact, in 8-bit broadcast D1 video, black is 16 (0x10) and white is 235 (0xEB), to allow for digitising analogue signals overshooting the limits due to ringing.[1]  10-bit the same, only multiplied by 4.  I forget about HD now, but similar to 10-bit SD as I recall.  No idea about HDR as that was invented after I left the industry.

My guess is that there's some 0-255 v. 16-235 fuckery going on, well, the HD equivalent anyway.
 1. No idea why it's asymmetrical, rather than, say, 16-239 (0x10-0xEF).  If I ever knew why I've long forgotten.

Captain Z

  • Die Hard or Die Tryin'
Re: C4 & C5 Down
« Reply #41 on: October 09, 2021, 08:31:54 PM »
4Music still hasnt recovered, at least not on freeview here. Some online/app TV listings are showing their usual schedule (Malcolm In The Middle, Taskmaster etc) but the channel is still showing The Box.

Alberon

  • His heart is an empty fridge
Re: C4 & C5 Down
« Reply #42 on: October 11, 2021, 05:02:51 PM »
Apparently ITV has crashed now. I can't be bothered to go check myself.

JamesTC

  • Helms calls for donut tax
Re: C4 & C5 Down
« Reply #43 on: October 11, 2021, 05:06:00 PM »
I doubt your average ITV viewer will be able to tell the difference.

JesusAndYourBush

  • Earnest silky coconut shell
    • http://www.google.com
Re: C4 & C5 Down
« Reply #44 on: October 11, 2021, 05:35:00 PM »
4Music still hasnt recovered, at least not on freeview here. Some online/app TV listings are showing their usual schedule (Malcolm In The Middle, Taskmaster etc) but the channel is still showing The Box.

At least on Freeview the guide tells you that they're showing videos because of the fault/disruption.

amateur

  • Not feeling it, tbh
Re: C4 & C5 Down
« Reply #45 on: October 12, 2021, 02:43:17 PM »
Channel 4 just borked again, froze during Countdown before cutting to an ad.

Blumf

  • Not long now
    • IGNORE ME!!!
Re: C4 & C5 Down
« Reply #46 on: October 12, 2021, 02:56:39 PM »
Apparently ITV has crashed now. I can't be bothered to go check myself.

Something to do with C4 channels returning their bitrate to normal, apparently.

https://twitter.com/a516digital/status/1447657128135626752/photo/1

JesusAndYourBush

  • Earnest silky coconut shell
    • http://www.google.com
Re: C4 & C5 Down
« Reply #47 on: October 12, 2021, 02:57:17 PM »
E4 was freezing last night, just for about 5-10 seconds then unfreezing, did it twice while I was watching.

mothman

  • I don't know why
Re: C4 & C5 Down
« Reply #48 on: October 12, 2021, 06:10:55 PM »
For about the past couple of weeks,  whenever we switch over to Channel 4 the volume is much lower than other channels - we need to increase it by at least 25% to make it audible. But I don’t see how the current issues could cause that..?

Re: C4 & C5 Down
« Reply #49 on: October 12, 2021, 08:38:50 PM »
...suppression system makers now baffle their nozzles...

I bet they do, the dirty bollocks

JesusAndYourBush

  • Earnest silky coconut shell
    • http://www.google.com
Re: C4 & C5 Down
« Reply #50 on: October 15, 2021, 08:26:48 PM »
Channels 33, 41, 49 & 51(the GREAT! Movies channels) are all scrolled up so the top of the picture (around a quarter) is at the bottom, with a black bar separating the 2 halves.  I waited until a commercial to see what happened and it was doing it then as well.

EDIT: It went back to normal a little after 9pm.
« Last Edit: October 15, 2021, 09:10:14 PM by JesusAndYourBush »

JesusAndYourBush

  • Earnest silky coconut shell
    • http://www.google.com
Re: C4 & C5 Down
« Reply #51 on: October 19, 2021, 12:19:31 AM »
When I have been watching things on C4 since the apocalypse everything looks like the contrast has been whacked right up to maximum. What should be totally black ends up a light grey.

I'd not noticed that until today, but noticed this evening that on E4 and one of the other Ch4 -owned channels the brightness/contrast was weird.

Re: C4 & C5 Down
« Reply #52 on: October 19, 2021, 10:49:17 AM »
New Channel 4 statement on why they haven't yet got subtitles back: https://www.channel4.com/press/news/whats-happened-access-services-channel-4.

icehaven

  • WORLD'S BIGGEST RIP OFF: $100 PER PERSON
Re: C4 & C5 Down
« Reply #53 on: October 19, 2021, 12:35:48 PM »
New Channel 4 statement on why they haven't yet got subtitles back: https://www.channel4.com/press/news/whats-happened-access-services-channel-4.

Strangely enough they've managed to get subtitles back on the adverts though, just not the shows.

Re: C4 & C5 Down
« Reply #54 on: October 19, 2021, 02:20:58 PM »
That's an interesting press release, thanks for that.  Implies that they're not going to go back with Red Bee once this is all done.  I wonder where this all-new facility they're building is?


Strangely enough they've managed to get subtitles back on the adverts though, just not the shows.

Probably because the advertisers will be supplying them with the adverts, whereas, from the sounds of it, all the hard drives that had the subtitle files for their regular programming got nuked.  Question is, why wasn't there a back up set of the same files at the disaster recovery site?


Re: C4 & C5 Down
« Reply #55 on: October 19, 2021, 02:36:25 PM »
There was. The backup also failed.

Re: C4 & C5 Down
« Reply #56 on: October 19, 2021, 02:49:58 PM »
Quality.

Re: C4 & C5 Down
« Reply #57 on: October 19, 2021, 02:56:38 PM »
The press release says that the system at the emergency backup failed. Saw something I think on Twitter that they and Channel 5 are having to re-source the subtitles for everything, which is a pretty major failing if true.

I didn't read it as them leaving Red Bee and that hasn't been reported elsewhere. There's a new Red Bee facility in Leeds I believe as part of Channel 4's move there. If that hasn't finished being commissioned then it would explain the reports of them not having a proper disaster recovery site at the time of the incident and having to manually play out programmes from another London facility instead.

buzby

  • Member
  • **
Re: C4 & C5 Down
« Reply #58 on: October 19, 2021, 03:08:05 PM »
There was. The backup also failed.
For that to happen the backup must have been a 'cold 'system. I wonder when the last time they tested it was.

That's why you have tape or offline backups, so you can rebuild the system from scratch if you have a double hardware failure. We still back everything up to LTO tape in our datacentre, which then get sent to a secure offsite facility.

Re: C4 & C5 Down
« Reply #59 on: October 19, 2021, 06:29:36 PM »
Cool, thanks for the info, steveh.

Tags: