We did discover that the bit rate for UHD averaged out at around 25Mb/s per viewer. We added this in to the total bandwidth predictions for the World Cup across the BBC to see how many concurrent s we could comfortably and found that we would need to limit access to the trial. The standard HD and mobile streams were our first priority so we needed to ensure the trial didn’t take bandwidth needed for normal iPlayer and BBC Sport viewing.
We decided we’d be comfortable with a maximum number of around 60,000 concurrent UHD viewers, but that the number would be continually re-assessed during the tournament depending on overall performance. We also estimated there were maybe 2 million devices that might be capable of playing the UHD streams, hence the warnings about the streams being available on a first come, first served basis.
The Group Stages
The World Cup kicked off for us on the Friday 15th June as Egypt took on Uruguay. Trivia fans will want to know that the first ever UHD World Cup goal on the BBC was scored by Gimenez for Uruguay in the 89th minute.
The first couple of games ran relatively smoothly from a production point of view. We were closely monitoring from audiences, via a survey and also through social media. The audience response was generally very positive, especially with regard to the picture quality. There were some comments about stuttering and juddering and speech getting out of sync despite viewers having high speed internet.
In the v Australia game on the Saturday, some video didn’t arrive in time to be served to the audience. Unfortunately, the audio did arrive as DASH separates the two. This led to the audio becoming one segment out of sync - about four seconds. We failed over to the back-up. If you left the stream and came back, the issue was fixed, but if you stayed on the stream it remained faulty.
In the later game between and Mexico, we had a router switch fail which caused more issues.
The quality of service that UHD demands of network infrastructure quickly roots out any problems, or components that are introducing latency. We replaced some optical connectors and noticed significant improvements in our logs and a reduction in complaints.
While we were fixing switches and networks, we took some other precautionary measures. We removed the lowest bit rate to give the distribution encoders less data to generate in each time period and we increased the wait time on the packagers. This increased our latency by a further 10 seconds. Not ideal but we wanted to keep the audience experience as smooth as possible.
We’d also noticed a number of comments on the size of the “BBC UHD” logo in the top right hand corner of the screen. This was set to a DPP (Digital Production Partnership) guideline but we all agreed it did look a bit big, so we reduced the size by 10 percent. We changed that on the Tuesday of the first week.
On the day of the first England game, 18th June, one of our contribution encoders had an issue during the preceding Belgium vs Panama match so we removed it from the chain and switched to the back-up. We used that for the rest of the tournament.
The England vs Tunisia game proved to be our most streamed live event up to that date. We stopped access to UHD in the first half for a short period of time as we re-balanced traffic across our CDNs to optimise performance. Things calmed down a bit and we re-enabled UHD access at half time. We had just over 35,000 UHD viewers at this point but were shipping around 5Tbits of data per second across all services.
We continued to work on our distribution and found and fixed further issues with networks and infrastructure along the distribution chain during the next few matches as we discovered them.
As the technical situation settled down, we felt confident enough to let the presentation team on BBC One mention the fact that there was UHD available. This was done by the commentators and an on-screen graphic. Doing so resulted in an immediate spike of 10,000 new viewers within a few seconds.
Two days later (27th of June) we had a serious external networking incident that caused a problems for viewers such as buffering and streams failing. We let the stream run, but there were only a few thousand watching. Things recovered for the second half of the South Korea vs game. We asked Sport not to mention us for a little while just in case.
Round of 16/Wimbledon
As we moved in to the round of 16 in the World Cup, we added Centre Court coverage from Wimbledon. This added a bit of complication as we didn’t have enough UHD encoders to provide 2x resilience for both events. This meant a degree of source swapping for the back-ups.
As the network issues had now been resolved, we put back the lowest bit rate and reduced our packager waiting time by 20 seconds. This got our latency down to around 45 seconds - but it was very dependent on what device you were using.
Audiences started to increase, both for UHD and in general. For the Belgium vs. Japan game, we peaked at 37,300 concurrent UHD viewers, including football and Wimbledon.
Quarter Finals
Things started to hot up now. For the quarter final between Brazil and Belgium on Friday 6th July, we had 41,600 concurrent UHD viewers at the peak (including Wimbledon viewers). We still hadn’t hit our theoretical maximum limit by full time though.
England vs. Sweden was on Saturday 7th June with a 3pm kick off. We hit our UHD cap of just over 60,000 concurrent viewers half way through the first half. We pretty much stayed at this level for the rest of the match.
Semi Finals
We ran UHD for the vs. Belgium semi final with no issues and just over 48,000 concurrent UHD viewers. The men’s semi-finals at Wimbledon provided a good soak test for our encoders and staff with play continuing until just after 11pm.
The Finals
The tennis final on Saturday 14th, and tennis and football finals on Sunday 15th July ran smoothly. A peak of 44,300 viewers saw Novak Djokovic and win their respective tournaments, including the first penalty awarded with the help of VAR in a World Cup final. We don’t know if the VAR was in UHD or not.
We stopped the last stream around 7pm and put live UHD to bed for now.
Extra Time
As well as the technical issues we addressed at the time, there were a couple of other common comments that I’d like to address now.
Latency
The elephant in the room for live internet streaming is latency. Well, it’s in the room if you’re watching on Freeview. If you’re online it’ll be a minute or two yet.
Latency is less of an issue for everyday viewing, but it comes in to sharp focus when we look at sport. The usual latency for online streaming is between 30 and 90 seconds depending on your viewing device. For UHD we got latency down to between 45 seconds and a few minutes but it was very variable, again depending on the device. Interestingly, when we asked viewers about latency in our survey, while they wanted it to be as low as possible, most said they wouldn’t trade it off against picture quality.
The problem is one of scale versus speed. It is possible to stream video very quickly over the internet. It’s used all the time for video calling. The problems with low latency streaming are that it generally only s small numbers of viewers, or it requires very proprietary hardware and software, or there is a significant reduction in quality or resilience. As more and more audiences move online this is becoming a significant issue for the whole industry.
Fortunately, it is also an issue the industry is beginning to address, both through developments by hardware and software vendors, and through the work of standards groups which the BBC is contributing to. Ultimately, we need to get the latency of IP delivery comparable to that of linear distribution and we, in BBC Platform and OTG have already started working through those issues with our partners in the industry.
It’s too dark
A couple of themes that came out of the were around the picture being too dark or the colours not being vibrant enough. Having worked with the production teams we think we have managed to improve some of the issues we found e.g. ensuring graphics are inserted at the correct level. For a more detailed assessment of some of the possible causes at the home end then please see this blog written by my R&D colleague Andrew Cotton.
It’s Coming Home
The UHD trial was a brilliant learning experience for us so we definitely consider it a success. It’s clear there’s lots more to learn, work on and improve, and massive events like the World Cup and Wimbledon really put our systems through their paces. We hope those that watched it had a good experience, and sincerest apologies for when it didn’t go so well. Our TV in the office buffered just as scored in the semi-final and I was livid so I do understand how it can be frustrating at times. Thank you for trying it if you did.
We built the trial in such a way that it can be re-used and improved upon. This will allow the next trial, whatever it may be, to take advantage of all our experiences and fixes we’ve implemented this time around. Live streaming Ultra HD over the internet is at a very early stage, but we’ve taken an important step with this trial. We will continue to build our capability and make sure future audiences have a free-to-air option for live Ultra HD and HDR.
