Inside Fox Sports Virtual Fans Effort with SMT, Silver Spoon, and Pixotope
Fox Sports coverage of the 2020 MLB season has featured one of the more interesting solutions for dealing with a stadium without fans: add in digital fans. And Dan Pack, managing director and partner of Silver Spoon, the company that gives each of the digital fans life via its Crowded software (with a tech assist from Pixotope and SMT), says the goal is to both fill the stands of MLB games and lay the groundwork for more augmented reality within a live sports production.
“The idea is to augment the broadcast, not take away from it,” he says of the efforts. “We want to show broadcasters that there is a lot of value in using AR.”
Mike Davies, Fox Sports, SVP, technical and field operations, says the visual difference of seeing an empty venue vs. seeing a venue full of fans, even if they are virtual, is striking.
“The big thing is that Fox isn’t trying to fool anyone here as everyone knows there are no fans,” he says. “But we are in the business of making a better presentation, and I think that this technology accomplishes this goal.”
Kirk Brown, SMT, COO, says that Fox, SMT, Pixotope, and Silver Spoon came together in a fairly intricate collaboration bringing together SMT’s technology that allows virtual graphics to be inserted with geospatial precision into the broadcast, Pixotope’s platform, based on the Unreal Engine 3D graphics real-time processor, and the Silver Spoons software allows for thousands of fans to be dynamically created and controlled instantly.
“For this project we have four cameras at every venue where we are taking the pan, tilt, zoom and focus from each camera and doing image stabilization,” says Brown. “That gives rock-solid information to Pixotope which then takes the animated graphics from Silver Spoon and places them in the precise location in the stadium so we can make the effects look as real as possible.”
The trick is to make sure the virtual fans are placed accurately within their seats. That process begins by taking scans of each stadium from MLB and Fox Sports that show exactly where the stands, tunnels, and other physical elements are located relative to seats.
“We performed the same thing for ‘The MLS is Back’ providing a series of virtual jumbotrons and virtual advertising on the far side and ends of the field,” says Brown.
Pack says the work that led to the Fox Sports implementation actually began for a different sport and a different network.
“At first we were half joking about it, but we kept cracking at it and had a little bit of time to develop and put this together for Fox,” he says.
The Crowded software runs in real time and at 60 fps and is produced onsite at each ballpark. Each character Silver Spoon creates begins with motion capture that records various states a fan will experience during a game.
Silver Spoon does a tremendous amount of motion capture in its studio and that is where the process of creating virtual fans begin: by capturing a variety of motions and reactions from fans.
“You start with a neutral position and if there is a pop up to the outfield the crowd stands up and they can start cheering or booing,” he explains. “It all has to happen in a way that looks natural and isn’t jarring.”
He describes the process of animating each character a bit like pulling puppet strings.
“It’s fun and built in a way that is scalable and customizable as we pre-load all the approved logos and build out a series of outfits, shirts, pants, etc. and then we can customize those on the fly with colors, and skin tones,” he explains. “And everything with crowd simulation is about making it feel random but at the same time you have control over everything.”
Director, the Pixotope system, is based on the Unreal Engine 3D graphics real time processor. Pack says the graphics GUI allows operators to communicate with all of the Unreal Engine machines at one time. Pack says the Pixotope Director platform is great because it can process multiple cameras at once so that cuts from one of the four cameras to another can be seamless and done much more efficiently.
As for getting the fans into the stands it all begins with the scan, tagging all of the seats, and then narrow down the virtual crowd to fit into the individual section.
“If there are any discrepancies we can paint in or erase part of the crowd by just moving the mouse over it,” he adds.
Even when fans return to the ballpark there will be uses for more AR technologies. Virtual mascots can be branded and placed into real-world environments. And things like “moments of exclusivity” where a stadium is Coca-Cola red or Pepsi blue can be taken to new heights. And, Super Bowl halftime shows will never be the same.
“We want to make it respectful of the game and also more encompassing,” says Pack.
Pack says the next goal is to reduce the footprint required on site, including the ability to do the insertion from a remote location.
“It’s all about lowering the barrier to entry,” he says. “Right now, the big issue is the hardware required to run it and if we can centralize the production broadcasters can use more machines at a lower cost per game.”
The current offering shows the work that still needs to be done (and eventually will be done given the trajectory of technology and processing power) to bring virtual fans to every corner of a ballpark. For example, placing virtual fans behind home plate needs another leap of capabilities so that the foreground objects, like the batter, umpire, and pitcher, can be separated from the background object, like the seats, and then virtual fans can be placed between the foreground and background objects.
Adds Brown: “A great team of people have put this together and no one has delivered this quality of live broadcast animated graphics with this level of precision before. It’s been exciting to do something new and challenging.”