Cutting-Edge Face Replacement ... on Ice

Blades of Glory's 3D Face-Switching on a Budget

Although the slick visual effects in DreamWorks’ figure-skating flick Blades of Glory easily slid past audiences, the filmmakers for this box-office hit would have been skating on thin ice without the skill of Rainmaker Visual Effects. Rainmaker’s Visual Effects Supervisor Mark Breakspear led the 150-person team that filled 3D stadiums with digital crowds and gave the four main actors Olympic-class skating moves — or rather, convinced the audience these skaters, not their stunt doubles, could perform an Olympian’s triple salchows, double axels, spins, lifts, and spirals.

Click the link below to see a Flash presentation (broadband recommended) featuring Rainmaker's work on the film, including videos and before-and-after slides.

Watch the Video Slideshow

Directed by Will Speck and Josh Gordon, the film stars Will Ferrell and Jon Heder as an unlikely male figure-skating pair. Writes Kevin Crust of the Los Angeles Times, "Whatever combination of choreography, camera trickery and special effects were required to render the over-the-top, hyper-real skate numbers, they're executed with wit and ingenuity.”

That wit and ingenuity was due, in part, to a lucky break. “Jon Heder cracked an ankle bone during practice, so we had a three-month hiatus while he healed,” says Breakspear. “That gave us an opportunity to previs every shot in the skating sequences, and we came up with new moves.”

To make the moves work, Rainmaker used new, state of the art techniques to replace the four stunt skaters’ faces with the actors’ faces. The techniques rely on a blend of commercial software and proprietary tools. “People have done fantastic face replacement in CG on mega-movies, but we didn’t have four years or a blockbuster budget,” Breakspear says. At first, the studio pushed for a 2D solution, but Breakspear convinced the directors that switching faces in 2D would limit their ability to shoot the skaters and restrict the skaters’ movements.

Fortunately, Rainmaker had a gold-medal face-replacement expert on its team, CG supervisor Kody Sabourin, who had worked on the legendary “Super Punch” shot in The Matrix: Revolutions. In that shot, accomplished with proprietary technology, Agent Smith’s face deforms in an extreme close-up when Neo lands a punch.

“Given my experience on the Matrix films and from working at ESC with people like George Borshukov and Kim Libreri, I knew the best way to approach this project was with animated textures that we captured at the same time we captured the performance,” he says.

Re-inventing Face Replacement

At first, Rainmaker tried to hire a studio or company to do the work. “We couldn’t find anyone we could outsource from,” Sabourin says. “People could capture the performance, but not the textures, or if they captured textures, the textures were low-res and not usable. Or, they’d texture-map a still frame on an animated mesh. I knew, based on my experience on Reloaded and Revolutions, that wouldn’t work.” As a result, Sabourin led a team at Rainmaker that developed the process used for Blades of Glory.

They began by taking digital photographs and making plaster casts of the actors’ faces. Then, they had the casts scanned in high resolution at XYZ RGB in Ottawa. Using the photographs and a 3D model created from the scan data in Autodesk’s Maya, visual effects artists determined where to place 102 tracking markers on each face. Before the performance capture session, makeup artists copied each marker’s position onto each actor’s face.

To capture the actors’ facial performance and animated textures simultaneously, Rainmaker devised a clever set up using three HD cameras, one film camera, and mirrors. “Imagine that the actor is sitting in a chair and around him are four cameras about two meters away,” says Breakspear. “Above and behind him are mirrors.”

The team captured the tracking markers with the three HD cameras, and recorded textures during the actors’ performances with the film camera, which they positioned in the center. Breakspear put lights behind silks to capture the faces with completely flat lighting. “Using the film camera was something that has never been done before,” Sabourin says. “Not even on the Matrix films. We could scan the film and have textures at 4K resolution if we needed, whereas with HD, we were capped at HD resolution.”

The actors performed to footage of his or her stunt double. “We had split screens on set so the actors could watch the stunt doubles in the plate and then watch themselves reacting,” says Sabourin. “The director had free reign.”

The mirrors doubled the data collection and provided textures from multiple angles. “With mirrors, rather than using more cameras, we could split one image into three,” says Sabourin. “We got textures from the front and both sides of the face, and an extra three cameras tracking the performance.” To match the frames once the cameras rolled, a time code generator linked all the cameras.

The effects team tracked the markers using RealViz’s MatchMover Pro and triangulated the actors’ facial motion into 3D space. “We ended up with 102 dots that looked somewhat like the actors’ faces moving in 3D space,” Sabourin says. “That is, you could make out many of the moving shapes, like their ears, throat, lips, eyes, and nose.”

Next, they applied the animation from the dots to animatable meshes created from the high resolution XYZ RGB scans of the actors’ facial casts. Proprietary code helped them transpose as much detail from the actor’s performance as possible. “That gave us an animated face mesh,” Sabourin says.

Weight maps helped make that movement believable. “We painted weight maps – grayscale texture maps – to define range of movement,” says Breakspear. The weight maps controlled how much influence each dot had over the geometry. “The skin on the nose doesn’t move far, so the weight map would be tight there,” explains Breakspear. “But a weight map for the cheek would be squashy.” Thus, if an anomalous dot tried to stray from reality, the weight maps pinned it in place.

The result gave them fully CG versions of the actors’ faces performing as they did on set. “It was a breakthrough,” says Breakspear of the team’s solution. “A eureka moment.”

Getting the Texture

After a final cleanup of the performance using hands-on animation — a tweak to animate the lips or the eyes a bit more, perhaps — the effects artists began texturing the faces. “We had textures captured at 24 frames per second that matched the performance we captured,” says Sabourin, “So we applied those animated textures to the mesh.” To tidy the textures and remove the tracking markers, the artists used Eyeon Software’s Digital Fusion, Right Hemisphere’s Deep Paint, and Adobe’s Photoshop.

Then, especially for close-up shots, they extracted high-resolution displacement maps from the XYZ RGB scans to add such details as scars and pores. “That, combined with the animated texture maps, gave us wrinkles,” says Sabourin. “Because we had 150 face replacements with unique performances to do, we gave the faces that were far away only the bare minimum, but a lot of the shots were close-ups.”

To match the lighting on the digital face to lights used when filming the skating doubles, Breakspear relied on HDRI. “For every shot of the doubles, we had shots of a gray ball, a chrome ball, and images taken with a fish eye camera,” he says. Even so, the lighting team faced unique challenges. “The ice created a big bouncing light environment. Our lighting team did a fantastic job making the digital faces look right.” For rendering, Rainmaker used Mental Ray.

Crowd Control

In addition to the facelifts for the skaters, Rainmaker also did digital surgery on the LA sports arena where the directors shot all live-action footage, but all the action in the film takes place in three other skating rinks in three different locations.

“Creating full CG stadium replacements with CG crowds was roughly half our work,” says Breakspear. Although the studio had devised a method to create crowds in 2D for the film She’s the Man [see F&V's coverage here], the camera moves needed for the swirling skating sequences dictated a 3D solution.

During the live-action shoot, 800 people and 5000 inflatable stuffies sat in the stands. “The stuffies worked well for a blurring background,” says Breakspear. “They look like real people. But in some shots we needed to have the people moving and clapping.”

For those shots, Rainmaker either replaced everything in the L.A. arena from the second level of seats up or created fully 3D stadiums by building digital sets in Maya. For the crowds, the studio used Massive software, building brains that signaled the fans to clap, cheer, stand up, sit down, walk through the aisles, and even order hot dogs. The CG characters moved using animation cycles created from motion data captured at the studio’s new division, Mainframe Entertainment, which Rainmaker acquired during 2006.

Beware of the 'Iron Lotus'

The climax of the film revolves around the “Iron Lotus,” a secret move never before successfully performed, that the characters played by Farrell and Heder try in front of a crowd of 15,000 screaming fans. “It has never been performed before because if a skater gets it one percent wrong, it could decapitate him,” says Breakspear. “We dreamed up this move during previz. When we explained it to our professional skating doubles, they wouldn’t speak to me for a week. They were teasing me. We all had a good laugh making it.”

Creating the Iron Lotus moves for the film involved face replacements, rig removals, CG crowds, CG stadiums, and green-screen composites. “Everything came together,” says Breakspear. “It was so dramatic. John Heder grabs Will’s leg and throws him in the air. Will spins in the air backwards. Then, John puts one leg in the air, does his own spin and jumps into the air ... and ... it’s so dramatic.”

That people walk out of the theater laughing, not wondering how the actors could possibly have performed their skating moves is a tribute to the VFX moves performed at Rainmaker. “Without our new face-replacement technology, we couldn’t have done it,” says Breakspear.