Irish director Catherine Owens arrived at New York visual effects house Spontaneous armed with Greek and Roman paintings, all reference material for a video U2 had commissioned for its live show. One painting depicting a woman with plants trailing out of her mouth tied directly into the lyrics of “Original of the Species,” a song about realizing and celebrating inner beauty. Spontaneous, which had worked with Owens and U2 in the past on lighting schemes and videos to be shown during concerts, used Maxon’s Cinema 4D to mock up animation of a woman with flowers that grow out of her mouth and circle her head to become a crown. “Everything took off from there,” says Spontaneous’ John Leamy, creative director and head of design. “The band got excited about making a video [as opposed to only using the animation during performances] and ultimately wanted to be in the environment with the woman.”

Lawrence Nimrichter, associate creative director and director of animation, suggested making 3D models of the band and motion-capturing Bono’s face so that he could sing to the woman within the 3D space. With only six weeks of production time available, the group thought back to a demo for Softimage’s Face Robot software, which they had first spied at Siggraph.

“It was just amazing stuff we had never seen before,” says Nimrichter. “It was a way of doing facial animation using motion capture that didn’t look stiff and unnatural, which is oftentimes how motion capture/facial animation tends to look in 3D. When it came time to start talking about doing this video and the concept around it being just faces, we thought this would be the perfect thing to use it for.”

Spontaneous got in touch with Softimage’s special projects group and, within days, Owens and Spontaneous were on a plane to meet the band in Chicago along with Dilip Singh from Softimage and Jeff Wilson from Blur Studios, who would both be wrangling the new technology. Also on hand in Chicago was Ottawa-based XYZ RGB.

“XYZ RGB have a very high-definition scanning technology, one that allows you to wave what looks a little bit like a supermarket check-out wand over a person’s facial geometry and collect a billion points of data to get a very dense and accurate mesh of the person or any object that you put in its path,” says Leamy. Each band member was laser-scanned, as was the head of Bono’s personal assistant, a woman named Katrina, who became the model for the woman who appears in the video.

Los Angeles-based House of Moves was also present to motion-capture Bono’s facial performance, capturing two performances of him singing as well as two passes of Katrina doing basic emotive gestures with her eyes and mouth.

With limited time, Owens, Spontaneous, and crew then did what Leamy deems “a down-and-dirty shoot” of Bono singing the song. Unhappy with the Chicago footage, they eventually re-shot the live segment in an extra room at Madison Square Garden with DP Tom Krueger. Director Mark Pellington, a friend of Owens who became a consultant during the project, re-transferred the live-action footage and volunteered some aesthetic treatments that gave the team more looks to work with.

Back in New York, Spontaneous began building a library of Katrina environments and elements in Softimage, including flowers and plants. R&D was done on how butterflies fly, and butterfly textures were scanned. But the main hurdle was realizing the models of the band in a compelling way.

The original plan was to use the scans of Bono to create a displacement map that would effect a low-polygon cage that would sync with Face Robot. “The result was a little too realistic and not magical enough,” says Leamy.

Instead, the team rebuilt Bono polygon by polygon, using the scanned technology as a template. “That model then had to be reconfigured to sync up with Face Robot’s specific cage structure,” says Leamy. “That was a bit of a hurdle ‘ trying to achieve a model that was going to work with Face Robot, but that would also be aesthetically pleasing and look enough like Bono that Bono would be happy.”

By combining the motion capture with Face Robot, the team was able to achieve fluid and natural-looking facial animation. “It felt like you had muscles and tissue moving and stretching,” says Nimrichter. “When you’re within Face Robot and you’re manipulating certain control points on the face, they are aligned to markers for the mocap. So if you pull a piece around the mouth, it ends up pulling the skin and the muscles all around the mouth in a very natural way, just like the muscles in the face would allow. You don’t have to go in and manually pull points all over the face to get this kind of animation.” Face Robot also allowed Spontaneous to easily adapt animation based on motion capture of Bono and transfer it onto Katrina’s face.

“I don’t know if we would’ve been able to do [the facial animation] very convincingly within the time that we had, without going this route,” says Nimrichter. “But Face Robot ended up working for many reasons. The fact that it uses the mocap was a big plus, but just to be able to get really good animation in a short amount of time was fantastic. Once we got our selects and we really went into animating, we had some finished shots two weeks later.”

The team exported Face Robot files as animation clips into XSI. “We were then able to re-time and time-stretch in the facial animation using the animation mixer. We were able to blend between different takes of mocap that had already been processed through Face Robot using XSI’s animation mixer. That was a key part of the pipeline as well,” says Nimrichter.

Visual effects director Andy Milkis did all compositing in Inferno, which he finds to be the perfect tool given its responsiveness with large amounts of data. “As a compositor, my work is only as good as the pieces that I get. I was never short for elements, pieces or passes. It didn’t matter if I needed 20 passes from CG for a shot; I always had them. It didn’t matter what the band threw at us at the last minute-whether it was a tear that had to come out an eye and roll down a cheek, or shadows-somehow they always made it out of the render farm exactly when I needed them.”

U2’s way of saying thank you? “We got free tickets to the last concert at Madison Square Garden,” says Milkis. “That was the ultimate payoff. We had just finished a 40-hour session, delivered it, and then all got into cars and went over to the Garden. And right before the band performed the song, Bono gave us all a shout out. I believe he called it a work of fine art. It was incredible.”