Henson’s Creature Shop and Real-Time Animation

At an event at Jim Henson’s Creature Shop, on the old Charlie Chaplin lot in Hollywood, Brian Henson greeted an enthusiastic group from the L.A. chapter of Siggraph and the Visual Effects Society. The topic was the real-time animation that the Creature Shop has successfully launched, most notably with PBS’ children’s series Sid the Science Kid. The event was sponsored by Autodesk, and both Siggraph L.A. co-chair Joan Collins Carey and VES executive director Eric Roth welcomed the crowd.

Suit performer for Sid the Science Kid

Jim Henson’s son Brian, whose appearance at the event was unexpected, described his father as a “technology-excited” person, who threw out his 8-track and added a brand new cassette player when they came with a microphone. Jim Henson was already thinking about the possibilities of computer animation in 1989, as was evident from an archived TV clip of “Waldo” shown to the crowd. “A two-minute piece took 120 hours to render on the computer,” said digital effects supervisor Steffen Wild. “It took another ten years for computers to catch up enough to create a image in real time.”

The possibility of directing animation exactly like live-action has been a sought-after goal among some animators, and Henson championed that possibility early on. That idea has come to fruition with the Henson Digital Puppetry Studio, whose roots go back to the Henson Performance Control System that enabled sophisticated, nuanced control of animatronics.

The Henson Digital Puppetry Studio enables real-time animation of characters based on a breakdown of tasks between the actor in a motion capture suit and a puppeteer who voices the character and manipulates everything above the neck with a digital control system that resembles practical puppet controls. That practical control system, called fittingly a waldo, enables very nuanced expressions. The suit can also capture nuances, including fingers.

"Waldo" control for facial expressions

“This is a whole new new way of doing TV/feature production,” said Henson. “This isn’t taking away from animation. This is different. It’s a performance medium.”

Head of production Kerry Shea introduced creative supervisor Peter Brooke; Wild; and new media producer Bret Nelson to add their observations about the Henson Digital Puppetry System. Brooke gave the back-story, describing his early days with Henson, working in 1988 on the TV series “The Storyteller.” At the time, animatronics were controlled by push-pull cables, with 8 or 10 performers to control one character. “It required coordination and diminished spontaneous performance,” he said. “Then we used radio controlled servers. The idea was to come up with the right software and mechanical device to operate the puppet.” From that came the Oscar Award-winning Henson Performance Control System–a mitt and a joystick (also called a waldo)–that enables a single puppeteer to control complex expressions.

Despite the advent of successful 3D CG characters, Brooke also spoke out in defense of the practical puppet. “There are directors who want to capture everything in camera,” he said. “The human connection is what makes puppets and animatronics appealing. Ironically the computer allows us, via rod removal, to go back to more simple, earlier puppetry techniques.”

The ideal balance, said Brooke, is marrying the practical and digital, in real-time, which is what the Henson Digital Puppetry Studio does. Wild noted that they’ve created a 3-camera sitcom environment in the 60×40-foot space, equipped with 38 Motion Analysis cameras. “We record entire takes,” said Wild. “We don’t record on a shot-by-shot basis. That includes all the body and facial movements and dialogue, in one single stream of data. ” Henson’s Creature Shop uses Autodesk’s Maya to get the data in and out of the pipeline. “Everything we’ve done to date is built in modules,” explained Wild. “So a creative producer can pick and choose among a wide array of technology: facial puppetry, camera layout, digital rehearsal, previsualization, and so on.” They can create up to six virtual characters and capture up to 4 cameras at the same time, using real camera operators to drive the virtual cameras.

Camera operators

Nelson noted that, for Jim Henson’s Creature Shop, the real-time aspect has always been necessary. “The puppeteer needs to see the camera tap because they need to see what’s going on on the screen,” he said. “CGI is just the next step in a logical progression.” Nelson also pointed out that the Digital Puppetry System is flexible and can be used to produce a lot of animation in a short time. The Creature Shop produced 50 animated segments of a cuckoo bird for PBS Sprout with two days of voiceover and two days on the stage.

“Someone said animation is documenting an event that actually never takes place,” said Nelson. “We set up the event and run it again and again until the event happens exactly the way we want it to happen.”