Robert Rodriguez’s Posse

Troublemaker Studios is as far from a typical Hollywood VFX house as it is from Hollywood itself. Aside from the fact that, unlike nearly all other VFX facilities used for feature films, it isn’t located in Los Angeles, New York or Northern California, a look at how this operation ticks reveals a combination of creativity and technology that makes for an unusual workflow

Most of the 60-some people who work at the Austin, Texas, production company founded by Robert Rodriguez and his wife and business partner, Elizabeth Avellán, don’t have titles. They don’t even have job descriptions. They simply have strengths and weaknesses. "There aren’t defined roles here," explains John Ford, who reluctantly describes himself as a digital artist. " Robert values people being multi-dimensional and able to wear different hats."

"I consider myself a conceptual artist," adds Chris Olivia. "But I build it all on the fly in 3D."

Clones of Troublemaker aren’t likely to be popping up around the country. There is, at last count, only one Robert Rodriguez. But other directors will no doubt customize their own studios to realize their own visions. Troublemaker proves that ubiquitous, affordable technology — bolstered by some clever custom code — can go a long way to creating a flexible production pipeline, and that a group of enthusiastic, committed and talented artists can avoid rigid job descriptions while staying efficient and making dazzling pictures.

Blurring the Lines

The lack of titles at Troublemaker reflects a bigger reality: Artists aren’t pigeonholed into tasks such as modeler or TD or rotoscoper. They even get the chance to add the occasional line of dialogue. Ford points out that the divide between 2D and 3D found in most studios is very blurred at Troublemaker. "The 3D artists do a lot of 2D work, which is pretty unusual and a very powerful tool," he says. "When I’m working out something in 3D, I will also be thinking in terms of how to work the problem out in 2D. It can be a time-saver to jump between the two." Alex Toader, whose strength is concept art, has found that "being part of every aspect of production" has helped him to be more creative in everything he does.

Troublemaker’s artists get a chance to work on sequences from beginning to end. The compartmentalized nature of most major VFX facilities simply doesn’t exist here. "You’re expected to do everything, and that’s been a huge learning experience," says Ford. "It’s not uncommon for one of us to get a shot, completely choreograph it, create an animatic, get a feel for Robert’s vision and then take it all the way to the end, finishing the shot." Olivia reports that he created 400 animatics, covering nearly every action sequence in The Adventures of Shark Boy and Lava Girl in 3-D. "This allowed me to touch every shot in the movie and work so closely with the director," he says. "I’d never been able to do that before."

Add to that the crazy brew of multiple projects running simultaneously. "It takes a while to get used to," admits Rodney Brunet, who worked on a violent scene in Sin City the same day as a scene set in the Land of Milk and Cookies for Shark Boy. "Our production schedules are so pressed that we’re always in crunch mode— we’ll never work on just one project at a time, and that can be bizarre.”

The geography of Rodriguez’s productions also offers an unusual twist: Troublemaker artists work upstairs while the productions are shot downstairs. The proximity encourages artistic involvement of the VFX crew throughout the shoot. "Sometimes we’d even show the actor what we need him to do by showing the animatic," says Toader.

Everyone involved agrees that this topsy-turvy way of working has little to do with technology. Troublemaker Studios is full of leading-edge gear, from 3D camera rigs to the latest HD cameras. But Troublemaker is really a reflection of Robert Rodriguez, a filmmaker known for his extensive collection of hats, from director, writer and cinematographer through to editor and composer. Rodriguez founded the facility as a visual playground. To bring the pictures in his head to life, Rodriguez needed pre-vis — lots of it — to work out the complexities of his VFX-heavy digital productions through concept design and animatics. "We work very closely with the director," says 2D supervisor Eric Pham." Robert looks at everything we do. In other houses, there’s a VFX supervisor. Here, he is our VFX supervisor and everything filters through him."

The Pre-vis Process

Picture a table ringed by the director and his merry band of pre-vis pranksters. Throw in a stack of paper and some pencils and watch the rounds of fast, fast sketching. Now move those sketches around the table, each artist looking at the others’ work, with the longest pause at Rodriguez’s seat.

That’s the pre-vis process at Troublemaker. The technology at this stage of the game is low-tech, but the pace is as quick as a line of 1s and 0s. Brunet reports that Rodriguez treats the artists as individuals in this early phase, "because he gets separate ideas from each of us. But once he picks and chooses, then we jump on it as a team," he says. "For the most part, he knows what he wants. We pull it out of his head, and we’ll go through 30 or 40 versions to get it the way he wants."

Translating those rough sketches to fully fleshed-out animatics takes teamwork. Though the concept artists furiously create imaginary worlds, the 2D artists pull perfect mattes, and the 3D artists create wireframe models at their PCs, they’re all ready to switch hats to help out a fellow artist with a knotty problem. Even tools programmer Sean Dunn has been known to pinch-hit as a 2D or 3D artist "when things get tough."

Animatics play center stage, not just for Rodriguez the director, but also for Rodriguez the editor. "Robert will give us a general idea of how he wants the camera to move," says Brunet. "But we’ll give him four or five different angles to give him coverage."

Many of the artists at Troublemaker met each other while working at Austin game developer Digital Anvil, an enterprise that Rodriguez was involved in. Many of them were first tapped by Rodriguez, who had observed them in their Digital Anvil jobs, to freelance on one of his films, which led to a full-time job. Since then, new hires have often come from the pool of colleagues and friends. Rodriguez likes to patronize local talent, helping support Austin’s— and Texas’— burgeoning creative film community.

A Basic Pipeline

The list of software at Troublemaker is deceptively simple. The CG pipeline is based on Softimage XSI, with a Discreet Flame and Combustion 4.0 from Autodesk for rotoscoping work and an older version of Apple Shake, which they are reluctantly planning to abandon in favor of a newer, PC-based system. For Shark Boy, some particle effects were created in Alias Maya, along with associated plug-ins (Glu3D from 3D Aliens) and Particle Illusion. The PCs come from Dell and the chipsets are from AMD, with whom Rodriguez has an exclusive agreement.

The DIY atmosphere at Trouble–maker extends to the two-person IT department. Kris Bushover and Jeff Acord were responsible for evolving Troublemaker Studios from analog to 4:2:2 to its current 4:4:4 status. The fact that the artists have absolutely nothing to say about that transition is testament to the fact that Bushover and Acord made it seamless.

The first challenge, remembers Bushover, was to build a scalable storage area network that was fast enough to make all the data files for two movies accessible at all times. "We also had to take into account that each 10-bit HD frame is 10.5 MB," he says. "When you’re looking at 800 frames per shot, 5000 frames per sequence, you’re getting into a significant chunk of storage."

Current SGI-based storage capacity is 8 TB, with a 58-processor render farm made up of the single-core AMD Opteron systems supplied by Verari. ("Dual-core was only recently introduced," explains Bushover. "We are working with AMD to install them in our production pipeline.") Dunn, who writes format-agnostic tools, reports that the main servers were made more efficient by adding the Rush distributed network render queue from Seriss. Switching from 8-bit to 10-bit also required ramping up hardware. "We use Flame, but now we have to use the [ Silicon Graphics ] Tezro workstation with the latest video system," says Acord.

The IT team is often asked to create innovative tools for bargain-basement prices. For example, when the artists wanted the ability to capture 10-bit video but didn’t want to spend $100,000, Acord researched and then built a solution using the Blackmagic Decklink card that’s capable of capturing 10-bit at 4:4:4 in a configuration with an array of inexpensive SCSI drives.

The variety of operating systems at Troublemaker is another challenge, says Bushover, who notes the difficulty of "just getting everything to play well with others." A lifesaver has been the open source community, in particular the Samba project (www.samba.org), which produces software that allows Linux-based servers to serve file systems to Windows clients. Data portability is via portable hard drives carried by couriers. "It’s still, weirdly enough, the most efficient way to transfer the data," says Acord. "We also run an FTP site, but if you have 20 GB worth of data, it isn’t practical to download."

The editorial facility (40 miles away) is set up with Avid workstations and a Unity SAN, and an Avid Xpress system at Troublemaker helps everyone keep track of the most current edits.

Shooting in 3D

Acquisition is all HD. Sin City was shot with the Sony HDC-F950 HD cameras, which captured 10-bit 4:4:4 to HDCAM SR. As a 3D film, Shark Boy posed other challenges, not the least of which was that the amount of imagery was doubled, creating twice the work. To create 3D plates, the production crew used the Cameron Reality Camera System (RCS), the result of a joint venture created by cinematographer Vince Pace’s company Pace Technology and filmmaker James Cameron.

Sebastian Vega, who’s been focus puller for four Rodriguez films, was the film’s "stereographer," a job that required minute attention to maintaining the camera’s mechanical system and tweaking the cameras’ calibrations in order to get the best 3D possible. The RCS is an HD system, and the Shark Boy production used two Sony F950s with T-cams, which separate the lens mount and CCD array from the bulk of the camera’s chassis. The lens mount and CCD array are fitted in a box the size of a child’s shoebox, says Vega. The rest of the camera, which is essentially a processing unit, is attached to the RCS by a 26-pin, 33-foot-long cable. ( Vega says Pace is currently working to replace the cable with fiber optics.)

Ordinarily, the stereographer can align elements by looking at their spatial relationships. But with Shark Boy shot entirely on green screen, Vega had no reference. "It’s hard to say what’s going on in the frame," he says. "I have to set the settings for the actor to fit seamlessly into the CG world, and if my settings are too aggressive or too relaxed, once they build the CG world, they may not fit."

Whatever Vega couldn’t finesse in production was tweaked in post by Dunn. The Troublemaker team created a special two-camera rig in XSI that mimicked the on-set cameras’ left eye and right eye. Dunn wrote a script that enabled information from the set, such as camera angle, lens, and convergence point in 3D space, to be plugged into the CG cameras. "That lets us match reality much better because we now have the same camera set-up in XSI as we do in reality," he says. "The CG camera is a mathematically perfect set-up, and when you bring in plates from the real world, they’re not perfect. You have to compensate by nudging things until they lock in the CG world." That "nudging" was done in 2D or 3D, says Dunn, "whatever gets it done faster." Dunn further facilitated the process by writing code to make the rendering queue 3D-aware.

Flexible Workflows

The production of Shark Boy and Lava Girl illustrates how a workflow grew out of Rodriguez’s inclusive creative process, enabled by an innovative technology pipeline. Pham recalls wearing many hats on Shark Boy. "I was designing the main logo with Robert for about two weeks, figuring out the fonts," he says. "Then Robert passed my designs to Chris Olivia to refine in 3D. I was also designing the main-on-end titles and, because of the heavy compositing work I had ahead of me, I asked [ tools programmer/artist ] Sean Dunn to take them over by adding texture and creating 3D geometries and extrusions."

Olivia recounts the experience of having designed and created animatics for the Shark cave sequence in Shark Boy— but he was "neck deep" in Sin City and didn’t have the time to pursue feedback. "I didn’t know if Robert thought the animatics were good or crap," he says. "I just moved on and did other things." Toader, Brunet and Olivia went downstairs to see what was cooking in the Shark Boy art department and were shocked. "All these images I’d done for the animatic were across the wall," he remembers. "It had become their bible, and 20 guys were working to create sets from it."

The number of shots finished at Troublemaker has increased with each film since Spy Kids 3, for which the studio finished about 100 shots. But the facility still sends most shots out of house: The Orphanage, CaféFX and Hybride helped bring Sin City to life. But 3D animatics are a useful blueprint for out-of-house shops as well as internal use. With the intensive use of animatics for Shark Boy, Troublemaker finished 140 shots— times two, for both the left and right eyes.

Ultimately, Troublemaker Studios shows how creative and technological forces can be customized to meet the unique needs of a director, a vision or a project. Born from Rodriguez’s interest in leveraging digital technology to create dramatic, graphical looks, Troublemaker is a living demonstration of the efficiencies and flexibilities that are gained when the creative process is frontloaded into detailed pre-vis. Sin City and Shark Boy and Lava Girl are very different results of a workflow in which "digital effects" are so integrated, from pre-production to post, that they are simply part of the filmmaking language.