DI Data Wranglers

How Digital Post Gurus Handle Negative Matching, Data Editing, and Herding 500,000 Files Through the Pipes

It’s been five years now since O Brother, Where Art Thou ushered in the era of the full digital intermediate. Since then, the DI has become a fixture both at the high end of the industry, where A-list colorists manipulating footage shot by A-list DPs at resolutions of 2K and higher, and on the low end, where indies make use of desktop tools like After Effects, Final Cut Pro and FinalTouch to get their DV, HD and 16mm images ready for prime time — or at least festival exhibition. To keep its reputation up, a DI facility has to prove itself through experience and stay on top through effective management of data. Facilities are working hard to keep processes under control, and hiring new data wranglers to work in a realm where metadata is everything.

To say the number of individual assets that a facility has to track has multiplied exponentially would be an understatement. Laser Pacific President Leon Silverman notes that a single DI project will typically consist of around 500,000 files. "Those half-million individually named files represent a few rolls of tape and film in the not-so-olden days," he says. "If you deal with multiple projects at once, there are millions of individual files that need to be managed, rearranged, tracked, rendered, stored and restored. And that’s not necessarily a skill you learn by being a tape operator." Instead, Silverman says, you learn it by becoming computer-centric and understanding network systems. Most importantly, he stresses, the process needs to seem absolutely seamless to filmmakers.

Become Computer-centric

"If you want to be serious about DIs, you need a full-time person" handling data-management issues, says Rainer Knebel, VP of digital intermediate services for Ascent Media’s Creative Services Group. That person works closely with the producers of a feature film project to keep different data files, and different versions of data files, on track. If a project is very complicated, or multiple projects are being done at the same time, more people will need to be given data-wrangling duty.

In some ways, the job descriptions are still being defined because standard practices in old-school post don’t translate to the new environment. "It’s all IT technology, all data," Knebel says. "Video was easy because we had SMPTE, and they had standards. With HD, 2K and 4K data, it’s not so easy. Fortunately, almost everyone is using DPX and the Cineon file format. But VFX houses like open formats, so our IT guys write scripts to convert OpenEXR to DPX and the other way around."

Handle Film Less

New relationships are developing inside the post house to keep track of the relationship between physical bits of film and data files— essentially, between keycode and timecode. Company 3 works with a negative-matching company called Computamatch, which has now moved in-house to make the process more efficient. "They are the new generation of negative matchers," says Siggy Ferstl, senior colorist. "Traditionally, all of your‘go’ takes used to be printed, and with digital dailies, fewer people are going to that expense. But it was the film print that had the keycode reference, so when the shots were pulled, they just cross-referenced the pulled negative with the print. There is human error in doing dailies work, and in an ideal world the lab would not join up any of the camera rolls— they would send them to Computamatch, which would log and join up the film before it goes to telecine."

That process reduces the number of roll changes that happen during the dailies process, and Computamatch, owned by Marilyn Sommer, checks and cross-checks the data to ensure accuracy as the master list is created. "If a shot has to be pulled for effects scanning, she maintains that and puts it into a database," explains Knebel. "It protects the film for us. We have a lot less dust-busting to do because it’s handled in a professional way, and we don’t have to do things twice."

Hire a Data Editor

Michael Christmann, a systems workflow architect and consultant who has helped Laser Pacific set up its DI pipeline, says the most important new development from a personnel standpoint may be the creation of a new position he calls the data editor. "Obviously, scanning operators and recording operators have existed for quite a while," he says. "The next layer in the chain is the data-management people, who sort out the issue of loading data offline and online, making it available to different people and making sure it’s structured in such a way that people follow the rules. It’s not new, but it becomes more and more of an issue.

"The next layer is the data editor, which is something similar to a video editor, like an Avid cutter. Those people operate a Lustre or a Spectre in preparation for the colorist. They get all the visual effects in, and they make the timeline work. They have to understand list-management and all sorts of other things, and this is a new job creation— the data editor is something that didn’t exist before."

Hope For Some Standards

Knebel sees de facto standards evolving as a result of what are becoming standard operating procedures in the industry, but wishes there were a few more SMPTE-style guidelines on the new frontier. "At the moment, not even the different products talk to each other," he says. "If you had a digital Betacam, it just worked. You were able to connect the SD output of a telecine to any VTR. Data is all a bit different. You can’t really take a conform on one product and apply it to a different manufacturer’s product. It’s also difficult to move jobs from one facility to a different facility— systems have to be calibrated, because they don’t talk to each other. You can’t just move a 3D look-up table from one facility to another and assume they can easily apply it."

Others trying to set up efficient, bullet-proof pipelines share Knebel’s concern about standards. "I think the post industry has to step in and say,‘We want you vendors to work more closely together and get more interoperability between your products,’" says Christmann, referring to the possible effects of efforts like the ASC’s color decision list proposal, being led by Lou Levinson and Josh Pines. "The ASC’s color decision list is the first step toward doing that— but it’s just a little tiny starting point. It doesn’t give you a lot, just nine primary parameters, but we need more of this activity to get the industry focused on interoperability.

"Once we start doing a lot of digital acquisition, it becomes more important," Christmann continues. "With film, you can still see something. You can look at the keycode and say,‘Here’s the frame I’m looking for.’ That’s not going to happen in the data world. There is no standard for metadata in the entire chain, and we are going to need that or else everybody is lost, and all the information is lost about what the DP wanted to do in the first place."

Christmann points to the Digital Cinema Initiative, which standardized models for digital cinema transmission, as a model for further standardization in the industry and notes that SMPTE has already developed recommendations for working with metadata elements in its W25 Wrappers and Metadata Committee. "If we follow that model, we create a branch for digital cinema from capturing through post and all the way through to delivery," he says. "If we can make use of the existing dictionaries of metadata, we are so far ahead— but we need to use them and implement them. And the post companies can push that a little bit, by refusing to buy products that don’t support the standards."

Calibrate, Calibrate, Calibrate

So how do you make the most of the systems that exist today? Calibration is key. "Before you do any DI project, you need to start asking yourself calibration questions," says Maurice Patel, product marketing manager for Autodesk Media and Entertainment. "How are all your different elements going to be calibrated? You need to understand all the different LUTs. There are LUTs in your scanners, in your printers, and in your software. LUT management and calibration are going to be the key concerns anyone doing a DI has to worry about."

The sheer volume of data can swamp a careless facility. "Someone has to look at the big picture," explains Da Vinci’s Rich Montez, senior colorist and director of the Da Vinci Academy. "Not only do you have to have someone cleaning up files to make sure a project is not taking up too much disk space, but you have to look at how data is moving from point A to point B. And that leads to resource management, which is probably one of the biggest gotchas that bites a facility.

"You pay for people to be there while data is moving from one place to another," Montez says, noting that a colorist may sit on his hands while waiting for a crucial piece of footage to become available, or maybe spend valuable time quality-checking it. And the next person down the line who has to work on the same footage is stuck waiting for the first guy to be done with it. "If you’re paying people overtime, it gets expensive. I can’t tell you how often you have an expensive colorist just sitting there for hours, waiting for something to load or to render."

Finally, as with any other new skill, DI facilities will learn by doing. "Facilities learn the hard way that color management is a huge undertaking," Montez cautions. "You have to have a good relationship with your lab, and the whole process takes time and work effort. All the facilities that are successful at making good DIs have made a lot of mistakes— and have learned from them."

Working at 4K, or How to Fill a SAN

The 2K DI process may seem like a trial by fire, but the data dragon roars even louder when you contemplate a 4K project. Each frame of a 2K project takes up more than 12 MB of storage, and each frame of a 4K project is more than 50 MB.

Any facility that works with major studios will assure you that it can do 4K work, but it’s only recently that studios have begun to demand it. At Efilm, for instance, the first 4K DI was Spider-Man 2, completed last year. Efilm’s next 4K DI was Ocean’s 12, and it was notable because even the FX work was completed at 4K (Spider-Man’s VFX were inserted at 2K, in part because the decision to go to 4K was made so late in the production process). Efilm’s success last year, combined with Digital Cinema Initiatives’ recent ratification of standards for 4K digital cinema and a general concern among Hollywood studios about high-quality archival materials, set the stage for a flurry of 4K activity at the facility this year.

"Last year we did two 4K shows," says Michael Cooper, Efilm’s VP of business development. "This year, the studios are looking at archival quality. They have digital archives with digital YCM separations, and we’re looking at digital cinema masters now that DCI has come up with standards for 4K digital cinema and 4K distribution. The studios say,‘Now that there are standards calling for 4K, we want to future-proof our asset’— which is the movie itself. So we expanded to do one 4K show. And to do four 4K shows, we had to add more storage and processing."

Currently being finished in 4K at Efilm are Disney’s Casanova, Universal’s Jarhead, Revolution’s Rent (see story in this issue), and Fox’s The Family Stone. Earlier this year, Columbia’s Stealth was finished at 4K for summer release.

Four times the data necessarily means four times the storage and bandwidth. Does this mean facilities get to charge four times as much for a 4K DI? Of course not! "For us, it’s just about a 10 to 15 percent [price] premium over 2K," says Cooper. And, of course, the delivery schedule remains the same. "Timewise, the workflow is traditionally five weeks from scanning until the film is done, and that five weeks is the same five weeks, whether it’s 2K or 4K." Efilm makes up the difference by chasing its own process as soon as the film scan starts. Once a full reel is ready for timing, the process can begin as soon as the director and/or DP are available, while the rest of the files are still being prepped.

And Cooper thinks efficient DI facilities can make a case for 4K as good value for money. "The DPs have been pushing for 4K because they want the best quality they can get for their image, and the director is right there with the DP," he says. "The studio wants the best quality — but they also want the finances and the timeline to work. If those can be controlled, they see big archival benefits."

Staying on Budget

"The big Hollywood productions create the looks that other people want to emulate," says Maurice Patel, product marketing manager for Autodesk Media and Entertainment. "The challenge is, how do you do that on a lower budget given that the process is so complex? Well, it’s easier if you stay in one format, such as going HD all the way. If you start combining formats, it becomes even more challenging."

Patel cites an Autodesk customer based in India, Prime Focus, as evidence that you don’t need big budgets to do smart DI work. "They do 10 DIs every two weeks," he says. "The movies are very low budget, but they’ve been driven by what’s happening with Hollywood releases. And they face very similar problems to the U.S., which is getting the process calibrated. And they give end-to-end service, with VFX and grading all in one environment, which avoids the challenges that happen when you break it up."

Laser Pacific offers InDI, an option for low-budget productions shot on 16mm or 35mm that combines the advantages of digital video and data-based workflows. The main cost savings comes from scanning the negative only once. "We scan the film from D-min to D-max, putting the full range of the negative onto HDCAM SR 4:4:4 tape," says Laser Pacific President Leon Silverman. "In the pathway for dailies, we put another set of Kodak image-science tools that include film LUTs that allow the editors and filmmakers to see their DI looking like a film print in dailies." Color-correction is applied to the footage in a traditional color-correction suite.

"The master elements you have will make a film that looks exactly like what you saw in the making of the master element— and this is not true in most low-budget HD environments, where you’re looking at monitors that are uncalibrated, with no sense of a film-out or how print density is emulated."