DI Data Wranglers

It’s been five years now since O Brother, Where Art
Thou ushered in the era of the full digital intermediate.
Since then, the DI has become a fixture both at the high end of the
industry, where A-list colorists manipulating footage shot by A-list
DPs at resolutions of 2K and higher, and on the low end, where indies
make use of desktop tools like After Effects, Final Cut Pro and
FinalTouch to get their DV, HD and 16mm images ready for prime time – or
at least festival exhibition. To keep its reputation up, a DI facility
has to prove itself through experience and stay on top through
effective management of data. Facilities are working hard to keep
processes under control, and hiring new data wranglers to work in a
realm where metadata is everything.

To say the number of individual assets that a facility has to track has
multiplied exponentially would be an understatement. Laser Pacific
President Leon Silverman notes that a single DI project will typically
consist of around 500,000 files. "Those half-million individually named
files represent a few rolls of tape and film in the not-so-olden days,"
he says. "If you deal with multiple projects at once, there are
millions of individual files that need to be managed, rearranged,
tracked, rendered, stored and restored. And that’s not necessarily a
skill you learn by being a tape operator." Instead, Silverman says, you
learn it by becoming computer-centric and understanding network
systems. Most importantly, he stresses, the process needs to seem
absolutely seamless to filmmakers.

"If you want to be serious about DIs, you need a full-time person"
handling data-management issues, says Rainer Knebel, VP of digital
intermediate services for Ascent Media’s Creative Services Group. That
person works closely with the producers of a feature film project to
keep different data files, and different versions of data files, on
track. If a project is very complicated, or multiple projects are being
done at the same time, more people will need to be given data-wrangling
duty.

In some ways, the job descriptions are still being defined because
standard practices in old-school post don’t translate to the new
environment. "It’s all IT technology, all data," Knebel says. "Video
was easy because we had SMPTE, and they had standards. With HD, 2K and
4K data, it’s not so easy. Fortunately, almost everyone is using DPX
and the Cineon file format. But VFX houses like open formats, so our IT
guys write scripts to convert OpenEXR to DPX and the other way around."

New relationships are developing inside the post house to keep track of
the relationship between physical bits of film and data files-
essentially, between keycode and timecode. Company 3 works with a
negative-matching company called Computamatch, which has now moved
in-house to make the process more efficient. "They are the new
generation of negative matchers," says Siggy Ferstl, senior colorist.
"Traditionally, all of your‘go’ takes used to be printed, and with
digital dailies, fewer people are going to that expense. But it was the
film print that had the keycode reference, so when the shots were
pulled, they just cross-referenced the pulled negative with the print.
There is human error in doing dailies work, and in an ideal world the
lab would not join up any of the camera rolls- they would send them to
Computamatch, which would log and join up the film before it goes to
telecine."

That process reduces the number of roll changes that happen during the
dailies process, and Computamatch, owned by Marilyn Sommer, checks and
cross-checks the data to ensure accuracy as the master list is created.
"If a shot has to be pulled for effects scanning, she maintains that
and puts it into a database," explains Knebel. "It protects the film
for us. We have a lot less dust-busting to do because it’s handled in a
professional way, and we don’t have to do things twice."

Michael Christmann, a systems workflow architect and consultant who has
helped Laser Pacific set up its DI pipeline, says the most important
new development from a personnel standpoint may be the creation of a
new position he calls the data editor. "Obviously, scanning operators
and recording operators have existed for quite a while," he says. "The
next layer in the chain is the data-management people, who sort out the
issue of loading data offline and online, making it available to
different people and making sure it’s structured in such a way that
people follow the rules. It’s not new, but it becomes more and more of
an issue.

"The next layer is the data editor, which is something similar to a
video editor, like an Avid cutter. Those people operate a Lustre or a
Spectre in preparation for the colorist. They get all the visual
effects in, and they make the timeline work. They have to understand
list-management and all sorts of other things, and this is a new job
creation- the data editor is something that didn’t exist before."

Knebel sees de facto standards evolving as a result of what are
becoming standard operating procedures in the industry, but wishes
there were a few more SMPTE-style guidelines on the new frontier. "At
the moment, not even the different products talk to each other," he
says. "If you had a digital Betacam, it just worked. You were able to
connect the SD output of a telecine to any VTR. Data is all a bit
different. You can’t really take a conform on one product and apply it
to a different manufacturer’s product. It’s also difficult to move jobs
from one facility to a different facility- systems have to be
calibrated, because they don’t talk to each other. You can’t just move
a 3D look-up table from one facility to another and assume they can
easily apply it."

Others trying to set up efficient, bullet-proof pipelines share
Knebel’s concern about standards. "I think the post industry has to
step in and say,‘We want you vendors to work more closely together and
get more interoperability between your products,’" says Christmann,
referring to the possible effects of efforts like the ASC’s color
decision list proposal, being led by Lou Levinson and Josh Pines. "The
ASC’s color decision list is the first step toward doing that- but it’s
just a little tiny starting point. It doesn’t give you a lot, just nine
primary parameters, but we need more of this activity to get the
industry focused on interoperability.

"Once we start doing a lot of digital acquisition, it becomes more
important," Christmann continues. "With film, you can still see
something. You can look at the keycode and say,‘Here’s the frame I’m
looking for.’ That’s not going to happen in the data world. There is no
standard for metadata in the entire chain, and we are going to need

that or else everybody is lost, and all the information is lost about
what the DP wanted to do in the first place."

Christmann points to the Digital Cinema Initiative, which standardized
models for digital cinema transmission, as a model for further
standardization in the industry and notes that SMPTE has already
developed recommendations for working with metadata elements in its W25
Wrappers and Metadata Committee. "If we follow that model, we create a
branch for digital cinema from capturing through post and all the way
through to delivery," he says. "If we can make use of the existing
dictionaries of metadata, we are so far ahead- but we need to use them
and implement them. And the post companies can push that a little bit,
by refusing to buy products that don’t support the standards."

So how do you make the most of the systems that exist today?
Calibration is key. "Before you do any DI project, you need to start
asking yourself calibration questions," says Maurice Patel, product
marketing manager for Autodesk Media and Entertainment. "How are all
your different elements going to be calibrated? You need to understand
all the different LUTs. There are LUTs in your scanners, in your
printers, and in your software. LUT management and calibration are
going to be the key concerns anyone doing a DI has to worry about."

The sheer volume of data can swamp a careless facility. "Someone has to
look at the big picture," explains Da Vinci’s Rich Montez, senior
colorist and director of the Da Vinci Academy. "Not only do you have to
have someone cleaning up files to make sure a project is not taking up
too much disk space, but you have to look at how data is moving from
point A to point B. And that leads to resource management, which is
probably one of the biggest gotchas that bites a
facility.

"You pay for people to be there while data is moving from one place to
another," Montez says, noting that a colorist may sit on his hands
while waiting for a crucial piece of footage to become available, or
maybe spend valuable time quality-checking it. And the next person down
the line who has to work on the same footage is stuck waiting for the
first guy to be done with it. "If you’re paying people overtime, it
gets expensive. I can’t tell you how often you have an expensive
colorist just sitting there for hours, waiting for something to load or
to render."

Finally, as with any other new skill, DI facilities will learn by
doing. "Facilities learn the hard way that color management is a huge
undertaking," Montez cautions. "You have to have a good relationship
with your lab, and the whole process takes time and work effort. All
the facilities that are successful at making good DIs have made a lot
of mistakes- and have learned from them."

The 2K DI process may seem like a trial by fire, but the data dragon
roars even louder when you contemplate a 4K project. Each frame of a 2K
project takes up more than 12 MB of storage, and each frame of a 4K
project is more than 50 MB.

Any facility that works with major studios will assure you that it can
do 4K work, but it’s only recently that studios have begun to demand
it. At Efilm, for instance, the first 4K DI was Spider-Man
2, completed last year. Efilm’s next 4K DI was
Ocean’s 12, and it was notable because even the FX
work was completed at 4K (Spider-Man‘s VFX were
inserted at 2K, in part because the decision to go to 4K was made so
late in the production process). Efilm’s success last year, combined
with Digital Cinema Initiatives’ recent ratification of standards for
4K digital cinema and a general concern among Hollywood studios about
high-quality archival materials, set the stage for a flurry of 4K
activity at the facility this year.

"Last year we did two 4K shows," says Michael Cooper, Efilm’s VP of
business development. "This year, the studios are looking at archival
quality. They have digital archives with digital YCM separations, and
we’re looking at digital cinema masters now that DCI has come up with
standards for 4K digital cinema and 4K distribution. The studios
say,‘Now that there are standards calling for 4K, we want to
future-proof our asset’- which is the movie itself. So we expanded to
do one 4K show. And to do four 4K shows, we had to add more storage and
processing."

Currently being finished in 4K at Efilm are Disney’s
Casanova, Universal’s Jarhead,
Revolution’s Rent (see story in this issue), and
Fox’s The Family Stone. Earlier this year,
Columbia’s Stealth was finished at 4K for summer
release.

Four times the data necessarily means four times the storage and
bandwidth. Does this mean facilities get to charge four times as much
for a 4K DI? Of course not! "For us, it’s just about a 10 to 15 percent
[price] premium over 2K," says Cooper. And, of course, the delivery
schedule remains the same. "Timewise, the workflow is traditionally
five weeks from scanning until the film is done, and that five weeks is
the same five weeks, whether it’s 2K or 4K." Efilm makes up the
difference by chasing its own process as soon as the film scan starts.
Once a full reel is ready for timing, the process can begin as soon as
the director and/or DP are available, while the rest of the files are
still being prepped.

And Cooper thinks efficient DI facilities can make a case for 4K as
good value for money. "The DPs have been pushing for 4K because they
want the best quality they can get for their image, and the director is
right there with the DP," he says. "The studio wants the best quality –
but they also want the finances and the timeline to work. If those can
be controlled, they see big archival benefits."

"The big Hollywood productions create the looks that other people want
to emulate," says Maurice Patel, product marketing manager for Autodesk
Media and Entertainment. "The challenge is, how do you do that on a
lower budget given that the process is so complex? Well, it’s easier if
you stay in one format, such as going HD all the way. If you start
combining formats, it becomes even more challenging."

Patel cites an Autodesk customer based in India, Prime Focus, as
evidence that you don’t need big budgets to do smart DI work. "They do
10 DIs every two weeks," he says. "The movies are very low budget, but
they’ve been driven by what’s happening with Hollywood releases. And
they face very similar problems to the U.S., which is getting the
process calibrated. And they give end-to-end service, with VFX and
grading all in one environment, which avoids the challenges that happen
when you break it up."

Laser Pacific offers InDI, an option for low-budget productions shot on
16mm or 35mm that combines the advantages of digital video and
data-based workflows. The main cost savings comes from scanning the
negative only once. "We scan the film from D-min to D-max, putting the
full range of the negative onto HDCAM SR 4:4:4 tape," says Laser
Pacific President Leon Silverman. "In the pathway for dailies, we put
another set of Kodak image-science tools that include film LUTs that
allow the editors and filmmakers to see their DI looking like a film
print in dailies." Color-correction is applied to the footage in a
traditional color-correction suite.

"The master elements you have will make a film that looks exactly like
what you saw in the making of the master element- and this is not true
in most low-budget HD environments, where you’re looking at monitors
that are uncalibrated, with no sense of a film-out or how print density
is emulated."