Hard Drive Storage for Producers

The More Firewire Drives The Better?

At least once a week I’m asked, "what hard drive should I buy?" A bewildering number of options make choosing dedicated storage for media files seem like a tough question. For a facility that needs a massive RAID system or storage area network to be shared by several workstations, it is. But for a producer or editor who just needs support for common SD and compressed HD formats on a single workstation, it isn’t. Once you know your throughput, capacity and redundancy needs, your choices become clear. Well, clearer.

How Fast?

Determine your throughput needs not by the format your camera records to tape (or card or optical disc), but by the highest-bandwidth format you regularly load onto your hard drives for general editing. If you shoot and edit DV, then you’re looking at 3.7 MB/sec. But if you transcode from HDV to DVCPRO HD, for example, base your calculations on the 14.8 MB/sec required for DVCPRO HD. For HDCAM, calculate based on whatever intermediate codec you use. Avid DNxHD, CineForm, uncompressed, 4:1 MJPEG and other options are all intermediate codecs, after all.

[A note about HDV and XDCAM: These MPEG-based formats may have average throughputs from 18 to 35 Mbps, but the throughput demand varies with the mix of I, B and P frames. So add a little padding to your data-rate needs for these formats.]

I then take the highest data rate I’ll regularly encounter and multiply by four. That will support reading and writing of a couple of video streams, a few channels of audio (actualities, VO, music), plus a graphics stream and overhead. Motion-graphics designers working with uncompressed or lightly compressed material may not be able to find a system that offers four times their format’s data rate. But if they can live with, or at least tolerate, non-real-time performance, there are still affordable options.

If you only occasionally online with uncompressed HD, and usually do that at a facility other than your own (have you seen the price of HD scopes?), this approach may work for you.

Video format (and network) data transfer rates are typically described as megabits per second (Mbps). Drive transfer rates are typically described in megabytes per second (MBps). Converting between the two is simple: Eight megabits equal one megabyte. The chart, on page 35, suggests that if you mainly edit in DV, a storage system that can sustain 15 MBps will do. If you edit DVCPRO HD, 60 MBps will keep you happy. Different intermediate codecs have different data-rate needs, but 120 MBps will keep a few streams of most formats humming along.

How Much?

Just as every problem expands to fill the available time, so our video expands to fill all available hard drive space.

The size of a storage system depends on your editing format’s data rate, the amount of material required for a typical project and the number of projects that will simultaneously be on the device.

I have yet to encounter a typical project, but I have found some typical source-to-final ratios. This isn’t the same as a shooting ratio. I reduce the effects of location imagorrhea with logging to trim down what ends up on hard drives.

For fiction and corporate work, 5:1 is a good benchmark for rough footage, graphics, render files and so on. For a documentary, the ratio may increase to 10:1 or 20:1. A motion graphic designer may have a higher source-to-final ratio than a documentary or corporate editor, but will also be working on shorter projects. So let’s call it a wash.

While a five-minute corporate piece shot and edited in DV may require only five gigabytes, a one-hour documentary edited in DVCPRO HD could require over a terabyte.

I work on multiple projects at the same time, usually one to three corporate jobs and one to three documentaries or short dramas. There’s the job I’m finishing, the job I’m working on and the job I’m sitting on while the client (and for the docs, that’s usually me) figures out what the heck they’re going to do. My corporate work is mostly standard definition (though that’s changing); the docs vary from DV to HDCAM originals with editing taking place in some format that consumes less than 75 GB per hour.

For me, one to three terabytes of mainline storage does the trick. And, in fact, each of my two main workstations (one Mac and one Windows) has a terabyte of main drive space for media files.

What Interface?

For video, the five leading computer-to-storage interfaces listed in general ascending order of performance are FireWire 400 (i.e., standard FireWire as found on DV cameras), FireWire 800, SATA, UltraSCSI and Fibre Channel. I’m mixing technologies here. Typical "FireWire" drives have either IDE or SATA mechanisms inside. Similarly, there are arrays built around IDE drives, but with SCSI interfaces to the computer. But most users group drives based on the type of cable they plug into the device.

For modern video formats, FireWire 400 is getting long in the tooth and slow on the track. Even with the recent Oxford Semiconductor 924 interface and a modern 500 GB SATA II drive, the best read performance I’ve seen with FireWire 400 was just over 30 MBps. That’s good for over four streams of DV and four of DVCPRO50, but not fast enough for reliable and efficient use with most other formats. I’ve seen Firewire 400 drives alternately play back HDV footage without a problem, but then drop HDV frames. However, I’ve always been able to reliably play back the same HDV footage on an internal SATA drive I’d investigate further if there weren’t better alternatives. But there are. For example, an external drive built around the Oxford 924 can offer FireWire 400 and 800, USB 2.0 and SATA ports. A single drive can give you good performance at home and great compatibility on the road.

FireWire 800 offers close to twice the throughput of FireWire 400 and is viable with more formats, as long as you have an independent FireWire bus that isn’t also supporting a VTR or camera. On most computers all FireWire ports use a single bus, and occasionally a camera’s or VTR’s FireWire signal will conflict with that of a FireWire drive. You can avoid the conflicts by installing a second FireWire bus through a PCI card in your desktop or a PC card or Express card in your laptop computer.

But if you don’t already have a spare FireWire 800 port, you could go to SATA. Big moderate-priced drives can sustain around 70 MBps, and you can combine drives for faster performance. You can get SATA cards for desktop and laptops. For my needs, SATA is the way to go. At least for this year. I don’t have anything against FireWire 400 drives. I have six right here. While I don’t use them for main online storage, I use them for backup, some location work and shipping media to collaborators.

Use a RAID System?

It comes down to speed, data security and cost. Two or more drives set up as a RAID 0, with files spread evenly across the drives, gives great performance at a low cost. For example, a single SATA drive that can transfer around 65 MBps (and that slows down as it gets full) could be matched with three similar drives into a RAID 0 array and transfer over 200 MBps.

But if any one of the drive mechanisms in a RAID 0 fails, data on all of them is lost. RAID levels 3, 5 and 6 add data security and redundancy by adding increasingly secure parity information needed to recreate files across drives. With RAID 6, you can theoretically have two drives fail and still not lose data.

There are other RAID levels, and each level has drawbacks and advantages. But overall, this data security often comes at the cost of reduced throughput and storage capacity, and always comes at the cost of increased, well, cost.

A 2-terabyte RAID 0 can cost $2,000. A 2-terabyte Fiber Channel RAID 3 or 5 system appropriate for video work can easily cost $4,000 to $7,000 or more. If you run a full-time facility with multiple seats and multiple editors, then a higher-level RAID system or storage area network is a good economic choice. Lost time is lots of lost money.

But my two workstations each have their own RAID 0 system, and I regularly back up the data on each to FireWire drives. It’s not a perfect solution and I plan to buy a higher-end RAID system in 2007, but for now it works.

What and Where to Buy

I used to track storage technology more closely than I do now. But knowing the best mechanism of yesteryear doesn’t mean I know the best choices today. Western Digital and Maxtor used to be on my do-not-fly list. Now some people have great experiences with them.

Tracking storage trends and building systems can be fun. But I have other hobbies, so these days I buy from video-savvy vendors. It saves time, doesn’t cost much extra and gives me a direct line to tech support and replacement parts if things go wrong (which they haven’t, knock on wood).

I don’t buy the lowest cost-per-gigabyte systems available at Best Buy. Video pros work their drives harder than consumer and business users do, and our systems, therefore, generate more heat. So even if the guy at the megastore says you can stick four more hard drives into your computer, or says he edits video with the FireWire drive that’s on sale today, you might not get what you need.

Many of the storage questions I get come from people whose FireWire drive just went kaput. It often turns out they bought a consumer FireWire drive that didn’t have an internal fan or heat sink adequate to keep cool a constantly working drive. And I’ve seen similar problems with some RAID systems built into computer cases.

But I’ve also seen reliable fanless FireWire drives and successful internal RAIDs. The difference is intelligent heat dissipation, and I find that intelligence most reliably with vendors and resellers who really understand the needs of video professionals.

Storage is cheap. Figure a buck or three a gigabyte. Or in video-sized terms, around $1,000 per terabyte for single drives and SATA RAID 0, around $3,000 per terabyte for multi-terabyte Fibre Channel systems. Compared to the guy in 1997 who rented out 9 GB hard drives for $125 per week, hard drive storage today is a bargain.

More Storage Articles in this Issue

- Tips for Implementing Storage

- Underdog editor Brett Schlaman: Why Archion RAID Takes a Bite Out of Time on an Effects-Heavy Avid Edit