Los Angeles Post Production Group: Preventing Media Disasters

Producer/Director/Journalist Cirina Catania and DriveSavers Chief Technology Officer Chris Bross spoke at the Los Angeles Post Production Group (LAPPG) conference. Together, they shared the story of Cirina’s devastating data loss of over 100TB and discussed ways to help other media professionals avoid the same situation.

Cirina: At 6 o’clock in the morning, I left the apartment to take a friend of mine to the airport who is flying to NAB [National Association of Broadcasters conference] and all my bags are packed that I’m getting ready to drive to NAB.
I had been working late that night and early again that morning to get ready. I had over a hundred terabytes of drives connected to my iMac—my brand-new, lovely iMac. I unmounted all the drives, which I just do because I don’t like anything chugging when I’m not there. I
unmounted all the drives. I left to go to the airport. Within an hour and a half of coming back, I walked into my apartment, I took my computer off asleep and this is what I saw: zero terabytes.
So all of those drives were full. They were all running. They all had media on them.
All of the copies of my of my life were in those drives. My films, my admin going back thirty years because I had gone paperless and I was scanning everything in. So I had legal contracts from when I worked at MGM, UA and UA. All of my contracts were in there. All of my signed location and personal releases were in there for all the films I’ve done over the years. Over 900 Final Cut 7 and 10 events and libraries. Everything was in there.
I called OWC. I called Tim Standing first, who runs SoftRAID, because I thought maybeSoftRAID had had something to do with this. And he said, “send me your log.” So we looked at the logs and it looked like at about 6:16 in the morning on that day there had been a command issued at the root level of the computer to erase everything that was attached—even though it was sleeping. So it brought it back from sleep and then erased it.
So these are some of the first seven drives that I sent out. You can see there are four twenty-four-terabyte drives, there’s a small—I think that’s a two terabyte WD Passport from the Amazon—there’s a G Speed RAID and there’s another four-terabyte [drive] that had some master media and master audio on it.
The guys at SoftRAID received my drives and Tim Standing at SoftRAID handed them over to mark James. He could do some forensics on it. Meanwhile, he said, “listen—we will lend you some drives,” which I will be forever grateful to all of these guys because they just saved my life with this.
You don’t think about it but, when you’re working, there are millions of files in there. And in one small partition alone there were over 385,000 files with no metadata, no date of creation, no file name except for the the new numbers that had been assigned, no folder structure.
So, for example, if I wanted to try to recover everything that I’ve shot since 2014 on the Wounded Warrior Chianti story, I have no way of doing that. So the only way to do it is to sit at your desk and literally open up every file one at a time, wait for it to load, wait for it to open or not. And most of them weren’t opening up. So then you just throw them in a folder that says “NG”, “NG”, “NG.” And then, you know, 30% of the time at that point, we were finding some of the files that were working. But most of the audio files would open up and then as I started to check them I would get two or three seconds into it and it would just go “chhhhh” because it was corrupted.
Now, OWC did manage to find a bunch of stuff.  They ran a couple of different programs—much, much more robust than anything you and I could do with something like photo recovery or anything we would do to try to recover this information. So OWC was great
and I was getting some stuff back but the sound files were a big, big problem. So that’s when OWC and SoftRAID called in DriveSavers.
And then I got the first drive back from DriveSavers and look what I got. I got a folder structure! That one thing that says “Master Media Files DO NOT TOUCH”—those are original media files for several of the projects. I could at least go in there now and start opening things
up and figure out what the damage really was because without going through millions of files now I could just go through a few thousand at a time.
Chris Bross: All of our data is money, right? And nowadays you’ve got to protect it. And budgeting for it and budgeting for backup and all this stuff has to be part of your initial purchase. It can’t be what you’re gonna do after you start production. You’ve got a plan for 2x or 3x the amount of storage you’re gonna need when you’re planning your shoot and budget it in from the beginning.  And sometimes that’s not even enough.
All hardware’s gonna expire at some point. It’s a question of putting in place a backup strategy like three, two, one: Three copies of your data, two different types of media and at least one off-site copy. That’s an absolute minimum of what you need to be doing. And when you’re running around with 150 terabytes, that’s a challenge. Right? But you still have to do it.
Not everything is always recoverable, you know. Serena just told you that. Well, sometimes you have a single drive and you have a catastrophic head crash and the drive starts making a bunch of noise like broken glass inside. Well, that’s broken data. And if you only have one point of failure and one copy of your data, you’re done. That’s it. Right? So having multiple layers, multiple schemes, multiple vendors of storage is really important to all of that.
The best thing you can really do if you lose data—we say stop, drop and roll. Just like you learned as a kid, you know, in a fire. In an emergency, get your hands off the keyboard. Don’t aggravate the problem. You’re getting nervous. You’re gonna make bad decisions. You’re gonna make the problem worse.
We’re open 24 hours a day, 365 days a year. Free phone advice from really smart people at DriveSavers. Call us anytime. We’ll tell you whether or not you have a real data loss on your hands or whether
or not it’s something you can deal with yourselves.
To your question about the cloud: The cloud is a great resource to back up some of your data but it doesn’t work for most people to back up all of the data, especially for this audience. Let’s also remember that the cloud is actually a big ugly gray building with no windows, air-conditioning and servers in it—and hard drives. So if you think the cloud is magic, it’s not. It’s a bunch of hard drives spinning in a data center. I know that because we do data recovery for cloud providers. You know, you should be nervous—it’s valid.
Cirina: The event that happened to me was that the internal server that I was running issued the root command to erase everything that was attached. That potentially could be a software incompatibility with the new OS. We don’t know what caused it. So I’m not saying
that NASA was in my computer, I’m just… Given some of the stuff that I’m working on, I’m thinking there’s a little possibility. But the bottom line is it got lost and it got degraded.
Chris Bross: Basically, a piece of driver software from a different drive… Let’s say you just went out and purchased a new drive off of the shelf at Best Buy. It comes with a piece of software you install. That piece of software helps you manage that drive. Right? Well, that particular piece of software seemed to intercept the commands to all of the drives that were on SoftRAID and basically step on it and say, “wait—you’re not a SoftRAID four drive, twenty-four terabyte thing. You’re four individual six terabyte drives that all want to be  initialized as four empty drives.” So this driver software took it upon itself—because it didn’t recognize the other drives—to say, “I don’t know who you are; therefore, I’m going to erase you and give you my name so I know what you are.”
Thank you all for your attention. Hopefully, we can prevent this from happening to you.