PDA

View Full Version : Need an organizing principle for footage library



mholt
03-24-2007, 09:44 AM
I have 3000 hours of film and video in numerous formats. I want a way to organize all of the material so I can use it to make moovies. Are there existing systems already in place, or do I have to invent this wheel.

My assumptions are: I need to digitize everything to hard drives in a RAID array. I need to catalog the footage scene by scene into a data base that is searchable. The data base needs to be linked to an editing program that will import the selected scenes.

Do I really need 100,000 gb of storage? Do stock footage libraries have such a system?

This is my life time of capture in numerous venues, locations and situations throughout the world. There are some worthy moovies in there. I want to be able to make them in full quality. I am seeking helpful advice.

unclemac
03-24-2007, 10:40 AM
Howdy and welcome aboard.

Big question. Will need big answers. I agree with everything you said, but you left out one issue. Backups.

You have to have backups. Bad stuff happens. Now your backups could be your orginal or master copies, even if they are all ananlog. You have to decide what the digitizing process is worth....would it be acceptable to do it over in the event of a catastophe, or would it make more sense to spend the money up front to create a backup, and not have to start over?

Ponder that one before moving forward. If you decide you want to have a digital backup, then we need to decide if that means a second RAID array, a tape library, or........

Gotta run for now. I think we can assist with ideas for the stoarge hardware. No expericence with video stoarge libraries, but let's see what others have to say. Will check in later.

despaxas
03-26-2007, 08:22 AM
Wow, 3000 hours? To make these video tapes/films into video files that are as small as possible but still in an editable and usable quality format, would almost have to use DV compression. It's very common and easy to capture analog video to DV files, but we will get into that later. The main problem seems to be storage, as unclemac has said. DV files are compressed 5:1, so that means a 1 hour tape will take up 12-13GB of space. So 3000hrs is about 36-39 thousand GB or 35-38 Terabytes (1,024GB = 1TB). So there is no way you will have all off the footage on one computer to edit everything at once at a high enough quality for a final product.

What filmmakers do in these situations, is capture everything in a very compressed format, sometimes at quarter screen resolution to save more space, edit everything they need (called Offline editing), and make an Edit Decision List (EDL) which is just a list of all the footage that they actually used and in what order it is in. Final Cut Pro can do this for you. Wipe the drives. Then they capture only that footage that they know they need into a high quality format and finish it (called Online editing).

There is a lot of labor involved in doing it this way, and it also means capturing everything twice, but in doing offline editing first, you can log all of your footage by color coding them in Final Cut Pro (like Good Take, B-Roll, Head Shots, etc) however I do not think there is a search function.

Another option may be to just make a database with all of the footage listed and their properties like location, type of shot, year, media, etc. and use it to search and then edit the result.

What formats are you using? Are any of them actually film on reels?

Whatever you decide, we can help you implement it, just let us know your thoughts.

mholt
03-26-2007, 09:18 AM
Yes there is about 400 hours of film on reels. Some of it has already been converted to dv tape. The film stuff starts about 1950 until 1975 when my trusty Boxex DL8 finally jittered its last delirium tremens. But not before having shot The Doors and Jefferson Airplane, Andy Warhol, Taylor Meade and Viva, John Chamberlain and Peter Voulkos. By '76 I moved to Super 8 film trying to shoot single system sound with it. I shot some 1/2" reel to reel Sony Rover stuff at the Montana Constititutional Convention in 1981, and went to 8mm video in 1985, Hi8 as soon as it was available and dvCam in '97 or so. I was in the Soviet Union when Michail Gorbachev was putched!

Once everything is digitized, I want to be able to have an idea, write a scenario and begin to build the moovie directly from the stock footage library held on the RAIDs. They should be linked to the edit system but stand alone and be available full time. Thumbnail sized preview is fine. I am beginning to use Final Cut, and I noticed it's database. I agree, I do not think it is connected to make it searchable. The program is a clumsy piece of crap, but the industry likes it, so I guess it is the game, for now.

An edit decision list is okay. I would like get the editing to be more direct if possible. But even that would be worlds better than cycling tape, which is what I have to do now.

I do not mind the idea of a back up system. I certainly like it better than relogging lost material from master tapes. It is a matter of money. But if we are talking 40tb instead of 100, then I my non existant budget for storage $100k could accomodate it. Ultimately, it costs what it has to cost, and I will beat the bushes for it, but first I need to get a plan and an idea that is doable. mx

TZ
03-26-2007, 09:35 AM
I thought about posting some rough translations: 3000 = 3 yrs of footage and go from there. Then I googled and saw that it might make sense to send it out (gradually and try the service) and have them convert it to DVDs.

The ideal system I envision is along the lines of Mac Pro or Xserve with the dual quad core (8 total) and 16-32GB RAM along with a pair of Fibre controllers connected to XRAID(s) for storage.

10 yrs?

Final Cut 6 due soon? Octal-Core Mac Pro workstation?
http://appleinsider.com/article.php?id=2585

unclemac
03-26-2007, 07:55 PM
Yeah, our in house video guy is really expecting FC6 soon, so it will interesting if it will help with this issue.

If we set aside the work station itself for now, what do you expect the specs for the storage to be? Things that would be nice to know before you put your $ down:

• Total volume of storage for existing archives
• Throughput/performance for the RAID
• Is this storage only, or room and performance required for rendering or conversion process (work space) too?

Xserve RAID was one of the most cost effective big storage devices last time I checked. They handle at least 750 GB HDs.

As for planning the storage itself, the Xserve RAID has two sides, which means two separate volumes, which means at least two different drives will mount on the desktop. If the video needs to be in one contiguous block, then there are better solutions out there, though probably more $$.

unclemac
03-26-2007, 08:02 PM
I was in the Soviet Union when Michail Gorbachev was putched!

Was in Moscow about a month before the wall came down myself. Nice folks. Mostly. ;)

mholt
03-27-2007, 08:17 AM
I envision the storage to be the archive. I do not expect it to be the working, rendering, editing platform. I figured I would need 100k gb but with compression perhaps it could be 40% of that, which makes me think I could have the back up archive that you suggested; if I can afford it.

I want the storage to be out of the traffic lane. And incorruptable by sleepy deletions and thoughtless keystrokes. It does not have to be particularly fast, I think. I will want complete random access so since it is so huge, it may need to be fast fast inorder to be useful. I would love to grab a shot from it and drop it into an edit time line. That would be ideal. I am willing to go the route suggested yesterday, where I identify what I want as an edit decision list, and let the system get all of my choices and drop them onto a clips pane or shot list or browser (FCP.) Then I could edit in the computer using its associated capture drives.

So that sounds like a good arrangement. RAID standalone for the Library connected to a Data Base interfaced with a non linear edit program. (Is this the "online" part?) I guess the edit program (above) could be different from the working edit (offline?) program. I am working my way into the ideas you guys have suggested so far. Please correct me if I am getting it wrong. I do not understand much about speeds. I think there is a "seek time" and a "throughput rate." The throughput rate has to be at least a certain volume, or the video gets scrambled. But I think these days, throughput is pretty well covered even by the 400 mbs systems. I think 800 is today's standard (for firewire) and I see no reason not to be able to operate at that speed. But I am not needing the latest bells and whistles here. What I want is functionality. My gut feeling is that when talking this size RAID array, speed is a big part of functionality.

If I select 400 shots from all over the RAID array, and they range from ten seconds to three minutes, making a shot list that is 5 hours long, how long will it take for that to capture? On one hand, I do not care. What I care is that the captured quality is equal to the quality of the original. The alternative, as suggested yesterday, would be to work with lower quality, and when the piece is completed, put that back into the system to recapture and render according to all of the decisions made during the "offline" editing process.

If I had to, I could accept that. It is antithetical to my experience. I would rather work with full quality when editing and not put the machine brain between my work, and the product. I think I work most surely when I see the product coming out of my hands.

Uncle, I did see you there in Red Square. I think you had a pry bar and were breaking into St. Basil's. I really would not have noticed except that I was looking at Lennin, lying in his glass coffin, and he rolled over and groaned, I looked up, pretty amazed, and just caught the flash of sunlight on the pry bar. Remember those two or three huge Russian mothers who had their winter coats wide open and their skirts pulled up over their heads, gettin a sun tan I guess, standing in that Moscow sun beam, in 1990? mx

despaxas
03-27-2007, 01:18 PM
DV compression has a data rate of only ~3.6MB/sec, so any new HD will be more than able to be used for capture and playback, however fast drives do speed up rendering tasks.

Mac OS 10.3 and 10.4 have a maximum volume size limit of 16 Terabytes, so you would have at least 5 volumes of 16TB each to get to 80TB (40TB storage + 40TB backup). I don't know anything about setting up a storage system that big, or if it can even be done yet, but HDs are just going to get bigger.

Hopefully a better system of searching through thousands of video clips will come out in FC6.

Have you been transfering the film yourself?

mholt
03-27-2007, 04:15 PM
despaxas

I have been using Bob Brodsky in a small town in northeastern Massachusetts for film to video transfer. He and Toni Treadway are small format film officianados with an organization called the International Society of 8mm Film, or somesuch. They are really nice, professional people.

He is very careful and meticulous. He is very expensive. His set up records on Beta SP, I think, but he also makes me a dv copy from the Beta SP. I never use the Beta tape. I would still go to to him for some of the work especially the stuff on Warhol and Morrison and Voulkos, because that stuff is special, and what I am down to now is outtakes that are full of splices and I am afraid that a more production operation might just tear it up. I also have eight or ten finished films of various lengths from that era, and I would probably go to him for that too, for the same reason.

But for the great bulk of the film (some is 16mm) I will probably try something closer to home, and less expensive. I have heard of a place in Seattle that does decent transfers for a reasonable price. If you have any suggestions, this excercise is an attempt to get as much of the information I need for this endeavor, together with as little wasted breath as possible. MX

despaxas
03-29-2007, 10:04 AM
It sounds like that is some pretty neat stuff. I wouldn't worry about the editing splices too much, the transfer equipment is much more gentle than a projector. The telecines that my friend uses are a Goko TC 201 and a TC 301 for regular and super8 film and have shutters. Their film path is a little cluttered and I thought that to be harder on the film, so I got a Goko TC-20 and TC-20R that has a straight film path and is very gentle on the film, but it also has a rotating prism instead of a shutter and so I sometimes get a "ghosting" effect in high motion, though it's rare. I came to the conclusion that the low speed modified telecines are the way to go. It's not as hard as you might think. Anyway, good luck.

mholt
03-29-2007, 12:23 PM
Thanks despaxas! Do you do transfers commercially, or for hire on occasion?

Does your machine prefer tape splices or cement? Do you copy through the splices or step around them? Toward the end of the film era, I was using a butt splice that cut on the frame line, so no frames were lost, and no splice lines showed. But that is a small part of my library. Otherwise it is about even conventional tape and cement. I would consider doing my own transfers, but really, I am kind of a slob and I don't spend alot of time keeping stuff clean and neat, so pretty soon my gear is full of crap and that bugs me enough that I just send stuff off to professionals, or people who care more about clean and neat than about art. I will google Goko TC 20 to see what I find though.

Question: What are the effects of compression? At the start of all this digital revolution, 4:1 was the standard; 10:1 began to develop artifacts when decompressed. What you suggested sounds like alot more than 10:1? It probably would not matter on an i Pod, but I am still a big screener so I want to shoot for that scale. Does decompression take alot of time? Is encoding and decoding the same as compression and decompression?

I do not know what the 16 terabyte limit means. My guy at Granite Digital says 1.2 tb drives are soon to be available. What do you think of GD RAID systems? Are the guys at GD savvy enough to help put together the final items I will need for my system? I care about cost, but when it comes to the difference between $500 and $479 I do not care. I care about loyalty, but for this sort of stuff, speed, quality and dependablity are way more important than a few hundred bucks. R & D and tech support are pretty important to me too, because I do not really have much technical knowledge.

I really appreciate the information you all are giving me. It helps to keep my thoughts directed. MX

unclemac
03-30-2007, 10:05 AM
Sounds like we need to work out format and compression so we can then extrapolate total volume of storage needed. Sorry, no help there.


Uncle, I did see you there in Red Square. I think you had a pry bar and were breaking into St. Basil's. I really would not have noticed except that I was looking at Lennin, lying in his glass coffin, and he rolled over and groaned, I looked up, pretty amazed, and just caught the flash of sunlight on the pry bar. Remember those two or three huge Russian mothers who had their winter coats wide open and their skirts pulled up over their heads, gettin a sun tan I guess, standing in that Moscow sun beam, in 1990?

Ha!

Nah, that wasn't me. I was there in the fall of '89, just before winter struck. Lenin was still napping peacefully, and there were plenty of guards to ensure his peace and quiet.

I spent more time haggling with the black marketeers in Red Square. Still have that Yuri Gagarin watch someplace..... :rolleyes:

mholt
03-30-2007, 03:55 PM
Uncle

I checked out the XRAID and Big Mac machine you mentioned. Man they are really stepping up for all of us videors. I guess the double cpu and disk drives allow doubling up the work without overloading the equipment? Like rendering at the same time as capturing? Is that right?

Are there other benefits? I mean, is processing faster generally, or is one processor still running along at it's speed without assistance from the other processor? I am not a production house, so deadlines are not where I live.

I like the idea that I do not have a stack of stuff waiting in line though. I think that is a formula for lost information. Maybe data dont get bored, but I dont really believe it. It acts as impatient as warm bodies sometimes.

But by the time I get this system figured out and put together, my G5 will probably be toast anyway, so I am inclined to get where I want to be within the next couple of years.

What about the X Raid? 147 tb limit seems huge to me, but only half again more than what I originally envisioned. Despaxas sounded like there is a limit to how much connectedness is possible no matter how big the storage device. I did not find a price for it either, but somewhere in that web site I saw $1.31/gb. That is about double what I am paying now. But I am not in a RAID array. I use individual hard drives daisy chained in twos. I think that is a formula for problems too, but mostly it seems to work.

I did lose one big drive to the death squads a month or so ago. Just siezed up. Lucky for me I had dumped most of it to tape a few weeks earlier! So your counsel for back up is well given, and taken too.

Catch a moovie, "Iraq in Fragments." Nicely done piece by an independent maker on that money pit. It's mostly about Man's inhumanity to man; the age old theme of literature. He shot on tape and transferred to film for release. I like that idea, but really, I am happy with the electronic presentation. It will all be digital soon anyway.

Again, Thanks to all you guys for talking to me about stuff I need to find out. mx

Greentree_uk
03-31-2007, 04:54 AM
I thought volume size was into the petabytes these days..... i may be wrong.

seems to me you need to be looking at a san architecture. a few machines connected to the same data arrays. xsan will do this for you. I am assuming you will want to use the machine for something other than capturing for the next year and a half!!. or even have more than one capturing. seems to me you need to think about exactly what you want to achieve. you also need to evaluate different technologies for connecting the raids, what is the fastest, what will be available in the future. obviously you aren't gonna buy 100TB in one swoop, prices of these are always coming down, xsan would make sense with the ability to grow volumeson the fly. what you need is fast scalable storage and one that won't be obsolete in 5 years.... if you are doing this right you will need to think about spares, power, cooling - it will extend the life of your equipment, and where are you gonna put it all. you will need a seperate airconditioned room for all the equipment you will have - it is noisy, fibre run into it from the workstations. a fibre switch a meta data controller, I think you would be best advised to speak to apple directly on a project of this size, they will be best placed to advise you on it. there's lots to talk about as mentioned before backing up this will be an issue, you need to have an off site backup, your medium could be an xserve raid and a whole bunch of drives you keep swapping out. I have no idea how valuable this footage is nor how you percieve to get the footage to a buyers market. I would think of a couple of people who can possibly do this for you. extensis - a product like portfolio server may allow you to catalogue every piece of footage and with net publish produce a web portal for it. there's also another company I know of can't think of the name, they have software called sports code and the product you need called studio code, it allows you to add comments to footage, tags etc. I think it can then be searched for etc.
one thing I will say is it's a big invetment and you want to make sure you get it right, if you get it wrong it could mean doing it all over again......

mholt
04-02-2007, 04:28 PM
Greentree
You are zeroing in on it for sure. Thanks for thinking of me. Thanks for thinking for me too! I can put together the space. I am not sure where I will come up with the money but I do have resources. Might have to sell a Picasso or something.

I have thought about how to deflect the huge time committment on the front end. I can arrange some interns and assistants to help out. Ultimately, it has to be my eyes and brain that does the viewing and directs the cataloging, because it is my work to make. I know the material, because it came through me in the first place. But much of that was long ago. I have less than two years to get the infrastructure in place. I figure another three years to log and capture. If in five years I am ready to produce, I will be pretty happy. I will still have fifteen years or so to get some serious movie making done.

Your suggestion to go directly to Apple is a good one. I have thought of it after talking to the knowledgeable folks here. Other people know much more about this stuff than I do. Every post brings me farther along this route. With your comments I can see that it is paying off. Please stick with me for awhile so that I do not just slip off some deep end without resolution.

I have no idea about the value. People tell me I have the best archive of this kind they have ever seen. But it is only of value to a narrow slice of the culture. The material has most of its value in what I make of it. I mean, the resource is there; it can be just another home movie, or it can be a seminal movie of my era. Honestly, I don't even think much about how or where to sell it. I just see that opportunity for getting this kind of thing out in front of people is happening in a way I never dreamed possible. I am ready to do the work. I need to put the studio and tools together and I am only impatient that I do not waste a bunch of time or activity.

Everything you said is exactly to the points I need to consider and organize. With the good will of those like you and others here, I might actually get this thing done. Thank you. mx

despaxas
04-03-2007, 02:32 PM
A volume is basically just a way to manage storage space, on a hard drive or anything else. Each hard drive in your computer usually will have it's own name and icon on the desktop, and the volume size for storage is the same as the hard drive size. But you can also take two (or more) identical hard drives and have their combined storage space available as only one volume, just as if you were looking at the desktop at one large hard disk. You can then store a large amount of files in what appears to be one place (called a RAID0 or striped). The volume size limit refers to how much storage this one place can hold. So in your case, to use 80TB of storage (40TB + 40TB backup) you would have at least 5 volumes, or what would appear to be 5 hard drives on the desktop, none of which larger than 16TB.

Yeah Greentree, other volume structures can be larger, but according to a google cached apple support page (original is under "maintenance") 10.3 and 10.4 with a MacOS Extended format have a 16TB volume size limit. Maybe that will change in Leopard.

It seems to break down like this:
7.5 - 4GB
7.5.2 - 2TB
10.2 - 8TB
10.3/4 - 16TB

The MacOS Extended maximum FILE size is apparently millions of terabytes, but you have to of course store it on a volume. Since the Drive utility can also format UNIX, what is the UNIX maximum volume size?

mholt
04-03-2007, 05:06 PM
Despaxas

Let me say one more time how much I appreciate you guys sticking with me on this. Too bad you have to spend your time answering my elementary questions. What is the need for "volumes?" Does the grouping of volumes extend to external drives? I currently use a bunch of external HDs plugging them in and out as I need to. I also have one hot swappable case and three drives that move in and out of it. I just had a drive drop dead, so I am thinking of putting a 750 gb drive in that tray. Any thing wrong with that? The case is 800mbs rated and I need the additional storage space. I am starting to build up "programs" of edited material that I convert to full quality quicktime moovies. Then I can mix and match as if they are scenes, when I want too. Mostly, it is just a fairly good way to give me versions of the material, or even very rough edits that I can just drag into FCX and deconstruct them if I want too. Does this seem like a reasonable way of working.

Greentree suggested going directly to Apple. After looking at Mac Pro and XRaid and X serve Raid, I get the idea that they are really savvy about this video handling stuff. I hate to start talking to sellers until I know enough about what I am shopping for to ask useful questions and know how to respond to good (or bad) answers. Besides, I do not feel pressed for hard decision making yet. I would rather get alot of various points of view before I lock up with a certain product. Like Greentree said, "you want to make sure you get it right."

My wife (whose judgement is nearly always right on target) said last night, that I should figure out what moovies I want to make, and build a process to achieve that goal. Well, am not very goal oriented. I am process, activity and idea oriented. I make about an hour worth of moovies every week, as it is. Things continually drop out of the fog and grab my hand to drag me around. Occasionally I get a brilliant flash and stuff congeals.The technique works, for me. I also get alot of stuff that may be cluttering up the path. But I am not very happy working any other way.

Final Cut for instance is an arcane and obtuse piece of junk for me to try to deal with. Everything is referential. Nothing seems real, and at the next to the last step, everything gets delayed while the program contemplates its navel for an hour and a half. It sort of drives me nuts. But it does not crash from being overloaded so I need that. In time I will get more in tune with its idiosyncracies, I hope.

Mostly I edit in iMovie and go as fast as I can til I get an hour in the time line. I work it down to thirty or forty minutes and convert it to quicktime in full quality, or take it to DVD if I want to show something. The problem with iMovie is you can't stack stuff, except a little bit of sound, and that takes alot of time to extract. Finally the program just starts smoking and whimpering and wont do any more work. I let it work itself out, and by morning, it usually is ready to go the next step. But the mainthing is, it is direct and straightforward and I can get alot of work done easily in one sitting.

You probably dont want to spend much time listening to me whine about my problems. But I think it might help if you to understand a bit about my technique or method of working. I am pretty productive, but I do not have control of all of my material. I have no system of cataloging and retrieval. When I want a scene that I know I have somewhere, I can not find it without cycling through hours of tape and frequently, I never find it that way. I think I am putting my original material at risk by handling it so much.

Sometimes I just give up in frustration and do something else because I lose track of the thread of my idea before I find the scene I know is lurking somewhere. Like the time when Yura Kabala killed a sheep in the orthodox way, and prepared it for roasting on a spit that they turned by hand for several hours over a bed of hot coals in the hills near Sorokino, Krasnoyarsk Region, Siberia. The Russian mafia killed Yura ten years ago. Nasty buggers those Russians. We might have been best off to have nuked them when we had the chance!

Well that scene remains to be played out another day. Someone made a new movie about old men and the future. Abstracts make it sound about like my worst nightmares. I am an old man, and I live in the civil war of that future. My hippie dreams from thirty years earlier, were alot nicer!

unclemac
04-03-2007, 08:55 PM
Well since we are just kickin' stuff around here, and the checkbook is not out yet, I want to throw out other options.

Nothing against Apple's Xserve/RAID system. Have this setup at work, using one side of the RAID for a 24/7 database, and the other side for a 24/7 email/groupware system. It has been very good for this purpose.

For this system, I would think space will be a much bigger issue. Apple, just like Dell, HP, IBM and others charge premium prices for each drive sled w/ a hard drive included. Kind of a hidden cost. Sure you can get into a base Apple RAID box for about $5000, but each drive you want to add you have to pay a premium. Unless the consumer reads the fine print, you might think you can add your own drives. Not really. The empty bays in a base model unit come with dummy or blank trays. Space holders that cannot accept hard drives. So what do you do when you need more drives? Buy these modules (http://store.apple.com/1-800-MY-APPLE/WebObjects/AppleStore.woa/wa/RSLID?mco=6C04E0B8&nplm=MA852G%2FA).

That's more than double what the bare drive costs.

So in order to get the largest drive (750 GB) that Apple currently offers, and 14 of them for max capacity, the cost is just over $14,000, plus tax. And you still need a fibre channel card plus a server (of any platform) to run it. So any way you slice it, you are pushing about $17,000 - at a minimum - to have one maxed out Xserve RAID setup.

Plus: rack, space, cooling, etc.

And what do you get, space-wise, for that chunk of change? Let's not forget that you need to have a RAID 5 configuration (or similar) to protect against drive failure, which eats the capacity of 1 entire drive. So 7 drives per side, but only 6 are concatenated into usable space. So then the max capacity is:

750 GB x 6 = 4500 GB, or 4.5 TB of space....before formatting - per side.

After formatting? Dunno.....maybe 4.3? 4.2? Anybody know the math here?

If this is not a boot volume, you can fill it to over 95% full. Got that from a senior Apple engineer in the flesh, regarding a video rendering and storage question.

So for the sake of simple math, you would end up with a bit under 4 TB of usable space per side on the xserve.

Let's also remember that both sides of the RAID are separate, although you will get ~4 TB * 2, you will not end up with ~8 TB of continuous space. No easy/safe way to combine the two volumes. What you would see is two volumes mounted on the server desktop, similar to having two different FW drives.

So approaching $20,000 for two solid ~4 TB volumes mounted on a solid server.

The XSAN may allow you to concatenate multiple Xserve RAID volumes into one, but I have no experince with that. The cost would run up quickly though.....not saying it would be a bad way to go, but simply out of my league.

While I am rambling on, I should point out that the Xserve RAID is long in the tooth, and some IT types are wondering when it will get updated. Would love to see them move from PATA to SATA drives, would love to see both sides able to see each other so the RAID controllers would be redundant and we could have one large volume from all 14 drives, and would love to see them offer at least 1 TB drives. If you don't have to have this up and running in the next couple months, I would consider a "wait and see" on the RAID itself. If you do talk with Apple directly, I would ask them, very bluntly.....when is the update coming?

unclemac
04-03-2007, 09:19 PM
So what else is available now?

There are always options. Recently set up one of these iSCSI boxes (http://www.kanotechnologies.com/products/XPDIP4X7504.cfm). A couple more hoops to jump through than the Apple RAID, but good setup guide, and good support.

Some features that caught my attention:

• 3 year warranty (apple give you 1 year unless you pay extra)
• Retail HDs with full 5 year warranty (Apple drives only have a 1 year warranty)
• The drive trays are real. Not dummies. Install whatever you want,whenever you want. Nothing else to buy.

Plugs into any Mac (or newer PC) with a second ethernet port. Cheaper than fibre channel. A bit of a steeper learning curve, but more feature rich, and has been really solid for the 3 months it has been up and running.

One gotcha is that 10.4 does not have built in support for iSCSI. The box comes with one iSCSI initiator (like a driver) for a Mac. There are two third party software vendors making Mac iSCSI initiators that I am aware of.....and the rumor mill suggests that it will be native in 10.5, which will be here before July 15th. According to me. :)

You could get 16 1 TB drives in there......and with one for parity and one for automatic failover (which is lacking from my Apple model above) you would still end up with about 14 TB of usable space before fomatting. And I am guessing for less than the Apple setup.

If you are curious, you might give them a call and grill them. I found their pre and post sales tech support to be very good. They can set it up however you like and quote you with as many drives of whatever size you need, then set it up and test, then ship.

Frankly, I hope Apple copies this model and puts their trademark attention to detail on it, especially the software to configure and monitor. This could be the next Xserve RAID design, and I would be thrilled if it was.

TZ
04-04-2007, 08:34 AM
Apple just came out with an 8 x 3GHz Mac Pro.

To use your Mac Pro to encode your home movie DVD collection to H.264 for viewing you can encode 2 or more movies at the same time. That should saturate your CPUs pretty quickly and can take a long time, depending on the size of your collection.

On my machine, it takes about 2 hours for a 90 min movie at the highest quality the iPod can take. The settings are 640 vertical pixels, deinterlaced with a 2 pass encoding and 1450kbps average bit rate in Mediafork. It uses about 150% CPU according to Activity Monitor, so if I run 2 encoding sessions in parallel, I use about 3 cores and I'm still done in 2 hours.

It compresses the movie down to a file of about 1GB. It looks gorgeous on the iPod and really nice on the big screen, almost indistinguishable from the original DVD.

So the new Mac Pro would have even more capability. (http://www.apple.com/macpro/performance.html)

Someone with experience with high-def video? Some reviews and benchmarks of RAID arrays with 400MB/sec? Review Mac Pro with Sonnet PM RAID (http://www.amug.com/amug-web/html/amug/reviews/articles/sonnet/mac-pro/)
PCIe Compatible (http://www.macgurus.com/productpages/sata/PCIeSATAHostCards.php)

On a lighter note, I just used Automator to convert some files that got downloaded as "jpg.jiff" to just ".jpg" and it was easy, no programming, and now I can import those files into iPhoto without trouble.

mholt
04-05-2007, 05:14 PM
Uncle

I am a bit baffled. I posted replies twice yesterday, but they did not enter. I am a novice to this technology, but up to then, it was working okay. I will try this once more.

The hardware is twofold:
CORE is catalog, capture, store, retrieve, backup.
CREATE is interface, import, edit, export, uses the data in the Core (and other sources) as scratch drives to build movies. Of course there is a bunch of software needed to schmooze the hardware party.

Two different systems and maybe concepts are necessary for this. In one sense, core stands alone, and is eternal. It should be well enough established to outlive multiple generations of create hardware and software upgrades.

Uncle, your thoughts about fast and wide resonate with me. Greentree suggested that "what you need is fast scalable storage and one that won't be obsolete in 5 years." He also said, "you want to make sure you get it right, if you get it wrong it could mean doing it all over again." You said, "If you don't have to have this up and running in the next couple months, I would consider a "wait and see" on the RAID itself." I am waiting and seeing on all of the elements until there is some resonance.

My time budget on getting the CORE together is 18 months. I want as much as I am able to afford by then, but mainly I want it to be the right arrangement of stuff. And if $20k gets the CORE in place with room for 15% of my footage, at least I can get started.

Tell me if I am thinking right. The RAID array is a connection among hard drives that helps them to work as a unit rather than as separate units. Is a volume a unit? It should be big and fast as possible so that videofiles can be moved quickly and with out lost data--a single term might be "easily." The Fiberoptical components are for the big and wide path for easy transport of videofiles. I do not think the CORE need to have particularly fast seek times, except that the "volumes" are so large that getting into all of the nooks and crannies to find those elusive and arcane files that I request, it has to be very fast on the seek. But once the data is located, I just need to get it home uncorrupted.

I have not yet figured out how much different RAID is from SAN. SAN seems like maybe it is more elegant and functional for such a large archive. Like, maybe it was created for this kind of a problem as a result of thinking it through, whereas RAID is an ad hoc solution created out of earlier technologies that were available. Striping certainly sounds like that. But I am guessing.

Well, for those of you who are sticking with me and who are willing to see me along this path, this is where you have brought me. My methods are to ask, and using the proffered suggestions, I do some research and move a baby step forward. I truly appreciate every one of your suggestions, and if any of you are seeing my situation clearly enough to synthesize it for me, I am indebted to you from the bottom of my heart.

I already think the discussion has been very worthwhile. Many clouds have cleared, and I think I can get where I am going. Right now I think that the specifics of the CORE are the most elusive, and it is dependent on where and when technology plateaus during the coming year. When I make that organizational decision about the hardware for the CORE it needs to be solid stuff that will be supported for the following decade, I think. A CORE that I can add to and work from with CREATIVE gear that is on the edge of R & D. Thank you all immensely. mx

TZ
04-05-2007, 06:42 PM
I think there are some photo layouts of how it would look to have something with one or more computers and RAID.

SAN = networked RAID in a shared environment.

I've been cleaning up some threads and ran across some in the RAID and Server FAQ area but might not be current. Xserve + Xraid + Xsan and terminology of what various RAID options are.

I have to see things or more.

Yesterday's 4-core MacPro/Xserve could successfully render or encode two streams. Eight cores will either cut the time down to 25% or handle 4 times as many operations, depending on how soon and well video editing software can fully exploit an 8 core system.

At one time it was primarily SCSI. Well, that gets expensive and limiting and has evolved into better Serial ATA and SAS or there is Fibre.

I don't think 1TB drives will be qualified and ready until they've been out for at least 6 months in the market, and proven.

Attaching ATA drives to SCSI use to be used by Xraid and HUGE systems and has moved into SATA.

I suspect that it will be fall this year when things get past the "1.0" stage with Leopard, new Mac Pros (beyond what was announced this week) as well as updated versions of FCP 6.x - everything and one will need their "SP1" update in time.

mholt
04-06-2007, 04:19 PM
I want a little clarity on volumes.

Despaxas said "you can also take two (or more) identical hard drives and have their combined storage space available as only one volume, ... You can then store a large amount of files in what appears to be one place (called a RAID0 or striped). The volume size limit refers to how much storage this one place can hold... 80TB of storage (40TB + 40TB backup) would (be) least 5 volumes, or what would appear to be 5 hard drives on the desktop, none larger than 16TB."

What is the problem with numerous volumes? I know that it is easier to drag stuff from various places in to the editor if it all comes from a single hard drive. FC does not care so much as long as every scratch drive is online, all of the time. But I do not have a way to plug five drives into my computer. I already daisy chain two drives at a time, but my Big mac only has 2 firewire ports.

Uncle said that "striped" is problematic because of a propensity for crashes. In my pea brain, that translates into a jerry-rigged deal that garbles the flow of data. What is the designed work around? The elegant solution. Does everyone agree that SAN will replace RAID for my CORE need?

I am converting all of my edited scenes into Quick time in full quality. When I need a scene, I import it directly into iMovie, or FCX. Of course FCX leaves it where it is and just uses the reference, whereas iMovie has to copy the QT file. I am learning more and more about the benefits of each method. Is this counter productive for the direction I am moving?

I mentioned earlier that I am badly in need of more HD space. I have an extra tray for my Hot Swap box, that will take a 750 gb hd. Granite Digital has a drive for $500. Is there any reason not to do it? Is that building "scalable storage for the future?"

The GD box is 800 mbs. I have several other swappable drives that go in and out of that box. What are the limitations of hot swap systems? mx

TZ
04-07-2007, 09:29 AM
Media Vault Fibre Channel Series knowledge base (http://kb.ciprico.com/lore/category.php?id=32)

Mac Pro Quad-channel 4Gb Fibre Channel

Apple has added a new option for fibre channel cards: Apple Quad-Channel 4Gb Fibre Channel PCI Express Card, price = $999. It shows up as a new option for the Mac Pro configure-to-order.

Port Multiplier PCIe SATA II Host Adapter (http://www.amug.org/amug-web/html/amug/reviews/articles/sonnet/e4p/)

A Review of the Sonnet Tempo SATA E4P Four Port External PCI Express Host Adapter
By Arthur Whalem Creative Cow ]AJA Kona (http://forums.creativecow.net/")

AJA Video SD Video Interface and Conversion (http://www.aja.com/)
Hardware Review - AJA Kona Card (http://www.kenstone.net/fcp_homepage/review_kona_card.html)

Background: The Kona card is a joint development between Aja (manufacturers of the physical card) and Black Magic Design (developer of the software codec
www.kenstone.net/fcp_homepage/review (http://www.kenstone.net/fcp_homepage/review_)kona

AJA Kona SD System Review by Marco Solorio (http://www.creativecow.net/articles/solorio_marco/kona/index.html)

CreativeCOW leader, Marco Solorio reviews the new AJA Kona SD v. 2.1. For those seeking the latest uncompressed capture card for your Final Cut Pro system
www.creativecow.net/articles/<wbr>solorio_marco/kona/index.html

CGSociety - AJA KONA 3 on AvP2 (http://features.cgsociety.org/story.php?story_id=3991)

AJA Video is pleased to announce that the KONA 3 video capture card is being used with Apple Final Cut Pro as the editing platform for the upcoming [B]...
RAID Types and Terminology (http://www.acnc.com/raid.html)

Redundant Arrays of Inexpensive Disks (http://www.storagereview.com/guide2000/ref/hdd/perf/raid/index.html)

TZ
04-07-2007, 10:24 AM
No one would recommend using FireWire for production.

We are talking about needing to add a PCI type controller and then some type of Burly external drive cases that do support 4-5-8 drives.

One volume in stripe is faster and easy to work with.

In your video editing, I take it then that you don't need to have minimum sustained read and writes of 220MB sec or faster?

Even for photo and video editing, it is normal to be using RAID to speed things up.

A Sonnet controller, using Port Multiplier, and 4 drive cabinets , each with 5 drives for total of 20 drives, 10TB data, could be two sets. And sustained I/O of nearly 400MB/sec.

mholt
04-07-2007, 09:33 PM
TZ

What I have is a G5 w 2gb ram OSX4.8 and 250 int HD. I have about 1.5 TB of external HD in 6 separate firewire HD. 3 of those drives are for an 800 hot swap case so I have to eject and swap trays to get access to that different data. Most of my footage is on tape, and I am building edited scenes of 3 to 30 minutes in iMovie that I convert to QT full quality and store on HD as my archive, or scratch disks. I use the edited scenes as a basis for movies I make. This is alot like making any art; some take a week to finish, others take a month. One longer form piece has been in the works for a year.

These are personal artform moovies. Their audience is limited, unknown really. Occasionally I get out an obviously saleable piece, and I can sell 500 or so at $50 each which gets my cost back. Mostly I have no idea who might be interested in my work. I do not really have time or inclination to do marketing.

I have 3000 hours of film and video archives to get into a digitized format so that I have access to all of my source material. Then I can begin to make the movie I have been shooting all of my life.

I do not know what I need? You guys have been very helpful in suggesting possibilities. I am looking for the essential hardware and software solution that will give me access to 100% of my footage with a way to search that library as if it were a data base, and have random access to it to retrieve scenes that meet the criteria of the search.

I will build a core of shots, scenes or clips and compose movies from the material available. If I need a shot or idea that I do not have, I can initiate another search, and draw up more scenes. The archives remain constant once all of the footage is digitized.

The need for speed has mostly to do with the amount of material to search when I set up the search criteria. It needs to be a sophisticated enough system that it is not bogged down by multiple criteria, and numerous segments of footage in the library that meet the criteria. Once the identified sequences are located, then some how they have to be viewable and if I want to import them into the project, I can import them from an edit decision list, I think. Or the archive can be the scratch disk from which the edit program works to build the movie. FC works that way. iMovie imports the source material into itself and works with the selected clips from within the program. I like some things about that, but I am leaning to working from the archive as if it were a huge scratch disk.

The bulk storage for the library is just a part of this thing that I need. The library is the substance of which the movies are made. Ultimately, the data base structure and how it connects between the archives (library) and the editing platform is probably the more complicated problem.

This discussion has been mostly limited to the massive storage arrangement necessary to archive the footage library so that the entire archive can be searched every time, wilth systematic access and retrieval. I want whatever I need. It does not have to be Production house quality with numerous workstations searching the same material all at the same time. I will be most likely be the only one using the resource. And I am not working against deadlines, and I do not have big budgets and fat cat producers breathing down my neck for the latest bit of edit babble.

This is independent art film making at its minimal limit. I just want the best set up I can put together to finish the work of my life. I have been at it for 45 years. I probably have another 15 or 20 to get it done.

Got any real ideas about how to help? mx

TZ
04-08-2007, 10:22 AM
Have you used or looked at/into AJA KONA System?

Minimum KONA 3 System: Power Mac G5 (dual) PCI-Express, 2GHz with 1GB or more RAM. Ensure your Macintosh has a PCI-Express slot for compatibility. Use a Fibre Channel or SCSI external RAID for uncompressed SD/HD storage.

Recommended KONA 3 System: Apple Mac Pro with Dual-Core Intel Xeon Processors (2 GHz or better) with a minimum of 1GB of RAM. Use a Fibre Channel or SCSI external RAID for uncompressed SD/HD storage.

Minimum KONA 3X System: Apple Power Mac G5 (dual) 2GHz with
1 GB or more RAM, PCI-X slot required (later G5s have these slots)

Recommended KONA 3X System: Apple Xserve 3 GHz Quad Xeon or Power Mac G5 (dual) 2.5 GHz or better with 2 GB or more RAM
PCI-X slot required (later G5s have these slots)

http://www.aja.com/html/products_macintosh.html

I would assume so... but give an idea of where you are.

If I was using FireWire, I would also - if I had an open PCI slot, be using a dedicated FW800 controller. The hot-swap sounds good. There are SATA hot swap cases and many of the rack or tower drive cases also support hot-swap drive trays.

mholt
04-09-2007, 04:20 PM
TZ

Thank you for the good information. No, I am not familiar with the KONA packages. I think that is a good place to start with Apple. I will check out the link you gave, and see what it tells me.

You ask where I am at. Mostly, I am at the beginning of this research project. I am trying to find out what is the best way to archive my footage library so that I can use it completely, randomly and on demand. The main problem is that it is a very large amount of video, and mucho TB drive space is needed.

We used to say, pick the software first, then get hardware to fit it. I think that still works, except that Apple seems to be the best game around, and I have no idea what software will let me do what I need with this material. So I am trying to establish a working relationship with final cut, and I know it is compatible with Apple.

But who has video file management software that can log video information into a data base that is then searchable according to the established criteria. I want to be more specific than Aristotle's 10 principles, but that would be a reasonable place to start in another world. The editing program has to be able to be driven by the data base to retrieve the scenes suggested by the search criteria. These scenes will be reviewed to determine which ones work for the program being made.

That is the software requirement. What hardware enables that application?

I like Kona arrangements you suggested. From what I have seen by following suggestions by your Guru's, the right stuff is out there. I am still putting together the pieces that connect the CPU processes with the storage devices. I like the idea of back up that senses when a drive is in danger of losing data, and immediately creates a back up before the drive crashes. At least, that is how I envision it. The other idea of a stand alone back up that replicates the archive works for me too. Once the archive is digitized, it may have little need for additional material to be added.

However, as I create edited pieces from the archives, I will want to save them as discreet new scenes that replace the original material. So, I may continue to need space to update the footage archive, even though at somepoint, I may not be originating new footage.

Does that explain where I am at, a little bit? MX

despaxas
04-10-2007, 08:58 AM
I just ran across this, I don't know much about it but it seems like and option for logging and searching video files.

http://www.imagineproducts.com/miva/merchant.mvc?Screen=CTGY&Store_Code=IP&Category_Code=MAC

The P2 Log works with Panasonic's P2 video standard which I know nothing about, but there is a DV Log for DV files. They both output to Final Cut via XML and EDLs. Who knows how user friendly it is, but you might want to look into it.

mholt
04-10-2007, 03:58 PM
despaxas

Thanks for the link. I checked it out, but I could not find out much on the first pass. Videography had a press release from them, but no real review.

Can you tell me more about volumes? Is it easier to move files within volumes? Does a search engine have to have a path to get in and out of discrete volumes? That would make searching the archive pretty cumbersome. But it looks like, no matter what, I will have to have more than one volume. Once I start talking to these video library archivists I guess I will find out how complicated a matter it is.

Do you have an idea for a good target for me? TZ suggested some stuff the other day, and I checked it out. It was a KONA board which is designed for the kind of things I am doing. And there were other pieces in that puzzle that show me how many different possibilities there are to choose from.

What about the idea of an 800mbs controller? I think there are many hardware items that are designed better than what came built into what I have. It makes sense that there would be. Tools made for general consumption are not likely to do specialized jobs as well as those especially made for the work. Do you see any reason not to get a 750gb drive to put in my extra hot swap tray? I am really needing to to something right away; why not 750 rather than 300?

Then issue of incrementally building storage solutions that I will be able to continue to use for many years in the future. What are my best prospects for that plan, if I am needing hardware now?

Thanks for checking in on me. If I am hogging space unfairly, someone needs to let me know. Otherwise I will just keep forging ahead. I really feel that I am starting to get some understanding of what I need to think about. That is a big help. mx

TZ
04-10-2007, 04:10 PM
Keep in mind building volumes that span multiple drives. While today there are 500GB drives, and 1TB due soon, I'd wait at least 4-6 months before jumping and using a drive with such new technology. Eight Bay RAID 5 (http://www.amug.org/amug-web/html/amug/reviews/articles/enhance/e8/) I'm sure Rick could recommend some competitive products, but a good review none the less using the Mac Pro's capabilities.

There is a review linked from my Mac Pro FAQ to a 400MB/sec storage solution using 8-12 drives in stripped RAID. You see it as one volume.

And that also makes it faster and easier to catalogue and index files, projects etc.

mholt
04-11-2007, 03:56 PM
TZ

It's a great review, thorough and extensive. I wish I understood more of it. I have only been working with firewire, so I do not know what ATA is or SATA. I really do not know how it is different from firewire, or why it matters. But I am depending on you guys to steer me right.

I understand that first generations are not trustworthy in this realm. So I try to wait until a couple of versions have come and gone before I get serious. Mostly, I cannot afford stuff when it first comes out, so I have to wait until I can get it second hand, or at a discontinued price. But occasionally I have saved up, and I can just go get, what I need.

I have had good luck using Granite Digital products. They are helpful, and friendly, and accessible. I know that they also have these big Raid arrays in various formats. Early on, they were using the "next generation" bridge boards, and I was recommended to them at that time. It might be five or six years ago now. I think they are still in that position, of moving to the next level of technology while it is happening. What do you know about their stuff? Since I am not a techie, I do not read much in the way of reviews, because most of it goes over my head. But when a statewide users group rates something 4.5 on a 5 scale, I assume it is a good product. mx

TZ
04-12-2007, 07:12 AM
Another review would be the new Hitachi 1,000GB (1TB = terrabyte) 7K1000 disk drive.

Hitachi offers up some fun facts about what you can store on a 1TB drive: 333,000 high-resolution, 3MB JPEG photos; 250,000 MP3s (at about 4MB apiece); about 250 hours of high-definition video--the equivalence of about 125 movies--encoded as 9 Mbps MPEG-4 video.

So what will you do with one terabyte of storage?

Reviewed: Hitachi's Massive One-Terabyte Hard Drive (http://www.pcworld.com/article/130598-1/article.html?tk=xlr8yourmac)
The new Deskstar 7K1000 can hold 150 high-def movies or more than 300,000 high-res photos.
Melissa J. Perenson, PC World
Tuesday, April 10, 2007 06:00 PM PDT

Data storage doesn't tend to elicit wows or water-cooler conversations, but that's about to change: Hitachi's new Deskstar 7K1000, the first hard drive to reach one terabyte, is here--and it's worth the hype.

Two 500GB drives would run $300-350. So $399 isn't so bad for a single drive. But this drive, as it is today, may not be ideal based on performance.

unclemac
04-12-2007, 10:36 AM
Just a quick note.....

So you are no fan of Final Cut. OK. Can't argue with that, 'cause I don't use it. New Final Cut Suite is on the way....who knows, maybe you will like it more, maybe less.

Besides iMovie, what else is there? I get you, you want to focus on the art, not the tech. Might rule out the other big player, Avid.

Just saw this today, and thought it looked interesting:

http://pomfort.com/silverstack/features.html

http://pomfort.com/framecalculator/

mholt
04-12-2007, 12:48 PM
uncle
Thanks for the link. I checked it out, and I will start a conversation with the company to see whether it benefits what I am trying to put together. Do you know anyone who is using it that might talk to me about it? Of course, I can pose the question on another thread.

Actually, I first learned non linear editing on Avid. It was all I knew, and it worked and it was a miracle and cost $100k. It was alot like getting back to film editing, with alot of equipment between me and the work. When I left the Avid platform, I edited tape on a sony evo 9700. I still use it every day, but not for editing. I use it for capturing Hi8 tape to dv tape or directly to my hard drive, sometimes.

In 1991 all of that stuff made mushy finished video and it was still linear and I just got disinterested. iMovie changed all of that. It was such a giant leap forward, and it worked so well for first cut editing, that I thought I had finally arrived at paradise.

Well, of course Paradise is still several hundred miles from Butte and in time I bumped up against the limits of iMovie, and got a little panicky. Everyone said, "You need Final Cut!" So I bought the book and soon realized that it would probably never be made into a moovie so I dumped Final Cut while I blubbered into my burbon thinking the video fun was over for me. I tried Premiere which was better, but it did not fit the Mac platform just right, I thought. Besides, I sort of hate to learn new software when what I have works well. I tried to work iMovie around to be better, but it is just not able to do what I need it to do, plus it crashes and gets bad hair if you ride it hard, even if you do not put it away wet.

I went back to Final Cut cause that's what everybody is using. And I am getting more acquainted and comfortable with FC. So maybe the nasty part of the honeymoon is over, and we can just get it on and have some fun, soon! It is probably mostly a compatablilty problem. I can see that it is more like what I need for editing and surely that is where I will end up. But in the mean time I can still complain about the hours of useless time spent trying to figure out an operation by 1)using FC help 2)going through the chapter in the book 3)just trying this and that and this other thing 4)calling up another FC user to ask them, etc. I am a thousand miles from any place to actually get a lesson or class in anything but Windows office or Turbo Tax.

The bigger question is whether Final Cut can interface with whatever log and search mechanism I come up with for getting control of my footage library. So far I have not spent much time looking into that. mx

TZ
04-12-2007, 01:53 PM
I'm a fan of books. Maybe something in Amazon: Final Cut Pro (http://www.amazon.com/s/?url=search-alias%3Dstripbooks&field-keywords=Final+Cut+Pro)

Final Cut Pro for Avid Editors (http://www.peachpit.com/books/product.asp?st=%7B0C38F087-3720-42FB-B5AE-2117E5FA96AF%7D&session_id=%7B5FDD3AA3-9729-40A8-9D4C-6649F4136E23%7D&product_id=%7B1F044B9A-9311-4DC9-9EFA-A0463E45FA1D%7D)
Diana Weynand
The quickest, most efficient way for experienced users of Avid Media Composer to learn Apple’s Final Cut Pro. Contains an illustrated step by step operational guide so you can get to work right away.

http://www.apple.com/finalcutstudio/

Xsan:
http://images.apple.com/xsan/pdf/20061115Xsan_Tech_Overview.pdf

Xserve RAID
http://www.apple.com/finalcutstudio/resources/white_papers/Xserve_RAID_dataSheet.pdf

Offline workflow for film using uncompressed HD
http://www.apple.com/finalcutstudio/resources/white_papers/L328398A-US_L328398A_HD_Workflow_FilmTV_FCP_wp.pdf

http://www.apple.com/finalcutstudio/resources/
http://www.apple.com/support/finalcutpro/index.html

FCP Discussions:
http://discussions.apple.com/category.jspa?categoryID=123

mholt
04-12-2007, 05:05 PM
Thanks for all the great leads today. I read through most of the links. Of course most of it goes right over my head, but some sinks in, and I am gaining on the curve, I think.

The links to apple info is very helpful. I think I will pass on Diana Weynand book. Hers on Final Cut was a bust; because it was as obtuse and indirect as FC itself--if you understand, they are a pair. But I guess my head is not wired quite that way. If I need to know something, I want it straight and simple--even simplistic. I will get the far ranging possibilities in time, but I need this right now and I do not want to spend three hours going through the primer to find out about the thirteen ways to do the same thing.

I prefer to work off the keyboard cause it is faster and less debilitating than using one hand on a mouse, and the other on the keyboard. So keystrokes are the path I follow if I can. Maybe a double joystick approach might work too, or an array of joysticks or track balls.

Can you suggest a stock footage library guru that is willing to talk about what sort of stuff they use to organize their information? I have done some email contact, but mostly no body had inclination to share information.

Thanks again you guys. It is terriffic that you are out there in our world. mx

unclemac
04-12-2007, 10:59 PM
No, don't know anybody using the pomfort stuff. Just stumbled across it, and thought it had couple features FC may not, the price is right, and there is a free demo so you can try before you buy. Way over my head......most of what I know of video revolves around youtube.

Will be interesting to know what they offer that FC does not, if any of their features or advantages appear in the next version of FC that everyone seems to agree will be a big update and is long over due.

Sounds like it may be too late for an annulment from FC, so you might also want to seek counseling:

http://www.apple.com/software/pro/training/self_paced.html

Am also curious what the heck Apple has up its sleeve for the next version of iMove, due in a month or two...

unclemac
04-13-2007, 12:05 AM
Trying to catch up on some of your questions.......


The hardware is twofold:
CORE is catalog, capture, store, retrieve, backup.
CREATE is interface, import, edit, export, uses the data in the Core (and other sources) as scratch drives to build movies.

100% agreed.

CREATE is easy. Buy a fast new Mac Pro, trick it out a bit, install FC, and edit away. Easy because it is off the shelf gear, cheaper and faster to setup of the two......and I don't have to pay for it or learn FC. :rolleyes:

CORE is the biggie. The holy grail, the arc of the covenant. As was stated, have to get it right the first time. Bigger, more $$, more to plan, configure, test and monitor. Have to make to hard choice up front: Hardware brands and types, and software tools and configuration to keep track of everything.



As to RAID vs. SAN for the CORE...

Not an either/or. For what you need to store, assuming we are talking live access and thus on hard drives (vs. on tape or DVDs or something), I would think everything but an enterprise grade hardware RAID controller is out of the question. This is what is used for storage of general purpose data (file servers, email servers, database servers, etc.) at large corporations - entities that are unwilling to have down time or lost data, with an entire backup regiment too.

Here is the best, short and sweet explanation of RAID versions I have come across. Note the different versions. 0 through 0+1, and that you can click on each one to see it, and read the pros and cons. You don't have to really understand this stuff, but it might help to visualize what the heck it is all about, and how your life's work will be stored:

http://www.acnc.com/04_01_05.html

The short version is, look at RAID 5 and RAID 3.....those are the most likely candidates. So if I may be so presumptuous as to narrow the field to enterprise grade RAID 3 or 5, we can narrow the scope of the search. Rules out all consumer and SOHO hardware, and all software choices for RAID.

Everybody here likes Granite Digital stuff, and support. Good folks. Trouble is, for the CORE, they don't offer what I consider enterprise grade RAID controllers.

There are two options for hardware RAID: controllers that are installed in the computer, and controllers that are installed in the storage array. In the PC world there are plenty of robust RAID controllers that can be added to computers. For Macs there are only a couple enterprise grade RAID controllers that can be installed in the Mac. Rick sells the one of the best right here at Mac Gurus, but it is a system designed and sold for CREATE type workstations, not massive storage of a CORE system.

There are a couple others hitting the Mac market now, and maybe one more coming that I know of still coming, but these are all too new and untested to my mind. Not sure any of them really address the issue of running a CORE system; seems like everybody is going after the CREATE market.

So what are we left with that is tried and true, enterprise grade, and can scale up to what........10 Tera bytes? 20? 50? Need to come up with that number to refine this search. Regardless, I would think you would be looking at only external storage devices with built in RAID controller(s). The Xserve RAID is likely the smallest amount of storage you would want to consider, assuming we are really looking at least 10 TB of total room. You should only consider name brand, tried and true, professionally engineered hardware. Don't forget that RAIDs need to be configured and monitored, so they need to include reliable, understandable Mac software and support, and it can't dry up and blow away next year, or for many years to come......so start-ups, home-builts, and no-names are out. This is the CORE we are talking about.

With me so far?

Next we can dig into what a SAN is.

Hint: everything RAID storage is, plus more.

mholt
04-13-2007, 11:55 AM
uncle

Thanks a great big bunch. I will review the link, and some links I got yesterday that I briefly scanned and realized that there was too much good information to do on the fly, but here comes the week end, and it is supposed to be snowing here, so a few hours online will not hurt my feelings. By now you surely do know how much I appreciate your attention to this idea. MX

Disc2
04-13-2007, 11:05 PM
Hi MX,
100,000GB x 2(1 backup) is what you need to capture standard definition. Its about appox. 2 min./GB with audio. And more than double if you will upconvert them to HD. How often will you retreive the files? Remember that you need fast hard drive raid for best quality and they are usually RAID 0. Not very forgiving to the amout work and time you'll capture 3000 hours. No matter what decision you go archiving 3000 hours and naming titles for each clip of 1 hour, you still have 3000 titles to read just to get a clip you need and 6000 titles for every 30 min clip you selected.
I guess if you have a spare 6x8ft of wall, just dump them to full size DVCAM tapes of 3 hours each and log them. You can also get the best DVCAM deck that is upgradable to HD output. Anyway, when you capture them draft you still have to redigitize the footage. You must have all your tapes time coded and still locate the original tape. (work x2)
But if you want to start editing them go for KONA SD/HD or Media100 HD. Both system can upconvert your SD footage to HD in realtime or SD to SD or HD to SD. Still, you need lots of RAID 0 storage and a deck that can record HD plus a fast RAID contoller to get realtime SD or HD playback and effects. Good Luck -gerry

System and Storage

--Sony DSR-2000A DVCAM Studio Edit Player/Recorder with Fire Wire, DV/DVCAM/DVCPRO Playback, RS-422, Edit Control Panel $13,806.00
--Sony DSBK-2020 DVCAM Master Series HD Upconverter Board for the DSR-2000/A VTR $3,600.00
--AJA Kona 3 - 12/10-Bit Analog/SDI, HD-SDI, Dual Link HD-SDI 4:4:4, HD and SD Video Editing PCIe Capture & Output Card with SD to HD, and HD to SD Up/Down-Conversion Suppor $2,700.00
--Medea VideoRaid 10/3200 FCR2X Copper - 3200GB (3.2TB) Removable 2GB-Fibre-Channel Dual-Channel Fail Safe Hard Drive Array $10,800
-- Media100 HD $6,000.00
--Sony PDV-184ME 184 Minute DVCAM Videocassette with Memory Chip $38.00 each
--Sony PDV-184N 184 Minute DVCAM Videocassette $32.00 each

TZ
04-15-2007, 11:50 AM
[Compressor] now features simplified cluster setup, extended to MPEG-2, H.264, Telestream Episode Pro plug-in with VC-1, WMV, FLV... dynamic filters with timecode overlays, animated motion watermarks.

Compressor 3 is 3x faster

Motion 3 -- this year we're embracing 3D, extended all the familiar tools into cameras, light sources, navigation of 3D space... create the kind of graphics you want to create. We're adding paint. Particles, video, pictures, pen -- speed, pressure sensitivity -- and throw behaviors on the strokes to bring that paint to live. Something shockingly complicated in the past is now easy: match moving."

"We introduced effects plugs -- they've added dozens of them. GPU-accelerated realtime playback of filters and effects."

comparing Uncompressed HD and ProRez: they look exactly the same according to our source
"We took some of the great technology in Shake and brought SmoothCam down to Final Cut Pro, removes camera shaking."

cost of IO-HD $3495 (confirmed by Engadget)
demo of Motion integration with Final Cut Pro
Motion templates inside Final Cut Pro
called the IO-HD
available in july
Realtime SD-HD conversion
hardware encoder box will get into prorez 422 automatically
introducing a HD-input device
Final Cut Pro can manipulate 4k files
Uncompressed HD, Pro-rez 422 format

AJA video system, an amazing device able to convert video on the fly to HD, it works with Mac too. Final Cut Studio: 3495$ also in July

FC studio and HD camcorders from Sony, Panasonic and other companies

final cut studio 2, can manage ProRes 422, non compressed HD video supported in 12 bits,
http://www.engadget.com/2007/04/15/apple-keynote-live-from-nab-2007/

Final Cut Server
- $999 for 10 concurrent users
- $1999 for unlimited users

Final Cut Studio 2
- Final Cut Pro 6, Motion 3, Soundtrack Pro 2

Apple ProRes 422 Format
- Next-gen Post Production format
- Uncompressed HD quality at SD file sizes
- 10-bit, 4:2:2, full raster, VBR

IO-HD
- Hardware box that provides realtime conversion into ProRes 422
- Available in July
- $3495

http://www.apple.com/finalcutstudio/motion/

mholt
04-16-2007, 09:25 AM
Disc 2

Thanks for the specific detail. Are you using something like this?

I think I have decided against converting everything to dvtape. It would take an extra two years, and I just do not think I have it in me. I considered it early on.

I want the material to be logged such that it is searchable by the fields set up in the logging database, with direct, random access to all of the library. That would be the ideal. I envision writing a scenario that references the logged data base fields (each clip has an address.) The data base would locate each of the referenced files; there may be several options as per the search. I could view the options, and capture the ones that best fit the idea. Or I could capture all of the referenced files and make decisions at the time of editing.

For example, "Boy and girl (red coat) running down a mountainside in the snow, camera moving, medium shot." "High mountain canyon, winter wind, mist blowing,camera pan, distant shot." "Ice bound river falls, pan down, medium shot." "Yellow dog running and leaping sagebrush, camera tracking, medium shot." "Crowd gathering along ice filled river, mixed shot." "Fog blowing in, river scene whiteout, distant shot." "Ice stalactites break from canyon walls, camera follow, close shot." "Winter river bank, red object in background."

Another scenario: "10/1/1991 to 12/1/1991." The search should generate about 18 hours of material shot in the Soviet Union from
Tallin Estonia to St Petersburgh Russia.

Using the archive as a scratch disc would be the most direct. Creating an EDL from the data search would be another way. Then a separate scratch disc could be used, which might allow the archive to live longer without failure. I doubt there would be more than one or two editors at any one time using this material. I do not expect to try to sell footage, or make it available beyond my own use.

My equipment is mostly dvcam because I was told that it would be most compatible with my idea of archiving and retrieval as discussed above. But I have not found the technology (software?) that would enable using it that way. At this point I think the data base, search and retrieval software will be separate and stand alone with an interface to the CORE (archive.) MacGurus have been helpful in sorting out my thinking about this CORE.

Of course, there are other less elegant options for searching and retrieving files. One would be totally off line by shooting the EDL in the dark and reviewing the capture after the fact. I will hold out for a more direct method, unless it is just not possible.

Please continue to review my idea, and respond with your input. I have no idea where the "CREATE" part of this system comes from, especially the data base and retrieval interface.

I will be in the field most of this week, and away from computer access, but I will check back thursday. mx

unclemac
04-16-2007, 10:26 AM
Thanks Disc2 for jumping in.

I have no direct experience with video of this magnitude, much less converting from film, so all advice from experience is welcome.

My experience falls along the lines of enterprise storage hardware and backup techniques, but maxes out at systems designed for several Terabytes, that could extend up to perhaps 20 TB.

Spoke to a friend that works at a small publisher. Been assisting them configure a new backup server and automatic LTO tape library. They use Portfolio (http://www.extensis.com/en/products/asset_management/product_infor,%20mation.jsp;jsessionid=YKQF5O2R4RE KZLAQAAUARSQ?locale=en_US&id=prod60006), though almost all graphics, not video. He did say he thougth it would work great for archiving video. I wonder how something like Portfolio would compare to the brand new Final Cut server?

In my mind, I see converting film to video with the CREATE setup, uneditied, and storing it on the CORE in an organized, cataloged structure. Sounds like a database anyway you slice it.

If the buget (making up numbers here.....) allows for 20 or even 30 TB of storage, but that is not nearly enough, then what? All of the film has to be digitized and stored somewhere. If live storage on a RAID is out of the question because of volume and/or budget limitations, what other options are there?

Convert only some film at a time for a specific project?

Is there any tape format that would be usable?

What about Blue Ray optical? One could build a library, and it would be easier to search (non-linear) than tape, harder to damage I would think, and up to 50GB per disk. Portable, and easy to make duplicates for backup or whatever.

Plenty of software options to build and organize an optical media library, so it should be easy to keep track of what is where. Blue Ray would be rather cheap to test (based on the scope of this project) - about $1000 to set up and start burning a few disks. How long would it take? Would you have to render the video first, then burn the disk? Burning could be unattended, but I have no idea how long it would take.....would the time involved be a deal breaker?

unclemac
04-17-2007, 10:41 AM
Two things caught my eye as requirements of a project of this scale.....not that you need to use this software, but what I would think of as "specs" for anything that would be considered for use:


Final Cut Server

The server includes a cross-platform client that "enables content browsing, review and approval from within a studio or over the Internet. Final Cut Server automatically catalogs large collections of assets and enables searching across multiple volumes via an intuitive user interface."

With the CORE being so large, this will be key. Any way we slice it, I think you will end up with multiple volumes.

Final Cut 6

Final Cut Pro 6, will feature a new format dubbed "ProRes 422," which Apple claims offers "uncompressed HD quality at SD file sizes and support for mixed video formats and frame rates in a single Timeline."

With the volume we are talking about, any format that will keep the quality you want, but cuts down on file size needs to be considered.....the larger the CORE, the more important this becomes.

Which leads me to ponder.......should we lay out a list of specs for the content before moving forward?

TZ
04-18-2007, 03:36 AM
Apple's new ProRes 422 video format (http://www.appleinsider.com/articles/07/04/18/a_closer_look_at_apples_new_prores_422_video_forma t.html)

Going into extreme detail, Apple has said its new HD video standard was designed not just to beat opposing formats but to clear bottlenecks -- some of which are stifling for video editors.

Those fortunate enough to work at a Mac Pro are poised to see even more of a benefit, Apple boasts. An improved half-resolution decoding method means that Final Cut Pro 6's unlimited real-time preview sees an exponential increase in the number of active previews: where only a single 1080i, 60Hz video would be visible on an eight-core Mac Pro in uncompressed form, the use of ProRes 422's high quality mode allows four. Dropping the quality to 720p at a film-level 24Hz sees the number of streams jump from three to fourteen on the same system and increases further with the normal 145Mbps ProRes bitrate.

this performance boost is explicitly adapted to multi-core systems. ProRes is said to scale almost linearly with the number of cores available to process video, cutting the latency in drawing a finished frame in half every time the available processing power doubles.

A Mac Pro with an HD SDI video card, used for capturing live broadcasts, can automatically transform raw HD into a ProRes 422 clip without losing visual data, optionally sending it back to other hardware at the same quality.

Disc2
04-18-2007, 02:15 PM
Hi again,
Just so happen that 3000 hours is is about 1/3 year of continuous different cable or sattelite programing you can watch. The magnitude of your work is super. Not good in math but tying to crunch numbers and will be really expensive.
I guess you have to determine how much quality you can afford. Do you want compressed or uncompressed (lossless) quality?
With your existing setup. Are you using firewire, SDI, or Svideo?
Do you want your work in SD or HD? I'm not sure if there are recorders commercialy that will record HD probably BluRay or the JVC Digital VHS or Apples DVD.
Guess I'm going too far and you plans are to archive your footages.
No matter which direction you go 3000 hours in realtime video still is 3000 + ejecting and rewinding tapes.

Another scenario: "10/1/1991 to 12/1/1991." The search should generate about 18 hours of material shot in the Soviet Union from
Tallin Estonia to St Petersburgh Russia.
This scenario alone will cost you 540GB of hard drive space in Standard Definition compressed 1:2. and you need to log each clip you need for your EDL. I hope that I'm wrong but theres no shortcut even with finalcut unless you have a bunch of people that will help you. Realtime uncompressed video RAID 0, won't do the job for archiving 3000 hrs. Unless you have a budget over 100,000.00 for hard drives youll be fine. My first SD setup with micronet years back, if one hard drive fails I can replace it immediately without loosing data.

March 1, 1999
The SRP of the Genesis system with dual RAID controllers and up to twenty-eight 18GB drive modules for a capacity of over 500GB starts at $108,845. A Genesis system with twenty-eight 36GB drive modules and up to one terabyte of storage with two Fibre-Channel host connections is available for $161,995 SRP. A Genesis system can also be configured with fourteen 36GB drive modules with a capacity up to 500GB for $81,435 SRP and as a 63GB to 250GB desktop tower configuration with seven 9GB to 36GB drive modules starting at $15,995 SRP. The seven-bay module is ideal for use in small to medium sized companies and as a RAID solution to complement Apple's new servers.
Now Hard Dives have more capacity, faster and cheaper. Just look at the advantages now. Good Luck again and I'll try to read more whats new this time.

mholt
04-24-2007, 05:44 PM
I posted a reply yesterday, but it got lost. Thanks for sticking with me. All of your input is helpful although, it seems that I am repeating my self quite abit. Maybe I am just not clear up front. Anyway I am back from the war with the Montana spring--winter to most of you. Spent last week out in the snow and sleet, I think I probably wrecked some electronics. They make great gear for people outside in the cold and wet, but equipment is not really made for the work. But it was beautiful, none the less. Green stuff is springing up through the snow, and inspite of the cold. Pretty weird which plants can stand it, and which ones cannot.

Disc 2

"how much quality can you afford. Do you want compressed or uncompressed (lossless) quality?"

I am not a nut about "quality," but I have good imagery, and I want it to be able to be projected and look good. The CORE (archive) must be as good as I can afford. DV Cam/mini DV to firewire hard drive seems to make a pretty good CORE image. It is second generation, and I am sure that it is compressed. If I can keep the CORE intact, it will be acceptable. If it is generated very many times in the digital realm, it gets artifacts that I do not care for. But I am okay with the second generation that results from copying the original tape to the hard drive through firewire. Of course, from Video 8 and Hi8, it is a third generation, but the second generation to DV tape appears to be a better image than the original analog. There is probably a way to go directly from analog 8 to hard drive without an intermediate step to dv tape. That would save time, and generation, I think.

"With your existing setup. Are you using firewire, SDI, or Svideo?
Do you want your work in SD or HD?"

I am using firewire, except when transferring V8 & Hi8, then I use Svideo. V8 does not benefit from Svideo, but I use it anyway just to keep from changing cables. Video 8 and Hi 8 never look like HD. So, I dont think that HD is critical for me.

The film would transfer to video in HD and look great. But the aspect is haywire. Ultimately, 16:9 will be the aspect I suppose so that will level some things. I do not intend to change out all of my gear for HD. I am at the last quarter of my life, and the edit is what I am trying to gear up for.

Think of the CORE (archive) as my stock footage library. It is the rushes from the entire lifetime shoot. That CORE is what I will make moovies with for the next 20 years, or until I drop dead between now and then. I have made a bunch of stuff with it already. Some of that I will recut knowing what I know now and having benefit of the energy that I had then.

I need to first transfer and catalog this material. Then I need to be able to go into it and pull out different scenes from it to build the moovies that reside within it. Some of the moovies will be straight ahead documentaries of whatever the activity was--People's Park in Berkeley; Andy and Paul making Lonesome Cowboys; artists in the soviet union losing their known way of life as communism drops dead all around them. Some of the moovies will be rhythm poems of flying, and driving and train riding and horses running and dogs leaping through sage brush jungles, tens of thousands of cranes or snowgeese moving around on the Platte River, or Freezeout Lake. Some will be stories of artists and cowboys and constitutional conventioneers and the rhythms of life. I want to have the entire library accessible and cataloged and unified for whatever impulses drive my editing direction.

The Core stands alone as the retrievable source. The Create side is the interface and the working tool set. From Create I can go to Core if I need more material. And when I have a finished edit, CREATE goes back to CORE to construct it if that is the way the system works. If I can retrieve from the CORE once and never go back to it, that is okay too. I think that would be a more expensive system. Maybe in the interests of cost, CREATE should use a more compressed format that produces an EDL referencing specific addresses in CORE and goes back to CORE to build the finished piece from the EDL.

I can see that this would mean a more complex CREATE module. But the editing systems out there now (like final Cut) are like that. So if it is the standard, it may actually be less expensive than trying to work in full quality all along the way.

This thing will not be used by several people at once. At most, my wife, and I will use it. It is our source for the purpose of making our moovies. I do not need a stable of editors to be able to access numerous parts of this at once. It is conceivable that CORE would go for months without being used. CORE is the unification mechanism to make all of this material easily available to me for moviemaking. Setting it up on the front end is a focused labor of watching, logging and organizing the imagery and ideas that exist in this body of captured motion picture material.

I envision spending several years (3-5) just putting CORE together. I can find some help for this once I have the organizing principle and data base formula in place. But it is also a way for me to review my life's work as an artful moovie maker and understand what is in there, and figure out how I can use it to tell the story of my era. It starts in 1940 and goes by way of the 60's into the second or third decade of the 21st century. I've been everywhere in that world and I think I have a unique view of it that is my own, and I want to be able to produce it into moovies.

I really thank you folks for helping me figure this thing out. Maybe it will be a model for others to use. Or maybe it will live and die with me. I do not care. But I need to get it together. Hang in there, we will get it done. mx

unclemac
04-24-2007, 07:10 PM
Hurry up and get going.........I wanna get the popcorn popping already. :p

mholt
04-25-2007, 12:42 PM
Uncle

You are a sweetheart! I could send you some stuff to wet your whistle, in the mean time.

Your suggestion to settle on the parameters for the quality of capture to the CORE is a good one. I am thinking about that. In my post yesterday, I yakked it up quite a bit. And I looked at the engaget site suggested by TZ. They are talking about stuff that unifies multiformat material into a single HD condition. It is too amazing to believe.

It makes me wonder if I want the CORE to be less than full quality. I mean, down the road, if equipment can take a mishmash and put it out at HD resolution, then compromising on the front end is foolish. But 100tb x2 is probably more than I can afford. I liked the 40tb x 2 suggested early on by Greentree I think. But that means alot more compression than what is on the original tape. Is there a happy medium? I need to study up on the specs I guess. What is 720 x 640? What is 1020 x 920? I know this is elementary stuff, but I need to get a reference.

I transferred a piece out of final cut last night as quicktime. It did not say Quicktime full quality and I put it into iMovie and made a dvd from it. It had some abberrant spots, but I think they were in an area that I had brought in originally from mpeg 2, (a dvd that was made entirely within the apple realm by a local apple officianado.) It looked okay on the TV this morning. I do not think it is a very good test, but I think the output was 720 x 640. Those are pixels, I guess. Is that SD? Is that less than SD? What size is HD? What size is the image on my DVCAM tape? Do pixels measure all of the pertinent information, for this discussion? Is there a better formula for talking about quality? mx

mholt
04-27-2007, 04:07 PM
Resolution of the stock footage library should be enough to allow better than 720 X 480 pixels which is dvd quality. The stuff I read says that HD is 1920 x 1080 or 1280 x 720. Those are both 16:9 aspect ratios. Television stuff is rated in lines. DVD is 480 lines. Some video cameras record more than 500 lines, I think. Anyway, with compression and decompression, artifacts get added, so I would like to avoid that as much as possible.

Please advise me about compression schemes and these new systems that can add information to boost SD to HD. Or is that just a gimmick? I want to output to show on a 17 foot wide screen, to television and to iPod. The stuff I have done to date blows up to movie screen size okay if I out put to DV Cam tape. I think that is really about the same as mini DV. There is just some additional data information on the tape. It aint Hollywood, but it is credible as a projected image for an 800 seat theater. I guess I want to continue to work to that spec.

The data base should handle 20 or 25 fields. I could get by with 10 or 15, I think, but I have not worked it out. I need the obvious stuff, who, what, where, when and some details like motion, color, season, direction of movement, pan, zoom follow/track, voice, music, noise.

I really need to see how some stock footage libraries are set up, I think. And I agree that the entire archive will be several volumes. But I do not know what sort of problems that imposes. MX

mholt
04-30-2007, 02:37 PM
Are RAID drives of a very specialized type?
Can hard drives for RAID arrays be hot swapped?
Can I use my existing hard drives to make a RAID array if I get a RAID enclosure?
If I need a RAID 3 array, does that change any of the above answers?

I am badly in need of an additional hard drive. I am cleaning up space on my drives by exporting finished pieces to dv tape. I would rather keep the finished pieces as stock footage for future moovies. If hard drives are RAIDable then how do I select a drive that is compatible with the future use. Please advise. mx

TZ
04-30-2007, 02:56 PM
Usually you want drives that have 5 yr warranty, and there are some that are designed more for RAID (and some that are not).

WD Enterprise are fine for RAID.
Seagate is what Rick handles and uses for customers.
Hitachi is also an option (and their 1TB drive looks to be 'interesting').

Generally you can use a drive for multiple purposes.

RAID 5, 6, 10, 50, or even 'just' stripped RAID0 are commonly used. RAID3 was a few years ago.

Hot-swap depends on the drive trays and case, and controller, not on the drive itself.

The following is one idea of what can be done

RAID Storage Subsystem
Norco DS-1220 (http://www.norcotek.com/DS-1220.php) combines two five bay SATA PM enclosures using bays 1-5 and 6-10 with a dual direct connect enclosure using bays 11 and 12.

Enclosure combines both SATA hard drive mounting methods. A ten drive SATA PM enclosure using ten identical model hard drives can provide a high performance striped RAID set with over 400MB per second read and write speeds.

A great use for this type of high performance striped RAID set is for working with large digital video or audio files. The striped RAID set provides high performance for editing and playback, but users still need a method for backing up important files.

This is where the two hot swap direct connect drive bays located in bays 11 and 12 come into use.
Once a user reaches a stopping point or a final edit they can use bays 11 and 12 with large hard drives for backup. Can you imagine using Seagate 750GB, or a pair of 1TB Hitachi hard drives, in these two bays to backup the days work?

Use a fast striped RAID set for editing and still have one or two SATA hard drive bays available for backup or import new content.

The ability of the Norco DS-1220 to support a ten drive striped RAID set and still have two drive bays available is an awesome feature. AMUG Review (http://www.amug.org/amug-web/html/amug/reviews/articles/norco/1220/)Features:
3U rack height.
Very cost effective near-line storage library.
4 port eSATA host interface (Silicon Image SiI3124 + SiI3726 Multipliers).
1.5 Gb/s SATA Gen 1 and 3 Gb/s SATA Gen 2 host and drive support.
Supports up to twelve 3.5" SATA-I or SATA-II disk drives;
Excellent cooling system ensures the drives reliability;
Hot-swappable disk drive tray with special designed power-off and lock mechanism;
Monitor and alarm annunciator for temperature and fans
Uses two Silicon Image 3726 chips.

mholt
05-01-2007, 04:34 PM
Thanks TZ;
Helpful information. I checked out the links. I am beginning to understand the hardware side of this problem a bit.

Can you direct me to some software solutions to interface the drive/storage component with the cataloging, data access and retrieval components? Is there an area of this forum that discusses the kind of software I am thinking of? Does anyone understand what I am looking for? It is the organizing principle part of this discussion. Thanks for all of your collective help to this point. mx

unclemac
05-01-2007, 05:09 PM
We can keep going right here, so everything is together.

Do you think you could describe all the functions you want? If it is a long wish list, you might also rate them....nice to have vs. essential?

mholt
05-04-2007, 01:02 PM
Uncle,

I am not sure what you mean by functions but I will try to describe what I envision.

CORE is my archive (a life time of film and video making material.) As I capture clips into the core I will log them by a set of terms. Each term will represent a searchable data field. A clip may require several terms to describe it. I think of these as terms: name, man, woman, child, animal, dog, horse, elk, vegetable, tree, grass, crop, mineral, stone, oil, dirt, chrystal, machine, auto, tractor, boat, ship, liquid, river, lake, ocean, oil, solid, wood, stone, steel, glass, ceramic, vapor, air, cloud, smoke, movement, pan left, tilt up, zoom in, follow, fast, slow, leftright, rightleft, bounce, jump, run, drive, roll, view, close, distant, med close, urban, street, building, town, city, rural, pastoral, mountain, valley, cliff, activity, play, work, dance, drive, walk, run, jump, objects, shovel, cards, rope, rifle, axe, pottery, painting, sculpture, kiln, welder, crucible, places, studio, home, museum, Yellowstone Park, Stolby, Yennissea, Hermitage, wilderness.

Maybe there will be 100 terms. That is a huge number of fields for a data base. Perhaps the "terms" must be more generic. Maybe another level of detail can happen within each term.

If I want a shot from a "sail boat on the Yennisea river with Rolf and Annie and a dog pan leftright", I type the phrase, and the system searches for the clips that fit all of the criteria. If I said "boat trip on a river with a man, woman and a dog, distant" it would probably retrieve the previous clips and other clips besides.

I could preview each of the clips and select a few to look at in full scale and motion, decide which of the three or four I wanted to use, and capture those clips into the editing program.

Something like this is what I envision. If I have to scale back from this level of sophistication, so be it.

I am working with at least three general needs:
One relates to films on specific topics where nearly all of the material will be from a set of shoots within a certain time frame.

One relates to films on specific topics where the material is separated in time and probably mixed in with other subject matter.

One relates to films that will draw imagery from any time and or place or topic as long as the clip fits the need of the moovie.

Well, of course, at this point, none of these are films. They are not really even videos. But they are movies and that is what I make. mx

mholt
05-17-2007, 03:42 PM
TZ,

thanks for thinking of my problem when replying to the timecode question.

I have talked briefly with people at apple. They suggest that I try Portfolio to see if that sort of thing will suit my needs. I reviewed Portfolio 8 on their site, and for $200 I am tempted to give it a whirl. At least I can try some things out with it, and when Final Cut Server gets a few versions under its belt, and some bugs out in the open, I may know whether to jump over there.

I am still up in the air about hardware, but I think you guys have me going in the right direction, and I can tweak that too as I see how heavy this system needs to be. Lots of fast number crunching capacity but I do not know how complicated it really needs to be. San may be overkill.

I keep Greentree's advice close to my brain. "Get the distance figured out before leaping the chasm." mx

unclemac
05-17-2007, 11:42 PM
Been meaning to get back over here, but, you know how life is. :rolleyes:

Been meaning to mention it...did I mention it?.....Portfolio: yes. Help a publishing company with their Mac, IT, backup and storage stuff, and they use Portfolio. Love it.

I asked one of their guys their what they thought of using it for a project such as this ;) and he said he thought it would be great. Their work is all still images and graphics, so he could not speak from video experience, but thought it would be a good tool for this. He also said it was great for grabbing and storing meta data (in a searchable way as a database). So I would give it a shot.

Still wonder how FC server fits in here....would it overlap Portflio, or compliment it? Will let you know if I find out.

As for hardware, you will need lots of room and reliability, so you will be getting a enterprise grade RAID something....SAN or not. Just got a quote on a Kano Technologies XSPAND RAID for the above publisher for backups that included 6TB of space, and it was right about $10,000, with Room to grow (empty drive bays). A pretty good price, all things considered.

We still don't know exactly how much space you will need until you have a format selected. The missing pieces include: how big will this video be once you finalize how it will be digitized, how much will you keep spinning on RAID(s), and what to do with the rest?

Oh, and if worse comes to worse, and none of the software out there let's you store and search the data and descriptions you want, it would not be that hard to make a slick little data base that would index all of your films the way you want, and seach the way you want. You could seach by the critera you want (based on your examples above), get a found set, further refine the found set, and narrow down to one or several matches. The data base would then give you an index numbet that you could find on your storage.....

Clips might have a numeric code, and you simply find and export the clip you need to the work station for copying/editing.

You could have fields to seach by with drop down menus of all possibilites:

Continent [Asisa, Africa, etc]
Country [USA, USSR, etc.]
State [CA, British Columbia, etc]
City [London, Moscow, etc.]
Location [Indoor, Outdoor
Environement [Woods, Mountains, Swamp, etc.]
Year [1972, 1973, etc.]
Season [Sring, Summer, etc.]
Person [Man, Woman, Boy, Girl]
Object [Gun, Pottery, Puzzle]
Activity or Action [Run, Sew, Pray, etc]

100 fields is no problem. If you could narrow down and list all the categories like my list, then you could fill all the data in those categories, and it would be pretty easy and logical to search, and easy to build the search functions.

This could be built for a nominal fee.....:D

Also saw this stuff, which is may be helpful:
http://www.studionetworksolutions.com/products/product_detail.php?pi=7
http://www.studionetworksolutions.com/products/product_detail.php?pi=8

Their hardware looks pretty good too. Might be a good resouce, since they are geared towards studio video work, whereas most larger storage solutions are more generic business storage based. These folks might have some expertise regarding tricks and best practices for video needs that one would not expect from most venders.

despaxas
05-18-2007, 10:56 AM
Resolutions can be misleading. It all depends on whether they are assuming square pixels or rectangular pixels in their dimentions. Here's the skinny:

All video before High Definition is considered Standard Definition.

The NTSC standard that Canada, the US, Mexico, parts of South America, Japan, etc. use is 525 horizontal lines of video. Because of the phosphorus used in early televisions, 30 frames per second video would flicker due to the upper portion of the screen darkening while the electron beam was displaying the lower portion. To solve this, the video was interlaced, meaning that one frame of 30fps video was split in half displaying the even lines (2,4,6,8, etc.) first, then coming back to the top of the screen and displaying the odd lines (1,3,5,7) for a total of 60 "fields" per second. When this was implemented, 60 fields per second happend equally over time so motion was continuous with each field. (That's why with interlaced video files stored on computers it's important to know which field is first [dominant], the even or odd lines, because othewise motion looks jerky since each pair of two fields are played backwards). The pixels on televisions were rectangular because of the space needed to represent color, and their color space is represented by YUV.

So now we come to try and digitize this stuff, and all hell breaks loose. Computers use square pixels, operate in the RGB color space, and don't interlace video. It used to be that video cards captured SD video as square pixels (640 horizontal and 480 vertical, a 4:3 aspect ratio) so it would look correct on computer monitors, but finally they moved to correctly capturing video as rectangular pixels, represented as 720x480 a 3:2 aspect ratio, and relied on the editing application to display it without distortion on a square pixel computer monitor. (Pro capture boards use a CCIR501 standard that is 720x486)

So the point is, it's all SD video, it just depends on your point of view as to how to represent it. Also, 720p is referred to as HD video, but it is just SD video playing back progressive content, which is only some movies at this point. Anything above 720x486 is HD video, whether interlaced or progressive.

If you have to $, transfering your film to an HD progressive camera is definitely the way to go, but storage would be a problem. A transfer system with a SD progressive camera (24p) would be second best, but I'm unsure of a telecine to do that (16~18fps to 24fps). Or just telecine to DV tape, and then from tape in camera to computer via firewire. That's how I do it, but I don't need to search through thousands of files :).

mholt
05-28-2007, 03:05 PM
Thank you for sticking with me. All of your advice is helpful. Uncle, I will try those links. And I am going to try Portfoilio 8 on a trial basis. I think I can make a few trial projects to simulate the variety of subjects in my library. Then I can try to use each of them in different ways.

I have also been thinking about the data base/catalog concept you suggested. I am not sure why I would reinvent a wheel, if it already exists. I guess it has to do with what it might cost to create the entire footage library and keep it on permanent Raid access. It seems especially like overkill if I only use a scene one time, and the rest of my life it is waiting in the wings, never to be called up again.

I guess that is an uncertainty for me now. It seems overindulgent to put all this stuff up knowing that I might only use each piece once. If it was a stock footage library for all the world to use, perhaps I could justify it. But lots of this stuff is licensed to me personally by verbal agreement at the time, on the fly, and I question whether that allows me to sell it to the world. I think I would need hard copy releases and signatures, which is not possible now that lots of the subjects have died or disappeared. It would take another lifetime to locate most of these people, I think.

Despaxes, thank you for your good input. I like the way you layed out the mechanical difference between SD and HD. Too often it is presented as if it is some sort of black magic. I have been checking into film to video transfers, and it seems that $0.20 per foot is cheap for reasonable quality. $0.50 is standard for HD quality ("Better than film".) That makes a 400 foot reel cost $80 to $200 for 15 to 20 minutes.

Actually, the cost may be less of a problem than the time it would take me to transfer the stuff myself. Time is looming as more and more of a hurdle that I need to have someone else step. There are some local communal interests developing, so it may be something I can cost share with other makers around here who also have film to transfer.

I will try out this transfer service though, to test the quality of their work. I hate to turn my back on Brodsky, but the time may have come. I will post the link for you to check out later in the week.

As I reread your last post, it sounds like going directly to computer is one option and the other is to go to tape. I figured that I would have to go from film to dv tape. And from analog tape (V8 and Hi 8) to dv tape. In either case, dv tape is the format from which the material will be captured to a hard drives Raid or other.

If the digital recording machine is capable of recording 720x486 do you think that is the best way to go? My problem with that is that most of my material is 4:3 aspect, whether film or video--well, the super 8 film is closer to 3:2 I think. I think about half of my film is S8. I am not sure how to interpret what you are saying about whether to use SD or HD in the film to tape transfer.

How does this affect the V8 and Hi8 to dv transfer? Do you think I would benefit from going HD with that too? Please clarify your thinking on this point.

Again, I thank you guys for sticking with me. I hope we can share some material. Let me know how to make that happen. mx

despaxas
05-29-2007, 01:55 PM
Film would be the only format that should be transferred to DV tape before being captured into a computer. I transfer film to DV tape first so that I can pause recording when a reel runs out, so the tape ends up with all the film images back-to-back. It also helps with film that breaks a lot. If you get it done somewhere they will most likely give the transferred film to you on DV tape also, probably for the same reasons. Here in Austin, .12 to .14 a foot is the average cost for film to MiniDV transfers, so .20 seems a little high unless it's a slower-than-realtime telecine. Although there is a lot of competition around here, so it's probably not the national average price.

S/VHS/C, 8mm/Hi8 tapes, BetaMax, or UMatic wouldn't gain anything by being converted to HD, and there's no reason to make a DV tape from them, so they can just be played back and captured into a computer via an analog to DV firewire converter like the ones found in most Sony and Canon DV cameras. Of course there are other stand alone options, but I have no experience with them. (Other than the JVC SVHS-MiniDV deck mentioned earlier, the HR-DVS1U is a great deck, but not good for analog to DV conversions via firewire since it requires an external time base corrector to ensure a constant DV stream over firewire).

8mm film would gain a little clarity if converted to HD, but if you are going to mix sources like film footage, then Hi8 tape video, then some more film footage or whatever in the same project, then they should all be the same resolution, all SD or all HD. Although with this many files, it's probably better to have them all be the same types of files anyway. SD DV files, DV tape and DVD video are all 720x480 3:2 as rect. pixels (the same as 640x480 4:3 as square pixels, but not usually used in referring to those formats). So in your case, Standard Definition DV files for everything seems the wisest route.

Also, there is an app called Metadata Hootenanny that searches through quicktime movie file metadata and acts like iTunes does searching through ID3 tags on MP3s.

It's here: http://www.applesolutions.com/bantha/MH.html

mholt
05-29-2007, 04:12 PM
Here is the web address of the transfer place I mentioned, Videoconversion Experts. I have no experience with the quality of their service, but it looks like they have a pretty good handle on the possibilities.

If you can suggest places in Austin to try, I will send a 3 minute reel to several different places to get some comparisons. I have also looked into the telecine units you suggested, and they are available around.

I dub to dv tape because I do not have a direct conversion piece of equipment. Maybe when I actually get set up to do the work, I will get the equipment required to convert directly.

Early on, I had a couple of cases of 3hr dvcam tapes, so it was no problem to copy analog video to dv tape, and not tie up my dv editing equipment. My mini dv deck is able to record the 3 hr tapes, and actually get 4 hours of mini dv signal onto a 184m dvcam tape. So, I got about three hundred hours of analog video to digital video by that means.

It is poorly indexed (no one but me to blame) but I figured to have to do that when it goes to hard drive anyway, so I was not careful about it. There's another reason to think things through completely before just plowing ahead. Ah Well!

I looked at the MetaHootenany website. Might be worth looking at the freeware to see its possibilities. Ultimately I am willing to drop a thousand for FC Server--or its like--if it really does have the capacity I think I need and want. Oops! there goes the Matisse sketch. Watch for it on Ebay!

Thanks again for your help. mx

despaxas
06-12-2007, 08:05 AM
I looked at their website and their higher end transfers use a slower-than-real-time transfer which is the best way. The places here in Austin, including myself, use real-time telecines because of the large amount of film to be done. It still amazes me how much film is still out there. Now they make it seem like real-time transfers are inadaquate, which I don't believe because people are continually wowed with the results, especially if they have gone somewhere else first and were not satisfied. That happened twice last week. One person called back crying because they were so happy with the new transfer. But these conversion people are correct in that the frame-by-frame restoration is the best way to remove scratches. So, IMO, the bottom line is that 8mm and Super8 film is well served with a quality real-time transfer, but rare film like yours that needs to look it's absolute best would benefit from a slower-than-real-time transfer like these guys advertise. Sending precious film through the mail during the summer heat is a little risky to me (although you are probably located in a place not as hot as Texas), but if you pack it in the center of an oversized box, it would probably be alright. Best of luck

Jeff

mholt
06-14-2007, 12:36 PM
Jeff

I also connected with another guy who proposes the best service. Bruce Mayfield, d.b.a. film-to-video.com.

I just heard an NPR report on the 67 Monterey Jazz Festival. It was the first big outing for Janis Joplin and Jimi Hendrix. The year ended up with Woodstock. We were on the road that summer heading from Missoula Mt to graduate school at the U of Az, by way of Seattle (Moby Grape, Jimi Hendrix) and San Francisco (Haight Ashbury) and shooting film all the way. It is a terriffic memory. I have some decent material from that trip, but I also missed a lot of stuff--too much LSD maybe, the distraction of a friend in SF who cut his throat trying unsuccessfully to do suicide and the theft of my 9 string guitar.

But what it made me think of is that I have a bunch of stuff from that year, and I ought to do a retrospective of some kind with it. If I do, I will need to transfer some film for the edit. Interested to help?

I am focused on finishing up this piece on Peter Voulkos this month, and I have a couple of other half done projects waiting, but the summer of luv might be a good project to wind up the year. MX

Let me know how we might connect to discuss possibilities. mx

despaxas
06-19-2007, 07:24 AM
Sure, shoot me an email since you have no listed contact info.