PDA

View Full Version : scratch, work, & storage: 2 vs. 3 separate drives (if no RAID)?



markwoods
05-28-2007, 12:12 AM
All the excellent advice I have read on this forum about the "workdisk/scratchdisk" method for Photoshop seems to apply to systems using a striped RAID array volume. Since RickS, among others, has advised me on the phone against going with RAID (because my particular G5's bus speed is too limited to benefit much from RAID), I would like to know how to implement the "workdisk/scratchdisk" method without RAID, if such a thing can even be done.

I understand that I should keep my OS and applications on the outside (fastest) volume of a first SATA drive with its own bus, and that I should point scratch to the outside (fastest) volume on a second SATA drive with its own bus, and i think i understand that this "scratch volume" should be kept entirely empty except for the temporary scratch file(s) (and with the possible additional exception of the one image file currently being worked on? Yes or no?).

What I especially don't understand is whether my "file save directory" for the one image file being currently worked on should be on the scratch volume, or on a different volume on a different, otherwise-completely-empty outside (fastest) volume on a third SATA drive with its own bus. In other words, in a RAID-free workdisk/scratchdisk method, is the workdisk ideally the same as the scratchdisk? or different?

Also, regarding the multiple Photoshop image files that I need to store while they're not being worked on: if I store these on a volume different from the scratch volume and the work volume, can that volume (the "cold-storage" volume, if you will) be on the same drive as the scratch volume and/or work volume, so long as it's the inside, second, slower, volume? If it can, should it be on the same drive as the scratch volume, or on the same drive as the work volume? Or since this is RAID-free, should the cold-storage volume just be the outside, fastest volume of the third SATA drive, where the one currently-worked-on file is both saved for work and stored long-term, so long as this volume is on a drive different from the scratch volume's drive, and so long as it remains half empty with at least 100GB of free space?

Or does my scenario's being RAID-free make it irrelevant (regarding speed) whether cold-storage, scratch, and currently-worked-on-file-saving are all located on the same volume, so long as it remains half empty with at least 100GB of free space?

Thank you,

Mark

Boots
05-28-2007, 06:14 AM
It depends..

Usually, when maximum RAM is installed, and the hard drives are fastest possible- the remaining bottleneck in fast Photoshop operations tends to be waiting on a file to finish saving or waiting on a file to finish opening. This is particular obvious with larger Photoshop files, and those which contain many layers, or are compressed TIFF files.

In a two-drive situation, it's usually best to have all files- system and data- on the first drive, and the scratch file on the second drive. You could use a second volume on the scratch drive as image file "cold storage", then use WDSD method by copying the image to be worked on to the scratch volume. You might see a slight improvement but this is usually evident only with a really fast RAID0 as scratch disk. And it can be a pain to manage the multiple versions of the resulting work.

If you throw a third drive into the mix- just as fast as the other two and on it's own channel- and use it just to store image files- then open and save the image files from there. You should see a bit of a speed boost.

Another possibility with a two-drive setup when working mostly with smaller files is to move Users (and all image files) to the second drive, and point the scratch disk to the default startup disk alongside the system. This too will improve open and save times slightly.

In practice, I found the WDSD method only really worthwhile when dealing with huge files. Nowadays, I usually do a RAID0 startup disk containing all files except scratch- which is on a separate 4-drive RAID0. With the image file on the striped startup, this gets me the fast open and save performance I'd get with the WDSD method using my 4-drive RAID0 scratch disk. And file management/backup is kept as simple as possible.

When I tested splitting the fast 4-drive RAID0 scratch disk into two volumes- scratch first volume, image file storage second volume- then opening and saving the image file from second volume- it just was not as fast as putting the image file to be worked on on the scratch volume; thus was born the "Work Disk/Scratch Disk Method".

markwoods
05-28-2007, 07:10 AM
[QUOTE]This is particular obvious with larger Photoshop files, and those which contain many layers, or are compressed TIFF files.

the files i was writing about are 1.5GB to 2GB files from scans of film, each of which will receive a day's worth of excruciatingly careful photoshop perfecting with 7 to 15 layers, then flattening, and more layers of careful sharpening trial-and-error. But i also often work quickly through 1000 files at a time of 10-50MB each shot digitally at weddings and parties and such.


In a two-drive situation, it's usually best to have all files- system and data- on the first drive, and the scratch file on the second drive.

I think i learned from your excellent Photoshop Accelleration pdf that this applies more to the 10-50MB files than to the 1.5GB-2GB files. Yes?


You could use a second volume on the scratch drive as image file "cold storage", then use WDSD method by copying the image to be worked on to the scratch volume. You might see a slight improvement but this is usually evident only with a really fast RAID0 as scratch disk. And it can be a pain to manage the multiple versions of the resulting work.

the nuance of this estimation is EXACTLY the kind of thing i was hoping for from you! THANK YOU!


If you throw a third drive into the mix- just as fast as the other two and on it's own channel- and use it just to store image files- then open and save the image files from there. You should see a bit of a speed boost.

"a bit," meaning appreciably more than the "slight improvement" you mention in the previous paragraph? or barely more? and when you write "use it just to store the image files," do you mean i should store only a few currently-being-worked-on files at a time there, or that i can load the drive to (for example) 50% full or 80% full with images that i then open and work on and save?


Another possibility with a two-drive setup when working mostly with smaller files is to move Users (and all image files) to the second drive, and point the scratch disk to the default startup disk alongside the system. This too will improve open and save times slightly.

yes, i remember this from your Photoshop Accelleration pdf. My currently-being-worked-on-image files, large and small, will never be on the startup disk. Maybe i could point scratch to the startup disk when i'm working on small files, and then remember to point scratch to a separate disk for large files, closing and re-opening photoshop after each pointing revision. does that seem practical to you?


In practice, I found the WDSD method only really worthwhile when dealing with huge files.

when you write "the WDSD method," do you mean that the WD and the SD are one and the same? or that the WD is on a drive separate from the SD? and is this equally true in a RAID scenario and in a RAID-free scenario?


Nowadays, I usually do a RAID0 startup disk containing all files except scratch- which is on a separate 4-drive RAID0. With the image file on the striped startup, this gets me the fast open and save performance I'd get with the WDSD method using my 4-drive RAID0 scratch disk. And file management/backup is kept as simple as possible.
When I tested splitting the fast 4-drive RAID0 scratch disk into two volumes- scratch first volume, image file storage second volume- then opening and saving the image file from second volume- it just was not as fast as putting the image file to be worked on on the scratch volume;

i think i understood this already from your other excellent writings on the subject, but, my problem is that i don't understand whether it applies to my RAID-free scenario. does it?


thus was born the "Work Disk/Scratch Disk Method".

in a RAID-free scenario, does this method entail putting the currently-worked-on-file on the scratch disk (which is also a work disk), or on a separate work disk?

sorry so confused,

Mark

TZ
05-28-2007, 07:24 AM
You need a Mac Pro for the work you do. Anything else is just a stop-gap measure.

markwoods
05-28-2007, 07:58 AM
You need a Mac Pro for the work you do. Anything else is just a stop-gap measure.

I appreciate your joining in, TZ. And I fear you are right.

But, of course, every system more than two years old is a stop-gap measure for the newest one, in some sense. We're always trying to make the most of what we have until the dam breaks: punctuated equilibrium is what some evolutionary biologists (and geologists) call it.

So, to answer the unwritten question, "how bad is it?":

Working on the thousands of small files shot digitally at weddings and other events, I have no complaints about my system's speed. Sure, I'd like to optimize, which is why I post to this forum, but I'm nowhere near needing to switch systems for that work.

Working on the 1.5GB-2GB files from scans that I prepare for exhibition-quality prints, (that's the size AFTER 7-15 layers), I do get impatient with the 10-minute save times, and the 5-minute open times, and so on. But I am hoping this will improve after I switch from 4-year-old drives (one of which, the drive that holds the startup volume, is only 80GB small) to new 500 GB Hitachi Deskstars and maxing out the RAM (from 2 GB to an unfortunately small 4GB) and getting help from you all on how to partition the drives most effectively. I thought $800 would by me at least another year of non-frustratingly efficient work on this crate. at which point $5000 would buy me a lot more than it does now. No?

Thank you sincerely for your input,

Mark

unclemac
05-28-2007, 11:20 AM
And the longer you wait to upgrade the more bang for the same buck you get, as there is always a newer faster machine right around the corner. Have to weigh that against the work you have to do now, and how much your time is worth....

TZ
05-28-2007, 04:02 PM
The Mac Pro is/was a major upgrade. And there will be improvements, but the basics are there.

Disk drives only do so much. And while you can use single drives on PCI because 75MB/sec writes is all that it will get, fine for backups and a lot of uses,

new drives only do a little on your system, too. You can't use 10K Raptor 150GB or new models which would help without PCI card. But might help and make sense. AND, you could justify them as an investment.

But SATA was new, in its infancy. And Apple made only the slowest progress for 3 yrs with that line.

The Mac Pro has a lot going for it, more than any G5 had. And upgradable. There really has not been something new and better right around the corner for years. Oh, a little speed bump, but nothing major.

A couple 500GB drives in RAID would help. $800 is 1/3 the price of a new Mac Pro.

G5 1.8DP vs Intel Xeon 2.66. 3.6 vs 10.4 GHz? That should and can yield 300%. Your work is limited by cpu, RAM as well. Surprised you didn't add more RAM earlier.

rwm
05-28-2007, 04:23 PM
............. But I am hoping this will improve after I switch from 4-year-old drives (one of which, the drive that holds the startup volume, is only 80GB small) to new 500 GB Hitachi Deskstars and maxing out the RAM..........A year and half ago I purchased a 74GB Raptor. :D WheW:D For a new boot drive on my QS.

I did not really have the need. Lots of surfing / researching small multiple projects. But the Raptor was obviously quicker and faster no matter if it was file transferring data or surfing it went faster. ;) Having it here hands on sold me. Work with boots on getting a Raptor in the mix.;)

markwoods
05-28-2007, 07:11 PM
[QUOTE]You can't use 10K Raptor 150GB or new models which would help without PCI card. But might help and make sense. AND, you could justify them as an investment.

I'm going to get a PCI card anyway, to solve the problem of the WD4000KD drive that turned out to be incompatible with my G5's internal bays. And possibly to host another, faster, SATA drive, since i need more storage space and work space anyway. I had considered buying a raptor, but RickS told me on the phone that he thought the Raptors wouldn't do much more for my system, given its limitations, than the Hitachi Deskstars, in terms of bang for the buck. also, I thought that if i were to put the money into a raptor, it would pay off most if used as my boot drive. But since the Raptors have the same incompatiblilty with my G5's internal bays as the WD4000KD does, i would have to connect the raptor via PCI card, and i've been told (not by RickS) that it was bad idea, less stable, to have one's regular boot drive be connected via PCI. No?


The Mac Pro has a lot going for it, more than any G5 had. And upgradable. There really has not been something new and better right around the corner for years. Oh, a little speed bump, but nothing major.

this does put things into perspective for me, and i appreciate it. Perhaps it means that I'll buy a discounted 2007 Mac Pro one year from now after squeezing more work from my G5!


A couple 500GB drives in RAID would help.

but RickS told me on the phone that RAID would barely help my system at all, at least not in a cost-effective way, since the bus speed of my particular G5 model is so limited.


$800 is 1/3 the price of a new Mac Pro.

except that only $160 of that (for the RAM) is stuff I can't take with me to the Mac Pro later: the rest is SATA drives I would take along. and I'd HAVE to buy extra drives and add them to a sttripped-down, barebones, $2499 Mac Pro anyway. And I'd have to buy 4GB of RAM for that Mac Pro, since $2499 only buys me 1GB of RAM. So this comparison seems more relevant:

$160 is 1/17th the price of a barebones Mac Pro.

I'm not expecting a speed surge from buying these drives for my G5; I'm just trying to learn how to use them most efficiently according to the WD/SD method.

I very much appreciate your perspective, TZ, and I thank you for writing!

Best,

Mark

markwoods
05-28-2007, 07:24 PM
...and thank you, too, boots, and rwm, and unclemac. I'm grateful for all of your help, too, past and (I hope) future.

I still don't understand some basics about what is meant by the WD/SD method in a non-RAID scenario, as I expressed above.

Hopefully,

Mark

Boots
05-28-2007, 09:33 PM
TZ put it most succinctly in that on the scale of what you're working with file size-wise, the hardware upgrades you have initiated and are contemplating for that pci-hobbled RAM-limited G5 will gain you only limited improvement.

With 2GB or even 4GB installed RAM, you're still going to be generating numerous swapfiles on the system startup disk when you work those relatively giant image files. That slows things down considerably even if you could add a fast RAID0 as a scratch disk..

The great thing about a G5 Quad or a Mac Pro is you can install up to 16GB RAM. That can help CS2 or CS3 hugely because you can let the OS cache scratch disk data in RAM for much faster performance- especially with those big image files.

Add a really fast independant RAID0 scratch disk and a really fast startup disk like the 150G Raptor to the mix and you've got an extremely powerful system capable of handling those big files with ease. It's just a quantam leap and though you might gain a few seconds here and there with the WDSD method or installing 4GB RAM or adding faster drives, unfortunately, your current rig will be capable of only incremental gains...

Having said that, to use the Work Disk/Scratch Disk Method, put a copy of the image file to be worked on onto the same volume as the first scratch disk which should be the first (or only) volume of the second hard drive. Open it and save changes to it from there. When finished working on the file, copy it to wherever you store your image files usually. Then erase scratch disk.

markwoods
05-28-2007, 09:53 PM
Having said that, to use the Work Disk/Scratch Disk Method, put a copy of the image file to be worked on onto the same volume as the first scratch disk which should be the first (or only) volume of the second hard drive. Open it and save changes to it from there. When finished working on the file, copy it to wherever you store your image files usually. Then erase scratch disk.

Thanks! But, just to make sure i understand, i'd like to reconcile this with your previous post in this thread:


You could use a second volume on the scratch drive as image file "cold storage", then use WDSD method by copying the image to be worked on to the scratch volume. You might see a slight improvement but this is usually evident only with a really fast RAID0 as scratch disk. And it can be a pain to manage the multiple versions of the resulting work.

If you throw a third drive into the mix- just as fast as the other two and on it's own channel- and use it just to store image files- then open and save the image files from there. You should see a bit of a speed boost.

It sounded to me before like you were saying this third drive option would be faster ("a bit" versus "so slight as to be not evident without RAID") than the two-drive method of copy-to-scratch-then-work-then-save-back-to-cold-storage-and-erase-scratch.

also, on the topic of this third drive option, when you write "use it just to store the image files," do you mean I should store only a few currently-being-worked-on files at a time there, or that i can load the drive to (for example) 50% full or 80% full (so long as i leave 100 GB free) with images that i then open and work on and save, and get the same speed as keeping it neary empty?

Also, thank you for the G5 Quad and Mac Pro advice, too. A Mac Pro is in my future, but probably not this year!

Best,

Mark

TZ
05-29-2007, 06:51 AM
From ANANDTECH (http://www.anandtech.com/guides/showdoc.aspx?i=2998&p=7) today's High-End Buyers' Guide: May 2007:
As mentioned in the introduction, the time to look at a new high-end system may not be perfect right now, with the P35 chipset introduction bringing DDR3 memory and a 1333MHz FSB into the mix.

Intel's X38 chipset is also right around the corner, just in time for Intel's Penryn processors.

Pending hardware introductions, however, will be looming every time one of these Guides are released.

As has been said repeatedly in the past, the "correct" time to update your system is when you have made the decision that your computer no longer meets your needs.

If that time is now, don't forget to visit the AnandTech Forums to ask any questions which you may have before making your purchase!

Visit our own Mac Pro FAQ (http://www.macgurus.com/forums/showthread.php?t=22544).

Boots
05-29-2007, 08:34 AM
It sounded to me before like you were saying this third drive option would be faster ("a bit" versus "so slight as to be not evident without RAID") than the two-drive method of copy-to-scratch-then-work-then-save-back-to-cold-storage-and-erase-scratch.


Mark- just email me with your phone number at boots911 at mac dot com. We can figure this out easier on the phone.

TZ
05-29-2007, 03:44 PM
Disk drives are something people do to optimize their workflow after they have the system that meets their needs, and usually can be optimized and configured.

But there is also a hierarchy of needs: CPU and memory, bandwidth to use.

Some reviews will offer an "ideal" or high-end workstation (or gaming) system, like ANANDTECH today which focused on gaming. Then your mid-range system.

On my G4, 3 * Raptors would use and saturate all the PCI bandwidth available. And limited to 2GB RAM. Hitting the "brick wall" early on.

From a practical standpoint, G5 1.8DP probably makes sense to RAID pair of 500GB Hitachi drives as boot drive + scratch (minimal OS and applications). And use FW800 drive for data, and even use one PCI SATA controller with another drive - because those are things you can do and use, will be useful for now and even if/when you get a new system (except PCI card which will change).

But having 6-8GB RAM and more cpu horsepower are going to help and have a payoff in what you can do and time it takes. Some of the performance benchmarks show Apple pro app results as well as CS3.

Xbench uses a G5 2.0DP as baseline 100 points. My Mac Pro is 2GHz (low end model) and rates about 150. Average 2.66 Mac Pros come in closer to 200 pts.

Boots
05-29-2007, 07:52 PM
I spoke with Mark today at length.
Sounds like he'll probably install more RAM and 2 or 3 large, fast drives for better overall I/O and file management without breaking the bank- and in future anticipation of a Mac Pro. Truth is, for most of what he presently does file-size wise, his current platform is adequate- though 4GB RAM installed and faster, larger internal drives will definitely help. The percentage of his workflow involving 1-2GB image files is fairly minimal.

We also touched on drive-partitioning issues and details of the WDSD method and other ways he can improve performance in his existing setup.

markwoods
05-30-2007, 02:08 AM
Thanks again to all of you, including TZ for your helpful post today, and most especially to boots for offering to talk on the phone, and then doing so, at length, and with great nuance and generosity (generosity of information and, more importantly, of spirit). I felt better about the world today after talking with boots: people really can help people for the good of helping people.

Based on my financial priorities, and my learnings from you all, and especially from talking with boots today, I'm leaning toward a three-fast-500GB-drive non-RAID arrangement for the last year of my old G5's life, not for the storage-space advantages, but rather for convenience and for the three independent buses with three fastest outside volumes on each:

Drive A (Hitachi 500GB Deskstar in internal bay)
vol 1: 120 GB (OS/apps/User folder for non-image files, never to grow past 50% of 120GB)
vol 2: remainder GB (drag-and-drop backup of digital images straight from CF cards, and other backup)

Drive B (Hitachi 500GB Deskstar in internal bay)
vol 1: 70 GB (scratch, kept always completely empty of non-scratchfile files, and periodically reformatted)
vol 2: remainder GB (daily Carbon Copy Cloner backup if Drive C)

Drive C (in external 2-drive eSATA enclosure: Hitachi 500GB Deskstar? OR Seagate Barracuda 7200.10 500GB SATAII? PLEASE SEE QUESTION BELOW!)
Just one volume. Currently-being-worked-on and soon-to-be-worked-on files from both DSLR and scans of 6x7-inch film, altogether never to exceed 66% of drive's capacity (I'll move files from this drive to Drive D when I'm done working on them). This means I would not copy image files onto scratch (drive B vol.1) for working on them: they'd stay on C, get opened on C, get worked on on C, get saved to C, finished on C, then eventually get moved to D for limbo storage.

Drive D (in external 2-drive eSATA enclosure: Western Digital WD4000KD 400GB drive [i needed to get this out of internal bay B, since it's incompatible with m G5, causing hang-ups and the need for rebooting]
Just one volume. Organized, pared-down storage of files recently processed on drive C but then moved and deleted from Drive C (once the job is completed). These files might need to be called back into service, but will eventually be moved to an external USB drive for cold archive storage. So Drive D is like a holding pattern, a limbo, if you will, for image files not current, but not ready to be archived. mostly it's a spillover, so i can be sure to always give Drive C whatever breathing room it needs.

Backups and archives (and backups of archives) will be on external FW drives when bootability is necessary, and on external USB drives, when bootability is not.

Please comment and question and object to this arrangement if you feel like it! I'd appreciate the input!

Also:

NEXT BIG QUESTION:

Should Drive C be a Hitachi 500GB Deskstar? or a Seagate Barracuda 7200.10 500GB SATAII? I ask because of this, from a MacGurus products page:
"Seagate Barracuda 7200.10 drives[:]....Because of great synchronous transfer rates, these are superior data storage drives. The 7200.10 trails the Hitachi performance in random access making the Hitachi a better choice for OS, applications and scratch drive use."

One competing priority might be the issue of whether i will, after some day buying a Mac Pro, wish i could, for some reason (involving RAID?), use three identical Hitachi drives together instead of two Hitachis and one Seagate. is this a concern? Otherwise, the seagate seems to make more sense for a live data storage and data working drive. No?

Thanks again,

Mark

TZ
05-30-2007, 05:39 AM
Eighteen months is a lifetime in storage. Choose by what meets your needs for today and the foreseeable year. Cross the Mac Pro when you get there. 500GB drives are $117 to $200 today.

I think you should see if RAID of your 2 internal 500GB Hitachi would help, stripping the first 100GB.

markwoods
05-30-2007, 02:47 PM
Thanks, TZ.

so am i right in reading that MacGurus description of the seagate as indicating that the seagate is better for my storage/workdisk needs?

Best,

Mark

TZ
05-30-2007, 03:05 PM
I didn't endorse using Seagate. I tried to steer you to using Hitachi drives in RAID.

And I find that having extra drives means I can use a couple in RAID and a couple more for backups (two minimum) along with boot drive or something else.

I am more hands-on and visual. I like to draw schematics of the drives and how to arrange things.

The simpler the better. You can get fancy or change things later as you experiment and after running some PShop Test benchmarks.

markwoods
05-31-2007, 07:47 AM
thanks again.

other opinions?

I think i will get two Hitachis, for OS/Apps and Scratch, respectively. But I'd like to get a third drive as a workdisk/storage disk. if i decide against doing the RAID thing in this setup, should i still choose Hitachi over Seagate for the workdisk/storage disk?

Is this Macgurus Store info not pointing me toward seagate for the workdisk/storage disk?:

"Seagate Barracuda 7200.10 drives[:]....Because of great synchronous transfer rates, these are superior data storage drives. The 7200.10 trails the Hitachi performance in random access making the Hitachi a better choice for OS, applications and scratch drive use."

TZ
05-31-2007, 07:55 AM
The forums and store are somewhat separate.

Apple 'fixed' a problem that plagued the Mac Pro and Seagate drives with revised logicboard. Up until then, it was questionable and users needed to rely on PCI adapters.

Using all one drive has option of using in RAID. Different drives have their own strong points and best environment.

MacGurus Store has sold Seagate drives forever, back when the SCSI Cheetah was king.

Don't forget to take into account backups for OS, data, emergency boot drive.

kaye
05-31-2007, 08:04 AM
Other opinion but the same as TZ sez, I vote for the Hitachis, simplicity and more options later on. And definitely a backup boot drive. k

markwoods
05-31-2007, 01:09 PM
Thank you, TZ and kaye, both of you, for offering me some more specific reasons for choosing this hitachi over this seagate based more specifically on the arrangement I'm envisioning.

to home in on the reasons now: when you speak of simplicity and more options later on, is only because RAID requires using drives of identical model together, and i might want to use RAID later on? or is there another kind of simplicity that i'm not considering?

ricks
05-31-2007, 01:43 PM
I like Seagate drives. All MacGurus servers use em.

Seagate support puts Hitachi to shame. If you ever need a replacement drive Seagate will make you feel like a king. Hitachi makes you jump through hoops to get a replacement drive. Very impersonal support.

However..

Both my workstations are crammed full of Hitachi drives for their better overall performance. Not to denigrate the Seagate drives, they just don't do random access as well. In all other ways the Seagate drives are awesome. By choice I'll take the extra performance of the Hitachi drives for a photoshop workstation.

markwoods
06-01-2007, 02:49 AM
thank you, ricks, for this balanced pros-and-cons evaluation!

you guys are all great. i really appreciate all the advice.