PDA

View Full Version : ATTO Quickbench Useless?



ricks
10-06-2001, 02:07 PM
Supposition; ATTO SCSI bench in useless for testing performance in high speed arrays.

Anytime a drive or array of drives reaches the magical number of 8mb of total buffer ram on board the drives, ATTO bench will then only read bus to drive controller ram (buffer) speed. The drives can be inert during write tests and you'll still get max bus performance results up to the buffer size.

I am a beginner in the RAID field, but I have read everything I could and done a ton of testing,building & experimenting in the last couple of months. As my raids go up the ladder in grouped buffer size I watch the results shift from buffer speed to actual 'read/write to disc speed' with complete predictability. I see no way to test a volumn with a combined buffer of 8mb or more using ATTO bench unless the buffer can be turned off.

When I see a SCSI raid built with 4 cheetahs and a dual channel card with results way up to the 200 mb/s range that's pretty hot stuff. But all the results indicate is how fast the buffers are.

In my just posted ATA Hardware raid topic, I used drives with 2mb buffers, the results show a complete shift at 4 mb from the buffer related peak rate to the drive related sustained rate (from 120mb down to 70mb). If I replaced the drives with 4mb buffered drives, all other factors being the same, I would get virtually identical results sustained vs. peak using the 8mb Quickbench (it would stay in the 120mb/s range). Yet my actual performance would still be the same sustained rate gotten from the 4mb test.

I think were kidding ourselves using the ATTO ATTO bench as a way to show how great our arrays are performing.

Speed IS its own reward.
Rick



[This message has been edited by ricks (edited 08 October 2001).]

dragon_x
10-06-2001, 06:37 PM
I know I've pointed this out a couple times. I'm sure others have realized this too. It does suck - it seems the benchmarks are stuck in the late 90's http://www.macgurus.com/ubb/eek.gif

QuickBench - from IntechUSA is slightly better - at 10MB, but still not close to where it should be. I would think there should be options for 16MB, 32MB, 64MB and probably even 128MB.

With new drives having 8MB cache an 8 drive RAID 0 would give about 64MB of cache... nice but most people with those massive setups are transfering gigabytes and more of data - maybe 256MB should be the limit.

It does seems like it would be fairly easy to have a nearly limitless test file size - just plug in a number and the app generates the test file and then runs it. The app may need more memory or something... but today most systems seem to have 384MB Minimum.

You can adjust the caches. I think SoftRAID has some settings for this - but it is not very clear and I do not tend to mess with it. ATTO and FWB probably have some options to change up the 'page mode' settings - which I think is what affects the cache - or maybe it is the cache? I have never heard of being able to turn off the disk cache but you can change up the segments... I guess turning off the cache is not a normal feature http://macgurus.com/infopop/emoticons/icon_wink.gif

I am not sure how much we are kidding ourselves. Just sitting on the peripherial of the computer/IT industry is enough to make one realize that things are MOSTLY HYPE - overpromised, underdeliver, overcomplicate and crash prone. This goes for Mac and Windows/x86 - but mostly for x86 as I work with that KRAP 5 days a week and I have YET to be impressed. I think we use it (well, I guess I dont http://macgurus.com/infopop/emoticons/icon_wink.gif ) because it is around and its really the only thing we have.

Like many benchmarks - it is important to take the readings with a grain of salt and also realize how the marks are arrived at. MacBench5 is acient, but can still give important info on some systems and configurations.



------------------
One Nation, under God; with liberty and justice for all.

kaye
10-06-2001, 09:30 PM
MB5 allows you to setup custom file sizes for sustained read and sustained write (as well as random read/write). I have setup a file size as large as 16MB but you have to convert to KB and at some point I remember having to up the memory for MB5 to get that 16MB. I never tried higher but I'm sure it is available. k

ricks
10-06-2001, 10:08 PM
Thanks Kaye, I'll give 'er a try.

Rick

Taffy.C
10-06-2001, 10:58 PM
Hidy ho. Where can I get Mac Bench?? I've looked all over the web and it seems to be discontinued coz it had issues with multiprocessing.

Question - if I have a 4 drive array with each drive having an 8M buffer, what test file size do I need to test the limitation Rick is talking about below?? I assume greater than 64MB. I'd actually like to test with much larger test file coz I will be reading and writing files of several GB. Can I go larger than 64MB in Macbench?

.... taffy.

kaye
10-07-2001, 02:54 AM
MB5 is on the Gurus ftp site and I see that, in addition to the installer, someone has created an image file to burn a CD which would be necessary for the CD-ROM test, Video tests, and the Graphics and Publishing Graphics tests. You only need the installer to run the Disk and Publishing Disk tests, Processor and FPU tests.

QuickBench is a product of Intech. The ATTO test via the Menu Bar's Benchmark Volume is called SCSI Benchmark.

I don't look at these test utilities and the fact that the drives have an 8MB buffer in the same light that Rick does. I see nothing wrong with his point of view, it is valid. My take on this is that we pay the extra money for this hi-speed RAM buffer, and yes it does help, but we are going to be using that buffer every day, so what is the point of finding a way of disabling what we paid for and what we intend to use every day? Why test by disabling such a feature? What would be the point? Further, where does this buffer get the data that fills it up? From the drive. The drive fills the buffer until the machine can use it and then continues to fill this buffer as it is being emptied. This allows for smoother operation of the drive during this process and smoother read/write thruput.

Either the QuickBench test or the ATTO test is a compilation of many file sizes beginning with small file sizes and progressing to 8MB with the ATTO test and 10MB with QuickBench v1.5. But during these tests you are running through a host of file sizes. You are not just stuffing thru an 8 or 10MB file size with either test. Those max sizes are just the last file size under test. Almost any of the tests you do, including the drive tests, is really a compilation also of many other factors, such as your SCSI card and your processor. I can stick one of these drives in another box with a slower G3/G4, and the results will be slower. Or use a slower SCSI bus or IDE bus and watch the performance drop. None of these drive tests is a pure raw drive test. Other factors affect the performance, not just buffer size.

Your drive and the computer that will run it, how fast will the drive perform for what you need or want. For me, I guess want is more appropriate because I don't need this performance. Until a couple of months ago, my main machine was a NuBus Power Mac with a G3 card and a bunch of fast drives, All the PCI machines were test beds. Anyway, that is how I see testing. k

TZ
10-07-2001, 08:13 AM
Testing for real results is complex. It is over-simplified and merely a 'standard' we all use to show - relative to other drives and setups - and isn't an exact or close to a performance benchmark. When I want that, I go to StorageReview.com and look at reports, benchmarks, and compare different drives.

MacBench may be a better choice. Tweaking settings for video and knowing that you have your setup optimized though is very important and can have an impact on time and quality. Having a tool (like softraid) that can optimize a drive for video helps. There is always the chance that tweaking can eek out better performance.

Should be possible to hack ExpressPro Tools to use say 16MB test size but then, using Photoshop and running various operations is one way to test the overall system, and extract whether one RAID setup over another makes a measureable difference.

Maybe with 10.1 there are or will be more options and test suite for database, file server, web server, or video and the like?

Gregory

ricks
10-07-2001, 06:06 PM
Thanks for all the feedback. My thoughts on this came from building an ATA raid this week and realizing that for all intents and purposes, if I had used 2 drives with 4mb buffers instead of the 2mb buffers all the tests we use to benchmark this raid would have indicated a 115 to 125 mb/s sustained write and read.

That would have been both thrilling for anyone wanting to build a inexpensive array, and misleading, because the sustained numbers of around 70-75 are more realistic. The difference was solely in my choice of drive/buffer size.

That leads me to wonder what actual large file performance is on any array, including state of the art SCSI w 8mb buffers.

We are comparing apples and oranges when we compare an array with less than 8mb buffer with an array with more using todays benchtests.


My brother, who helps run one of Apples web portals, says he can fairly easily write us a unix script that will measure a file to disk and from disc. I am going to pursue this and see what we get. It'll have to run in OS10 but that's not a big deal probably.

I would really like to see how drives perform with 1/2 gig file sizes. Maybe file transfers can be read to a ram-disk as a method of testing HD performance? The slowest part of the data path should be the HD in this case, what do you think?

Anyone doing videos or production photo work is going to be interested in huge file transfers, not piddly little 8meg transfers.

For anything I am doing with my computing, other than the search(lust) for performance, my 70ish thru-put is blazingly fast. Everyones help has been a wonderous thing for me, eye opening to say the least. I thought I was looking at a simple, logical problem "drive performance", I have found through just this thread that just defining the term 'drive performance' is complicated at best. Measuring it, subject to many, many variables and most of all viewpoints of the testers

Thanks
Rick.

SF Crunch
10-08-2001, 01:04 AM
Yeah I don't believe QuickBench is Atto's utility, as Kaye mentioned...is it?

SF Crunch
10-08-2001, 01:12 AM
Q: When will SoftRAID support FireWire drives?
SoftRAID does not currently support generic FireWire drives.

Does ATTO, or any others?

ricks
10-08-2001, 01:35 AM
Yes, I was mistakenly calling ATTOs benchtest 'Quickbench'.
My apologies for the confusion. I was using and talking about ATTOs product. However I am not complaining about the quality of any of the products mentioned. I am only trying to figure out how to use the data we get from them.

Rick

SF Crunch
10-08-2001, 01:50 AM
I see...not saying that your complaining...just don't wanna be confused...;-)

TZ
10-08-2001, 08:51 AM
www.storagereview.com (http://www.storagereview.com) looks at WD ATA drive that now offers an 8MB cache and looks at impact on performance. Seems to offer real benefit and be equivelant to what a 10k ATA drive might deliver. Also looks at Barracuda IV and lack of noticeable effect of enabling noise silencing (which in North America is enabled by default - you may want to configure to disable but need access to Windows to do so).

Cache matters. But it also increases cost.

Gregory

kaye
10-08-2001, 09:41 AM
Duh...I must be losing my mind. There is a very simple way to disable both read and write onboard cache for either SCSI or ATA drives. HDT's FWB Configure, which accesses the Mode Page Parameters, allows you to make the changes. Have not tried it yet but will do so today after screwing up my courage and hopefully nothing else. k

ricks
10-08-2001, 01:32 PM
kaye,
You are one brave dude. I agree that doing a Page Mode mod (turning off cache) would give us more information to go on.(is 'cache' and 'buffer' interchangable here?)

BUT, you have modified my thought processes considerably with your postings on this thread. The cache is an integral part of disk drive performance, testing with it off would only give us information that we could extrapolate with. Leaving us with having to guess real performance again. My ignorance was showing when I simplified things too much by saying "test with buffer off". Your comments about cache are true.

I think the correct test procedure is to get Intech or ATTO to rewrite and up the test size, or write our own.

I know, I'm a pain in the butt, my wife's been saying it for years.

Rick



[This message has been edited by ricks (edited 08 October 2001).]

kaye
10-08-2001, 07:25 PM
Rick,

Yes onboard cache and buffer are the same. Just have to differentiate the onboard stuff with System Disk Cache which is of course RAM. Getting Intech to raise the file size above 10MB could be a problem. I asked/begged for greater than 1MB back when that was their max file size. It took a long while (and the weight of MacGurus) before they decided to offer the 10MB size with QuickBench v1.5.

I did disable both read and write cache for both drives on my 2x G2 X15 (8MB buffer each) striped RAID, both drives connected to one channel on a UL3D, OS9.2.1. Before doing so, I retested the RAID in the ATTO EPT test, 8MB, Sample Size 2, no System Disk Cache.

Read/Write cache enabled:
PR-127.77---PW-119.31
SR-113.02---SW-116.03

Interesting to note that the graphs only showed those ridiculous peaks at the beginning up to 1MB then dropped and started rising. So those dual buffers are full by 1MB of the 8MB test. I remember reading the ATTO pdf files that the test begins at a file size of 8KB and progresses to 8MB. It does not say what all the sizes are. Some are of course selectable, but not all. One other note is that the Sample Size 2 means that each file size is run twice. You could select Sample Size 4 or 8 which would rapidly fill the buffers.

I used FWB Configure to change the Mode Page Parameters:

Read/Write cache disabled:
PR-115.86---PW-108.73
SR-113.15---SW-105.88

Interesting to note that SR is actually a tad higher, SW of course is less, and some say that SW is the true measure of drive performance. The graphs are rougher, more jagged, because the buffers are not there to dampen the thruput. The graph drop is still at 1MB then gradual increase.

The ATTO docs say to try Sample Size 8 for larger file size transfer performance testing. So I tried that at 8MB:

Read/Write cache disabled:
PR-115.90---PW-108.53
SR-113.57---SW-106.10

Last, I did check that FWB Configure would disable read/write buffers with ATA drives. It will tho I did not do it nor test it. If you do, remember that you have to do it individually for all drives on a RAID. Then, when you are done, remember to enable read/write buffers on all drives. k

ricks
10-08-2001, 07:48 PM
Thanks Kaye,
Lots less difference than I expected. Great test. I will try this with the ATA drives and see what happens.

I will keep after my brother to get a test program written for a large file transfer in unix for OSX. One problem will be the type of file transfers done, sounds like it's more complex than just throw 250mb at the disk and then read it back.

Your post, as always, drives me to want to know more. My deepest appreciation. The only negative, makes me want to own some X15s.(come spring I will!)

Rick

kaye
10-09-2001, 10:28 AM
Rick,

This morning I ran the same tests on my slave ATA/100 drive connected to the internal ATA/66 bus, G4-800DP. Could not tell any difference with read/write cache enabled or disabled except that disabled, both graphs are far more jagged. This is with the ATTO test, 8MB, Sample Size 2, no System Disk Cache. Except for the graphs you wouldn't even know whether the cache was enabled or disabled, IBM 60GB 60GXP, 7200rpm, 2MB cache. k

Santilli
11-22-2001, 06:07 AM
Maybe this utility was designed for scsi, only?
gs