Results 1 to 10 of 10

Thread: Windows 10 - Burly 5Bays - Hard Drive problems, Best Practices questions

  1. #1
    Join Date
    Dec 2010
    Posts
    10

    Default Windows 10 - Burly 5Bays - Hard Drive problems, Best Practices questions

    Hello,

    I am a photographer, I have been following a backup system that I thought was great and smart, but recent events have me wondering. I am posting here because I am using the 5bay Burly Storage boxes.

    I have a windows 10 pc with 5 internal drives, c:, second drive and 3 studio drives. We have a 5 drive array box with one old archives drive and then 4 back up drives for the old archives drive and for the 3 studio drives that are in the computer.

    We have a second set of the backup drives, every night our 4 backup drives are mirrored from our 4 main drives, the next day I swap the backup with set 2 and take the set 1 home. Every day I rotate drives. . .

    So the issue that has me worried, a number of months ago I lost the 2012 folder with all those images off of the old archives drive. The loss had propagated to the backup drives. I did have an old drive around that had 2012 on it so no lost images but worrisome. That drive was not flagged by windows 10 as having errors.

    Last weekend I had a few of the backup drives report errors from windows and lost data. Even had a backup drive with a lot more free space with no errors reported. The difference in this was a couple of large 260Gb disk image backup files of the windows boot disk that I had on our studio files drive that were no longer on the backup. Checked the drive and the drive said no errors, ran disk check and then errors were found, fixed and files still missing. I could have some of this paragraph backwards in terms of when errors were reported and such but it is mostly right - I took some notes when I realized it could be a reoccurring issue.

    So the quandary, I love mirroring because it is so simple and quick to swap in a backup when a drive goes bad. But if a drive can lose folders . ? Any way to easily catch these kinds of problems before we are screwed?

    I use Hard Disk Sentinel to check the health and SMART history of the my drives. Is there software that can catch hard drive file system problems when it is not due to a disk error? Does hardware raid have better drive monitoring or file allocation table management, so it makes sense to include raid in the mix? (I know, still need to back it up).

    I noticed when reformatting a drive that windows 10 now has a new drive file system that automatically repairs errors on the fly, Refs, supposed to be the next generation of drive format so maybe smart to switch all drives to that? If I move to raid, ntfs or refs?

    Of course I need to replace any questionable drives . . but down the road I could be in this pickle again and not as lucky. . . so my real focus in this post is long term best practices!

    I really do appreciate all help.

    Thanks!
    Roger

  2. #2
    Join Date
    Aug 2001
    Location
    Grangeville, ID USA
    Posts
    8,953

    Default

    I'll do some thinking on this.... Windows 10 is not really my area of operations, but doing some research would do me good. I have Windows 10 on a couple test systems here, I just never spent more than hardware verification time on them. Between Apple having a new OS every year and M$ doing it every couple years, getting hard to stay up. I'll see what I can dig up.

    There are some other folks here that may have better ideas - and sooner. That would be most welcome.

    Rick
    molṑn labe'
    "I am a mortal enemy to arbitrary government and unlimited power. I am naturally very jealous for the rights and liberties of my country, and the least encroachment of those invaluable privileges is apt to make my blood boil."
--Ben Franklin

  3. #3
    Join Date
    Dec 2010
    Posts
    10

    Default

    Quote Originally Posted by ricks View Post
    I'll do some thinking on this.... Windows 10 is not really my area of operations, but doing some research would do me good. I have Windows 10 on a couple test systems here, I just never spent more than hardware verification time on them. Between Apple having a new OS every year and M$ doing it every couple years, getting hard to stay up. I'll see what I can dig up.

    There are some other folks here that may have better ideas - and sooner. That would be most welcome.

    Rick
    Thanks Rick,

    Last night had a 4TB backup drive, windows reported problems and on reboot and repaired overnight. This morning the drive works fine but now has 2.47 TB free space instead of the 110 GB of free space it should have .... currently running some tests on the drive using Hard Disk Sentinel.

    A little more background on the computer ...

    It was built by Maingear in 2011, a gaming computer company that also builds workstations. It has the Intel DP67BG motherboard on it. As they were building the computer Intel had a recall on the boards (Sata ports 2 3 4 and 5 worked but were found to not be reliable) and then released the use of the boards for vendors who were putting the boards in computers that only used the Sata 0 and Sata 1 ports only. Maingear called me and gave me the choice of waiting or having the computer shipped with their fix that they said would be fine. Their fix was to include a 4 port SATA card in a PCIe slot. The card is a SY-PEX 40013 Raid card with a J-micron chip set.

    In the computer I am using the Newer eSATA 6G-4e PCIe 2.3 Controller card that I got from you guys. Is it possible there could be a conflict between the raid cards, where they work but occasionally cause a problem?

    The Maingear computer is overclocked by Maingear at what they say is a safe level. They have extra cooling and fans in a very clean box and I don't have any drives or anything that I can see reporting overheating - windows hasn't had operating system problems that I would expect if there was a problem. But, I was reading through the help pages on the Hard Disk Sentinel web site and found a comment that hard disk problems can be caused by over clocking. I am wondering what kind of chance that might the case in this scenario?

    This computer is our main retouching computer and also hosts all of our file sharing of all our main drives - so it can run lightroom with the images local to the computer. It is fast and solid and doesn't appear to slow down at all when it is being used and other computers are accessing files on it so I am hoping not to mess with the bios to remove the overclocking, but if it could be causing a problem I figure I got to put the idea out there

    Thanks

  4. #4
    Join Date
    Aug 2001
    Location
    Grangeville, ID USA
    Posts
    8,953

    Default

    That kind of tells me what is going on. You need to leave 20% of your drive as free space. You are leaving 2% free space. What is happening is the directory is getting overwritten as the drive runs out of space. Do not under any circumstances run a 4TB drive with much less than 800GB free space.

    When the directory gets overwritten, the data just disappears, like it was never there.


    Rick
    molṑn labe'
    "I am a mortal enemy to arbitrary government and unlimited power. I am naturally very jealous for the rights and liberties of my country, and the least encroachment of those invaluable privileges is apt to make my blood boil."
--Ben Franklin

  5. #5
    Join Date
    Dec 2010
    Posts
    10

    Default

    Quote Originally Posted by ricks View Post
    That kind of tells me what is going on. You need to leave 20% of your drive as free space. You are leaving 2% free space. What is happening is the directory is getting overwritten as the drive runs out of space. Do not under any circumstances run a 4TB drive with much less than 800GB free space.

    When the directory gets overwritten, the data just disappears, like it was never there.


    Rick
    Thanks Rick,

    Totally hear you and will have to figure out how to change our methodology but I am suspicious that there could be other factors, Our S drive that lost large disk image files a couple of weeks ago was nowhere near full.

    I have decided to change our directory structure for images ... right now we have an active sessions directory and then when we deliver the order we delete unneeded files and move to archive folders by year. Theoretically anything in the active session folder that we have missed fallowing up on will be caught. In real life when we get busy we get behind on archiving and then it gets out of hand and the drive starts getting full ... a guilt ridden viscous circle ... then we start archiving and our current archive drive starts getting full. I am going to change to an only by the year system and screw the guilt, if we miss deleting some files we could have deleted our time is more valuable! Gheeeez, this is starting to feel like a 'hello, my name is Roger and here at our studio we are addicted to archive guilt' or some such 12 step program....

    Since we have the resources with our 2 5bay Burly boxes, I am thinking I should just create Raid(s) so that we can have one big drive and not have to move things around so much - and if lose a drive we just keep on trucking while the hot spare is rebuilt. Currently our back up is to swap drives out of the Burly, but if that goes away because the box is being used for the Raid then it feels like the simplest backup is to use smaller usb passport drives. All of the data on the dives and then when an old year is taken off line a duplicate on another passport drive. If the passport drives are only plugged in to backup then their life cycle is much different than what we have now (currently our backup drives are always on when they our in, which is half the time).

    If we go Raid on the Burly enclosures we can remove the extra drives from inside the computer and no longer even need the Raid card that came with the computer. The Raid controller will have more management of the drives instead of a pass through roll (I know, making assumptions here) but guessing that it might help ...

    Anyway Raid with 2 5bay enclosres - mostly image files and lightroom catalogs with a few large disk image files. Raid 6 or 10 seems to be the smartest from what I read scouring the net. So Raid in each box and separate or combine boxes into a large raid ... best practices? I can figure out the drive sizes that I need to accommodate the hot spares and whatever Raid setup is used. Everything will be backed up on the passport drives but no one like downtime so looking for a good safe setup that will serve up the files nicely

    I am assuming to test after the Raid was set up, if we still had anything questionable happen we could move our Raid card, Burly Boxes and drives to our 2nd production computer and it would work there also for testing? Does the Raid set up go with the drives as long as it is the same controller?

    Thanks for letting me sort through this out load and thank you for the help!

    Thanks!

  6. #6
    Join Date
    Aug 2001
    Location
    Grangeville, ID USA
    Posts
    8,953

    Default

    If you build a RAID, the RAID is stored on the drive but unless a pure software RAID, you will have to use the same RAID card.
    molṑn labe'
    "I am a mortal enemy to arbitrary government and unlimited power. I am naturally very jealous for the rights and liberties of my country, and the least encroachment of those invaluable privileges is apt to make my blood boil."
--Ben Franklin

  7. #7
    Join Date
    Dec 2010
    Posts
    10

    Default

    Quote Originally Posted by ricks View Post
    If you build a RAID, the RAID is stored on the drive but unless a pure software RAID, you will have to use the same RAID card.
    Thanks!

    Any consensus on Raid 6 or 10 or other Raid setup that might be the smartest for image files and Lightroom catalogs? I can guess on my own but am wondering if there is a prevailing thought among the photographers here on this form ... I have read on different sites that different uses lend themselves to different setups.

    Thanks,
    Roger

  8. #8
    Join Date
    Aug 2001
    Location
    Grangeville, ID USA
    Posts
    8,953

    Default

    RAID5 and RAID6 don't really have anything to do with HOW they are used but with how the RAID is managed.

    A RAID5 is a striped RAID plus parity to protect from a drive failure. This is not a backup but instead is intended to increase uptime. This is usually the fastest of the 'redundant' type RAIDs. In my opinion, RAID5 is most often best implemented with a hardware RAID controller, not in software pr in firmware with an 'engine' built into a host card or motherboard. RAID5 also requires a very good mature management utility to handle any problems. As complex as RAID5 is, its failure mode is usually corruption of the RAID structure. There are not all that many high end RAID management utilities - Promise, ATTO, Areca, LSI are the most common high end.

    A RAID6 is a striped RAID plus two sets of parity data. Same as a RAID5 only protection from 2 drive failures. More complex than a RAID5, it also requires significant processing to create and or run off the parity data.

    RAID 10 is the most wasteful but simplest. It is two striped RAIDs that are mirrored. Like all mirrors, for every data drive there is also a mirror drive. Mirrors are NOT considered a backup as the two mirrors are siamese twins. Anything happens to one, instantly happens to the other. I consider this the least useful. It will also not have any kind of management tools as it is so simple. A corruption can happen, in which case there is nothing to repair it except a reformat.

    Simple striped RAID0. Fast, no protections. Simple. No need for hardware. Drive failure = data loss.


    In any case, any system must be backed up. With a good backup, any will work fine.
    molṑn labe'
    "I am a mortal enemy to arbitrary government and unlimited power. I am naturally very jealous for the rights and liberties of my country, and the least encroachment of those invaluable privileges is apt to make my blood boil."
--Ben Franklin

  9. #9
    Join Date
    Dec 2010
    Posts
    10

    Default

    Quote Originally Posted by ricks View Post
    RAID5 and RAID6 don't really have anything to do with HOW they are used but with how the RAID is managed.

    A RAID5 is a striped RAID plus parity to protect from a drive failure. This is not a backup but instead is intended to increase uptime. This is usually the fastest of the 'redundant' type RAIDs. In my opinion, RAID5 is most often best implemented with a hardware RAID controller, not in software pr in firmware with an 'engine' built into a host card or motherboard. RAID5 also requires a very good mature management utility to handle any problems. As complex as RAID5 is, its failure mode is usually corruption of the RAID structure. There are not all that many high end RAID management utilities - Promise, ATTO, Areca, LSI are the most common high end.

    A RAID6 is a striped RAID plus two sets of parity data. Same as a RAID5 only protection from 2 drive failures. More complex than a RAID5, it also requires significant processing to create and or run off the parity data.

    RAID 10 is the most wasteful but simplest. It is two striped RAIDs that are mirrored. Like all mirrors, for every data drive there is also a mirror drive. Mirrors are NOT considered a backup as the two mirrors are siamese twins. Anything happens to one, instantly happens to the other. I consider this the least useful. It will also not have any kind of management tools as it is so simple. A corruption can happen, in which case there is nothing to repair it except a reformat.

    Simple striped RAID0. Fast, no protections. Simple. No need for hardware. Drive failure = data loss.


    In any case, any system must be backed up. With a good backup, any will work fine.
    Thanks Rick!

  10. #10
    Join Date
    Feb 2001
    Location
    on the landline, Mr. Smith
    Posts
    7,766

    Default

    Sounds like you are on the right path.

    To add another perspective, you might think about replicating your archive to something else, so that if you lose the box, or the RAID controller, you still have quick access to your data without needing to rebuild the exact box and RAID controller. Full redundancy without any dependance. Moving away from a simply mirror can be intimidating, but at a certain point, the weaknesses start to out weigh the benefits....like not having versions of backups to catch a problem before it gets "mirrored" over to the backup, or more importantly, not being able to go back in time to recover from a point before the mirror.

    Lots of ways to do that. Some new and interesting things in the last few years.

    One way to consider is with a dedicated file system that has built in snap shots and more robust error checking that either a Win of Mac box has. One I am keeping an eye on (will be running some tests soon) is the newer file system on the more robust boxes by Synology. Something like this can do a RAID 5 (and others types too, including their hybrid RAID which has the great feature of being able to grow a volume by replacing His one at a time with bigger models), but most importantly their BTRFS file system should be more robust than any desktop to help protect priceless data. Their backup features and options are compelling.

    I also like that you can buy the bare box and install your own drives. With something like this a dedicated backup server, you can work on your high performance station and local Burly with confidence that you have a secondary, fully separate backup archive with the ability to see versions, which is a compelling choice for me.

    Not the only option. Other things could include a dedicated back up application that includes versioning. Lots of choice there too.
    "Imagine if every Thursday your shoes exploded if you tied them the usual way. This happens to us all the time with computers, and nobody thinks of complaining." -- Jef Raskin

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •