Page 1 of 2 1 2 LastLast
Results 1 to 20 of 25

Thread: HFS+ and OS X file-copy commands

  1. #1
    Join Date
    Jun 2002
    Location
    Campbell, CA, USA
    Posts
    732

    Default

    (I ran a search under OSX General for HFS and didn't find anything on this subject -- maybe it's here without the HFS term appearing.)

    I'm working on a utility script to copy home directories around on OS X installs and a friend pointed out that the base Unix file-copying commands don't see the resource (and maybe info) forks of files on HFS+ volumes.

    I ran a quick test with a source folder full of JPEGs with custom icons (which are stored in the resource fork in HFS+) and every time, even on the same file system, the custom icons were not there in the destination files.

    Commands checked: cpio, gnutar, rsync, ditto

    I was reminded of Gregory's comments about CCC problems and the point that CCC calls on ditto (used to call on ditto??) to do the actual copying. Hmmmm.....

    I'm starting the hunt now to try to find an OS X command -- something that can be invoked from Terminal or a shell script -- which copies resource (and info) forks along with the data fork.

    FYI, while losing custom icons and previews for JPEGs can cause one to grumble about having to recreate them, other files like SimpleText, Word, Excel docs, .smi images, and OS 9 applications, all get badly broken if their resource forks are ripped out of them.

    If you already know the command -- standard of versiontrackered -- to handle this task, I'd surely appreciate hearing of it from you! If I can find one, I'll post about it too.

    Jazzbo

  2. #2
    Join Date
    Jan 2001
    Location
    Mobius Strip
    Posts
    13,045

    Default

    I guess this means that 'cp' doesn't cut it, huh?

    I have a couple books and none of them 'touch' ditto so I'm not recommending "Unleashed" anymore.

    tar, dump, cpio, or even backup?

    ditto -arch m68k src dst
    ????????Thin all fat files to Motorola 68000 architecture during copy

    OPTIONS

    -arch archVal Thin multi-architecture binaries ("fat
    binaries") to the specified architec-
    ture. If multiple -arch options are
    specified then the resulting destina-
    tion file will be multi-architectural
    containing each of the specified
    architectures (if they are present in
    the source file). archVal should be
    specified as "m68k", "ppc", "i386",
    "hppa", or "sparc".

    it looks like ditto recognizes the resource fork from this option example.

    Then again, whenever I copy jpeg to a new drive I lose preview icons.

    Gregory

  3. #3
    Join Date
    Aug 2001
    Location
    Grangeville, ID USA
    Posts
    9,142

    Default

    ?See if I get this right, because we use HFS we have a data fork. Unix File System doesn't have that fork and Unix commands don't, at this point, seem to be able to deal with that data fork.

    ?If we had chose UFS for our storage method we would not have this same problem. Makes me wonder if that's why Apple has the Home Directory glued to the root Directory. Also makes me wonder if any of the current backup schemes are actually able to do a clean HFS backup.

    Rick

  4. #4
    Join Date
    Jun 2002
    Location
    Campbell, CA, USA
    Posts
    732

    Default

    The bit in ditto about archs is that one can construct "fat binaries", which is to say a compiled program (ls, rm, mach_kernel, ...) that has more than one CPU chip's machine code in it. Copy the command to a Sun SPARC, the ELF CPU chip's code is there; copy it to an Intel, there's the 386 code elsewhere in the same file to run on that CPU chip. "ditto -arch something" deletes the machine code for all but the specified architecture as it copies the file from source to dest. Drops the file-sizes by about half if they're fat for two archs (say, PPC and i386) originally and you only need the PPC machine code in your destination directory.


    'Way back before anyone contemplated mixing Apples with Unices, the MFS file system had three "branches" to a file: a data fork, an info fork for things like invisibility bit and create date and Creator:Type codes, and a resource fork for storing file resources like custom cursors and icons, local fonts, etc. The first alteration was HFS (Hierarchical) so you could have more than one file of the same name on the same disk (in different folders) and the last was HFS+ for very large (by comparison) disk volumes to have the capability of a bazillion little files instead of just a few huge files. Still multiple pieces to the files, and perhaps the info-fork "disappeared" into a more Unix-ish file-slot, but still Apple-Double has the resource fork as before. Fonts, custom-icons, layout instructions, doc templates, formatting or calc macros...

    Unix files are all basically what the Apple-Double system stores in the "data fork". Just the plain ol' file contents.

    If you set up a disk in OS X as a UFS volume, OS 9 can't access it. Where would that leave Classic? If you always set up your filesystems as HFS+, both 9 and X can traverse and use them. A lot easier than trying to explain "Well, your ufs volume that holds your Home directory is invisible when you're booted in 9 but not the other way around..." and "Don't copy files from your HFS+ volumes to your UFS volumes and expect them to be useful in 9 when you copy 'em back."

    What I haven't been able to figure out is *how* the resource-fork is both stored on-disk and hidden from the stock Unix commands! Canon: "In Unix, *everything* is a file." In this case, I'm wondering where or how.

    We will find an answer to dealing with Apple-Doubles on the Unix side of the monitor-screen! Gotsta!

    Jazzbo

  5. #5
    Join Date
    Jan 2001
    Location
    Mobius Strip
    Posts
    13,045

    Default

    UFS won't launch or run some programs. And it is the resource fork that is troublesome.

    I just tried CCC 1.3.1 'one more time' this time (hopefully) just to a .dmg file of 2.6GB.
    Use to work fine and makes a nice backup of the system - do it as soon as you have the
    basics and again when you have it clean but customized at the latest version.

    As soon as CCC began, and was copying ".DS_Store" there was a kernel panic.
    How in creation can that happen? CHMOD and ditto? Mike B has removed himself.
    AT least his former email address(es) that worked yesterday no longer and are 'fatal.'

    Jazzbo, be gentle with us on this, okay? Glad to test it, but my best guess is to try
    it with 10.2 which will likely be different enough you may have to do things differently.

    Actually, the tips on Bombich's home page for moving /Users and the scripts,
    should at least be a good start. That and MacSlash and MacOSXHints.
    ]http://www.bombich.com/mactips/index.html] Mac Tips - Bombich [/url]
    has a page on moving /Home.

    Gregory

  6. #6
    Join Date
    Jun 2002
    Location
    Campbell, CA, USA
    Posts
    732

    Default

    Breakthrough!

    A google search got me to
    http://www.westwind.com/reference/OS...s-folders.html

    when someone mentioned the CpMac command, which is the standard Unix cp command rebuilt to handle resource forks. The problem is that CpMac's only distributed in the Developer package.

    Well.... even better, the page documents an option for ditto, -rsrc or -rsrcFork, that is NOT in ditto's man page, but tells ditto to include resource forks when copying. Since ditto's on the standard base installation, Bingo!

    Testing/Confirmation tonight,
    Jazzbo

  7. #7
    Join Date
    Jun 2002
    Location
    Campbell, CA, USA
    Posts
    732

    Default

    The .DS_Store file simply hold OS X Finder's window-layout data for a folder, so unless that particular .DS_Store file, the directory it was in, or the directory or filesystem CCC was trying to copy it to were horribly damaged, I can't imagine that the file itself, being a .DS_Store, caused the panic.

    There's something else strangely amiss, Gregory!

    >UFS won't launch or run some programs. And it is the resource fork that is troublesome.

    UFS file-systems are single-file (not Apple-Double) only. In essence, everything on a UFS is purely data fork; no resource fork at all. Some OS X facilities -- Cocoa apps, I believe -- get around that by storing resource forks in normally-hidden files with "._" prepended to the filename. You'll see that on your iDisk in OS X if, in Terminal, you run 'ls -l' (ell-ess dash-ell) against any iDisk directory (folder).

    This is the only solution I know for UFS volumes to handle files with resource forks in OS X, and that *only* works for Cocoa apps.

    Jazzbo

  8. #8
    Join Date
    Jun 2002
    Location
    Campbell, CA, USA
    Posts
    732

    Default

    Hallelujah, I'm a bum!

    ditto -rsrcFork works!

    Note that this solves it for copying disk-to-disk (or directory-to-directory, even on the same disk). It does NOT allow the creation of an archive file for later unpacking. I guess for that, StuffIt Deluxe may be needed.

    Jazzbo

  9. #9
    Join Date
    Aug 2001
    Location
    Grangeville, ID USA
    Posts
    9,142

    Default

    Now that is very cool. Back to the grindstone with you.

  10. #10
    Join Date
    Aug 2002
    Posts
    1,010

    Default

    Awesome work, Jazz!

  11. #11
    Join Date
    Jun 2002
    Location
    Campbell, CA, USA
    Posts
    732

    Default

    Yeah, but we'll see if I can successfully remember how to convert my scripts from using "find | cpio" to using ditto. (R, r, r)

    Thanks for the support during the hunt,
    Jazzbo

  12. #12
    Join Date
    May 2000
    Location
    wherever I hang my hat
    Posts
    3,575

    Default

    dang.

  13. #13
    Join Date
    Aug 2002
    Posts
    1,010

    Default

    I haven't actually tried that CCC 1.3.1 yet-
    Not sure if this relates to Gregory's catastrophic experiece, but here's the major bug that 1.3.1 purportedly fixes:
    quote:

    "CCC version 1.3 modified the permissions on my source and target volume such that I can no longer copy items to the root directories".? Fix: Only the root directories, not their contents are affected by this bug. To repair the problem, simply type "sudo chmod 1775 /" in the Terminal while you are booted from the affected drive. This problem has been fixed with CCC version 1.3.1.


    On the upside, I very successfully copied a new user from the default (/Users) to /Volumes/Users/new_user :

    cd /Volumes/volume_name/Users (I had previously moved my Users but wanted to add a new user)
    mkdir new_user

    cd /
    sudo ditto -v -rsrcFork /Users/old_user /Volumes/Users/new_user

    Went into netinfo and and told the new user where to find its new home...

    As an aside, I used rm for the very first time. Boy, were my knees knocking when I eliminated the old users from the default position! All went OK- I decided to keep the parent folder /Users; not sure if X would automatically remake that directory if it had to- I think it does..

    Jazzbo, I was curious of your comment in, I think, another post regarding using ditto logged in as root rather than sudo from cli..
    You're saying not to use sudo in Terminal while doing ditto, but rather to create a root password and log in through netinfo?

  14. #14
    Join Date
    Jan 2001
    Location
    Mobius Strip
    Posts
    13,045

    Default

    I am still not 'ready' to give CCC another shot.

    My drives are on a SCSI controller that isn't fully supported. Jag is just around the corner. FWB Backup Toolkit did such a lousey job that I had to use Apple's Repair Privilages 1.1 so I could even launch applications! (funny that I wasn't in a Catch-22 bind, if I can't launch, how did Repair manage to on its own?).

    If Jag doesn't have full support for Adaptec 39160, then I'll put UL3S in its place (which does work in 7300 and beige but the 39160 doesn't).

    OS X does need to have the 'old' ~/Users on the boot volume, and yea, knocking knees on those rm commands is common!

    I just read someone say in another (RAID 5 thread) to use scripts (ditto?) for FM server backups rather than the supplied program.

    Gregory

  15. #15
    Join Date
    Jun 2002
    Location
    Campbell, CA, USA
    Posts
    732

    Default

    My recommendation on actually logging in to the desktop as root when you want to ditto /Users actually has little to do with ditto, itself, copying files and directories, but the knee-shaking 'rm' command afterwards.

    Say I login as jazzbo, go to Terminal, sudo, ditto /Users/jazzbo to /Volumes/Other/Users/jazzbo, and them 'rm -r /Users/jazzbo'. What happens to the resources currently open to support my desktop login as jazzbo?

    Since root's home directory isn't under /Users, if you're logged in as root, you haven't got anything using current contents of /Users.

    Caveats: It's not certain that /Users/* isn't in use if your system may have active rlogin, telnet, or ssh sessions, if there are active NFS, AFP, or Samba exports, or if you have crontabs or daemons running from within /Users.

    Jazzbo

  16. #16
    Join Date
    Aug 2002
    Posts
    1,010

    Default

    Thanks Jazzbo, that makes a lot of sense.

    So its probably safe or safer to rm the old /User/user_name booted from another user...
    But I can see considering the caveats, that probably the safest would be as root from the desktop of the admin user.
    How can I verify my home as root? If I hit command+up arrow in SNAX or in Finder with invisibles turned on, will that point me to the root home directory?
    Or, if I open a terminal shell, will that open as local host root?
    I'll try these experiments later tonight after going over how to log in as root...
    Thinking of root as another user has given me a new leap forward here..
    Thanks, Jazz

  17. #17
    Join Date
    Jun 2002
    Location
    Campbell, CA, USA
    Posts
    732

    Default

    Yep, you're on the right track.

    'root' is just another user: uid=0 (!). OS X is the only Unix I've ever seen where it wasn't also required that the root user be the very first entry in the passwd file.

    If you're logged in at Desktop level as the root user, open Terminal and then enter

    pwd

    which means "tell me my present working directory". Since Terminal windows start out in the home directory of the user running it, if you're logged in as 'root', you'll find that 'pwd' reports either /private/var/root or /var/root (the same directory since on OS X, /var is a symbolic link aimed at the /private/var directory).

    The safest thing is to do the copy and remove of /Users contents when logged into the Desktop as 'root', since root's home directory is NOT under /Users.

    Jazzbo

    [This message has been edited by Jazzbo (edited 20 August 2002).]

  18. #18
    Join Date
    Aug 2002
    Posts
    1,010

    Default

    Ok, this is just totally exciting-
    I haven't had a chance to try any of this out yet, nor have I deliberately logged in as root with a specific task to accomplish, but:

    What about booting into single-user mode. What is that? Can I aply the same commands? Can I log in as root? Wouldn't that be the ultimate situation where none of the disks are actually mounted yet (?), or (maybe only the boot device is mounted?) is that totally different? Do I need the GUI to get to root. I've heard I can run fsck -y on all the disks, not just the boot- by typing in their identifying code (sorry, don't know the term for it yet) after typing fsck -y.

    If I'd spent the time learning Unix commands and Terminal that I spent tracking down 3rd-party Unix-manipulating utilities like SwapCop, CCC, Xoptimize, etc., I'd have saved a bunch of dough!

  19. #19
    Join Date
    Jun 2002
    Location
    Campbell, CA, USA
    Posts
    732

    Default

    Heya, Boots!

    Single-user mode is a command-line only environment (as you said), specifically designed for maintenance. In OS X, it comes up with nothing but the root disk mounted, and it's mounted read-only. If you have multiple disks or partitions, you'll only get guidance on getting the root disk checked and remounted read/write.

    Most of the time, the things you want to accomplish as root are best done not in single-user mode, but either:

    1. As yourself, in Terminal, use either sudo or su to become root long enough to do your work;

    2. Log into the Desktop as root, launch Terminal, do your thing, log out.

    The reason is that the "run control" scripts (/etc/rc, /etc/rc.common) for MultiUser boot-up do a lot of carefully-ordered things to make sure the entire system is ready for work, including filesystem repairs and mounts, device-file access, memory and process management, logging, and so on.

    So, in single user mode, you have to manually invoke any of those things you need, with the correct arguments and in the right order, to do anything more than root-disk maintenance. And even to do that, you'll need to fsck (filesystem check) and remount it read/write.

    Obviously, I'm recommending that you do most of your rooting around (ahem) in MultiUser mode. Yes, libraries and log directories and networks and so on are in use, but *usually* if you want to change those, you want to be booted from a different disk or partition while you, again in MultiUser mode, do something to the boot-disk in question.

    Just a generally safer and certainly more complete environment.

    Not intending to be a wet blanket,
    Jazzbo

    [This message has been edited by Jazzbo (edited 21 August 2002).]

  20. #20
    Join Date
    Aug 2002
    Posts
    1,010

    Default

    That is a very cogent explanation of the proper approach, Jazzbo.

    The implication I take it is that if one knows the commands real well, you can do this work in single user, except that it really doesn't gain you much beyond fsck because your boot volume is mounted anyway and you have to do a bunch of extra work just to get to the plate..

    I'm going to have to get a more thorough book than the Missing Manual. Not that I've learned all it has to offer- from from it. But some text that gets to the bottom of how the entire system is set up...

    Plus booting in Verbose should be educational...

    ..........so that's what rc stands for- run control. One thing I want to try when I get more understanding of the issues involved, is to move /Applications to the top of my Users partition, which is on a different drive from the boot. I figure I'll gain from the buffer on the second disk....(actually, all of my non-Apple installed apps are at /Volumes/volumename) but I've heard it has to be done just so, a mod of the etc/fstab. And even this may not prevent problems with application installers...

    So this is another benefit to having a second boot volume: boot from that, log in as root, and work on the first boot volume.

    Anyway this is just the beginning for me- lottsa learning to do!
    Thanks again for all your help.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •