PDA

View Full Version : Remote OS installs - The Sequel



unclemac
04-28-2005, 11:38 AM
Just had some pretty serious success with updating 48 ibooks with RadMind (http://rsug.itd.umich.edu/software/radmind/).

We had a two day window to get 48 ibooks updated. The update entails moving from 10.3.3 to 10.3.8, plus most every app had an update, service pack, or new version, plus admin specific tweaks to the OS. All of the users were travelers or location directors that are spread all over the US, and a couple from London UK. They were here locally for an annual training session, so a small and finite window. Fun.



The Problem:

Previous to RadMind, we would have backed up the data from all the users (we have 1 admin and 2 managed accounts on all machines) to FW drives, wiped the drive, and reimaged it with Netrestore, and then manually moved the data back, and then semi-automatically reconfigured the setting for the specific machine. Netrestore is an awesome product; it takes us about 10 minute to boot the recipient Mac to Target Mode, wipe the drive with Disk Utility, and reimage with Netretore.

The time consuming part is the backing up and restoring all the data the user(s) have - perfectly. At first glance, it doesn't sound too bad: grab each user's Desktop and Documents folders, plus the Shared folder and that's it, right? Not that easy. Don't forget about browser bookmarks, and a myriad of user specific preferences that vary by application, and niceties such as customized dictionaries. And that is just the user specific information that the users care about.

Over and above that, there are a handful of machine specific settings that are critical too. Network preferences, including preconfigured Locations. On top of that, Apple uses byhost preferences (that are MAC address specific) for some items, and therefore do not move correctly when a volume is cloned...so there is always considerable cleanup and configuration to do. Moving data and config specifics is about 2 hours worth of work per machine, and is labor intensive enough to be fairly error prone.



Work Group Manager

If that isn't enough (and believe me, it is!), we are now using Work Group Manger from OS X Server to manage non-admin accounts. This includes all of the same features that are included with the Accounts Preference Pane in OS X, plus other features that are very handy for a managed account. Things like locking the dock so a user cannot add or delete anything items, plus being able to lock a user out of any Preference pane as needed. Plus you can easily modify users and groups permissions, have instant access to Netinfo WGM is a great tool. I have know about it for years, as has every Server admin type....what I did not know until MacWorld SF 2005 was that any Mac OS X box can use this app. I had blindly assumed that it only ran on Server, as there are several other apps in Server that will not run on non-server versions of OS X. This is the the golden missing rossetta stone that we have been searching for high and low for years...under my nose all along. Doah! Apple should be pushing this tool - no, they should include it with every copy of OS X.

To get it, one must dig up the link to Server Admin Tools and download it... All 272 MB of it. Lot's of other goodies included to wade through, but WGM is in there:

Here is the link to the most recent full download I could find (article ID: 120280): http://docs.info.apple.com/article.html?artnum=120280

So with WGM installed and configured on our new master image, we needed to preserve these settings and add them to all the machines, as well as everything else. One more thing that cloning can sometimes break, mangle, and even butcher.



The Solution:

RadMind.

Radmind is a completely different concept than other products for OS X, like CCC, Restore, RsyncX, SuperDuper, and the like. All of the these cloning/imaging apps are designed to primarily clone entire volumes. Although you *can* clone only certain directories, the real issue is being able to easily tell what needs to be updated, and get those components updated, without disturbing any user data or preferences, or any of the other components that *don't* need to be updated. All the cloning software we have used over the last few years is not designed to do this. Radmind is.

From the RadMind Manual:


Radmind (Remote Administration Daemon) is a client management system that allows you to create a specific setup on a single Mac OS X system and then be able to implement the same setup on multiple clients. Most importantly, radmind enables you to install updates andnew apps on a single Mac and then force the other Macs to inherit the same configuration. It can be set to automatically bring back systems to a pristine state every night as a way to distribute new system updates on demand.

Radmind supports multiple configurations so one radmind server can handle several departments in your organization, each with its own setup and applications. At its core, radmind operates as a tripwire; it is able to detect differences between the server and the client to any managed file system object, e.g. files, directories, links, etc. However, Radmind goes further than just integrity checking: once a difference is detected, radmind can optionally take action.

This is ideal for small to large businesses as well as schools and universities. radmind not only lets you upgrade and keep all systems the same, it also lets you down grade if you need to. radmind is generally useful if you have three or more Macs that need to run similar or identicalconfigurations. You can use radmind to combat any application or system corruption and even deliberate mis-configuration by simply running the radmind update session. When used with checksums, radmind also verifies the integrity of files and any damaged ones are replaced.

RadMind is just about perfect for our needs. But wait, there's more...it can "synchronize" any machine anywhere on a LAN or WAN, with the only restriction being the time it takes to push out the files and changes, and that is merely a function of bandwidth. This is huge.

We have about 40+ remote locations, plus a further 20-30 people that travel and are on location in some very remote areas. Had two travelers out in the tundra on the North Slope in Alaska last year. They had two options to get to work: local Mail plane or a snow mobile. We are talking remote. If their ibooks need an update or has some issue related to say OS/app compatibility, or maybe a permissions issue, they have a serious problem that could take well over a week to get resolved if they have ship the ibook back to us.

So RadMind over IP sounds very tempting indeed.

But be warned, it is involved. it is a simple concept, but not simple to understand the working details of nor implement. Radmind itself isn't too bad, but as it looks at every file and directory on an entire volume (includig all of the invisible ones), it is up to the admin type - a coworker and myself in this case - to decide what files to manage, and which files not to manage. Without rewriting the manual, I will get a brief overview of the concept and the key terms that must be undstood to use it.


More details to follow.

Nicolas
04-29-2005, 03:22 AM
Those RAD tools are buildin every Linux Distri.
Nice to have such a powerfull tool for OSX.

Thanks again.

Regards

Nicolas

unclemac
04-29-2005, 08:13 AM
Yes sir. Scary powerful.

Here comes part two.

unclemac
04-29-2005, 09:30 AM
Transcripts


RadMind is built around a underlying principle: The Radmind server holds and serves documents that list every file on a volume called transcripts. The transcripts are similar to a bill of materials for an installer: They document what should and should not be on a machine. Along with the transripts, Radmind also generates the actual loadset of files from the master client, which are then uploaded to the server to be pushed out to other machines.

Using the transcripts generated from the master client (the machine image we want to push out), Radmind compares a machine designated to be updated to the master client image, then generates a list of things to change, which includes all items to be added, all items to be deleted, and all items to be modified. Radmind clients then download and install the necessary components from the loadset on the server required to mirror the condition of the transcripts from the server, as well as delete and modify the correct items - all in the proper order.

To function, two transcripts are required: One positive, and one negative.



Positive Transcript: Any item listed in a positive transcript will be be fully managed. This means the attributes of the item itself (and its contents) will be verified against the model on the server, and changed to exactly match the server image if it differs from it. Every item that is on the machine needs to be in the positve transcript; any item on the machine and not on the positive transcript will be deleted.

Negative Transcript: Any contents of item listed in a negative transcript will not be managed. In most cases, items in the negative transcripts are only managed in the sense that Radmind ensures their existence and attributes, but does not manage their contents, whether the object is a file or a folder.


Radmind will build the positive transcript automatically from your "golden master" client that you want to copy. They also have two prebuilt negative transcripts that get you about 80 - 90% of the way, and cover almost all of the common OS attributes that one would want to not manage (not change). So if you get that far, all that is left is to customize the negative transcript to fit your image and requirements.

Easier said than done. Finding and including the correct files and folders can be like searching for a needle in a stack of needles at times.... Here is where you really need to know OS X inside and out. Do you know where to find the preference file that controls the desktop/screensaver Preference Pane? Can you find and add the file that contains the authorization code for MS Office 2004? The list goes on and on...

And another wrinkle: see how everything to be left alone or added to a machine must be in one of the transcripts? Well, kinda. Once Radmind compares the existing items and structure to the master transcripts it generate a transcript that documents the actual changes to be applied to this particular machine. You can manually edit this transcript before applying the changes.

So say a user has Palm desktop installed in their Applications folder, and your master image does not have Palm on it. If you don't manually delete the Palm directory and all of it's components from the apply script for this machine before it runs, it will delete the Palm directory. Ouch.

Wanna see a negative transcript?

Here is one of the default negative transcripts provided by the RadMind organization. Lines that start with "f" are files and lines that start with "d" are directories (folders). Remember that these are the files that are not managed, and that directories are managed but their contents are not. All the lines that start with # are simply comments they added so we can all have some idea what the items actually do within the OS. All of you lurking Unix gurus will recognize that the permissions for each file are list here in their numerical represention too. That's right: you can repair permisssions (assuming they are correct on your master image) by running Radmind!

I cleaned it up a bit to make it easier to read:

# A template negative transcript for managing machines running
# Mac OS X 10.3 on personal desktop or laptop computers.


# Stores trash on the root volume
d /.Trashes 1333 0 80

# Apparently associated with journaling
a /.hotfiles.btree 0600 0 80

# Used by the system and NFS
d /.vol 0444 0 0

# Caches and other directories containing volatile items
d /Library/Caches 1777 0 80
d /Library/ColorSync/Profiles/Displays 0775 0 80
d /Library/Logs 0775 0 80

# Some property lists that tend to get touched frequently
f /Library/Preferences/.GlobalPreferences.plist 0644 501 80
d /Library/Preferences/SystemConfiguration 0755 0 80
f /Library/Preferences/com.apple.loginwindow.plist 0644 0 80
d /Network 0755 0 0
d /System/Library/Caches 0755 0 0

# Kernel extensions cache files. Created on boot if missing or older than /System/Library/Extensions.
f /System/Library/Extensions.kextcache 0644 0 0
f /System/Library/Extensions.mkext 0644 0 0

# User space. Comment it out if you want to manage it (not recommended).
d /Users 1775 0 80

# mount point for other drives
d /Volumes 1777 0 80
d /automount 0755 0 80

# /dev is a pseudo-filesystem (devfs) created on reboot
d /dev 0555 0 0

# mach symbol file created in /etc/rc on reboot
f /mach.sym 0444 0 80

# Printer configurations. Placed here so desktop users can add and delete printers as it becomes necessary.
d /private/etc/cups/ppd 0755 0 26
f /private/etc/cups/printers.conf 0600 0 26

# Used to turn on and off system services
f /private/etc/hostconfig 0444 0 0

# Web sharing configuration
f /private/etc/httpd/httpd.conf 0644 0 0
d /private/etc/httpd/users 0755 0 0

# Printer file manager by CUPS
f /private/etc/printcap 0644 0 0

# world-writable temporary space
d /private/tmp 1777 0 0

# Contains databases of system information, such as NetInfo.
d /private/var/db 0755 0 0

# System log directory
d /private/var/log 0755 0 0

# Location of radmind files on the client
d /private/var/radmind/client 0755 0 0

# root user's home directory
d /private/var/root 0750 0 0

# Directories containing temporary items
d /private/var/run 0775 0 1
d /private/var/spool 0755 0 0
d /private/var/tmp 1777 0 0

# Virtual memory
d /private/var/vm 0755 0 0

# whatis database. Rebuilt weekly by
/private/etc/periodic/weekly/500.weekly
f /usr/share/man/whatis.db 0644 0 0



In our first production run we were able to get about 98% of what we needed in our negative script to avoid changing specific files and folders, and that was good enough to use the tool for this project. Would have loved to get it closer to perfect, but time was working against us, so we had to go with "good enough". And even though it was less than perfect, it was still much better than any other tool we had available. To our first run, we added over 200 more lines to the above defualt "desktop" transcript.

We used a 1.6 G5 imac as the server, and were able to connect and image as many as 11 ibooks at a time. The image time was about 1 hour and 45 minutes, give or take, and did not appear to be any slower doing the 11 as opposed to just a couple at once. Maybe we could have done 15 or more at once, but we were ran out of space, network drops, and power outlets.

Our positive transcript had about 82,760 line items in it, and of those we managed just over 62,000 lines. So we "updated" about 3/4 of the files on the ibooks.

Kinda like unscrewing the radiator cap and driving a new car under it. :kickass:

Chris Billington
04-29-2005, 11:23 AM
Amazing stuff. Hyoodge.
CB :eek: :cool: :dance:

Macaholic
04-29-2005, 08:54 PM
Nice work Uncle,
Seems you are breaking new ground and excited. :D
So many possibilities,
And nicely documented,
Unix huh.....
Tried to digest this today...
http://developer.apple.com/macosx/xcode2.html
Thanks,



Dave

unclemac
04-29-2005, 09:37 PM
Yes indeed.

We have been dabbling a bit with the Developer tools too. Fun stuff there as well.

Other little nuggets I need to write up too, like the right way to modify items to be able to allow (or lock out) managed users from having to authenticate to use a specific app.

Those MacWorld classes are starting to pay dividends. :)

unclemac
06-28-2005, 05:05 PM
Not much progress here for some time, cause other wheels were squeaking away....

Jumped back in last week and found a substantial update to the OS X Radmind tools. Speed has jumped up dramatically:


Upload a new Positive Transcript:


Old Version - 1 hour, 15 min [+/-]
New Version - 12 min [+/-]

Update a machine from 10.3.3 image to 10.3.8 image:


Old version - 1 hour, 25 min [+/-]
New Version - 20 min [+/-]


Wicked Fast. Lots of minor bug fixes too.

Starting to actually figure how it works as well, as opposed to copying everybody elses work. More fun to come.

TZ
10-20-2005, 05:23 AM
Multicast ASR: The Fastest Way to Manage Mac OS X Deployments (http://www.informit.com/articles/article.asp?p=419258&rl=1)

unclemac
10-20-2005, 10:30 AM
Thanks for the link TZ. Very good stuff.

Not sure how much we can use at work, cause unlike most schools, we need to keep all data, and most configs (printer lists, etc) while updating the OS and apps. ASR, from what I understand, is usually used to drive a complete image onto a machine, to start fresh. We already use NetRestore for that, however we have been doing lcoal via FW, not over the network. May be time to get stuff networked for the base image.

---

Been making slow but steady progress with RadMind, and it is the real deal. Sounds kinda obvious, but the hardest part, once the methodolgy and procedures sink in, is understanding what file does what.

I updated a hacked up machine just the other day that had a transcript (the log of changes to make: files to add, delete, or modify) of 166,000 lines. A transcript for a "clean" machine that just needs normal updates is typically only 50-60,000 lines.

Easy to edit, but what to edit makes for a steep learning curve.

It went well, although I had to reinstall Office 2004. But I know why that happened, how to fix it, and honesty....it's one of the easiest problems to fix.

We are looking at serious video conferencing, so we will likely end up with a VPN or even a private IP network, which should ease the process of RadMinding remote Macs over the LAN. I hope.