Recovering from Disaster

Some people may not call it a “disaster”, but in today’s digital world the loss of digital data, is in most people’s minds, a disaster. Loss of a carefully crafted music library is one thing, but what about family photos that can never be re-created, or the years of work invested in a dissertation? Obviously there is a need to safe guard this data.

The following is an account of my experience with my 15” Powerbook G4 failing 31 December, 2007. I awoke that morning not suspecting anything was wrong, looking forward to ushering in the new year, although not looking forward to ending my vacation. However, when I tapped my trackball to wake up the screen, I was greeted with… nothing. No response. Rebooting brought my computer back on, but things were not right. In addition to constant waiting for file system operations, some of my desktop scripts were yielding unusual results. Then finally it hung completely again and I was force to do a hard power-down. All subsequent attempts at booting resulted in the prohibitory sign (grey circle with a slash through it) appearing. It wouldn’t boot at all. My computer was dead.

Triage

My first step was to determine what was causing the boot issues. A cursory look online told me that sign I was getting indicated the computer couldn’t find the boot volume. OK, I tried to boot from my Leopard install DVD. It would give me the same result. Nothing. I tried all the standard tricks: I reset my PRAM and my NVRAM and reset all that I could in Open Firmware. Holding Option (⌥) during the boot process provides a list of available boot sources. Mine showed both the hard disk and the Install DVD, both of which it identified correctly by name. One, both, nothing.

Without DVD booting, I feared that the logic board might have suffered. The glimmer of hope was that I was able to run the hardware tester on my original system discs. Quick test. Extended test. Both said everything was fine. Both also completely skipped my hard drive in their scans. An interesting development. Unfortunately at that point, I had to catch a plane back to Bermuda. I packed up the carcass of my computer and all my accessories, it was a sad moment.

Returning to Bermuda, I was getting depressed about the computer. I didn’t have the tools required to really do anything about it, and that was frustrating. So after a couple of days I brought my dead laptop into work. I figured, at least there I would have the tools to take it apart if it came to that. On Friday 4 January, I even tried to boot off of a colleague’s Macbook Pro. Again, no luck. Saturday had me returning to the office since that was where my computer was. I brought a couple of blank CDs this time in hopes of downloading and burning a Linux LiveCD or a recovery CD for PowerPC. I had good experience with these types of discs for problems with other people’s computers in the past.

Even here I had very little success. I could get into a boot loader, but the systems would complain of an error reading from the filesystem on the CD. My colleague Joseph, who is a serious Linux tinkerer helped me a lot that day. In addition to examining what little information we could glean from the LiveCD, he even went out to the store on his scooter when we realized that we didn’t have a Torx screwdriver. I remember clearly when he returned, success written on his face, and said, “It’s surgery time.” Yes, it was certainly looking that way.

Surgery

This is the part that separates the boys from the men. Or perhaps I should say those with warranties and those without. Any power user can try holding a few different key strokes when booting or even burn an ISO of a recovery CD. Some might even brave taking the side of their case off a desktop machine, but it takes a special breed of dedication to pry open a notebook computer. Fortunately I have experience with laptop autopsies. I’ve assisted on one involving a Sony Viao, and then spent several hours resurrecting an iBook G4 with a dead hard drive–similar to my problem. Unfortunately, the ardour of the iBook opening was actually dissuading me from taking a peak inside my own computer. I had heard that Powerbook were easier to open, but I was still reluctant.

Reluctant that is, until Joseph returned with the set of allen keys which was the missing tool in my kit at that point. I’m not sure who at Apple decided that it would be better to use some unusually obscure screw head for only two out of twenty-three screws that I needed to remove–but they were wrong. The remaining Phillips size 0 screws were small, but still not unusual in the IT circuit. Fortunately, as with my iBook experience, I found the Internet to be full of wonderful resources, including step by step guides to open Powerbooks. I used one from ifixit which was targeted for those users looking to replace a hard drive… which was pretty much what I was going after, except I didn’t have a replacement drive.

Deftly removing tiny screw after tiny screw, I removed enough to be able to lift the top cover. There is a bit of prodding required around the DVD drive, which was the worst part of the iBook as well. This was far easier because at that point you have the rest of the top pried up. A single narrow ribbon cable is all that connects the top case to the logic board. Keep an eye on this, because the connector just snaps in place lightly on the logic board, so if you lift the top off too far, it will disconnect. It doesn’t seem to harm it, but if you don’t reconnect you won’t be able to turn the computer back on. Another tapered ribbon cable connects the IDE connector on the drive to the tiny receptor on the logic board. You will also notice a fair bit of yellow tape holding various connectors onto the logic board. A low-tech solution, but it seems to work. I needed to peel back some of the tape before I could disconnect the hard drive from the computer. Once I had the hard drive out, I was ready to try again. Obviously I wouldn’t be able to access my files, but would I be able to boot from a CD?

In short, yes! Instead of completely reassembling the computer without the hard drive, I just gently set the top case on the bottom part of the computer. I turned it on and it started the boot process. Only this time it was able to read from the CD and boot fully. Oh, this was definitely a start towards something positive. I shut down, unplugged and once again peered under the hood. I decided to try plugging the drive back in, just in case the connection to the logic board was loose. That wasn’t the case, as soon as the drive was reattached, the problem returned.

Joseph pointed out that it still might be the ribbon cable or the hard drive. However, thinking back, the hard drive is really your best bet because they are so much more likely to fail. At this point it was late Saturday evening, and I wasn’t going to be able to buy a replacement or find any way to test the problem further until Monday. So I left my countless screws and dissembled laptop on the side of desk and headed home. I was back into work on Sunday for my fix of the Internet and I was feeling bolstered by the previous day’s success. Although my computer was in pieces beside me, I was feeling that the end was in sight.

Monday morning, I ducked out from work to purchase a USB enclosure for 2.5 inch drives. Since no one had one for me to borrow, I figured it would be a small price to pay to test the drive further. I hooked my drive to the IDE connector of the enclosure and plugged it into another co-workers Macbook Pro. It mounted but was only showing up as 5GB total size, far short of the actual 100GB. No, something was wrong with the disk. When I tried to access it from Disk Utility on my Leopard Install Disc, the enclosure started making a loud beeping noise. Not good. Fine, I would buy a replacement. It was end of the work day at this point and I headed back to the store where I got the enclosure. They had one 2.5 inch EIDE drive left! A 120GB 5400rpm Western Digital. It was 20 GB more than my dead drive and it was at a decent price. I bought it.

Returning and plugging it in, I tried booting from the install DVD again. It booted. This was it, the atmosphere at my desk was palpable. I checked in Disk Utility that the drive was visible; yes it was. I clicked Install, but it didn’t show up as a target. Damn. So close. Then I went back into the Disk Utility and formatted the drive with a HFS+ Journalled partition. This time the partition showed up as an installation target. I clicked go, and I was about one hour away from a functioning system. A small miracle was being enacted at my desk. Leopard installed while my computer was still in pieces and booted without problem. I shut the computer down and sewed her up. Ready to take back home and nurse back to full health.

Recovery

As I mentioned in my introduction, digital data is the key asset nowadays. I had my computer running, but my former hard drive was toast. No data.

Enter Time Machine. Time Machine is the new backup software system included with Mac OS X 10.5 Leopard. And it is great based on one simple reason. People want to use it. If you plug in an external drive to a Leopard system for the first time, it will ask you if you want to use it for Time Machine. You click Yes and you are done. It just does the right thing. It creates a full incremental backup of your drive without getting in your way. Granted, if your drive is full, the first backup does take some time, but after that, backing up is quite transparent. Fortunately I had started using it when I installed Leopard in early November. I’m glad I did, it literally pulled me from the fire on this one.

When I got my computer home, I booted it up and it came on as expected. Perfect, the reassembly hadn’t caused any problems. The moment of truth, I plugged in my external drive and it mounted without a problem. And there it was, a perfect replica of my old hard drive sitting there… several times over. I should mention at this point, that when I was finishing up the installation of Leopard, the installer asked me whether I would like to migrate old data from, among other options, a Time Machine backup. This means that you could basically startup again where you left off. I’m not sure how the Migration Assistant actually assists you in choosing what you want brought over, I’ve always done it manually. I’ve heard mixed reviews in the past about that process breaking a few things, but if you want to start up on a new installation where you left off before, it is likely the quickest and easiest way. But as I mentioned in a previous post, I was interested in moving over slowly, only taking things that I actually need.

Of course, moving over this way isn’t for the feint of heart either. Documents and files are easy to bring over, as are most applications. Some applications however can have extensive configuration that you have built up over time. All of this trickiness boils down to one primary folder: ~/Library/. This is the folder which holds all of your preferences, configuration files and in some case even application data. A wholesale copy is defeating the purpose of a clean new system, but what is important?

If you take a look in a clean install, there are a number of folders that are in there by default. The list can be daunting but there are some important ones to consider. There should be a sub folder of the library called Application Support. This is where applications can store miscellaneous bit of information and data to support their operation. Things that are specific to a user, and that shouldn’t be clobbered when an upgrade is installed. Some software will have a folder directly inside the Library, like Safari, but in general, they should be in Application Support. The other folder inside the library to make note of is Preferences. These will be revisited often.

I have been keeping track of what software I have reinstalled and what procedures I have gone through to recover any data I need from my backup. I’m not going to talk about all of it here, but some of the initial parts might apply to most people and could be helpful.

Things I’m Restoring to Philyra

• First things first–setup Internet access. In my case it meant entering my PPPoE settings again, but depending on your networking environment, this might just be a matter of plugging in an ethernet cable or entering a wireless password. Your network has a password right?

• Install Camino. I prefer the Camino browser to Safari. This is for historical reasons, and I suspect Safari 3 would suit me quite well. I downloaded the newest version. I will talk about bringing over bookmarks and other bits in a moment.

• Install Skype. This is one of the few programs which I have startup when I login. Fortunately your contact list and account details are stored remotely, which enables you to simply log on to Skype and go. For the uninitiated, Skype is an Internet telephony application with a nice instant messaging component. It is the only IM that I use. Also of note, is that the Mac version has a much nicer interface than the Windows version.

• Install Quicksilver. You either love Quicksilver, or you hate it. I know I don’t use it to its full potential, but it is something I need on my system.

• Install Growl. This useful notification service is really a nice addition to my workflow. I would recommend re-downloading this one unless you have an old .dmg file for it. It installs as a preference pane so is a bit different than your average application.

At this point I have my recovery command centre up and running. Ready to proceed with some of the more heavy lifting and tweaking. Obviously the default applications that come with Leopard are already on my system at this point: iTunes, Mail, etc. It is Mail which is my next goal.

• Copy over email prefs, accounts and messages. Easier said than done perhaps. Fortunately Apple has a support document which discuss backing up and restoring important files. It is for Tiger, but the premise is the same in Leopard. In my case I wanted all of my accounts, mailboxes and actual email messages returned. If this isn’t your desire, read the support document. At this point I needed to venture into my backed up files. The path to those files will depend on you drive name, but it will be something like this.

  /Volumes/<backup drive>/Backups.backupdb/<computer name>/<timestamp>/<old drive>/Users/<username>/Library/

  Now that is quite a mouthful. All of these values are your old values if they aren’t the same any more. The timestamp that you choose should be recent enough to contain all the data you want. Each one contains a snapshot of your files at that time. Once in the appropriate folder, you can drag over the Mail subfolder into your new local ~/Library/. You might want to grab Mail Downloads while you are there as well. There is just one more file which is hiding back inside the Preferences subfolder. You can go there on your backup drive and locate the file com.apple.mail.plist. Drag this file into the Preferences folder on your local drive. Basically when I say copy over a preference file, this is what I mean. You also needn’t worry about messing up you backup because it is, quite appropriately, read-only.

• Copy over Keychains. Here is something that you wouldn’t miss until it was gone. You Keychain is the encrypted file where all of your passwords live. If you have ever clicked on the “Store in Keychain” checkbox in any number of programs such as browsers, than you have important items in the Keychain. Perhaps you don’t rely on yours much, but I do. I have a separate non-login keychain that I manually have to open which contains a number of my various, mostly online logins. Unlike my default, login keychain which is unlocked all the time I’m logged in, this Keychain only opens (decrypts) when I need to access it. This gives a little extra piece of mind. But it also means that my passwords, which I randomly generate and often never see, are trapped in there. The keychains are appropriately enough in a folder called, Keychains. In this case I recommend making a copy of your new login.keychain before copying over the content of your backup. Once copied, you should be able to open Mail.app and things should be fine. Mine took a while to re-index my email, but once that was done, I was in checking my accounts without problem.

• Configure Quicksilver hotkeys for iTunes. I was finding it a bit tedious working in stark silence, so I wanted to bring over some of my music. I don’t really care about my old playlists per se, so I was planning to take over my music as I has desire to listen to it. I might mount my backup library in iTunes at some point and back it up to DVD properly, but at the moment I just copied in a couple of albums. Since I had Growl and Quicksilver, I had all I needed to resume my music listening workflow. I made sure the iTunes plugin was installed in Quicksilver, along with the Growl plugin and the Notification Hub. I only use one hot key: Play/Pause, which I assign to ⌥-⌘-Space – after disabling the Spotlight window which normally uses that.

• Copy over Terminal preferences. At this point I went back into my Preferences in my backups and found com.apple.Terminal.plist. At this point, since you are there, if there are any other .plist files that you think you might want, like org.mozilla.camino.plist, you can bring those across now too. If you want to restore you bookmarks in Camino, they are in the Application Support/Camino folder in the Library, in a file called bookmarks.plist. You may want to bring that whole Camino folder over if you want all of your cookies, history and other files. It is up to you. I didn’t.

• Install a selection of my favourite fonts. I noticed when I opened Terminal with my old configuration that it looked odd, this was because I was using a non-standard font. So I took the opportunity to launch FontBook. I wanted to add my fonts back going through the proper channels, and since I had so many before I was only going to install the ones I used most, for now. I browsed to the Library/Fonts folder in my backup home directory and found the ones I wanted. FontBook added them, and once I relaunched Terminal, I was all set.

• I noticed that my Developer Tools weren’t installed once I started playing around on the command line. So I put the Leopard install disc back in and went to the optional installs area. From here I first installed X11. X11 is a windowing server in the unix world and is needed for some ported software from other unix systems. Matlab is a good example. Most likely won’t need this, but if you do, it is on the install disc. I also installed the Developer Tools at this point. Xcode, along with a host of command line utilities and a few other development applications are installed with this. Took about 25 minutes to install. Again, home users don’t need to worry about this.

• Run Software Update. There was an update for 10.5.1 along with Quicktime and a couple others. Reboot.

• I use a Microsoft Natural Ergonomic Keyboard 4000, and when I plugged it in at this point, I noticed that my extra features didn’t work correctly, so I went to the Microsoft site to download Microsoft Desktop. This also required a reboot.

• Download/install VLC for all of my video watching needs.

• Download Transmit and Unison. Usually I only buy software if I use it, so these programs are ones that make an early reappearance.

• Install TextMate. The ultimate in great software, I am lost without it. This is the program I missed most throughout this whole ordeal. I copied over my Bundles and other preferences. And of course, with my trusty text editor, I need my file to edit, so I copied over a large number of my web sites, which I keep in /web/clients/ on the root of my drive. In order to get the sites working I need to copy over my Apache config files. Since I decided to retire some of the sites I hadn’t touched in years, I had to remove these Virtual Hosts from my vhosts.conf file. Don’t forget the /etc/hosts file as well. Flush caches, start Apache and… in business.

• But what about my database files? More data trapped in the ether. My last installation of MySQL was version 4.1, and I was hoping to upgrade, but I only had the binary server data files. Couldn’t find a quick answer online as to whether they were binary compatible with version 5.0, so I download and install MySQL 4.1, from source. I won’t explain this here because there is a nice article at Hivelogic about just this. I copied the database files into the new installation and ran mysql_dump --all-databases to get an SQL representation which would be more portable.

• Uninstall MySQL 4.1 – install MySQL 5.0. Import my data. All the databases seem to be there, with all the tables. Good stuff. There were a few small tweaks, like making sure that MySQL and PHP were in synch with regards to where the mysql.sock was located. Once in agreement, the local versions of my websites began to hum with life once again.

At this point I am happily running along with my development environment pretty much all set up the way I like it. A computer is a dynamic entity with new data constantly arriving and old data being lost in a collection of magnetized bits. As my last post about Ruby alluded, there might be a new exploration beginning into the world of Rails, but time will tell.

Summary

It was a rough week without my computer. Being without Internet is nothing compared to being without a computer. But if anything can be learned from this experience, it is that regardless of how careful you are, disaster can strike, and you are always going to handle it better if you are prepared. I had backups, Time Machine saved me on that one, and you better believe that I will continue to use Time Machine and advocate it to anyone who will listen.

I only hope that this article is useful to someone. I suspect I’ll never know, but it was cathartic to write and might be somewhat enlightening or entertaining for my usual readers. Cheers.