Base | Lab

From time to time, I embark on what can sensibly only be called a lab experiment. This section is where I keep my notes.

2006 01 29

Base | Lab | iTunes


Brainless backup

I recently had an email conversation with an otherwise happy user of my tip on how to manually move your music files, whilst retaining track usage data in which he mentioned backing up his files.

For one thing it has never occured to me to do that, but that is not uncommon for some other Best Practices that I can often be found beating mere mortals over the head with. What can I say... physician heal thyself, it seems.

Anyway, the conversation prompted me to think of ways in which one could have iTunes help out in this mundane task. Below are some thoughts, which are clearly just the ramblings of a sleep-deprived geek. Enjoy them as such. :)

Option 1: Full backups. For people with DVD-RW hardware or media to spare who also enjoy the sound of spinning burners in the background. (Or whose music collection is ludicrously modest by today's everyone-is-a-pirate standards.)

  1. Create a Smart Playlist that has everything in it. This might be 'Kind contains MP3' or 'Bit Rate is greater than 0 kbps' Whatever works for you.
  2. Burn your playlist to DVD. iTunes will ask you for more media if the file content is bigger than what can fit one disc.

I would suggest you always keep two sets of backup discs and toggle them.

Option 2: Full incremental.

This is basically the same idea as option 1, but by maintaining a Playlist that selects on 'Added after [date of your last backup]', you can save yourself the trouble of backing up everything for the sheer excitement of it.

Option 3: Automatic

Now this is the actual no-brainer I mentioned in the title. Works like this:

  1. Set up a Smart Playlist selecting on everything. (see above)
  2. Limit it to whatever medium you intend to use, say 4.6GB for DVD. (you may have to experiment a bit to get a good number; the 4.7GB on the DVD case is a bit misleading) and do so by most recently added.
  3. Burn away.

What this will do is grab the track you added most recently. If you merely buy a track every now and then or rip the occasional CD, this will result in a stack of DVDs that will have a fair number of tracks backup up several times over, as they get pushed along the queue when new files come in.

Caveats: if you add more than (say) 4.5GB inbetween backup sessions, you obviously miss out on some tracks. Increasing either the size of the playlist (for instance to the size of 2 DVDs) or the frequency of your backups may help.

Also, you may find that one track might be backed up dozens of times, where others exist on only one DVD. That may bother some, but not others.

To the question why you would even consider using iTunes to do your backups, rather than some file-based tool to just grab directories off disk, I would say that outside of niftiness and convenience, you will not have to hunt across your disk to find your files; iTunes will do that for you. Also, creating multi-disc backups is not the easiest thing in the world in some burn apps..

A definite downside is that you may or may not be able to control any form of directory structure on the backup discs, unless you agree to having iTunes manage your music files.

2005 12 14

Base | Lab | iTunes


Overlapping tastes

I share a living space with my wife and family (a concept not unheard of, I guess). For the time being and seeing as the eldest is about 2.5 years old, the children have very little to say about what songs get played. Actually, I make a concious effort to expose them to the full scope of my own taste in music (which has in the past attracted comments like 'stretches the meaning of eclectic')

In any case, the only relevant tastes I need to deal with are my own and my wife's. Fortunately, mine encompasses all of hers.

What I did to create the possibility to switch between her taste and mine, wilst maintainting the cool dynamic of my Shuffle playlist, is to assign meaning to four ratings:

no stars
This track has not been assessed yet for this purpose.
1 star
Definitely outside my wife's taste
2 stars
Possibly inside my wife's taste
3 stars
Definitely inside my wife's taste

For my wife's automatic music playing needs, I use a smart playlist that is exactly the same as TheShuffle, but with the added rule that tracks must be rated two stars or higher.

One nice aspect of this mechanism is that by adjusting the rating, she can influence tracks appearing in her playlist or not. Another (deliberate) effect is that I can codify my unsureness about her liking a certain track.

Last (but rather obscure) benefit of the overall setup is that my main playlist reacts not only to itself being played (in the intened way), but also to her playlist being used, or any other track selection mechanism, for that matter.

Base | Lab | iTunes


Controlled randomness

As promised earlier, I will tell something about the rather elaborate stack of (smart) playlists that I use in iTunes.

First, let me state the Ultimate Goal that started me on this road: to have a playlist that I could blindly use, but that would both be as random as possible, but with a moderate preference for newer (i.e. more recently added) tracks. I got the idea off a mailing list, where someone had his currently playing tune mentioned in his email footer. Someone commented on the appearance that he had but one or two CDs and was apparently listening to them non-stop.

He explained in an answer to that comment that it was merely a coincidence, but not a strange one, as his playlists favoured newer tracks. It was never explained how he he achieved such a dynamic, but I was intrigued to the point of being hooked. In fact, I ditched WinAmp in favour of iTunes the very same day.

Over the course of time since then, I have tried a variety of algorithms, improving and tweaking as I went along. I should admit to trial-an-error, spurred on by the occasional insight, rather than systematic research. In any case, here is my thinking, so that you can either save yourself the trouble or correct my thinking.

The first thing to note is that any setup should be somewhat tuned to the amount of tracks in your database. Not as much the higher numbers, but rather the starting point of someone ripping all five of their CDs.

Second is the notion that to do anything above creating a very basic dynamic involves layering Smart Playlists. This will undoubtly result in a few playlists that you will use and just as many that merely form the building blocks. Recent incarnations of iTunes have a 'folder' feature, that I use, for this very reason, to hide those building blocks, since I will never use them for playing as they are.

OK, building blocks, then.

The most crucial playlist I use is one called 'Recent'. It's 'formula':

  • Match any of the following rules:
  • kind contains MPEG
  • Limit to 2 hours selected by most recently played
  • Math only checked songs
  • Live updating

In English this means: any MP3s that are considered active and have been played for the last two hours of playing.

Next up is a list called 'Exclusion'. The name is pretty accurate: it consists of a list of rules that are all like 'Playlist is not [Foo]. The idea is that in my library, there are a number of tracks that I do want to keep, but that I really only ever want to hear on purpose and thus never as part of any Smart Playlist. I manage this by maintaining a number of manual (i.e. normal) playlists, such as 'Spoken Word', 'Children's music', 'Stuff only my wife actually likes', etc. I find that using membershp of a certain playlist is easier to manage than maintaining multi-value 'Genre' tags. For one thing, I can use the interface with drag & drop for track management. Other than that, the difference is negligible.

In any case, to enable the concept of 'never automatically play any of these tracks' I collect these playlists in one called 'Exclusion'.

The next playlist I have called 'potential', but the name is rather meaningless in the current context. Its 'formula':

  • Match all of the following rules:
  • Playlist is not Exclusion
  • Playlist is not Recent
  • Limit to 10 hours selected by least often played
  • Match oly checked songs
  • ive updating

In English: figure out which tracks I want to hear at all, but leave out the ones that recently have been played. From all those, grab ten hours of the ones played the least often.

Whilst playing tracks, the dynamic emerges that tracks get played, get excluded for some time (determined by the limit in 'Recent') and then re-enter the pool. At some point, however, they are played too often and drift outside of the time limit set in 'Potential'. This is where the size of the library comes in: if you have just a dozen hours or so total playing time to work with (or you aplpy this mechanism to a subset of your library), you should use smaller values. Beware that large values (resulting in playlist with thousands of tracks) require lots CPU resource every time a track ends and mulitple playlists get (re)processed.

Obviously this dynamic is not nearly enough for the demanding bastard that I am. :)

The final playlist (and the one I actually use the most) is called 'TheShuffle' and looks like this:

  • Match the following rule:
  • Playlist is Potential
  • Limit to 54 songs selected by random>
  • Match only checked songs
  • Live updating

The net effect in terms of the playing dynamic, is that over time, the play counts of the tracks in the database levels out. In the meantime, by picking 54 songs (the amount which fills one screen on my laptop with iTunes maximised) out of ten hours playing time at random, I very much avoid repeating patterns in the order in which tracks are played.

The last element to throw into the mix (and somewhat of the reason I add the randomising layer) is that I play the 54-song list in descending order of track time, with the list set to repeat. Because of the 'live updating' set on all Smart Playlists, replacing songs can get added above or below the current one playing. It may thus take anywhere from 4 hours (a typical total playing time for 54 songs) to ten hours (in case the newly added songs are biased towards the 4-5 minute range).

I would be interested to do a study in larger libraries, but I think there is a weak correlation between song length (playing time in iTunes parlance) and genre. My technique thus groups genres dynamically based on that weak correlation.

Base | Lab | iPod


I now belong

Today's my birthday and the direct consequence of that is that I am now a lucky owner of a white iPod Mini. My better half not only shelled out for that desirable piece of kit, but also threw in a black leather cover (with a plastic sheet in front of the screen) and a USB charger.

I shall now be entering the world of PodCasts... :)

Base | Lab | iTunes


Avoid very large playlists

A while ago, I had intermittent but persistent problems with iTunes (on WinXP) maxing out the CPU to the point where the music would start skipping playback.

At first (as I recorded here) I thought the issue was with a USB vs FireWire connection to the external drive on which the music files are stored. Using USB instead of the FireWire surely made the problem worse, as did the fact that the file system was severely fragmented.

So, I defragmented the drive and things were better--but not over just yet.

Really by coincidence, I found out what the actual culprit must have been: automatic playlists that have around 8000 songs each (largely overlapping, off course).

I had about a dozen of those lists and they used the 'Last played' data as a criterium. In effect, after every song, each of those lists would have to be updated. The process of updating involves (from what I can tell from file movements and memory usage) building a catalog of songs in memory, wich then get written to temporary files, which then, in turn, get written out to the XML file in the iTunes directory.

Open that file (or a copy thereof) in your favourite text editor some time. You will find that not just the 'recipe' for your automatic playlists is listed, but also a list of the the actual songs that make up the list. Not very elegant, I agree.

I removed the playlists (they were a mere byproduct of playing around with iTunes anyway) and my CPU maxing problems disappeared.

Base | Lab | iTunes


Avoid USB with large libraries in iTunes

For as long as I have been using it, I have been having problems with iTunes stopping playback due to lack of CPU resources. I have long thought that I was just hammering the machine too hard and that it was annoying, but maybe just to be expected. Not anymore.

My music library resides on an external 250GB drive, with USB2.0 and FireWire connection options. Mainly because I keep the USB hub 'n cable spaghetti in a bunch at the back of my desk, I opted for the USB connection, because the FireWire port is located on the left side of the laptop, too close to the keyboard for comfort.

Just acting on a hunch, I decided that since I have been using a wireless USB keyboard anyway, as well as a PC-card connected CD-ROM drive sticking out on the same side, the comfort argument is not valid anymore and that it might be a good idea to switch the drive to FireWire.

The hypothesis is that the high CPU burn of iTunes and other applications has something to do with the fact that I have a ton of gadgets hanging of USB ports. It all works, but it eats cycles. since FireWire is a standalone bus system, it will need a lot less resources. I don't expect a big speed gain, other than what is achieved by not having the CPU clipping at 100%, but it is still worth a try.

A reboot later and I am now due to smack myself on the head for not figuring this out sooner. iTunes' CPU usage used to jump to 90-100% on almost every use of the interface, and then again at every change of song. It now barely gets over 35%. I managed to push it to 65% by having it scan the entire library for new songs.

Moral of the story: if you use iTunes and your collection is of a significant size (which seems to make almost all operations expensive for some reason), avoid USB connections to your storage media.

(note: Apple users can stop their sniggering in the back, now. I know you will have been using FireWire as a first choice anyway)

Base | Lab | MCSE | Servers


Disabled DHCP

Neither of these machines needs to be a DHCP server, since from their point of view, there already is one on the network.

In fact, the DHCP service is provided by the VMWare networking subsystem.

I removed the DHCP role from both boxes.

Base | Lab | MCSE | Servers


Oh, what a glorious sight...

Two virtual machines, running simultaneously, all boote dup in under five minutes. Pity they don't fit side by side on the host desktop, but it still is a magnificent sight; I can't help but thinking about the enourmous amount of complex software that has to work together just for these two windows to do what they are doing right now. (i.e. nothing much)

The NIC icons on both of the machines are constantly flashing; I can't wait to have them up and running like this and to slap a network sniffer inbetween them.

Base | Lab | MCSE | Servers


Small tweaks

Individually, the servers will perform less, but together they should be prevented from pushing the host overboard. So I tweaked the settings a little bit:

  • RAM was decreased to 196MB per machine. Why that number? Because I like it, that's why. :)
  • Both servers are now wired to use the same virtual CD-ROM drive. I assume those drives are configured to be read-only anway, so that should not be a problem in itself.
  • I synchronised the configurations: I disabled all the fancier features of VMWare, like snapshotting and drag-n-drop between the host and the guest.
  • Inside both guests, I decreased the desktop resolution to 800x600. Looks like crap, but I need to save on resources.

Base | Lab | MCSE | Servers


Two on one

After days of sslloowwllyy installing servers, I am now in a position to have them running side by side on the host machine. Fingers crossed...

Base | Lab | MCSE | Servers | Server02


OS Installation

With the newly created bootable ISO a a virtual CD-ROM drive, I kicked off the installation of this second server. Sure enough, things went by quicker than the installation of the first one, but that might just as well be because of not using the external hard drive anymore.

I busied myself with reading about hotfixes and slipstreaming and every now and then entering a few details for the installation process.

I used the same options as for Server01, other than (obviously) the server name.

On several occasions, I got the impression that all activity inside the virtual machine had stopped, only to 'wake up' again when I switched the focus back to the window. Also, I switched back on various occasions when the installation was waiting at a prompt. Not in the least, I was also downloading some fairly large files and playing music through iTunes. All in all, I think these were not ideal circumstances for a fast installation. Nevertheless, it took only a little over an hour and a half to complete, including the installation of VMWare tools. (every reboot takes around 10 minutes, so time does go by quickly)

Base | Lab | ISOs


I created a bootable ISO

This post is really only a placeholder for the result: I managed to create a bootable ISO file from the Windows Server 2003 installation disk, but it has been nothing short of an ordeal to get to this point. I will try to record my adventures at a later time, using the notes below (which you can ignore for the time being).

Nero Express will only take a boot image from a floppy drive, ISO buster can extract everything and then some, but can't create bootable images, mkisofs should be able to, but consistenly fails to respond to what I believe to be a corect command line, MagicISO would probably work painlessly if it would not be crippled to less than 300MB. Undisker did the trick, even though it claimed errors at the beginning of the process.

Base | Lab | MCSE | Servers | VMWare


Using an ISO image instead of a CD drive

Whilst I was on the VMWare site looking for the exact command line parameters to use in a custom shortcut for Server01, I came across the marvellous idea to create an ISO file from a CD-ROM and mount that ISO file as a virtual disk in a virtual machine.

That would allow me to do all kinds of install without having to even have the Windwos Server 2003 CD available, or even the drive.

What's more, the transfer of files at installation time is likely to go a lot faster than when reading from an external device.

Base | Lab | MCSE | Servers | Server01


Setting up the paging file

The situation is a bit weird in this case, but I think the basic idea still holds: putting the OS files and the Virtual Memory Paging File (commonly known as a swap file) on the same partition is asking for trouble, if only because of fragmentation.

At the very least, I always try to create a seperate partition to use for swap, if not an actual seperate disk. The maximum wisdom in these matters is to divide your swap requirement over as much bus capacity as possible, but avoiding otherwise busy areas. So: not on the same partition as the OS. If there are two physical disks, then one-third on the disk that also has the OS and the rest on the other disk. Never on two partitions on the same disk; that would just make the head move around like mad and slow things down all around.

In this case, everything actually resides on the same file system on the same disk in the Host OS, so the logic hardly applies. My only reasons for creating a seperate virtual disk and using that is that the fragmentation issue is still relevant and that it is more manageable, in case I decide to expand the disk setup in future.

Like real ones, virtual disks can be swapped in and out of a system or replaced with bigger or smaller ones, but doing the same with partitions requires much more difficult procedures and low-level fiddling with tools like PartitionMagic. All in all, I would just a much avoid having to that altogether.

In the end, I formatted the (very much overcapacitated) swap drive and moved the swap file from the OS drive onto the swap drive.

Base | Lab | MCSE | Servers | Server01


Disks moved to the host's hard drive

As outlined earlier, I have moved the virtual disks to the hard drive of the host machine, my laptop. The assumption was that that would make a difference in performance. Well ... let's just say that my instincts were right: the Virtual machine with Server 2003 installed has been tuned down to 'merely' 384MB of RAM and it still starts up three times faster than it did when its disks were physically on the external drive.

Base | Lab | iTunes


How to preserve library data when moving music files

Updated June 2007!

I think iTunes is great as a player, because it allows me to build all kinds of so-called 'Smart' playlists, using some easy rules that can be combined to wonderfully complex things, all just in order to come up with the right song to play next.

Problem is, this being Apple software, it has a tendency to give you the choice of:

  1. letting it control your life (shut up and be happy) or
  2. giving you absolutely nothing to work with if you decide to opt out of preconfigured bliss.

Specifically: if you allow iTunes to 'organise your music folder' (meaning: totally butcher any ideas that you might have yourself about what makes up a logical way of organising your music files) you are allowed to move the library (i.e. the folder(s) that hold the music files) around to another place on your drive, to other disks or even to a network drive.

The reverse is the actual problem: move a music file and iTunes will treat it as a new file, which obviously resets play counts and 'Last Played' dates.

Here are a few things I tried and found NOT to work:

  • Edit the 'iTunes Music Library.xml' file with your favourite text editor only to find it ignored first and overwritten second.
  • Delete the binary 'iTunes Music Library.itl' file and your entire library will vanish.
  • Import a previously exported or a modified XML file and the entries will be treated as new files.

I searched a lot of fora, but all relevant posts and discussions I could find are either OS X oriented, using AppleScript that (AFAIK) does not run on Windows or advising you to take the bliss option outlined above. I did read something about AppleScripts possibly working on Windows, but it would be confined to within iTunes itself, meaning that moving stuff around in the Windows file system is out of reach. Running AppleScripts that try to do things with the Finder (the OS X interface to the file system) are likely to do nothing if you're lucky and serious damage if you are not.

Admittedly by means of sheer luck on a what-does-this-button-do-mission I have stumbled on to a working solution by myself! Hurrah!

The trick is very simple: do not delete the 'itl' file, but rather damage it. To be sure, just empty it. Upon startup, iTunes will try to load its database and fail (because you have damaged it). In an effort to repair it, it will (re)import the XML file--including our precious play statistics. A message follows that the itl file has unfortunately become corrupt and all that could be done has been tried. It even saves a copy of your emptied file. :)

Given that the XML stores not only music data, but also the playlist definitions, you are good to go from this point on.

Please see below for a number of gotchas and observed side-effects.

Instructions for technically timid folk (on Windows!):

  1. Shut down iTunes
  2. Make a backup copy of these files:
    • iTunes Music Library.xml (on my system, it is located in My Documents/My Music/iTunes)
    • iTunes Music Library.itl (on my system, it is located in My Documents/My Music/iTunes)
    • If you have the disk space to spare and you are truly paranoid, backup your actual music files as well.
  3. Open the itl file in Notepad (or any other text editor, but not in Word or any other word processor).
  4. Select all (Ctrl+A) and hit Delete. You should now have an empty file; save it and close the text editor.
  5. In Explorer, verify that the file size is indeed now 0Kb. I haven't tried partially damaging the file, but this method is a sure-fire way.
  6. If you haven't already, do whatever you want to do to your music folder(s): move them all to a new drive, shuffle some of them around, etc.
  7. Open the xml file in your favourite text editor and modify the paths as they should be, reflecting the new location of your music files. Global search and replace comes to mind as a quick way of making these changes. Don't worry about mistakes--you did take a backup, did you?
  8. Save the file and fire up iTunes. Instead of starting normally, it will import the xml file. I have some 6500-odd entries in it and a dozen playlists. With the files on an external USB2.0 drive, it takes a few minutes for this to complete, which is indicated by the warning message about how iTunes discovered that your library file had gotten damaged and how it tried to save the day. Click OK.
  9. If all is well, you should now be looking at your library, with all data intact and the locations of the music files updated.

Some additional notes:

The location of files is dependant on your system defaults and any specific choices you may have made during installation of iTunes. In any case, you should be aware of the following:

  • The iTunes database files (the .itl and .xml files) are installed in your system's 'My Music' folder. Typically, this is a subfolder of 'My Documents'. Note that this location can be different for different users of the same computer. I do not know what happens in a situation where there is a 'Shared Documents' type of setup.
  • The music files (typically, your mp3 files) can be anywhere on your system. If you have let iTunes 'manage' your music library, it will have put (a copy of) those files in subfolders of the top-level library folder.
    This is where the confusion sets in: the default configuration is for the top-level library folder to be the same folder in which the database files reside. You CAN (easily) change the location of the library folder (in the preferences) but you CAN NOT change the location of the database files. At least, not from within iTunes and not without mucking about with profile defaults in Windows itself.
  • Changing the top-level library folder through the preferences only tell iTunes where to expect existing files and where to put new ones. That is, if you have configured it to manage your files for you. If you have not told it to manage your files, changing the preference does (I think) nothing for the music files.
  • Since the Album Art functionality was introduced, there is another tree of subfolders underneath the database folder, containting the image files for your albums. I honestly don't know how the location of those files changes (or not) if you change preferences. Just be aware that they are there, for the time being.
  • Moving the whole folder around (someone moved their entire 'My Music' folder to an external drive) confuses the hell out of iTunes. The actual program files (the software, rather than the data) are stored elsewhere (on Windows, 'C:\Program Files' by default). When starting iTunes, it can tell (from the Registry on Windows, in some similar way on OS X) where the database files are supposed to be located. The database files will point the way to the music files.
    At best, iTunes might be blunt and recreate the database files (empty!) at the place from where you have just moved them. Other weird side effects may occur. Bottom line: do NOT move the database files, but ONLY the music files themselves.

Some more notes and comments, partially inspired by emails I have received over the years:

  • Beware of some text editors that are unicode aware or have any other reason to stick (normally) invisible characters in the first few bytes of your files. People have reported crashes using my method and at least one person mentioned that UltraEdit had done just this. Removing the extra garbage from the text file using another editor solved the crashing.
  • If you have a large collection, NotePad is likely not up to the task. If you find that you can not open your XML with NotePad, try a text editor of heavier caliber, such as UltraEdit or LargeEdit. I personally use vim, which I suppose is akin to long range missiles in terms of caliber. :)
  • I have the setting enabled to normalise the volume of all tracks. Upon re-importing the xml, iTunes will 'Determine song volume' on all the songs. Depending on the size of your library, this might take a while, but it is otherwise harmless. Just let it do its thing.
  • Since iTunes started handling album art work, the setting for 'automatically downloading missing art work' might trigger a similar automatic process, taking another good bit of time to complete.
  • The same applies to the option for 'determining gapless playback'.
  • My method works on both Windows and OS X.
  • I have had confirmation of the method working on just about every version of iTunes up until the latest (at the time of writing this, June 2007). I have reason to believe that it will remain working for a while longer, because of the following: the xml file exist for a purpose, which is to allow Spotlight and other applications on the OS X platform access to iTunes' data. This is a cornerstone of cross-application functionality in OS X, on which a good many third party programs depend. I speculate that the existence of the XML file is pointless on Windows, but since iTunes is developed for the two platforms simultaneously, it has been overall easier for the development team to just leave it in. (rather than having to maintain two different architectures). Also, it seems to me that the 'repair' facility depends on the xml file to be present. My best guess is that as long as the cross-application API on OS X continues to use the XML mechanism, my method will keep working.

  • I have briefly looked into the way playlists are stored in the XML file. At first glance, it seems to be in the format of actually listing the IDs of tracks that belong to a particular playlist, along with copies of the metadata. This holds true even for dynamic playlists. When you change around some of the meta-data (such as the location of the file) these lists seem to get upset. I have had reports of playlists only partially surviving the process described above.
  • Podcast subscriptions are reported to break. I don't use them myself, so I can not give you any more specifics than that.
  • Someone (named Greg) mentioned that the 'Date added' data element will be reset, since the repairing is seen as re-importing. I have not verified this and actually have my doubts about it.
  • Someone (named Paul) moved an external drive from one machine to the next (I gather a new installation) and after installation dropped in his 'old' database files. Since all the paths to the music files were kept identical between installations, it worked straight away.
  • Be sure to exit out of iTunes cleanly at least once, so that it will rebuild the itl and xml files to be in sync again. I can't say what will happen if you don't, but based on past experience, I would say that only Bad Things can happen from that.
  • I have received feedback from a handful of happy users whose playlists have been saved from forced re-arrangement. If you use the tip above, do me a favour and send me a message at Oh, and mention iTunes or SchmolleWorld in the subject, so that you don't get caught in my spamfilter straight away. :)
  • I can't claim having done a proper test, but the timing of the feedback suggests that the above should work on at least the last version or two of iTunes. Specifics in your feedback are welcome.
  • From one or two fora I can deduce that this trick will work on Mac OS X. There is no itl file, though. Rather, an iTunes Library and an iTunes Music Library.xml file. You need to have you Finder set to exposing file extensions to see this, obviously.
  • In a recent version, Apple introduced this method. All good and well--if you let iTunes organise your library. So bummer about that one.

Base | Lab | MCSE | Servers | Server01


Networking up and running

Soon after trying to do the first thing that involved networking (connecting from the host to the guest using Remote Desktop, to be precise) I found out that networking was not operational.

The first thing that I did was to configure the VMWare networking subsystem. Sort of like a virtual machine provides a machine's hardware, there is a service installed that provides the machine's networking environment. This means a NIC inside the virtual machine, but also one or more NICs on the host. VMWare is able to act as a bridge, connecting the virtual 'internal' network to actual outside NICs, thus providing virtual machines with connectivity to the rest of the world, beyond the host.

I opted for host-only networking (which effectively disables outside access for the virtual machine; the concept is like plugging a cable into a router that is local to this small network.

To avoid confusion between the private IP range on my home network and the private IP range on the host-based network, I set the DHCP range to use the network.

Apparently, newly configured Windows 2003 servers don't even bother looking for a DHCP server, so I changed the TCP/IP properties to 'Obtain an IP address auomatically'. I have often wondered why Microsoft don't just call this option 'Use DHCP when available', which would be a much more accurate description of what it actually does. Oh, well, it works anyway.

With the NIC disabled and re-enabled, Server01 has now got an IP address that is actually on the network it is on and I can ping the box from the host. Strangely enough, I couldn't seem to do that the other way around at first. Then it hit me that I usually have Norton Internet Security running, which is (among other things) a firewall, which undoubtly blocks incoming ICMP messages.

I say undoubtly, because making sure is just too much work to find out. I reckon I won't be using that product for very much longer, especially when XP SP2 comes out with its vastly improved built-in personal firewall. NIS seems to be wanting to do the right thing (configure all applications exactly according to their needs and only allowing incoming traffic when there is a corresponding application running) but the interface is so incredibly clunky that I can't seem to deal with it. Also, it appears to have a very hard time remembering settings. But that is the topic of another rant, another time.

After disabling the firewall and bouncing the host NIC (to get the right IP address from the reconfigured DHCP server) I had connectivity in two directions and I was able to set up a Remote Desktop session.

Base | Lab | MCSE | Servers | Server01


I really have to move the virtual disks

This setup really is too slow to work with and I can not help but think that the external disk is the biggest culprit. So there is nothing left to do but move the virtual disks (2 times 2GB, although on the actual host file system they are much smaller, following usage inside the guest) to the laptop's hard drive.

That does present me with a problem that I had really wanted to avoid having to solve: moving my MP3s off that disk, so that enough space is cleared up. Moving them in itself is not that big a deal: I have them backed up on the external drive as it is anyway and 99.9% of the time that I am playing music it is at home, with the laptop hooked up to a HiFi system anyway.

The only problem is that in my past experience and from what I have read on various fora, iTunes will do anything in its power that it can to treat moved files as new ones, thus losing play count and 'last played' dates. I have an intricate system of 'Smart Playlists' in use that depend on these two data items heavily and it has taken me literally months to get the whole Library sufficiently randomised for it to work--hardly something I want to give up easily. Time for a sub-project...

Base | Lab | MCSE | Servers | Server01


Domain setup

After restarting from the VMWare tools installation, at least the screen looks decent and the mouse pointer is responsive. Other than that, it is a very slow affair. It looks like I will have to shut down most or all of the activity on the host machine, in order to conserve CPU resource. This ought to be fun when I have two or even three of these running. :)

As per instructions, I create the domain name, which automatically sets the NetBIOS name of this box to CONTOSO. I let the install do whatever it feels like next, which seems to include a reboot. See you a bit later, then...

Base | Lab | MCSE | Servers | Server01


Installing VMWare tools

This is so nifty: it used to be that you had to do some tricky stuff with the interface and the OS, which would replace your floppy drive temporarily with a built-in virtual drive. All right, but it did require some manual actions on behalf of the user.

Now, it is just a menu option in the interface. A drive will appear within Windows and it will autorun the setup. I opt for the custom option, so that I can opt out of the SCSI acceleration (don't need it) and the host-guest file sharing option (don't want it, since I will be doing all kinds of network stuff on these boxes and I do not need surplus confusion).

Installing is a snap; a reboot is advised, so I do that.

Base | Lab | MCSE | Servers | Server01


first logon

Sure enough, the box rebooted by itself and I was able to log on. Now for the few initial pieces of setup and I will have sucessfully installed a Windows Server 2003 Enterprise Edition machine. :)

The OS rightly complains about a very low resolution and color setting. It offers to correct it automatically and I take the offer. Neverthelss, I should install VMWare tools first.

Base | Lab | MCSE | Servers | Server01


OS basic configuration

As per instructions, I configure the OS for the right keyboard and time zone. I also leave the network settings to 'not belonging in a domain, but part of workgroup 'WORKGROUP'.

I supply a password for the administrator account. The book says it will accept either very good paswords or total blanks. I tried something inbetween and sure enough, the thing complained about it. It did give me the option to proceed anyway, which might mean that the book is in error. I'll try that another time, for now I have just put in an acceptable password.

The installation continues with copying files and configuring the installation. It seems to take long times without any obvious activity, barring the small squares in the bottom right corner that light up in sequence. The timer claims something like 27 minutes to go, but the underlying algorithm does not seem to be very informed about the relation between time passed and the time needed to actually perform the tasks ahead.

Nevertheless, it is all sailing smoothly ahead.

Base | Lab | MCSE | Servers | Server01


OS: Initial setup

As expected, with the Windows Server 2003 Enterprise Edition CD in the external drive, the Virtual Machine happily boots up and starts the setup procedure from CD.

Unlike I expected, the initial file transfer is taking a very long time. In fact, I have done more than a few installations of Windows 2000 from the four-floppy disk set that you can create, and in my recollection, that went through noticeably faster.

Maybe the problem is bus contention: the disk on which the virtual disks are stored is an external drive on a USB2.0 connection. fast enough for continuous drive oprations, but it eats system resources when multiple processes are accessing files, because unlike with an IDE controller, the OS must do a lot of work to manage the USB traffic.

I can't say for sure without doing the whole thing twice with a stopwatch, but killing eMule (which has dozens of files open on that same drive) seems to speed things up a little bit. Still darn slow, though.

As per the instructions in the book, modified for my situation, I configure the drives and partitions. I am leaving the swap partition unformatted for now; that can be done in Disk Administrator at a later time. Besides, there is no obvious option at this point to format both partitions.

Base | Lab | MCSE | Servers | Server01


Hardware parameters

I want to have a seperate disk to use as a swap drive for my virtual servers. That makes the total set of parameters:

  • 788MB of RAM. I have 1GB in the host machine, so it is available. At the time when I will be running more than one server, it is easily enough decreased. Until that time, more RAM makes the VM a lot faster.
  • The primary hard drive (master on IDE0) is a 2GB drive. The drive intended for swap is also 2GB, on IDE1.
  • Networking is limited to the 'host-only' option, because even though there is hardly anything to damage on my network, I want to limit the networked exposure these servers get. It isn't too difficult to accidentally hook them up with a live Internet connection and these things are not patched at all. Given the time constraints that I am working under, a heavy worm or virus infection is just too much fun to be dealing with.
    I am assuming that the host-only option means that two virtual machines running on the same host will be able to communicate with each other. If not, settings are changed easily enough. (why people bother with real hardware, I don't know... :)

Base | Lab | MCSE | Servers | Server01


Starting installation

WIth everything I need downloaded, it is time to start the installation. A tad later than I had hoped (I had hoped for 10:00 am this morning :) but the night is still young and this process is going to take a fairly long time of waiting. It should blend in with the rest of my evening just fine.

Base | Lab | MCSE | Servers | VMWare


No, wait, hang on

Rather than wait for a web master to get back to me, I figured I might as well have a browser window babysit the site that I am interested in. So I called up the page again and whilst it was still loading (or not, rather) I set an autoreload on the tab.

Contrary to my expectations, the page loaded, although it surely took its time. I proceeded to download the binary, cancel the download in eMule and sent a message to the web master to never mind about full outage.

Base | Lab | MCSE | Servers | VMWare


Alternative means

Normally, I shy away from this kind of thing, but given the circumstances I will forgive myself.

Seeing as I can't seem to reach the download section of the VMWare site, I have looked for a copy of the binaries on the eDonkey network. As luck would have it, it also had me stumble onto a key generator. That will save me from having to register for an eval key with no intention of buying the product. Actually, it saves VMWare more time than me. I promise (as I have done before---and made good upon) to buy a licensed copy as soon as either I can afford it or I can get an employer to fork out for it.

Base | Lab | MCSE | Servers | VMWare


Getting a copy of the binary

Simple enough: head over to and find the binary download section.

No such luck: for some reason, I could not get to the actual page to download a binary copy. In fact, it seems like everythign that has anything to do with the 'vmwarestore' just is not responding. It is not completely dead either: FireFox would have times out much earlier if there would have been an actual network problem. It seems like the web server is just keeping the client hanging on.

I confirmed my findings in several ways and killed off other programs that might be choking the network on my end (such as eMule, which has a habit of exhausting network sockets). Other web sites seem to be fine, though. As is, in fact, most of the VMWare site itself. I have sent an email to the web master to alert them of this problem.

Base | Lab | MCSE | Servers | VMWare



The initial idea I had was that I would blow the dust off the Linux box in the server cabinet in the attic and create myself some Virtual Machines for this purpose; I still have a copy of VMWare (properly licensed and all) on that machine, from way back when.

There were some problems with that plan, though:

  • Although it has 748-something MB of RAM, the CPU is a lowly 800MHz P3. (at best, it might even be a lesser spec).
  • The copy of VMWare is of the 1.x or 2.x variety; not likely to support Windows Server 2003.
  • The disks in that machine do not have a great deal of space available, because I have been dumping all kinds of stuff on there for ages. Nothing that could not be cleaned out or archived to CDs or something, but that would still be several days' worth of sifting through files.

Then it hit me: my primary workstation is a laptop with a 1.6MHz Centrino and a whopping 1GB of RAM. Also, I have a 250GB external harddrive attached. Disk performance over USB2.0 will not be fantastic (maybe I shouls switch it to FireWire to see if that makes a big difference) but with that amount of RAM, it is likely to be allright. Better, in any case, than anything I could possibly do on that Linux box.

What's more: I can download an eval copy of VMWare 4.5.1 and be assured of the latest and greatest in features and support.

Base | Lab | MCSE | Servers


Knowledge first, practice later

Thinking about the trade-off between spending lots of time now on getting a hardware platform up and running and continue working through the books, leaving the hands-on exercises until later, I am leaning heavily towards the latter option.

I will start building the environment on Wednesday afternoon, a time when studying quietly is not an option anyway on account of the cleaning lady and the vacuumer.