Unlike Windows or OS X, there are no automatic updates for our website when a security problem is found in the software we use; it’s up to me to do the updates. Because we’re pretty busy, I let this slide for about 6 months, long enough for a security vulnerability to be discovered in Gallery, the software we use to share our photos.

The software is fixed now and I’m in the process of re-doing all of the customizations we had, hopefully I’ll finish it this week.

People often ask how this works so I thought I’d explain how I do my job. Briefly my job is to develop a part of the next version of Windows. A few things make my telecommute possible. None of these are unique to Microsoft, perhaps you’ll find some tips in here you can use to figure out how you could work from home (no matter how far from the office that is) and convince your employer that it will actually work.

Good internet access
We have the fastest internet connection I could get in our neighborhood, which turned out to be 10mb ADSL. Considering our neighborhood is at least 1500 years old this seems pretty good. The bandwidth is sufficient for me to be able to maintain an up-to-date copy of the source code for Windows on my home computer, which I use to create a new test version of Windows every day or two. Typically this means transferring about 2gb of data each day.

I connect to the internal Microsoft network using the normal buzzwords – a VPN that requires two factor authentication and IPSEC.

Good internet access these days also means really cheap international phone calls via VOIP, I use the phone (in addition to email) to communicate with my teammates.

Fast and inexpensive PC’s
Windows is not a small software project (according to these guys there are 50 million lines of code in Windows), to be able to compile the source code requires either a lot of time or a very fast PC. Most of my teammates in Redmond only compile the part of the project they work on and get a copy of the rest of it from one of our file servers; even with a good internet connection that isn’t feasible for me. A complete copy of Windows suitable for development is about 25gb of files – this includes files to make debugging easier in addition to other files to help with testing. In short a huge amount of data to try to download over an ADSL connection. I compile my own copy of Windows instead, using a fast but surprising inexpensive PC.

For the curious, I currently use a Dell Dimension 9200, with an Intel Core 2 Quad cpu, 4gb of memory and 4 10krpm Raptor drives, 3 of these are arranged as a stripe set (aka RAID 0) to make the disks perform even faster. The whole setup cost about $1500. From start to finish it takes just under 12 hours for me to produce a new installable version of Windows.

I also have PC’s at home that I use to test my new versions of Windows.

Remote controlled PC’s
I have a set of PC’s in a datacenter in Redmond, they allow me to share data with my teammates and minimize the data I need to transfer over the network – I don’t want to send 25gb of data to Redmond any more than I want to download it. On any given day the size of the changes to the source code I have are on the order of 100kb; I transfer the small source changes and compile them in Redmond when I need to share my changes with other team members.

I access my machines remotely using Terminal Services for Windows for most of my work. I use some of the machines for testing new versions of Windows, so I install new versions of Windows on PC’s in Redmond; this is tricky because installing a new version of Windows at times involves needing to interact with the BIOS of the PC. I use an IP KVM which allows me to control the PC even when it is in BIOS. And finally, sometimes things go really wrong (especially when I’m testing new changes) and the PC hangs or blue-screens. I use a remote controlled power strip so that I can turn the PC on and off remotely to reset it.

There is also a lab staff that I can call upon if things are really messed up and I need someone to physically do something to one of my PC’s. That is rare, however I have had one machine catch on fire before and I wasn’t able to deal with that one myself from Spain.

Good management
My manager and his manager (and his manager and his manager too) and I have all worked together for a long time; we have a solid foundation of trust and expectations and how to communicate those to each other when they are (or aren’t) being met. Technology aside, I have found this to be essential.

I visit the offices in Redmond two times a year for a week or two at a time, this helps tremendously in terms of keeping my network of colleagues strong and allows me to meet new team members too.

We still use the same system I described, but have added another backup storage to it, this time on the web.  By good fortune the same people that host our website also allow us a huge amount of storage, it increases each month we are a customer.  At this point we have over 500mb of storage available, enough to consider backing up our data to their servers too.  

There is a great unix tool for web backups, in fact robocopy that I mentioned before is a simpler version of it.  The tool is rsync, and what it does is incrementally mirror one folder to another, allowing it to be very efficient and only upload things that have changed to our server.  I use it on my Mac which comes with rsync, if you have use Windows you need to get a package of Unix tools and install it, I like Cygwin, its free.

We back up our data to an account on our web server, but that account is not web-accessible, eg you can’t reach it via http or ftp, meaning that while it is stored on a server on the internet its not generally accessible to anyone but us; I prefer that but its also possible to make it web-accessible if you want, but I wouldn’t recommend it if you have a choice.

Backing up becomes simple; from your command prompt, change directories to the place you want to back up, in my case the parent directory that contains my photos directory.  I then do this (substitute the my’s for your account and host) to put the files in the photos directory on our server.  Note that the trailing /’s on the folder names are important.  I get prompted to enter the account password on my server and then the upload begins.  This uses a secure (encrypted) channel to upload the data.

rsync -e ssh -av  photos/ my-account-name@my-web-host.com:photos/

We have a lot of data to upload to our server so it never gets there in one upload session.  No matter, rsync as I said only uploads things that have changed, so when we restart an upload I use the same command and it automatically skips the files it already uploaded and picks up where it left off from the last upload.

I won’t be installing the next software update on my Mac until a few weeks have passed, next time someone else can waste a day if it goes wrong.

We returned home and as seems normal these days all of our computers wanted to install the latest updates that had been released during the 2 weeks we were gone.  I do most of my personal work (including writing this stuff) on a Mac mini, the majority of our computers run some form on Windows.  

The Windows updates went fine of course (disclaimer: I work for Microsoft and help write Windows), but sadly the Mac was rendered unbootable by the “update” to OS X 10.4.11.  It looks like I wasn’t alone – http://apple.slashdot.org/article.pl?sid=07/11/27/0040206 talks a bit about it but focuses on bootcamp, sadly that’s a red herring as I don’t have bootcamp.  But, as Apple reminds me with smug taglines like “Think Different,” using and owning a Mac put me in an elite club of people who are in the know.  So I thought different and it helped; while I wasted the better part of a day copying, imaging, installing and trying to save all my data and make the Mac boot again I realized a lot of people were having a much different day, probably even having a fun.  But they don’t own a Mac.

Caution/Cuidado, random boring technical junk to follow.  I thought I’d give a brain-dump of the system we use at our house, perhaps it is something you can use too, or, even better, I’ve missed something and you’ll tell me how to fix it before another one of our hard drives crashes.

We have a central server in our house with a RAID-5 array of 3 400GB hard drives, giving us roughly 800GB of space.  The only catch is you have to monitor this, its designed to keep working even if a drive fails and it does just that without a hitch – until a second drive fails, then the whole array needs to be recovered.  Our server is running Windows Server 2003, which has a neat feature known as “Timewarp” or “Volume Snapshots”.  This automatically saves versions of files for later recovery in a very space-efficient way.  Eg. if we accidentally delete a photo or otherwise corrupt it we can revert back to the older version from any PC in the house without having to find the copy on our backup drive.

We back up our server onto removable hard drives.  I use a generic external 1394 drive case which I plug various drives into when I need to do a backup.  My backup software is simple, a tool called “Robocopy” in the Windows Resource Kit, you can get it free here.  It has a mode of copying that will mirror one directory to another, perfect for backing up data.  

One key of our backup system is that it is on external hard drives and they are not connected to the server except when doing a backup.  This helps protect from virus’s and power spikes.  We have multiple backup drives and not all of them are stored in our home in case of a real disaster.  

The other key is the data is stored on the backup drives in a plain format – when you plug one of our backups into any other PC you can immediately access the data without needing special restore software.

We lose 1-2 hard drives per year (we have a lot of hard drives).  Expecting it to happen and practice with it happening rather than dreading it has kept us safe from data loss for a long time, I hope it continues.