This post is about my thoughts and experiences as a software developer, leading up to my use of Linux and building my first non-Microsoft based Web application.

In The Begining

I’m a non-CompSci grad – having started my academic life with a college diploma in biology (a Canadian community college is the equivalent to what would have been called a ‘polytechnic’ in the UK). After about four years of working as a laboratory assistant, I decided that biology wasn’t for me, and moved into IT.

I started the hard-way – in PC support, and slowly worked my way up from there. I had a couple of lucky breaks along the way, which combined with a lot of hard work, meant that I was eventually able to pass for a ‘nearly’ competent IT professional (whatever that means).

After several years of ICT related employment (and having overseen a couple of medium sized software projects), I decided that I needed to go back to the coalface, and learn more about software development. I started by writing desktop applications in VB6, followed by C# and ASP.NET. I also sat the required exams for the Microsoft MCSD and MCDBA certifications (.NET versions).

I was firmly in the Microsoft camp. In fact even before my switch from biology to IT, I’d been building XT compatible clones – feeding DOS and Windows floppies into the disk drives of my home-built machines. It all seemed like good fun at the time. Later – having earned some certs, and with a few projects under my belt – I thought I was doing all right, earning okay money on a couple of contracts, and thinking things were generally headed in the right direction.

Ch-Ch-Ch-Ch-Changes

But then two things happened. The first was that Microsoft unleashed a wave of frameworks and technology – part of what was then called WinFX, including Longhorn, WPF (Avalon), WCF (Indigo), WWF (its workflow offering). The second was that a few years later I decided to go back to school.

I remember thinking at the time that by going back to school I was simply going to rack-up a decent qualification – something that would help me to progress, and maybe even help prepare for the day when I would hang-up my keyboard and move back into management (or something else altogether). What I hadn’t anticipated, was how hard it was going to be to study as well as stay on top of what was happening in the Microsoft world. Things were changing fast, and I was struggling to keep up. Like some of my contemporaries, I was also trying to stay abreast of the broader trends in software development. I’d started reading some of Fowler’s stuff. And I was reading a lot about the challenges of managing software development projects – challenges that are unique to building systems that reach out and touch all aspects of an organization. I’d read ‘GoF’, Kent Beck, Eric Evans and others. I watched the TDD, BDD, and Agile zealots rise to glory. I was also following the raging WS-* debate, and beginning to understand the significance of Roy Fielding’s work.

I was beginning to acquire a deeper understanding of the software development process as a whole – in particular, an appreciation that the major challenges for most projects are not technology related. To this day the two most important texts I’ve read on the topic are The Mythical Man Month by Frederick P. Books, and PeopleWare by Tom DeMarco. Incredible when you think that The Mythical Man Month was first published in 1975. Read them both, and you’ll understand why they’re important in the field of software and systems.

Somewhere between studying, and trying to keep up with things in general, I began to experiment with Linux. I’d just started a course on computer security – which included Linux and Windows security. Thanks to virtualization technology, spinning-up a desktop Linux VM was a breeze; although apart from using the VMs for a couple of exercises on my course, not at lot happened at this point. I was still a bigoted Microsoft developer, convinced that C# was a ‘real’ developer’s language, and that Linux, Apache, MySQL and PHP were for kids. Actually that’s a little harsh. I wasn’t really bigoted, at least not in the literal sense – but humans are a tribal bunch, and we like to stick to what we know. Often when our beliefs in what we know are challenged, we resort to derision and are dismissive of alternative points of view.

Things began to change when my computer security course started looking at historical operating systems, combined with some basic principles of machine organization and computing. It was then that I realized my entire frame of reference in computing had been shaped by the tools, messages and strategy coming from Redmond. It was a myopic point of view. I discovered that a lot of the messages coming out of Microsoft were simply re-hashed versions that had originated elsewhere, and were designed to keep people on its platform. When Microsoft decided it was time to push a point of view that suited its technology, it did; and as far as message delivery goes – it excels at it. Remember the ‘impedance mismatch’ between general purpose programming languages and SQL? That was the message de jour when LINQ was released. Why not earlier? It was weird to see so many Microsofties ‘on message’ at the time.

Of course Redmond wasn’t the only thing influencing my cognitive map of software development. My ‘up-to-date’ view of what was going on in the development community was shaped largely by the list of RSS blog feeds I’d subscribed to. I learned a veritable ton from excellent bloggers and technical writers. But I was also becoming suspicious of a smaller group of writers whose mandate was clearly one of self-promotion, and who had little interest in helping the development community at large. Whether this is unique to the Microsoft development community or not – I was getting tired of the BS, troll-baiting, and general angst and confusion that seemed to be the result of mixed messages from the so called ‘thought leaders’ in the community. Rob Conery made a comment at one point that (paraphrased) the level angst and confusion surrounding data access – “is just weird” – and he was right. Remember the open letter to Microsoft – ‘ADO .NET Entity Framework Vote of No Confidence’? What better example of the level of hyperbole that exists between Microsoft and its community developers (acknowledging that the intent of the letter was to try and help the tool evolve). And then there was Scott Bellware.

Another important event in my transition from a Microsoft-only approach was when I attended my first Barcamp developers’ conference here in Bangkok. I was impressed with the energy and enthusiasm shown by the small army of open source software developers I’d met there. I wasn’t quite ready to put on my Richard Stallman acolyte’s robe (it turns out not many are, but that’s a different story) – but there was some very cool stuff going on. I also couldn’t help but notice that many of the successful Microsoft open source projects, were in fact ports of successful projects that had come from other languages and platforms; like Lucene, Hibernate, and Velocity to name a few.

The tipping point came when I was involved in a large .NET project that can only be described as the quintessential death-march. Death-march projects aren’t unique to users of Microsoft technology – but this one happened to be using those tools. And no one in the organization had heard of Richard Stallman, or REST, or had even the slightest interest in Agile; not that those alone are the basis for establishing the maturity of a software development team, but it was an indication that the team, like a lot of other teams, had been living in the Microsoft echo-chamber for too long. For me personally – it was hard – really hard. And most of what was hard had nothing to do with writing code. There were problems in requirements, which had led to classical problems of oversight and late realizations – the stuff of any difficult project. But what I found interesting in hindsight was the level of friction in tools, frameworks, testing and deployment – all of which conspired to fight against each other, creating an exhausting drag on the project.

Lessons Learned

After a bruising project, it was time to complete my studies and write my thesis. I’d sworn privately, that I would never allow myself to become worn-down or as affected by any project ever again – even if it meant a fundamental change in direction (although not quite ready for 410 Gone).  Fortunately, my next role was a fairly low-stress part-time engagement, which allowed me to devote the rest of my time to writing my final report. What transpired over the course of those happy nine months was that I built and deployed a Linux-based production server, and wrote my first Web application based on an open source framework.

And after all of that, this is what I’ve learned:

First, Linux, Apache, PostgreSQL and PHP are not (just) for kids. Linux has turned out to be the lion of the Internet. Look at the latest Netcraft Webserver survey – 65% of the Web is powered by the Apache HTTP Server. Back in the late 90s and for most of the 2000s (and while Microsoft was still mucking around with IIS4, 5 and 6), the Apache HTTP Server community was benefiting from a world class webserver (with a real url re-write module) along with all of the other tools and technology that fit together to form seamless and frictionless development and deployment environments (before Microsoft developers even began talking about ‘friction’ in their projects). I’ve come to realize that what’s happening now in the *nix world is simply too big to ignore, and I don’t think I would work for, or consider hiring, anyone that didn’t possess at least a rudimentary understating of this fact.

Second, I mentioned above that the major challenges for most projects are not technology related. In one of the first texts I’d read by Fowler, he describes three fundamental risks to any project: 1) requirements risk, 2) skills risk, and 3) technology risk. I think requirements and skills risks conspire to create the most trouble for any project, but technology is still important. Successful projects can be, and are, run using Microsoft technology, but remember this: Microsoft’s only objective is to sell software licenses, and to do what ever is required to continue to sell software licenses. And that is all. It’s a company that has been arguably over-compensated for decades from a license-based revenue stream that is the result of its dominance in the desktop computer market. Nor has Microsoft been shy about using its position of dominance to destroy competition, and in the process, innovation. Its attempt to ‘create’ a developer community is a top-down effort and is the antithesis of how communities should be formed. The developer division in Redmond still largely decides what the strategy will be for the development of Microsoft-based frameworks and technology. When compared to the level of activity and innovation in other communities, it might be reasonable to describe a dependency on the Microsoft platform as a technological risk – both in terms of vendor lock-in, and in terms of the maturity of the development community as a whole; and that these risks should be assessed as with any other risk to a project. If a project is at a stage where it can freely choose its underlying technology, then I believe there are healthier tools and communities in the non-Microsoft world. Two great cases in point are the Ruby and Node.js communities on GitHub (and elsewhere). Looking at the speed of development and contributions to either of these communities, and it’s clear that while feathers might still be ruffled – good projects and tools quickly rise to the top, while mediocre projects fall to the side. It’s fascinating to watch.

Third, anecdotal evidence suggests that as many as 70% of all software projects fail to meet their stated objectives. It’s not a fun industry to work in, and death-march projects will continue to exist in all of the major technological camps. The success of any project will be determined by a subtle combination of maturity, experience and insight along with realistic estimates of the ‘true’ cost of creating or implementing software based solutions. In theory, the up-front analysis of a project is meant to be ‘technology agnostic’. In practice, most shops are in one technological camp or another. I think at this point however, at least one indicator of the likely maturity and experience of a potential supplier, or team – is their genuine willingness to consider a range of technologies and solutions including open source platforms and tools like Linux, PostgreSQL, PHP, Python and Ruby to name a few.

Fourth, I wish I’d started my career in software development at the ‘web-face’, maybe as a Perl/CGI scripter.

Over the coming months I’ll be publishing a series of blog posts on how I got started with Linux, Vim, and a bunch of other tools that, for me at least, made computing fun again.

Stay tuned…

I think the decision not to include NFC technology in the iPhone 4S was based on two factors.

The first, was that Apple was likely catering to the wishes of the two major US mobile carriers – AT&T and Verizon.  NFC technology punches another large hole in the influence mobile operators will have over handset technology and services. And almost all of the major carriers around the world have tried at some point to leverage their network and account/payment/billing infrastructure in order to tie customers to their business. Remember when you were going to be able to buy a coke from a vending machine by just entering your mobile number? And then there was M-Pay in the UK. Most schemes have failed – but the mobile operators are still trying. And they’ve had a huge influence on handset design as a result. Verizon refused to sell handsets that had WiFi initially. What they want is for everything to go through their network – and ideally their billing infrastructure too. But they stink at it. WiFi allows handset owners to bypass the mobile network, and so will NFC. So this is probably a last gasp by the mobile operators in the States to try and continue to own the ‘smart-card’ (aka SIM card) and see if they can come up with an alternative scheme using existing hardware/infrastructure in the meantime.

The second reason is that Apple probably don’t mind having a “significant feature” to add to their upgrade path.

Now that I’m in the clear, I’m able to publish the report online – which can be found here at the following link - 3-D Secure: A critical review of 3-D Secure and its effectiveness in preventing card not present fraud. It can be read as HTML, or downloaded in PDF form (with larger figures and illustrations) from here. I think the executive summary, introduction and conclusion read fairly well, and might be of general interest.

In another life – I’d love to spend more time on human computer interaction for security related tasks (HCI-Sec). It’s a fascinating field and I found some really interesting material on the topic.

I also wanted to say thank you to all of those who participated in the survey; in particular to Paul Stack (@stack72 ) and Craig Murphy (@CAMURPHY) for helping to drum-up participants.

Enjoy….

1. Quicksilver: This application launcher is the business. It’s been around for ages, and does what it says on the tin. It looks a lot like ‘Gnome Do’, and I love it. A very fast, and very convenient way to launch apps, scripts, open documents and a lot more. It’s also free. An alternative to Quicksilver is Alfred – which also looks nice – although I’ve yet to take it for a spin.
2. Pathfinder from Cocoatech: Cocoatech calls this “The Finder, reinvented” – and it’s nothing less than that. For me – it takes all the pain out of Finder, and creates a file and folder management shell that reminds me of Total Commander for Windows. Having folders sort before files should be part of Mac OS X out-of-the-box – but for some strange reason it isn’t. Pathfinder takes care of that nicely. There’s also an excellent split window view, preview view, and options to override all of the default key-binding behaviors – including ‘Enter to Launch’ as opposed to renaming a file or folder. Most importantly for hi-res displays – all the standard View Options are available, allowing you to change font and icon size to suit. I can barely use a Mac without it and it’s well worth the price of admission.
3. Little Snitch: This is a great utility that tells you when any software installed on your Mac is trying to ‘call home’. It’s the opposite of a Firewall. Instead of preventing stuff from ‘coming in’ to your computer – Little Snitch will tell you whenever an app is trying to ‘get out’ – and allow you to decide if you want to allow it to or not. The network monitor that comes with Little Snitch is brilliant. If you’ve ever wondered what was causing all of your network activity, when you weren’t actually using your computer, Little Snitch will tell you.
4. iStats Menu from bjango: Want to keep an eye on memory usage, disk activity, cpu activity and system or cpu temperature? iStats Menu will do that nicely thanks. I discovered iStats menu after looking for a simple hard disk activity light. I wanted to know when my disk was idle, or reading, or writing (and potentially thrashing). Activity monitor does a good job – but iStats Menu had so many nice features and extras packed into the app that I forked-out for it. As a bonus – I really like their menu bar clock, and additional world clocks that can be added to the drop-down. Another utility app that’s well worth the price of admission.
5. Tunnelblick: For road-warriors that need access to their small office, or corporate virtual private networks (VPNs) Tunnelblick will do nicely. Tunnelblick is an OpenVPN client. It’s super stable, easy to set-up and works great. The only catch of course is that your office or corporate VPN must be an OpenVPN SSL based VPN (as opposed to say an IPSec VPN). Most modern day firewalls support OpenVPN and so you may be using an OpenVPN without knowing it. Contact your IT or systems support people to find-out, and recommend Tunnelblick to them if you’re using an OpenVPN.
6. TrueCrypt: Everyone should have some sort of personal information security strategy – although most people don’t, or are relying on paid-for utility apps and 3rd party browser plugins. I couldn’t do the topic justice in a short post like this suffice to say that it’s kind of important to know where you are keeping your sensitive information like bank account details, web site logins and passwords, software registration details and serial numbers etc. Mac OS X ships with FileVault and this is likely to be a practical solution for full disk encryption on a Mac – although I’ve not yet tried it. I like TrueCrypt a lot because it allows me to create a single ‘confidential’ volume that I can take with me, and mount on any system that supports TrueCrypt - and that means Windows, Linux, and Mac OS X. The only downside to TrueCrypt, is that it’s not quite as user friendly as I’d like it to be on the Mac. Before you can install it you need something called FUSE. To quote WikiPedia –  ”Filesystem in Userspace (FUSE) is a loadable kernel module for Unix-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code.” OK – so you don’t really have to know what that means – suffice to say that you need FUSE installed before you can install TrueCrypt. And MacFUSE will do just the trick.
7. ClamXav: This might be a somewhat risky recommendation – but I don’t want any anti-virus software running on my computer.  By running, I mean any real-time monitoring or background processing that will take up CPU cycles, or interfere with the general performance of my computer. What I want is an on-demand anti-virus scanner that I can use whenever anyone hands me a disk, or when I download a file from the internet. ClamXav lets me do just that. ClamXav is a graphical front-end to the OpenSource ClamAV virus scanning system that is used on many mail gateways and servers on the Internet. I love ClamXav‘s interface – it’s simple to use, and also does what it says on the tin. If you do want real-time file system monitoring, ClamXav has an option to turn on sentry monitor – which will do that for you.
8. AppCleaner: Mac OS X doesn’t have a default ‘application uninstall’ subsystem, nor do many apps that you install by dragging into the Applications folder have an uninstall option (which is a little weird). There are a couple of application removal tools. AppCleaner which is free, and appears to do the job. There’s also AppZapper which looks good as well, but is not free.
9. SuperDuper: Two popular file based backup utilities for Mac OS X are SuperDuper and Carbon Copy Cloner. I was tempted to use my own rsync script – but SuperDuper was nice and clean, simple to use, allows for pre- and post-processing scripts – and will make the sync’d external backup disk bootable as well.

I’m also using Vim – which I have to say, I’m liking a lot (my dotfiles and Vim settings, plugins etc. are here). I’m also trying to synchronise settings between each machine, ideally using the same terminal and font settings. On Linux this has been a challenge since Gnome and KDE handle fonts a little differently. Gnome has also made the Tango project and theme its default for the Gnome Terminal emulator. This is a great colour theme that softens system colours using a more ‘pastel’ palette. However in order to get this to work under KDE you need to download the following colorscheme. Save it as Tango.colorscheme and place it in your ~/.kde4/share/apps/konsole/ directory. It will then show up under the Settings –> Configure Profiles –> Appearance tab for Konsole.

A matching terminal colour palette combined with my custom Vim colour scheme (daring-dark.vim) – and I was close to having matching systems. The trick was finding a matching font. Monaco was pretty good under Ubuntu – but it was broken under openSUSE (even the Linux version).

And then I discovered Bitstream Vera Sans Mono via the Gnome project. A brilliant font and perfect rendering under Gnome Terminal, Konsole, and even the Terminator! (which is an awesome cross-platform terminal emulator).

And so there it is. I still think Monaco looks just a tad better in Visual Studio – but Vera Sans Mono on Linux with the Tango colour theme, and my custom Vim colours – is gorgeous.

I’ve been doing some research on storage options for a media company who are in desperate need of an easy-to-use, safe and large capacity storage solution.

I spent the best part of a week looking at various offerings, when it became clear that LTO and LTFS were going to hit the sweet spot.

The LTO specification is currently at LTO-5 – which provides 1.5 TB of uncompressed storage to the linear tape format and data transfer speeds of up to 140MB per second. Impressive. That’s about 500GB per hour.

But here’s the good part.

Until now – all tape backup systems required their own non-standard and proprietary index in order to catalogue and locate data on a tape. So for example once you start using Symantec’s Backup Exec – or anyone else’s software – you’re stuck using it until you’re ready to rotate your backup stock. Even if the data is in tar format – you still need to know ‘where’ the tar files are located on the tape. With archive material, there’s a risk that years from now your data may be on a tape that requires software that might not exist, or that you’re going to have to pay for in order to access data that belongs to you.

Enter LTFS.

With LTO-5 a new open file system has been introduced, which means that a tape that has been backed up on a LTFS compliant system – can be read and restored from another. Great.

What’s more – IBM, and HP (to start) have released freely available LTFS software for LTO-5 tape drives for Mac OS X and Linux. Even better news for Mac OS X users since support for enterprise class backup software on Mac OS X is thin (with some specialty solutions from people like BRU-PE and the very expensive PresStore system from Archiware).

The initial releases of LTFS doesn’t replace an enterprise class backup and archive system – however it does allow smaller companies to begin using LTO 5 LTFS now – knowing that their assets are safe in a long shelf-life and open standard.

Win win for LTO-5.

For the last few days I’ve been unable to sync my Outlook 2007 Contacts with my iPhone – and it was driving me nuts.

I tried reinstalling iTunes 9.1, tried the Edit –> Preferences –> Devices – Reset Sync History option. I tried checking all my recurring dates in Outlook – since this was reported in the past as a problem.

In the end – the answer was to roll back to iTunes 9.03 – by doing the following.

As per http://support.apple.com/kb/HT1923 …

Use the Control Panel to uninstall iTunes and related software components in the following order:

1. iTunes
2. QuickTime
3. Apple Software Update
4. Apple Mobile Device Support
5. Bonjour
6. Apple Application Support (iTunes 9 or later)

I then deleted ALL of the Apple and Apple Computer folders in all of the user profiles – under

<UserName>\AppData\Local
<UserName>\AppData\LocalLow
<UserName>\AppData\Roaming

Reboot my Windows 7 PC – and then downloaded a copy of iTunes 9.0.3 from here… http://www.oldapps.com/itunes.php

Installed 9.03 – rebooted again – and presto – it worked.

You’ve wasted nearly a day of my time Apple – which is worth more to me than your phone and your terrible software.

/// <summary>
/// Let's synchronise some sheep...
/// </summary>
/// <param name="newSheep"></param>
/// <param name="db"></param>
/// <returns></returns>
public IEnumerable<Sheep> SynchronizeSheep(IEnumerable<Sheep> newSheep, IDbService db)
{
    IEnumerable<Sheep> oldSheep = db.GetSheeps();

    foreach (var delete in oldSheep.Except(newSheep))
    {
        db.DeleteSheep(delete.Id);
    }

    foreach (var update in newSheep.Intersect(oldSheep))
    {
        db.UpdateSheep(update);
    }

    foreach (var addition in newSheep.Except(oldSheep))
    {
        db.InsertSheep(addition);
    }

    return db.GetSheeps();
}

Other sticky features include being able to just type my search terms into the address bar – and execute a search (not unique to Chrome of course IE8 and Firefox both do this). And then the little details – like the screen shot shown here. While visiting one of my regular podcasts with Chrome – just clicking on the MP3 file – from what I’m assuming is a progressive download – launched a very nice little audio player. I don’t know why it’s there – or what ‘auto-magical’ stuff happened under the hood – it just worked. I also like the dark theme I found for Chrome and just ‘clicked’ and it was in – nice and dark and less painful to the eye – just like my dev IDE.

There are bigger issues and tectonic industry shifts at work here. I couldn’t write about them all with enough zest or knowledge to be of interest in this short post (like the future of Firefox now that the Mozilla Foundation’s largest funding organisation – Google – has their own browser) – suffice to say that for me at least- Chrome is sticky – and I’m using it more and more.