The Long Road Home

10
Nov
2011

The Long Road Home

I’ve toyed with a few titles for this post, including - ‘My Kingdom for a Backslash’, or ‘Fish Don’t Know They’re in Water’. In the end I felt ‘The Long Road /home’ was the most fitting. This post is about my thoughts and experiences as a software developer, leading up to my use of Linux and building my first non-Microsoft based Web application.

 

In The Begining

I’m a non-CompSci grad - having started my academic life with a college diploma in biology (a Canadian community college is the equivalent to what would have been called a ‘polytechnic’ in the UK). After about four years of working as a laboratory assistant, I decided that biology wasn’t for me, and moved into IT. I started the hard-way – in PC support, and slowly worked my way up from there. I had a couple of lucky breaks along the way, which combined with a lot of work, meant that I was eventually able to pass for a ‘nearly’ competent IT professional.

After several years of ICT related employment (and having overseen a couple of medium sized software projects), I decided that I needed to go back to the coalface, and learn more about software development. I started by writing desktop applications in VB6, followed by C# and ASP.NET. I also sat the required exams for the Microsoft MCSD and MCDBA certifications (.NET versions). I was firmly in the Microsoft camp. In fact even before my switch from biology to IT, I’d been building XT compatible clones – feeding DOS and Windows floppies into the disk drives of my home-built machines. It all seemed like good fun at the time.

Later – having earned some certs, and with a few projects under my belt – I thought I was doing all right, earning okay money on a couple of contracts, and thinking things were generally headed in the right direction.

Ch-Ch-Ch-Ch-Changes

But then two things happened. The first was that Microsoft unleashed a wave of frameworks and technology – part of what was then called WinFX, including Longhorn, WPF (Avalon), WCF (Indigo), WWF (its workflow offering). The second was that a few years later I decided to go back to school. I remember thinking at the time that by going back to school I was simply going to rack-up a decent qualification – something that would help me to progress, and maybe even help prepare for the day when I would hang-up my keyboard and move back into management (or something else altogether).

What I hadn’t anticipated, was how hard it was going to be to study as well as stay on top of what was happening in the Microsoft world. Things were changing fast, and I was struggling to keep up. Like some of my contemporaries, I was also trying to stay abreast of the broader trends in software development. I’d started reading some of Fowler’s stuff. And I was reading a lot about the challenges of managing software development projects – challenges that are unique to building systems that reach out and touch all aspects of an organization. I’d read ‘GoF’, Kent Beck, Eric Evans and others. I watched the TDD, BDD, and Agile zealots rise to glory. I was also following the raging WS-* debate, and beginning to understand the significance of Roy Fielding’s work.

I was beginning to acquire a deeper understanding of the software development process as a whole - in particular, an appreciation that the major challenges for most projects are not technology related. To this day the two most important texts I’ve read on the topic are The Mythical Man Month by Frederick P. Books, and PeopleWare by Tom DeMarco. Incredible when you think that The Mythical Man Month was first published in 1975. Read them both, and you’ll understand why they’re important in the field of software and systems.

Somewhere between studying, and trying to keep up with things in general, I began to experiment with Linux. I’d just started a course on computer security – which included Linux and Windows security. Thanks to virtualization technology, spinning-up a desktop Linux VM was a breeze; although apart from using the VMs for a couple of exercises on my course, not at lot happened at this point. I was still a bigoted Microsoft developer, convinced that C# was a 'real' developer’s language, and that Linux, Apache, MySQL and PHP were for kids. Actually that’s a little harsh. I wasn’t really bigoted, at least not in the literal sense – but humans are a tribal bunch, and we like to stick to what we know. Often when our beliefs in what we know are challenged, we resort to derision and are dismissive of alternative points of view.

Things began to change when my computer security course started looking at historical operating systems, combined with some basic principles of machine organization and computing. It was then that I realized my entire frame of reference in computing had been shaped by the tools, messages and strategy coming from Redmond. It was a myopic point of view. I discovered that a lot of the messages coming out of Microsoft were simply re-hashed versions that had originated elsewhere, and were designed to keep people on its platform. When Microsoft decided it was time to push a point of view that suited its technology, it did; and as far as message delivery goes – it excels at it. Remember the ‘impedance mismatch’ between general purpose programming languages and SQL? That was the message de jour when LINQ was released. Why not earlier? It was weird to see so many Microsofties ‘on message’ at the time. Of course Redmond wasn’t the only thing influencing my cognitive map of software development. My ‘up-to-date’ view of what was going on in the development community was shaped largely by the list of RSS blog feeds I’d subscribed to. I learned a veritable ton from excellent bloggers and technical writers.

But I was also becoming suspicious of a smaller group of writers whose mandate was clearly one of self-promotion, and who had little interest in helping the development community at large. Whether this is unique to the Microsoft development community or not – I was getting tired of the BS, troll-baiting, and general angst and confusion that seemed to be the result of mixed messages from the so called ‘thought leaders’ in the community. Rob Conery made a comment at one point that (paraphrased) the level angst and confusion surrounding data access – “is just weird” – and he was right. Remember the open letter to Microsoft - ‘ADO .NET Entity Framework Vote of No Confidence’? What better example of the level of hyperbole that exists between Microsoft and its community developers (acknowledging that the intent of the letter was to try and help the tool evolve). And then there was Scott Bellware.

Another important event in my transition from a Microsoft-only approach was when I attended my first Barcamp developers’ conference here in Bangkok. I was impressed with the energy and enthusiasm shown by the small army of open source software developers I’d met there. I wasn’t quite ready to put on my Richard Stallman acolyte’s robe (it turns out not many are, but that’s a different story) - but there was some very cool stuff going on. I also couldn’t help but notice that many of the successful Microsoft open source projects, were in fact ports of successful projects that had come from other languages and platforms; like Lucene, Hibernate, and Velocity to name a few.

The tipping point came when I was involved in a large .NET project that can only be described as the quintessential death-march. Death-march projects aren’t unique to users of Microsoft technology – but this one happened to be using those tools. And no one in the organization had heard of Richard Stallman, or REST, or had even the slightest interest in Agile; not that those alone are the basis for establishing the maturity of a software development team, but it was an indication that the team, like a lot of other teams, had been living in the Microsoft echo-chamber for too long. For me personally – it was hard – really hard. And most of what was hard had nothing to do with writing code. There were problems in requirements, which had led to classical problems of oversight and late realizations – the stuff of any difficult project. But what I found interesting in hindsight was the level of friction in tools, frameworks, testing and deployment – all of which conspired to fight against each other, creating an exhausting drag on the project.

Lessons Learned

After a bruising project, it was time to complete my studies and write my thesis. I’d sworn privately, that I would never allow myself to become worn-down or as affected by any project ever again – even if it meant a fundamental change in direction (although not quite ready for 410 Gone).  Fortunately, my next role was a fairly low-stress part-time engagement, which allowed me to devote the rest of my time to writing my final report.

What transpired over the course of those happy nine months was that I built and deployed a Linux-based production server, and wrote my first Web application based on an open source framework.

And after all of that, this is what I’ve learned:

First, Linux, Apache, PostgreSQL and PHP are not (just) for kids. Linux has turned out to be the lion of the Internet. Look at the latest Netcraft Webserver survey - 65% of the Web is powered by the Apache HTTP Server. Back in the late 90s and for most of the 2000s (and while Microsoft was still mucking around with IIS4, 5 and 6), the Apache HTTP Server community was benefiting from a world class webserver (with a real url re-write module) along with all of the other tools and technology that fit together to form seamless and frictionless development and deployment environments (before Microsoft developers even began talking about ‘friction’ in their projects).

I’ve come to realize that what’s happening now in the *nix world is simply too big to ignore, and I don’t think I would work for, or consider hiring, anyone that didn’t possess at least a rudimentary understating of this fact.

Second, I mentioned above that the major challenges for most projects are not technology related. In one of the first texts I’d read by Fowler, he describes three fundamental risks to any project: 1) requirements risk, 2) skills risk, and 3) technology risk. I think requirements and skills risks conspire to create the most trouble for any project, but technology is still important.

Successful projects can be, and are, run using Microsoft technology, but remember this: Microsoft’s only objective is to sell software licenses, and to do what ever is required to continue to sell software licenses. And that is all. It’s a company that has been arguably over-compensated for decades from a license-based revenue stream that is the result of its dominance in the desktop computer market. Nor has Microsoft been shy about using its position of dominance to destroy competition, and in the process, innovation. Its attempt to ‘create’ a developer community is a top-down effort and is the antithesis of how communities should be formed.

The developer division in Redmond still largely decides what the strategy will be for the development of Microsoft-based frameworks and technology. When compared to the level of activity and innovation in other communities, it might be reasonable to describe a dependency on the Microsoft platform as a technological risk – both in terms of vendor lock-in, and in terms of the maturity of the development community as a whole; and that these risks should be assessed as with any other risk to a project. If a project is at a stage where it can freely choose its underlying technology, then I believe there are healthier tools and communities in the non-Microsoft world.

Two great cases in point are the Ruby and Node.js communities on GitHub (and elsewhere). Looking at the speed of development and contributions to either of these communities, and it’s clear that while feathers might still be ruffled – good projects and tools quickly rise to the top, while mediocre projects fall to the side. It’s fascinating to watch.

Third, anecdotal evidence suggests that as many as 70% of all software projects fail to meet their stated objectives. It’s not a fun industry to work in, and death-march projects will continue to exist in all of the major technological camps. The success of any project will be determined by a subtle combination of maturity, experience and insight along with realistic estimates of the ‘true’ cost of creating or implementing software based solutions. In theory, the up-front analysis of a project is meant to be ‘technology agnostic’. In practice, most shops are in one technological camp or another. I think at this point however, at least one indicator of the likely maturity and experience of a potential supplier, or team – is their genuine willingness to consider a range of technologies and solutions including open source platforms and tools like Linux, PostgreSQL, PHP, Python and Ruby to name a few.

Fourth, I wish I’d started my career in software development at the 'web-face', maybe as a Perl/CGI scripter.

Over the coming months I’ll be publishing a series of blog posts on how I got started with Linux, Vim, and a bunch of other tools that, for me at least, made computing fun again. Stay tuned…

Category: 

Comments

how many years until since you change from biology to today? how you realize biology wasnt your thing? exactly what happened?

Hi saboralion - for one thing (at the time at least) biologists weren't making very much money - which is a shame. For another - I would have had to go back to school for at least another five or six years if I was going to progress with a career in biology. That was over 18 years ago...

The first multi-user, multi-tasking OS I worked on, was a Microsoft OS, called Xenix, a UNIX clone. It worked. Eventually, SCO bought it out. It amazes me that Microsoft, having worked with Xenix, could end up selling OSes based on DOS and VMS. I put it down to their not being able to own the product and having to pay royalties to AT&T. They have a special kind of greed; they want it all. I did some Windows programming way back when, but when I realised that Microsoft was promoting six different ways to access databases and that each version of Visual Basic was incompatible with the one before, I gave up. I learnt PHP, compiled FreeTDS and started to write web queries to an SQL Server database using Apache. My life suddenly got better. I know nothing about design patterns, Agile programming or any of the other buzzwords flying around nowadays. I'm a script writer basically. I like the freedom of the whole FOSS movement. I hate being told "you can't do that. you have to buy a license". Hate that. That's why I use free software. Life's too short for Microsoft's crap.

Hi Stan - interesting that you mention Xenix. I was going to include a family tree of OSs - and Xenix was going to be the point at which Microsoft might end up having wished they'd turned left, instead of right. Imagine having a decent commercial desktop OS (for which I wouldn't mind paying 'something') combined with the best of all the GNU and FreeBSD userland tools and apps. That's what Apple did - and it's the reason the Mac is such a popular machine for hackers today.

Can you imagine MS being happy with half of all WindowX developers using Emacs instead of DevStudio? Or LaTeX instead of Word? It may have been a very conscious left turn when they dropped Xenix and went on with Windows.

FOSS is cool, I totally agree :). It's dynamic, there is so much happening, and people are devoted to write good software. Having never been in the Microsoft camp, I have no idea how things work there. But, even with FOSS, software is written by people and companies. And they have their own road maps and interests. Things don't always run smoothly, there is lots of discussion and friction. The barriers to entry are higher than anyone would expect. FOSS developers are somewhat elitist. As long as you are a user of FOSS, this doesn't matter. It's only when you start thinking about contributing to FOSS that this might become an issue. Anyway, I just wanted to leave a comment and say hi :). It was great meeting you at Barcamp BKK and I wish you all the best for your future projects!

Interesting piece. I look forward to the follow up. Open source is where it is happening.

"mediocre projects fall to the side". When MS even *hints* at dropping a tech, people scream from the rooftops. Given that they've got paying customers to keep happy, they end up keeping around a lot of stuff that maybe should have been dropped earlier. But... they're really too big to act with the speed that other camps operate on. They're a victim of their own success.

If you mean the Eric Evans who wrote Domain Driven Design, then you've linked to the wrong Eric Evans. The DDD Eric is at http://twitter.com/#!/ericevans0 and http://domainlanguage.com.

Thanks - fixed!

I'm torn. SMB seems 'stuck' in many ways on Microsoft. The tech support is available and the release cycles predictable and appropriatly glossy with 'market share'. This is comfortable. And where a lot of money is. For those of us that just want to get something done and have it work for years, (pick your flavor of linux) is hard to improve upon. Selling this up the stack and having it integrate seamlessly with powerbuilder, the flavor of the week SQL server, and the flavor of the month MS Phone server proves to be enough 'real world' that I've observed many smaller shops not even considering Linux. This is disappointing as I believe it does many things better and more reliably with fewer resources if one is willing to put in the time. Some shops have more money then time; and that hurts innovation.

That is a really good article on the software world and my own progress through it has mirrored your own quite closely: I started off as a language graduate of all things! I still work commercially in the Microsoft world but have become more and more interested over the last couple of years in the *nix / OSS side of the development world especially erlang and am certainly not a Microsoft acolyte. There's a lot of really good .Net-based technology out there (ASP.Net MVC is a great case in point) but there's equally a lot of great non-Microsoft technology and I would like to be able to combine the best of the two rather than picking a silo to work in. But anyway, really, really good article and thanks for posting - lots of food for thought!

Thanks for the feedback Andrew and Dan. I'm hoping the next few posts on the steps I went through to get up and running, will be of help to those who are ready to give Linux and an open source stack a try.

nice article. Funnily enough i'm also mirgrating over the the linux and open source world. Been a windows developer for quite a few years now and am getting dissillusioned by the Redmond way of things. Haven't moved over to dev work on Linux yet but I am going to start soon!

Great post. I love Linux...all flavors!

I'm happy such a view is getting bigger audience and acceptance. In 90's I've almost decided to cancel my IT study because I was sick and tired of bloated M$ techonology buzzwords all hipe around "known & repackaged" things. M$ documentation was a horror. M$ APIs were horror. Tools were a crap that hardly worked for what it was supposed to (somebody remembers Visual Source Safe - can't beleive it, but people used that, engineers!, "because it had a GUI"! oh god...). Luckily Linux came soon (enough) and I'm really enjoying my work today. I've returned from management "back" to creative work in engineering field thanks to FOSS. And I earn well. So, I'm lucky lucky bastard...

Open source presents its own counterparts to dependency. Vendor lock-in is not a danger in just proprietary systems. To use your examples, if your web-site runs on Node.js and you want to go with another platform later on, you have no other options, you'll have to rewrite your site from scratch. Same with Ruby. If you choose the Ruby platform, your Ruby code won't run on anything but Ruby. If you don't like the direction some FOSS platform is going, do you have the time and resources to change the source code to meet your needs? Do you have any more influence over that direction with a FOSS product than a MS developer does with MS' products? I doubt it. In the end, it's all a gamble. As for the maturity of the .NET community, you imply that it isn't as mature as the FOSS communities. You're kidding, right?

Robert S. You seems to get offended. But do you think Ruby is an OS? You can install Ruby on any operating system I think. So you can run Ruby code anywhere you want. There is nothing from scratch. However It is not a discussion on what software or framework you can use, but it is more on OS-dependencies and trap, the blind-folding of MS and the herd of people thinking MS is the only sacred place to live the life. I am just posting your words here: "do you have the time and resources to change the source code to meet your needs?" And ask: have you ever tried to change Microsoft's Visual Studio produced codes? Do you know how MFC's preprocessor directive works? Have you ever tried to touch them? Just to write "Hello world" do you know how many lines of code is produced in Visual Studio Ultimate 2011? You may not be able to discover the reason why your software is not working the way it is supposed to work. That is not in a case in open-source community. You know why something is going wrong, and then you also know how to correct it. In Microsoft's case it is a nightmare. You will have to shuttle between "managed" and un-managed codes. In open-source there is nothing mis-managed.

Nice post, it's amazing how you can get lost in the 'Microsoft echo chamber' and forget that there is a whole world out there. It becomes difficult when you also have to convince other colleagues that are also lost (both tech and non-tech) that there are alternatives. There is fear of the unknown there, and I think that Microsoft have also played to the 'skills risk' category by putting fear in the enterprise that if you aren't developing in .NET then you will have trouble finding good developers because "everyone is doing .NET"

We tried to switch from M$ SQL to Postgre. Spent significant time testing just to find that Postgre driver for .NET makes simple inserts and updates 8 times slower than with MS SQL. Wrote to their teams and only silence in response. I am not mentioning other problems Postgre has. So we rather pay premium for something that works and is supported. As fas as I know My SQL does not even have foreign key constraints? We use plenty of other services provided over net, and from our experience, if their service runs on open source - expect all sorts of problems caused by low competence levels... Tried Open Office (was turned off with counterproductive M$ ribbon interface), but had to uninstall it or risk documents being mangled :( To the date, all our attempts to ditch M$ resulted in time=money loss. Perhaps the only exception was OpenProject, but again it does not export to MS Project format. I am not theorizing, just sharing our REAL LIFE experience...

MySQL does have foreign key constraints if you use the InnoDB engine. That said, yes it is difficult to switch if you are trying to integrate non MS platforms into an MS environment, I know from experience as well. A lot of commercial vendors like MS (it's not only MS) try to tie you to their platform so it seems like nothing else works, but that is not the case, it's just hard to make other things work well with their stack of technologies. MS even has a long history of breaking other products on their platform so you will use theirs instead (not saying that is the Postgre .NET provider issue). For our company I built our web and intranet off of MySQL, Apache, and Ubuntu servers, this was previously an ASP.NET, MSSQL and IIS site. Our intranet and public web are now faster and easier to manage. Yes I'm sure there are incompetent open source people, just like there are in any field you look at. If you want to be sure you get someone who knows what they are talking about get paid support, which MySQL has and so does Ubuntu (and so do other large open source projects). Yes I agree that some open source projects can be counter productive but there are others that are very professional (Apache, Hadoop, MySQL, various Linux distros, etc). Kudos for trying and thinking outside the box, hope you keep an open mind in the future and just use the best tool for the job, whether MS or other.

Thanks for the thoughtful feedback Lucas.

Thanks for a really great article! I've been doing C# and SQL dev for a while now, and I've also been watching their new technologies roll out. Recently I took on some extra work with a small company that was pretty open on what technologies they wanted to use, and I initially installed kubuntu with apache, php, mysql and an open source project that fitted their needs, to save them some money... This was probably the best thing I could have done, because Linux has grown on me. Now I'm looking into php and mysql more seriously.

Hi Brad. That was in effect the leap I took as well. It was tough going initially - but it's paid off in spades. The client was on a budget as well - and so not having to worry about server OS licensing costs was an added bonus.

Reading other comments I notice that technical people might not understand business realities. To keep turning profit M$ needs to leave room for improvement in each version of every product. You can clearly see that they will do improvements in one area which is a selling point, and make some other areas worse than before. In the next version these revamped areas will be a selling point. Or they create a DLL hell with MDAC and then sell something that solves it and creates other problems. Architecture of current Windows systems is such that it will keep bloating the system with updates and service packs. Eventually the only workable solution will be buying the next version. In current economic system that behavior is an unavoidable evil. Goal of every business is turning maximum profit. The only things we can blame M$ for is that their marketing machine is superior to brains of IT managers who decide on technologies to use. During their Win7 TV marketing campaign in Australia they were showing IT-illiterate people stating that Win7 was their idea. As IT professionals we kind of agreed :) Microsoft still has plenty of things that really work for most scenarios. The common mistake of IT managers is buying what Microsoft is marketing, not what the business really needs. The sad thing is that we looked, tried, and still did not find better alternative to core, well-established Microsoft technology.

Sounds like it sums up my programming life: VB to VB.Net, then to C#, then to C/C++; then, out of a desire to hack my android phone, I got involved with linux, and discovered a whole new world. Still, after nearly three years' involvement in all the things on the *NIX camp, I still find myself unable to drop Windows; after all, Windows is still the dominant end-user PC operating system, whereas Linux/UNIX systems have too sharp a learning curve or (seemingly) lack of features, and Mac's are too expensive for the average end-user. Microsoft technologies are also great, there's no denying that, only it's not the only great thing. As long as Microsoft's still alive, I can't ignore it.

Interesting article - I made the switch to nix about a decade ago, having once been an MS "fan boy" too - Windows NT beta tester, no less. These days I am less ardently hostile to MS than I used to be - I wrote about that here http://cartesianproduct.wordpress.com/2011/09/17/microsoft-are-not-the-enemy/ - but could never see myself going back. Incidentally, there is nothing particularly wrong about the design of Windows NT and its children, but the issue for me is freedom to choose what works and not have it dictated by one commercial interest.

Thanks for the link Adrian - interesting post.

Great article! I am looking forward to seeing the "roadmap". I am currently following a similar path. Excel Macros, Asp (Classic), Vb .NET and now C# .NET. I am experimenting with LAMP at home, incliding the mono project for c#. Limited time means that if I choose to go the OSS route, I need a good solid direction to go in with the right starting point. Seeing your personal roadmap would help little minnows like me. :-)

Hi Gareth - that's the idea behind the series really. There are many routes into FOSS, Linux/FreeBSD, and lots of good articles out there already. I'd just like to publish the path I took - hoping that might help anyone else that tries to do the same.

Excellent! Keep it up. Running Ubuntu 10.04 Server as a base at the moment at home for testing, now I just need to think of something to use it for so I can get going. (Other than a remote storage system for my iPad via ssh). (Just spent the last 10 minutes shouting uncontrollably at Visual Studio 2011). Does FOSS come with extra coffee?

I tend to disagree with this post. I particularly like the .Net environment. I agree that the MS way to do stuff may not be suitable for everyone (since it does require some investment in time and money up front), but the truth is that .Net is a trully inspiring environment as a whole, not just for web development. I agree that a lot of the ideas put into .Net have been implemented in other languages as well, but incorporating these into .Net is not a negative. On the contrary, we now have a very solid and powerfull environment on which we build software. Also for all those that hate Microsoft because it is the "cool" thing to do, I just want to say that however Microsoft works, it started, it grew and it still is a company helping developers by providing cheap development tools, great development environments with great support while targetting enterprises for the revenue (an almost zero revenue is produced from developer products).

Hi Savvas, I agree that the .NET environment works, the CLR is impressive, and the framework libraries are extensive. I wasn't suggesting otherwise. But the motivation for producing these tools, is clearly tied to keeping developers, and users, on the Microsoft platform. It's good to have choice. And I think being able to choose freely between a Linux based project, or a Microsoft based project - is a good thing.

looking forward to seeing more