58bits - tech

Google Custom Search Engine and Google Site Search – In 30 Seconds

Anthony Bouch — Mon, 23 Nov 2009 18:11:56 GMT

A project I’ve been working on recently required a search feature. After a quick look around I thought I’d give a hosted solution a try – using Google Site Search.

After almost a full day of banging my head against the wall – here’s what I discovered…

I) The only difference between Site Search and Google Custom Search – is that you pay for Site Search and you get on-demand indexing – so you’re guaranteed (in theory) to have all of the pages in your site indexed.

II) Once you’ve paid for Site Search – the control panel is effectively the Custom Search Engine Search control panel – with the on-demand option present.

III) There are three ‘levels’ of control you have over the results on your site.

Here’s a quick tour of each – assuming you’ve read just enough of the docs (Here’s a link to the Google AJAX Search API) to know that you need to link to the jsapi JavaScript library – load the search api via: google.load('search', '1'); and then declare your OnLoad function .

Note – for all three options it’s a good idea to have the following style sheet at the top of all your test pages – since you get some ‘freebie’ and fast styling as a result.

http://www.google.com/cse/style/look/default.css

1) google.search.CustomSearchControl – with just two lines of code – you can have a custom search page up and running (well not quite two lines – but these are the most important two).

var customSearchControl = new google.search.CustomSearchControl('013652945859801643433:fdsfsqjnea4');
customSearchControl.draw('targetDiv');

The parameter to the CustomSearchControl is the CSE id for the custom search engine you’ve created – and calling ‘draw’ will render the completed form and search results area to the page. The CustomSearch control has a built in ‘searcher’ based on the settings from your CSE control panel. CustomSearchControl is however a little difficult to customize. I found it hard to turn off the default ‘search’ form and place the results where I wanted them. You can add additional searchers – like images and video search via:

customSearchControl.addSearcher(new google.search.ImageSearch(), options);

The main point to note here – is that CustomSearchControl is ‘driven’ by the CSE control panel – including filter options, look and feel etc. all based on the search engine unique ID that you’ll also find in the CSE control panel.

This is custom search with training-wheels on.

2) google.search.SearchControl – is the next level down in abstraction – and here’s the ‘Hello World’ example from the docs. With SearchControl – you have to add searchers to the control and then you can tell the control to ‘draw’ itself – just like with the CustomSearchControl. However it’s a little easier to customize. For example:

this.control = new google.search.SearchControl();
// set up for CSE result sets
this.control.setResultSetSize(google.search.Search.FILTERED_CSE_RESULTSET);

// configure search control
var options = new google.search.SearcherOptions();
options.setExpandMode(GSearchControl.EXPAND_MODE_OPEN);
options.setRoot(resultsDiv)

var searcher = new google.search.WebSearch();
searcher.setSiteRestriction("01365294584480dsadad3:ffsfsfseqjnea4");
searcher.setUserDefinedLabel("Search Results");
searcher.setLinkTarget(google.search.Search.LINK_TARGET_SELF);

this.control.addSearcher(searcher, options);

this.control.draw();

Note the options.setRoot(resultsDiv) highlighted line. This does two things – 1) It sends the output of the search to the ‘resultsDiv’ and 2) it turns off the submit form – which means you can now control the input to search – from your own form – placed anywhere on the page as long as you handle the OnSubmit event and call

this.control.execute(‘formInputValue’);

Again if you link to http://www.google.com/cse/style/look/default.css – you’ll get some instant results that look ok.

3) google.search.Search – is the lowest level abstraction – the raw searcher class – and although you still get some good stuff here, you will have to take care of rendering the results to the page. In this case you will need to handle the input and output for every class of searcher you use – Web, Local, Image or Video – all of which are derived from google.search.Search. For example…

// Our searcher instance.
var searcher = new google.search.WebSearch();

searcher.setSiteRestriction("013652945859801616623:ecbgsqjnea4");
searcher.setResultSetSize(google.search.Search.LARGE_RESULTSET);
searcher.setUserDefinedLabel("QCC Search Results");
searcher.setLinkTarget(google.search.Search.LINK_TARGET_SELF);
searcher.setSearchCompleteCallback(this, searchComplete, null);

var searchForm = document.getElementById("search");
if (searchForm.q.value) {
    searcher.execute(searchForm.q.value);
}

Here's a link to a complete example of a raw search with output being handled by the developer (which is in another area of the Google docs – the Code Playground).

While there’s more plumbing required here – you’re still off to a good start. Unlike the sample above for raw handling of the results – you can use the .html property of the result object in the results array – which will tie up again nicely with the linked style sheet - and you’ll have a ‘Goggle-like’ – good-to-go search experience in your site. Like this…

function searchComplete() {
    // Check that we got results            
    if (searcher.results && searcher.results.length > 0) {
        // Grab our content div, clear it.
        var resultsDiv = document.getElementById('results');
        resultsDiv.innerHTML = '';
        // Loop through our results, printing them to the page.
        var results = searcher.results;
        for (var i = 0; i < results.length; i++) {
            // For each result write it's title and image to the screen
            var result = results[i];
            resultsDiv.appendChild(result.html);
        }

        // Now add the paging links so the user can see more results.
        addPaginationLinks(searcher);
    }
}

And that's it. There is documentation available for all of this – but it’s in various locations. The first examples you will see if you come to search via the CSE control panel – will be the CustomSearchControl. The hard part was the ‘orientation’ exercise that I had to go through to understand what the heck I was using, where it fit in the bigger picture of the search API, and how I could create a more customized search experience. The docs are a bit all over the shop in this case – and so it took me nearly a day to write some POC tests and properly understand the relationship between google.search.CustomSearchControl, google.search.SearchControl and good.search.Search.

Hope this post helps to reduce the pain for anyone else that’s new to Google Search and would like to implement a tailored search experience.

We Come to Bury IE6

Anthony Bouch — Wed, 04 Nov 2009 13:15:21 GMT

When the desired death of a browser makes it into a full-page editorial of IEEE Spectrum – you know that the issue is finally starting to gain some traction. The tech blogosphere communities have been making plenty of noise about this for a while now. And I’d love someone to do an ‘economic’ impact analysis for IE6 – both in terms of development and security related incidents.

I think the Spectrum article (and the referring blog post from Digg) does a good job of describing the issues – corporate restrictions being high on the list of reasons that prevent a user from upgrading. That said I’d rank Microsoft’s failed Vista strategy which resulted in downgrade options to Windows XP being offered by manufacturers, along with the number of illegal and non-updating XP installations out there as just as large a contribution.

Despite my earlier and naïve attempt to ban IE6 from my site I’ve now successfully bared the browser using IE conditional comments – which is the only reliable way you can detect this version of IE (you can’t sniff for it reliably). I then display the following :-)

Nikon ViewNX Broken on Windows 7

Anthony Bouch — Sat, 24 Oct 2009 12:15:25 GMT

I posted my views earlier on Nikon software – and I’m now pretty sure that Nikon ViewNX is broken on Windows 7. Below is a link to the native Process Monitor log file – filtered for ViewNX after a complete uninstall and re-install to the latest version of ViewNX 1.5.0 – running on the RTM of Windows 7. ViewNX was still performing an endless loop of I/O and registry requests.

I stopped recording after 1.7 million events (a 182MB log file – compressed to 21MB below).

ViewNX.zip

:-(

Formal Versus Agile Methodologies

Anthony Bouch — Fri, 25 Sep 2009 23:28:41 GMT

I’ve been an IEEE and Computer Society member for a couple of years now. Trying to keep up with the IEEE publications along with my RSS Reader and tech news has been a challenge – so I’m sure I’ve missed some good stuff along the way.

However, an article in the September 2009 edition of ‘Computer’ caught my eye: “Formal Versus Agile: Survival of the Fittest” by Sue Black, Paul P. Boca, Jonathan P. Bowan, Jason Gorman and Mike Hinchey.

It’s a great article with good history and an insightful assessment of the current ‘methodologies’ landscape. Unfortunately the article is only available to subscribers of ‘Computer’ at the moment. I hope it will be published more widely in the near future since I think it has a wide appeal.

The author’s comparison of RAD and Agile was particularly interesting – and the following statement about Agile practises is worth quoting…

Most teams purporting to be doing agile software development are not applying the level of technical rigor necessary to succeed at it. Most ‘agile’ teams have actually only adopted Scrum’s project-management practise and have failed to effectively adopt ‘the hard disciplines’ like test-driven development, refactoring, pair programming, simple design (writing the simplest code possible to satisfy the customer’s requirements), and continuous integration.

Their description of the major misconception that ‘Agile = Fast’ was also insightful (when it should be about being responsive to ‘change’).

A great read if you can get hold of it.

Projects

Anthony Bouch — Tue, 01 Sep 2009 22:09:39 GMT

I’m pretty sure that anyone who actually likes to ‘make stuff’ and has been a programmer at least once in their life – has got a short list of pet projects that he/she has started – and maybe even finished :-)

Here are a few of my ‘pets’.

CodeDom class generator for VS2005/2008

This was an early attempt of mine at writing a Visual Studio Add-in - designed to create a class from a database query, table or stored procedure. There’s a ton of stuff like this out there – but I wanted something that generated the class from the DataTable – SchemaTable of an IReader – so that a class could be generated from any query or shape for use with DTOs, reporting, or a simple domain model. You can choose from C#, VB or C++ to generate the class as well as a number of styles of classes – from a built-in IReader constructor, attribute based mapper, or plain old CLR object (POCO). There also doesn’t have to be any data in the result set for this to work since the SchemaTable will be returned regardless.

The Add-in also adds an ‘Open Folder’ option to the context menu of all solutions, projects, folders, etc. in Solution Explorer. This feature exists as of VS 2008 – but I wanted it at the top of the context menu – not at the bottom.

You can download the Add-in here at - FiftyEightBitsAddin.zip Just unzip and place the contents of the zip file into any one of the Add-in locations for VS2008.

PreCode Code Snippet Plugin for WLW

This is an open source project hosted at Codeplex – a plugin for Windows Live Writer that helps to format code snippets before pasting them into a blog post (or elsewhere). It was written for the excellent SyntaxHighlther from Alex Gorbatchev. It’s a WPF app – and also includes a standalone desktop version for pasting formatted code into any app (ideal for Stack Overflow). Please don’t look at this as a reference WPF app though – it’s more than just a little ‘hacky’. The app that follows is my ‘better’ WPF effort. Visit http://www.codeplex.com/precode for more details, the source code and the installer.

Gallery Builder – Desktop and Web

If you follow the Photos link above – you’ll see the results of this application. Yet another gallery builder app – however I’ve spent a bit of time on this one – choosing a specific architecture that I think represents a good balance between desktop and web server processing for small to medium size collections of photos. I’ll be hosting a Wiki soon with docs and the full download including the HttpModule that is used display galleries online (along with a sample app and extensibility features). The module can be integrated into any site and includes full Atom and Rss syndication support, nested galleries, as well as good caching/304 response support.

A pre-release version of the desktop application can be downloaded from the link below (although it’s not a whole lot of good without the web component). Hand rolling a complete set of control templates for a custom theme in WPF was a great way to get to know the inside of WPF.

NOTE: If there are no other WPF apps on your machine – and this application is the first WPF application you start – the ‘cold’ loading time for a WPF app is very long. Subsequent application loading times are what you’d expect.

GalleryBuilder_0.9.9.1000.zip

Yet another Task Management App - DotNetTime

Yet another task/project list manager; only I actually use this one. This is an ASP.Net Model View Presenter application (and hopefully my last Webforms app). I use this to track my own time working on projects or just to ‘monitor’ my own time in general. Implementing multi-language support was a good learning exercise – and until recently this had been my ‘workbench’ throw anything at it hobby app. Now that charting controls are available free from Microsoft – I’ll be adding some charts to the reporting page. Feel free to register and have a play – it’s very basic – but it does what I need. Be gentle too if you’re into web hackery– this was built before my formal introduction to the world of application security and so I wouldn’t be surprised at all if there were XSS or CSRF vulnerabilities (although there shouldn’t be any SQL injection weaknesses – he says).

http://www.58bits.com/dnt

Get XElement Value with Default

Anthony Bouch — Sun, 02 Aug 2009 01:24:20 GMT

Thanks to this post at .Net Dojo, here’s a utility method for getting a safe value from an XElement, returning a default value if the element was not found.

public static T GetElementValue<T>(XElement parent, string elementId, T defaultValue) where T : IConvertible
{
    XElement element = parent.Element(elementId);
    if (element != null)           
        return (T)Convert.ChangeType(element.Value, typeof(T), CultureInfo.InvariantCulture);           
    else
        return defaultValue;   
}

I’ve been using this combined with .Net 3.0 object initializers to re-hydrate objects from Xml with safe values, like…

var myClass = new MyClass() 
{ 
    Name = GetElementValue(parent, "Name", String.Empty),
    IsDefault = GetElementValue(parent, "IsDefault ", true),
    YearsLeft = GetElementValue(parent, "Yearsleft", 100)
}

ASP.NET MVC 304 Not Modified Filter for Syndication Content

Anthony Bouch — Sun, 26 Jul 2009 06:35:50 GMT

Earlier I posted about a compression filter for ASP.NET MVC that was the product of two other posts on compression and QValues. I’ve also seen a couple of posts describing different approaches to creating syndication content via ASP.NET MVC – RSS or Atom feeds. Here are two suggestions – one at DeveloperZen and the other at Stack Overflow that both use an RssActionResult subclass of ActionResult.

What was missing was a 304 Not Modified filter – so that feeds that have not changed can return a 304 Not Modified HTTP Response - saving bandwidth and improving performance of the consumer.

Here’s my attempt at putting it all together using a custom NotModifiedFilter, the CompressionFilter and a SyndicationActionResult class.

First the NotModifiedFilter. This article at the Embarcadero Developer Network was very helpful (along with the W3C standard for 304 Not Modified).

Here’s the filter.

public class NotModifiedFilterAttribute : ActionFilterAttribute
{
    public override void OnResultExecuted(ResultExecutedContext filterContext)
    {
        var response = filterContext.HttpContext.Response;
        var request = filterContext.HttpContext.Request;           

        if ((IsSourceModified(request, response) == false))
        {
            response.SuppressContent = true;
            response.StatusCode = 304;
            response.StatusDescription = "Not Modified";
            // Explicitly set the Content-Length header so the client doesn't wait for
            // content but keeps the connection open for other requests
            response.AddHeader("Content-Length", "0");
        }
    }
}

And here’s the helper method that tests for modified content.

private static bool IsSourceModified(HttpRequestBase request, HttpResponseBase response)
{                  
    bool dateModified = false;
    bool eTagModified = false;

    string requestETagHeader = request.Headers["If-None-Match"] ?? string.Empty;
    string requestIfModifiedSinceHeader = request.Headers["If-Modified-Since"] ?? string.Empty;            
    DateTime requestIfModifiedSince;                    
    DateTime.TryParse(requestIfModifiedSinceHeader, out requestIfModifiedSince);
    
    string responseETagHeader = response.Headers["ETag"] ?? string.Empty;
    string responseLastModifiedHeader = response.Headers["Last-Modified"] ?? string.Empty;
    DateTime responseLastModified;
    DateTime.TryParse(responseLastModifiedHeader, out responseLastModified);

    if (requestIfModifiedSince != DateTime.MinValue && responseLastModified != DateTime.MinValue)
    {
        if (responseLastModified > requestIfModifiedSince)
        {
            TimeSpan diff = responseLastModified - requestIfModifiedSince;
            if (diff > TimeSpan.FromSeconds(1))
            {
                dateModified = true;
            }
        }
    }
    else
    {
        dateModified = true;
    }

    //Leave the default for eTagModified = false so that if we
    //don't get an ETag from the server we will rely on the fileDateModified only           
    if (String.IsNullOrEmpty(responseETagHeader) == false)
    {
        eTagModified = responseETagHeader.Equals(requestETagHeader, StringComparison.Ordinal) == false;
    }

    return (dateModified || eTagModified);    
}

And lastly – here’s my SyndicationActionResult class which uses a Func<FeedData> delegate to get the feed data – and so can be used for any type of feed – Rss, Atom, Sitemaps etc. (FeedData contains the feed content as well as the last modified date and the ETag). Not sure if I’m guilty of cargo cult programming here since I was a little unsure about setting the ‘Last-Modified’ header manually instead of using response.Cache.SetLastModified() method. However the latter was not showing up in the HttpContext of the OnResultExecuted method in the filter. Maybe the runtime moves this value into the headers collection later in the pipeline.

public class SyndicationActionResult : ActionResult
{
    public Func<FeedData> ActionDelegate { get; set; }
    
    public override void ExecuteResult(ControllerContext context)
    {
        if (ActionDelegate != null)
        {
            var response = context.HttpContext.Response;
            var data = ActionDelegate.Invoke() as FeedData;
            if (data != null)
            {
                response.ContentType = data.ContentType;
                response.AppendHeader("Cache-Control", "private");
                response.AppendHeader("Last-Modified", data.LastModifiedDate.ToString("r"));
                //response.Cache.SetLastModified(data.LastModifiedDate);
                response.AppendHeader("ETag", String.Format("\"{0}\"", data.ETag));
                response.Output.WriteLine(data.Content);
                response.StatusCode = 200;
                response.StatusDescription = "OK";
            }
        }
    }
}

Note that we write the content to the response.Output stream - however if the NotModifiedFilter tells us nothing has changed – then this will be suppressed – (although it is important that the ETag remains).

And very lastly - here's the Action in my SyndicationController class that puts it all together, and in this case – is used to provide the aggregate Atom feed on my home page at http://www.58bits.com.

[AcceptVerbs(HttpVerbs.Get)]
[NotModifiedFilter(Order = 1)]
[CompressFilter(Order = 2)]
public SyndicationActionResult SiteFeedAtom()
{
    return new SyndicationActionResult() { ActionDelegate = SyndicationHelper.GetAggregateAtomFeed };
}

As Scott sometimes says - it may be “poo”… or it maybe useful. At least it seems to be working ok for me, although suggestions on how any of it might be improved would be greatly appreciated.

The Best Windows 7 Feature

Anthony Bouch — Fri, 17 Jul 2009 16:13:03 GMT

I’m still freak’n over how solid, fast and all-round great Windows 7 is. The first time ever an OS upgrade (without repaving the machine) actually feels like I did a fresh install.

Vista drove me nuts – both because of its sluggishness, and because of a totally failed file and folder management strategy. At the height of my frustration I was experimenting with Total Commander, Opus 9 and a couple of other file management applications in an effort to rid my self of the the ‘built in’ windows explorer. Folders did not remember the view or view preferences between sessions. Even worse was Vista’s attempt to guess the contents of a folder and ‘adapt’ accordingly. Maybe one day when we’re all in the cloud – or when the file system has been abstracted away by a content and keyword management system we won’t need conventional access to the file system per se – but we’re not there yet – and I do.

Here’s what they’ve done in Windows 7 – which is exactly right. They’ve separated their attempt at managing views based on content – from a regular file management window (and all explorer windows always remember the last view used).

So for example – if I’m looking at photos via the Picture Library – it looks like this.

…with extra columns for tags and picture information. It’s a media specific view and that’s fine. Of course it could be a thumbnails view – but the important thing is that in details view – content specific columns appear.

If I need a ‘real’ file system window though – say because I want to quickly sort by file type or some other file system criteria – then I navigate to the same folder via the file system using a ‘regular’ explorer window (Windows Key – E) – which looks like this.

The default view for folders when you’ve entered them not via a Library but via the file system – is the ‘General Items’ view – which works like you’d expect it to when you need a ‘regular’ file management window.

What a relief. It works. It makes sense, and I can use Windows Explorer again.

Technorati Tags: Windows 7

How to ban Internet Explorer 6 from your Site Using the IIS7 Url Rewrite Module

Anthony Bouch — Sat, 11 Jul 2009 10:39:08 GMT

UPDATE 12/07/2009 - As Shirley Boyle sang - "I dreamed a dream". The exercise below was great for learning how to use the IIS7 Url Rewrite Module - but there is a long list of user agents that contain the string "compatible; MSIE 6.0;" including the MSN spider and some transparent proxies. Internet Explorer 6 is the browser that refuses to die. :-(

ORIGINAL POST - This one took a little while to figure out along with a careful read of the reference docs for the IIS7 Url Rewrite Module. I was banging my head against the wall for a few minutes with {USER_AGENT} before reading the fine print… the HTTP prefix bit in particular.

All dash (“-”) symbols in the HTTP header name are converted to underscore symbols (“_”).
All letters in the HTTP header name are converted to capital case.
“HTTP_” prefix is added to the header name.
For example, in order to access the HTTP header “user-agent” from a rewrite rule, you can use the {HTTP_USER_AGENT} server variable.

And here’s the result…

<rule name="BlockIE6" stopProcessing="true">
  <match url=".*" />
  <conditions>
    <add input="{HTTP_USER_AGENT}" pattern="MSIE 6" />
    <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" />
    <add input="{URL}" negate="true" pattern="ie6" />
  </conditions>
  <action type="Redirect" url="ie6" redirectType="Permanent" />
</rule>

It’s really really important that you do two things: 1) Prevent the redirect from redirecting static content – like script files and style sheets, ala the {REQUEST_FILENAME} negative condition, and 2) You also check the Url to see if it’s the redirected Url – i.e. they’ve already been redirected – otherwise you’ll get an endless redirect loop. In this case I’m sending them to the url “ie6” which is an action on a route in ASP.Net MVC – displaying a download page with some ‘alternatives’ :-)

The Functional Renaissance

Anthony Bouch — Wed, 08 Jul 2009 18:09:39 GMT

I recently watched Luca Bolognese’s presentation at last years PDC on F#. A really great presentation.

I’ve also been listening to Steve Vinoski’s podcasts on the The Functional Web.

There’s definitely stuff up in this area now. Coming from a non-comp-sci background – and only ever having written statically typed and imperative code (sad I know) – I’m curious to see where this will all lead. Haskell , Erlang, and Scala are all on the rise. CouchDB (backed I think by Lucene for search) and even Yaws (yet another web server) – are just two of several significant projects written in Erlang.

I think it was Martin Fowler who said that the best way to be a better programmer – is to learn another language, and F# is now on my list of things to do.

Technorati Tags: fsharp,haskell,erlang,scala

iPoo

Anthony Bouch — Wed, 08 Jul 2009 15:47:00 GMT

What will those crazy folks at Apple think of next?

BlackBerry Tour

Anthony Bouch — Sun, 05 Jul 2009 05:43:41 GMT

I'm still in watch and wait mode - being torn between the iPhone 3GS and a BlackBerry device.

I partly want to own an iPhone just to be part of the iPhone phenomena - not to mention use the device that has totally changed the landscape of mobile communication.

I also really admire the BlackBerry devices. I noticed a few reviews for the new BlackBerry Tour - and thought 'oh that's nice'. Great specs - a refinement of the BlackBerry World Edition and Bold all rolled up into one - and then the bottom fell out. No WiFi. ? WTF? Here’s a good review of the Tour.

Still waiting and watching....

Technorati Tags: blackberry,iphone

How to Compile a WPF Application as a Class Library

Anthony Bouch — Thu, 25 Jun 2009 06:17:09 GMT

Charles Petzold’s book, Applications = Code + Markup - received some critical press for not being very pretty. And it’s true that there’s a lot of text and code listings but (and as you’ll see below), he was methodical.

I recently wanted to upgrade a Windows Live Writer plug-in I’d written to use the Windows Presentation Framework (WPF) – instead of WinForms and a custom UI library.

However my first attempt to compile the application as a Class Library resulted in the following error…

“Library project file cannot specify ApplicationDefinition element”.

Thankfully I’d remembered the opening chapters of Charles’ book where he spent considerable time explaining the creation of the application class and initial window. The trick in this case is to remove the App.xaml file. You then need to create your own start-up class – like Program.cs – as shown below, and the application will compile fine as either a Windows Application or Class Library. In the example below extra parameters are passed to the constructor of the main window which is opened as a Dialog – and then checked for the DialogResult value when the window is closed.

public class Program : Application
{
    [STAThread]
    public static void Main()
    {
        var app = new Program();
        app.Run();
    }

    protected override void OnStartup(StartupEventArgs args)
    {
        base.OnStartup(args);

        var window = new PreCodeWindow(new PreCodeSettings(), PreCodeWindow.Mode.StandAlone);
        window.ShowDialog();
        if (window.DialogResult.HasValue && window.DialogResult.Value)
        {
            System.Diagnostics.Debug.WriteLine("OK");
            Clipboard.SetText(window.Code);
        }
        else
        {
            System.Diagnostics.Debug.WriteLine("Not OK");
        }        
    }
}

The downside to this approach is that you no longer have an application level location for placing resource dictionaries and other shared objects - and so these will need to be placed in each window as required.

IIS7 StaticContent and ClientCache Settings

Anthony Bouch — Sat, 13 Jun 2009 00:58:35 GMT

Yet another great feature of IIS7 - declarative static content cache settings. It took a little detective work to find the settings for this feature – including a helpful post from Jörg Jooss.

I wanted to be able to set the client cache settings for the static content of a site I host at ORCS Web. In the past – unless offered by the control panel of your shared hosting provider – this would have meant emailing support and asking them to set the client cache header settings for the required directory in IIS6. In IIS7 this can be set now in the <system.webserver> section.

Note: If you’re going to try this on your local machine first, you will need to unlock the staticContent section in IIS7 (from the machine level) using the following appcmd…

appcmd unlock config /section:staticContent

You can also use appcmd to create the static content cache settings as shown in Jörg’s post above, which will add the correct section to your web.config – or you can directly edit/create the web.config file for the required directory.

I chose to hand edit the web.config – creating a new web.config located in the directory with the static content (i.e. images, style sheets, script files – all under the ‘assets’ directory in my case)

<?xml version="1.0"?>
<configuration>
  <system.webServer>
    <staticContent>
      <clientCache cacheControlCustom="private" cacheControlMode="UseMaxAge" cacheControlMaxAge="3.00:00:00" />
    </staticContent>
  </system.webServer>
</configuration>

A couple of things to note about these settings. Firstly – you can add any custom cache control header setting using the cacheControlCustom attribute (e.g. no-store, must-revalidate, private, public etc). Secondly using the http 1.1 max-age setting – you can specify days – by using the 1.00 notation for the hours portion of the hh:mm:ss timespan value. So the setting above will set the max-age value to 3 days – or 259200 seconds when you see it in the response header.

Bing – Fast, Refreshing and Relaxing

Anthony Bouch — Tue, 02 Jun 2009 15:51:41 GMT

A little experimenting with Microsoft’s new search engine Bing - and I’m sold.

Even TechCrunch thinks it’s a hit.

What’s more – it comes in four different flavours!

When was the last time you searched and felt refreshed and relaxed too! Not even Google can compete with this one. Google in a bottle? Refreshing Google? Not even close...

Have a Bing today…. :-)

NOTE: Not kidding about the ‘fast’ part. Don’t know what’s up with Google over the past few months – both their regular search page and access to features such as analytics and webmaster tools are crawling. Bing – so far at least – has been super quick.

MSc in Information Security

Anthony Bouch — Wed, 20 May 2009 02:52:54 GMT

Whew – well after a tough few months I’ve finally completed the coursework towards my masters degree in Information Security - taught by the Information Security Group (ISG) at Royal Holloway, (via the University of London External System).

The online syllabus is available from the external system and the ISG also provides an overview of the programme here.

In addition to the syllabus and ISG’s introduction - I thought that a personal summary might be useful for those considering the programme.

As per the syllabus – there are four compulsory modules, two optional modules and a final project. A module is really a course, composed of roughly ten units, and lasts the full academic year – from October to the final exam in May. Here’s a description of the compulsory modules (accurate as of 2008/09 academic year) along with my two chosen optional modules.

IC01 Security Management

This course addresses the major themes of security management, including people, processes and technology with particular emphasis on the role of policy in helping to shape an organisation’s security management strategy. The ISO 27000 series of standards (ISO 27001 and 27002 in particular) are covered in detail. COBIT, ITIL, ISF SOGP along with the DPA, RIPA, and relevant EU directives are examined. So too is the impact of industry specific regulations such as Sarbanes Oxley, Turnbull, Basel II, and HIPAA. Principles of risk assessment (quantitative/qualitative), compliance, audit, and management including excellent lectures from industry leaders help to round out what is probably the most important message of the course; namely the importance of creating organisational awareness and a culture of information security that includes all aspects of the business (and not just the technology). Some of the material on the course was a little dated but relevant nevertheless. A ‘must read’ companion to this course is Bruce Schneier’s ‘Secrets and Lies.’

IC02 An Introduction to Cryptography and Security Mechanisms

This was a really great course in cryptography and security mechanisms. Pitched at just the right level for non-maths graduates the course explains the roles of all the major cryptographic primitives, including symmetric key cryptography (block and stream ciphers), hashes, message authentication codes, asymmetric (public) key cryptography and digital signatures. Historical algorithms along with well known algorithms like DES, AES, RSA, Diffie-Hellman and others are explained in detail (including worked examples of RSA modulus, public and private key calculations). The use of these primitives as mechanisms for providing higher level security services like data integrity, data origin authentication, entity authentication (unilateral and mutual authentication protocols) and non-repudiation is also explained in detail. If you can get a copy – you really should ready Stephen Levy’s book ‘Crypto - Secrecy and Privacy in the New Code War’ for a brilliant side-by-side narrative of everything that happened in the crypto world from 1975 onwards. It puts the course into perspective and explains exactly what was happening from the 70s to the 90s in the struggle between governments attempting to regulate crypto, and the commercial interests of the private sector. Simon Singh’s ‘The Code Book’ is also another great companion read.

IC03 Network Security

Also a great course on the fundamentals of network communications security including ISO 7498-2, as well as detailed analysis of network, transport, and application level security protocols (IPSec, SSL/TLS, SSH, Kerberos etc.) Other unit subjects included biometrics, email security, wireless networking, firewalls, GSM/UMTS security and an introduction to intrusion detection systems (IDS). One of the texts for this course is Douglas Comer’s brilliant book - 'Computer Networks and Internets (5th Edition)'.

IC03 Computer Security

A bit like IC02 above – there was some important history in this course in terms of the evolution of computer security, operating systems and formal security models including Bell Lapadula, Biba, Chinese Wall, and Clark-Wilson. For me at least – a real eye opener and it filled in a lot of important gaps – especially in terms of access control and some of the things I’d previously worked on in the areas of authorisation and workflow. Units on security in Pentium architecture, Unix/Linux, IBM z/OS and Windows helped to round off theory with real world and practical examples of applied computer security.

OPT5 Secure Electronic Commerce and Other Applications

I’d assumed (which turned out to be partially correct) that this course would be close to my comfort zone given my background in Web and e-commerce development. The course was ok – but I think could be structured better. It’s a pretty large subject (especially the ‘other applications’). The course’s primary objective is to link security specific methodologies to the traditional software development lifecycle as well as emphasise the different security approaches that might be required depending on the context of the application. Information flow analysis, threat modelling and application level risk assessment are covered in the opening units. The TETRA system is used as an initial case study, followed by units on Web application security, Web services security and an excellent unit on identity management. The final units introduce smart cards and the EMV standard. Lots of help from IC02 in this module as secure protocols are described in several units.

OPT12 Smart Cards/Tokens Security and Applications

This course was new to the distance learning programme this year (although taught at the ISG for several years). The accompanying text – ‘Smart Cards, Tokens, Security and Applications’ edited by two RHUL faculty members, Keith Mayes and Konstantinos Markantonakis – is a tour de force on all things in the smart card world. I’ll never look at my ‘chip and pin’ credit card the same way again. A really great course, benefiting hugely from being fresh and up-to-date. Topics included smart card production, smart card development, smart card for mobile communications (GMS/UMTS), banking and finance (EMV), RFID and contactless smart cards, smart cards in video broadcasting (DVB), an introduction to trusted platform modules (TCG/TPM), the Common Criteria (ISO 15408) for evaluation assurance, ID cards, e-passports and a fascinating unit on smart card attacks and countermeasures from Jacques Fournier at Gemalto.

In Summary

Looking back on the last two years, there were times when I wasn’t entirely sure how it was all going to fit together. With the benefit of hindsight I can clearly see the aims of the programme, and how the ISG has attempted to strike a balance between history, fundamentals, and applied techniques in information security.

The ISG has also put a lot of work into creating the distance learning version of this degree. They’ve experimented with multimedia and an online format which for the most part works well (although I still lean towards printed material and my trusty highlighter). Some of the material however (in particular a couple of the lectures in IC01) is starting to show its age and could do with a refresh – or at the least with a yearly update to the introduction for each module.

The course has totally changed the way I view the ‘information age’, information security, and privacy (having now taken the ‘red pill’). I’m extremely glad I enrolled and looking forward to putting what I’ve learned into practise on future projects. Also looking forward to taking my time and working on my final project at a more relaxed pace over the next year or so.

To be continued…

Windows 7 RC 7100 Upgrade from Vista

Anthony Bouch — Thu, 14 May 2009 05:55:27 GMT

Well I figured since the RC was generally available – I’d test the upgrade process from Windows Vista to Windows 7. I chose my ThinkPad T61p as the victim. This is my fully loaded portable dev environment including BitLocker encrypted volumes - and I was a little sceptical at first. A new OS usually means re-paving the machine, but I thought I’d try the upgrade and see how it went.

Get Ready

After a complete system backup to a removable drive I launched the setup program from the DVD – from within Vista (not via a booted DVD). Windows 7 setup reported MagicISO, LifeCam, and Skype as software that might not work after the upgrade – so I uninstalled these first. BitLocker had to be disabled as well (not volume decrypted – just disabled). SQL Server 2005 was also on the list of ‘might not work’ after upgrade, but I decided to try upgrading to SQL 2008 Developer Edition afterwards.

Make Yourself a Coffee

First word of advice – make yourself a coffee – grab some donuts too – and find something to read, or watch. The upgrade took over three hours to complete. The ‘Gathering files, settings, and programs’ stage took over an hour alone. At 18% of ‘Expanding Windows files’ – the PC rebooted and continued to expand files – but not before 30 minutes of ‘Gathering additional information before expanding files…’

Transferring files, settings and programs – 1,138,187 files, settings, and programs to be exact – took another hour (and two reboots).

Incredibly – it worked!

First Impressions

I’ve not been following the Windows 7 story that closely – but most of the reviews report performance improvements, and I like the new task bar and general UI improvements a lot. Will be interesting to see how it goes over the next few days of general use and abuse. A couple of noticeable changes though – disabling Windows Search – also removes the search function from the start menu – it didn’t used to. Also Superfetch – which was on my list of disk thrashing culprits, is enabled by default – and so I’ve turned this off again (the I/O at start-up because of Superfetch was horrendous in Vista – will have to experiment to see if it’s any better in Windows 7)

UPDATE: If you’ve upgraded from Vista as I have – and have previously installed a LifeCam VX-700 – uninstall the LifeCam software before the upgrade and be sure to delete the following file as well C:\Windows\inf\OEM16.INF (check that this is the LifeCam driver inf file) – otherwise the installation of the LifeCam will fail in Windows 7 when it finds the previous LifeCam inf file.

More soon…

RFIDs on the Brain

Anthony Bouch — Sun, 10 May 2009 04:59:20 GMT

Speechless really. I’m not often lost for words – but this presentation by Patrick Dixon of Siemens leaves me speechless. Thanks to RFIDs on the Brain from Boing Boing (and Douglas Rushkoff via ‘Joe’).

GZip and Deflate Compression Filter for ASP.Net MVC

Anthony Bouch — Sat, 02 May 2009 21:17:21 GMT

I’d previously created a compression processor for dynamic content – using this excellent post for QValues by Dave Transom. Also just just discovered this post on creating an action filter for compression by Kazi Manzur Rashid.

So combining the two and we have:

public class CompressFilter : ActionFilterAttribute
{
    public override void OnActionExecuting(ActionExecutingContext filterContext)
    {
        HttpRequestBase request = filterContext.HttpContext.Request;

        // load encodings from header
        QValueList encodings = new QValueList(request.Headers["Accept-Encoding"]);

        // get the types we can handle, can be accepted and in the defined client preference
        QValue preferred = encodings.FindPreferred("gzip", "deflate", "identity");

        // if none of the preferred values were found, but the
        // client can accept wildcard encodings, we'll default
        // to Gzip.
        if (preferred.IsEmpty && encodings.AcceptWildcard && encodings.Find("gzip").IsEmpty)

            preferred = new QValue("gzip");

        HttpResponseBase response = filterContext.HttpContext.Response;
        // handle the preferred encoding
        switch (preferred.Name)
        {
            case "gzip":
                response.AppendHeader("Content-encoding", "gzip");
                response.Filter = new GZipStream(response.Filter, CompressionMode.Compress);
                break;

            case "deflate":
                response.AppendHeader("Content-encoding", "deflate");
                response.Filter = new DeflateStream(response.Filter, CompressionMode.Compress);
                break;

            case "identity":
                break;
            default:
                break;
        }
    }
}

Which when applied as below to any action method will enable compression if it’s supported by the browser.

[AcceptVerbs(HttpVerbs.Get)]
[CompressFilter]
public ContentResult SiteMap()
{
    return new ContentResult
               {
                   Content = SyndicationHelper.GetAggregateSiteMap(),
                   ContentType = "application/xml; charset=UTF-8"
               };
}

Download here: QValueAndCompressionFilter.zip

Why is Google Fuzz Testing My Site

Anthony Bouch — Fri, 10 Apr 2009 17:36:49 GMT

This is weird – was looking at some of the reports in the Webmaster Tools I use from Google – and there were a lot (several hundred) Web crawl 404 not founds. Google reports that its spider was failing to find pages that I’ve never published – and which contain values that are incrementing over a range as parameter inputs in the url – 692-708 in the snapshot below.

Best ASP.Net Web Hosting

Anthony Bouch — Wed, 08 Apr 2009 13:09:20 GMT

Trying to find a good ASP.Net web hosting company that represents the balance I need between cost and features turned out to be harder than I thought.

I recently signed up with Discount ASP.Net – and you know what – for just ten dollars a month – they offer excellent value for money, great support, a good in-house control panel and good bandwidth. They’ve kept their setup very simple and standard – with just one type of account (Computer security design principle number three – assurance vs. complexity).

On a single account, you have one root site – one domain and as many additional domains pointing to that ‘same’ root content as you like. You cannot point domain names to subdirectories – so you’ll need to re-write yourself if you want another domain name pointing to different content.

But then I discovered their ‘standard’ application pool recycle settings, which are: 1) You are idle for twenty minutes, 2) the worker process exceeds a 200MB working set, or 3) CPU utilization exceeds 70% for more than 3 minutes. To be honest these are all more than adequate for a single site, blog, or the standard five page company website. But dasBlog is a little memory hungry, and with two blogs and a photo gallery – my app pool was doing flip-flops many times a day.

So I searched and searched – and was generally unimpressed with the list of hosting companies ranked by some of the reviews. Sites were either unprofessional – or there was enough mixed reviews to put me off. Others were just too vague about what was in the package and others were not on W2K8/IIS7 yet (and I need IIS7).

So it came down to ORCS Web Hosting who are in a Peak 10 data centre – and Server Intellect – who are in a data centre located in the the Infomart Hotel in Dallas – now a part of DCI Technology Holdings.

Server Intellect truly impressed me with their pre-sales support and they were super keen to get the business (even if it was just me). And their Tier 2 plan allows for multiple wwwroot directories and domain names – which is sweet. However not yet having a shared plan on W2K8/IIS7, combined with a personal recommendation for ORCS Web Hosting from this guy, swung me towards ORCS Web.

ORCS Web Platinum Shared Hosting is not that different from the Discount ASP.Net setup in so far as you get a single root site and unlimited additional domain and mail aliases, but that’s where the similarities end. The app pool has no idle time limit, no CPU limit, and an 800MB working set threshold. Data transfer allowance is 100 GB/month – and the site is in a data centre with fat pipes from…

...multiple Tier-1 Internet carriers with multiple fully-lit OCx connections.

…served in part by Level 3 Communications.

So that’s where I am now… although – it was a bumpy start. ORCS appear to have built, or had built, a new control panel which was err… buggy. Support was fine though and we got the wrinkles ironed out.

So ORCS Web hosting is my new home in cyberspace for the moment; nice view, plenty of room, and fast - will be here until the clouds come along :-)