A project I’ve been working on recently required a search feature. After a quick look around I thought I’d give a hosted solution a try – using Google Site Search.

After almost a full day of banging my head against the wall – here’s what I discovered…

I) The only difference between Site Search and Google Custom Search – is that you pay for Site Search and you get on-demand indexing – so you’re guaranteed (in theory) to have all of the pages in your site indexed.

II) Once you’ve paid for Site Search – the control panel is effectively the Custom Search Engine Search control panel – with the on-demand option present.

III) There are three ‘levels’ of control you have over the results on your site.

Here’s a quick tour of each – assuming you’ve read just enough of the docs (Here’s a link to the Google AJAX Search API) to know that you need to link to the jsapi JavaScript library – load the search api via: google.load('search', '1'); and then declare your OnLoad function .

Note – for all three options it’s a good idea to have the following style sheet at the top of all your test pages – since you get some ‘freebie’ and fast styling as a result.

http://www.google.com/cse/style/look/default.css

 

1) google.search.CustomSearchControl – with just two lines of code – you can have a custom search page up and running (well not quite two lines – but these are the most important two).

var customSearchControl = new google.search.CustomSearchControl('013652945859801643433:fdsfsqjnea4');
customSearchControl.draw('targetDiv');

The parameter to the CustomSearchControl is the CSE id for the custom search engine you’ve created – and calling ‘draw’ will render the completed form and search results area to the page. The CustomSearch control has a built in ‘searcher’ based on the settings from your CSE control panel. CustomSearchControl is however a little difficult to customize. I found it hard to turn off the default ‘search’ form and place the results where I wanted them. You can add additional searchers – like images and video search via:

customSearchControl.addSearcher(new google.search.ImageSearch(), options);

The main point to note here – is that CustomSearchControl is ‘driven’ by the CSE control panel – including filter options, look and feel etc. all based on the search engine unique ID that you’ll also find in the CSE control panel.

This is custom search with training-wheels on.

 

2) google.search.SearchControl – is the next level down in abstraction – and here’s the ‘Hello World’ example from the docs. With SearchControl – you have to add searchers to the control and then you can tell the control to ‘draw’ itself – just like with the CustomSearchControl. However it’s a little easier to customize. For example:

this.control = new google.search.SearchControl();
// set up for CSE result sets
this.control.setResultSetSize(google.search.Search.FILTERED_CSE_RESULTSET);

// configure search control
var options = new google.search.SearcherOptions();
options.setExpandMode(GSearchControl.EXPAND_MODE_OPEN);
options.setRoot(resultsDiv)

var searcher = new google.search.WebSearch();
searcher.setSiteRestriction("01365294584480dsadad3:ffsfsfseqjnea4");
searcher.setUserDefinedLabel("Search Results");
searcher.setLinkTarget(google.search.Search.LINK_TARGET_SELF);

this.control.addSearcher(searcher, options);

this.control.draw();

Note the options.setRoot(resultsDiv) highlighted line. This does two things – 1) It sends the output of the search to the ‘resultsDiv’ and 2) it turns off the submit form – which means you can now control the input to search – from your own form – placed anywhere on the page as long as you handle the OnSubmit event and call

this.control.execute(‘formInputValue’);

Again if you link to http://www.google.com/cse/style/look/default.css – you’ll get some instant results that look ok.

 

3) google.search.Search – is the lowest level abstraction – the raw searcher class – and although you still get some good stuff here, you will have to take care of rendering the results to the page. In this case you will need to handle the input and output for every class of searcher you use – Web, Local, Image or Video – all of which are derived from google.search.Search. For example…

// Our searcher instance.
var searcher = new google.search.WebSearch();

searcher.setSiteRestriction("013652945859801616623:ecbgsqjnea4");
searcher.setResultSetSize(google.search.Search.LARGE_RESULTSET);
searcher.setUserDefinedLabel("QCC Search Results");
searcher.setLinkTarget(google.search.Search.LINK_TARGET_SELF);
searcher.setSearchCompleteCallback(this, searchComplete, null);

var searchForm = document.getElementById("search");
if (searchForm.q.value) {
    searcher.execute(searchForm.q.value);
}

Here's a link to a complete example of a raw search with output being handled by the developer (which is in another area of the Google docs – the Code Playground).

While there’s more plumbing required here – you’re still off to a good start. Unlike the sample above for raw handling of the results – you can use the .html property of the result object in the results array – which will tie up again nicely with the linked style sheet - and you’ll have a ‘Goggle-like’ – good-to-go search experience in your site. Like this…

function searchComplete() {
    // Check that we got results            
    if (searcher.results && searcher.results.length > 0) {
        // Grab our content div, clear it.
        var resultsDiv = document.getElementById('results');
        resultsDiv.innerHTML = '';
        // Loop through our results, printing them to the page.
        var results = searcher.results;
        for (var i = 0; i < results.length; i++) {
            // For each result write it's title and image to the screen
            var result = results[i];
            resultsDiv.appendChild(result.html);
        }

        // Now add the paging links so the user can see more results.
        addPaginationLinks(searcher);
    }
}

And that's it. There is documentation available for all of this – but it’s in various locations. The first examples you will see if you come to search via the CSE control panel – will be the CustomSearchControl. The hard part was the ‘orientation’ exercise that I had to go through to understand what the heck I was using, where it fit in the bigger picture of the search API, and how I could create a more customized search experience. The docs are a bit all over the shop in this case – and so it took me nearly a day to write some POC tests and properly understand the relationship between google.search.CustomSearchControl, google.search.SearchControl and good.search.Search.

Hope this post helps to reduce the pain for anyone else that’s new to Google Search and would like to implement a tailored search experience.

When the desired death of a browser makes it into a full-page editorial of IEEE Spectrum – you know that the issue is finally starting to gain some traction. The tech blogosphere communities have been making plenty of noise about this for a while now. And I’d love someone to do an ‘economic’ impact analysis for IE6 – both in terms of development and security related incidents.

I think the Spectrum article (and the referring blog post from Digg) does a good job of describing the issues – corporate restrictions being high on the list of reasons that prevent a user from upgrading. That said I’d rank Microsoft’s failed Vista strategy which resulted in downgrade options to Windows XP being offered by manufacturers, along with the number of illegal and non-updating XP installations out there as just as large a contribution.

Despite my earlier and naïve attempt to ban IE6 from my site I’ve now successfully bared the browser using IE conditional comments – which is the only reliable way you can detect this version of IE (you can’t sniff for it reliably). I then display the following :-)

I posted my views earlier on Nikon software – and I’m now pretty sure that Nikon ViewNX is broken on Windows 7. Below is a link to the native Process Monitor log file – filtered for ViewNX after a complete uninstall and re-install to the latest version of ViewNX 1.5.0 – running on the RTM of Windows 7. ViewNX was still performing an endless loop of I/O and registry requests.

I stopped recording after 1.7 million events (a 182MB log file – compressed to 21MB below).

ViewNX.zip

:-(

I’ve been an IEEE and Computer Society member for a couple of years now. Trying to keep up with the IEEE publications along with my RSS Reader and tech news has been a challenge – so I’m sure I’ve missed some good stuff along the way.

However, an article in the September 2009 edition of ‘Computer’ caught my eye: “Formal Versus Agile: Survival of the Fittest” by Sue Black, Paul P. Boca, Jonathan P. Bowan, Jason Gorman and Mike Hinchey.

It’s a great article with good history and an insightful assessment of the current ‘methodologies’ landscape. Unfortunately the article is only available to subscribers of ‘Computer’ at the moment. I hope it will be published more widely in the near future since I think it has a wide appeal.

The author’s comparison of RAD and Agile was particularly interesting – and the following statement about Agile practises is worth quoting…

Most teams purporting to be doing agile software development are not applying the level of technical rigor necessary to succeed at it. Most ‘agile’ teams have actually only adopted Scrum’s project-management practise and have failed to effectively adopt ‘the hard disciplines’ like test-driven development, refactoring, pair programming, simple design (writing the simplest code possible to satisfy the customer’s requirements), and continuous integration.

Their description of the major misconception that ‘Agile = Fast’ was also insightful (when it should be about being responsive to ‘change’).

A great read if you can get hold of it.

I’m pretty sure that anyone who actually likes to ‘make stuff’ and has been a programmer at least once in their life – has got a short list of pet projects that he/she has started – and maybe even finished :-)

Here are a few of my ‘pets’.

CodeDom class generator for VS2005/2008

This was an early attempt of mine at writing a Visual Studio Add-in - designed to create a class from a database query, table or stored procedure. There’s a ton of stuff like this out there – but I wanted something that generated the class from the DataTable – SchemaTable of an IReader – so that a class could be generated from any query or shape for use with DTOs, reporting, or a simple domain model. You can choose from C#, VB or C++ to generate the class as well as a number of styles of classes – from a built-in IReader constructor, attribute based mapper, or plain old CLR object (POCO). There also doesn’t have to be any data in the result set for this to work since the SchemaTable will be returned regardless.

 

The Add-in also adds an ‘Open Folder’ option to the context menu of all solutions, projects, folders, etc. in Solution Explorer. This feature exists as of VS 2008 – but I wanted it at the top of the context menu – not at the bottom.

You can download the Add-in here at - FiftyEightBitsAddin.zip Just unzip and place the contents of the zip file into any one of the Add-in locations for VS2008.

 

PreCode Code Snippet Plugin for WLW

This is an open source project hosted at Codeplex – a plugin for Windows Live Writer that helps to format code snippets before pasting them into a blog post (or elsewhere). It was written for the excellent SyntaxHighlther from Alex Gorbatchev. It’s a WPF app – and also includes a standalone desktop version for pasting formatted code into any app (ideal for Stack Overflow). Please don’t look at this as a reference WPF app though – it’s more than just a little ‘hacky’. The app that follows is my ‘better’ WPF effort. Visit http://www.codeplex.com/precode for more details, the source code and the installer.

 

 

Gallery Builder – Desktop and Web

If you follow the Photos link above – you’ll see the results of this application. Yet another gallery builder app – however I’ve spent a bit of time on this one – choosing a specific architecture that I think represents a good balance between desktop and web server processing for small to medium size collections of photos. I’ll be hosting a Wiki soon with docs and the full download including the HttpModule that is used display galleries online (along with a sample app and extensibility features). The module can be integrated into any site and includes full Atom and Rss syndication support, nested galleries, as well as good caching/304 response support.

 

A pre-release version of the desktop application can be downloaded from the link below (although it’s not a whole lot of good without the web component). Hand rolling a complete set of control templates for a custom theme in WPF was a great way to get to know the inside of WPF.

NOTE: If there are no other WPF apps on your machine – and this application is the first WPF application you start – the ‘cold’ loading time for a WPF app is very long. Subsequent application loading times are what you’d expect.

GalleryBuilder_0.9.9.1000.zip

 

Yet another Task Management App - DotNetTime

Yet another task/project list manager; only I actually use this one. This is an ASP.Net Model View Presenter application (and hopefully my last Webforms app). I use this to track my own time working on projects or just to ‘monitor’ my own time in general. Implementing multi-language support was a good learning exercise – and until recently this had been my ‘workbench’ throw anything at it hobby app. Now that charting controls are available free from Microsoft – I’ll be adding some charts to the reporting page. Feel free to register and have a play – it’s very basic – but it does what I need. Be gentle too if you’re into web hackery– this was built before my formal introduction to the world of application security and so I wouldn’t be surprised at all if there were XSS or CSRF vulnerabilities (although there shouldn’t be any SQL injection weaknesses – he says).

http://www.58bits.com/dnt

Thanks to this post at .Net Dojo, here’s a utility method for getting a safe value from an XElement, returning a default value if the element was not found.

public static T GetElementValue<T>(XElement parent, string elementId, T defaultValue) where T : IConvertible
{
    XElement element = parent.Element(elementId);
    if (element != null)           
        return (T)Convert.ChangeType(element.Value, typeof(T), CultureInfo.InvariantCulture);           
    else
        return defaultValue;   
}    

I’ve been using this combined with .Net 3.0 object initializers to re-hydrate objects from Xml with safe values, like…

var myClass = new MyClass() 
{ 
    Name = GetElementValue(parent, "Name", String.Empty),
    IsDefault = GetElementValue(parent, "IsDefault ", true),
    YearsLeft = GetElementValue(parent, "Yearsleft", 100)
}

Earlier I posted about a compression filter for ASP.NET MVC that was the product of two other posts on compression and QValues. I’ve also seen a couple of posts describing different approaches to creating syndication content via ASP.NET MVC – RSS or Atom feeds. Here are two suggestions – one at DeveloperZen and the other at Stack Overflow that both use an RssActionResult subclass of ActionResult.

What was missing was a 304 Not Modified filter – so that feeds that have not changed can return a 304 Not Modified HTTP Response - saving bandwidth and improving performance of the consumer.

Here’s my attempt at putting it all together using a custom NotModifiedFilter, the CompressionFilter and a SyndicationActionResult class.

First the NotModifiedFilter. This article at the Embarcadero Developer Network was very helpful (along with the W3C standard for 304 Not Modified).

Here’s the filter.

public class NotModifiedFilterAttribute : ActionFilterAttribute
{
    public override void OnResultExecuted(ResultExecutedContext filterContext)
    {
        var response = filterContext.HttpContext.Response;
        var request = filterContext.HttpContext.Request;           

        if ((IsSourceModified(request, response) == false))
        {
            response.SuppressContent = true;
            response.StatusCode = 304;
            response.StatusDescription = "Not Modified";
            // Explicitly set the Content-Length header so the client doesn't wait for
            // content but keeps the connection open for other requests
            response.AddHeader("Content-Length", "0");
        }
    }
}

And here’s the helper method that tests for modified content.

private static bool IsSourceModified(HttpRequestBase request, HttpResponseBase response)
{                  
    bool dateModified = false;
    bool eTagModified = false;

    string requestETagHeader = request.Headers["If-None-Match"] ?? string.Empty;
    string requestIfModifiedSinceHeader = request.Headers["If-Modified-Since"] ?? string.Empty;            
    DateTime requestIfModifiedSince;                    
    DateTime.TryParse(requestIfModifiedSinceHeader, out requestIfModifiedSince);
    
    string responseETagHeader = response.Headers["ETag"] ?? string.Empty;
    string responseLastModifiedHeader = response.Headers["Last-Modified"] ?? string.Empty;
    DateTime responseLastModified;
    DateTime.TryParse(responseLastModifiedHeader, out responseLastModified);

    if (requestIfModifiedSince != DateTime.MinValue && responseLastModified != DateTime.MinValue)
    {
        if (responseLastModified > requestIfModifiedSince)
        {
            TimeSpan diff = responseLastModified - requestIfModifiedSince;
            if (diff > TimeSpan.FromSeconds(1))
            {
                dateModified = true;
            }
        }
    }
    else
    {
        dateModified = true;
    }

    //Leave the default for eTagModified = false so that if we
    //don't get an ETag from the server we will rely on the fileDateModified only           
    if (String.IsNullOrEmpty(responseETagHeader) == false)
    {
        eTagModified = responseETagHeader.Equals(requestETagHeader, StringComparison.Ordinal) == false;
    }

    return (dateModified || eTagModified);    
}

And lastly – here’s my SyndicationActionResult class which uses a Func<FeedData> delegate to get the feed data – and so can be used for any type of feed – Rss, Atom, Sitemaps etc. (FeedData contains the feed content as well as the last modified date and the ETag). Not sure if I’m guilty of cargo cult programming here since I was a little unsure about setting the ‘Last-Modified’ header manually instead of using response.Cache.SetLastModified() method. However the latter was not showing up in the HttpContext of the OnResultExecuted method in the filter. Maybe the runtime moves this value into the headers collection later in the pipeline.

public class SyndicationActionResult : ActionResult
{
    public Func<FeedData> ActionDelegate { get; set; }
    
    public override void ExecuteResult(ControllerContext context)
    {
        if (ActionDelegate != null)
        {
            var response = context.HttpContext.Response;
            var data = ActionDelegate.Invoke() as FeedData;
            if (data != null)
            {
                response.ContentType = data.ContentType;
                response.AppendHeader("Cache-Control", "private");
                response.AppendHeader("Last-Modified", data.LastModifiedDate.ToString("r"));
                //response.Cache.SetLastModified(data.LastModifiedDate);
                response.AppendHeader("ETag", String.Format("\"{0}\"", data.ETag));
                response.Output.WriteLine(data.Content);
                response.StatusCode = 200;
                response.StatusDescription = "OK";
            }
        }
    }
}

Note that we write the content to the response.Output stream - however if the NotModifiedFilter tells us nothing has changed – then this will be suppressed – (although it is important that the ETag remains).

And very lastly - here's the Action in my SyndicationController class that puts it all together, and in this case – is used to provide the aggregate Atom feed on my home page at http://www.58bits.com.

[AcceptVerbs(HttpVerbs.Get)]
[NotModifiedFilter(Order = 1)]
[CompressFilter(Order = 2)]
public SyndicationActionResult SiteFeedAtom()
{
    return new SyndicationActionResult() { ActionDelegate = SyndicationHelper.GetAggregateAtomFeed };
}

As Scott sometimes says -  it may be  “poo”… or it maybe useful. At least it seems to be working ok for me, although suggestions on how any of it might be improved would be greatly appreciated.

I’m still freak’n over how solid, fast and all-round great Windows 7 is. The first time ever an OS upgrade (without repaving the machine) actually feels like I did a fresh install.

Vista drove me nuts – both because of its sluggishness, and because of a totally failed file and folder management strategy. At the height of my frustration I was experimenting with Total Commander, Opus 9 and a couple of other file management applications in an effort to rid my self of the the ‘built in’ windows explorer. Folders did not remember the view or view preferences between sessions. Even worse was Vista’s attempt to guess the contents of a folder and ‘adapt’ accordingly. Maybe one day when we’re all in the cloud – or when the file system has been abstracted away by a content and keyword management system we won’t need conventional access to the file system per se – but we’re not there yet – and I do.

Here’s what they’ve done in Windows 7 – which is exactly right. They’ve separated their attempt at managing views based on content – from a regular file management window (and all explorer windows always remember the last view used).

So for example – if I’m looking at photos via the Picture Library – it looks like this.

…with extra columns for tags and picture information. It’s a media specific view and that’s fine. Of course it could be a thumbnails view – but the important thing is that in details view – content specific columns appear.

If I need a ‘real’ file system window though – say because I want to quickly sort by file type or some other file system criteria – then I navigate to the same folder via the file system using a ‘regular’ explorer window (Windows Key – E) – which looks like this.

The default view for folders when you’ve entered them not via a Library but via the file system – is the ‘General Items’ view – which works like you’d expect it to when you need a ‘regular’ file management window.

What a relief. It works. It makes sense, and I can use Windows Explorer again.

UPDATE 12/07/2009 - As Shirley Boyle sang - "I dreamed a dream". The exercise below was great for learning how to use the IIS7 Url Rewrite Module - but there is a long list of user agents that contain the string "compatible; MSIE 6.0;" including the MSN spider and some transparent proxies. Internet Explorer 6 is the browser that refuses to die. :-(

ORIGINAL POST - This one took a little while to figure out along with a careful read of the reference docs for the IIS7 Url Rewrite Module. I was banging my head against the wall for a few minutes with {USER_AGENT} before reading the fine print… the HTTP prefix bit in particular.

All dash (“-”) symbols in the HTTP header name are converted to underscore symbols (“_”).
All letters in the HTTP header name are converted to capital case.
“HTTP_” prefix is added to the header name.
For example, in order to access the HTTP header “user-agent” from a rewrite rule, you can use the {HTTP_USER_AGENT} server variable.

And here’s the result…

<rule name="BlockIE6" stopProcessing="true">
  <match url=".*" />
  <conditions>
    <add input="{HTTP_USER_AGENT}" pattern="MSIE 6" />
    <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" />
    <add input="{URL}" negate="true" pattern="ie6" />
  </conditions>
  <action type="Redirect" url="ie6" redirectType="Permanent" />
</rule>

It’s really really important that you do two things: 1) Prevent the redirect from redirecting static content – like script files and style sheets, ala the {REQUEST_FILENAME} negative condition, and 2) You also check the Url to see if it’s the redirected Url – i.e. they’ve already been redirected – otherwise you’ll get an endless redirect loop. In this case I’m sending them to the url “ie6” which is an action on a route in ASP.Net MVC – displaying a download page with some ‘alternatives’ :-)

I recently watched Luca Bolognese’s presentation at last years PDC on F#. A really great presentation.

I’ve also been listening to Steve Vinoski’s podcasts on the The Functional Web.

There’s definitely stuff up in this area now. Coming from a non-comp-sci background – and only ever having written statically typed and imperative code (sad I know) – I’m curious to see where this will all lead. Haskell , Erlang, and Scala are all on the rise. CouchDB (backed I think by Lucene for search) and even Yaws (yet another web server) – are just two of several significant projects written in Erlang.

I think it was Martin Fowler who said that the best way to be a better programmer – is to learn another language, and F# is now on my list of things to do.