December 6, 2010

Exception data evolved

As detailed in my earlier post about exception signatures I and my colleagues take exceptions from our production servers seriously. In addition to grouping them by signatures (which helps a lot and makes triage much more pleasant) and logging them in our internal bug tracking software we also try to add relevant debug information whenever we throw an exception.

The BCL team in their infinite wisdom added the Exception.Data property. This property is simply an IDictionary which allows storing key and value pairs of any type. By default this is an empty collection which means you don’t have to worry about it being null.

Typical usage

public void LogonUser(string username, string password, string domain)
{
    try
    {
        DataStore.LogOnUser(username, password, domain);
    }
    catch (Exception exc)
    {
        var ae = new ApplicationException("Underlying logon failed, see InnerException", exc);
                
        ae.Data["username"] = username;
        ae.Data["domain"] = domain;

        throw ae;
    }
}

When the exception reaches our internal trac site it might look something like the image below (in our case the data dictionary will actually be added as a comment but you get the point).

example-data

This has proven to be is incredibly useful for debugging purposes but it is a bit tedious to actually write the code. Storing the exception in a variable just to get access to the data property just didn’t feel right to me and writing the same boilerplate .Data[“xyz”] = xyz was just boring.

Step 1 – The AddData extension method.

I started off by creating an extension method for System.Exception called AddData. AddData looks a bit like this (very simplified, see the end of the post for the real deal)

public static Exception AddData(this Exception exception, string key, object value)
{
    exception.Data.Add(key, value);
// whohoo, chaining! return exception; }

This allowed me to save a few keystrokes over the first method and I would end up with something like this instead

catch(...) {
    throw new ApplicationException("Underlying logon failed, see InnerException", exc)
        .AddData("username", username)
        .AddData("domain", domain);
}

It might not look like much but it sure helps when you want to add debug data to an already existing exception, it saves you from having to store the reference and so on but it still doesn’t solve the problem with having to write the argument names twice. This isn’t only a nuisance when writing the code; it’s very easy for the key and value to get out of sync if you’re doing refactoring since refactoring tools will only change the variable name, not the key.

My ideal solution would be something like this

catch(...) {
    throw new ApplicationException("Underlying logon failed, see InnerException", exc)
        .AddData(username)
        .AddData(domain);
}

And have the AddData method automatically infer the proper key name but since that’s not possible I had to find another way.

Step 2 – Taking a cue from ASP.NET MVC

We continued using the AddData extension method above for a couple of weeks before it dawned on me that I could use anonymous types to skip the parameter name duplication. Anonymous types have the great ability of being able to “infer” property names when initialized.

var x = new { username = username, password = password }
// Is the same as
var x = new { username, password }

With this in mind I wrote an AddData extension method which accepts a single object and then uses reflection to iterate over all properties and adds their property names and values into the exception data dictionary.  This allowed me to rewrite my code yet again.

catch(...) {
    throw new ApplicationException("Underlying data store logon failed, see InnerException", exc)
        .AddData(new { username, password });
}

Neat, isn’t it? This is the exact same technique that ASP.NET MVC uses for route-declarations, attributes in html helpers and more.

Reflection? Isn’t that horribly slow

Not like in the olden days. It comes with a cost but you shouldn’t be too worried since you’re probably not throwing exception often enough for it to matter anyway (and if you are, then you have bigger problems).

Teh codez

using System;
using System.ComponentModel;
using System.Diagnostics;

namespace freakcode.Extensions
{
    /// 
    /// Extension methods related to instances of System.Exception and inherited objects.
    /// 
    public static class ExceptionExtensions
    {
        /// 
        /// Adds the supplied debug data to the exceptions data dictionary and returns
        /// the exception allowing chaining.
        /// 
        /// The exception type, you should not need to specify this explicitly
        /// The exception.
        /// The key of the debug value to be inserted into the exceptions data dictionary.
        /// The value to be inserted into the exceptions data dictionary.
        /// key is null
        /// An element with the same key already exists in the Data dictionary
        public static T AddData<T>(this T exception, string key, object value) where T : Exception
        {
            if (exception == null)
                throw new ArgumentNullException("exception");

            if (key == null)
                throw new ArgumentNullException("key");

            /* Key or value is not serializable (or key is null). The default internal structure which
             * implements the IDictionary is going to throw an exception in Add() so instead of 
             * throwing another exception while preparing to throw the first one we silently ignore the
             * error. Unless we're building in debug mode that is, then we'll fail. */
            if (value != null && !value.GetType().IsSerializable)
            {
                Debug.Fail("Attempt to add non-serializable value to exception data");
            }
            else
            {
                exception.Data.Add(key, value);
            }

            return exception;
        }

        /// 
        /// Adds the each property name and value from the supplied object to the exceptions data dictionary and returns
        /// the exception allowing chaining.
        /// 
        /// The exception type, you should not need to specify this explicitly
        /// The exception.
        /// An object from where properties will be read and added to the exception debug data collection.
        /// key is null
        /// An element with the same key already exists in the Data dictionary
        public static T AddData<T>(this T exception, object values) where T : Exception
        {
            if (values == null)
            {
                // Some really nasty things can happen if you start throwing exceptions in the middle
                // of throwing exceptions so unless we're in debug more we'll just silently ignore it.
                Debug.Fail("Argument 'values' was null!");
            }
            else
            {
                foreach (PropertyDescriptor descriptor in TypeDescriptor.GetProperties(values))
                    exception.AddData(descriptor.Name, descriptor.GetValue(values));
            }
            
            return exception;
        }
    }
}
kick it on DotNetKicks.com

Licensing information

December 5, 2010

Calculating week numbers in Google Spreadsheets Revisited

In my previous entry Calculating week numbers in Google Spreadsheets I explored and presented a way of retrieving the week number of a given date. A couple of weeks ago a guy named Frederik Claes send me the best bug report from a non-programmer that I've ever received. He pointed out an error in an edge-case date that wasn't included in the test suite. After having looked at the problem and read up on the problem I found two errors.

First: the algorithm

The algorithm I used didn't comply with the specifications and for some dates it would return an invalid week number. While it had the advantage of being easy to read and verbose it was obvious that it would have to be replaced. An improved algorithm is currently being discussed on the talk-page of the wikipedia article and after a couple of verifications I ended up reimplementing my script using it instead.

Second: Time zone conversion

This is a classic example of a Works On My Machine problem. It turns out (and for good reasons to) that Google Apps Scripts run in a (potentially) different timezone from the actual spreadsheet. This is a major problem when dealing with dates since what looks like 2009-01-01 in the spreadsheet could very well have been converted to 2008-12-31 when it reaches the script. While I haven't had the time to fully investigate the possible workarounds I think I've found an acceptable way of transforming the date within the script to it's proper timezone.

The script

I've sent the script to Google for review so hopefully it'll be available through the Apps Script gallery within a couple of days. Until then you can copy the script from this blog and insert it to your sheets.

// by Markus Olsson (http://www.freakcode.com). 
// See http://www.freakcode.com/projects/google-spreadsheet-week-numbers
// The script is licensed under the MIT License. Basically you're free to do whatever you want with it. Attribution not necessary but appreciated.

// Update 2011-05-02:
//   Now returns current week when no arguments are given.
//   Adding attempt to automatically convert non-date objects, see code
//   Using 2 space indentation since that seems to be the proper indentation level for apps scripts
//
// Update 2010-12-05: 
//   Using the algorithm suggested by DR J R Stockton on http://en.wikipedia.org/wiki/Talk:ISO_week_date#Algorithms and
//   some of the JS implementation provided by DR Stockton on http://www.merlyn.demon.co.uk/weekcalc.htm#JS.
//
//   This fixes the erroneous week numbers reported by Frederik Claes.
//
//   Also fixed proper conversion between spreadsheet and script time zones since scripts now have explicit 
//   time zones (they didn't when I first started this script).
//
// This method is provided as an easy to use extension to the function namespace of Google Spreadsheet. 
// Since it was written a couple of people have reported that it is possible to get the week number from a
// date cell without resorting to scripts formula:
//    =VALUE(MID(TEXT( A1; "yyw") ; 3 ;5)) 
// Use whichever method you prefer, I think =WeekNumber(A1) is more readable but that could be because of the
// work I've put in into this script ;)
//
function WeekNumber(date, inIsoStringFormat)
{
  // Allows this method to be called without arguments to return the
  // current week. Shorter version of =WEEKNUMBER(TODAY())
  if(arguments.length < 1)
    date = new Date();

  // For a short while when I first started writing this script date cells 
  // wasn't given as pure js date objects so you had to convert them first. This 
  // hasn't been an issue for me for ages but some comments on my blog suggests 
  // that explicitly convertering into a Date instance has helped them. Maybe they're
  // using old spreadsheets, maybe not. But I can't foresee that adding this extra 
  // safeguard should cause any major problems (famous last words).
  if(Object.prototype.toString.call(date) !== '[object Date]')
    date = new Date(date);

  var activeSpreadsheet = SpreadsheetApp.getActiveSpreadsheet();
  var spreadsheetTimeZone = activeSpreadsheet.getSpreadsheetTimeZone();  

  // Google apps will automatically convert the spreadsheet date into the timezone
  // under which the script is running, this will revert it to the spreadsheet date and
  // at the same time truncate hours, minutes and seconds.
  date = new Date( Utilities.formatDate(date, spreadsheetTimeZone, "MMM d, yyyy") );
    
  // Get the week day where 1 = Monday and 7 = Sunday
  var dayOfWeek = ((date.getDay() + 6) % 7) + 1;

  // Locate the nearest thursday
  date.setDate(date.getDate() + (4 - dayOfWeek));
  
  var jan1 = new Date(date.getFullYear(), 0, 1);

  // Calculate the number of days in between the nearest thursday and januari first of 
  // the same year as the nearest thursday.
  var deltaDays = Math.floor( (date.getTime() - jan1.getTime()) / (86400 * 1000) )

  var weekNumber = 1 + Math.floor(deltaDays / 7);
  
  if(inIsoStringFormat)
    return jan1.getFullYear() + "-W" + (weekNumber < 10 ? "0" + weekNumber : weekNumber);
        
  return weekNumber;
}

January 20, 2010

Calculating week numbers in Google Spreadsheets

I've fixed a couple of bugs and made some changes to the script. The code on this page is obsolete and only preserved for historical purposes. Please see my updated blog post for the updated code

I use an excellent little app called TimeTracker to keep track of how much time I spend at work. When I've collected too much overtime I try to compensate by taking a day off (unfortunately for me I haven’t exactly been great at balancing this in the past).

TimeTracker has an export feature that mails me a CSV file with all the data I need to produce a report suitable for showing to my boss should he ever complain about me not working enough.

I use the Google Spreadsheets to input and calculate how much I've been working. The problem is that I wanted to sum up how much I work in any given week but I couldn't find a function that, given a date, returns the ISO 8601 week number.

Writing it myself: The bad way

My first attempt at writing it myself was both messy and naive

=ROUNDDOWN((INT(B2-DATE(YEAR(B2);1;1))+(WEEKDAY(DATE(YEAR(B2);1;1);3)))/7)+1)

While this function has worked for me during the few months that I've been using it it won't work properly across new years eve, it doesn't take leap years into account and I'm sure there's a bunch of other things that's wrong with it.

But that was before I found out that it is possible to write Google spreadsheet functions in JavaScript! It's called Google Apps Script and it's JavaScript which you can write that gets executed server-side in the Google cloud.

Please note that unfortunately this only seems to be available to Google Apps users (i.e. not for standard Google accounts) at the moment. There seems to be some kind of a petition for opening up the scripts feature to the public that you might want to check out.
- As of 2010-03-11 Apps script is available to all docs users!

Writing it myself: The right way

The intertubes are littered with lots and lots of examples of how to calculate week numbers in JavaScript but very few of the nifty ones (the one-liners) seem to properly handle the 2010 transition correctly (2010-01-03 is week 53 and 2010-01-04 is week 1) so I decided to more or less translate the pseudo code at Wikipedia to JavaScript and ultimately Google Apps Script.

The solution is quite verbose and there's a couple of tricks that I haven't bothered implementing. But it works.

This code is obsolete and will fail for certain dates. It is only preserved for historical purposes. Please see my updated blog post for the updated code

// Built from pseudo code at:
// http://en.wikipedia.org/wiki/ISO_week_date#Calculating_the_week_number_of_a_given_date
// by Markus Olsson (http://www.freakcode.com). See http://blog.freakcode.com/2009/12/calculating-week-numbers-in-google.html.
function WeekNumber(inDate)
{
    function IsLeapYear(year)
    {
        return (year % 4 == 0) && ((year % 100 != 0) || (year % 400 == 0))
    }

    function OrdinalDays(inDate)
    {
        var ordinalDatesPerMonth = [0, 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334];
        var ordinalDatesPerMonthInLeapYear = [0, 31, 60, 91, 121, 152, 182, 213, 244, 274, 305, 335];

        var ordinals = ordinalDatesPerMonth;

        if(IsLeapYear(inDate.getFullYear()))
            ordinals = ordinalDatesPerMonthInLeapYear;
  
        return ordinals[inDate.getMonth()] + inDate.getDate();
    }
      
    // Get the week day where 1 = Monday and 7 = Sunday
    var weekDay = ((inDate.getDay() + 6) % 7) + 1;
    var ordinal = OrdinalDays(inDate);

    var weekNum = Math.floor(((ordinal - weekDay) + 10) / 7);

    // If the week number equals 0, it means that the 
    // given date belongs to the preceding (week-based) year.
    if(weekNum == 0)
        return 53;

    // If 53 is obtained, we must check that the date is not actually 
    // in week 1 of the following year.
    if(weekNum == 53 && weekDay < 4)
        return 1;

    return weekNum;
}

I've made a small test suite available as a spreadsheet.

Trick

If you've got lots and lots of cells performing week number calculations you can select the cells, hit CTRL+C (or copy through the menu) then only paste the value (Edit menu -> Paste values only). There's really no need to calculate it over and over again since the likelihood of a major change to the ISO spec is pretty low ;)

Update 2010-01-20 (2010-W3): Google made some changes to Google Apps Scripts that broke the script. The sample above and the test suite has been updated.

Update 2010-03-11 (2010-W10): Google released the Google Apps Scripts gallery today and simultaneously enabled apps script for regular docs users. The apps gallery makes it possible to publish apps script! Great work by the Apps script team! I've submitted the week number script to the gallery and I'm currently waiting for them to approve it.

Update 2010-03-16 (2010-W11): Got an approval mail from Google, the script is now available in the script gallery!

weeknumbers-in-google-apps-script-gallery

Licensing information

January 8, 2010

Pretty automatically updating lighttpd server status or “mod_status on steroids”

At work we use the excellent lighttpd (or lighty) web server as a frontend reverse proxy that performs SSL offloading, and  some other neat tricks before our visitors reach our backend servers. It’s been working like a charm.

We’ve been using mod_status to keep track of how much traffic goes through our frontend server. mod_status is one of those little apps that gets the job done but not so much more.

server-status

It contains all the relevant information but I wanted to display this data on a separate monitor to as a way to keep track of how much traffic we where getting in real time.

Since it’s a static html page there wasn’t much interactivity there. So I whipped out my vim-editor and wrote a little helper page that dynamically polls data from the server-status page and displays it in a little more full-screen friendly way. You can see the result below (click on it to enlarge). It will automatically refresh the data once every 5 seconds.

server-status-pretty

Go ahead and download the source. It’s a single html file and all you have to do is to configure your lighty server so that the server status is enabled for a directory to which you’ve got write access to. Ours looks like this:

$SERVER["socket"] == "192.168.30.1:80" {
    status.status-url = "/server-status"
    server.document-root = "/var/www/status"
    mimetype.assign = ( ".html" => "text/html" )
    index-file.names = ( "pretty-status.html" )
}

Then you simply put the file in that page and it should just magically work. Note that the file loads the Google hosted jQuery library so if you don’t have access to the internet (on the machine your viewing from) you’ll have to download the jQuery script and place it along side the page.

Things on the todo list include pretty error messages when the data can’t be fetched and perhaps a nice little demo-mode.

[Download the source], [“Project page at freakcode.com”]

Licensing information

November 23, 2009

Google Maps version 3 intellisense

Intellisense is one of those killer-features of Visual Studio. I love that I can just drop in an assembly in VS and start poking around in an editor to get a feel of the API. If the API i properly documented and layed out in a .netty way I can immediately start working with it.

The flipside of intellisense is that once you get used to it you feel severely limited when you encounter a project without intellisense. That's what happened to me today.

I was integrating the excellent Google Maps library into one of our websites and decided that I should use the latest and greatest release - Google Maps v3. I fired up VS and started hacking away in a .js file until i got flashbacks from back in the days when I developed PHP. I was toggling between writing code in my IDE and reading the api documentation in my browser. It was horrifying!

-vsdoc to the rescue

At some point I realized that since I can get intellisense for jQuery I should be able to get it for Google Maps as well so I started googling and found this single commit project at codeplex that did offer an intellisense file for Google Maps v2 but not for v3.

That's when I started looking at writing my own intellisense file. I tried to validate the Google API reference in the W3 Validator only to find out that the king of all web doesn't always produce the best markup.

Not to fear; I've written a couple of screen scraping utilities before and I know that there's one project that makes dealing with badly formatted html a breeze: Html Agility Pack. The agility pack parses almost any amount of bad markup and gives you a nice little XPath interface for poking around inside the DOM.

The result

After some hackish (screen scrapers are always messy) code I was able to produce a basic intellisense file that covers most of the Google Maps API.

The files

The js intellisense file and usage instructions can be found over at http://www.freakcode.com/projects/google-maps-v3-vsdoc.

kick it on DotNetKicks.com

Licensing information

July 25, 2009

Introducing exception signatures

At work we develop, maintain and host a web solution which consists of a fair amount of assemblies and spans across several websites. Although we might like to think that we develop perfect code that certainly isn't always the case. And as with all non-djb applications ours sometimes generates a few exceptions.

Now please don't think of us as bad people, exceptions are an unfortunate part of nature and not all exceptions are the result of our bad programming, quite often we get exceptions from external libraries, from asp.net (viewstate exceptions, forbidden input exceptions and so on).

Keeping track of exceptions

Quite early on in our development process we decided that we needed a way of reacting to these exceptions as they happened. Our solution was to implement global exception handlers which sent an email to us whenever an unhandled exception occurred.

This worked great and in our early days it was more than enough. We could immediately react to an exception and we could often deploy a fix rapidly.

If you don't keep track of your exception at all you're probably in trouble. See this post by Jeff Atwood for a better explanation of why than I could ever come up with.

Keeping track of large amounts of exceptions

When our sites had been live for a while and started to get some serious use we noticed that it was getting really hard to keep track of all incoming exception mails. From looking at the stack trace we had no idea if an exception had been resolved, if someone was working on it, it's priority and so on. Troubled by the need to keep a bunch of stack traces in (my) memory I started thinking about ways of solving the problem.

My idea was to convert the exceptions into issue tickets. We use Trac at work so I started looking at simply redirecting all of our exception email there. Getting the exception mails converted to tickets wasn't really a problem but it didn't help at all with organization since every exception spawned a new ticket and we still had to manually merge the duplicates.

Enter Exception Signatures

In order to enable organization I started looking at generating a fingerprint for every exception. Some sort of string generated by looking at the stack traces of the exception, its inner exceptions and meta information of each exception. I wanted a solution which could be intelligent enough to differentiate between two exceptions thrown from the same place but for different reasons (i.e., the algorithm should look at the entire exception chain instead of only the base exception target site).

Handling dynamic data in exception messages

I also wanted it to be able to detect and disregard dynamic data in exception messages. Ever heard of Exception.Data? That's where we store extra information that we want to tack along for debugging purposes. Exception.Data is great but sometimes it's to much work and you just want to throw in some debugging data in the Exception.Message string like this:

public void RefundPurchase() {
    if(PurchaseState != "completed")
        throw new InvalidOperationException(
            "Invalid state for refund: " + PurchaseState
        );
}

In order to handle these cases I made the algorithm aware of three "pretty standard" ways of including dynamic data in exception messages.

  1. Text after a colon ':'. Invalid state for refund: inprogress
  2. Text within single and double quotes. Invalid state for refund "inprogress"
  3. Text within balanced parenthesis, brackets and braces. Invalid state for refund (state was inprogress)

Handling localized exceptions

I also wanted localized exceptions to produce the same signature when possible. We have a windows app used by some of our customers and we wanted exceptions from them to produce the same signature regardless of their OS language. To accommodate this I added some custom code for exceptions which contain an ErrorCode (SocketException and Win32Exception) and use that instead of the message.

Human semi-friendly signatures

My last requirement was that the signatures generated would be short so that we could display them to our users when they encounter one of our error-pages. This enables our customer service to provide status updates to calling customers by simply looking up the signature in trac and looking at its status.

As such the signatures are no-crazy-signs 8 character hex strings, like "617c1de5" and "1570d6d7"

Conclusion and possible solution

I think I've covered all my requirements and I must say that it has worked great for us. All of our exceptions end up in trac where we can prioritize, assign and catalogue them. We even have a great way of knowing when exceptions that we think we've squashed re-appear because then the ticket just get re-opened.

Jeff Atwood wrote about using exception logs as a de-facto to do list a while ago and that's exactly what we have accomplished using this technique. We even did it long before he published his post but no one will ever believe that =)

What about ELMAH?

I have not used ELMAH personally but I've certainly heard many great things about it. From what I've read about ELMAH it seems that they have some great features for filtering unwanted exceptions but I haven't seen anything about grouping exceptions. The ideal thing would of course be if existing frameworks like ELMAH used this implementation to provide simple grouping.

Our end result

trac-tickets

image

This is an example of how it looks when we encounter an exception in our system. Note that although the exception was thrown twice there’s only one ticket.

Get the code and abuse it

This is the part which makes my stomach ache... The code, complete with a couple of unit tests and an example program plus more information is available at http://www.freakcode.com/projects/exception-signatures.

Licensing information

kick it on DotNetKicks.com

February 10, 2009

IIS7 Certificate binding issue

I recently tried installing a certificate into IIS7 on an new Windows 2008 server at work. The import whent well but when I tried to bind a site to use the new certificate I got a dialog box with the following message

---------------------------
Add Site Binding
---------------------------
There was an error while performing this operation.

Details: 

A specified logon session does not exist. It may already have been terminated. (Exception from HRESULT: 0x80070520)
---------------------------
OK   
---------------------------

(Note: The above text was produced by simply focusing the dialog window and pressing CTRL+C; really neat trick!)

After a lot of debugging/googling I managed to find a solution; you have to check the "Allow this certificate to be exported" box when you import the cert. Really strange and really not good practice but until I find another solution that's the way it has to be. If you've got another solution please comment!