Showing posts with label code. Show all posts
Showing posts with label code. Show all posts

December 6, 2010

Exception data evolved

As detailed in my earlier post about exception signatures I and my colleagues take exceptions from our production servers seriously. In addition to grouping them by signatures (which helps a lot and makes triage much more pleasant) and logging them in our internal bug tracking software we also try to add relevant debug information whenever we throw an exception.

The BCL team in their infinite wisdom added the Exception.Data property. This property is simply an IDictionary which allows storing key and value pairs of any type. By default this is an empty collection which means you don’t have to worry about it being null.

Typical usage

public void LogonUser(string username, string password, string domain)
{
    try
    {
        DataStore.LogOnUser(username, password, domain);
    }
    catch (Exception exc)
    {
        var ae = new ApplicationException("Underlying logon failed, see InnerException", exc);
                
        ae.Data["username"] = username;
        ae.Data["domain"] = domain;

        throw ae;
    }
}

When the exception reaches our internal trac site it might look something like the image below (in our case the data dictionary will actually be added as a comment but you get the point).

example-data

This has proven to be is incredibly useful for debugging purposes but it is a bit tedious to actually write the code. Storing the exception in a variable just to get access to the data property just didn’t feel right to me and writing the same boilerplate .Data[“xyz”] = xyz was just boring.

Step 1 – The AddData extension method.

I started off by creating an extension method for System.Exception called AddData. AddData looks a bit like this (very simplified, see the end of the post for the real deal)

public static Exception AddData(this Exception exception, string key, object value)
{
    exception.Data.Add(key, value);
// whohoo, chaining! return exception; }

This allowed me to save a few keystrokes over the first method and I would end up with something like this instead

catch(...) {
    throw new ApplicationException("Underlying logon failed, see InnerException", exc)
        .AddData("username", username)
        .AddData("domain", domain);
}

It might not look like much but it sure helps when you want to add debug data to an already existing exception, it saves you from having to store the reference and so on but it still doesn’t solve the problem with having to write the argument names twice. This isn’t only a nuisance when writing the code; it’s very easy for the key and value to get out of sync if you’re doing refactoring since refactoring tools will only change the variable name, not the key.

My ideal solution would be something like this

catch(...) {
    throw new ApplicationException("Underlying logon failed, see InnerException", exc)
        .AddData(username)
        .AddData(domain);
}

And have the AddData method automatically infer the proper key name but since that’s not possible I had to find another way.

Step 2 – Taking a cue from ASP.NET MVC

We continued using the AddData extension method above for a couple of weeks before it dawned on me that I could use anonymous types to skip the parameter name duplication. Anonymous types have the great ability of being able to “infer” property names when initialized.

var x = new { username = username, password = password }
// Is the same as
var x = new { username, password }

With this in mind I wrote an AddData extension method which accepts a single object and then uses reflection to iterate over all properties and adds their property names and values into the exception data dictionary.  This allowed me to rewrite my code yet again.

catch(...) {
    throw new ApplicationException("Underlying data store logon failed, see InnerException", exc)
        .AddData(new { username, password });
}

Neat, isn’t it? This is the exact same technique that ASP.NET MVC uses for route-declarations, attributes in html helpers and more.

Reflection? Isn’t that horribly slow

Not like in the olden days. It comes with a cost but you shouldn’t be too worried since you’re probably not throwing exception often enough for it to matter anyway (and if you are, then you have bigger problems).

Teh codez

using System;
using System.ComponentModel;
using System.Diagnostics;

namespace freakcode.Extensions
{
    /// 
    /// Extension methods related to instances of System.Exception and inherited objects.
    /// 
    public static class ExceptionExtensions
    {
        /// 
        /// Adds the supplied debug data to the exceptions data dictionary and returns
        /// the exception allowing chaining.
        /// 
        /// The exception type, you should not need to specify this explicitly
        /// The exception.
        /// The key of the debug value to be inserted into the exceptions data dictionary.
        /// The value to be inserted into the exceptions data dictionary.
        /// key is null
        /// An element with the same key already exists in the Data dictionary
        public static T AddData<T>(this T exception, string key, object value) where T : Exception
        {
            if (exception == null)
                throw new ArgumentNullException("exception");

            if (key == null)
                throw new ArgumentNullException("key");

            /* Key or value is not serializable (or key is null). The default internal structure which
             * implements the IDictionary is going to throw an exception in Add() so instead of 
             * throwing another exception while preparing to throw the first one we silently ignore the
             * error. Unless we're building in debug mode that is, then we'll fail. */
            if (value != null && !value.GetType().IsSerializable)
            {
                Debug.Fail("Attempt to add non-serializable value to exception data");
            }
            else
            {
                exception.Data.Add(key, value);
            }

            return exception;
        }

        /// 
        /// Adds the each property name and value from the supplied object to the exceptions data dictionary and returns
        /// the exception allowing chaining.
        /// 
        /// The exception type, you should not need to specify this explicitly
        /// The exception.
        /// An object from where properties will be read and added to the exception debug data collection.
        /// key is null
        /// An element with the same key already exists in the Data dictionary
        public static T AddData<T>(this T exception, object values) where T : Exception
        {
            if (values == null)
            {
                // Some really nasty things can happen if you start throwing exceptions in the middle
                // of throwing exceptions so unless we're in debug more we'll just silently ignore it.
                Debug.Fail("Argument 'values' was null!");
            }
            else
            {
                foreach (PropertyDescriptor descriptor in TypeDescriptor.GetProperties(values))
                    exception.AddData(descriptor.Name, descriptor.GetValue(values));
            }
            
            return exception;
        }
    }
}
kick it on DotNetKicks.com

Licensing information

January 8, 2010

Pretty automatically updating lighttpd server status or “mod_status on steroids”

At work we use the excellent lighttpd (or lighty) web server as a frontend reverse proxy that performs SSL offloading, and  some other neat tricks before our visitors reach our backend servers. It’s been working like a charm.

We’ve been using mod_status to keep track of how much traffic goes through our frontend server. mod_status is one of those little apps that gets the job done but not so much more.

server-status

It contains all the relevant information but I wanted to display this data on a separate monitor to as a way to keep track of how much traffic we where getting in real time.

Since it’s a static html page there wasn’t much interactivity there. So I whipped out my vim-editor and wrote a little helper page that dynamically polls data from the server-status page and displays it in a little more full-screen friendly way. You can see the result below (click on it to enlarge). It will automatically refresh the data once every 5 seconds.

server-status-pretty

Go ahead and download the source. It’s a single html file and all you have to do is to configure your lighty server so that the server status is enabled for a directory to which you’ve got write access to. Ours looks like this:

$SERVER["socket"] == "192.168.30.1:80" {
    status.status-url = "/server-status"
    server.document-root = "/var/www/status"
    mimetype.assign = ( ".html" => "text/html" )
    index-file.names = ( "pretty-status.html" )
}

Then you simply put the file in that page and it should just magically work. Note that the file loads the Google hosted jQuery library so if you don’t have access to the internet (on the machine your viewing from) you’ll have to download the jQuery script and place it along side the page.

Things on the todo list include pretty error messages when the data can’t be fetched and perhaps a nice little demo-mode.

[Download the source], [“Project page at freakcode.com”]

Licensing information

November 23, 2009

Google Maps version 3 intellisense

Intellisense is one of those killer-features of Visual Studio. I love that I can just drop in an assembly in VS and start poking around in an editor to get a feel of the API. If the API i properly documented and layed out in a .netty way I can immediately start working with it.

The flipside of intellisense is that once you get used to it you feel severely limited when you encounter a project without intellisense. That's what happened to me today.

I was integrating the excellent Google Maps library into one of our websites and decided that I should use the latest and greatest release - Google Maps v3. I fired up VS and started hacking away in a .js file until i got flashbacks from back in the days when I developed PHP. I was toggling between writing code in my IDE and reading the api documentation in my browser. It was horrifying!

-vsdoc to the rescue

At some point I realized that since I can get intellisense for jQuery I should be able to get it for Google Maps as well so I started googling and found this single commit project at codeplex that did offer an intellisense file for Google Maps v2 but not for v3.

That's when I started looking at writing my own intellisense file. I tried to validate the Google API reference in the W3 Validator only to find out that the king of all web doesn't always produce the best markup.

Not to fear; I've written a couple of screen scraping utilities before and I know that there's one project that makes dealing with badly formatted html a breeze: Html Agility Pack. The agility pack parses almost any amount of bad markup and gives you a nice little XPath interface for poking around inside the DOM.

The result

After some hackish (screen scrapers are always messy) code I was able to produce a basic intellisense file that covers most of the Google Maps API.

The files

The js intellisense file and usage instructions can be found over at http://www.freakcode.com/projects/google-maps-v3-vsdoc.

kick it on DotNetKicks.com

Licensing information

July 25, 2009

Introducing exception signatures

At work we develop, maintain and host a web solution which consists of a fair amount of assemblies and spans across several websites. Although we might like to think that we develop perfect code that certainly isn't always the case. And as with all non-djb applications ours sometimes generates a few exceptions.

Now please don't think of us as bad people, exceptions are an unfortunate part of nature and not all exceptions are the result of our bad programming, quite often we get exceptions from external libraries, from asp.net (viewstate exceptions, forbidden input exceptions and so on).

Keeping track of exceptions

Quite early on in our development process we decided that we needed a way of reacting to these exceptions as they happened. Our solution was to implement global exception handlers which sent an email to us whenever an unhandled exception occurred.

This worked great and in our early days it was more than enough. We could immediately react to an exception and we could often deploy a fix rapidly.

If you don't keep track of your exception at all you're probably in trouble. See this post by Jeff Atwood for a better explanation of why than I could ever come up with.

Keeping track of large amounts of exceptions

When our sites had been live for a while and started to get some serious use we noticed that it was getting really hard to keep track of all incoming exception mails. From looking at the stack trace we had no idea if an exception had been resolved, if someone was working on it, it's priority and so on. Troubled by the need to keep a bunch of stack traces in (my) memory I started thinking about ways of solving the problem.

My idea was to convert the exceptions into issue tickets. We use Trac at work so I started looking at simply redirecting all of our exception email there. Getting the exception mails converted to tickets wasn't really a problem but it didn't help at all with organization since every exception spawned a new ticket and we still had to manually merge the duplicates.

Enter Exception Signatures

In order to enable organization I started looking at generating a fingerprint for every exception. Some sort of string generated by looking at the stack traces of the exception, its inner exceptions and meta information of each exception. I wanted a solution which could be intelligent enough to differentiate between two exceptions thrown from the same place but for different reasons (i.e., the algorithm should look at the entire exception chain instead of only the base exception target site).

Handling dynamic data in exception messages

I also wanted it to be able to detect and disregard dynamic data in exception messages. Ever heard of Exception.Data? That's where we store extra information that we want to tack along for debugging purposes. Exception.Data is great but sometimes it's to much work and you just want to throw in some debugging data in the Exception.Message string like this:

public void RefundPurchase() {
    if(PurchaseState != "completed")
        throw new InvalidOperationException(
            "Invalid state for refund: " + PurchaseState
        );
}

In order to handle these cases I made the algorithm aware of three "pretty standard" ways of including dynamic data in exception messages.

  1. Text after a colon ':'. Invalid state for refund: inprogress
  2. Text within single and double quotes. Invalid state for refund "inprogress"
  3. Text within balanced parenthesis, brackets and braces. Invalid state for refund (state was inprogress)

Handling localized exceptions

I also wanted localized exceptions to produce the same signature when possible. We have a windows app used by some of our customers and we wanted exceptions from them to produce the same signature regardless of their OS language. To accommodate this I added some custom code for exceptions which contain an ErrorCode (SocketException and Win32Exception) and use that instead of the message.

Human semi-friendly signatures

My last requirement was that the signatures generated would be short so that we could display them to our users when they encounter one of our error-pages. This enables our customer service to provide status updates to calling customers by simply looking up the signature in trac and looking at its status.

As such the signatures are no-crazy-signs 8 character hex strings, like "617c1de5" and "1570d6d7"

Conclusion and possible solution

I think I've covered all my requirements and I must say that it has worked great for us. All of our exceptions end up in trac where we can prioritize, assign and catalogue them. We even have a great way of knowing when exceptions that we think we've squashed re-appear because then the ticket just get re-opened.

Jeff Atwood wrote about using exception logs as a de-facto to do list a while ago and that's exactly what we have accomplished using this technique. We even did it long before he published his post but no one will ever believe that =)

What about ELMAH?

I have not used ELMAH personally but I've certainly heard many great things about it. From what I've read about ELMAH it seems that they have some great features for filtering unwanted exceptions but I haven't seen anything about grouping exceptions. The ideal thing would of course be if existing frameworks like ELMAH used this implementation to provide simple grouping.

Our end result

trac-tickets

image

This is an example of how it looks when we encounter an exception in our system. Note that although the exception was thrown twice there’s only one ticket.

Get the code and abuse it

This is the part which makes my stomach ache... The code, complete with a couple of unit tests and an example program plus more information is available at http://www.freakcode.com/projects/exception-signatures.

Licensing information

kick it on DotNetKicks.com

September 12, 2008

ToTitleCase

Not to long ago I had to write a method to convert lowercase names to proper case for display on a website I was working on. I started rolling my own but then I thought that such a feature would be nice to have in the framework. So I hit google with a query and sure enough the framework authors had thought of it. The method was well hidden though ;)

CultureInfo.InvariantCulture.TextInfo.ToTitleCase("hELLo wORLd");
// Will return "Hello World"

This fits quite well in as an extension method

/// <summary>
/// Creates a new string with Title Case (ie "hEllO wORLd" becomes  "Hello World") using the Invariant Culture
/// </summary>
/// <param name="s">The string to convert</param>
/// <returns>The string in title case</returns>
public static string ToTitleCaseInvariant(this string s)
{
    return ToTitleCase(s, CultureInfo.InvariantCulture);
}

/// <summary>
/// Creates a new string with Title Case (ie "hEllO wORLd" becomes  "Hello World")
/// </summary>
/// <param name="s">The string to convert</param>
/// <returns>The string in title case</returns>
public static string ToTitleCase(this string s)
{
    return ToTitleCase(s, CultureInfo.CurrentCulture);
}

/// <summary>
/// Creates a new string with Title Case (ie "hEllO wORLd" becomes  "Hello World")
/// </summary>
/// <param name="s">The string to convert</param>
/// <param name="ci">The culture to use when creating title case</param>
/// <returns>The string in title case</returns>
public static string ToTitleCase(this string s, CultureInfo ci)
{
    if (s == null)
        throw new ArgumentNullException("s");

    return ci.TextInfo.ToTitleCase(s);
}

Now you can just call the ToTitleCase on your string objects like this:

var s = "george washington";
s.ToTitleCase();
kick it on DotNetKicks.com

Licensing information

September 4, 2008

High precision performance measurement

Sometimes you need to get a feel for how performant a specific method is. I've seen lots of developers use DateTime.Now for this. Like this:

DateTime start = DateTime.Now;
PerformWork();
TimeSpan ts = DateTime.Now - start
Console.WriteLine("Time taken: {0}ms", ts.TotalMilliseconds);

Now, there's a couple of problems with this. First off, DateTime.Now performs way worse than DateTime.UtcNow since it has to get the UTC time and the calculate timezone and DST offsets.

The second and more concerning problem is that DateTime has a resolution of about 15ms, it can't be more precise than that. Now, in development code you could of course get around this by running your code a couple of thousand time to compensate for the low precision but what if you're code can't easily be run more than one time (perhaps it performs expensive and hard-to-recover database work).

Stopwatch to the rescue

The System.Diagnostics.Stopwatch class provides high-precision measurement of elapsed time. Using hardware frequency counters it achieves nanosecond precision and it's really easy to use.

Stopwatch sw = Stopwatch.StartNew();
PerformWork();
sw.Stop();

Console.WriteLine("Time taken: {0}ms", sw.Elapsed.TotalMilliseconds);

The stopwatch falls back on DateTime.UtcNow if your hardware doesn't support a high frequency counter. You can check to see if Stopwatch utilizes hardware to achieve high precision by looking at the static field Stopwatch.IsHighResolution.

This post is essentially a ripoff of my answer to a question regarding time measurement on the kickass Q/A site stackoverflow.com (not open to public yet).

Licensing information

kick it on DotNetKicks.com

August 31, 2008

StartsWithAnyOf extension method

Here comes another great string extension for improved readability. You all know how to check if a string starts with another string (yep, it's the StartsWith method I'm talking about).

But what if you wan't to check if a string starts with any of a series of strings? Well, you'd have to do something like this.

string name = "Mr. Markus Olsson"
var l = new List<string> { "Dr", "Mr", "Ms" };
bool found;
foreach(string s in l) {
    if(name.StartsWith(l)) {
        found = true;
        break;
    }
}

Or, you could use lambdas for a much more elegant solution

string name = "Mr. Markus Olsson"
var l = new List<string> { "Dr", "Mr", "Ms" };
bool found = l.Exists(prefix => name.StartsWith(prefix));

That's pretty cool, right? The .Exists method on the list object takes a Predicate as a parameter and executes that predicate with each element as it's first argument until it finds a match. Coolio indeed but we can do better readability wise.

Enter StartsWithAnyOf extension methods

/// <summary>
/// Checks to see if the string starts with any of the supplied strings
/// </summary>
/// <param name="s">The string to check for a start value</param>
/// <param name="strings">One or more strings</param>
/// <returns>True the strings starts with any of the supplied strings, false otherwise</returns>
public static bool StartsWithAnyOf(this string s, params string[] strings)
{
    if (s == null)
        throw new ArgumentNullException("s");

    if (strings == null)
        throw new ArgumentNullException("strings");

    if (strings.Length == 0)
        throw new ArgumentOutOfRangeException("strings", "You must supply one or more strings");

    return Array.Exists(strings, (prefix => s.StartsWith(prefix)); 
}

/// <summary>
/// Checks to see if the string starts with any of the supplied strings
/// </summary>
/// <param name="s">The string to check for a start value</param>
/// <param name="strings">One or more strings</param>
/// <returns>True the strings starts with any of the supplied strings, false otherwise</returns>
public static bool StartsWithAnyOf(this string s, List<string> strings)
{
    if (s == null)
        throw new ArgumentNullException("s");

    if (strings == null)
        throw new ArgumentNullException("strings");

    if (strings.Count == 0)
        throw new ArgumentOutOfRangeException("strings", "You must supply one or more strings");

    return strings.Exists(x => s.StartsWith(x));
}

This allows us to rewrite our code to

string name = "Mr. Markus Olsson"
bool found = name.StartsWithAnyOf("Dr.", "Mr.", "Ms.");

Readability in a nutshell. Variations of these extension methods includes an override that takes a StringComparison in order to allow for case insensitive lookup. The EndsWithAnyOf method is of course also a must.

Update: as mattias pointed out there's a bit of code duplication here but that's intentional, read why
Update 2: I changed my mind again after discussing it with mattias and for the sake of readability and code-duplication I've decided to wrap the first method into a call of the second.
Update 3: James Curran pointed out that the Array class have an Exist method and that's of course what you want for the string array. Thanks James!

Licensing information

kick it on DotNetKicks.com

August 30, 2008

Collection.IsNullOrEmpty

Ah, string.IsNullOrEmpty. It was love at first sight when I found that simple yet extremely useful method. It's quite often you want check if a string is null or empty. Instead of checking if the string is null and then checking if it is of zero length (or god forbid checking if it equals string.Empty) you can call this this method and it does it for you. Not only does it save you from unnecessary keystrokes, it also increases readability.

string s = null;
if(!string.IsNullOrEmpty(s))
    PerformWork(s);
instead of
string s = null;
if(s != null && s.Length > 0)
    PerformWork(s);

Well, you probably get my point, it's awesome. But I found that I still write if-statements checking for nullity and length of lists, hashsets and other collections. I can't quite figure out why the BCL team didn't include a corresponding method for collections (or perhaps they did? Enlighten me plix!). Don't despair though, uncle Markus is here to help.

using System.Collections;

namespace freakcode.Utils
{
   public static class Collection
   {
       /// <summary>
       /// Checks if the supplied collection is null or empty
       /// </summary>
       /// <param name="il">The collection</param>
       /// <returns>True if the collection is null or empty, false otherwise</returns>
       public static bool IsNullOrEmpty(ICollection ic)
       {
           return (ic == null || ic.Count == 0);
       }

   }
}

It's so simple it's almost embarrassing to post but it sure increases readability and it's a really great method to integrate and enforce in you own repository of utility methods.

var employees = new List<string>() {"Bob", "Alice", "Tom" };
if(Collection.IsNullOrEmpty(employees))
    FireTheirAsses(employees);

You could rename/copy this to List.IsNullOrEmpty if you think that produces cleaner and more readable code.

Licensing information

kick it on DotNetKicks.com