Generation 5 » Dot Net

First-Class Functions and Logical Negation in C#

Paul Houle — Mon, 09 Mar 2009 13:50:39 +0000

Introduction

Languages such as LISP, ML, oCaml F# and Scala have supported first-class functions for a long time. Functional programming features are gradually diffusing into mainstream languages such as C#, Javascript and PHP. In particular, Lambda expressions, implicit typing, and delegate autoboxing have made C# 3.0 an much more expressive language than it’s predecssors.

In this article, I develop a simple function that acts on functions: given a boolean function f, F.Not(f) returns a new boolean function which is the logical negation of f. (That is, F.Not(f(x)) == !f(x)). Although the end function is simple to use, I had to learn a bit about the details of C# to ge tthe behavior I wanted — this article records that experience.

Why?

I was debugging an application that uses Linq-to-Objects the other day, and ran into a bit of a pickle. I had written an expression that would return true if all of the values in a List were valid phone numbers:

[01] dataColumn.All(Validator.IsValidPhoneNumber);

I was expecting the list to validate successfully, but the function was returning false. Obviously at least one of the values in the function was not validating — but which one? I tried using a lambda expression in the immediate window (which lets you type in expression while debugging) to reverse the order of matching:

[02] dataColumn.Where(s => !ValidatorIsValidPhoneNumber(x));

but that gave me the following error message:

[03] Expression cannot contain lambda expressions

(The expression compiler used in the immediate window doesn’t support all of the language features supported by the full C# compiler.) I was able to answer find the non matching elements using the set difference operator

[04] dataColumn.Except(dataColumn.Where(Validator.IsValidPhoneNumber).ToList()

but I wanted an easier and more efficient way — with a logical negation function, I could just write

[05] dataColumn.Except(F.Not(Validator.IsValidPhoneNumber).ToList();

Try 1: Extension Method

I wanted to make the syntax for negation as simple as possible. I didn’t want to even have to name a class eliminate the need to name a class, so I tried the following extension method:

[06] static class FuncExtensions {
[07]    public static Func Not(this Func innerFunction) {
[08]       return ()  => innerFunction();
[09]    }
[10]    ...
[11]   }

I was hoping that, given a function like

[12] public static boolean AlwaysTrue() { return true };

that I could negate the function by writing

[13] AlwaysTrue.Not()

Unfortunately, it doesn’t work that way: the extension method can only be called on a delegate of type Func. Although the compiler will “autobox” function references to delegates in many situations, it doesn’t do it when you reference an extension method. I could write

[14] (Func AlwaysTrue).Not()

but that’s not a pretty syntax. At that point, I tried another tack.

Try 2: Static Method

Next, I defined a set of negation functions as static methods on a static class:

[15] public static class F {
[16]    public static Func Not(Func innerFunction) {
[17]       return () => !innerFunction();
[18]    }
[19]
[20]    public static Func Not(
[21]       Func innerFunction) {
[22]          return x =>!innerFunction(x);
[23]    }
[24]
[25]    public static Func Not(
[26]       Func innerFunction) {
[27]          return (x,y) => !innerFunction(x,y);
[28]    }
[29]
[30]    public static Func Not(
[31]       Func innerFunction) {
[32]           return (x, y, z) => !innerFunction(x, y, z);
[33]    }
[34]
[35]    public static Func Not(
[36]       Func innerFunction) {
[37]          return (x, y, z, a) => !innerFunction(x, y, z, a);
[38]    }
[39] }

Now I can write

[40] F.Not(AlwaysTrue)() // always == false

[41] Func testFunction = (a,b,c,d) => (a+b)>(c+d)
[42] F.Not(testFunction)(1,2,3,4)

which is sweet — the C# compiler now automatically autoboxes the argument to F.Not on line [40]. Note two details of how type inference works here:

The compiler automatically infers the type parameters of F.Not() by looking at the arguments of the innerFunction. If it didn’t do that, you’d need to write
```
F.Not(testFunction)(1,2,3,4)
```
which would be a lot less fun.
On line [40], note that the compiler derives the types of the parameters a,b,c, and d using the type declaration on the right hand side (RHS)
```
Func
```
you can’t write var on the RHS in this situation because that doesn’t give the compiler information about the parameters and return values of the lambda.

Although good things are happening behind the scenes, there are also two bits of ugliness:

I need to implement F.Not() 5 times to support functions with 0 to 4 parameters: once I’ve done that, however, the compiler automatically resolves the overloading and picks the right version of the function.
The generic Func<> and Action<> delegates support at most 4 parameters. Although it’s certainly true that functions with a large number of parameters can be difficult to use and maintain (and should be discouraged), this is a real limitation.

Extending IEnumerable

One of the nice things about Linq is that you can extend it by adding new extension methods to IEnumerable. Not everbody agrees, but I’ve always liked the unless() satement in Perl, which is equivalent to if(!test):

[42] unless(everything_is_ok()) {
[43]   abort_operation;
[44] }

A replacement for the Where(predicate) method that negates the predicate function would be convenient my debugging problem:

I built an Unless() extension method that combines Where() with F.Not():

[45] public static IEnumerable Unless(
[46]    this IEnumerable input, Func fn) {
[47]       return input.Where(F.Not(fn));
[48]     }

Now I can write

[49] var list = new List() { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };
[50] var filtered = list.Unless(i => i > 3).ToList();

and get the result

[51] filtered = { 1,2,3 }

Conclusion

New features in C# 3.0 make it an expressive language for functional programming. Although the syntax of C# isn’t quite as sweet as F# or Scala, a programmer who works with the implicit typing and autoboxing rules of the compiler can create functions that act on functions that are easy to use — in this article we develop a set of functions that negate boolean functions and apply this to add a new restriction method to IEnumerable.

Using Linq To Tell if the Elements of an IEnumerable Are Distinct

Paul Houle — Fri, 13 Feb 2009 21:15:38 +0000

The Problem

I’ve got an IEnumerable that contains a list of values: I want to know if all of the values in that field are distinct. The function should be easy to use a LINQ extension method and, for bonus points, simply expressed in LINQ itself

One Solution

First, define an extension method

01   public static class IEnumerableExtensions {
02        public static bool AllDistinct(this IEnumerable input) {
03            var count = input.Count();
04            return count == input.Distinct().Count();
05        }
06    }

When you want to test an IEnumerable, just write

07 var isAPotentialPrimaryKey=CandidateColumn.AllDistinct();

Analysis

This solution is simple and probably scales as well as any solution in the worst case. However, it enumerates the IEnumerable twice and does a full scan of the IEnumerable even if non-distinct elements are discovered early in the enumeration. I could certainly make an implementation that aborts early using a Dictionary to store elements we’ve seen and a foreach loop, but I wonder if anyone out there can think of a better pure-Linq solution.

Subverting XAML: How To Inherit From Silverlight User Controls

Paul Houle — Tue, 10 Feb 2009 20:31:21 +0000

The Problem

Many Silverlighters use XAML to design the visual appearance of their applications. A UserControl defined with XAML is a DependencyObject that has a complex lifecycle: there’s typically a .xaml file, a .xaml.cs file, and a .xaml.g.cs file that is generated by visual studio. The .xaml.g.cs file is generated by Visual Studio, and ensures that objects defined in the XAML file correspond to fields in the object (so they are seen in intellisense and available to your c# code.) The XAML file is re-read at runtime, and drives a process that instantiates the actual objects defined in the XAML file — a program can compile just fine, but fail during initialization if the XAML file is invalid or if you break any of the assumptions of the system.

XAML is a pretty neat system because it’s not tied to WPF or WPF/E. It can be used to initialize any kind of object: for instance, it can be used to design workflows in asynchronous server applications based on Windows Workflow Foundation.

One problem with XAML, however, is that you cannot write controls that inherit from a UserControl that defined in XAML. Visual Studio might compile the classes for you, but they will fail to initialize at at runtime. This is serious because it makes it impossible to create subclasses that let you make small changes to the appearance or behavior of a control.

Should We Just Give Up?

One approach is to give up on XAML. Although Visual Studio encourages you to create UserControls with XAML, there’s nothing to stop you from creating a new class file and writing something like

class MyUserControl:UserControl {
      public MyUserControl {
      var innerPanel=new StackPanel();
      Content=innerPanel;
      innerPanel.Children.Add(new MyFirstVisualElement());
      innerPanel.Children.Add(new MySecondVisualElement());
      ...
}

UserControls defined like this have no (fundamental) problems with inheritance, since you’ve got complete control of the initialization process. If it were up to me, I’d write a lot of controls like this, but I work on a team with people who design controls in XAML, so I needed a better solution.

Or Should We Just Cheat?

If we still want to use XAML to define the appearance of a control, we’ve got another option. We can move the XAML into a control that is outside the inheritance hierarchy: the XAML control is then contained inside another control which doesn’t have restrictions as to how it is used:

The XamlPanelInnards.xaml.cs code-behind is almost completely empty: it contains only the constructor created by Visual Studio’s XAML designer. XamlPanel.cs contains a protected member called InnerControl which is of type XamlPanelInnards. InnerControl is initialized in the constructor of XamlPanel, and is also assigned to the Content property of the XamlPanel, to make it visible. Everything that you’d ordinarily put in the code-behind XamlPanelInnards.xaml.cs goes into XamlPanel.cs, which uses the InnerControl member to get access to the internal members defined by the XAML designer.

Step-by-Step Refactoring

Let’s imagine that we’ve got an existing UserControl implemented in XAML called the XamlPanel. We’d like to subclass the XamlPanel so we can use it for multiple purposes. We can do this by:

Renaming XamlPanel to OldXamlPanel; this renames both the *.xaml and *.xaml.cs files so we can have them to look at
Use Visual Studio to create a new “Silverlight User Control” called XamlPanelInnards. This will have both a *.xaml and *.xaml.xs file
Copy the contents of OldXamlPanel.xaml to XamlPanelInnards.cs. Edit the x:Class attribute of the element to reflect the new class name, “XamlPanelInnards”
Use Visual Studio to create a new class, called XamlPanel.cs. Do not create a XamlPanel.xaml.cs file!
Copy the constructor, methods, fields and properties from the OldXamlPanel.xaml.cs file to the XamlPanel.cs file.
Create a new private field in XamlPanel.cs like
```
private XamlPanelInnards InnerControl;
```

Now we modify the constructor, so that it does something like this:

public XamlPanel() {
   InnerControl = new XamlPanelInnards();
   Content = InnerControl;
   ... remainder of the constructor ...
}

Most likely you’ll have compilation errors in the XamlPanel.cs file because there are a lot of references to public fields that are now in the InnerControl. You need to track these down, and replace code that looks like
```
TitleTextBlock.Text="Some Title";
```
with
```
InnerControl.TitleTextBlock.Text="SomeTitle";
```
Any event handler attachments done from the XAML file will fail to work (they’ll trigger an error when you load the application.) You’ll need to convert
in the XAML file to
```
InnerControl.PressMe.Click += PressMe_OnClick
```
in the constructor of XamlPanel.
UI Elements that are defined in XAML are public, so there’s a good chance that other classes might expect UI Elements inside the XamlPanel to be accessable. You’ve got some choices: (a) make the InnerControl public and point those references to the InnerControl (yuck!), (b) selectively add properties to XamlPanel to let outside classes access the elements that they need to access, or (c) rethink the encapsulation so that other class don’t need direct access to the members of XamlPanelInnards.
There are a few special methods that are defined in DependencyObject and FrameworkElement that you’ll need to pay attention to. For instance, if your class uses FindName to look up elements dynamically, you need to replace
```
FindName("Control"+controlNumber);
```
with
```
InnerControl.FindName("Control"+controlNumber);
```

So far this is a refactoring operation: we’re left with a program that does what it already did, but is organized differently. Where can we go from here?

Extending The XamlPanel

At this point, the XamlPanel is an ordinary UserControl class. It’s initialization logic is self-sufficient, so it can inherit (it doesn’t necessarily have to derive from UserControl) and be inherited from quite freely.

If we want to change the behavior of the XamlPanel, for instance, we could declare it abstract and leave certain methods (such as event handlers) abstract. Alternately, methods could be declared as virtual.

A number of methods exist to customize the appearance of XamlPanel: since XamlPanel can see the objects inside XamlPanelInnards, it can change colors, image sources and text contents. If you’re interested in adding additional graphical element to the Innards, the innards can contain an empty StackPanel — children of XamlPanel can Add() something to the StackPanel in their constructors.

Note that you can still include a child XamlPanel inside a control defined in XAML by writing something like

you’re free to make XamlPanel and it’s children configurable via the DependencyObject mechanisms. The one thing that you lose is public access to the graphical element inside the StackPanelInnards: many developers would think that this increase in encapsulation is a good thing, but it may involve a change in the way you do things.

The community at silverlight.net has pointed me to a few other articles about XAML, inheritance and configuring Custom XAML controls.

In a lavishly illustrated blog entry, Amyo Kabir explains how to make a XAML-defined control inherit from a user-defined base class. It’s the converse of what I’m doing in this article, but it’s also a powerful technique: I’m using it right now to implement a number of Steps in a Wizard.

Note that Silverlight has mechanisms for creating Custom Controls: these are controls that use the DependencyObject mechanism to be configurable via XAML files that include them. If you’re interested in customizing individual controls rather than customizing subclasses, this is an option worth exploring.

Conclusion

XAML is a great system for configuring complex objects, but you can’t inherit from a Silverlight class defined in XAML. By using a containment relation instead of an inheritance relation, we can push the XAML-configured class outside of our inheritance hierarchy, allowing the container class to participate as we desire. This way we can have both visual UI generation and the software engineering benefits of inheritance.

What do you do when you’ve caught an exception?

Paul Houle — Wed, 27 Aug 2008 15:19:03 +0000

Abort, Retry, Ignore

This article is a follow up to “Don’t Catch Exceptions“, which advocates that exceptions should (in general) be passed up to a “unit of work”, that is, a fairly coarse-grained activity which can reasonably be failed, retried or ignored. A unit of work could be:

an entire program, for a command-line script,
a single web request in a web application,
the delivery of an e-mail message
the handling of a single input record in a batch loading application,
rendering a single frame in a media player or a video game, or
an event handler in a GUI program

The code around the unit of work may look something like

[01] try {
[02]   DoUnitOfWork()
[03] } catch(Exception e) {
[04]    ... examine exception and decide what to do ...
[05] }

For the most part, the code inside DoUnitOfWork() and the functions it calls tries to throw exceptions upward rather than catch them.

To handle errors correctly, you need to answer a few questions, such as

Was this error caused by a corrupted application state?
Did this error cause the application state to be corrupted?
Was this error caused by invalid input?
What do we tell the user, the developers and the system administrator?
Could this operation succeed if it was retried?
Is there something else we could do?

Although it’s good to depend on existing exception hierarchies (at least you won’t introduce new problems), the way that exceptions are defined and thrown inside the work unit should help the code on line [04] make a decision about what to do — such practices are the subject of a future article, which subscribers to our RSS feed will be the first to read.

The cause and effect of errors

There are a certain range of error conditions that are predictable, where it’s possible to detect the error and implement the correct response. As an application becomes more complex, the number of possible errors explodes, and it becomes impossible or unacceptably expensive to implement explicit handling of every condition.

What do do about unanticipated errors is a controversial topic. Two extreme positions are: (i) an unexpected error could be a sign that the application is corrupted, so that the application should be shut down, and (ii) systems should bend but not break: we should be optimistic and hope for the best. Ultimately, there’s a contradiction between integrity and availability, and different systems make different choices. The ecosystem around Microsoft Windows, where people predominantly develop desktop applications, is inclined to give up the ghost when things go wrong — better to show a “blue screen of death” than to let the unpredictable happen. In the Unix ecosystem, more centered around server applications and custom scripts, the tendency is to soldier on in the face of adversity.

What’s at stake?

Desktop applications tend to fail when unexpected errors happen: users learn to save frequently. Some of the best applications, such as GNU emacs and Microsoft Word, keep a running log of changes to minimize work lost to application and system crashes. Users accept the situation.

On the other hand, it’s unreasonable for a server application that serves hundreds or millions of users to shut down on account of a cosmic ray. Embedded systems, in particular, function in a world where failure is frequent and the effects must be minimized. As we’ll see later, it would be a real bummer if the Engine Control Unit in your car left you stranded home because your oxygen sensor quit working.

The following diagram illustrates the environment of a work unit in a typical application: (although this application accesses network resources, we’re not thinking of it as a distributed application. We’re responsible for the correct behavior of the application running in a single address space, not about the correct behavior of a process swarm.)

The Input to the work unit is a potential source of trouble. The input could be invalid, or it could trigger a bug in the work unit or elsewhere in the system (the “system” encompasses everything in the diagram) Even if the input is valid, it could contain a reference to a corrupted resource, elsewhere in the system. A corrupted resource could be a damaged data structure (such as a colored box in a database), or an otherwise malfunctioning part of the system (a crashed server or router on the network.)

Data structures in the work unit itself are the least problematic, for purposes of error handling, because they don’t outlive the work unit and don’t have any impact on future work units.

Static application data, on the other hand, persists after the work unit ends, and this has two possible consequences:

The current work unit can fail because a previous work unit caused a resource to be corrupted, and
The current work unit can corrupt a resource, causing a future work unit to fail

Osterman’s argument that applications should crash on errors is based on this reality: an unanticipated failure is a sign that the application is in an unknown (and possibly bad) state, and can’t be trusted to be reliable in the future. Stopping the application and restarting it clears out the static state, eliminating resource corruption.

Rebooting the application, however, might not free up corrupted resources inside the operating system. Both desktop and server applications suffer from operating system errors from time to time, and often can get immediate relief by rebooting the whole computer.

The “reboot” strategy runs out of steam when we cross the line from in-RAM state to persistent state, state that’s stored on disks, or stored elsewhere on the network. Once resources in the persistent world are corrupted, they need to be (i) lived with, or repaired by (ii) manual or (iii) automatic action.

In either world, a corrupted resource can have either a narrow (blue) or wide (orange) effect on the application. For instance, the user account record of an individual user could be damaged, which prevents that user from logging in. That’s bad, but it would hardly be catastrophic for a system that has 100,000 users. It’s best to ‘ignore’ this error, because a system-wide ‘abort’ would deny service to 99,999 other users; the problem can be corrected when the user complains, or when the problem is otherwise detected by the system administrator.

If, on the other hand, the cryptographic signing key that controls the authentication process were lost, nobody would be able to log in: that’s quite a problem. It’s kind of the problem that will be noticed, however, so aborting at the work unit level (authenticated request) is enough to protect the integrity of the system while the administrators repair the problem.

Problems can happen at an intermediate scope as well. For instance, if the system has damage to a message file for Italian users, people who use the system in the Italian language could be locked out. If Italian speakers are 10% of the users, it’s best to keep the system running for others while you correct the problem.

Repair

There are several tools for dealing with corruption in persistent data stores. In a one-of-a-kind business system, a DBA may need to intervene occasionally to repair corruption. More common events can be handled by running scripts which detect and repair corruption, much like the fsck command in Unix or the chkdsk command in Windows. Corruption in the metadata of a filesystem can, potentially, cause a sequence of events which leads to massive data loss, so UNIX systems have historically run the fsck command on filesystems whenever the filesystem is in a questionable state (such as after a system crash or power failure.) The time do do an fsck has become an increasing burden as disks have gotten larger, so modern UNIX systems use journaling filesystems that protect filesystem metadata with transactional semantics.

Release and Rollback

One role of an exception handler for a unit of work is to take steps to prevent corruption. This involves the release of resources, putting data in a safe state, and, when possible, the rollback of transactions.

Although many kinds of persistent store support transactions, and many in-memory data structures can support transactions, the most common transactional store that people use is the relational database. Although transactions don’t protect the database from all programming errors, they can ensure that neither expected or unexpected exceptions will cause partially-completed work to remain in the database.

A classic example in pseudo code is the following:

[06] function TransferMoney(fromAccount,toAccount,amount) {
[07]   try {
[08]      BeginTransaction();
[09]      ChangeBalance(toAccount,amount);
[10]      ... something throws exception here ...
[11]      ChangeBalance(fromAccount,-amount);
[12]      CommitTransaction();
[13]   } catch(Exception e) {
[14]      RollbackTransaction();
[15]   }
[16] }

In this (simplified) example, we’re transferring money from one bank account to another. Potentially an exception thrown at line [05] could be serious, since it would cause money to appear in toAccount without it being removed from fromAccount. It’s bad enough if this happens by accident, but a clever cracker who finds a way to cause an exception at line [05] has discovered a way to steal money from the bank.

Fortunately we’re doing this financial transaction inside a database transaction. Everything done after BeginTransaction() is provisional: it doesn’t actually appear in the database until CommitTransaction() is called. When an exception happens, we call RollbackTransaction(), which makes it as if the first ChangeBalance() had never been called.

As mentioned in the “Don’t Catch Exceptions” article, it often makes sense to do release, rollback and repairing operations in a finally clause rather than the unit-of-work catch clause because it lets an individual subsystem take care of itself — this promotes encapsulation. However, in applications that use databases transactionally, it often makes sense to push transaction management out the the work unit.

Why? Complex database operations are often composed out of simpler database operations that, themselves, should be done transactionally. To take an example, imagine that somebody is opening a new account and funding it from an existing account:

[17] function OpenAndFundNewAccount(accountInformation,oldAccount,amount) {
[18]    if (amount
It’s important that the TransferMoney operation be done transactionally,  but it’s also important that the whole OpenAndFundNewAccount operation be done transactionally too,  because we don’t want an account in the system to start with a zero balance.
A straightforward answer to this problem is to always do banking operations inside a unit of work, and to begin, commit and roll back transactions at the work unit level:
[26] AtmOutput ProcessAtmRequest(AtmInput in) {
[27]    try {
[28]       BeginTransaction();
[29]       BankingOperation op=AtmInput.ParseOperation();
[30]       var out=op.Execute();
[31]       var atmOut=AtmOutput.Encode(out);
[32]       CommitTransaction();
[33]       return atmOut;
[34]    }
[35]    catch(Exception e) {
[36]       RollbackTransaction();
[37]       ... Complete Error Handling ...
[38]    }
In this case, there might be a large number of functions that are used to manipulate the database internally, but these are only accessable to customers and bank tellers through a limited set of BankingOperations that are always executed in a transaction.
Notification
There are several parties that could be notified when something goes wrong with an application,  most commonly:

the end user,
the system administrator,  and
the developers.

Sometimes, as in the case of a public-facing web application, #2 and #3 may overlap. In desktop applications, #2 might not exist.
Let’s consider the end user first. The end user really needs to know (i) that something went wrong, and (ii) what they can do about it. Often errors are caused by user input: hopefully these errors are expected, so the system can tell the user specifically what went wrong: for instance,
[39] try {
[40]   ... process form information ...
[41]
[42]    if (!IsWellFormedSSN(ssn))
[43]       throw new InvalidInputException("You must supply a valid social security number");
[44]
[45]    ... process form some more ...
[46] } catch(InvalidInputException e) {
[47]    DisplayError(e.Message);
[48] }
other times, errors happen that are unexpected. Consider a common (and bad) practice that we see in database applications: programs that write queries without correctly escaping strings:
[49] dbConn.Execute("
[50]   INSERT INTO people (first_name,last_name)
[51]      VALUES ('"+firstName+"','+lastName+"');
[52] ");
this code is straightforward, but dangerous,  because a single quote in the firstName or lastName variable ends the string literal in the VALUES clause,  and enables an SQL injection attack.  (I’d hope that you know better than than to do this, but large projects worked on by large teams inevitably have problems of this order.) This code might even hold up well in testing, failing only in production when a person registers with
[53] lastName="O'Reilly";
Now,  the dbConn is going to throw something like a SqlException with the following message:
[54] SqlException.Message="Invalid SQL Statement:
[55]   INSERT INTO people (first_name,last_name)
[56]      VALUES ('Baba','O'Reilly');"
we could show that message to the end user, but that message is worthless to most people. Worse than that, it’s harmful if the end user is a cracker who could take advantage of the error — it tells them the name of the affected table, the names of the columns, and the exact SQL code that they can inject something into. You might be better off showing users something like:

and telling them that they’ve experienced an “Internal Server Error.”  Even so,  the discovery that a single quote can cause an “Internal Server Error” can be enough  for a good cracker to sniff out the fault and develop an attack in the blind.. What can we do? Warn the system administrators. The error handling system for a server application should log exceptions, stack trace and all. It doesn’t matter if you use the UNIX syslog mechanism,  the logging service in Windows NT,   or something that’s built into your server,  like Apache’s error_log.  Although logging systems are built into both Java and .Net,  many developers find that Log4J and Log4N are especially effective.
There really are two ways to use logs:

Detailed logging information is useful for debugging problems after the fact. For instance, if a user reports a problem, you can look in the logs to understand the origin of the problem, making it easy to debug problems that occur rarely: this can save hours of time trying to understand the exact problem a user is experiencing.
A second approach to logs is proactive: to regularly look a logs to detect problems before they get reported. In the example above, the SqlException would probably first be thrown by an innocent person who has an apostrophe in his or her name — if the error was detected that day and quickly fixed, a potential security hole could be fixed long before it would be exploited.  Organizaitons that investigate all exceptions thrown by production web applications run the most secure and reliable applications.

In the last decade it’s become quite common for desktop applications to send stack traces back to the developers after a crash: usually they pop up a dialog box that asks for permission first. Although developers of desktop applications can’t be as proactive as maintainers of server applications, this is a useful tool for discovering errors that escape testing, and to discover how commonly they occur in the field.
Retry I: Do it again!
Some errors are transient: that is, if you try to do the same operation later, the operation may succeed. Here are a few common cases:

An attempt to write to a DVD-R could fail because the disk is missing from the drive
A database transaction could fail when you commit it because of a conflict with another transaction: an attempt to do the transaction again could succeed
An attempt to deliver a mail message could fail because of problems with the network or destination mail server
A web crawler that crawls thousands (or millions) of sites will find that many of them are down at any given time: it needs to deal with this reasonably, rather than drop your site from it’s index because it happened to be down for a few hours

Transient errors are commonly associated with the internet and with remote servers; errors are frequent because of the complexity of the internet, but they’re transitory because problems are repaired by both automatic and human intervention. For instance, if a hardware failure causes a remote web or email server to go down, it’s likely that somebody is going to notice the problem and fix it in a few hours or days.
One strategy for dealing with transient errors is to punt it back to the user: in a case like this, we display an error message that tells the user that the problem might clear up if they retry the operation. This is implicit in how web browsers work: sometimes you try to visit a web page, you get an error message, then you hit reload and it’s all OK. This strategy is particularly effective when the user could be aware that there’s a problem with their internet connection and could do something about it: for instance, they might discover that they’ve moved their laptop out of Wi-Fi range, or that the DSL connection at their house has gone down for the weekend.
SMTP, the internet protocol for email, is one of the best examples of automated retry. Compliant e-mail servers store outgoing mail in a queue: if an attempt to send mail to a destination server fails, mail will stay in the queue for several days before reporting failure to the user. Section 4.5.4 of RFC 2821 states:
   The sender MUST delay retrying a particular destination after one
   attempt has failed.  In general, the retry interval SHOULD be at
   least 30 minutes; however, more sophisticated and variable strategies
   will be beneficial when the SMTP client can determine the reason for
   non-delivery.

   Retries continue until the message is transmitted or the sender gives
   up; the give-up time generally needs to be at least 4-5 days.  The
   parameters to the retry algorithm MUST be configurable.

   A client SHOULD keep a list of hosts it cannot reach and
   corresponding connection timeouts, rather than just retrying queued
   mail items.

   Experience suggests that failures are typically transient (the target
   system or its connection has crashed), favoring a policy of two
   connection attempts in the first hour the message is in the queue,
   and then backing off to one every two or three hours.
Practical mail servers use fsync() and other mechanisms to implement transactional semantics on the queue: the needs of reliability make it expensive to run an SMTP-compliant server, so e-mail spammers often use non-compliant servers that don’t correctly retry (if they’re going to send you 20 copies of the message anyway, who cares if only 15 get through?) Greylisting is a highly effective filtering strategy that tests the compliance of SMTP senders by forcing a retry.
Retry II: If first you don’t succeed…
An alternate form of retry is to try something different. For instance, many programs in the UNIX environment will look in many different places for a configuration file: if the file isn’t in the first place tried, it will try the second place and so forth.
The online e-print server at arXiv.org has a system called AutoTex which automatically converts documents written in several dialects of TeX and LaTeX into Postscript and PDF files.  AutoTex unpacks the files in a submission into a directory and uses chroot to run the document processing tools in a protected sandbox. It tries about of ten different configurations until it finds one that successfully compiles the document.
In embedded applications,  where availability is important,  it’s common to fall back to a “safe mode” when normal operation is impossible.  The Engine Control Unit in a modern car is a good example:

Since the 1970′s,   regulations in the United States have reduced emissions of hydrocarbons and nitrogen oxides from passenger automobiles by more than a hundred fold.  The technology has many aspects,  but the core of the system in an Engine Control Unit that uses a collection of sensors to monitor the state of the engine and uses this information to adjust engine parameters (such as the quantity of fuel injected) to balance performance and fuel economy with environmental compliance.
As the condition of the engine,  driving conditions and composition of fuel change over the time,  the ECU normally operates in a “closed-loop” mode that continually optimizes performance.   When part of the system fails (for instance,  the oxygen sensor) the ECU switches to an “open-loop” mode.  Rather than leaving you stranded,  it lights the “check engine” indicator and operates the engine with conservative assumptions that will get you home and to a repair shop.
Ignore?
One strength of exceptions,  compared to the older return-value method of error handling is that the default behavior of an exception is to abort,  not to ignore.  In general,  that’s good,  but there are a few cases where “ignore” is the best option.  Ignoring an error makes sense when:

Security is not at stake,  and
there’s no alternative action available,  and
the consequences of an abort are worse than the consequences of avoiding an error

The first rule is important,  because crackers will take advantage of system faults to attack a system.  Imagine,  for instance,  a “smart card” chip embedded in a payment card.  People have successfully extracted information from smart cards by fault injection:  this could be anything from a power dropout to a bright flash of light on an exposed silicon surface.  If you’re concerned that a system will be abused,  it’s probably best to shut down when abnormal conditions are detected.
On the other hand,  some operations are vestigial to an application.  Imagine,  for instance,  a dialog box that pops when an application crashes that offers the user the choice of sending a stack trace to the vendor.  If the attempt to send the stack trace fails,  it’s best to ignore the failure — there’s no point in subjecting the user to an endless series of dialog boxes.
“Ignoring” often makes sense in the applications that matter the most and those that matter the least.
For instance,  media players and video games operate in a hostile environment where disks,  the network, sound and controller hardware are uncooperative.  The “unit of work” could be the rendering of an individual frame:  it’s appropriate for entertainment devices to soldier on despite hardware defects,  unplugged game controllers,  network dropouts and corrupted inputs,  since the consequences of failure are no worse than shutting the system down.
In the opposite case,  high-value systems and high-risk should continue functioning no matter what happen.  The software for a space probe,  for instance,  should never give up.  Much like an automotive ECU,  space probes default to a “safe mode” when contact with the earth is lost:  frequently this strategy involves one or more reboots,  but the goal is to always regain contact with controllers so that the mission has a chance at success.
Conclusion
It’s most practical to catch exceptions at the boundaries of relatively coarse “units of work.” Although the handling of errors usually involves some amount of rollback (restoring system state) and notification of affected people, the ultimate choices are still what they were in the days of DOS: abort, retry, or ignore.
Correct handling of an error requires some thought about the cause of an error: was it caused by bad input, corrupted application state, or a transient network failure? It’s also important to understand the impact the error has on the application state and to try to reduce it using mechanisms such as database transactions.
“Abort” is a logical choice when an error is likely to have caused corruption of the application state, or if an error was probably caused by a corrupted state. Applications that depend on network communications sometimes must “Retry” operations when they are interrupted by network failures. Another form of “Retry” is to try a different approach to an operation when the first approach fails. Finally, “Ignore” is appropriate when “Retry” isn’t available and the cost of “Abort” is worse than soldiering on.
This article is one of a series on error handling.  The next article in this series will describe practices for defining and throwing exceptions that gives exception handlers good information for making decisions.  Subscribers to our RSS Feed will be the first to read it.

Stop Catching Exceptions!

Paul Houle — Fri, 01 Aug 2008 01:45:24 +0000

Motivation

It’s clear that a lot of programmers are uncomfortable with exceptions [1] [2]; in the feedback of an article I wrote about casting, it seemed that many programmers saw the throwing of a NullReferenceException at a cast to be an incredible catastrophe.

In this article, I’ll share a philosophy that I hope will help programmers overcome the widespread fear of exceptions. It’s motivated by five goals:

Do no harm
To write as little error handling code as possible,
To think about error handling as little as possible
To handle errors correctly when possible,
Otherwise errors should be handled sanely

To do that, I

Use finally to stabilize program state when exceptions are thrown
Catch and handle exceptions locally when the effects of the error are local and completely understood
Wrap independent units of work in try-catch blocks to handle errors that have global impact

This isn’t the last word on error handling, but it avoids many of the pitfalls that people fall into with exceptions. By building upon this strategy, I believe it’s possible to develop an effective error handling strategy for most applications: future articles will build on this topic, so keep posted by subscribing to the Generation 5 RSS Feed.

The Tragedy of Checked Exceptions

Java’s done a lot of good, but checked exceptions are probably the worst legacy that Java has left us. Java has influenced Python, PHP, Javascript, C# and many of the popular languages that we use today. Unfortunately, checked exceptions taught Java programmers to catch exceptions prematurely, a habit that Java programmers carried into other languages, and has result in a code base that sets bad examples.

Most exceptions in Java are checked, which means that the compiler will give you an error if you write

[01] public void myMethod() {
[02]    throw new ItDidntWorkException()
[03] };

unless you either catch the exception inside myMethod or you replace line [01] with

[04] public void myMethod() throws ItDidntWorkException {

The compiler is also aware of any checked exceptions that are thrown by methods underneath myMethod, and forces you to either catch them inside myMethod or to declare them in the throws clause of myMethod.

I thought that this was a great idea when I started programming Java in 1995. With the hindsight of a decade, we can see that it’s a disaster. The trouble is that every time you call a method that throws an exception, you create an immediate crisis: you break the build. Rather than conciously planning an error handling strategy, programmers do something, anything, to make the compiler shut up. Very often you see people bypass exceptions entirely, like this:

[05] public void someMethod() {
[06]    try {
[07]       objectA.anotherMethod();
[08]    } catch(SubsystemAScrewedUpException ex) { };
[09] }

Often you get this instead:

[10]    try {
[11]       objectA.anotherMethod();
[12]    } catch(SubsystemAScrewedUp ex) {
[13]        // something ill-conceived to keep the compiler happy
[14]    }
[15]    // Meanwhile,  the programmer makes a mistake here because
[16]    // writing an exception handler broke his concentration

This violates the first principle, to “do no harm.” It’s simple, and often correct, to pass the exception up to the calling function,

[17] public int someMethod() throws SubsystemAScrewedUp {

But, this still breaks the build, because now every function that calls someMethod() needs to do something about the exception. Imagine a program of some complexity that’s maintained by a few programmers, in which method A() calls method B() which calls method C() all the way to method F().

The programmer who works on method F() can change the signature of that method, but he may not have the authority to change the signature of the methods above. Depending on the culture of the shop, he might be able to do it himself, he might talk about it with the other programmers over IM, he might need to get the signatures of three managers, or he might need to wait until the next group meeting. If they keep doing this, however, A() might end up with a signature like

[18] public String A() throws SubsystemAScrewedUp, IOException, SubsystemBScrewedUp,
[19]   WhateverExceptionVendorZCreatedToWrapANullPointerException, ...

This is getting out of hand, and they realize they can save themselvesa lot of suffering by just writing

[20] public int someMethod() throws Exception {

all the time, which is essentially disabling the checked exception mechanism. Let’s not dismiss the behavior of the compiler out of hand, however, because it’s teaching us an important lesson: error handling is a holistic property of a program: an error handled in method F() can have implications for methods A()…E(). Errors often break the assumption of encapsulation, and require a global strategy that’s applied consistently throughout the code.

PHP, C# and other post-Java languages tend to not support checked exceptions. Unfortunately, checked exception thinking has warped other langages, so you’ll find that catch statements are used neurotically everywhere.

Exception To The Rule: Handling Exceptions Locally

Sometimes you can anticipate an exception, and know what exact action to take. Consider the case of a compiler, or a program that processes a web form, which might find more than one error in user input. Imagine something like (in C#):

[21] List errors=new List();
[22] uint quantity=0;
[23] ... other data fields ...
[24]
[25] try {
[26]   quantity=UInt32.Parse(Params["Quantity"]);
[27] } catch(Exception ex) {
[28]   errors.Add("You must enter a valid quantity");
[29] }
[30]
[31] ... other parsers/validators ...
[32]
[33] if (errors.Empty()) {
[34]    ... update database,  display success page ...
[35] } else {
[36]    ... redraw form with error messages ...
[37] }

Here it makes sense to catch the exception locally, because the exceptions that can happen on line [22] are completely handled, and don’t have an effect on other parts of the application. The one complaint you might make is that I should be catching something more specific than Exception. Well, that would bulk the code up considerably and violate the DRY (don’t repeat yourself) principle: UInt32.Parse can throw three different exceptions: ArgumentNullException, FormatException, and OverflowException. On paper, the process of looking up the “Quantity” key in Params could throw an ArgumentNullException or a KeyNotFoundException.

I don’t think either ArgumentNullException can really happen, and I think the KeyNotFoundException would only occur in development, or if somebody was trying to submit the HTML form with an unauthorized program. Probably the best thing to do in either case would be to abort the script with a 500 error and log the details, but the error handling on line [24] is sane in that it prevents corruption of the database.

The handling of FormatException and OverflowException, in the other case, is fully correct. The user gets an error message that tells them what they need to do to fix the situation.

This example demonstrates a bit of why error handling is so difficult and why the perfect can be the enemy of the good: the real cause of an IOException could be a microscopic scratch on the surface of a hard drive, and operating system error, or the fact that somebody spilled a coke on a router in Detroit — diagnosing the problem and offering the right solution is an insoluble problem.

Fixing it up with finally

The first exception handling construct that should be on your fingertips is finally, not catch. Unfortunately, finally is a bit obscure: the pattern in most languages is

[38] try {
[39]    ... do something that might throw an exception ...
[40] } finally {
[41]    ... clean up ...
[42] }

The code in the finally clause get runs whether or not an exception is thrown in the try block. Finally is a great place to release resources, roll back transactions, and otherwise protect the state of the application by enforcing invariants. Let’s think back to the chain of methods A() through F(): with finally, the maintainer of B() can implement a local solution to a global problem that starts in F(): no matter what goes wrong downstream, B() can repair invariants and repair the damage. For instance, if B()’s job is to write something into a transactional data store, B() can do something like:

[43] Transaction tx=new Transaction();
[44] try {
[45]    ...
[46]    C();
[47]    ...
[48]    tx.Commit();
[49] } finally {
[50]    if (tx.Open)
[51]        tx.Rollback();
[52] }

This lets the maintainer of B() act defensively, and offer the guarantee that the persistent data store won’t get corrupted because of an exception that was thrown in the try block. Because B() isn’t catching the exception, it can do this without depriving upstream methods, such as A() from doing the same.

C# gets extra points because it has syntactic sugar that makes a simple case simple: The using directive accepts an IDisposable as an argument and wraps the block after it with a finally clause that calls the Dispose() method of the IDisposable. ASP.NET applications can fail catastrophically if you don’t Dispose() database connections and result sets, so

[53] using (var reader=sqlCommand.ExecuteReader()) {
[54]   ... scroll through result set ...
[55] }

is a widespread and effective pattern.

PHP loses points because it doesn’t support finally. Granted, finally isn’t as important in PHP, because all resources are released when a PHP script ends. The absense of finally, however, encourages PHP programmers to overuse catch, which perpetuates exception phobia. The PHP developers are adding great features to PHP 5.3, such as late static binding, so we can hope that they’ll change their mind and bring us a finally clause.

Where should you catch exceptions?

At high levels of your code, you should wrap units of work in a try-catch block. A unit of work is something that makes sense to either give up on or retry. Let’s work out a few simple examples:

Scripty Command line program: This program is going to be used predominantly by the person who wrote it and close associates, so it’s acceptable for the program to print a stack trace if it fails. The “unit of work” is the whole program.

Command line script that processes a million records: It’s likely that some records are corrupted or may trigger bugs in the program. Here it’s reasonable for the “unit of work” to be the processing of a single record. Records that cause exceptions should be logged, together with a stack trace of the exception.

Web application: For a typical web application in PHP, JSP or ASP.NET, the “unit of work” is the web request. Ideally the application returns a “500 Internal Error”, displays a message to the user (that’s useful but not overly revealing) and logs the stack trace (and possibly other information) so the problem can be investigated. If the application is in debugging mode, it’s sane to display the stack trace to the web browser.

GUI application: The “unit of work” is most often an event handler that’s called by the GUI framework. You push a button, something does wrong, then what? Unlike server-side web applications, which tend to assume that exceptions don’t involve corruption of static memory or of a database, GUI applications tend to shut down when they experience unexpected exceptions. [3] As a result, GUI applications tend to need infrastructure to convert common and predictable exceptions (such as network timeouts) into human readable error messages.

Mail server: A mail server stores messages in a queue and delivers them over a unreliable network. Exceptions occur because of full disks (locally or remote), network failures, DNS misconfigurations, remote server falures, and an occasionaly cosmic ray. The “unit of work” is the delivery of a single message. If an exception is thrown during delivery of the message, it stays in the queue: the mail server attempts to resend on a schedule, discarding it if it is unable to deliver after seven days.

What should you do when you’ve caught one?

That’s the subject of another article. Subscribe to my RSS feed if you want to read it when it’s ready. For now, I’ll enumerate a few questions to think about:

What do tell the end user?
What do you tell the developer?
What do you tell the sysadmin?
Will the error clear if up if we try to repeat this unit of work again?
How long would we need to wait?
Could we do something else instead?
Did the error happen because the state of the application is corrupted?
Did the error cause the state of the application to get corrupted?

Conclusion

Error handling is tough. Because errors come from many sources such as software defects, bad user input, configuration mistakes, and both permanent and transient hardware failures, it’s impossible for a developer to anticipate and perfectly handle everything that can go wrong. Exceptions are an excellent method of separating error handling logic from the normal flow of programs, but many programmers are too eager to catch exceptions: this either causes errors to be ignores, or entangles error handling with mainline logic, complicating both. The long term impact is that many programmers are afraid of exceptions and turn to return values as an error signals, which is a step backwards.

A strategy that (i) uses finally as the first resort for containing corrupting and maintaining invariants, (ii) uses catch locally when the exceptions thrown in an area are completely understood, and (iii) surrounds independent units of work with try-catch blocks is an effective basis for using exceptions that can be built upon to develop an exception handling policy for a particular application.

Error handling is a topic that I spend entirely too much time thinking about, so I’ll be writing about it more. Subscribe to my RSS Feed if you think I’ve got something worthwhile to say.

The Semantics of Dictionaries, Maps and Hashtables

Paul Houle — Thu, 17 Jul 2008 16:41:05 +0000

Introduction

The first language I used that put dictionaries on my fingertips was Perl, where the solution to just about any problem involved writing something like

$hashtable{$key}=$value;

Perl called a dictionary a ‘hash’, a reference to the way Perl implemented dictionaries. (Dictionaries are commonly implemented with hashtables and b-trees, but can also be implemented with linked-list and other structures.) The syntax of Perl is a bit odd, as you’d need to use $, # or % to reference scalar, array or hash variables in different contexts, but dictionaries with similar semantics became widespread in dynamic languages of that and succeeding generations, such as Python, PHP and Ruby. ‘Map’ container classes were introduced in Java about a decade ago, and programmers are using dictionaries increasingly in static languages such as Java and C#.

Dictionaries are a convenient and efficient data structure, but there’s are areas in which different mplementations behave differently: for instance, in what happens if you try to access an undefined key. I think that cross-training is good for developers, so this article compares this aspect of the semantics of dictionaries in four popular languages: PHP, Python, Java and C#.

Use cases

There are two use cases for dictionaries, so far as error handling is concerned:

When you expect to look up undefined values, and
When you don’t

Let’s look at three examples:

Computing A Histogram

One common use for a dictionary is for counting items, or recording that items in a list or stream have been seen. In C#, this is typically written something like:

[01] var count=Dictionary();
[02] foreach(int i in inputList) {
[03]   if (!counts.Contains(i))
[04]       count[i]=0;
[05]
[06]   count[i]=count[i]+1
[07] }

The Dictionary count now contains the frequency of items inputList, which could be useful for plotting a histogram. A similar pattern can be used if we wish to make a list of unique items found in inputList. In either case, looking up values that aren’t already in the hash is a fundamental part of the algorithm.

Processing Input

Sometimes, we’re getting input from another subsystem, and expect that some values might not be defined. For instance, suppose a web site has a search feature with a number of optional features, and that queries are made by GET requests like:

[08] search.php?q=kestrel
[09] search.php?q=admiral&page=5
[10] search.php?q=laurie+anderson&page=3&in_category=music&after_date=1985-02-07

In this case, the only required search parameter is “q”, the query string — the rest are optional. In PHP (like many other environments), you can get at GET variables via a hashtable, specifically, the $_GET superglobal, so (depending on how strict the error handling settings in your runtime are) you might write something like

[11] if ($_GET["q"])) {
[12]     throw new InvalidInputException("You must specify a query");
[13] }
[14]
[15] if($_GET["after_date"]) {
[16]  ... add another WHERE clause to a SQL query ...
[17] }

This depends, quite precisely, on two bits of sloppiness in PHP and Perl: (a) Dereferencing an undefined key on a hash returns an undefined value, which is something like a null. (b) both languages have a liberal definition of true and false in an if() statement. As a result, the code above is a bit quirky. The if() at line 11 evaluates false if q is undefined, or if q is the empty string. That’s good. However, both the numeric value 0 and the string “0″ also evaluate false. As a result, this code won’t allow a user to search for “0″, and will ignore an (invalid) after_date of 0, rather than entering the block at line [16], which hopefully would validate the date.

Java and C# developers might enjoy a moment of schadenfreude at the above example, but they’ve all seen, written and debugged examples of input handling code that just as quirky as the above PHP code — with several times the line count. To set the record straight, PHP programmers can use the isset() function to precisely test for the existence of a hash key:

[11] if (isset($_GET["q"]))) {
[12]     throw new InvalidInputException("You must specify a query");
[13] }

The unusual handling of “0″ is the kind of fault that can survive for years in production software: so long as nobody searches for “0″, it’s quite harmless. (See what you get if you search for a negative integer on Google.) The worst threat that this kind of permissive evaluation poses is when it opens the door to a security attack, but we’ve also seen that highly complex logic that strives to be “correct” in every situation can hide vulnerabilities too.

Relatively Rigid Usage

Let’s consider a third case: passing a bundle of context in an asynchronous communications call in a Silverlight application written in C#. You can do a lot worse than to use the signatures:

[14] void BeginAsyncCall(InputType input,Dictionary context,CallbackDelegate callback);
[15] void CallbackDelegate(ReturnType returnValue,Dictionary context);

The point here is that the callback might need to know something about the context in which the asynchronous function was called to do it’s work. However, this information may be idiosyncratic to the particular context in which the async function is called, and is certainly not the business of the asynchronous function. You might write something like

[16] void Initiator() {
[17]   InputType input=...;
[18]   var context=Dictionary();
[19]   context["ContextItemOne"]= (TypeA) ...;
[20]   context["ContextItemTwo"]= (TypeB) ...;
[21]   context["ContextItemThre"] = (TypeC) ...;
[22]   BeginAsyncCall(input,context,TheCallback);
[23] }
[24]
[25] void TheCallback(ReturnType output,Dictionary context) {
[26]   ContextItemOne = (TypeA) context["ContextItemOne"];
[27]   ContextItemTwo = (TypeB) context["ContextItemTwo"];
[28]   ContextItemThree = (TypeC) context["ContextItemThree"];
[29]   ...
[30] }

This is nice, isn’t it? You can pass any data values you want between Initiator and TheCallback. Sure, the compiler isn’t checking the types of your arguments, but loose coupling is called for in some situations. Unfortunately it’s a little too loose in this case, because we spelled the name of a key incorrectly on line 21.

What happens?

The [] operator on a dot-net Dictionary throws a KeyNotFoundException when we try to look up a key that doesn’t exist. I’ve set a global exception handler for my Silverlight application which, in debugging mode, displays the stack trace. The error gets quickly diagnosed and fixed.

Four ways to deal with a missing value

There are four tools that hashtables give programmers to access values associated with keys and detect missing values:

Test if key exists
Throw exception if key doesn’t exist
Return default value (or null) if key doesn’t exist
TryGetValue

#1: Test if key exists

PHP:    isset($hashtable[$key])
Python: key in hashtable
C#:     hashtable.Contains(key)
Java:   hashtable.containsKey(key)

This operator can be used together with the #2 or #3 operator to safely access a hashtable. Line [03]-[04] illustrates a common usage pattern.

One strong advantage of the explicit test is that it’s more clear to developers who spend time working in different language environments — you don’t need to remember or look in the manual to know if the language you’re working in today uses the #2 operator or the #3 operator.

Code that depends on the existence test can be more verbose than alternatives, and can be structurally unstable: future edits can accidentally change the error handling properties of the code. In multithreaded environments, there’s a potential risk that an item can be added or removed between the existance check and an access — however, the default collections in most environment are not thread-safe, so you’re likely to have worse problems if a collection is being accessed concurrently.

#2 Throw exception if key doesn’t exist

Python: hashtable[key]
C#:     hashtable[key]

This is a good choice when the non-existence of a key is really an exceptional event. In that case, the error condition is immediately propagated via the exception handling mechanism of the language, which, if properly used, is almost certainly better than anything you’ll develop. It’s awkward, and probably inefficient, if you think that non-existent keys will happen frequently. Consider the following rewrite of the code between [01]-[07]

[31] var count=Dictionary();
[32] foreach(int i in inputList) {
[33]   int oldCount;
[34]   try {
[35]       oldCount=count[i];
[36]   } catch (KeyNotFoundException ex) {
[37]       oldCount=0
[38]   }
[39]
[40]   count[i]=oldCount+1
[41] }

It may be a matter of taste, but I think that’s just awful.

#3 Return a default (often null) value if key doesn’t exist

PHP:    $hashtable[key] (well,  almost)
Python: hashtable.get(key, [default value])
Java:   hashtable.get(key)

This can be a convenient and compact operation. Python’s form is particularly attractive because it lets us pick a specific default value. If we use an extension method to add a Python-style GetValue operation in C#, the code from [01]-[07] is simplified to

[42] var count=Dictionary();
[43] foreach(int i in inputList)
[44]   count[i]=count.GetValue(i,0)+1;

It’s reasonable for the default default value to be null (or rather, the default value of the type), as it is in Python, in which case we could use the ??-operator to write

[42] var count=Dictionary();
[43] foreach(int i in inputList)
[44]   count[i]=(count.GetValue(i) ?? 0)+1;

(A ?? B equals A if A is not null, otherwise it equals B.) The price for this simplicity is two kinds of sloppiness:

We can’t tell the difference between a null (or default) value associated with a key and no value associated with a key
The potential of null value exports chaos into the environment: trying to use a null value can cause a NullReferenceException if we don’t explictly handle the null. NullReferenceExceptions don’t bother me if they happen locally to the function that returns them, but they can be a bear to understand when a null gets written into an instance variable that’s accessed much later.

Often people don’t care about 1, and the risk of 2 can be handled by specifying a non-null default value.

Note that PHP’s implementation of hashtables has a particularly annoying characteristic. Error handling in php is influenced by the error_reporting configuration variable which can be set in the php.ini file and other places. If the E_STRICT bit is not set in error_reporting, PHP barrels on past places where incorrect variable names are used:

[45] $correctVariableName="some value";
[46] echo "[{$corectValiableName}]"; // s.i.c.

In that case, the script prints “[]” (treats the undefined variable as an empty string) rather than displaying an error or warning message. PHP will give a warning message if E_STRICT is set, but then it applies the same behavior to hashtables: an error message is printed if you try to dereference a key that doesn’t exist — so PHP doesn’t consistently implement type #3 access.

#4 TryGetValue

There are quite a few methods (Try-* methods) in the .net framework that have a signature like this:

[47] bool Dictionary.TryGetValue(K key,out V value);

This method has crisp and efficient semantics which could be performed in an atomic thread-safe manner: it returns true if finds the key, and otherwise returns false. The output parameter value is set to the value associated with the key if a value is associated with the key, however, I couldn’t find a clear statement of what happens if the key isn’t found. I did a little experiment:

[48] var d = new Dictionary();
[49] d[1] = 5;
[50] d[2] = 7;
[51] int outValue = 99;
[52] d.TryGetValue(55, out outValue)
[53] int newValue = outValue;

I set a breakpoint on line 53 and found thate the value of outValue was 0, which is the default value of the int type. It seems, therefore, that TryGetValue returns the default value of the type when it fails to find the key. I wouldn’t count on this behavior, as it is undocumented.

The semantics of TryGetValue are crisp and precise. It’s particularly nice that something like TryGetValue could be implemented as an atomic operation, if the underyling class is threadsafe. I fear, however, that TryGetValue exports chaos into it’s environment. For instance, I don’t like declaring a variable without an assignment, like below:

[54] int outValue;
[55] if (d.TryGetValue(55,outValue)) {
[56] ... use outValue ...
[57] }

The variable outValue exists before the place where it’s set, and outside of the block where it has a valid value. It’s easy for future maintainers of the code to try to use outValue between lines [54]-[55] or after line [57]. It’s also easy to write something like 51], where the value 99 is completely irrelevant to the program. I like the construction

[58] if (d.Contains(key)) {
[59]    int value=d[key];
[60]    ... do something with value ...
[61] }

because the variable value only exists in the block [56]-[58] where it has a defined value.

Hacking Hashables

A comparison of hashtables in different languages isn’t just academic. If you don’t like the operations that your language gives you for hashtables, you’re free to implement new operations. Let’s take two simple examples. It’s nice to have a Python-style get() in PHP that never gives a warning message, and it’s easy to implement

[62] function array_get($array,$key,$defaultValue=false) {
[63]   if (!isset($array[$key]))
[64]      return $defaultValue;
[65]
[66]   return $array[$key];
[67] }

Note that the third parameter of this function uses a default value of false, so it’s possible to call it in a two-parameter form

[68] $value=array_get($array,$key);

with a default default of false, which is reasonable in PHP.

Extension methods make it easy to add a Python-style get() to C#; I’m going to call it GetValue() to be consistent with TryGetValue():

[69] public static class DictionaryExtensions {
[70]   public static V GetValue(this IDictionary dict, K key) {
[71]      return dict.GetValue(key, default(V));
[72]   }
[73]
[74]   public static V GetValue(this IDictionary dict, K key, V defaultValue) {
[75]      V value;
[76]      return dict.TryGetValue(key, out value) ? value : defaultValue;
[77]   }
[78] }

Conclusion

Today’s programming languages put powerful data structures, such as dictionaries, on your fingertips. When we look closely, we see subtle differences in the APIs used access dictionaries in different languages. A study of the different APIs and their consequences can help us think about how to write code that is more reliable and maintainable, and informs API design in every language

Extension Methods, Nulls, Namespaces and Precedence in C#

Paul Houle — Thu, 03 Jul 2008 16:34:25 +0000

Introduction

Extension methods are the most controversial feature that Microsoft has introduced in C# 3.0. Introduced to support the LINQ query framework, extension methods make it possible to define new methods for existing classes.

Although extension methods can greatly simplify code that uses them, many are concerned that they could transform C# into something that programmers find unrecognizable, or that C#’s namespace mechanisms are inadequate for managing large systems that use extension methods. Adoption of the LINQ framework, however, means that extension methods are here to stay, and that .net programmers need to understand how to use them effectively, and, in particular, how extension methods are different from regular methods.

This article discusses three ways in which extension methods differ from regular methods:

Extension methods can be called on null objects without throwing an exception
Extension methods cannot be called inside of a subclass without the use of ‘this’
The precedence rules for extension methods

The Menace of Null Values

The treatment of null values is one of the major weaknesses of today’s generation of languages. Although C# makes it possible to make nullable versions of value types, such as int? and guid?, there’s nothing in C# or Java like the “NOT NULL” declaration in SQL. As a result, handling nulls is a significant burden to writing correct code. Consider the simple case where you want to write

[01] someObject.DoSomething();

(where DoSomething is an ordinary instance method) When I type something like this, Resharper often highlights the line of code to warn me that someObject might be null. In some cases I might be confident that it never will, but if there is any change that it will be null, I’ll need to write something like

[02] if(someObject!=null) {
[03]    someObject.DoSomething();
[04] }

or maybe

[05] if(someObject==null)
[06]    return;
[07] someObject.DoSomething();

Alternatively I could accepted that an exception could be thrown by the invocation and decide to catch it (or not catch it) elsewhere in the application. In two cases out of three, one line of code gets bulked up to three. Worse than that, I need to make a decision at that point about what to when there’s an error condition — each decision is a case where somebody can make the wrong decision. Even if coders make the wrong decision 5% of the time, that would be 50 time bombs in your code for every 1000 method invocations. (Oliver Steele works out a particularly outrageous but common case where it takes 10 lines of null-checking code to protect 1 line of working code.)

Extension Methods Can Accept Null Values

What does this have to do with extension methods?

Unlike ordinary instance methods, extension methods do not automatically throw an exception if you call them on a null-valued object. Depending on your point of view, this can be (i) a gotcha, or (ii) a useful tool for simplifying your code. Here’s a little example:

[08] namespace ExtensionMethodTest {
[09]
[10]   static class ObjectExtension {
[11]       static public bool IsNull(this object o) {
[12]            return o == null;
[13]       }
[14]    }
[15]
[16]    class Program {
[17]        static void Main(string[] args) {
[18]            String s1 = "hello";
[19]            Console.WriteLine(s1.IsNull());
[20]            String s2 = null;
[21]            Console.WriteLine(s2.IsNull());
[22]            Console.WriteLine(s2.ToUpper());
[23]        }
[24]    }
[25] }

This example does something a bit bold: it attaches an extension method to object, adding an extenson method to every object in the system. This method, object.IsNull() returns true if object is null and false if it isn’t. Some people might see this as a nice example of syntactic sugar, others may see it as reckless. What’s important is that it works: if you run this program from the command line, line [21] will print ‘true’, while line [22], which uses an ordinary method, will throw a NullReferenceException.

Events and Extension Methods for Delegates

Chris Brandsma works out a practical example of how extension methods can be used to fix a broken and dangerous API. That is, the event handling mechanism commonly used in C#:

[26] public eventEventHandler OnLoadData;
[27] ...
[28] OnLoadData += SomeEventHandler;
[29] ...
[30] OnLoadData(this, argument);

OnLoadData is a MulticastDelegate. You can attach an unlimited number of real delegates to it. The sample above works great if you attach at least one delegate, but it fails with a NullReferenceException if you don’t. Perhaps this isn’t a problem for you, because you’re smart and you write

[31] if (OnLoadData==null) {
[32]     OnLoadData(this,argument)
[33] }

Unfortunately, there are two little problems with that. First, none of us program in a vacuum, so many of us will end up having to maintain or use objects where somebody forgot to include a null check. Secondly, the example between lines [31] and [33] isn’t thread safe. It’s possible that a method can be removed from OnLoadData between the time of the null check and the call!

It turns out that extension methods can be added to delegates, so Chris created a really nice extension method called Fire() that encapsulates the error check code between 31-33. Now you can just write the code you wanted to write:

[34] OnLoadData.Fire(this,argument);

and be confident that knowledge about threads and quirks of the type system is embedded in an extension method.

You must use this to access an extension method inside a subclass

Suppose you’re building a Silverlight application and you’d like your team to have an important method that incorporates something tricky on their fingertips. For instance, suppose you’re implementing error handling in an event handler that’s responding to a user-initiated event or an async callback. You can always write

[35] if(... something wrong...) {
[36]    ... several lines of code to display dialog box ...
[37]    return;
[38] }

But this is something that (i) programmers don’t want to do to begin with, (ii) that programmers will have to do tens or hundreds of times, and (iii) isn’t going to be in the main line of testing. It’s a quality problem waiting to happen. It’s imperative, therefore, to reduce the amount of code to do the right thing as much as possible… To make it easier to do the right thing than to do the wrong thing. It’s tempting to define an extension method like:

[39] public static void ErrorDialog(this UserControl c, string message) {
[40]    throw new ErrorMessageException(message);
[41] }

and catch the ErrorMessageException in the global error handler. (The “method that doesn’t return” is effective, because it avoids the need to repeat the return, which occassionaly seems to vanish when people write repetitive error handling code.) You’d think that this simplifies the code inside the UserControls you write to:

[42] if (... something wrong...)  {
[43]    ErrorDialog(...);
[44] }

But it turns out that line [43] doesn’t actually work, and you need to write

[45] if (... something wrong...) {
[46]    this.ErrorDialog(...);
[47] }

in which case you might as well use an ordinary static method on a helper class.

What’s wrong with extension methods?

I’ve seen two arguments against extension methods: (i) extension methods could make code hard to understand (and hence maintain) and (ii) extension methods are vulnerable to namespace conflicts. I think (i) is a specious argument, but (ii) is serious.

I think (i) splits into two directions. First there’s the practical problem that a programmer is going to see some code like

[48] String s="somebody@example.com";
[49] if (s.IsValidEmailAddress()) {
[50]     ... do something ...
[51] }

and wonder where the heck IsValidEmailAddress() comes from, where it’s documented, and so forth. Practically, Visual Studio understands extension methods well, so a user that clicks on “Go To Definition” is going to get a quick answer.

Going further, however, one can imagine that extension methods could transform C# unrecognizably: I think of a friend of mine who, in the 1980′s, liked FORTRAN better than C, and abused preprocessor macros so he could write C code that looked like FORTRAN. This is connected with a fear of lambda expressions, and other features that derive from functional programming. For instance, that beginning programmers just won’t get it.

We’ll see how it all works out, but I think that new features in C# are going to help the language evolve in a way more like jquery and prototype have transformed javascript. Microsoft is bringing concepts that have been locked in the ivory tower for decades into the mainstream: all programming languages are going to benefit in the long term.

Extension methods, precedence and namespaces

Here’s the killer.

I can make extension methods available by just adding a namespace to my .cs file with a using directive. The compiler scans the namespace for extension methods in static classes, and makes them available. Pretty easy, right? Well, what happens if two extension methods with the same name get declared in two namespaces which get included in the same file? What if we define an extension method on class A, but there’s a conventional method with the same name on class B? What if file One.cs uses namesspace C, and Two.cs uses namespace D, so that ThisExtensionMethod means something different in One.cs and Two.cs?

There are real problems in how extension methods interact with namespaces. These problems aren’t as fatal as namespace conflicts were in C (and C++ before namespaces), but they are for real.

One answer is to avoid the use of extension methods entirely, but that causes the loss of the benefits. Anyone who uses extension methods should take a close look at the C# version 3.0 specification and think about how precedence rules effect their work:

(i) Instance methods take precedence over extension methods. The definition of an instance method makes extension methods with the same name inaccessable. This happens at the level of methods, not method groups, so two methods with the same name but different signatures can be handled by an extension method and instance method respectively.
(ii) Once the compiler tries extension methods, processing works backwards from the closest enclosing namespace declaration outward, trying extension methods defined in using groups.
(iii) The compiler throws an error when there are two or more extension methods that are candidates for a spot.

Matt Manela demonstrates an interesting example on the MSDN forums. With three examples, he demonstrates that the existence of an instance method (that overrides both extension methods) will suppress the generation of an error message about a conflict between extension methods. This indicates that potentially conflicting extension methods in two namespaces will only cause an error if an attempt is made to use them.

Mitigating Namespace Conflicts

Overall, conflicts between extension methods in different namespaces will not result in catastrophic consequences:

The compiler raises an error if there is any ambiguity as to which extension method to apply at a particular invocation — code won’t silently change behavior upon adding a new namespace in a using directive.
The compiler does not throw an error if potentially conflicting extension methods are declared in two different namespaces including in distinct using directives if those extension methods are not used — therefore, conflicting extension methods won’t automatically prevent you from using any namespaces you choose.
If there is a conflict, either between two extension methods or an extension method and an instance methods, you can always call a specific extension method like an ordinary static example. For instance, in the case above:

ObjectExtension.IsNull(someObject);

You won’t end up in a situation where an extension method becomes unavailable because of a conflict — you’ll just be forced to use an uglier syntax. I do see two real risks:

You can end up using an extension method that you don’t expect if you’re not keeping track of which using directives are in your file, and
An instance method can silently shadow an extension method. A change in the definition of a method could cause the behavior of a (former) extension method cal to change in a suprising way. On the other hand, this could be a useful behavior if you’d like a subclass to override a behavior defined in an extension method.

A common bit of advice that I’ve seen circulating is that extension methods should be defined in separate namespaces, so that it would be possible to include or not include extension methods associated with a namespace to avoid conflicts. I think this is based on superstition, for, as we’ve seen, conflicting extension methods do not preclude the use of two namespaces; this advice is certainly not followed in the System.Linq namespace, which defines a number of valuable extension methods in the System.Linq.Enumerable static class.

Conclusion

We’re still learning how to use extension methods effectively. Although extension methods have great promise, they’re difference from ordinary instance methods in a number of ways. Some of these, like the difference in null handling, are minor, and could potentially be put to advantage. Others, such as the interaction with namespaces in large projects, are more challenging. It’s time to start building on our experiences to develop effective patterns for using extension methods.

Getting back to the UI Thread in Silverlight 2 Beta 2

Paul Houle — Wed, 25 Jun 2008 20:54:24 +0000

The problem

Silverlight 2 Beta 2 has changed the concurrency model for asynchronous communications. In Silverlight 2 Beta 1, asynchronous requests always returned on the UI Thread. This was convenient, since updates to the user interface can only be done via the UI thread. As of Silverlight 2 Beta 2, asynchronous callbacks are fired in worker threads that come from a thread pool: although this potentially allows for better performance via concurrency, it increases the risk for race conditions between callbacks – more importantly, some mechanism is necessary to make code that updates the user interface run in the UI thread.

A solution

It’s straightforward to execute a function in the UI thread by using the Dispatcher property of any ScriptObject The tricky part is that ScriptObjects are part of the user interface, so you can only access the Dispatcher property from the UI thread. At first this seems like a chicken-and-egg situation: you need a Dispatcher to get to the UI thread, but you need to be in the UI thread to get a Dispatcher.

This dilemma can be resolved by accessing a Dispatcher in your App.xaml.cs file and stashing it away in a static variable on application startup:

private void Application_Startup(object sender, StartupEventArgs e) {
    ...
    UIThread.Dispatcher = RootVisual.Dispatcher;
}

UIThread is a simple static class:

public static class UIThread {
    public static Dispatcher Dispatcher {get; set;}
    public static void Run(Action a) {
        Dispatcher.BeginInvoke(a);
    }
}

At some point in the future, you can transfer execution to the UIThread by scheduling a function to run in it.

public class ProcessHttpResponse(...) {
   ...
   UIThread.Run(UpdateUserInterface);
}

The parameter of Run is an Action delegate, which is a function that returns void and takes no parameters. That’s fine and good, but what if you want to pass some parameters to the function that updates the UI. The usual three choices for passing state in asynchronous applications apply, but it’s particularly easy and fun to use a closure here:

public class ProcessHttpResponse(...) {
    ...
    var elementToUpdate=...;
    var updateWithValue=...;

    UIThread.Run(delegate() {
        UpdateUserInterface(elementToUpdate,updateWithValue)
    });
}

When to return?

If your application is complex, and you have nested asynchronous calls, you’re left with an interesting question: where is the best place to switch execution to the UI thread? You can switch to the UI Thread as soon as you get back from an HttpRequest or a WCF call and you must switch to the UI Thread before you access any methods or properties of the user interface. What’s best?

It is simple and safe to switch to the UI Thread immediately after requests return from the server. If you’re consistent in doing this, you’ll never have trouble accessing the UI thread, and you’ll never have trouble with race conditions between returning communication responses. On the other hand, you’ll lose the benefit of processing responses concurrently, which can improve speed and responsiveness on today’s multi-core computers.

It’s straightforward to exploit concurrency when a requests can be processed independently. For instance, imagine a VRML viewer written in Silverlight. Displaying a VRML would require the parsing of a file, the construction of the scene graph and the initialization of a 3-d engine, which may require building data structures such as a BSP Tree. Doing all of this work in the UI Thread would make the application lock up while a model is loading — it would be straightforward, instead, to construct all of the data structures required by the 3-d engine, and attach the fully initialized 3-d engine to the user interface as the last step. Since the data structures would be independent of the rest of the application, thread safety and liveness is a nonissue.

Matters can get more complicated, however, if the processing of a request requires access to application-wide data; response handlers running in multiple threads will probably corrupt shared data structures unless careful attention is paid to thread safety. One simple approach is to always access shared data from the UI Thread, and to transfer control to the UI Thread with UIThread.Run before accessing shared variables.

Conclusion

Silverlight 2 Beta 2 introduces a major change in the concurrency model for asynchronous communication requests. Unlike SL2B1, where asynchronous requests executed on the user interface thread, SL2B2 launches asynchronous callbacks on multiple threads. Although this model offers better performance and responsiveness, it requires Silverlight programmers to explicitly transfer execution to the UI thread before accessing UI objects: most SL2B1 applications will need to be reworked.

This article introduces a simple static class, UIThread, which makes it easy to schedule execution in the UI Thread.

Prefix-casting versus as-casting in C#

Paul Houle — Fri, 13 Jun 2008 21:23:22 +0000

Introduction

This is a story of two types: GenericType and SpecificType, where GenericType is a superclass of SpecificType. There are two types of explicit cast in C#:

The Prefix cast:

[01] GenericType g=...;
[02] SpecificType t=(SpecificType) g;

The as cast:

[03] GenericType g=...;
[04] SpecificType t=g as SpecificType;

Most programmers have a habit of using one or the other — this isn’t usually a conscious decision, but more of a function of which form a programmer saw first. I, for instance, programmed in Java before I learned C#, so I was already in the prefix cast habit. People with a Visual Basic background often do the opposite. There are real differences between the two casting operators which can make a difference in the reliability and performance of your application.

Prefix casting: Reliable Casting

The major difference between prefix- and as-casting is what happens when the cast fails. Imagine, for instance, that g is really an instance of AnotherSpecificType, which is not a subclass of SpecificType. In this case, the prefix-cast throws an exception at line [2] — however, the as-cast returns null when the cast fails, letting the execution of the program barrel on.

It’s easier to implement correct error handling in programs that use prefix-casting, and programs that use prefix-casting are easier to debug. Prefix-casting causes an exception to be thrown at the moment of the cast, where the problem is obvious. As-casting, on the other hand, can cause an exception to be thrown at the moment where the SpecificType t is referenced. The used of the SpecificType can be far away in the program: it can be in another method, or another class — it could even happen hours after the cast is performed. Be it in development or production, bugs caused by corrupted data structures can be maddeningly difficult to find and correct.

As-casting: Fast Casting

If it’s harder to write correct programs with as-casting, why would anybody use it? For one thing, as-casting is faster than prefix casting by quite a lot. Benchmarks show that as-casting is about five times faster than prefix casting. That said, the performance of casting is only going to matter in the innermost of loops of most programs. The fastest kind of casting is no casting at all: it’s best to use generics to eliminate the need for casting when possible and to move casting outside loops. (Generics also improve the reliability of your code because they help C#’s static type checking catch mistakes.)

There are some cases where as-casting could be convenient, for instance, when you expect the cast to fail sometimes. Often I like to ‘tag’ classes with interfaces that specify certain behaviors. For example,

[05] public Interface IBoppable {
[06]     void Bop();
[07] }

Now i might want to loop through a bunch of Thing objects, and bop all the ones that implement IBoppable: it’s reasonable to use as-casting here:

[08] List list=...
[09] foreach(Thing thing in list) {
[10]    List boppable=thing as IBoppable;
[11]    if (boppable !=null) {
[12]        boppable.Bop()
[13]    }
[14] }

It’s OK to use as-casting if you’re going to check to see if the value is null immediately after the cast. The above code is correct, but has the bad smell that the boppable variable continues to exist in the block after the cast… It’s still there for a later maintainer to use erroneously. In cases like this, code can be made clearer with the is operator:

[15] List list=...
[16] foreach(Thing thing in list) {
[17]    if(thing is IBoppable) {
[18]        ((IBoppable) boppable).Bop()
[19]    }
[20] }

(Speed freaks can use as-cast on line 18, as we know it’s not going to fail.)

The pattern of testing for null after an as-cast has a few problems. (i) It doesn’t distinguish between the case of the original object being null from the case of the original object being the wrong type and (ii) correct error handling often requires more contorted logic than using an exception — and once you added test logic, you’ve lost the speed advantage of as-casting.

Conclusion

C# offers two casting operators: the prefix-cast and the as-cast. Although the two operators compile to different op-codes in the CLR, the practical difference between them is in how they handle failed casts. Prefix-cast throws an exception on cast failure, while as-cast returns null. It’s easier to implement correct error handling when you use prefix cast, because it doesn’t require manual checks for null values that can cause problems in distant parts of your program. Prefix-cast should be the default cast operator on your fingertips, that you use for everyday situations — reserve as-cast for special cases where performance matters. (For best performance, however, eliminate the cast entirely.)

Keeping Track Of State In Asynchronous Callbacks

Paul Houle — Mon, 02 Jun 2008 17:13:43 +0000

When you’re writing applications that use asynchronous callbacks (i.e. Silverlight, AJAX, or GWT) you’ll eventually run into the problem of keeping track of the context that a request is being done in. This isn’t a problem in synchronous programming, because local variables continue to exist after one function calls another function synchronously:

int AddToCount(int amount,string countId)  {
   int countValue=GetCount(countId);
   return countValue+amount;
}

This doesn’t work if the GetCount function is asynchronous, where we need to write something like

int AddToCountBegin(int amount,string countId,CountCallback outerCallback) {
     GetCountBegin(countId,AddToCountCallback);
}

void AddToCountCallback(int countValue) {
    ... some code to get the values of amount and outerCallback ...
    outerCallback(countValue+amount);
}

Several things change in this example: (i) the AddToCount function gets broken up into two functions: one that does the work before the GetCount invocation, and one that does the work after GetCount completes. (ii) We can’t return a meaningful value from AddToCountCallback, so it needs to ‘return’ a value via a specified callback function. (iii) Finally, the values of outerCallback and amount aren’t automatically shared between the functions, so we need to make sure that they are carried over somehow.
There are three ways of passing context from a function that calls and asynchronous function to the callback function:

As an argument to the callback function
As an instance variable of the class of which the callback function is a class
Via a closure

Let’s talk about these alternatives:

1. Argument to the Callback Function

In this case, a context object is passed to the asynchronous function, which passes the context object to the callback. The advantage here is that there aren’t any constraints on how the callback function is implemented, other than by accepting the context object as a callback. In particular, the callback function can be static. A major disadvantage is that the asynchronous function has to support this: it has to accept a state object which it later passes to the callback function.

The implementation of HttpWebRequest.BeginGetResponse(AsyncCallback a,Object state) in the Silverlight libraries is a nice example. If you wish to pass a context object to the AsyncCallback, you can pass it in the second parameter, state. Your callback function will implement the AsyncCallback delegate, and will get something that implements IAsyncResult as a parameter. The state that you passed into BeginGetResponse will come back in the IAsyncResult.AsyncState property. For example:

class MyHttpContext {
	public HttpWebRequest Request;
        public SomeObject FirstContextParameter;
        public AnotherObject AnotherContextParameter;
}

protected void myHttpCallback(IAsyncResult abstractResult) {
	MyHttpContext context = (MyHttpContext) abstractResult.AsyncState;
	HttpWebResponse Response=(HttpWebResponse) context.Request.EndGetResponse(abstractResult);
}

public doHttpRequest(...) {
	...
        MyHttpContext context=new MyHttpContext();
	context.Request=Request;
	context.FirstContextParameter = ... some value ...;
	context.AnotherContextParameter = .. another value ...;
	Request.BeginGetResponse();
	Request.Callback(myHttpCallback,context);
}

Note that, in this API, the Request object needs to be available in myHttpCallback because myHttpCallbacks get the response by calling the HttpWebResponse.EndGetResponse() method. We could simply pass the Request object in the state parameter, but we’re passing an object we defined, myHttpCallback, because we’d like to carry additional state into myHttpCallback.

Note that the corresponding method for doing XMLHttpRequests in GWT, the use of a RequestBuilder object doesn’t allow using method (1) to pass context information — there is no state parameter. in GWT you need to use method (2) or (3) to pass context at the RequestBuilder or GWT RPC level. You’re free, of course, to use method (1) when you’re chaining asynchronous callbacks: however, method (2) is more natural in Java where, instead of a delegate, you need to pass an object reference to designate a callback function.

2. Instance Variable Of The Callback Function’s Class

Functions (or Methods) are always attached to a class in C# and Java: thus, the state of a callback function can be kept in either static or instance variables of the associated class. I don’t advise using static variables for this, because it’s possible for more than one asynchronous request to be flight at a time: if two request store state in the same variables, you’ll introduce race conditions that will cause a world of pain. (see how race conditions arise in asynchronous communications.)

Method 2 is particularly effective when both the calling and the callback functions are methods of the same class. Using objects whose lifecycle is linked to a single asynchronous request is an effective way to avoid conflicts between requests (see the asynchronous command pattern and asynchronous functions.)

Here’s an example, lifted from the asynchronous functions article:

    public class HttpGet : IAsyncFunction
    {
        private Uri Path;
        private CallbackFunction OuterCallback;
        private HttpWebRequest Request;

        public HttpGet(Uri path)
        {
            Path = path;
        }

        public void Execute(CallbackFunction outerCallback)
        {
            OuterCallback = outerCallback;
            try
            {
                Request = (HttpWebRequest)WebRequest.Create(Path);
                Request.Method = "GET";
                Request.BeginGetRequestStream(InnerCallback,null);
            }
            catch (Exception ex)
            {
                OuterCallback(CallbackReturnValue.CreateError(ex));
            }
        }

        public void InnerCallback(IAsyncResult result)
        {
            try
            {
                HttpWebResponse response = (HttpWebResponse) Request.EndGetResponse(result);
                TextReader reader = new StreamReader(response.GetResponseStream());
                OuterCallback(CallbackReturnValue.CreateOk(reader.ReadToEnd()));
            } catch(Exception ex) {
                OuterCallback(CallbackReturnValue.CreateError(ex));
            }
        }
    }

Note that two pieces of context are being passed into the callback function: an HttpWebRequest object named Request (necessary to get the response) and a CallbackFunction delegate named OuterCallback that receives the return value of the asynchronous function.

Unlike Method 1, Method 2 makes it possible to keep an unlimited number of context variables that are unique to a particular case in a manner that is both typesafe and oblivious to the function being called — you don’t need to cast an Object to something more specific, and you don’t need to create a new class to hold multiple variables that you’d like to pass into the callback function.

Method 2 comes into it’s own when it’s used together with polymorphism, inheritance and initialization patterns such as the factory pattern: if the work done by the requesting and callback methods can be divided into smaller methods, a hierarchy of asynchronous functions or commands can reuse code efficiently.

3. Closures

In both C# and Java, it’s possible for a method defined inside a method to have access to variables in the enclosing method. In C# this is a matter of creating an anonymous delegate, while in Java it’s necessary to create an anonymous class.

Using closures results in the shortest code, if not the most understandable code. In some cases, execution proceeds in a straight downward line through the code — much like a synchronous version of the code. However, people sometimes get confused the indentation, and, more seriously, parameters after the closure definition and code that runs immediately after the request is fired end up in an awkward place (after the definition of the callback function.)

    public class HttpGet : IAsyncFunction
    {
        private Uri Path;

        public HttpGet(Uri path)
        {
            Path = path;
        }

        public void Execute(CallbackFunction outerCallback)
        {
            OuterCallback = outerCallback;
            try
            {
                HttpWebRequest request = (HttpWebRequest)WebRequest.Create(Path);
                Request.Method = "GET";
                Request.BeginGetRequestStream(delegate(IAsyncResult result) {
	            try {
                        response = request.EndGetResponse(result);
                        TextReader reader = new StreamReader(response.GetResponseStream());
                        outerCallback(CallbackReturnValue.CreateOk(reader.ReadToEnd()));
                    } catch(Exception ex) {
                        outerCallback(CallbackReturnValue.CreateError(ex));
                    }

            },null); // <--- note parameter value after delegate definition

            }
            catch (Exception ex)
            {
                outerCallback(CallbackReturnValue.CreateError(ex));
            }
        }
    }

The details are different in C# and Java: anonymous classes in Java can access local, static and instance variables from the enclosing context that are declared final — this makes it impossible for variables to be stomped on while an asynchronous request is in flight. C# closures, on the other hand, can access only local variables: most of the time this prevents asynchronous requests from interfering with one another, unless a single method fires multiple asynchronous requests, in which case counter-intuitive things can happen.

Conclusion

In addition to receiving return value(s), callback functions need to know something about the context they run in: to write reliable applications, you need to be conscious of where this information is; better yet, a strategy for where you’re going to put it. Closures, created with anonymous delegates (C#) or classes (Java) produce the shortest code, but not necessarily the clearest. Passing context in an argument to the callback function requires the cooperation of the called function, but it makes few demands on the calling and callback functions: the calling and callback functions can both be static. When a single object contains both calling and callback functions, context can be shared in a straightforward and typesafe manner; and when the calling and callback functions can be broken into smaller functions, opportunities for efficient code reuse abound.

Generation 5 » Dot Net

First-Class Functions and Logical Negation in C#

Introduction

Why?

Try 1: Extension Method

Try 2: Static Method

Extending IEnumerable

Conclusion

Using Linq To Tell if the Elements of an IEnumerable Are Distinct

The Problem

One Solution

Analysis

Subverting XAML: How To Inherit From Silverlight User Controls

The Problem

Should We Just Give Up?

Or Should We Just Cheat?

Step-by-Step Refactoring

Extending The XamlPanel

Related articles

Conclusion

What do you do when you’ve caught an exception?

Abort, Retry, Ignore

The cause and effect of errors

Repair

Release and Rollback

Notification

Retry I: Do it again!

Retry II: If first you don’t succeed…

Ignore?

Conclusion

Stop Catching Exceptions!

Motivation

The Tragedy of Checked Exceptions

Exception To The Rule: Handling Exceptions Locally

Fixing it up with finally

Where should you catch exceptions?

What should you do when you’ve caught one?

Conclusion

The Semantics of Dictionaries, Maps and Hashtables

Introduction

Use cases

Computing A Histogram

Processing Input

Relatively Rigid Usage

Four ways to deal with a missing value

#1: Test if key exists

#2 Throw exception if key doesn’t exist

#3 Return a default (often null) value if key doesn’t exist

#4 TryGetValue

Hacking Hashables

Conclusion

Extension Methods, Nulls, Namespaces and Precedence in C#

Introduction

The Menace of Null Values

Extension Methods Can Accept Null Values

Events and Extension Methods for Delegates

You must use this to access an extension method inside a subclass

What’s wrong with extension methods?

Extension methods, precedence and namespaces

Mitigating Namespace Conflicts

Conclusion

Getting back to the UI Thread in Silverlight 2 Beta 2

The problem

A solution

When to return?

Conclusion

Prefix-casting versus as-casting in C#

Introduction

Prefix casting: Reliable Casting

As-casting: Fast Casting

Conclusion

Keeping Track Of State In Asynchronous Callbacks

1. Argument to the Callback Function

2. Instance Variable Of The Callback Function’s Class

3. Closures

Conclusion