Introduction

Extension methods are the most controversial feature that Microsoft has introduced in C# 3.0.  Introduced to support the LINQ query framework,  extension methods make it possible to define new methods for existing classes.

Although extension methods can greatly simplify code that uses them,  many are concerned that they could transform C# into something that programmers find unrecognizable,  or that C#’s namespace mechanisms are inadequate for managing large systems that use extension methods.  Adoption of the LINQ framework,  however,  means that extension methods are here to stay,  and that .net programmers need to understand how to use them effectively,  and,  in particular,  how extension methods are different from regular methods.

This article discusses three ways in which extension methods differ from regular methods:

1. Extension methods can be called on null objects without throwing an exception
2. Extension methods cannot be called inside of a subclass without the use of ‘this’
3. The precedence rules for extension methods

The Menace of Null Values

The treatment of null values is one of the major weaknesses of today’s generation of languages.  Although C# makes it possible to make nullable versions of value types,  such as int? and guid?,  there’s nothing in C# or Java like the “NOT NULL” declaration in SQL.  As a result,  handling nulls is a significant burden to writing correct code.  Consider the simple case where you want to write

[01] someObject.DoSomething();

(where DoSomething is an ordinary instance method)  When I type something like this,  Resharper often highlights the line of code to warn me that someObject might be null.  In some cases I might be confident that it never will,  but if there is any change that it will be null,  I’ll need to write something like

[02] if(someObject!=null) {
[03]    someObject.DoSomething();
[04] }

or maybe

[05] if(someObject==null)
[06]    return;
[07] someObject.DoSomething();

Alternatively I could accepted that an exception could be thrown by the invocation and decide to catch it (or not catch it) elsewhere in the application.  In two cases out of three,  one line of code gets bulked up to three.  Worse than that,  I need to make a decision at that point about what to when there’s an error condition — each decision is a case where somebody can make the wrong decision.  Even if coders make the wrong decision 5% of the time,  that would be 50 time bombs in your code for every 1000 method invocations.  (Oliver Steele works out a particularly outrageous but common case where it takes 10 lines of null-checking code to protect 1 line of working code.)

Extension Methods Can Accept Null Values

What does this have to do with extension methods?

Unlike ordinary instance methods,  extension methods do not automatically throw an exception if you call them on a null-valued object.  Depending on your point of view,  this can be (i) a gotcha,  or (ii) a useful tool for simplifying your code.  Here’s a little example:

[08] namespace ExtensionMethodTest {
[09]
[10]   static class ObjectExtension {
[11]       static public bool IsNull(this object o) {
[12]            return o == null;
[13]       }
[14]    }
[15]
[16]    class Program {
[17]        static void Main(string[] args) {
[18]            String s1 = "hello";
[19]            Console.WriteLine(s1.IsNull());
[20]            String s2 = null;
[21]            Console.WriteLine(s2.IsNull());
[22]            Console.WriteLine(s2.ToUpper());
[23]        }
[24]    }
[25] }

This example does something a bit bold:  it attaches an extension method to object,   adding an extenson method to every object in the system.  This method,  object.IsNull() returns true if object is null and false if it isn’t.  Some people might see this as a nice example of syntactic sugar,  others may see it as reckless.  What’s important is that it works:  if you run this program from the command line,  line [21] will print ‘true’,  while line [22],  which uses an ordinary method,  will throw a NullReferenceException.

Events and Extension Methods for Delegates

Chris Brandsma works out a practical example of how extension methods can be used to fix a broken and dangerous API.  That is,  the event handling mechanism commonly used in C#:

[26] public eventEventHandler<EventArgs> OnLoadData;
[27] ...
[28] OnLoadData += SomeEventHandler;
[29] ...
[30] OnLoadData(this, argument);

OnLoadData is a MulticastDelegate.  You can attach an unlimited number of real delegates to it.  The sample above works great if you attach at least one delegate,  but it fails with a NullReferenceException if you don’t.  Perhaps this isn’t a problem for you,  because you’re smart and you write

[31] if (OnLoadData==null) {
[32]     OnLoadData(this,argument)
[33] }

Unfortunately,  there are two little problems with that.  First,  none of us program in a vacuum,   so many of us will end up having to maintain or use objects where somebody forgot to include a null check.   Secondly,  the example between lines [31] and [33] isn’t thread safe.  It’s possible that a method can be removed from OnLoadData between the time of the null check and the call!

It turns out that extension methods can be added to delegates,  so Chris created a really nice extension method called Fire() that encapsulates the error check code between 31-33.   Now you can just write the code you wanted to write:

[34] OnLoadData.Fire(this,argument);

and be confident that knowledge about threads and quirks of the type system is embedded in an extension method.

You must use this to access an extension method inside a subclass

Suppose you’re building a Silverlight application and you’d like your team to have an important method that incorporates something tricky on their fingertips.  For instance,  suppose you’re implementing error handling in an event handler that’s responding to a user-initiated event or an async callback.   You can always write

[35] if(... something wrong...) {
[36]    ... several lines of code to display dialog box ...
[37]    return;
[38] }

But this is something that (i) programmers don’t want to do to begin with,  (ii) that programmers will have to do tens or hundreds of times,  and (iii) isn’t going to be in the main line of testing.  It’s a quality problem waiting to happen.  It’s imperative,  therefore,  to reduce the amount of code to do the right thing as much as possible…  To make it easier to do the right thing than to do the wrong thing.   It’s tempting to define an extension method like:

[39] public static void ErrorDialog(this UserControl c, string message) {
[40]    throw new ErrorMessageException(message);
[41] }

and catch the ErrorMessageException in the global error handler.  (The “method that doesn’t return” is effective,  because it avoids the need to repeat the return,  which occassionaly seems to vanish when people write repetitive error handling code.)  You’d think that this simplifies the code inside the UserControls you write to:

[42] if (... something wrong...)  {
[43]    ErrorDialog(...);
[44] }

But it turns out that line [43] doesn’t actually work,  and you need to write

[45] if (... something wrong...) {
[46]    this.ErrorDialog(...);
[47] }

in which case you might as well use an ordinary static method on a helper class.

What’s wrong with extension methods?

I’ve seen two arguments against extension methods:  (i) extension methods could make code hard to understand (and hence maintain) and (ii) extension methods are vulnerable to namespace conflicts.  I think (i) is a specious argument,  but (ii) is serious.

I think (i) splits into two directions.  First there’s the practical problem that a programmer is going to see some code like

[48] String s="somebody@example.com";
[49] if (s.IsValidEmailAddress()) {
[50]     ... do something ...
[51] }

and wonder where the heck IsValidEmailAddress() comes from,  where it’s documented,  and so forth.  Practically,  Visual Studio understands extension methods well,  so a user that clicks on “Go To Definition” is going to get a quick answer.

Going further,  however,  one can imagine that extension methods could transform C# unrecognizably:  I think of a friend of mine who,  in the 1980′s,  liked FORTRAN better than C,  and abused preprocessor macros so he could write C code that looked like FORTRAN.   This is connected with a fear of lambda expressions,  and other features that derive from functional programming.  For instance,  that beginning programmers just won’t get it.

We’ll see how it all works out,  but I think that new features in C# are going to help the language evolve in a way more like jquery and prototype have transformed javascript.  Microsoft is bringing concepts that have been locked in the ivory tower for decades into the mainstream:  all programming languages are going to benefit in the long term.

Extension methods,  precedence and namespaces

Here’s the killer.

I can make extension methods available by just adding a namespace to my .cs file with a using directive.  The compiler scans the namespace for extension methods in static classes,  and makes them available.  Pretty easy,  right?  Well,  what happens if two extension methods with the same name get declared in two namespaces which get included in the same file?  What if we define an extension method on class A,  but there’s a conventional method with the same name on class B?  What if file One.cs uses namesspace C,  and Two.cs uses namespace D,   so that ThisExtensionMethod means something different in One.cs and Two.cs?

There are real problems in how extension methods interact with namespaces.  These problems aren’t as fatal as namespace conflicts were in C (and C++ before namespaces),  but they are for real.

One answer is to avoid the use of extension methods entirely,  but that causes the loss of the benefits.  Anyone who uses extension methods should take a close look at the C# version 3.0 specification and think about how precedence rules effect their work:

(i) Instance methods take precedence over extension methods.  The definition of an instance method makes extension methods with the same name inaccessable.  This happens at the level of methods,  not method groups,  so two methods with the same name but different signatures can be handled by an extension method and instance method respectively.
(ii) Once the compiler tries extension methods,  processing works backwards from the closest enclosing namespace declaration outward,  trying extension methods defined in using groups.
(iii) The compiler throws an error when there are two or more extension methods that are candidates for a spot.

Matt Manela demonstrates an interesting example on the MSDN forums.  With three examples,  he demonstrates that the existence of an instance method (that overrides both extension methods) will suppress the generation of an error message about a conflict between extension methods.  This indicates that potentially conflicting extension methods in two namespaces will only cause an error if an attempt is made to use them.

Mitigating Namespace Conflicts

Overall,  conflicts between extension methods in different namespaces will not result in catastrophic consequences:

1. The compiler raises an error if there is any ambiguity as to which extension method to apply at a particular invocation — code won’t silently change behavior upon adding a new namespace in a using directive.
2. The compiler does not throw an error if potentially conflicting extension methods are declared in two different namespaces including in distinct using directives if those extension methods are not used — therefore,  conflicting extension methods won’t automatically prevent you from using any namespaces you choose.
3. If there is a conflict,  either between two extension methods or an extension method and an instance methods,  you can always call a specific extension method like an ordinary static example.  For instance,  in the case above:

ObjectExtension.IsNull(someObject);

You won’t end up in a situation where an extension method becomes unavailable because of a conflict — you’ll just be forced to use an uglier syntax.  I do see two real risks:

1. You can end up using an extension method that you don’t expect if you’re not keeping track of which using directives are in your file,  and
2. An instance method can silently shadow an extension method.  A change in the definition of a method could cause the behavior of a (former) extension method cal to change in a suprising way.  On the other hand,  this could be a useful behavior if you’d like a subclass to override a behavior defined in an extension method.

A common bit of advice that I’ve seen circulating is that extension methods should be defined in separate namespaces,  so that it would be possible to include or not include extension methods associated with a namespace to avoid conflicts.  I think this is based on superstition,  for,  as we’ve seen,  conflicting extension methods do not preclude the use of two namespaces;  this advice is certainly not followed in the System.Linq namespace,  which defines a number of valuable extension methods in the System.Linq.Enumerable static class.

Conclusion

We’re still learning how to use extension methods effectively.  Although extension methods have great promise,  they’re difference from ordinary instance methods in a number of ways.  Some of these,  like the difference in null handling,  are minor,  and could potentially be put to advantage.  Others,  such as the interaction with namespaces in large projects,   are more challenging.  It’s time to start building on our experiences to develop effective patterns for using extension methods.

The problem

I got started with relational databases with mysql,  so I’m in the habit of making database changes with SQL scripts,  rather than using a GUI.  Microsoft SQL Server requires that we specify the name of a unique constraint when we want to drop it.  If you’re thinking ahead,  you can specify a name when you create the constraint;  if you don’t,  SQL Server will make up an unpredictable name,  so you can’t write a simple script to drop the constraint.

A Solution

In the spirit of “How To Drop A Primary Key in SQL Server“,  here’s a stored procedure that queries the data dictionary to find the names of any unique constraint on a specific table and column and drop them:

CREATE PROCEDURE [dbo].[DropUniqueConstraint]
    @tableName NVarchar(255),
    @columnName NVarchar(255)
AS
    DECLARE @IdxNames CURSOR
   
    SET @IdxNames = CURSOR FOR
        select sysindexes.name from sysindexkeys,syscolumns,sysindexes
            WHERE
                syscolumns.[id] = OBJECT_ID(N'[dbo].['+@tableName+N']')
                AND sysindexkeys.[id] = OBJECT_ID(N'[dbo].['+@tableName+N']')
                AND sysindexes.[id] = OBJECT_ID(N'[dbo].['+@tableName+N']')
                AND syscolumns.name=@columnName
                AND sysindexkeys.colid=syscolumns.colid
                AND sysindexes.[indid]=sysindexkeys.[indid]
                AND (
                    SELECT COUNT(*) FROM sysindexkeys AS si2
                    WHERE si2.id=sysindexes.id
                    AND si2.indid=sysindexes.indid
                )=1

    OPEN @IdxNames
    DECLARE @IdxName Nvarchar(255)
    FETCH NEXT FROM @IdxNames INTO @IdxName    

    WHILE @@FETCH_STATUS = 0 BEGIN
        DECLARE @dropSql Nvarchar(4000)

        SET @dropSql=
            N'ALTER TABLE  ['+@tableName+ N']
                DROP CONSTRAINT ['+@IdxName+ N']'
        EXEC(@dropSql)
           
        FETCH NEXT FROM @IdxNames
        INTO @IdxName
    END
CLOSE @IdxNames
DEALLOCATE @IdxNames

Usage is straightforward:

EXEC [dbo].[DropUniqueConstraint]
    @tableName='TargetTable',
    @columnName='TargetColumn'

This script has a limitation:  it only drops unique constraints that act on a single column,  not constraints that act on multiple columns.   It is smart enough,  however,  to not drop multiple-column constraints in case one of them involves @columnName.

Feedback from SQL stored procedure wizards would be mostly certainly welcome.

The problem

Silverlight 2 Beta 2 has changed the concurrency model for asynchronous communications.  In Silverlight 2 Beta 1,  asynchronous requests always returned on the UI Thread.  This was convenient,  since updates to the user interface can only be done via the UI thread.  As of Silverlight 2 Beta 2,  asynchronous callbacks are fired in worker threads that come from a thread pool:  although this potentially allows for better performance via concurrency,  it increases the risk for race conditions between callbacks –  more importantly,  some mechanism is necessary to make code that updates the user interface run in the UI thread.

A solution

It’s straightforward to execute a function in the UI thread by using the Dispatcher property of any ScriptObject The tricky part is that ScriptObjects are part of the user interface,  so you can only access the Dispatcher property from the UI thread.  At first this seems like a chicken-and-egg situation:  you need a Dispatcher to get to the UI thread,  but you need to be in the UI thread to get a Dispatcher.

This dilemma can be resolved by accessing a Dispatcher in your App.xaml.cs file and stashing it away in a static variable on application startup:

private void Application_Startup(object sender, StartupEventArgs e) {
    ...
    UIThread.Dispatcher = RootVisual.Dispatcher;
}

UIThread is a simple static class:

public static class UIThread {
    public static Dispatcher Dispatcher {get; set;}
    public static void Run(Action a) {
        Dispatcher.BeginInvoke(a);
    }
}

At some point in the future,  you can transfer execution to the UIThread by scheduling a function to run in it.

public class ProcessHttpResponse(...) {
   ...
   UIThread.Run(UpdateUserInterface);
}

The parameter of Run is an Action delegate,  which is a function that returns void and takes no parameters.  That’s fine and good,  but what if you want to pass some parameters to the function that updates the UI.  The usual three choices for passing state in asynchronous applications apply,  but it’s particularly easy and fun to use a closure here:

public class ProcessHttpResponse(...) {
    ...
    var elementToUpdate=...;
    var updateWithValue=...;

    UIThread.Run(delegate() {
        UpdateUserInterface(elementToUpdate,updateWithValue)
    });
}

When to return?

If your application is complex,  and you have nested asynchronous calls,  you’re left with an interesting question:  where is the best place to switch execution to the UI thread?  You can switch to the UI Thread as soon as you get back from an HttpRequest or a WCF call and you must switch to the UI Thread before you access any methods or properties of the user interface.  What’s best?

It is simple and safe to switch to the UI Thread immediately after requests return from the server.  If you’re consistent in doing this,  you’ll never have trouble accessing the UI thread,  and you’ll never have trouble with race conditions between returning communication responses.  On the other hand,  you’ll lose the benefit of processing responses concurrently,  which can improve speed and responsiveness on today’s multi-core computers.

It’s straightforward to exploit concurrency when a requests can be processed independently.  For instance,  imagine a VRML viewer written in Silverlight.  Displaying a VRML would require the parsing of a file,  the construction of the scene graph and the initialization of a 3-d engine,  which may require building data structures such as a BSP Tree.  Doing all of this work in the UI Thread would make the application lock up while a model is loading — it would be straightforward,  instead,  to construct all of the data structures required by the 3-d engine,  and attach the fully initialized 3-d engine to the user interface as the last step.  Since the data structures would be independent of the rest of the application,  thread safety and liveness is a nonissue.

Matters can get more complicated,  however,  if the processing of a request requires access to application-wide data;  response handlers running in multiple threads will probably corrupt shared data structures unless careful attention is paid to thread safety.  One simple approach is to always access shared data from the UI Thread,  and to transfer control to the UI Thread with UIThread.Run before accessing shared variables.

Conclusion

Silverlight 2 Beta 2 introduces a major change in the concurrency model for asynchronous communication requests.  Unlike SL2B1,  where asynchronous requests executed on the user interface thread,  SL2B2 launches asynchronous callbacks on multiple threads.  Although this model offers better performance and responsiveness,  it requires Silverlight programmers to explicitly transfer execution to the UI thread before accessing UI objects:  most SL2B1 applications will need to be reworked.

This article introduces a simple static class,  UIThread,  which makes it easy to schedule execution in the UI Thread.

Introduction

This is a story of two types: GenericType and SpecificType, where GenericType is a superclass of SpecificType. There are two types of explicit cast in C#:

The Prefix cast:

[01] GenericType g=...;
[02] SpecificType t=(SpecificType) g;

The as cast:

[03] GenericType g=...;
[04] SpecificType t=g as SpecificType;

Most programmers have a habit of using one or the other — this isn’t usually a conscious decision, but more of a function of which form a programmer saw first. I, for instance, programmed in Java before I learned C#, so I was already in the prefix cast habit. People with a Visual Basic background often do the opposite. There are real differences between the two casting operators which can make a difference in the reliability and performance of your application.

Prefix casting: Reliable Casting

The major difference between prefix- and as-casting is what happens when the cast fails. Imagine, for instance, that g is really an instance of AnotherSpecificType, which is not a subclass of SpecificType. In this case, the prefix-cast throws an exception at line [2] — however, the as-cast returns null when the cast fails, letting the execution of the program barrel on.

It’s easier to implement correct error handling in programs that use prefix-casting, and programs that use prefix-casting are easier to debug. Prefix-casting causes an exception to be thrown at the moment of the cast, where the problem is obvious. As-casting, on the other hand, can cause an exception to be thrown at the moment where the SpecificType t is referenced. The used of the SpecificType can be far away in the program: it can be in another method, or another class — it could even happen hours after the cast is performed. Be it in development or production, bugs caused by corrupted data structures can be maddeningly difficult to find and correct.

As-casting: Fast Casting

If it’s harder to write correct programs with as-casting, why would anybody use it? For one thing, as-casting is faster than prefix casting by quite a lot. Benchmarks show that as-casting is about five times faster than prefix casting. That said, the performance of casting is only going to matter in the innermost of loops of most programs. The fastest kind of casting is no casting at all: it’s best to use generics to eliminate the need for casting when possible and to move casting outside loops. (Generics also improve the reliability of your code because they help C#’s static type checking catch mistakes.)

There are some cases where as-casting could be convenient, for instance, when you expect the cast to fail sometimes. Often I like to ‘tag’ classes with interfaces that specify certain behaviors. For example,

[05] public Interface IBoppable {
[06]     void Bop();
[07] }

Now i might want to loop through a bunch of Thing objects, and bop all the ones that implement IBoppable: it’s reasonable to use as-casting here:

[08] List<Thing> list=...
[09] foreach(Thing thing in list) {
[10]    List boppable=thing as IBoppable;
[11]    if (boppable !=null) {
[12]        boppable.Bop()
[13]    }
[14] }

It’s OK to use as-casting if you’re going to check to see if the value is null immediately after the cast. The above code is correct, but has the bad smell that the boppable variable continues to exist in the block after the cast… It’s still there for a later maintainer to use erroneously. In cases like this, code can be made clearer with the is operator:

[15] List<Thing> list=...
[16] foreach(Thing thing in list) {
[17]    if(thing is IBoppable) {
[18]        ((IBoppable) boppable).Bop()
[19]    }
[20] }

(Speed freaks can use as-cast on line 18, as we know it’s not going to fail.)

The pattern of testing for null after an as-cast has a few problems. (i) It doesn’t distinguish between the case of the original object being null from the case of the original object being the wrong type and (ii) correct error handling often requires more contorted logic than using an exception — and once you added test logic, you’ve lost the speed advantage of as-casting.

Conclusion

C# offers two casting operators: the prefix-cast and the as-cast. Although the two operators compile to different op-codes in the CLR, the practical difference between them is in how they handle failed casts. Prefix-cast throws an exception on cast failure, while as-cast returns null. It’s easier to implement correct error handling when you use prefix cast, because it doesn’t require manual checks for null values that can cause problems in distant parts of your program. Prefix-cast should be the default cast operator on your fingertips, that you use for everyday situations — reserve as-cast for special cases where performance matters. (For best performance, however, eliminate the cast entirely.)

One of my pet peeves with Microsoft’s SQL Server is that it requires you to know the name of constraints and indexes that you’re dropping. This isn’t a problem when you’re working with the management studio, because it looks at the database metadata and generates code like

ALTER TABLE [dbo].[someTable]  DROP CONSTRAINT [PK__someTabl__3214EC07271AA44F]

It’s less convenient, however, when you’re writing a set of migration scripts in SQL to implement changes that you make over the database in time. Specifically, if you create the table twice in two different databases, the hexadecimal string in the name of the key will be different — the ALTER TABLE statement will fail when you try to drop the index later, since the name of the key won’t match.

Here’s a stored procedure that looks up the name of the primary key in the system catalog and uses dynamic SQL to drop the index:

CREATE PROCEDURE [dbo].[DropPrimaryKey]
    @tableName Varchar(255)
AS
    /*
       Drop the primary key on @TableName

http://gen5.info/q/

       Version 1.1
       June 9, 2008
    */

    DECLARE @pkName Varchar(255)

    SET @pkName= (
        SELECT [name] FROM sysobjects
            WHERE [xtype] = 'PK'
            AND [parent_obj] = OBJECT_ID(N'[dbo].['+@tableName+N']')
    )
    DECLARE @dropSql varchar(4000)

    SET @dropSql=
        'ALTER TABLE [dbo].['+@tableName+']
            DROP CONSTRAINT ['+@PkName+']'
    EXEC(@dropSql)
GO

Once you've loaded this stored procedure,  you can write

EXEC [dbo].[DropPrimaryKey] @TableName='someTable'

It’s just that simple. Similar stored procedures can be written to convert fields from NOT NULL to NULL and do other operation which required a named constraint.

When you’re writing applications that use asynchronous callbacks (i.e. Silverlight, AJAX, or GWT) you’ll eventually run into the problem of keeping track of the context that a request is being done in. This isn’t a problem in synchronous programming, because local variables continue to exist after one function calls another function synchronously:

int AddToCount(int amount,string countId)  {
   int countValue=GetCount(countId);
   return countValue+amount;
}

This doesn’t work if the GetCount function is asynchronous, where we need to write something like

int AddToCountBegin(int amount,string countId,CountCallback outerCallback) {
     GetCountBegin(countId,AddToCountCallback);
}

void AddToCountCallback(int countValue) {
    ... some code to get the values of amount and outerCallback ...
    outerCallback(countValue+amount);
}

Several things change in this example: (i) the AddToCount function gets broken up into two functions: one that does the work before the GetCount invocation, and one that does the work after GetCount completes. (ii) We can’t return a meaningful value from AddToCountCallback, so it needs to ‘return’ a value via a specified callback function. (iii) Finally, the values of outerCallback and amount aren’t automatically shared between the functions, so we need to make sure that they are carried over somehow.
There are three ways of passing context from a function that calls and asynchronous function to the callback function:

1. As an argument to the callback function
2. As an instance variable of the class of which the callback function is a class
3. Via a closure

Let’s talk about these alternatives:

1. Argument to the Callback Function

In this case, a context object is passed to the asynchronous function, which passes the context object to the callback. The advantage here is that there aren’t any constraints on how the callback function is implemented, other than by accepting the context object as a callback. In particular, the callback function can be static. A major disadvantage is that the asynchronous function has to support this: it has to accept a state object which it later passes to the callback function.

The implementation of HttpWebRequest.BeginGetResponse(AsyncCallback a,Object state) in the Silverlight libraries is a nice example. If you wish to pass a context object to the AsyncCallback, you can pass it in the second parameter, state. Your callback function will implement the AsyncCallback delegate, and will get something that implements IAsyncResult as a parameter. The state that you passed into BeginGetResponse will come back in the IAsyncResult.AsyncState property. For example:

class MyHttpContext {
	public HttpWebRequest Request;
        public SomeObject FirstContextParameter;
        public AnotherObject AnotherContextParameter;
}

protected void myHttpCallback(IAsyncResult abstractResult) {
	MyHttpContext context = (MyHttpContext) abstractResult.AsyncState;
	HttpWebResponse Response=(HttpWebResponse) context.Request.EndGetResponse(abstractResult);
}

public doHttpRequest(...) {
	...
        MyHttpContext context=new MyHttpContext();
	context.Request=Request;
	context.FirstContextParameter = ... some value ...;
	context.AnotherContextParameter = .. another value ...;
	Request.BeginGetResponse();
	Request.Callback(myHttpCallback,context);
}

Note that, in this API, the Request object needs to be available in myHttpCallback because myHttpCallbacks get the response by calling the HttpWebResponse.EndGetResponse() method. We could simply pass the Request object in the state parameter, but we’re passing an object we defined, myHttpCallback, because we’d like to carry additional state into myHttpCallback.

Note that the corresponding method for doing XMLHttpRequests in GWT, the use of a RequestBuilder object doesn’t allow using method (1) to pass context information — there is no state parameter. in GWT you need to use method (2) or (3) to pass context at the RequestBuilder or GWT RPC level. You’re free, of course, to use method (1) when you’re chaining asynchronous callbacks: however, method (2) is more natural in Java where, instead of a delegate, you need to pass an object reference to designate a callback function.

2. Instance Variable Of The Callback Function’s Class

Functions (or Methods) are always attached to a class in C# and Java: thus, the state of a callback function can be kept in either static or instance variables of the associated class. I don’t advise using static variables for this, because it’s possible for more than one asynchronous request to be flight at a time: if two request store state in the same variables, you’ll introduce race conditions that will cause a world of pain. (see how race conditions arise in asynchronous communications.)

Method 2 is particularly effective when both the calling and the callback functions are methods of the same class. Using objects whose lifecycle is linked to a single asynchronous request is an effective way to avoid conflicts between requests (see the asynchronous command pattern and asynchronous functions.)

Here’s an example, lifted from the asynchronous functions article:

    public class HttpGet : IAsyncFunction<String>
    {
        private Uri Path;
        private CallbackFunction<String> OuterCallback;
        private HttpWebRequest Request;

        public HttpGet(Uri path)
        {
            Path = path;
        }

        public void Execute(CallbackFunction<String> outerCallback)
        {
            OuterCallback = outerCallback;
            try
            {
                Request = (HttpWebRequest)WebRequest.Create(Path);
                Request.Method = "GET";
                Request.BeginGetRequestStream(InnerCallback,null);
            }
            catch (Exception ex)
            {
                OuterCallback(CallbackReturnValue<String>.CreateError(ex));
            }
        }

        public void InnerCallback(IAsyncResult result)
        {
            try
            {
                HttpWebResponse response = (HttpWebResponse) Request.EndGetResponse(result);
                TextReader reader = new StreamReader(response.GetResponseStream());
                OuterCallback(CallbackReturnValue<String>.CreateOk(reader.ReadToEnd()));
            } catch(Exception ex) {
                OuterCallback(CallbackReturnValue<String>.CreateError(ex));
            }
        }
    }

Note that two pieces of context are being passed into the callback function: an HttpWebRequest object named Request (necessary to get the response) and a CallbackFunction<String> delegate named OuterCallback that receives the return value of the asynchronous function.

Unlike Method 1, Method 2 makes it possible to keep an unlimited number of context variables that are unique to a particular case in a manner that is both typesafe and oblivious to the function being called — you don’t need to cast an Object to something more specific, and you don’t need to create a new class to hold multiple variables that you’d like to pass into the callback function.

Method 2 comes into it’s own when it’s used together with polymorphism, inheritance and initialization patterns such as the factory pattern: if the work done by the requesting and callback methods can be divided into smaller methods, a hierarchy of asynchronous functions or commands can reuse code efficiently.

3. Closures

In both C# and Java, it’s possible for a method defined inside a method to have access to variables in the enclosing method. In C# this is a matter of creating an anonymous delegate, while in Java it’s necessary to create an anonymous class.

Using closures results in the shortest code, if not the most understandable code. In some cases, execution proceeds in a straight downward line through the code — much like a synchronous version of the code. However, people sometimes get confused the indentation, and, more seriously, parameters after the closure definition and code that runs immediately after the request is fired end up in an awkward place (after the definition of the callback function.)

    public class HttpGet : IAsyncFunction<String>
    {
        private Uri Path;

        public HttpGet(Uri path)
        {
            Path = path;
        }

        public void Execute(CallbackFunction<String> outerCallback)
        {
            OuterCallback = outerCallback;
            try
            {
                HttpWebRequest request = (HttpWebRequest)WebRequest.Create(Path);
                Request.Method = "GET";
                Request.BeginGetRequestStream(delegate(IAsyncResult result) {
	            try {
                        response = request.EndGetResponse(result);
                        TextReader reader = new StreamReader(response.GetResponseStream());
                        outerCallback(CallbackReturnValue<String>.CreateOk(reader.ReadToEnd()));
                    } catch(Exception ex) {
                        outerCallback(CallbackReturnValue<String>.CreateError(ex));
                    }

            },null); // <--- note parameter value after delegate definition

            }
            catch (Exception ex)
            {
                outerCallback(CallbackReturnValue<String>.CreateError(ex));
            }
        }
    }

The details are different in C# and Java: anonymous classes in Java can access local, static and instance variables from the enclosing context that are declared final — this makes it impossible for variables to be stomped on while an asynchronous request is in flight. C# closures, on the other hand, can access only local variables: most of the time this prevents asynchronous requests from interfering with one another, unless a single method fires multiple asynchronous requests, in which case counter-intuitive things can happen.

Conclusion

In addition to receiving return value(s), callback functions need to know something about the context they run in: to write reliable applications, you need to be conscious of where this information is; better yet, a strategy for where you’re going to put it. Closures, created with anonymous delegates (C#) or classes (Java) produce the shortest code, but not necessarily the clearest. Passing context in an argument to the callback function requires the cooperation of the called function, but it makes few demands on the calling and callback functions: the calling and callback functions can both be static. When a single object contains both calling and callback functions, context can be shared in a straightforward and typesafe manner; and when the calling and callback functions can be broken into smaller functions, opportunities for efficient code reuse abound.

For years people have talked about an “impedance mismatch” between relational databases and object-oriented languages. Big answers have been proposed to this problem, such as object-relational mapping, but we’ve yet to see a big answer that’s entirely satisfactory — this article is one of a series that tries to break the problem into a set of smaller problems that have simple answers.

Many modern languages, such as Java, Perl, PHP and C#, are derived from C. In C, string literals are written in double quotes, and a set of double quotes can’t span a line break. One advantage of this is that the compiler can give clearer error messages when you forget the close a string. Some C-like languages, such as Java and C# are like C in that double quotes can’t span multiple lines. Other languages, such as Perl and PHP, allow it. C# adds a new multiline quote operator, @”" which allows quotes to span multiple lines and removes the special treatment that C-like languages give to the backslash character (\).

The at-quote is particularly useful on the Windows platform because the backslash used as a path separator interacts badly with the behavior of quotes in C, C++ and many other languages. You need to write

String path="C:\\Program Files\\My Program\\Some File.txt";

With at-quote, you can write

String path=@"C:\Program Files\My Program\Some File.txt";

which makes the choice of path separator in MS-DOS seem like less of a bad decision.

Now, one really great thing about SQL is that the database can optimize complicated queries that have many joins and subselects — however, it’s not unusual to see people write something like

command.CommandText = "SELECT firstName,lastName,(SELECT COUNT(*) FROM comments WHERE postedBy
=userId AND flag_spam='n') AS commentCount,userName,city,state,gender,birthdate FROM user,user
Location,userDemographics WHERE user.userId=userLocation.userId AND user.userId=userDemographi
cs.user_id AND (SELECT COUNT(*) FROM friendsReferred WHERE referringUser=userId)>10 ORDER BY c
ommentCount HAVING commentCount>3";

(line-wrapped C# example) Complex queries like this can be excellent, because they can run thousands of times faster than loops written in procedural or OO languages which can fire off hundreds of queries, but a query written like the one above is a bit unfair to the person who has to maintain it.

With multiline quotes, you can continue the indentation of your code into your SQL:

command.CommandText = @"
   SELECT
       firstName
      ,lastName
      ,(SELECT COUNT(*) FROM comments
         WHERE postedBy=userId AND flag_spam='n') AS commentCount
      ,userName
      ,city
      ,state
      ,gender
      ,birthdate
   FROM user,userLocation,userDemographics
   WHERE
      user.userId=userLocation.userId
      AND user.userId=userDemographics.user_id
      AND (SELECT COUNT(*) FROM friendsReferred
         WHERE referringUser=userId)>10
   ORDER BY commentCount
   HAVING commentCount>3
";

Although this might not be quite quite as cool as LINQ, it works with Mysql, Microsoft Access or any database you need to connect to — and it works in many other languages, such as PHP (in which you could use ordinary double quotes.) In languages like Java that don’t support multiline quotes, you can always write

String query =
   " SELECT"
  +"    firstName"
  +"   ,lastName"
  +"   ,(SELECT COUNT(*) FROM comments"
  +"      WHERE postedBy=userId AND flag_spam='n') AS commentCount"
  +"   ,userName"
  +"   ,city"
  +"   ,state"
  +"   ,gender"
  +"   ,birthdate"
  +" FROM user,userLocation,userDemographics"
  +"    WHERE"
  +"       user.userId=userLocation.userId"
  +"       AND user.userId=userDemographics.user_id"
  +"       AND (SELECT COUNT(*) FROM friendsReferred"
  +"          WHERE referringUser=userId)>10"
  +" ORDER BY commentCount"
  +" HAVING commentCount>3";

but that’s a bit more cumbersome and error prone.

CORRECTION:  The threading model in Silverlight has changed as of Silverlight 2 Beta 2.  It is now possible to initiate asynchronous communication from any thread,  however,  asynchronous callbacks now run in “new” threads that come from a thread pool.  The issues in this article still apply,  with two additions:  (1) the possibility of race conditions and deadlocks between asynchronous callback threads and (2) all updates to user interface components must be done from the user interface thread.  (Fortunately,  it’s easy to get back to the UI thread.)  Subscribe to our RSS Feed to keep informed of breaking developments in Silverlight development.

There’s a lot of confusion about how asynchronous communication works in RIA’s such as Silverlight, GWT and Javascript. When I start talking about the problems of concurrency control, many people tell me that there aren’t any concurrency problems since everything runs in a single thread. [1]

It’s important to understand the basics of what is going on when you’re writing asynchronous code, so I’ve put together a simple example to show how execution works in RIA’s and how race conditions are possible. This example applies to Javascript, Silverlight, GWT and Flex, as well as a number of other environments based on Javascript. This example doesn’t represent best practices, but rather what can happen when you’re not using a proactive strategy that eliminates concurrency problems:

In the diagram above, execution starts when the user pushes a button (a). This starts the user interface thread by invoking an onClick handler. The user interface thread starts two XmlHttpRequests, (b) and (c). The event handler eventually returns, so execution stops in the user interface thread.

In the meantime, the browser still has two XmlHttpRequests running. Callbacks from http requests, timers and user interfaces go into a queue — they get executed right away if the user interface thread is doing nothing, but get delayed if the user interface thread is active.

Http request (b) completes first, causing the http callback for request (b) to start. Had something been a little different with the web browser, web server or network, request (c) could have returned first, causing the callback for request (c) to start. If the result of the program depends on the order that the callbacks for (b) and (c) run, we have a race condition. The callback for http request (b) starts a new http request (d), which runs for a long time.

In the meantime, the user is moving the mouse and triggers a mouseover event while the request (b) callback is running. Right after the request (b) callback completes, the web browser starts the UI thread, which causes a mouseover event handler (e) to run. Note that the user can trigger user interface events while XmlHttpRequests are running, causing event handlers to run in an unpredictable order: if this causes your program to malfunction, your program has a bug.

While the event handler (e) is running, request (c) completes: like the mouseover event, this event is queued and runs once event handler (e) completes. Before (e) completes, it starts a new http request (f). The browser looks into the event queue when (e) completes, and starts the callback for (c). Http request (f) completes while callback (c) is running, gets queued, and runs after (c) is running.

At the end of this example, the callback for (f) completes, causing the UI thread to stop. The http request (c) is still in flight — it completes in the future, somewhere off the end of the page.

This example did not include any timers, or any mechanism of deferred execution such as DeferredCommand in GWT or Dispatcher.Invoke() in Silverlight. This is but another mechanism to add callback references to the event queue.

As you can see, there’s a lot of room for mischief: http requests can return in an arbitrary order and users can initiate events at arbitrary times. The order that things happen in can depend on the browser, it’s settings, on the behavior of the server, and everything in between. Some users might use the application in a way that avoids certain problems (they’ll think it’s wonderful) and others might consistently or occasionally trigger an event that causes catastrophe. These kind of bugs can be highly difficult to reproduce and repair.

Asynchronous RIAs have problems with race conditions that are similar to threaded applications, but not exactly the same. Today’s languages and platforms have excellent and well documented mechanisms for dealing with threads, but today’s RIAs do not have mature mechanisms for dealing with concurrency. Over time we’ll see libraries and frameworks that help, but asynchronous safety isn’t something that can be applied like deodorant: it involves non local interactions between distant parts of the program. The simplest applications can dodge the bullet, but applications beyond a certain level of complexity require an understanding of asynchronous execution and the consistent use of patterns that avoid trouble.

[1] Although it is possible to create new threads in Silverlight, all communication and user interface access must be done from the user interface thread — many Silverlight applications are single-threaded, and adding multiple threads complicates the issue.

Mat Byrne recently posted source code for a dynamic domain object in PHP which takes advantage of the dynamic nature of PHP.  It’s a good example of how programmers can take advantage of the unique characteristics of a programming language.

Statically typed languages such as C# and Java have some advantages:  they run faster and IDE’s can understand the code enough to save typing (with your fingers),  help you refactor your code,  and help you fix errors.  Although there’s a lot of things I like symfony,  it feels like a Java framework that’s invaded the PHP world.  Eclipse would help you deal with the endless getters and setters and domain object methods with 40-character names in Java,  Eclipse.

The limits of polymorphism are a serious weakness of today’s statically typed languages.  C# and Java apps that I work with are filled with if-then-else or case ladders when they need to initialize a dynamically chosen instance of one of a set of classes that subclass a particular base class or that implement a particular interface.  Sure,  you can make a HashMap or Dictionary that’s filled with Factory objects,  but any answer for that is cumbersome.  In PHP,  however,  you can write

$class_name="Plugin_Module_{$plugin_name}";
$instance = new $class_name($parameters);

This is one of several patterns which make it possible to implement simple but powerful frameworks in PHP.

Mat,  on the other hand,  uses the ‘magic’ __call() method to implement get and set methods dynamically.  This makes it possible to ‘implement’ getters and setters dynamically by simply populating a list of variables,  and drastically simplifies the construction and maintainance of domain objects.  A commenter suggests that he go a step further and use the __get() and __set() method to implement properties.  It’s quite possible to implement active records in PHP with a syntax like

$myTable = $db->myTable;
$row = $myTable->fetch($primaryKey);
$row->Name="A New Name";
$row->AccessCount = $row->AccessCount+1;
$row->save();

I’ve got an experimental active record class that introspects the database (no configuration file!) and implements exactly the above syntax,  but it currently doesn’t know anything about joins and relationships.  It would be a great day for PHP to have a database abstraction that is (i) mature,  (ii) feels like PHP,  and (iii) solves (or reduces) the two-artifact problem of maintaining both a database schema AND a set of configuration files that control the active record layer.

The point of this post isn’t that dynamically typed languages are better than statically typed languages,  but rather that programmers should make the most of the features of the language they use:  no PHP framework has become the ‘rails’ of PHP because no PHP framework has made the most of the dynamic natures of the PHP language.

Oliver Steele writes an excellent blog about coding style,  and has written some good articles on asynchronous communications with a focus on Javascript.

Minimizing Code Paths In Asynchronous Code,  a recent post of his,  is about a lesson that I learned the hard way with GWT that applies to all RIA systems that use asynchronous calls.  His example is the same case I encountered,  where a function might return a value from a cache or might query the server to get the value:   an obvious way to do this in psuedocode is:

function getData(...arguments...,callback) {
   if (... data in cache...) {
      callback(...cached data...);
   }
  cacheCallback=anonymousFunction(...return value...) {
     ... store value in cache...
     callback(...cached data...);
  }

   getDataFromServer(...arguments...,cacheCallback)

}

At first glance this code looks innocuous,  but there’s a major difference between what happens in the cached and uncached case.  In the cached case,  the callback() function gets called before getData() returns — in the uncached case,  the opposite happens.  What happens in this function has a global impact on the execution of the program,  opening up two code paths that complicate concurrency control and introduce bugs that can be frustrating to debug.

This function can be made more reliable if it schedules callback() to run after the thread it is running in completes.  In Javascript,  this can be done with setTimeout().   In Silverlight use System.Windows.Threading.Dispatcher.  to schedule the callback to run in the UI thread.