Twitter Joins Me

I’ve watched Twitter from a distance for the past year or so,  sometimes making fun of it in blog comments,  but I never actually joined.

Last week I was looking at my web server log with the good old tail -f,  and found that several other bloggers had hotlinked the copy of the twitter fail whale that was in my old “What do you do if you catch an exception?” post.  It turns out that my copy of the whale currently ranks #1 in Google Image Search.  It’s not bringing in a vast amount of traffic,  but it seems to be really engaging people,  because Google blog search is finding a new reference to the image just about every day.

I’ve been spending a lot of time developing sites where that’s the whole idea:  to build virtuous circles where people find images,  put them on their sites,  link back to my site,  which attracts more visitors.  Sometimes you can succeed at this without trying,  but making a business out of it is a matter of being lucky consistently.

After that I broke down and joined twitter.  My username on twitter is paul_houle;  Right now it seems like a strange and lonely place,  but I can see some discipline in expressing oneself in 140 characters.  I’ve noticed quite a few characters already:  everything from the very corporate people who tweet in calculated sound bites to people that tweet like /dev/random.  Perhaps I can’t do anything about the “strange” bit,  but perhaps I can about the “lonely” part.  If you like the things that I blog about,  you’re certainly invited to follow me,  and I’m interested in following like minded people.

Manipulate HTML Forms With Silverlight 2

Remote Control


Lately I’ve been working on a web application based on Silverlight 2.  The application uses a traditional web login system based on a cryptographically signed cookie.  In early development,  users logged in on an HTML page,  which would load a Silverlight application on successful login.  Users who didn’t have Silverlight installed would be asked to install it after logging in,  rather than before.

Although it’s (sometimes) possible to determine what plug-ins a user has installed using Javascript,  the methods are dependent on the specific browser and the plug-ins.  We went for a simple and effective method:  make the login form a Silverlight application,  so that users would be prompted to install Silverlight before logging in.

Our solutionn  was to make the Silverlight application a drop-in replacement for the original HTML form.  The Silverlight application controls a hidden HTML form:  when a user hits the “Log In” buttonin the Silverlight application,  the application inserts the appropriate information into the HTML form and submits it.  This article describes the technique in detail. Continue Reading »

require(), require_once() and Dynamic Autoloading in PHP


I program in PHP a lot,  but I’ve avoided using autoloaders,  except when I’ve been working in frameworks,  such as symfony,   that include an autoloader.  Last month I started working on a system that’s designed to be part of a software product line:  many scripts,  for instance,  are going to need to deserialize objects that didn’t exist when the script was written:  autoloading went from a convenience to a necessity.

The majority of autoloaders use a fixed mapping between class names and PHP file names.  Although that’s fine if you obey a strict “one class,  one file” policy,  that’s a policy that I don’t follow 100% of the time.  An additional problem is that today’s PHP applications often reuse code from multiple frameworks and libraries that use different naming conventions:  often applications end up registering multiple autoloaders.  I was looking for an autoloader that “just works” with a minimum of convention and configuration — and I found that in a recent autoloader developed by A.J. Brown. Continue Reading »

Nested Sets, PHP, Verb Objects and Noun Objects


Controversy persists to this day about the relative merits of dynamic languages such as PHP and Python versus static languages such as C# and Java.  We’re finding more and more that the difference isn’t so much about static or dynamic typing,  but more about the cultures of different languages.   In this article,  I discuss an efficient representations of SQL trees in a database,  an algorithm for creating that representation,  and a PHP implementation.  The PHP implementations uses objects in a way foreign to many developers:  rather than using objects to represent nouns (data),  it uses a class to represent a verb (an algorithm.)  I make the case that programmers shouldn’t feel compelled to create new classes to represent every data item:  that verb objects often provide the right level of abstraction for many tasks.

The Presenting Problem

Lately I’ve been collecting pictures of animals, and decided that incorporating the taxonomic database from ITIS would be a big help. I’m interested in asking questions like “What are all the species underneath the Tapiridae family?” The ITIS database uses the adjacency list representation, where each row contains a column that references the primary key of a parent row. Algorithms for the adjacency list are well known, but are awkward to implement in SQL since it takes multiple SQL statements to traverse a tree. Continue Reading »

What do you do when you’ve caught an exception?

Abort, Retry, Ignore

This article is a follow up to “Don’t Catch Exceptions“, which advocates that exceptions should (in general) be passed up to a “unit of work”, that is, a fairly coarse-grained activity which can reasonably be failed, retried or ignored. A unit of work could be:

  • an entire program, for a command-line script,
  • a single web request in a web application,
  • the delivery of an e-mail message
  • the handling of a single input record in a batch loading application,
  • rendering a single frame in a media player or a video game, or
  • an event handler in a GUI program

The code around the unit of work may look something like

[01] try {
[02]   DoUnitOfWork()
[03] } catch(Exception e) {
[04]    ... examine exception and decide what to do ...
[05] }

For the most part, the code inside DoUnitOfWork() and the functions it calls tries to throw exceptions upward rather than catch them.

To handle errors correctly, you need to answer a few questions, such as

  • Was this error caused by a corrupted application state?
  • Did this error cause the application state to be corrupted?
  • Was this error caused by invalid input?
  • What do we tell the user, the developers and the system administrator?
  • Could this operation succeed if it was retried?
  • Is there something else we could do?

Although it’s good to depend on existing exception hierarchies (at least you won’t introduce new problems), the way that exceptions are defined and thrown inside the work unit should help the code on line [04] make a decision about what to do — such practices are the subject of a future article, which subscribers to our RSS feed will be the first to read.

Continue Reading »

Converting A Synchronous Program Into An Asynchronous Program


One of the challenges in writing programs in today’s RIA environments  (Javascript, Flex, Silverlight and GWT)  is expressing the flow of control between multiple asynchronous XHR calls.  A “one-click-one-XHR” policy is often best,  but you don’t always have control over your client-server protocols.  A program that’s simple to read as a synchronous program can become a tangle of subroutines when it’s broken up into a number of callback functions.  One answer is program translation:  to manually or automatically convert a synchronous program into an asynchronous program:  starting from the theoretical foundation,  this article talks about a few ways of doing that.

Thibaud Lopez Schneider sent me a link to an interesting paper he wrote, titled “Writing Effective Asynchronous XmlHttpRequests.” He presents an informal proof that you can take a program that uses synchronous function calls and common control structures such as if-else and do-while, and transform it a program that calls the functions asynchronously. In simple language, it gives a blueprint for implementing arbitrary control flow in code that uses asynchronous XmlHttpRequests.

In this article,  I work a simple example from Thibaud’s paper and talk about four software tools that automated the conversion of conventional control flows to asynchronous programming.  One tool,  the Windows Workflow Foundation, lets us compose long-running applications out of a collection of asynchronous Activity objects.  Another two tools are jwacs and Narrative Javascript,  open-source   translators that translated pseudo-blocking programs in a modified dialect of JavaScript into an asynchronous program in ordinary JavaScript that runs in your browser.

Continue Reading »

Stop Catching Exceptions!


It’s clear that a lot of programmers are uncomfortable with exceptions [1] [2]; in the feedback of an article I wrote about casting, it seemed that many programmers saw the throwing of a NullReferenceException at a cast to be an incredible catastrophe.

In this article, I’ll share a philosophy that I hope will help programmers overcome the widespread fear of exceptions. It’s motivated by five goals:

  1. Do no harm
  2. To write as little error handling code as possible,
  3. To think about error handling as little as possible
  4. To handle errors correctly when possible,
  5. Otherwise errors should be handled sanely

To do that, I

  1. Use finally to stabilize program state when exceptions are thrown
  2. Catch and handle exceptions locally when the effects of the error are local and completely understood
  3. Wrap independent units of work in try-catch blocks to handle errors that have global impact

This isn’t the last word on error handling, but it avoids many of the pitfalls that people fall into with exceptions. By building upon this strategy, I believe it’s possible to develop an effective error handling strategy for most applications: future articles will build on this topic, so keep posted by subscribing to the Generation 5 RSS Feed.

Continue Reading »

The Multiton Design Pattern


Many people have independely discovered a new design pattern, the “Multiton”, which, like the “Singleton” is an initialization pattern in the style of the Design Patterns book. Like the Singleton, the Multiton provides a method that controls the construction of a class: instead of maintaining a single copy of an object in an address space, the Multiton maintains a Dictionary that maps keys to unique objects.

The Multiton pattern can be used in systems that store persistent data in a back-end store, such as a relational databases. The Multiton pattern can be used to maintain a set of objects are mapped to objects (rows) in a persistent store: it applies obviously to object-relational mapping systems, and is also useful in asynchronous RIA’s, which need to keep track of user interface elements that are interested in information from the server.

An alternate use case of Mulitons, seen in the “Multicore” version of the PureMVC framework, is the extension of the Singleton pattern to support multiple instances of a system in a single address space.

As useful as the Multiton pattern is, this article explains how Multitons use references in a way that doesn’t work well with conventional garbage collection. Multitons are a great choice when the number of Multitons is small, but they may leak memory unacceptablely when more than a few thousand are created. Future posts will describe patterns, such as the Captive Multiton, that provide the same capabilities with more scalable memory management — subscribe to our RSS feed to keep informed.

Continue Reading »

The Role Of The Model in Silverlight, GWT and Javascript


When people start developing RIA’s in environments such as Silverlight, GWT, Flex and plain JavaScript, they often write asynchronous communication callbacks in an unstructured manner, putting them wherever is convenient — often in an instance member of a user interface component (Silverlight and GWT) or in a closure or global function (JavaScript.)

Several problems almost invariably occur as applications become more complex that force the development of an architecture that decouples communication event handlers from the user interface: a straightforward solution is to create a model layer that’s responsible for notifying interested user interface components about data updates.

This article uses a simple example application to show how a first-generation approach to data updates breaks down and how introducing a model-view split makes for a reliable and maintainable application.

(This is one of a series of articles on RIA architecture: subscribe to the Gen5 RSS feed for future installments.)

Example Application: Blogging And The Category Dropdown

Imagine a blogging application that works like the WordPress blog used on this site. This application consists of a number of forms, one of which is used to write a new post:

This form lets you fill out two text fields: a title and the body of the post. It also contains a dropdown list of categories, and gives you the option of adding a new category. Categories are represented (server-side) in a table in a relational database that looks like:

[01] CREATE TABLE categoryList (
[02]     id                 integer primary key auto_increment,
[03]     name               varchar(255)
[04] ); Continue Reading »

The Semantics of Dictionaries, Maps and Hashtables


The first language I used that put dictionaries on my fingertips was Perl, where the solution to just about any problem involved writing something like


Perl called a dictionary a ‘hash’,  a reference to the way Perl implemented dictionaries.  (Dictionaries are commonly implemented with hashtables and b-trees,  but can also be implemented with linked-list and other structures.)  The syntax of Perl is a bit odd, as you’d need to use $, # or % to reference scalar,  array or hash variables in different contexts,  but dictionaries with similar semantics became widespread in dynamic languages of that and succeeding generations, such as Python, PHP and Ruby.  ‘Map’ container classes were introduced in Java about a decade ago,  and programmers are using dictionaries increasingly in static languages such as Java and C#.

Dictionaries are a convenient and efficient data structure, but there’s are areas in which different mplementations behave differently: for instance,  in what happens if you try to access an undefined key.   I think that cross-training is good for developers,  so this article compares this aspect of the semantics of dictionaries in four popular languages:  PHP,  Python,  Java and C#.

Use cases

There are two use cases for dictionaries, so far as error handling is concerned:

  1. When you expect to look up undefined values, and
  2. When you don’t

Let’s look at three examples:

Computing A Histogram

One common use for a dictionary is for counting items, or recording that items in a list or stream have been seen. In C#, this is typically written something like:

[01] var count=Dictionary<int,int>();
[02] foreach(int i in inputList) {
[03]   if (!counts.Contains(i))
[04]       count[i]=0;
[06]   count[i]=count[i]+1
[07] }

The Dictionary count now contains the frequency of items inputList, which could be useful for plotting a histogram. A similar pattern can be used if we wish to make a list of unique items found in inputList. In either case,  looking up values that aren’t already in the hash is a fundamental part of the algorithm.

Processing Input

Sometimes, we’re getting input from another subsystem, and expect that some values might not be defined. For instance, suppose a web site has a search feature with a number of optional features, and that queries are made by GET requests like:

[08] search.php?q=kestrel
[09] search.php?q=admiral&page=5
[10] search.php?q=laurie+anderson&page=3&in_category=music&after_date=1985-02-07

In this case, the only required search parameter is “q”, the query string — the rest are optional. In PHP (like many other environments), you can get at GET variables via a hashtable, specifically, the $_GET superglobal, so (depending on how strict the error handling settings in your runtime are) you might write something like

[11] if ($_GET["q"])) {
[12]     throw new InvalidInputException("You must specify a query");
[13] }
[15] if($_GET["after_date"]) {
[16]  ... add another WHERE clause to a SQL query ...
[17] }

This depends, quite precisely, on two bits of sloppiness in PHP and Perl: (a) Dereferencing an undefined key on a hash returns an undefined value, which is something like a null. (b) both languages have a liberal definition of true and false in an if() statement. As a result, the code above is a bit quirky. The if() at line 11 evaluates false if q is undefined, or if q is the empty string. That’s good. However, both the numeric value 0 and the string “0″ also evaluate false. As a result, this code won’t allow a user to search for “0″, and will ignore an (invalid) after_date of 0, rather than entering the block at line [16], which hopefully would validate the date.

Java and C# developers might enjoy a moment of schadenfreude at the above example, but they’ve all seen, written and debugged examples of input handling code that just as quirky as the above PHP code — with several times the line count. To set the record straight, PHP programmers can use the isset() function to precisely test for the existence of a hash key:

[11] if (isset($_GET["q"]))) {
[12]     throw new InvalidInputException("You must specify a query");
[13] }

The unusual handling of “0″ is the kind of fault that can survive for years in production software:  so long as nobody searches for “0″,  it’s quite harmless.  (See what you get if you search for a negative integer on Google.)  The worst threat that this kind of permissive evaluation poses is when it opens the door to a security attack,  but we’ve also seen that highly complex logic that strives to be “correct” in every situation can hide vulnerabilities too.

Relatively Rigid Usage

Let’s consider a third case: passing a bundle of context in an asynchronous communications call in a Silverlight application written in C#. You can do a lot worse than to use the signatures:

[14] void BeginAsyncCall(InputType input,Dictionary<string, object> context,CallbackDelegate callback);
[15] void CallbackDelegate(ReturnType returnValue,Dictionary<string,object> context);

The point here is that the callback might need to know something about the context in which the asynchronous function was called to do it’s work. However, this information may be idiosyncratic to the particular context in which the async function is called,  and is certainly not the business of the asynchronous function. You might write something like

[16] void Initiator() {
[17]   InputType input=...;
[18]   var context=Dictionary<string,object>();
[19]   context["ContextItemOne"]= (TypeA) ...;
[20]   context["ContextItemTwo"]= (TypeB) ...;
[21]   context["ContextItemThre"] = (TypeC) ...;
[22]   BeginAsyncCall(input,context,TheCallback);
[23] }
[25] void TheCallback(ReturnType output,Dictionary<string,object> context) {
[26]   ContextItemOne = (TypeA) context["ContextItemOne"];
[27]   ContextItemTwo = (TypeB) context["ContextItemTwo"];
[28]   ContextItemThree = (TypeC) context["ContextItemThree"];
[29]   ...
[30] }

This is nice, isn’t it?  You can pass any data values you want between Initiator and TheCallback. Sure,  the compiler isn’t checking the types of your arguments,  but loose coupling is called for in some situations.  Unfortunately it’s a little too loose in this case,  because we spelled the name of a key incorrectly on line 21.

What happens?

The [] operator on a dot-net Dictionary throws a KeyNotFoundException when we try to look up a key that doesn’t exist.   I’ve set a global exception handler for my Silverlight application which,  in debugging mode,  displays the stack trace.  The error gets quickly diagnosed and fixed.

Four ways to deal with a missing value

There are four tools that hashtables give programmers to access values associated with keys and detect missing values:

  1. Test if key exists
  2. Throw exception if key doesn’t exist
  3. Return default value (or null) if key doesn’t exist
  4. TryGetValue

#1: Test if key exists

PHP:    isset($hashtable[$key])
Python: key in hashtable
C#:     hashtable.Contains(key)
Java:   hashtable.containsKey(key)

This operator can be used together with the #2 or #3 operator to safely access a hashtable.  Line [03]-[04] illustrates a common usage pattern.

One strong advantage of the explicit test is that it’s more clear to developers who spend time working in different language environments — you don’t need to remember or look in the manual to know if the language you’re working in today uses the #2 operator or the #3 operator.

Code that depends on the existence test can be more verbose than alternatives,  and can  be structurally unstable:  future edits can accidentally change the error handling properties of the code.  In multithreaded environments,  there’s a potential risk that an item can be added or removed between the existance check and an access — however,  the default collections in most environment are not thread-safe,  so you’re likely to have worse problems if a collection is being accessed concurrently.

#2 Throw exception if key doesn’t exist

Python: hashtable[key]
C#:     hashtable[key]

This is a good choice when the non-existence of a key is really an exceptional event.  In that case,  the error condition is immediately propagated via the exception handling mechanism of the language,  which,  if properly used,  is almost certainly better than anything you’ll develop.  It’s awkward,  and probably inefficient,  if you think that non-existent keys will happen frequently.  Consider the following rewrite of the code between [01]-[07]

[31] var count=Dictionary<int,int>();
[32] foreach(int i in inputList) {
[33]   int oldCount;
[34]   try {
[35]       oldCount=count[i];
[36]   } catch (KeyNotFoundException ex) {
[37]       oldCount=0
[38]   }
[40]   count[i]=oldCount+1
[41] }

It may be a matter of taste,  but I think that’s just awful.

#3 Return a default (often null) value if key doesn’t exist

PHP:    $hashtable[key] (well,  almost)
Python: hashtable.get(key, [default value])
Java:   hashtable.get(key)

This can be a convenient and compact operation.  Python’s form is particularly attractive because it lets us pick a specific default value.  If we use an extension method to add a Python-style GetValue operation in C#,  the code from [01]-[07] is simplified to

[42] var count=Dictionary<int,int>();
[43] foreach(int i in inputList)
[44]   count[i]=count.GetValue(i,0)+1;

It’s reasonable for the default default value to be null (or rather,  the default value of the type),  as it is in Python,  in which case we could use the ??-operator to write

[42] var count=Dictionary<int,int>();
[43] foreach(int i in inputList)
[44]   count[i]=(count.GetValue(i) ?? 0)+1;

(A ?? B equals A if A is not null,  otherwise it equals B.)   The price for this simplicity is two kinds of sloppiness:

  1. We can’t tell the difference between a null (or default) value associated with a key and no value associated with a key
  2. The potential of null value exports chaos into the environment:  trying to use a null value can cause a NullReferenceException if we don’t explictly handle the null.  NullReferenceExceptions don’t bother me if they happen locally to the function that returns them,  but they can be a bear to understand when a null gets written into an instance variable that’s accessed much later.

Often people don’t care about 1,  and the risk of 2 can be handled by specifying a non-null default value.

Note that PHP’s implementation of hashtables has a particularly annoying characteristic.  Error handling in php is influenced by the error_reporting configuration variable which can be set in the php.ini file and other places.  If the E_STRICT bit is not set in error_reporting,   PHP barrels on past places where incorrect variable names are used:

[45] $correctVariableName="some value";
[46] echo "[{$corectValiableName}]"; // s.i.c.

In that case, the script prints “[]” (treats the undefined variable as an empty string) rather than displaying an error or warning message.  PHP will give a warning message if E_STRICT is set,  but then it applies the same behavior to hashtables:  an error message is printed if you try to dereference a key that doesn’t exist — so PHP doesn’t consistently implement type #3 access.

#4 TryGetValue

There are quite a few methods (Try-* methods) in the .net framework that have a signature like this:

[47] bool Dictionary<K,V>.TryGetValue(K key,out V value);

This method has crisp and efficient semantics which could be performed in an atomic thread-safe manner:  it returns true if finds the key,  and otherwise returns false.  The output parameter value is set to the value associated with the key if a value is associated with the key,  however,  I couldn’t find a clear statement of what happens if the key isn’t found.  I did a little experiment:

[48] var d = new Dictionary<int, int>();
[49] d[1] = 5;
[50] d[2] = 7;
[51] int outValue = 99;
[52] d.TryGetValue(55, out outValue)
[53] int newValue = outValue;

I set a breakpoint on line 53 and found thate the value of outValue was 0,  which is the default value of the int type.  It seems,  therefore,  that TryGetValue returns the default value of the type when it fails to find the key.  I wouldn’t count on this behavior,  as it is undocumented.

The semantics of TryGetValue are crisp and precise.  It’s particularly nice that something like TryGetValue could be implemented as an atomic operation,  if the underyling class is threadsafe.  I fear,  however,  that TryGetValue exports chaos into it’s environment.  For instance,  I don’t like declaring a variable without an assignment,  like below:

[54] int outValue;
[55] if (d.TryGetValue(55,outValue)) {
[56] ... use outValue ...
[57] }

The variable outValue exists before the place where it’s set,  and outside of the block where it has a valid value.  It’s easy for future maintainers of the code to try to use outValue between lines [54]-[55] or after line [57].  It’s also easy to write something like 51],  where the value 99 is completely irrelevant to the program.  I like the construction

[58] if (d.Contains(key)) {
[59]    int value=d[key];
[60]    ... do something with value ...
[61] }

because the variable value only exists in the block [56]-[58] where it has a defined value.

Hacking Hashables

A comparison of hashtables in different languages isn’t just academic.  If you don’t like the operations that your language gives you for hashtables,  you’re free to implement new operations.  Let’s take two simple examples.  It’s nice to have a Python-style get() in PHP that never gives a warning message,  and it’s easy to implement

[62] function array_get($array,$key,$defaultValue=false) {
[63]   if (!isset($array[$key]))
[64]      return $defaultValue;
[66]   return $array[$key];
[67] }

Note that the third parameter of this function uses a default value of false,  so it’s possible to call it in a two-parameter form

[68] $value=array_get($array,$key);

with a default default of false,  which is reasonable in PHP.

Extension methods make it easy to add a Python-style get() to C#;  I’m going to call it GetValue() to be consistent with TryGetValue():

[69] public static class DictionaryExtensions {
[70]   public static V GetValue<K, V>(this IDictionary<K, V> dict, K key) {
[71]      return dict.GetValue(key, default(V));
[72]   }
[74]   public static V GetValue<K, V>(this IDictionary<K, V> dict, K key, V defaultValue) {
[75]      V value;
[76]      return dict.TryGetValue(key, out value) ? value : defaultValue;
[77]   }
[78] }


Today’s programming languages put powerful data structures,  such as dictionaries,  on your fingertips.  When we look closely,  we see subtle differences in the APIs used access dictionaries in different languages.  A study of the different APIs and their consequences can help us think about how to write code that is more reliable and maintainable,  and informs API design in every language

kick it on