Retryable actions in C#

Download the Code

2012-10-18 18:00 Edit: I was just informed via Twitter that the source code for this article is no longer available.  I had the source on my SkyDrive.  I will attempt to find it in the next couple of days.  I will post an update with a new link to the source code.  I will most likely post the code to GitHub.

2012-10-18 22:30 Edit: I have uploaded a variation of my retry source code to GitHub. See the download link above.

In inevitably there will be times during the execution of your application where you will encounter an exception. Most common unavoidable types of exceptions are caused when accessing an external resource such as a database or web service. The resource may be known to have occasional expected. A SQL Server Cluster fail-over may only take 5-10 seconds, however, during the fail-over, your application will get a number of different variations of SQLException, varying from connection refused, database not available, etc.

Your database access code may start of with something like the following,

   1:  try

   2:  {

   3:      ReadDatabase();

   4:  }

   5:  catch (SqlException)

   6:  {

   7:      // return error to user?

   8:  }

Your first thought may be to catch the known exception type and try the action again, such as,

   1:  try

   2:  {

   3:      ReadDatabase();

   4:  }

   5:  catch (SqlException)

   6:  {

   7:      try

   8:      {

   9:          ReadDatabase();

  10:      }

  11:      catch (SqlException)

  12:      {

  13:          // failed again, return error to user?

  14:      }

  15:  }

If there is only one place in your code you have to do this, then this approach may be acceptable. However, more likely there will be a fair number of places you need to catch an exception and try again. In each of these places, the conditions and behavior will not be the same. You will want to handle retrying of certain blocks of code in a consistent manner. Enter Retryable....

Introducing Retryable

Retryable is a class that allows a consistent and extensible way to automatically retry code that is commonly subject to exceptions. Before showing some code, there are are few things you should keep in mind before blindly wrapping your error prone code with retry functionality.

The first is side effects. Will the retry cause unexpected delays or program hangs? Retry waits should be kept to an minimum. Second, with automatic retrying code there are two design decisions to consider

• In what case do you want to retry the action.
• How much time should elapse between attempts

The Retryable class has a number of overloads of the Execute method. The simplest is
method

public static void Execute(Action action);

This method can be called in any of the following ways,

// using anonymous method

Retryable.Execute(delegate() { ReadDatabase(); });

// using lambda

Retryable.Execute(() => ReadDatabase());

// using method group...

Retryable.Execute(ReadDatabase);

But wait! What about those design decisions? How many times and in what conditions does the it retry? How long does wait between attempts? Before answer this, I must first introduce couple of other components.

IRethrowEvaluator

IRethrowEvaluator is an interface that is called when an exception is thrown by the Action target. It has a single method named Rethrow,

   1:  public interface IRethrowEvaluator

   2:  {

   3:      bool Rethrow(int retryCount, Exception exception);

   4:  }

If the Rethrow method returns true, the exception will be rethrown, otherwise the Action target will be retried. Implementations can examine the retryCount and exception parameters to determine if it is an appropriate reason to retry. If your code expects database connection issues or deadlock to manifest as a SQLException, you should only retry if the SQLException represents a connection problem or deadlock. If the SQLException represents an invalid query or permission problem, it does not make sense to retry. You need to be clear on what conditions you are trying to recover from.

ISleepEvaluator

ISleepEvaluator is an interface that is called if the IRethrowEvaluator return false. The ISleepEvaluator interface is very similar to IRethrowEvaluator,

   1:  public interface ISleepEvaluator

   2:  {

   3:      int SleepMilliseconds(int retryCount, Exception exception);

   4:  }

It should be pretty clear what this method is supposed to do. It gets the same parameters as IRethrowEvaluator. It returns the number of milliseconds to sleep before the next call to action is attempted. You can adjust the amount of time to wait based on how many items the action has been retried and what caused it to fail the last time.

Putting it all together

The examples above did not pass instances of either of these interfaces. There a number of overloads to the Execute method,

public static void Execute(Action action, IRethrowEvaluator rethrowEvaluator, ISleepEvaluator sleepEvaluator)

If IRethrowEvaluator is not specified, the default implementation will retry 5 times, i.e., retryCount == 4, on any type of Exception.

If ISleepEvaluator is not specified, the default implementation will use a random exponential delay with minimum of 5ms. I discovered this technique on JD Conley's blog post Functional Optimistic Concurrency in C#. The implementation is pretty elegant,

   1:  private static int SleepMilliseconds(int retryCount, int minimumDelay)

   2:  {

   3:      Random random = new Random(Guid.NewGuid().GetHashCode());

   4:      return random.Next((int)Math.Pow(retryCount, 2) + minimumDelay,
               (int)Math.Pow(retryCount + 1, 2) + 1 + minimumDelay);

   5:  }

You can implement IRethrowEvaluator and ISleepEvaluator as simple or as complex as you require. However, most of the time, you may want very simple conditions and waits. To assist in these common situations, you can use the helper static classes, ThrowIf and SleepFor,

   1:  Retryable.Execute(

   2:      ReadDatabase,

   3:      ThrowIf.RetryCountIs(9),

   4:      SleepFor.Milliseconds(250));

   5:   

   6:  Retryable.Execute(

   7:      ReadDatabase,

   8:      ThrowIf.RetryCountIs(9),

   9:      SleepFor.RandomExponential(250));

  10:   

  11:  Retryable.Execute(

  12:      ReadDatabase,

  13:      ThrowIf.Custom((retryCount, exception) => retryCount == 9 && exception.GetType() == typeof(SqlException)),

  14:      SleepFor.Custom((retryCount, exception) => (retryCount + 1) * 100));

The Retryable class has a number of overloads taking Action, Action<T>, Action<T1, T2>, etc and Func<TResult>, Func<T, TResult>, giving the user many options. Note that not all possible combinations have been implemented. I have only implemented it overloads that I required.

I have found this set of classes very useful in situations where occasional expected exceptions are expected but can be recovered from retrying the action. Consumers of these classes have ability to control what should happen when an exception is raised and evaluate if the action should be retried and now long to wait between attempts.

Your thoughts?