Thursday, March 11, 2010

Retryable actions in C#


Download the Code

2012-10-18 18:00 Edit: I was just informed via Twitter that the source code for this article is no longer available.  I had the source on my SkyDrive.  I will attempt to find it in the next couple of days.  I will post an update with a new link to the source code.  I will most likely post the code to GitHub.

2012-10-18 22:30 Edit: I have uploaded a variation of my retry source code to GitHub. See the download link above.

In inevitably there will be times during the execution of your application where you will encounter an exception. Most common unavoidable types of exceptions are caused when accessing an external resource such as a database or web service. The resource may be known to have occasional expected. A SQL Server Cluster fail-over may only take 5-10 seconds, however, during the fail-over, your application will get a number of different variations of SQLException, varying from connection refused, database not available, etc.

Your database access code may start of with something like the following,

   1:  try
   2:  {
   3:      ReadDatabase();
   4:  }
   5:  catch (SqlException)
   6:  {
   7:      // return error to user?
   8:  }

Your first thought may be to catch the known exception type and try the action again, such as,

   1:  try
   2:  {
   3:      ReadDatabase();
   4:  }
   5:  catch (SqlException)
   6:  {
   7:      try
   8:      {
   9:          ReadDatabase();
  10:      }
  11:      catch (SqlException)
  12:      {
  13:          // failed again, return error to user?
  14:      }
  15:  }

If there is only one place in your code you have to do this, then this approach may be acceptable. However, more likely there will be a fair number of places you need to catch an exception and try again. In each of these places, the conditions and behavior will not be the same. You will want to handle retrying of certain blocks of code in a consistent manner. Enter Retryable....

Introducing Retryable


Retryable is a class that allows a consistent and extensible way to automatically retry code that is commonly subject to exceptions. Before showing some code, there are are few things you should keep in mind before blindly wrapping your error prone code with retry functionality.

The first is side effects. Will the retry cause unexpected delays or program hangs? Retry waits should be kept to an minimum. Second, with automatic retrying code there are two design decisions to consider
  • In what case do you want to retry the action.
  • How much time should elapse between attempts

The Retryable class has a number of overloads of the Execute method. The simplest is
method

public static void Execute(Action action);

This method can be called in any of the following ways,

// using anonymous method
Retryable.Execute(delegate() { ReadDatabase(); });
// using lambda
Retryable.Execute(() => ReadDatabase());
// using method group...
Retryable.Execute(ReadDatabase);

But wait! What about those design decisions? How many times and in what conditions does the it retry? How long does wait between attempts? Before answer this, I must first introduce couple of other components.

IRethrowEvaluator

IRethrowEvaluator is an interface that is called when an exception is thrown by the Action target. It has a single method named Rethrow,

   1:  public interface IRethrowEvaluator
   2:  {
   3:      bool Rethrow(int retryCount, Exception exception);
   4:  }

If the Rethrow method returns true, the exception will be rethrown, otherwise the Action target will be retried. Implementations can examine the retryCount and exception parameters to determine if it is an appropriate reason to retry. If your code expects database connection issues or deadlock to manifest as a SQLException, you should only retry if the SQLException represents a connection problem or deadlock. If the SQLException represents an invalid query or permission problem, it does not make sense to retry. You need to be clear on what conditions you are trying to recover from.

ISleepEvaluator

ISleepEvaluator is an interface that is called if the IRethrowEvaluator return false. The ISleepEvaluator interface is very similar to IRethrowEvaluator,

   1:  public interface ISleepEvaluator
   2:  {
   3:      int SleepMilliseconds(int retryCount, Exception exception);
   4:  }

It should be pretty clear what this method is supposed to do. It gets the same parameters as IRethrowEvaluator. It returns the number of milliseconds to sleep before the next call to action is attempted. You can adjust the amount of time to wait based on how many items the action has been retried and what caused it to fail the last time.

Putting it all together

The examples above did not pass instances of either of these interfaces. There a number of overloads to the Execute method,

public static void Execute(Action action, IRethrowEvaluator rethrowEvaluator, ISleepEvaluator sleepEvaluator)

If IRethrowEvaluator is not specified, the default implementation will retry 5 times, i.e., retryCount == 4, on any type of Exception.

If ISleepEvaluator is not specified, the default implementation will use a random exponential delay with minimum of 5ms. I discovered this technique on JD Conley's blog post Functional Optimistic Concurrency in C#. The implementation is pretty elegant,

   1:  private static int SleepMilliseconds(int retryCount, int minimumDelay)
   2:  {
   3:      Random random = new Random(Guid.NewGuid().GetHashCode());
   4:      return random.Next((int)Math.Pow(retryCount, 2) + minimumDelay,
               (int)Math.Pow(retryCount + 1, 2) + 1 + minimumDelay);
   5:  }

You can implement IRethrowEvaluator and ISleepEvaluator as simple or as complex as you require. However, most of the time, you may want very simple conditions and waits. To assist in these common situations, you can use the helper static classes, ThrowIf and SleepFor,

   1:  Retryable.Execute(
   2:      ReadDatabase,
   3:      ThrowIf.RetryCountIs(9),
   4:      SleepFor.Milliseconds(250));
   5:   
   6:  Retryable.Execute(
   7:      ReadDatabase,
   8:      ThrowIf.RetryCountIs(9),
   9:      SleepFor.RandomExponential(250));
  10:   
  11:  Retryable.Execute(
  12:      ReadDatabase,
  13:      ThrowIf.Custom((retryCount, exception) => retryCount == 9 && exception.GetType() == typeof(SqlException)),
  14:      SleepFor.Custom((retryCount, exception) => (retryCount + 1) * 100));

The Retryable class has a number of overloads taking Action, Action<T>, Action<T1, T2>, etc and Func<TResult>, Func<T, TResult>, giving the user many options. Note that not all possible combinations have been implemented. I have only implemented it overloads that I required.

I have found this set of classes very useful in situations where occasional expected exceptions are expected but can be recovered from retrying the action. Consumers of these classes have ability to control what should happen when an exception is raised and evaluate if the action should be retried and now long to wait between attempts.

Your thoughts?

15 comments:

said...

your sample code link is broken

Phil Bolduc said...

Sorry about that, I have updated the link. Please let me know if you have any further issues.

mehul said...

i got this piece working with a hello world.. mail me on trivedimehulk@gmail.com if somebody wants...

Anonymous said...

Nice. I like this approach a lot. I have done a little tidying up and added some extension methods so now I can just call Action.Retry() or Func.Retry(). Very slick.

Unknown said...

When trying to compile the code, it fails on all "Guard." objects with the following error:

The name 'Guard' does not exist in the current context

Am I missing a reference?

Anir said...

Mike - You need to reference the Microsoft.Practices.Unity namespace
You can read more about it here:
http://msdn.microsoft.com/en-us/practices/default.aspx

Kiquenet said...

any sample application, full source code ? thx

Andrew Allen said...

Phil:

What's the license/permission to use on this code? Don't want to assume public domain-ness...

@BertCraven: could you post your code (and license/permission to use)?

Andrew

Bernhard Richter said...

This is a great idea. I specially like how you use static methods to provide implementations for the required dependencies.

When it comes to the Execute methods and their signatures, I think you can simplify this to a pretty large extend.

You mentioned in the article that you have only implemented the Execute overloads you need.


The fact is that you really just need two overloads.

One that takes an Action delegate and one that takes a Func delegate. It does not matter how many parameters the target method have, the arguments will always be known up front.

static void Main(string[] args)
{
Retryable.Execute(() => MethodWithTwoArtguments("A","B"));
}

private static string MethodWithTwoArtguments(string value1, string value2)
{
return value1 + value2;
}

This effectively means that a lot of Execute overloads can be removed from the current implementation and things can be kept even simpler :)

Phil Bolduc said...

@Unknown feel free to use this code for any purpose. If I need to specify a license, I would probably go with a BSD style one.

Phil Bolduc said...

@Bernhard Richter thanks for the feedback.

Anonymous said...

Thanks for that, works a treat.

I found it useful to extend it so that it can take a list of rethrow-evaluators, so that the rethrow-conditions are composable rather than complex conditional logic.

Jerry said...

The sample code link no longer exist?

Unknown said...

Download Link is not working

Anonymous said...

Thank you this was a lifesaver!