A Pattern Heavy PHP Circuit Breaker Pattern

Thanks to the article Why your PHP App NEEDS a Circuit Breaker I recently learnt about the Circuit Breaker pattern and it also encouraged me to purchase the book "Release It!" which first described the pattern. The Circuit Breaker is a stability pattern that helps to control and regulate access to resources outside of your application. It does this in a similar way to a real Circuit Breaker which will cut the flow of electricity if the circuit is ever overloaded preventing your appliances from bursting into flames. In the Circuit Breaker pattern, if a failure is detected when trying to access another resource the 'switch' is opened preventing all further connections to that resource. The benefits of this are twofold. Firstly it prevents a failed, or failing resource from being further overloaded while it attempts to recover. Secondly, once a failure is detected your system will fail quickly so your users don't have to sit through a potentially long time-out period.

I think the concept of the pattern is great, however I felt that the implementation described in that article left something to be desired. The first issue I have is it has been hard coded to use a MySQL database to look after the state of each circuit. This leaves your database exposed and unprotected by the pattern. The second issue; I feel the use case is too complicated, which you can see here

$cb = new CircuitBreaker('myapp');
$cb->logging = true;

if($cb->isClosed()) {
  if(!$res = @file_get_contents("http://litfuel.net/tutorials/testserver.php")) {
     $cb->fail();
  } else {
     $cb->success();
  }
  var_dump($res);
} else {
  echo 'CIRCUIT TRIPPED! NO CALLS MADE';
  return false;
}

While this code works perfectly, it does require an understanding of what the pattern does and how it works. The bigger problem seems to be that my brain is broken. For some reason, the 'open' state being the failure state feels wrong to me. I know exactly how a circuit breaker works, and the states are based on this, but while studying this pattern I continually got the 'open' and 'closed' states mixed up. I know that if I was to use this code I would get the states the wrong way round and wonder why my system was no longer working.

This got me thinking about a better way to use a circuit breaker and I came up with the following use case.

function getFile($url)
{
  if (!$res = @file_get_contents($url))
      throw new Exception('Cannot connect to server');
  } else {
      return $res;
  }
}

try {
  $cb = new CircuitBreaker("myapp");
  $res = $cb->execute("getFile", "http://litfuel.net/tutorials/testserver.php");
  var_dump($res);
} catch (Exception $e) {
  // Could not fetch the file. Handle the error
}

As you can see this does not require any knowledge of whether the circuit is open or closed, or even what that means. The circuit breaker will protect the resource throwing an exception on error, that exception being either a real resource failure exception, or an open circuit exception.

Now that we have a use case, how do we go about implementing the nitty gritty of a Circuit Breaker? What follows is code I came up with for a Zend Framework application I have been playing with. If you decide to use it outside of Zend Framework you will need to remove the references to Zend specific libraries such as Zend_Exception. The code snippets I have included in this article are pretty incomplete. I have only included enough to get the idea across. A full implementation can be downloaded from here.

The Circuit Breaker pattern works by having three states: Open, Closed, and Half-Open. If the circuit is closed then everything is working as normal, resources are passed through it without a problem. If the resource fails too many times the circuit moves to an Open state. At that point the circuit will always throw an exception preventing any attempts to access the resource. After a certain amount of time has passed the circuit moves to a Half-Open state and tentatively allows one attempt to get the resource. If that attempt succeeds it moves back to a Closed state, otherwise it flips back to an Open state. This behavior makes it a perfect candidate for the State Pattern.

The first step in creating this is to define the Circuit State's abstract class. Each state needs to be able to respond to three different points during the execution of the resource code.

1. The point just before execution, where a state can move from Open to Half-Open.
2. An Exception, where a failure count is incremented and the state can move to the Open state.
3. Resource Called, where the state moves to a Closed state and the failure count is reset.

This gives us a very simple abstract class that looks like this

abstract class App_CircuitBreaker_State_Abstract
{
protected $_circuitbreaker;

public function __construct(App_CircuitBreaker $circuitbreaker)
{
 $this->_circuitbreaker = $circuitbreaker;
}

public function onCalling()
{}

public function onCalled()
{
 $this->_circuitbreaker->resetFailures();
}

public function onException(Exception $e)
{
 $this->_circuitbreaker->incrementFailure();
}
}

Each state class is then responsible for changing the circuit breaker's state depending on how successful the code call was. The Closed state is only concerned with exceptions which is when the state is moved to Open.

public function onException()
{
parent::onException();

if ($this->_circuitbreaker->thresholdReached()) {
 $this->_circuitbreaker->setState('open');
}
}

The Open state changes the circuit breaker state to Half just before the code is called.

public function onCalling()
{
parent::onCalling();

// If the timeout has passed set the circuit to half open. Else throw exception.
if (time() > $this->_circuitbreaker->getTimeout() + $this->_circuitbreaker->getTrippedtime()) {
 $this->_circuitbreaker->setState('half');
} else {
 throw new App_Exception_CircuitBreaker('Circuit is open');
}
}

The Half state switches back to Open or Closed depending the success or failure of the code call

public function onException()
{
parent::onException();

$this->_circuitbreaker->setState('open');
}

public function onCalled()
{
parent::onCalled();

$this->_circuitbreaker->setState('closed');
}

Now for the CircuitBreaker class. The execute function is where all the interesting stuff happens and the state pattern keeps it pretty simple.

class App_CircuitBreaker
{
// Snip for brevity

public function execute()
{
 // Alert the state that we are about to call the code
 $this->_state->onCalling();

 try {
      // Get the function and the arguments.
      $args = func_get_args();
      $function = array_shift($args);
      $returnval = call_user_func_array($function, $args);
 } catch (Exception $e) {
     // Alert the state that we caught an exception then rethrow
     $this->_state->onException();

     throw $e;
 }

 // Alert the state that the code was called successfully
 $this->_state->onCalled();

 return $returnval;
}
}

So what we have now is a simple state based circuit breaker pattern,which is pretty useless as there is nothing persistent about it. We need a storage mechanism. This part is pretty basic. To do this a storage adapter is required. This is just another object that implements the following interface

interface App_CircuitBreaker_Store_Interface
{
public function load();
public function save();

public function getState();
public function getFailures();
public function getTimeout();
public function getTrippedtime();
public function getThreshold();

public function setState($state);
public function setFailures($failures);
public function setTimeout($timeout);
public function setTrippedtime($time);
public function setThreshold($threshold);
}

The circuit breaker constructor should then take the storage object and the application id to load all the information back into the circuit breaker. Then whenever any of this information is changed it should be saved back into the store, like so.

public function __construct($store)
{
$this->_store = $store;

// Load the circuit breaker
$this->_store->load();
$this->setState($this->_store->getState());
$this->_failures = $this->_store->getFailures();
$this->_timeout = $this->_store->getTimeout();
$this->_trippedtime = $this->_store->getTrippedtime();
$this->_threshold = $this->_store->getThreshold();
}

public function save()
{
$this->_store->setState($this->_state->getName());
$this->_store->setFailures($this->_failures);
$this->_store->setTimeout($this->_timeout);
$this->_store->setTrippedtime($this->_trippedtime);
$this->_store->setThreshold($this->_threshold);

$this->_store->save();
}

This adapter pattern will then allow you to use whatever you like to save the circuit breaker's state. MySQL, sqlite, or memcache.

There you have it. The important parts of what I hope is a neat and flexible implementation of the Circuit Breaker pattern. To make the use case described at the beginning of this article even better you should probably move the $store and $cb objects into a singleton pattern class reducing the actual execute call it to something like the following

My_CB::getInstance('myapp')->execute("getFile", "http://litfuel.net/tutorials/testserver.php");

The other thing missing from this code is a logger. All state changes should be logged somewhere to enable some system debugging when a failure occurs. The logger should also be implemented using an adapter.

As mentioned earlier a full implementation can be downloaded from here. Please have a look at it as it contains a lot of additional boring code not touched on in this article. Please feel free to leave questions or comments about how this article could be improved, or how the code can be made better.