Automatic Deadlock retry Aspect with Spring and JPA/Hibernate

I’m currently working on a project that is converted from being a Mainframe application, to a Java web/batch application. We don’t ‘big bang’ into production, so the Mainframe and the Java code will work next to each other for a fairly amount of time. Since we have multiple batch processes and many simultaneous users, we start seeing deadlock errors in certain parts of the application. Some specific parts have to take a pessimistic lock, this is where it goes wrong.
Since a deadlock is an error that can be solved by repeating the action, we decide to build in a retry mechanism to restart the transaction if it got rolled back.
I started of with creating an Annotation. This annotation will mark the entry point that we want to retry in case of a deadlock.

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface DeadLockRetry {
    /**
     * Retry count. default value 3
     */
    int retryCount() default 3;
}

The retry count is a value you can supply together with your annotation, so you can specify the number of times we want to retry our operation.
Using AOP we can pick up this annotation an let us surround the method call with a retry mechanism.

@Around(value = "@annotation(deadLockRetry)", argNames = "deadLockRetry")

So lets view the aspect, we start with adding an @Aspect annotation on top of our class, this way it is configured to be an Aspect.
We also want to implement the Ordered interface. This interface lets us order our aspect. We need this to surround our Transactional aspect. If we don’t surround our Transaction, we will never be able to retry in a new transaction, we would be working in the same (marked as rollback only) transaction.
The rest of the code is pretty straight forward. We create a loop where we loop until we have more retries than we should have. Inside that loop we proceed our ProceedingJoinPoint and catch the PersistenceException that JPA would throw when a deadlock would occur. Inside the catch block we check if the error code is a deadlock error code.
Off course we could not directly configure the database specific error codes inside our aspect, so I’ve created an interface.

/**
 * Interface that marks a dialect aware of certain error codes. When you have to
 * do a low level check of the exception you are trying to handle, you can
 * implement this in this interface, so you can encapsulate the specific error
 * codes for the specific dialects.
 * 
 * @author Jelle Victoor
 * @version 05-jul-2011
 */
public interface ErrorCodeAware {
    Set<Integer> getDeadlockErrorCodes();
}

We already have custom hibernate dialects for our database and database to be, so this let me configure the error codes in the Dialect implementations. It was a bit tricky to get the current dialect. I injected the persistence unit, since we are outside a transaction, and made some casts to get my dialect. The alternative was to use a custom implementation of the ErrorCodeAware interface, not using the dialects. We could inject the needed ErrorCodeAware implementation based on our application context. This added another database specific injection, which added another point of configuration. This is why I chose to store it in our custom dialect.

private Dialect getDialect() {
        final SessionFactory sessionFactory = ((HibernateEntityManagerFactory) emf).getSessionFactory();
        return ((SessionFactoryImplementor) sessionFactory).getDialect();
    }

The only thing left is to configure the aspect, mind the order of the transaction manager and the retry aspect

	<tx:annotation-driven order="100" transaction-manager="transactionManager" />
	<bean id="deadLockRetryAspect" class="DeadLockRetryAspect">
		<property name="order" value="99" />
	</bean>

Now when I have a deadlock exception, and I’ve added this annotation, the transaction will rollback and will be reexecuted.

/**
 * This Aspect will cause methods to retry if there is a notion of a deadlock.
 * 
 * <emf>Note that the aspect implements the Ordered interface so we can set the
 * precedence of the aspect higher than the transaction advice (we want a fresh
 * transaction each time we retry).</emf>
 * 
 * @author Jelle Victoor
 * @version 04-jul-2011 handles deadlocks
 */
@Aspect
public class DeadLockRetryAspect implements Ordered {
    private static final Logger LOGGER = LoggerFactory.getLogger(DeadLockRetryAspect.class);
    private int order = -1;
    @PersistenceUnit
    private EntityManagerFactory emf;

    /**
     * Deadlock retry. The aspect applies to every service method with the
     * annotation {@link DeadLockRetry}
     * 
     * @param pjp
     *            the joinpoint
     * @param deadLockRetry
     *            the concurrency retry
     * @return
     * 
     * @throws Throwable
     *             the throwable
     */
    @Around(value = "@annotation(deadLockRetry)", argNames = "deadLockRetry")
    public Object concurrencyRetry(final ProceedingJoinPoint pjp, final DeadLockRetry deadLockRetry) throws Throwable {
        final Integer retryCount = deadLockRetry.retryCount();
        Integer deadlockCounter = 0;
        Object result = null;
        while (deadlockCounter < retryCount) {
            try {
                result = pjp.proceed();
                break;
            } catch (final PersistenceException exception) {
                deadlockCounter = handleException(exception, deadlockCounter, retryCount);
            }
        }
        return result;
    }

    /**
     * handles the persistence exception. Performs checks to see if the
     * exception is a deadlock and check the retry count.
     * 
     * @param exception
     *            the persistence exception that could be a deadlock
     * @param deadlockCounter
     *            the counter of occured deadlocks
     * @param retryCount
     *            the max retry count
     * @return the deadlockCounter that is incremented
     */
    private Integer handleException(final PersistenceException exception, Integer deadlockCounter, final Integer retryCount) {
        if (isDeadlock(exception)) {
            deadlockCounter++;
            LOGGER.error("Deadlocked ", exception.getMessage());
            if (deadlockCounter == (retryCount - 1)) {
                throw exception;
            }
        } else {
            throw exception;
        }
        return deadlockCounter;
    }

    /**
     * check if the exception is a deadlock error.
     * 
     * @param exception
     *            the persitence error
     * @return is a deadlock error
     */
    private Boolean isDeadlock(final PersistenceException exception) {
        Boolean isDeadlock = Boolean.FALSE;
        final Dialect dialect = getDialect();
        if (dialect instanceof ErrorCodeAware && exception.getCause() instanceof GenericJDBCException) {
            if (((ErrorCodeAware) dialect).getDeadlockErrorCodes().contains(getSQLErrorCode(exception))) {
                isDeadlock = Boolean.TRUE;
            }
        }
        return isDeadlock;
    }

    /**
     * Returns the currently used dialect
     * 
     * @return the dialect
     */
    private Dialect getDialect() {
        final SessionFactory sessionFactory = ((HibernateEntityManagerFactory) emf).getSessionFactory();
        return ((SessionFactoryImplementor) sessionFactory).getDialect();
    }

    /**
     * extracts the low level sql error code from the
     * {@link PersistenceException}
     * 
     * @param exception
     *            the persistence exception
     * @return the low level sql error code
     */
    private int getSQLErrorCode(final PersistenceException exception) {
        return ((GenericJDBCException) exception.getCause()).getSQLException().getErrorCode();
    }

    /** {@inheritDoc} */
    public int getOrder() {
        return order;
    }

    /**
     * Sets the order.
     * 
     * @param order
     *            the order to set
     */
    public void setOrder(final int order) {
        this.order = order;
    }
}

Extending Spring JUnit Runner

For our integration tests we had to use the development database. On that database there are some batch parameters stored and one of those parameters is a cutoffTime. As the name suggest, our business logic can not be processed after that time, so we had to create a mechanism to make sure that the tests are not executed, else the build would fail.
The first thing we did was a quick fix. We added an if-statement to the test to see what time it is and if the test can be executed. As some of you already know, an empty method annotated with @Test will execute and be successful. This is not correct, when someone breaks code that is in those tests, the test should not be marked as successful. We should be able to add the @Ignore annotation on the tests when the cutOffTime is passed.

I looked into this problem and the first thing that came to my mind was to use Annotations, because it is meta-data and not test functionality. JUnit4 has a very easy way to handle annotations, so I decided to go that way.

/**
 * The CutOffTime annotation..
 */
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
public @interface CutOffTime {
    String cutOffTime();
}

I want this annotation to cancel all my tests in a testcase, not just a method, that is why i put the ElementType on TYPE.

Our integration tests are running with the SpringJUnit4ClassRunner so I extended this. I have overridden the invokeTestMethod method, because this is the one executing each test. Here is the source code

/**
 * The Class IntegrationTestClassRunner. This class is a specific class runner
 * for the payments. There is a cutoffTime defined in the database that decides
 * if a payment that can be made or not. This can be applied by the
 * {@link CutOffTime} annotation
 *
 * @author Jelle Victoor
 * @version 14-apr-2010
 */
public class IntegrationTestClassRunner extends SpringJUnit4ClassRunner {

    /**
     * The Constructor.
     *
     * @param clazz
     *            the clazz
     *
     * @throws InitializationError
     *             the initialization error
     */
    public IntegrationTestClassRunner(final Class<?> clazz) throws InitializationError {
        super(clazz);
    }

    /** {@inheritDoc} */
    @Override
    protected void invokeTestMethod(final Method method, final RunNotifier notifier) {
        Annotation[] classAnnotations = classAnnotations();
        Boolean ignore = Boolean.FALSE;
        for (Annotation classAnnotation : classAnnotations) {
            if (classAnnotation instanceof CutOffTime) {
                if (shouldIgnore(((CutOffTime) classAnnotation).cutOffTime())) {
                    ignore = Boolean.TRUE;
                }
                break;
            }
        }
        if (!ignore) {
            super.invokeTestMethod(method, notifier);
        } else {
            notifier.fireTestIgnored(methodDescription(method));
        }
    }

    /**
     * Decides to ignore the test
     *
     * @param cuttOffTime
     *            the cutt off time
     *
     * @return true, if should ignore
     */
    private boolean shouldIgnore(final String cuttOffTime) {
        int hour = Integer.parseInt(StringUtils.substring(cuttOffTime, 0, 2));
        int minutes = Integer.parseInt(StringUtils.substring(cuttOffTime, 2, 4));
        return new LocalTime().isAfter(new LocalTime(hour, minutes));
    }
}

As you can see, the classAnnotation() method will give you all the annotations that are on your class *and are specified with @Retention(RetentionPolicy.RUNTIME)*. I simply check if the annotation is found on the class. When I’m done checking if the annotation is found and if the test should be ignored, i can fireTestIgnored on the notifier or execute the test. We just annotate our tests with @RunWith(IntegrationTestClassRunner.class).
This way the test should not fail or succeed, but now you can simply see that the test is ignored. The statistics are clean :-).

I know that the whole purpose of this extention is odd, but we are forced into this position. With this post I’m trying to show you that extending a test framework like JUnit isn’t hard at all and you can use it in every way you want to.

Java EE annotations in Spring

Since I began programming I noticed that annotations kept gaining popularity. Implementing a marker interface is in many cases replace by adding an Annotation.

Even in Spring. Spring added the support tor the java EE annotations. Here’s a little example. Instead of implementing the initializingBean Interface, you can simply use theĀ @PostConstruct van Java EE 5.

All callback handler used in EE5+ are possible in Spring. you only have to configure theĀ CommonAnnotationBeanPostProcessor in your spring context.

/** {@inheritDoc} */
public void afterPropertiesSet() throws Exception {
}

becomes

/** Postconstuction method */
@PostConstruct
public void myMethod() {
}

It is a small nuance, but I personally like this way of working. I don’t chose annotations over interfaces in every case, but when we are talking about callback handlers, I think it the better way. Your implementation doesn’t include strange spring specific methods, but methods that make sense to you and your other developers. It just has a little annotation above it.