Automatic Deadlock retry Aspect with Spring and JPA/Hibernate

I’m currently working on a project that is converted from being a Mainframe application, to a Java web/batch application. We don’t ‘big bang’ into production, so the Mainframe and the Java code will work next to each other for a fairly amount of time. Since we have multiple batch processes and many simultaneous users, we start seeing deadlock errors in certain parts of the application. Some specific parts have to take a pessimistic lock, this is where it goes wrong.
Since a deadlock is an error that can be solved by repeating the action, we decide to build in a retry mechanism to restart the transaction if it got rolled back.
I started of with creating an Annotation. This annotation will mark the entry point that we want to retry in case of a deadlock.

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface DeadLockRetry {
    /**
     * Retry count. default value 3
     */
    int retryCount() default 3;
}

The retry count is a value you can supply together with your annotation, so you can specify the number of times we want to retry our operation.
Using AOP we can pick up this annotation an let us surround the method call with a retry mechanism.

@Around(value = "@annotation(deadLockRetry)", argNames = "deadLockRetry")

So lets view the aspect, we start with adding an @Aspect annotation on top of our class, this way it is configured to be an Aspect.
We also want to implement the Ordered interface. This interface lets us order our aspect. We need this to surround our Transactional aspect. If we don’t surround our Transaction, we will never be able to retry in a new transaction, we would be working in the same (marked as rollback only) transaction.
The rest of the code is pretty straight forward. We create a loop where we loop until we have more retries than we should have. Inside that loop we proceed our ProceedingJoinPoint and catch the PersistenceException that JPA would throw when a deadlock would occur. Inside the catch block we check if the error code is a deadlock error code.
Off course we could not directly configure the database specific error codes inside our aspect, so I’ve created an interface.

/**
 * Interface that marks a dialect aware of certain error codes. When you have to
 * do a low level check of the exception you are trying to handle, you can
 * implement this in this interface, so you can encapsulate the specific error
 * codes for the specific dialects.
 * 
 * @author Jelle Victoor
 * @version 05-jul-2011
 */
public interface ErrorCodeAware {
    Set<Integer> getDeadlockErrorCodes();
}

We already have custom hibernate dialects for our database and database to be, so this let me configure the error codes in the Dialect implementations. It was a bit tricky to get the current dialect. I injected the persistence unit, since we are outside a transaction, and made some casts to get my dialect. The alternative was to use a custom implementation of the ErrorCodeAware interface, not using the dialects. We could inject the needed ErrorCodeAware implementation based on our application context. This added another database specific injection, which added another point of configuration. This is why I chose to store it in our custom dialect.

private Dialect getDialect() {
        final SessionFactory sessionFactory = ((HibernateEntityManagerFactory) emf).getSessionFactory();
        return ((SessionFactoryImplementor) sessionFactory).getDialect();
    }

The only thing left is to configure the aspect, mind the order of the transaction manager and the retry aspect

	<tx:annotation-driven order="100" transaction-manager="transactionManager" />
	<bean id="deadLockRetryAspect" class="DeadLockRetryAspect">
		<property name="order" value="99" />
	</bean>

Now when I have a deadlock exception, and I’ve added this annotation, the transaction will rollback and will be reexecuted.

/**
 * This Aspect will cause methods to retry if there is a notion of a deadlock.
 * 
 * <emf>Note that the aspect implements the Ordered interface so we can set the
 * precedence of the aspect higher than the transaction advice (we want a fresh
 * transaction each time we retry).</emf>
 * 
 * @author Jelle Victoor
 * @version 04-jul-2011 handles deadlocks
 */
@Aspect
public class DeadLockRetryAspect implements Ordered {
    private static final Logger LOGGER = LoggerFactory.getLogger(DeadLockRetryAspect.class);
    private int order = -1;
    @PersistenceUnit
    private EntityManagerFactory emf;

    /**
     * Deadlock retry. The aspect applies to every service method with the
     * annotation {@link DeadLockRetry}
     * 
     * @param pjp
     *            the joinpoint
     * @param deadLockRetry
     *            the concurrency retry
     * @return
     * 
     * @throws Throwable
     *             the throwable
     */
    @Around(value = "@annotation(deadLockRetry)", argNames = "deadLockRetry")
    public Object concurrencyRetry(final ProceedingJoinPoint pjp, final DeadLockRetry deadLockRetry) throws Throwable {
        final Integer retryCount = deadLockRetry.retryCount();
        Integer deadlockCounter = 0;
        Object result = null;
        while (deadlockCounter < retryCount) {
            try {
                result = pjp.proceed();
                break;
            } catch (final PersistenceException exception) {
                deadlockCounter = handleException(exception, deadlockCounter, retryCount);
            }
        }
        return result;
    }

    /**
     * handles the persistence exception. Performs checks to see if the
     * exception is a deadlock and check the retry count.
     * 
     * @param exception
     *            the persistence exception that could be a deadlock
     * @param deadlockCounter
     *            the counter of occured deadlocks
     * @param retryCount
     *            the max retry count
     * @return the deadlockCounter that is incremented
     */
    private Integer handleException(final PersistenceException exception, Integer deadlockCounter, final Integer retryCount) {
        if (isDeadlock(exception)) {
            deadlockCounter++;
            LOGGER.error("Deadlocked ", exception.getMessage());
            if (deadlockCounter == (retryCount - 1)) {
                throw exception;
            }
        } else {
            throw exception;
        }
        return deadlockCounter;
    }

    /**
     * check if the exception is a deadlock error.
     * 
     * @param exception
     *            the persitence error
     * @return is a deadlock error
     */
    private Boolean isDeadlock(final PersistenceException exception) {
        Boolean isDeadlock = Boolean.FALSE;
        final Dialect dialect = getDialect();
        if (dialect instanceof ErrorCodeAware && exception.getCause() instanceof GenericJDBCException) {
            if (((ErrorCodeAware) dialect).getDeadlockErrorCodes().contains(getSQLErrorCode(exception))) {
                isDeadlock = Boolean.TRUE;
            }
        }
        return isDeadlock;
    }

    /**
     * Returns the currently used dialect
     * 
     * @return the dialect
     */
    private Dialect getDialect() {
        final SessionFactory sessionFactory = ((HibernateEntityManagerFactory) emf).getSessionFactory();
        return ((SessionFactoryImplementor) sessionFactory).getDialect();
    }

    /**
     * extracts the low level sql error code from the
     * {@link PersistenceException}
     * 
     * @param exception
     *            the persistence exception
     * @return the low level sql error code
     */
    private int getSQLErrorCode(final PersistenceException exception) {
        return ((GenericJDBCException) exception.getCause()).getSQLException().getErrorCode();
    }

    /** {@inheritDoc} */
    public int getOrder() {
        return order;
    }

    /**
     * Sets the order.
     * 
     * @param order
     *            the order to set
     */
    public void setOrder(final int order) {
        this.order = order;
    }
}

5 thoughts on “Automatic Deadlock retry Aspect with Spring and JPA/Hibernate”

  1. You seem to be doing it wrong. Deadlocks are down to errors in your application, and this is just papering over the problems. There are several things you can do to prevent deadlocks. The first is to make sure that all your read queries are inside functions annotated to be read-only transactions: @Transactional(readOnly = true). Secondly make sure your write transactions cover the smallest amount of work possible. Finally, make sure you’re not using a transactional isolation level higher than you need.

  2. This is a mechanism to support 2 applications using the same database. We have no control over the Mainframe application, only over the Java application. This is why we need this aspect, it is only used where needed. The pessimistic lock that we have to take on one of the tables is needed for Id generation. When the mainframe takes a pessimistic lock and the java side takes a optimistic lock, we get duplicate Id errors.
    Your remarks are correct, but not applicable in the context of our application.

  3. I have a highly concurrent transactional application where I have run into the same deadlocks. I will disagree strongly that deadlocks are application errors. They _can_ be caused by poorly written code, but they can also appear in perfectly nice, best-practices code. So, yes, you will need a retry. The problem is that it will only work cleanly if you have a luxury to clean Hibernate session. otherwise, your are doomed to random deadlock-related problems.

    For example, consider simple code:

    MyClass myObject=session.load(MyClass.class, myId)
    myObject.setX(myObject.getX()+10)

    If you are going to retry this code, the variable myObject.x may not refresh, and you’ll end up incrementing the value again

    1. This will only work if you place your retry around your whole transaction. In this case you would have to get a new session, otherwise you will get the result you describe in your comment.
      Mind the order I have given with my aspect, this takes care of the new transaction problem.

  4. Jelle

    You are an absolute legend. I’ve spent two days messing with our code to make the transations smaller, change propogation approaches and isolation approaches. I refactored POJOs to not cascade to make them smaller, but this ended up being the only solution.

    Our system had these issues when using a load testing tool that ensured transactions were posting at exactly the same time at 100 per second, so I don’t think the other comments here are valid. I think under serious load the deadlocks will occur?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>