Continuous Deployment Practice #6: Automate Rollback

You have just completed the deploy of latest version on your production environment, but something was wrong, and after a few minutes/hours the business guys ask you to revert to the previous version. You was smart, and did a backup of the latest version in production in a tar.gz. Next you manually copied the backup and deployd it to all your servers…. without making mistakes, because you had a perfect documentation for this procedure, and you remembered well all the steps to rollback… isn’t?

Antipattern: use a backup to rollback an application to a previous version.

The recipe: automate the rollback in the same way you automate the deploy

In the last post we see how to use business metrics to validate a deploy, using a mix of software instrumentation and PaaS services. But what to do when a deploy validation fails?

You have at least 2 different aspects to consider:

• Changes at “behaviour” level (your source code)
• Changes at “data” level (your database

Rollback of Source Code

At onebip.com we use a simple trick: every deploy is in different new folder, with a unique generated name (provided by our deployment pipeline tool). After a successful and validated deploy we update a symbolic link “onebip” to the working directory of the latest version of the application. To rollback the code, we just revert the symbolic link to the previous version, a matter of few seconds.

But how to know which is previous version? we create a previousVersion file with a very simple command

<echo msg="Preparing version info file in ${deploy.basedir}" />
<phingcall target="exec-pssh">
  <property name="ssh.host.remote" value="${ws.hosts.argument}" />
  <property name="ssh.command"
            value="echo `readlink ${deploy.basedir}` > ${deploy.basedir.number}/previousVersion" />
</phingcall>
<echo msg="Version info OK!" />

Now the rollback becames easy:

<target name="rollback">
    <echo msg="Rollback" />
    <exec command="echo '-H ${ssh.username}@${ws.host}'" outputProperty="ws.hosts.argument" />
    <phingcall target="exec-pssh">
        <property name="ssh.host.remote" value="${ws.hosts.argument}" />
        <property name="ssh.command"
                  value="
                        cd ${deploy.basedir} &amp;&amp;
                        PREVIOUS_VERSION=`cat previousVersion` &amp;&amp;
                        echo PREVIOUS_VERSION is $PREVIOUS_VERSION &amp;&amp;
                        if [ ! -d $PREVIOUS_VERSION ];
                        then
                            echo FAILURE PREVIOUS_VERSION $PREVIOUS_VERSION does not exist, we can not rollback;
                            exit 255;
                        fi &amp;&amp;
                        cd .. &amp;&amp;
                        rm -fR ${deploy.basedir} &amp;&amp;
                        ln -s $PREVIOUS_VERSION ${deploy.basedir} &amp;&amp;
                        ACTUAL_VERSION=`readlink ${deploy.basedir}` &amp;&amp;
                        if [ $ACTUAL_VERSION != $PREVIOUS_VERSION ];
                            then
                                echo FAILURE ACTUAL_VERSION: $ACTUAL_VERSION, PREVIOUS_VERSION: $PREVIOUS_VERSION;
                            else
                                echo OK link ok actual: $ACTUAL_VERSION;
                        fi
                        " />
    </phingcall>
    <echo msg="Rollback done" />
</target>

Rollback of Persisted Data

The management of data is a bit harder, because you have to manage the schema changes, in the persisted data and at application level.

A specific post on the subject will coming soon… for now let assume that you can simply run a rollback script and revert schema and data on you database.

Test the Rollback automation

An important point to remember is that the rollback must be tested, as you do with any other feature. The current process thaw we use is very simple: at every deploy in the “Integration Test” environment, we do the following steps:

• deploy new version
• rollback
• deploy new version again

Conclusions

This is a lesson learn from the first “not working” rollback we did in production…. now we want to be sure that rollback works every time, so we automated it and dry run the rollback procedure in deployment pipeline.

What are you thoughts on the subject? How do you manage rollbacks? Let me know 🙂