Simonluca Landi

You have just completed the deploy of latest version on your production environment, but something was wrong, and after a few minutes/hours the business guys ask you to revert to the previous version. You was smart, and did a backup of the latest version in production in a tar.gz. Next you manually copied the backup and deployd it to all your servers…. without making mistakes, because you had a perfect documentation for this procedure, and you remembered well all the steps to rollback… isn’t?

Antipattern: use a backup to rollback an application to a previous version.

The recipe: automate the rollback in the same way you automate the deploy

In the last post we see how to use business metrics to validate a deploy, using a mix of software instrumentation and PaaS services. But what to do when a deploy validation fails?

You have at least 2 different aspects to consider:

• Changes at “behaviour” level (your source code)
• Changes at “data” level (your database

Rollback of Source Code

At onebip.com we use a simple trick: every deploy is in different new folder, with a unique generated name (provided by our deployment pipeline tool). After a successful and validated deploy we update a symbolic link “onebip” to the working directory of the latest version of the application. To rollback the code, we just revert the symbolic link to the previous version, a matter of few seconds.

But how to know which is previous version? we create a previousVersion file with a very simple command

<echo msg="Preparing version info file in ${deploy.basedir}" />
<phingcall target="exec-pssh">
  <property name="ssh.host.remote" value="${ws.hosts.argument}" />
  <property name="ssh.command"
            value="echo `readlink ${deploy.basedir}` > ${deploy.basedir.number}/previousVersion" />
</phingcall>
<echo msg="Version info OK!" />

Now the rollback becames easy:

<target name="rollback">
    <echo msg="Rollback" />
    <exec command="echo '-H ${ssh.username}@${ws.host}'" outputProperty="ws.hosts.argument" />
    <phingcall target="exec-pssh">
        <property name="ssh.host.remote" value="${ws.hosts.argument}" />
        <property name="ssh.command"
                  value="
                        cd ${deploy.basedir} &amp;&amp;
                        PREVIOUS_VERSION=`cat previousVersion` &amp;&amp;
                        echo PREVIOUS_VERSION is $PREVIOUS_VERSION &amp;&amp;
                        if [ ! -d $PREVIOUS_VERSION ];
                        then
                            echo FAILURE PREVIOUS_VERSION $PREVIOUS_VERSION does not exist, we can not rollback;
                            exit 255;
                        fi &amp;&amp;
                        cd .. &amp;&amp;
                        rm -fR ${deploy.basedir} &amp;&amp;
                        ln -s $PREVIOUS_VERSION ${deploy.basedir} &amp;&amp;
                        ACTUAL_VERSION=`readlink ${deploy.basedir}` &amp;&amp;
                        if [ $ACTUAL_VERSION != $PREVIOUS_VERSION ];
                            then
                                echo FAILURE ACTUAL_VERSION: $ACTUAL_VERSION, PREVIOUS_VERSION: $PREVIOUS_VERSION;
                            else
                                echo OK link ok actual: $ACTUAL_VERSION;
                        fi
                        " />
    </phingcall>
    <echo msg="Rollback done" />
</target>

Rollback of Persisted Data

The management of data is a bit harder, because you have to manage the schema changes, in the persisted data and at application level.

A specific post on the subject will coming soon… for now let assume that you can simply run a rollback script and revert schema and data on you database.

Test the Rollback automation

An important point to remember is that the rollback must be tested, as you do with any other feature. The current process thaw we use is very simple: at every deploy in the “Integration Test” environment, we do the following steps:

• deploy new version
• rollback
• deploy new version again

Conclusions

This is a lesson learn from the first “not working” rollback we did in production…. now we want to be sure that rollback works every time, so we automated it and dry run the rollback procedure in deployment pipeline.

What are you thoughts on the subject? How do you manage rollbacks? Let me know 🙂

You deploy the application on your production environment and… surprise: doesn’t work!

Antipattern: start from your dev environment and scale up adding more servers, load balancers, db replicas etc…

While this seems a standard way to design and release a new application, it will just postpone the issues the you will have when you will scale up your infrastructure to production.
Let me clarify: at a first glance, this can be confused with “over design”, and we all know this is wrong, but you know that application will run in a production environment, serving real users, so you have to design starting from your production environment, and solve the issues that arise here first (fail fast and learn fast, as always).

The recipe: scale down from production

Here is a list of design considerations, emerged from experiences at my company.
One of our application, based on PHP+MongoDB, run on “standard” setup for a production environment: one domain name is resolved to 2 different IPs, allocated to 2 load balancers running HAProxy; the load balancers direct the traffic to 2 or more web servers running Apache+PHP, connected to a 3 nodes MongoDB replica set (no need for a shard yet). All the machines are virtual machines, built using RighScale templates.

Using this methodology, can “scale down” the infrastructure simply collapsing the 3 layers (LB, FrontEnd, BackEnd) in less virtual machines, still using the same full stack: for instance, our “Integration Tests” environment runs the full stack on a single machine, but HAProxy and a MongoDB replica set are included, even if not required. The apparently complexity of this setup is simplyfied by using “Server Templates” and virtual machines, than can be instantiated from scratch with just a simple click of a button.

The development environment is a virtual machine too, running on each of our laptop; we are working to have the dev virtual machine built from a RightScale template too, but at least we can share a “master” image to all the evs when something changes in the setup (an article on that coming soon…)
This practice saved our life a few times:

• Starting from v.1.3, the MongoDB driver for PHP requires a different configuration for replica set: we discovered that BEFORE going in production enviroment (yes… we should read the change log…)
• We have cron jobs that must run as “environment singleton” (just job running in a given enviroment): again, we discoverd this need BEFORE going in production and addressed it.
• Because we have multiple front end web servers in production, we need to address the “session management”: the final decision was to design the application “sessionless”. If we would have followed the classic approch, we should have addressed the “session management” in prodution enviroment in some way: storing sessions on DB, using server affinity in load balancer, etc… in summary: a more complex infrastructure.

Conclusions

Starting from design your production enviroment and scale it down is a practice to anticipate the troubles, and solve them at “development” stage.
Usign modern technologies like server virtualization and automated infrastructure provisioning, simplify this practice.

This is what we are learning in the Company that I work for, where we are moving from an “svn update /var/www” anti-pattern to a deployment pipeline in 4 different environments, from Dev to Production.

Here you can find real solutions and how-to that worked on our environment.

Practices that we follow for Continuous Deployment

As soon as we feel that we are suffering in some way, we try to find a Practice to mitigate this sufferance. Here what we did until now

Practice #1: Same software stack from dev to production environment
Mitigate: “on my PC works!”

Practice #2: Dedicated Continuous Integration environment for unit test
Mitigate: “on my PC works!”

Practice #3: Deploy on the same way from dev to production environment
Mitigate: deployment to production fails

Practice #4: Smoke test the deployments
Mitigate: major configuration errors

Practice #5: Use Business Metrics (KPI) to validate deploy
Mitigate: deployment to production secceded, but business was impacted

Practice #6: Automate Rollback
Mitigate: production downtime after a failed deploy

Practice #7: Keep deployment pipeline fast
Mitigate: too much time required to go in production

Practice #8: Scale down, not up
Mitigate: “on my PC works!”

(updated soon…)