We found that this is the best way to mitigate the usual impedance mismatch between “devs” and “operations”: deployment to production WILL fail, even if you write the best documentation possible, with a full description of all the steps required, even if the “operations” guys write some script to execute the procedures.
The real solution that worked in our environment was to create a single deploy script that is used to deploy the application in all the environments in the deployment pipeline, from dev to production environment.
The recipe to deploy on the same way from dev to production environment
As our application stack is based on PHP running on Linux, was natural for us to implement the deploy script using phing, scp and ssh
The workflow is:
- build an “artifact” (environment independent)
- copy the artifact to the destination environment in an new folder, let’s say “application.233”
- copy the configuration files required for the destination environment
- setup all the required permissions on destination environment
- run the “migration” scripts required to update the database, if required
- move a symbolic link “application” from “application.232” to “application.233”
- apache will now see the new files and serves the new application
We deploy in all the environment using this same workflow, so even to deploy on “localhost” machine, we run “scp” to copy files and “ssh” to setup permissions, run migrations, etc… and because we have multiple servers on the each environment, we use the “parallel-ssh” and “parallel-scp” utilities.
Let’s see some of the magic that we have in the phing build.xml file…
How to use parallel-scp to copy on multiple servers
Here is the snippet from build.xml
<target name="exec-pscp">
<exec
command="pscp ${scp.host.remote} -O LogLevel=ERROR -O UserKnownHostsFile=/dev/null -O StrictHostKeyChecking=no -O IdentityFile=${ssh.privkeyfile} -t 0 ${file} '${todir}'"
outputProperty="pscp.output"
/>
<echo msg="pscp.output: ${pscp.output}" />
<condition property="pscp.failed">
<contains substring="FAILURE" string="${pscp.output}" />
</condition>
<fail if="pscp.failed" message="Remote copy failed: ${pscp.output}" />
</target>
The property ${scp.host.remote} contains the list of the servers to deploy into, for instance “-H root@server1 -H root@server2”
As you can see, we store the output of the pscp command in the property “pscp.output”, check the result of the operation and fail the build if required.
How to use parallel-ssh to remote execute commands on multiple servers
A similar trick is used to remote execute commands on all the servers in the environment. Here is the snippet:
<target name="exec-pssh">
<exec
command="pssh ${ssh.host.remote} -O LogLevel=ERROR -O UserKnownHostsFile=/dev/null -O StrictHostKeyChecking=no -O IdentityFile=${ssh.privkeyfile} -P -i -t 0 '${ssh.command}'"
outputProperty="pssh.output"
/>
<echo msg="pssh.output: ${pssh.output}" />
<condition property="pssh.failed">
<contains substring="FAILURE" string="${pssh.output}" />
</condition>
<fail if="pssh.failed" message="Remote command failed: ${pssh.output}" />
</target>
Again, the property ${scp.host.remote} contains the list of the servers to deploy and we check the result of the operation looking at the “outputProperty” of command.
The deploy worlflow
here is an extract from the build.xml, showing the main workflow for the deploy.
<target name="deploy">
<echo msg="Copying ${artifact.tgz.path} archive to environment ${deploy.environment} hosts:${fs.host}" />
<phingcall target="exec-pssh">
<property name="ssh.host.remote" value="${fs.hosts.argument}" />
<property name="ssh.command" value="rm -Rf ${fs.deploy.basedir.number}; mkdir -p ${fs.deploy.basedir.number}" />
</phingcall>
<phingcall target="exec-pscp">
<property name="scp.host.remote" value="${fs.hosts.argument}" />
<property name="todir" value="${fs.deploy.basedir.number}" />
<property name="file" value="${artifact.tgz.path}" />
</phingcall>
<echo msg="${artifact.tgz.name} copied to environment ${deploy.environment} host:${fs.host}" />
<!-- we unzip the application artifacts in a new fresh location, -->
<!-- set permissions and and move a symlink to this location. -->
<!-- in this way we have a local backup of previuos version, useful for -->
<!-- a fast rollback (just move the link to the previous version -->
<echo msg="Estracting archive to ${deploy.basedir.number} on host:${fs.host}" />
<phingcall target="exec-pssh">
<property name="ssh.host.remote" value="${fs.hosts.argument}" />
<property name="ssh.command"
value="tar -xzf ${fs.deploy.basedir.number}/${artifact.tgz.name} -C ${fs.deploy.basedir.number};
rm -f ${fs.deploy.basedir.number}/${artifact.tgz.name}" />
</phingcall>
<echo msg="copying configs from configs/${deploy.environment} to ${fs.deploy.basedir.number}/code/configs on host:${fs.host}" />
<phingcall target="exec-pssh">
<property name="ssh.host.remote" value="${fs.hosts.argument}" />
<property name="ssh.command" value="mkdir -p ${fs.deploy.basedir.number}/code/configs" />
</phingcall>
<phingcall target="exec-pscp">
<property name="scp.host.remote" value="${fs.hosts.argument}" />
<property name="todir" value="${fs.deploy.basedir.number}/code/configs" />
<property name="file" value="configs/${deploy.environment}/*.ini" />
</phingcall>
<echo msg="done" />
…. more steps here…. and then
<echo msg="================================================================================" />
<echo msg="Final step: Linking ${deploy.basedir.number} to ${deploy.basedir}" />
<echo msg="================================================================================" />
<phingcall target="exec-pssh">
<property name="ssh.host.remote" value="${ws.hosts.argument}" />
<property name="ssh.command"
value="rm -fR ${deploy.basedir};
ln -s ${deploy.basedir.number} ${deploy.basedir};
" />
</phingcall>
<echo msg="================================================================================" />
<echo msg="Checking link..." />
<phingcall target="check-link" />
<echo msg="Linking OK!" />
<echo msg="================================================================================" />
</deploy>
Conclusions
I can show here just a few snippets of our build.xml, but I think you can get the basics from what I show you:
- Write a single deploy script that can be used to deploy in every environment in the same way
- Start from solving problems related to your production environment early (EG: multiple servers, minimize downtime, easy rollback, etc..)
- Alway check the results of the operations run and fail as required.
Feel free to share your comments and thoughts
