After you deploy your application on production environment, you should be sure that everything is up an running fine, and that your business still run as expected. You look at you logs and everything is fine: no error is present and the application flows regularly, then you run a manual test and it works as expected.
The day after you receive a phone call: you daily income from is half of the day before, and all the people in the business team is angry with you…
Antipattern: validate your deploy looking at your logs and running a manual test of your application.
The recipe: use business metrics to validate a deploy
At onebip.com we found that a better way to validate if a deploy can be considered successful or not is to measure some business kpi after a deploy for a few minutes (2 minutes is enough): if the kpi values still in the normal range, than the deploy is considered “good”, otherwise an automatic rollback to previous version is run.
Here an example from our phing build.xml file:
<target name="validate-deploy-by-kpi" depends="prepare">
<echo msg="Validation by kpi for deploy ${build.number} started..." />
<exec command="php ${project.basedir}/code/libraries/Onebip/Monitoring/validateDeploy.php" outputProperty="validate.deploy.kpi.output" />
<echo msg="validate.deploy.output: ${validate.deploy.kpi.output}" />
<condition property="validate.kpi.failed">
<contains substring="FAILED" string="${validate.deploy.kpi.output}" />
</condition>
<fail if="validate.kpi.failed" message="Validation by kpi on deploy ${build.number} failed. We should rollback" />
<echo msg="Validation by kpi for deploy ${build.number} completed" />
</target>
The KPIs used for deploy validation are the same that we use for h24x7application monitoring: we instrumented the software using some probes (for the curious people, some details are shared here on DZone) and we send application metrics to Datadog, as SaaS solution born to track metrics and events.
To validate the deploy, we just query the Datadog service (2 minutes after the deploy) using a custom php script able to manage the http authentication and retrieve the most important business metrics, then we check if the last values are inside a predetermined confidence band, and decide for the “go/no-go”.
Here an example: the 2 red vertical bars mark the start/end of the “validation by KPI” step
Conclusions
Instrument your software to track business metrics, and then use the same metrics to validate if a deploy can be considered successful or not.
