The first problem than you can have after a deployment is that…. the configuration files for your software are somehow wrong: a wrong database password, a wrong connection url, a wrong log file name… more or less every configuration parameter in your application can be wrong.
Antipattern: prepare a configuration file for the production environment, deploy the application and look at the logs.
Well, you know what happen now: the application sometimes fails to start and you have to spend your time looking at your log files, while the application is down and your services not available to users.
We found a simple solution for that: introduce a smoke test stage into the deployment pipeline.
The recipe to smoke test your deployment
First of all, what is a Smoke Test? According to Wikipedia,
a smoke test proves that “the pipes will not leak, the keys seal properly, the circuit will not burn, or the software will not crash outright
The idea is simple: read the configuration parameters and validate them in the actual environment. For instance:
- search all the database connection parameters (hostname, database name, username, password, etc…) and try to make a real connection to that database.
- search all the remote URLs your application uses (APIs, webservices, etc..) and try to connect to them.
- search all the folder names and file names and check the existence and the permissions.
You can just create a script in your preferred language, execute it before going live with the new release (remember the Practice #3: Deploy on the same way from dev to production environment) and abort the build in case of a test failure: your application still at the actual release and continues to be available to your users.
It’s very import to execute the smoke test script in the enviroment where your application run, because you want to be sure that the resources (database, external API, etc..) are available from the same host where the application is installed, so, if you have multiple servers, run the smoke tests from all the servers in the array.
Here some real how-to, taken from our smoke test scripts.
How to smoke test a MongoDB connection
Here an example script:
$ smoke-test-mongo-connection.sh srv1:27017,srv2:27017,srv3:27017
#!/bin/bash
# smoke-test-mongo-connection.sh
#
IFS=',' read -ra mongoConnections <<< "$1"
mongoConnection=${mongoConnections[0]}
mongoOutput=$(mongo $mongoConnection --eval 'status = db.serverStatus(); print(status["ok"]);' --quiet)
if [ "$mongoOutput" != "1" ]; then
echo "MongoDB not reachable at: $mongoConnection"
exit 1
fi
echo "Check configuration OK"
How to smoke test folder permissions
Just run a script like this:
$ smoke-test-log-permissions.sh path/to/my/logs theuser
#!/bin/bash
# smoke-test-log-permissions.sh
#
applicationLogPath=$1
applicationLogPathPermissions=$2
applicationLogPathOutput=$(stat --format=%U $applicationLogPath)
if [ "$applicationLogPathOutput" != "$applicationLogPathPermissions" ]; then
echo "Permissions or owner of file $applicationLogPath are wrong: $applicationLogPathOutput"
exit 1
fi
echo "Check configuration OK"
How to smoke test an URL
You can just execute a script like this:
$ smoke-test-url.sh http://api.exmaple.com
#!/bin/bash
# smokte-test-url.sh
#
statusOutput=$(curl --insecure --silent -o /dev/null -w %{http_code} $url)
if [[ "$statusOutput" == "000" || "$statusOutput" == "40*" || "$statusOutput" == "50*" ]]; then
echo "$url not reachable"
exit 1
fi
echo "Check configuration OK"
Conclusions
As you can understand, I can show here just a few rows of our deployment smoke test script, but I think you can get the idea:
- Parse the configuration files
- Try to instantiate database connections, connect to URLs and writing to files
- fail the build if one resource is not available
I would like to see your comments and thoughts here
