The Zen of Deployment Automation
I work with a lot of users using our Octopus Deploy server on a daily basis. It seems that not all, but many of our users don’t fully grasp the philosophy behind automated deployments so I thought a rant was in order. If you’re using Octopus Deploy or even thinking about implementing it, Octopus is indeed a great first step into automation – but that’s it. It’s the first step – not the final step. Simply using Octopus Deploy doesn’t mean you’re automating your deployments. How so?
I find that many users, misunderstand deployment automation because they simply translate the manual steps of what was previously done (sometimes verbatim) into Octopus Deploy. Many users are confused as their previous manual deployments had always worked before, but within Octopus their deployments can fail seemingly at random. Some users try to get very creative, even beyond their understanding of what they intended to do or the purpose of the steps they’re executing. Ultimately what users fail to realize is that the concepts of manual deployments and automated deployment are completely incompatible with one another. People like to think that there’s some overlap between the two, but frankly there isn’t. In Octopus Deploy, you need construct your deployment process from scratch – not just recreate your previous manual process. In order to be successful in automating your deployments, you need to design your deployment process from a minimalist zen-like implementation with a “fire-and-forget” approach.
Too often I see some of our users deploying by manually skipping steps for certain environments. While some may say they like the level of control (or whatever reason they can think of), this is not considered an automated deployment. If anything creating large deployment processes with specific steps for specific environments is a recipe for disaster. For example, one team that tried deploying to staging had run into a repeated failure as one of their files was locked in their installation directory. Out of the 43 steps in their deployment process, they forgot to stop IIS for their staging environment. This example may be easy to fix but the core lesson behind this is: don’t rely on someone to reliably configure your deployments. Human error is far more possible than people admit. It may seem like it’s easy to enable/disable steps – but that isn’t automation. The general purpose of automation is to remove human interaction – not to extend it.
Here’s an example of common (simplified) example of a deployment process I see from time to time. Note on the left side (blue) is a poor, lengthy, intricate series of steps that are specific towards an environment or a particular server. On the right side (green), steps are consolidated into simple, encompassing, and adaptive to any environment. The blue side can/may represent what was done manually before Octopus Deploy. The manual steps do not translate well at all in terms of automation, reliability and speed.
I’ve seen deployment processes that have exceeded 40, 50 and 60 steps; these steps require users to enable/disable steps for a given environment. This old tried-and-true manual way of deploying project is really a curse in disguise. In spite of those users not realizing the logical gaps between manual and automated processes – there’s nothing about a manual process that translates to automation. From manual intervention steps, to backing up projects, backing up web.configs and moving them back into place after a NuGet package contents have been installed, to manually moving files around, users often don’t realize that automation requires a completely different perspective from how they used to implement their deployments.
I want to point out that not all of our users misuse Octopus; many use it very well. Users and/or teams that do well typically ask a lot of questions, learn new techniques and try out different approaches repeatedly and consistently deploy day by day across their environments. They put the time and effort in and it shows. I can see the consistency of consecutive successful deployments that have been promoted between their environments. I also see the deployment times gradually decreasing with respect to newer releases. The biggest sign of a team that uses Octopus Deploy well: their production deployments go smoothly.
The reason I’m stating this philosophical stance of how automated deployments need to be treated in terms of a zen-like minimalist perspective is due to the fact that many of our users and teams still lean on bad manual habits of before and treat automation no differently. Using Octopus Deploy to automate your deployments means you have to orchestrate steps, variables and NuGet packages in such a way that guarantees 100% certainty that the deployments will work as expected – with no manual configuration or intervention. This is the “fire-and-forget” approach of deployment automation. Any possible deviations in configurations or unstructured application changes are just vectors for possible failure. But as I’ve learned over time – most users don’t see it that way.
I’ve come to accept the fact that a minority of users may view Octopus Deploy as an extension or rather a “dip” of their toes into the systems administration pool without getting their feet wet. All the settings they wanted to tinker with and change on the fly (within their desired time frame) they now have the keys to the kingdom to do so. Of course, with that power comes the requisite knowledge of how to weld everything properly but the comprehension that’s required from said users I’ve found to be severely lacking.
Octopus Deploy makes deployments very easy to orchestrate – perhaps too easy. Having the Octopus Step Template Library also allows users with minimal to no PowerShell knowledge at all to do more systems administration type tasks with zero understanding of the actual inner workings. Is this dangerous or bad? Insert typical tech answer here: it depends. Should developers know more about IIS, Windows Services, SQL Server, etc.? Absolutely. But that responsibility is on those whom have the ability to deploy at will and there’s the catch. The friction I’m seeing with users and teams that struggle with automating their deployments is a direct result of the fact that they don’t fully understand the technology that they’ve been shielded from for so long. That’s why they keep sticking to the same routine as before – they don’t really know what to do from scratch because they don’t know how these systems actually work. So what can you do to move towards faster, reliable deployment automation?
1. Assess – Take an accurate inventory/stock of your knowledge. Acknowledge what you know but more importantly what you don’t know. If you don’t know IIS, now’s the time to learn. Not familiar with sc.exe for installing your Windows Service? Saddle up. You know PowerShell but don’t know NuGet? Now’s the time to tinker with NuGet.exe and your NuSpec file. I don’t know every little bit of technology I work with, but I can immediately recognize when I’m out of my depth. I have no shame nor arrogance about it, but I make sure to be true to myself in acknowledging that which I don’t know. From there I have to start asking questions – good questions too. From there I can proceed further and start figuring out a way to understanding what I don’t know.
2. Learn how to learn – If you don’t know something, start by researching. Books, websites, blogs, whatever. There’s so much information available it’s merely a matter of taking the time to do so and dig around. If you’re not very good at researching, the only way to get better is to practice – so why not start now? Additionally, one of the best things about practicing is that it should lead you to ask more questions relevant to the problems you may be having. The more questions you have, the better to answer them. When you get good at asking questions, you’ll be able to focus on the more important questions as they arise – and eventually you may solve problems faster than before simply by learning to ask the right questions.
3. Practice – The cure to these ailments is practice. After that? Adjustments and.. you guessed it: more practice. And then more learning followed by practice. I view every deployment in non-production environments as an opportunity to prepare for a production deployment, but it’s also an opportunity for you to learn more about the systems you’re working with. Reading manuals may be in your near future – and it’s a good thing. There’s no magic bullet for this – if you’re automating your deployments you’re going to have to understand more about your systems than you did as a developer as well as your packaging and your configurations – intimately.
4. Communicate – If you still can’t figure out something or need help, ask! Not asking isn’t going to help anyone – especially you. But be aware – asking for help means you need to put forth the time and effort of trying to solve your problems as well. Simply stopping on every minor road bump isn’t going to get you to your goal any quicker and ultimately, you won’t really learn anything new if you’re constantly depending on others.
Every now and then I’m reminded of a quote Carl Sagan once said about humans and technology.
We have also arranged things so that almost no one understands science and technology. This is a prescription for disaster. We might get away with it for a while, but sooner or later this combustible mixture of ignorance and power is going to blow up in our faces
The quote may be a little hyperbolic, but the sentiment is fitting. My goal of this rant isn’t to intimidate or pressure (okay maybe there is a little pressure here), but to educate and promote a proper deployment automation philosophy. Whether you’re using Octopus Deploy or not, deployment automation is always worth doing. And while I prefer Octopus Deploy, the principal of Zen-like philosophy with the “fire-and-forget” approach really is a winning combination that could be applied beyond deployment automation. Like anything else in life, starting is the hardest part but like everything else, it can be mastered.