As you know, developers regularly deploy software. I remember that earlier, the deployment was a huge, important event. Everyone prepared for it in a special way: notified users wrote long documentation about a new version.
I as well remember how all these deployments turned out regular failures. The database was frozen. Users could not log in. Kernel panic. Unreasonably growing LA. Graying admin. Smoking one by one cigarette and very nervous team lead.
Later on, engineers understood that software should be deployed more often and even better to do this after every push. Besides, the delivery should be invisible to users. They just get some bugs fixed, or a new feature appears that they, as usual, do not notice and then learn about it from the blog of the project. This is called “Continuous Delivery”.
How to understand that deploy is addled
I think now the biggest part of web-oriented / micro service/ cloud software is deployed continuously. This process became so usual that for 5-6 years of active usage of continuous integration, we literally got used to the thought that is the only right way to deploy software.
Recently, I had to update a bot, that for 2 years regularly sends statistics on service to Slack. At one moment, the bot started to stumble with the data source, and along with valid information, we had HTTP 404. I decided that I can just comment on one invocation, and nothing will be broken.
I found the repository and opened the code, started to recall the contents, and even run tests and they were successful. I commented on unnecessary invocation and was going to deploy.
Then a very interesting thing happened. I started to feel a strange smell somewhere deep inside (this means that code is difficult to read and, therefore, difficult to support). You are preparing to deploy and suddenly understand that deploy is addled. Do you understand what I mean? There are sausages, but you are afraid to eat it. Do you feel? Butter was kept in the deep freeze, but you do not want to put it on the bread.
The established notion of Code Smell can be as well applied to the case, calling it "Deploy Smell". But I would call "Deploy Smell" the deploy that is new but smells bad, i.e. the deployment is regular but done in a wrong way. In our case the deployment process is enough adequate; however, it was not used for a long time and, therefore, it "smells bad".
I suggest introducing the notion of "Addled Deploy". This is the situation when it is necessary to secure yourself against the deploy that had not been done a long time ago. The situation when you do "addled deploy" and do not secure yourself against it will be called "addle".
How to avoid addled deploy situation
For example, you may not to do "addled deploy" on a Friday afternoon and deploy software on Monday in the morning, when all DevOps has coffee and are not going to leave their working places. This means that if everything will be broken, they will be able to revert changes. Here, you will find out how the MadOps team works.
"Addled deploy situation". This can be the header of the worklog where developers will describe everything they experienced after "addled deploy" to production.
UPDATE by Vadim Glebov: The situation is well-known. I do not have CD in my project, we deploy to production once in 2 weeks, deploy is semi-automated (admin manually runs scripts and reviews the results). But not to get "addled deploy" CI automatically deploys to demo 3 times per day. This approach allows me to have problem free deploy to production since I have time to reveal and fix problems with deploy. In conclusion, not to get "addled deploy" it is necessary to deploy the software at least in test regime.
UPDATE by Vasilii Alferov: This is not related to deploying only, there are also operations that are often forgotten such as back up data recovery, switching to the cold regime, and etc. All operations should be regularly and automatically tested not to become addle.
Latest articles here
