One of the critical factors contributing to the success of your software is your ability to learn and adapt in order to improve your application. The faster you can move, the better off you’ll be. This can affect you on several levels, but it’s imperative that you have the means to confidently update and release new code even during peak hours. You have to be able to ship, and anything that prevents that will hurt your business.
For an application with thousands of users, any change is going to upset someone. The good news is that most of the complaints will be knee-jerk reactions to the fact that something changed. Some of your power users spend a lot of time in your application, and they’ll develop a muscle memory for some tasks. So even if you improve a given task, they may be upset if that change interrupts their habits. This makes it difficult to change, but if you put yourself in their shoes, it’s fair. Empathize and work with your customers to reduce the negative impact of changes.
Some customers won’t like your changes, but this comes with the territory. You can’t dismiss their complaints, but don’t let the fear of a few complaints outweigh the benefits to the thousands of other users. You can lessen complaints by approaching any dramatic changes with caution. I’m not saying you should avoid significant changes, but if you can take small steps toward these changes over time, you’ll not only make fewer waves, but you’ll also have smaller, safer, and more contained releases.
One tactic I found to work well with customers after a big change was to empathize over the impact to their work, and then politely ask them to give it a week. More often than not, I’d hear back that they appreciated the improvements once they got used to them. Occasionally, I’d get much more tangible feedback about what they didn’t like, which made it easier to fix the problem so they’d be fully on board with the changes.
The last thing you need is for a key piece of code to break as the result of an update. To be confident your updates aren’t going to break anything, thoroughly test any code going into production.
Use automated testing and continuous integration. They might seem tedious on the surface, but the first time they save you from a regression bug, you’ll be glad you had them. The more confident you are that your code works well, the more confident you can be about releasing updates.
The larger the update, the bigger the risk. There will be times where you’ll need to make significant and far-reaching updates. But the more you minimize this and update small pieces regularly, the safer you’ll be. With small updates, even if something were to break, there’s a better chance it would be isolated and simple to fix. With a big release, you risk having several things go wrong at once.
Releasing a series of smaller updates working towards a bigger update can also reduce some of the shock to customers when the changes happen. If your app evolves slowly, changes are easy to adapt to, but if several important functions change all at once, customers can face real struggles adapting.
If code confidence lets you trust your code, release confidence gives you assurance that you can push updates during peak hours. That’s not to say there aren’t exceptions: when it comes to updates that include database changes or other significant upgrades, you probably shouldn’t release those during peak hours. But with everyday updates, you ought to have complete confidence that your releases will deploy seamlessly (or that you can recover quickly and reliably if they don’t).
Anecdote: Release Paralysis at Sifter
Sifter uses Capistrano to manage releases. Scaling it to manage multi-server releases was just one of the changes we made when we migrated to a multi-server environment. Shortly after the move, new releases began to have hiccups. At first, they weren’t particularly noticeable, but as time went on we started to realize that our search process was going offline for ten to fifteen minutes each time we released new code. During that time, customers couldn’t perform searches–and they didn’t like it one bit.
I was juggling a lot of tasks right then, and the solution wasn’t obvious. Rather than try to fix it, I dealt with it by releasing less frequently and only on weekends. Before long, I had become hesitant to release new code.
I had been trying to find the problem, but I was also neck-deep in another update I had wanted to release. I’ll spare you the technical details of what had gone awry, but the release paralysis brought on by the problem made everything worse. Even small bug fixes were delayed by a day until I could release them during slow periods. But after a couple of weeks of problems, I finally put everything else on hold to get the releases back in line.
Testing and Automated Deployment
One of the best ways to increase your confidence is to have a reliable–and nearly identical–staging or QA environment. If you can test deployments in your staging environment, you’ll be able to work out any kinks even when you’re adding new components to your deployment process and scripts.
The key to seamless deployments is automation. If you have to run more than one command to deploy your software, it’s not automated. It’s worth the investment to create an automated and reliable release process. In an ideal world, you could release many times each day, and your customers would never know if they didn’t see the changes.
Shipping a feature isn’t the end of it–it’s only the beginning. When you make changes, customers are affected, and they’ll inevitably have questions and some adjustment to make. If you ever make significant changes, be ready to provide some great support to help customers. Listen to their feedback. Write help documents. Create videos. Launching a feature is about much more than putting code on a server. It’s about every detail that affects and relates to customers. Plan to help when you have big changes coming. My rule of thumb was to plan a cushion week after big releases to work through support, bugs, and other enhancements and suggestions from customers.
After seeing and talking with many teams over the years, I know that the more you invest in these processes and tools, the better your software will be. You’ll have fewer bugs, fewer fires to put out, and greater capacity to put out those fires. It may seem tedious to set up all these processes, but once they’re in place you’ll find they’re more than worth the investment.