The title of this article is sort of odd, because the very idea of "agile" is flexibility. But that's the point, I want to emphasize the flexibility of agile. One of the core concepts in agile is the idea of "Responding to change over following a plan". This can't be overemphasized. Customers/users expect resilience, and the ability to change their minds.
The most important thing I've learned in 3 decades of software development is that it's all guesswork until our code hits the real world. They're not "requirements specs". They're guesses. That's not an "architecture". It's a guess. Once you wrap your head around that, you realise that the most valuable thing you can deliver to your customers is the ability to CHANGE THEIR MINDS. If that's not right at the top of your priorities, you have failed at software development. - Jason Gorman https://twitter.com/jasongorman/status/1413255915973795840
This is one of the fundamentals of what the agile community has recognized to be true. When thinking in a waterfall kind of way, where everything's done through massive requirements, analysis, design, development, and delivery phases, we ALWAYS deliver the wrong product. Note I said "always", not sometimes, but "always"! Worse yet, we deliver it late, and we deliver it for a greater cost.
The corollary of all of this is that if we've come to this epiphany of understanding what Gorman has said, we realize it also implies that we deliver quickly, and regularly. As I like to say, deliver to production from day 1. If we literally deploy a "Hello World" application into production from day one, and we deliver software changes daily to that production environment, the changes come in bite sized chunks. Bite sized chunks let users determine if we're going in the right direction, without being overwhelmed with a bunch of features that have all been released at once. If we deliver with a big bang waterfall approach, all sorts of side-effects arise from that. The users don't even really know whether it's the product they want, until they fully start using it in production, and gain significant experience using it. It can even take months to realize that it's the software causing them to move so slow in the daily work they do. Because it's new to them, and it is perhaps the software that is overly complicated, it takes them time to recognize what is slowing them down. By then it's too late, because we've spent months developing a product they didn't even want. What the heck!
Improving Efficiency by Reducing Work Size
Two important agile principles that counter the thinking that we should be doing large batches of work, with estimates, roadmaps, etc, from the Twelve Principles of Agile Software are...
Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. agilemanifesto.org
Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. agilemanifesto.org
I would wager that most organizations today are running their software development operations and processes at 20% efficiency, if that. In fact, it's a well known fact that IT organizations or departments frequently go 15x over budget and time on their projects. Why? Because they do a ton of requirements gathering, analysis, design, development, and delivery. They have to work hard deploying lousy software that they had to rush into production, only to find out it wasn't what the users wanted. It's funny, because they literally "rush" into production, and yet it's months or years later than a true agile process would have had the same software launch.
If each change is a day or two of work, the most we have to throw away is a day or two of work if we get it wrong. Furthermore, because it's so small, we don't often get it wrong. The recent change might not be ideal, but it's easily adjusted, because the changes were tiny. If we turn it into a week, there's more uncertainty, and the uncertainty grows exponentially past a week development cycle time. This is because the cyclomatic complexity of the code deployed per change increases. When we finally deliver it to production, it's damn near impossible that it's correct. Humans can't handle the increasing levels of complexity, we start making huge mistakes. Many times we do not even know it until it hits production. We also find out many days or weeks after deployment that it's not what the users wanted, or it has a bug. When the changes are tiny, there's no learning curve for the users, so they know right now that it's either right or wrong.
What's interesting about all of this is that waterfall and agile have some core things in common. The scale is different, but so is how agile deals with analysis, design, and development. In agile, we still engage in requirements gathering (stories), analysis, design, development, and delivery. The difference is, they happen continuously throughout the project. We just do them in such tight repetitive micro loops that there's rarely any wasted time. We do them on the order of hours or days at most, and very specifically not big long term plans or roadmaps.
It is the large scale of waterfall that causes so much failure, because humans
aren't good at predicting the outcomes or plans of large amounts of work. In an
agile organization, roadmaps have nearly no purpose, if we're attaching
dates to them, and are firm that we must meet those dates. There are too many
variables in software development to be able to predict far into the future.
Roadmaps can be useful for knowing what we are planning, but the when is
impossible for humans to predict or adhere to without compromising the system
in development. With agile, those roadmaps are ultimately just a burn down
chart. Burn down charts can be somewhat predicting when we'll reach certain
goals, but they too lose their accuracy the further into the future we go.
Ironically, working with agile provides a better, more well understood timeline than waterfall ever could. With waterfall, if something needs to change, there's no time for analysis/design, we just change it and move forward, to the project's detriment. Agile integrates requirements, analysis, design, and development so tightly, that when something needs to change, we are able to easily accommodate that change, because it usually has a micro impact on the project. This is especially true if we are using TDD, as TDD ensures proper coupling and cohesion of code, making it easy to shift the code around to accommodate the new requirements.
It's important to consider how the development cycle times work in a waterfall vs agile approach. Keep in mind these are just estimates, as agile tightly integrates all phases in a loop with each other; meaning they aren't phases, they are repetitive continuous cycles. Some organizations call their waterfall process agile, when it's just waterfall relabeled agile, plain and simple. It's not agile that fails them when they do that, it's the same old waterfall. This table will help with understanding that working in smaller more frequent releases will help aide in more successful delivery with less waste.
| Process | Waterfall Time | Agile Time |
|---|---|---|
| Requirements | Weeks to months | Hours at most |
| Analysis | Weeks, sometimes months | Couple of hours |
| Design | Weeks | Minutes, but ongoing through the entire development cycle, with evolutionary design through TDD |
| Development | Weeks, months, or even years | Hours or days, preferably less than a week with a strong preference for a couple of days. |
| Delivery | Plan the release for many days, possibly weeks, but usually fails to launch, and requires a great deal of effort to get it running. | Seconds to minutes; failures are rare, as everything is automated and is deployed to prod regularly |
One might tend to think that this means you can have long-drawn-out plans and roadmaps, while only implementing individual stories in a micro cycle fashion. That isn't agile, and it's one of the core reasons that waterfall has failed repeatedly for many decades. Humans just can't predict the future very well. So if our intention is for our roadmaps to be concrete in any way, then we're just looking for failure. It's a universal truth that everyone knows, humans make mistakes, and the mistake making will never end. Save yourself the headache, learn the ways of real agile.
The main difference between waterfall and agile thinking is that the waterfall people imagine that they can get things right. - Allen Holub
Responding to change over following a plan
One of the big drawbacks of waterfall, with it's long term plans, long development cycles, etc, is that humans almost always fall victim to logical bias. When there's a big plan, the thinking is "we must follow it". It happens almost every time. We have put so much time and effort into following that plan, that we outright reject the thought of abandoning it, even when it's the wrong plan. We just want to get it done. Abandoning the plan feels like it's the wrong thing to do, even when it's frequently the best thing to do. This is called the Sunk Cost Fallacy. It happens when we believe that we can't abandon or change the plan because of the amount of time, effort, cost, etc, that we've already put into the project. We believe it's just not worth changing at this point, because the cost is already high, let's finish it. But, when your time investment is minutes or hours, suddenly we're like "let's try again".
What is agile? Look up the word in the dictionary, and you've got a pretty good start right there. It's the ability to change direction quickly and with very little effort. These long term plans, big projects, etc, don't allow for agility at all. They literally fight agility and try to bring a very rote process to the work. This results in no-one wanting to change direction. Being agile is very specifically about being able to quickly and easily change direction. When we have processes in place that prevents changing direction, it stands to reason we are not agile at all. Waterfall on the other hand is all about rote process, where changing direction either doesn't happen at all, or is very slow to happen. NOT AGILE!!!
Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. agilemanifesto.org
Agile solves this, by tightening the loop, and shrinking the plan size down to a
micro level. When a plan is wrong, and it's only 2-3 days of effort, everyone on
the team is perfectly happy
saying Yeah, this isn't working, let's do something entirely different. This
is because there's very little time/cost investment, and therefore no emotional
investment. Bias tends to creep in when there are emotions involved. By doing
small batches of work, we reduce not only the cost of investment, but the
emotional investment, and therefore the frequency in which we become biased.
Yeah, this isn't working, let's do something different
I've heard people say things like "If you get the scrum plan wrong, then you're doing something wrong. The estimates should be right. Why are your estimates off?". Not only is this a toxic notion, but the person doesn't even realize how they don't really believe in agile, they believe in waterfall. Yes, that's the key point, except not the way they think; everyone is bad at estimating. There are those that believe they aren't, but really they are just internalizing the "gaming of the system", and not realizing that they game the system. I've recognized from the beginning that I have to game the system to make a good estimate. One of the most successful ways of gaming the estimates is to figure out what needs to be done at a relatively high level. Then, make the estimate based on if the stars, and the planets were all to align. Then just multiply it by a fairly large number. Doing it any other way is just "pretending" at best. We humans can't estimate large batches of work; oh, did I say that already?. Obviously the solution is to not engage in large batches of work. Focus on doing small batches of work with a regular cadence. Over time, we find that we're getting things done way more quickly, because there's less "messing around" so to speak.
Mindset
If we look up the definition of agile on google (define agile), we find out it is not a noun, it is not a verb, it is an adjective. It is defined as "able to move quickly and easily". It's so easy to adopt the agile mindset. Stop asking "What is agile?", and replace that question with "Is what we are doing here agile?". Ask that question repeatedly, with every action, every process, every piece of code; EVERYTHING! If you do, your eyes will be opened to how badly you need agile, and how not agile your organization really is. Unfortunately, it will also depress you!
able to move quickly and easily - google definition (define agile)
Why do most organizations not implement real agile? I believe it has to do with their mindsets. Management is typically stuck in such an endlessly biased cycle of believing that old school waterfall/command and control style of management is a good idea. They have such heavy "process" thinking that they can't get their mind around the idea of letting go, and implementing lightweight agile processes to get the work done. Ironically, these lightweight agile processes would make them look absolutely amazing to the customers/users, their staff, etc. Yet they fight real agility, because they think there is no "process" or " control" over the work; which couldn't be further from the truth.
For example, how many organizations today have a manual change management process using a Change Advisory Board (CAB)? Let's just say that it's super common. CAB doesn't improve stability of production systems. CAB doesn't improve security of production systems. CAB doesn't improve anything about production systems in any way shape or form. What CAB does is slow down the change process, with no tangible benefits at all. This is such a well known fact, and yet managers frequently hold on to their CAB process like it's their last meal. It's because they are biased about what they consider to be successful outcomes of a project, based on their past heavy waterfall processes. Well let's forget about personal opinion for a moment, and see what the experts have to say.
We found that external approvals were negatively correlated with lead time, deployment frequency, and restore time, and had no correlation with change fail rate. In short, approval by an external body (such as a manager or CAB) simply doesn't work to increase the stability of production systems, measured by the time to restore service and change fail rate. However, it certainly slows things down. It is, in fact, worse than having no change approval process at all.
- Nicole Forsgren, Accelerate.
I could go on and on about bad process, but this article is long enough. If an organization is unable to see that processes like CAB, and others, are strangling their agility, they aren't agile; PERIOD.
Waterfall was an accident, literally. Thirty years ago, Winston Royce was writing a paper to refute the idea of the common waterfall diagram of today. Unfortunately, by using that diagram, it caused people to use it as if it was the way things should be. Waterfall was born out of an article that was refuting it before it was even born; how ironic is that. The past 50 years of managers advocating for super heavy process, was caused by an accident.
- See Clean Agile by Robert C. Martin