I've been discussing agile concepts with various folk. Here's some of my thoughts. Some of them are reiterations of things I've already said, but it doesn't hurt to rehash important concepts. :D
Prefer Shorter Timescales
Cycles of a couple of weeks to a couple of months, is a very old sentiment. When the agile manifesto was being composed, they were just starting a paradigm shift in how to efficiently deliver software solutions. It's evolved a lot more since those days. These days, everyone says deploy multiple times per day. I say, deploy from day one, starting with a “Hello World” application. We've put zero code in, we're just saying “hi”, and then we can begin by writing our first test, writing our first code to make that test pass, etc.
Responding to change over following a plan
Waterfall and agile do not mix at all, they are opposites of each other. Waterfall thinking brings roadmaps, estimates, enormous projects, and a host of other things that are not something an agile org values. They don't value them because they...
- create human bias that makes us want to follow a plan, over responding to change.
- create unattainable goals which may not match reality.
It's not that we don't have some rough plans of where we want to go, and if we want to call it a roadmap, okay, whatever. But, it's probably best to adopt another name, because roadmaps are a waterfall thing. Having a roadmap, with an expectation that you're actually going to follow it, creates a tonne of waste. We have to spend an enormous amount of time planning and estimating things that we may never even do. Then we have to update those plans when we realize they were not based in reality. Once we're heading down that trail, humans end up getting "committed" to the plan, and everyone naturally resists changing it, even when it really needs to change.
This is why some of the front-runners of agile, such as Allen Holub and Woody Zuill suggest that work arises out of working, not out of planning.
It is in the doing of the work that we discover the work that we must do. Doing exposes reality. - Woody Zuill.
When we plan in advance I see there as being two possible outcomes.
- We discover reality and adjust our plan.
- We ignore reality and follow the plan.
And, as Woody said, we bias ourselves by advanced planning, so in essence we will always follow the flawed plan to some degree; which is really not good. For some more insight, see this twitter post and scroll to the top.
The odd thing is, if we have to adjust our plan to match reality, that means we've created waste by making it necessary to adjust our plan. That implies a lot of things, such as updating work break down structures, plan documents, etc; all unnecessary processes in an agile context.
Working Software is the primary measure of progress
An agile iteration (“working product”) is not necessarily a "complete feature". The important thing is that we've deployed, the users can see it, use it, and provide feedback. If it's enhancing their work, then it's usable. You want the feedback from users, in a real production environment. In other words, QA or UAT is useful, but isn't enough, because those are typically canned use cases for testing purposes. Only production deploys can give you the real feedback you need to make user focused decisions in a timely matter. Note I said "user focused", because if they are not specifically based on input and feedback from the users, then they aren't the correct decisions.
So, if we deploy several times a day, but our typical story is two days, then we're getting feedback throughout that two days, thereby making it highly likely we'll deliver what the users actually wanted by the time the two days is complete. This quick deployment cycle is all about getting feedback quickly, so that we can make informed decisions. In other words, responding to change, as users figure out what they liked or didn't like. The faster you get feedback, the more decisions you can make, and the faster you can deliver an incrementally better solution. But, what's most important is that the users start experiencing how the new features are changing the way they work, and they can do that as early as possible. With that production experience, they really start to think hard: Hmm, this part is really nice! This other part would maybe be better this way. Again, users cannot get that feel/experience in a QA or UAT environment.
I've heard people say that with agile you end up with the same product that you intended with a waterfall approach. In some ways this is true, but it is very far from the complete picture. With waterfall, it's more often that we find out down the road that we needed different things than we thought we did, and now we've wasted time implementing some unnecessary thing, or implementing a necessary thing incorrectly. With agile, we end up delivery with a focus on actual needs of users, thereby saving time/money. This is because the users decided that's enough for now, and we can move on to something else. The Minimum Viable Feature is complete, so let's move on. It saves a lot on wasted time. Waterfall always causes us to think we need to complete the plan. That's why the agile manifesto says “Responding to change over following a plan”, because feeling a need to complete the plan, even when it's wrong, is a waterfall thing. Once you get into the sunk cost fallacy of waterfall, everyone wants to continue on that trail, even if they realize it's not what is needed. Human bias kicks in, and we stick to the plan, no matter the cost, as that's the comfortable charted road.
The flip side of this is if we're always traveling through uncharted territory, we get used to it, comfortable with it, etc. We get to our destination far more quickly than taking a big long planned route, and everyone becomes comfortable with that. Efficiency goes through the roof when adopting proper agile mindsets!
API Service Layers, an example
Let's say you're engaged in a big digital transformation, and you realize you need an API service layer for a large third party product, so that you can build add-ons for it without touching the vendor's systems.
The Waterfall Approach
A waterfall manager thinks about projects like this on a big scale. They consider how difficult it will be, and try to figure out how long it's going to take, what is actually needed, etc. With waterfall thinking, this is entirely the way projects like this work. We try to figure out what's needed in advance of actually knowing. We then end up trying to figure out how long it will take without actually having any way of knowing (we don't really know what's needed remember). All of this is guess work 100% of the time, without exception.
The Agile Approach
With agile thinking, we approach products like this very differently. We discover that we need feature X of the vendor system to be exposed to applications we need to write, because the vendor system just doesn't offer the feature we need. We'll call this application "Discovery", since it's our first need that we're 100% certain of. We then engage on discovering why we need this data exposure, and what exactly needs to be exposed, but we do this on a micro-scale.
So, for example, let's say we need biographical information. Obviously we almost every app we write is going to need this. The waterfall mindset would be to try and identify every possible use of this biographical data API, which would take months, and then implement it. But with agile, we know Discovery only needs first name, last name, email address, and login id. So, we start by writing the absolute simplest biographical data API service layer, which exposes, yep, you guessed it, first name, last name, email address, and login id. Now this is going to take many months, obviously; okay, joking, it'll take 2-3 days max. Perhaps, simply because we know more biographical information is needed for other apps we're going to write, we'll spend 1-2 weeks on it, but hopefully you get the point. The outcome will be...
- A usable biographical information service.
- A template of how to implement other service layers. They may or may not conform to the exact same template, but it's a good starting point.
- A running biographical information service that we can modify for future needs.
We start to discover that some services need birthdate, social insurance number/social security number (SIN/SSN), etc, etc. Maybe we start to realize that the service needs some access controls, so that our "Discovery" app can call it for first name, last name, email, and student ID, but another app can also obtain birthdate, or SIN/SSN. We now have a new need, so we begin to implement that.
Notice how this approach is "product" focused, rather than "project" focused. Projects tend to get crazy amounts of requirement, while being comprised of a bunch of guess work. Products tend to be about needs, which are usually founded in what the user has asked for. Humans tend to automatically be focused on reality when working at a product level, while focused on possibilities while working at a project level. Possibilities are not reality, they are possible outcomes. In our example we had concrete 100% absolute needs. We had these certainties because we were dealing with things at a micro-scale, rather macro-scale.
Summary
I find that many folks are afraid of a real agile mindset. Waterfall makes them feel safe. So, they implement a waterfall + agile approach, which isn't agile at all, but they call it agile. The only recommendation I would give to them is to "try it", and be determined to make it work. Frequently our fears are based entirely inside our mindsets, not on reality. In the case of agile they are based on faulty agile practices not working, so we try and mix waterfall and agile together, rather than working hard to "get it right". If you think about it, waterfall reinforces human bias, and agile (micro deployments) is the antidote.
One of the most fundamentally true quotes for architecture of any system, which has stood the test of time, is Gall's Law...
A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. - Gall's Law
By this we know that to "Go big or go home" is in essence "Go big and fail".