Most organizations start their Agile-Lean journey with Software Development, typically using scrum. This makes sense for a lot of reasons:
- The lean software movement started in programming with the seminal work of the Poppendiecks (2003 Lean Software Development).
- Developers are likely IT organization’s early-adopters, eager to stay current in their chosen field.
- There is neither a great financial hurdle nor significant cultural change required to bring something like Scrum into the organization (assuming there is a person willing and able to be product owner).
Water-Scrum-Fall
Assembling a product owner, a scrum master, developers and testers to prioritize a backlog and start short iterative cycles may not pose any significant challenge, but many organizations have a culture and inertia that forces embedding scrum development into a siloed IT organization with functional departments executing some standard requirements phase, a test phase, a build phase and a release phase. This type of development, the industry has taken to derisively label Water-Scrum-Fall.
Dead End? Some may view this state a necessary development stage towards Lean IT, but I caution it more often becomes a dead end. To appreciate this I think it helps to look at the value stream of the Software Development Life Cycle.
In the figure I have deconstructed the value stream to separately highlight activities for Requirements, Development, Test, and Build & Release. Moreover, I have abstracted a lot of detail and most of the inevitable development/test cycling.
Note the value stream has extensive periods focused on requirements gathering, testing, build and release, in addition to the period of time devoted to development. If the goal is set to reduce the overall Lead Time from a request to customer availability – our attention should be on minimizing the Work In Process (WIP).
The value stream begins with activities concerning requirements gathering. Note that in addition to the value-added work of defining requirements, we represent non-value added activities such as the identification of extraneous wish list requirements (since everyone knows with waterfall you only get one chance to propose possible requirements). “Necessary” non-value added work some organizations mandate is to have an executive review the list of requirements (typically consuming lead time waiting on the executive to get around to it). For most organizations a project will not start prior to identifying most of the key resources, so we likely have developers assigned to this effort (possibly even charging to this project) waiting for the approved requirements while working on other tasks which will be dropped once the requirements are ready. A lengthy requirements phase necessarily extends our Lead Time.
Moreover, we likely have WIP building up with handoffs between functional departments. For example, if the organization has a separate testing function performing non-functional and UAT tests this creates WIP which likely will require additional development rework. When an organization is not performing Continuous Integration an extensive Integration Phase is required where all the developers scramble to get their feature branches into main. Once the team gets though integration there is generally a special build team for the staging environment with more testing performed. Finally the team can petition to get on the next Release Train which will move the code into production. This can add months to lead time, which only begs management to squeeze the developers to add a few more features which hopefully will be adequately tested.
Question: How does agile programming focused on increasing throughput of development impact overall Lead Time in water-scrum-fall?.
ANSWER: Agile Development in this context may be minimal. Looking at the current state map, a 25% reduction in Development will have a minimal impact on overall Lead Time if the remainder of the value stream is not changed. Moreover, if Test, Build and Release cannot keep up with increases in software throughput we may actually be making the situation worse. Any reduction in the development process time may end up building queues for downstream steps.
Lean IT strives to examine the overall Value Stream and identify constraints. Possibly in this example, the lengthy requirements phase and being constrained to quarterly releases have the greatest effect on the overall Lead Time.
The Fallacy? Although we think we are making good headway tackling the development phase using scrum, the constraints holding us back are actually elsewhere. The gains the development team makes will never be appreciated by the customer who only sees a slight incremental improvement in Lead Time and Project Cost. In fact, the organization may erroneously conclude that agile programming is a hoax since the end result is not materially better than the old way. The fallacy we make is thinking we can optimize any process in the value stream without identifying that we are optimizing the constraining process. This is of course the Theory of Constraints which any business or engineering student recognizes. When we eliminate the lead time and non-value work outside of development so that development approaches 65-70% of the total work then our customer will be impressed with a 25% improvement in software development.
Starting your Lean IT journey with agile programming is fine as long as you communicate to your organization that it represents just the first step. In doing so, clients that appreciate bigger improvements may await next steps.