Tuesday, October 20, 2009

The Symptom of Repeated Success

This is a post about something I have heard about happening from some distance. Organizations working with scrum or similar methodologies reach the conclusion that the methodology does not fit them. This is a hypothesis about why that tends to happen and a bit of reasoning around the problem.

As developers begin working with scrum or other agile methodologies it often appears as if they manage to achieve a productivity boost and gain features at a faster rate than they were used to. The teams really like this and find it to be a good practice. There is however one problem surfacing relatively often which present itself as a positive symptom of continuous and rapid success. However, behind this seemingly great facade hides a problem that seems eeringly common. The rapid increase in productivity and success is often a symptom of well... failure.
Now I'll have to explain, yay!
The problem is that the team is not achieving manageable failures. Every backlog item which comes from the product owner is successfully implemented and done within a sprint or two of work. Often each sprint contain several backlog items and the product takes on several new features in rapid succession. Now there is one of two possible states that you are in, one of these is more probable to be true and the other is likely closer to resembling a joyous pipe dream:
1: Awesome!!
  • The production is going on rails and creating fully tested and reliable user value with every backlog item which is started. The Product Owner is a domain expert in the field, has incredible insights in design and production techniques and is well connected with the market.
2: Hmm, what did the user feedback say again? Does that really match our Definition of Done? Or what does it really say?
  • The testing of user value does not inform the team about how much value they have been producing and they are developing features blindly.
If your team is experiencing this symptom of the repeated success you have a statistically troublesome scenario. The problem is that for the "Awesome!!" result to really occur you need to have one of the best Product Owners in the world. These guys are quite rare. You might also be part of an organization which is building an existing product within an existing market, this is the metaphorical equivalent of a Nuclear Power plant. The problem is known and the solution is known.
The user Values you develop are not known beforehand
It is quite likely that you are developing a product which aims for a niche market or a low-cost alternative within an existing market. In these cases your Product Owner will need to use the development team to run experiments against the market to find out how to handle the unknown factors. Be especially aware if you are making a new product in a new market, that means your hypothesis will be wrong almost all the time.
As any scientist will know you will not always validate all hypothesis as being true from their first experiments. This is where the failure of agile and scrum screams its loudest, at least from my perspective as Game Designer. I know that even the best of designers will only have a theoretical foundation which pre-validates a portion of the designs. A generous estimate says that you will be roughly correct maybe 70% of the time, but each feature contain several sub-hypothesis which in turn has about the same probability of success. This means that more than 20% of your Backlog will be dead wrong. And now I am being very nice about that number...
Features which are not valid user value is waste which is expensive to clean up
If you are part of an agile team which fails to invalidate the hypothesis within the Backlog be aware that you are most likely producing a huge design debt within the product you are working on. Not only are you failing to find the defects before they impact the user experience, but you are also failing to understand where the real user value actually reside and are thereby effectively torpedoing future design efforts aimed at cleaning up the debt.
Personally I would advice that the only reliable way to step out of this trap if you are sitting in it is to treat design debt just as you treat technical debt. Design Debt most commonly rears its ugly head as increased complexity within the user experience. Once there are many features of surmised value within the product which has been deployed it becomes quite difficult to separate the ones which have user value from the ones which don't. This is a type of debt which grows fast with compounding interests because the probability that any future backlog actually does anything good to the user value drops with the complexity of the already faulty design. This is well supported by how system complexity also increases exponentially with the number of components as neatly described by Metcalfe's Law.
The way out of Debt is slow and painful
You will need to slow down and begin to refactor the user value through effective measurement within the product. The more common attempt is to increase the volume of features which has the result of increasing the volume of produced waste.
I am not too sure about development teams which have had the ability to climb out of this type of mess. The most common exit strategy I hear people deploy is to package the failed product as "valuable experience" then starting over with a new vision, mostly to achieve the same experience next time. There are also other tricks of business magic which some have been able to perform where they can sell the work done to wealthy customers.
Conclusion
Beware of agile teams that do not report failures from the interpretation of relevant feedback from valid tests. These projects are quite likely to be creating delayed waste which have impact enough to be fatal to the project. When working on a particular backlog item ask yourself: "How would the statement that this feature is waste be defeated by a valid proof to the contrary?"
If the best answer you think you will get is a whole lot of opinions then you have a good reason to help improve the development process.

No comments:

Post a Comment