The Symptom of Repeated Success

This is a post about something I have heard about happening from some distance. Organizations working with scrum or similar methodologies reach the conclusion that the methodology does not fit them. This is a hypothesis about why that tends to happen and a bit of reasoning around the problem.

As developers begin working with scrum or other agile methodologies it often appears as if they manage to achieve a productivity boost and gain features at a faster rate than they were used to. The teams really like this and find it to be a good practice. There is however one problem surfacing relatively often which present itself as a positive symptom of continuous and rapid success. However, behind this seemingly great facade hides a problem that seems eeringly common. The rapid increase in productivity and success is often a symptom of well... failure.

Now I'll have to explain, yay!

The problem is that the team is not achieving manageable failures. Every backlog item which comes from the product owner is successfully implemented and done within a sprint or two of work. Often each sprint contain several backlog items and the product takes on several new features in rapid succession. Now there is one of two possible states that you are in, one of these is more probable to be true and the other is likely closer to resembling a joyous pipe dream:

1: Awesome!!

• The production is going on rails and creating fully tested and reliable user value with every backlog item which is started. The Product Owner is a domain expert in the field, has incredible insights in design and production techniques and is well connected with the market.

2: Hmm, what did the user feedback say again? Does that really match our Definition of Done? Or what does it really say?

• The testing of user value does not inform the team about how much value they have been producing and they are developing features blindly.

If your team is experiencing this symptom of the repeated success you have a statistically troublesome scenario. The problem is that for the "Awesome!!" result to really occur you need to have one of the best Product Owners in the world. These guys are quite rare. You might also be part of an organization which is building an existing product within an existing market, this is the metaphorical equivalent of a Nuclear Power plant. The problem is known and the solution is known.

The user Values you develop are not known beforehand

It is quite likely that you are developing a product which aims for a niche market or a low-cost alternative within an existing market. In these cases your Product Owner will need to use the development team to run experiments against the market to find out how to handle the unknown factors. Be especially aware if you are making a new product in a new market, that means your hypothesis will be wrong almost all the time.

As any scientist will know you will not always validate all hypothesis as being true from their first experiments. This is where the failure of agile and scrum screams its loudest, at least from my perspective as Game Designer. I know that even the best of designers will only have a theoretical foundation which pre-validates a portion of the designs. A generous estimate says that you will be roughly correct maybe 70% of the time, but each feature contain several sub-hypothesis which in turn has about the same probability of success. This means that more than 20% of your Backlog will be dead wrong. And now I am being very nice about that number...

Features which are not valid user value is waste which is expensive to clean up

If you are part of an agile team which fails to invalidate the hypothesis within the Backlog be aware that you are most likely producing a huge design debt within the product you are working on. Not only are you failing to find the defects before they impact the user experience, but you are also failing to understand where the real user value actually reside and are thereby effectively torpedoing future design efforts aimed at cleaning up the debt.

Personally I would advice that the only reliable way to step out of this trap if you are sitting in it is to treat design debt just as you treat technical debt. Design Debt most commonly rears its ugly head as increased complexity within the user experience. Once there are many features of surmised value within the product which has been deployed it becomes quite difficult to separate the ones which have user value from the ones which don't. This is a type of debt which grows fast with compounding interests because the probability that any future backlog actually does anything good to the user value drops with the complexity of the already faulty design. This is well supported by how system complexity also increases exponentially with the number of components as neatly described by Metcalfe's Law.

The way out of Debt is slow and painful

You will need to slow down and begin to refactor the user value through effective measurement within the product. The more common attempt is to increase the volume of features which has the result of increasing the volume of produced waste.

I am not too sure about development teams which have had the ability to climb out of this type of mess. The most common exit strategy I hear people deploy is to package the failed product as "valuable experience" then starting over with a new vision, mostly to achieve the same experience next time. There are also other tricks of business magic which some have been able to perform where they can sell the work done to wealthy customers.

Conclusion

Beware of agile teams that do not report failures from the interpretation of relevant feedback from valid tests. These projects are quite likely to be creating delayed waste which have impact enough to be fatal to the project. When working on a particular backlog item ask yourself: "How would the statement that this feature is waste be defeated by a valid proof to the contrary?"

If the best answer you think you will get is a whole lot of opinions then you have a good reason to help improve the development process.

Dangers of iterative production techniques, part 1 of n

As I mentioned in a previous post http://gamesartdesign.blogspot.com/2009/03/iterating-towards-value.html there are, from my personal perspective, several ways to cut an iterative process. This post is about general risks with any of these, and likely other, cuts.

Risk 1: Feedback conditioning

This is a risk which occurs whenever you keep getting similar feedback through a series of iterations. The shape it takes is much like what happens to us humans when we get accustomed to living close to a paper manufacturing plant which stinks of sulfurous chemicals. Our brains get used to the initially nasty experience and we get used to it. A game development team which gets familiar with some feedback and determine that it is part of reality will make a worse product than they could.

Risk 2: Reliance on implicit feedback loops

When the measurement is implicit as with the case for how the subjective components of the product are experienced it is quite likely that the team will fail to understand when they reach the top of the value s-curve and keep on investing in things which are done. For us who spend our time with the modern art form of game design this is horribly difficult to communicate. I also believe it is more difficult for the other art forms in game development as well than what it is for the older more established media. How do you know when you should make a new animation rather than keep on iterating the current one?

Risk 3: Lack of direction

This is the most often mentioned risk which comes from a combination of lacking vision or the communication of a vision which is hard for the team and business to understand. As a consequence of a poorly communicated vision the team lacks a foundation for understanding even explicit feedback loops and will progress erratically.

Risk 4: Going in circles

When the team has too little room to maneuver they rick failure to understand how to change the next experiment in the iterative process to improve the feedback state of a production. This comes appear to often come as a consequence to another failure which cascades down as an unmanageable cost of an effective cycle time. If changes to the product reach a certain volume the next change will be driven by risk aversion, risk elimination is likely to continuously surface with the same solution over and over.


Do you think this list should be extended, feel welcome to add your own risks and I’ll try bake them into part 2. ^^