The Symptom of Repeated Success

This is a post about something I have heard about happening from some distance. Organizations working with scrum or similar methodologies reach the conclusion that the methodology does not fit them. This is a hypothesis about why that tends to happen and a bit of reasoning around the problem.

As developers begin working with scrum or other agile methodologies it often appears as if they manage to achieve a productivity boost and gain features at a faster rate than they were used to. The teams really like this and find it to be a good practice. There is however one problem surfacing relatively often which present itself as a positive symptom of continuous and rapid success. However, behind this seemingly great facade hides a problem that seems eeringly common. The rapid increase in productivity and success is often a symptom of well... failure.

Now I'll have to explain, yay!

The problem is that the team is not achieving manageable failures. Every backlog item which comes from the product owner is successfully implemented and done within a sprint or two of work. Often each sprint contain several backlog items and the product takes on several new features in rapid succession. Now there is one of two possible states that you are in, one of these is more probable to be true and the other is likely closer to resembling a joyous pipe dream:

1: Awesome!!

• The production is going on rails and creating fully tested and reliable user value with every backlog item which is started. The Product Owner is a domain expert in the field, has incredible insights in design and production techniques and is well connected with the market.

2: Hmm, what did the user feedback say again? Does that really match our Definition of Done? Or what does it really say?

• The testing of user value does not inform the team about how much value they have been producing and they are developing features blindly.

If your team is experiencing this symptom of the repeated success you have a statistically troublesome scenario. The problem is that for the "Awesome!!" result to really occur you need to have one of the best Product Owners in the world. These guys are quite rare. You might also be part of an organization which is building an existing product within an existing market, this is the metaphorical equivalent of a Nuclear Power plant. The problem is known and the solution is known.

The user Values you develop are not known beforehand

It is quite likely that you are developing a product which aims for a niche market or a low-cost alternative within an existing market. In these cases your Product Owner will need to use the development team to run experiments against the market to find out how to handle the unknown factors. Be especially aware if you are making a new product in a new market, that means your hypothesis will be wrong almost all the time.

As any scientist will know you will not always validate all hypothesis as being true from their first experiments. This is where the failure of agile and scrum screams its loudest, at least from my perspective as Game Designer. I know that even the best of designers will only have a theoretical foundation which pre-validates a portion of the designs. A generous estimate says that you will be roughly correct maybe 70% of the time, but each feature contain several sub-hypothesis which in turn has about the same probability of success. This means that more than 20% of your Backlog will be dead wrong. And now I am being very nice about that number...

Features which are not valid user value is waste which is expensive to clean up

If you are part of an agile team which fails to invalidate the hypothesis within the Backlog be aware that you are most likely producing a huge design debt within the product you are working on. Not only are you failing to find the defects before they impact the user experience, but you are also failing to understand where the real user value actually reside and are thereby effectively torpedoing future design efforts aimed at cleaning up the debt.

Personally I would advice that the only reliable way to step out of this trap if you are sitting in it is to treat design debt just as you treat technical debt. Design Debt most commonly rears its ugly head as increased complexity within the user experience. Once there are many features of surmised value within the product which has been deployed it becomes quite difficult to separate the ones which have user value from the ones which don't. This is a type of debt which grows fast with compounding interests because the probability that any future backlog actually does anything good to the user value drops with the complexity of the already faulty design. This is well supported by how system complexity also increases exponentially with the number of components as neatly described by Metcalfe's Law.

The way out of Debt is slow and painful

You will need to slow down and begin to refactor the user value through effective measurement within the product. The more common attempt is to increase the volume of features which has the result of increasing the volume of produced waste.

I am not too sure about development teams which have had the ability to climb out of this type of mess. The most common exit strategy I hear people deploy is to package the failed product as "valuable experience" then starting over with a new vision, mostly to achieve the same experience next time. There are also other tricks of business magic which some have been able to perform where they can sell the work done to wealthy customers.

Conclusion

Beware of agile teams that do not report failures from the interpretation of relevant feedback from valid tests. These projects are quite likely to be creating delayed waste which have impact enough to be fatal to the project. When working on a particular backlog item ask yourself: "How would the statement that this feature is waste be defeated by a valid proof to the contrary?"

If the best answer you think you will get is a whole lot of opinions then you have a good reason to help improve the development process.

Feedback pipelines

Game production use the word ”pipeline” for the technology and behaviors needed for adding data to the end product. This is a great and useful component to the iterative process. There is a system which can be optimized for speeding up the time needed to change the data from one state to another. I have no real idea of how much of the budget for a large scale production goes into developing this pipeline. I guess it is significant and well motivated.

Now to the point of the argument, the investment made in the feedback pipeline is most likely much smaller for most game projects. This is not well motivated and I believe it surfaces as a tremendous cost, for successful project it means your staff needs to be incredibly competent and able to manage implicit iteration without excessive waste. For failed projects it is the one major cause for failure.

I knowingly use very strong words here because I want the notion challenged so I learn something useful from it.

User Value measuring is art, craft or science

Measuring user ualue can be considered to be more or less scientific. There are arguments for the scientific approach, but there are also arguments for an artistic approach especially when the values you need to measure are subjective.

There are some parts of user value you definately can measure with a scientific methodology, if you can do so it makes sense to use the available scientific methods. Most common entertainment productions will usually not have this ability readily available. Using an artistic approach towards measuring user value would in my book be the last resort, and whenever going down this route expect a lot of failures.

>skips past what a craft is.

The measuring of user value appears to best fit as a craft.

User Value, by looking at users

With a definition of design fully focused on creating user value it appears important to have a decent understanding of what user value is. The most important aspect is that it needs to be something you can measure.

The first and most useful method comes through looking at and listening to users, or testers who reasonably well represent users. This method is a bit ugly because it will not give you much hard data. The lack of hard data will inhibit the ability to communicate the result to anyone without insight into the actual test itself. People you need to communicate the result with will have to trust your interpretation of the behavior exhibited by the user during the test. You are already fully loaded with test analysis hardware for this particular type of test, use it. We’ll look at methods for scaling up this type of test later but for now think small.

You can start measuring the value of any design as soon as you can find a test subject. A good start may be to talk about the idea with someone. This is extremely cheap, but the test output is very unreliable. However if your design gets shot down by such a primitive test then go back and start over. Or adjust accordingly, possibly by replacing the test subject with another tester.

If your cheapest test is successful it might be time to invest a bit of effort into the design process. What kind of investment you will benefit from depends a lot on the idea itself and your own skills. At this point I tend to find myself needing to develop my own understanding of the idea which is getting developed so I sketch up a series of takes on the idea. If the idea is associated with a game type of product or if the idea is the whole game itself I tend to find a primitive sketch of interface components to be a decent start. This helps me get a better grip on what actions the user should be taking. The end result does not inherit any particular components from this simple sketch, but it gives me a better fundament for testing the idea on more test users.

If the idea survives this far and if it is simple enough it is time for a functional prototype, some paper prototyping is a good idea to start with. If the idea has a close relative implemented in an existing product you can look at altering that product to fit the idea or if you got plenty of time and access to the right skills develop a prototype in code. However most ideas are critically dependant on the value of other ideas, after you got the first idea to the point of where you can invest some time in it you are likely better off measuring the value of the other ideas which it depends on. Take your time and go through each idea until you have collected a firm picture of all the relevant ideas and their individual value before promising any end result to anyone.

Keep in mind that ideas are cheap, plentiful and most often bad. Avoid promising that an idea has value until after you have verified its value against a decent representative of users. By continously measuring the things you do against test users you increase your chances of designing successfully. When you fail to get a successful test from an effort you have produced waste.