Managing hypotheses with Mythbusters

Most design or innovation projects start off the same way: you have a lot of uncertainty about a lot of things and gradually you reduce this uncertainty - perhaps through research, testing or prototyping.

But how can you and the project team obtain a comprehensive overview of the assumptions that you are working with? And how do you communicate these to outside stakeholders? Introducing: Mythbusters.

Why bother?

Assumptions accompany product development at every stage, both implicit and explicit. Getting some of these assumptions wrong could mean the difference between success and failure for your product.

By creating a shared understanding of hypotheses within the entire team, everyone will gain a more comprehensive understanding of the direction of product development as well as user research. Therefore, there's no longer a need for internal debates on allocating resources, as everyone has a shared understanding of resource management; “Why the hell are we spending resources on doing this? Oh, that’s why.”

As a project evolves, so do the hypotheses, which is why it makes sense to have a living document that charts this evolution and can demonstrate the progress made in validating or disproving our assumptions.

Enter Mythbusters

The Mythbusters document is essentially a list of hypotheses that surround a particular project. There are many ways to generate hypothesis statements, so choose your favourite. Once you have your hypotheses, you can put them in a spreadsheet and give them each an identifying number:

Not all hypotheses are created equal - some are more critical to the outcome of a project than others - so it makes sense to also prioritise them. We usually prioritise them from 1 to 3, with 1 being highest priority and 3 being lowest, but you can use pretty much any scale you want. When doing this, you can try to ask the question “How bad would it be for the project if we get this hypothesis wrong?” - if the answer is “it would very probably kill the project”, then it’s a 1,  if it’s more like “it might mean this major feature does not really make sense any more” then it’s a 2, and if it’s “would be annoying, but it’s not really a huge deal at this stage” then it’s a 3.

OK, now that you have listed and prioritised your hypotheses, it’s time to start rating them based on how confident you are about whether they are true or false. You do this by creating 5 columns labelled:

• Definitely False
• Probably False
• Keep Digging
• Probably True
• Definitely True

Even before any major research has begun, the product team will probably have some thoughts about where on the spectrum each hypothesis lies. It can be a fun exercise to put an ‘X’ in one of the columns for each one based on the gut feeling of the team just to see how this changes as the discovery and validation phase progresses.

You should now have something that looks like this:

The next step is to set up other columns that might be relevant to the project. I find it’s often useful to have the possibility to add notes, which validation methods you intend to use and also links to research sources (like interview videos or transcripts in Dovetail or UserBit) - but you can add wherever makes most sense for your project - go nuts!

The final step is to could how many hypotheses you have in each stage by using the formula:

=COUNTA()

at the end of each column.

Once you have done all that, you might have something that looks like this:

Congratulations! You’ve just created the first version of your Mythbusters document.

Managing the Moving Hypotheses

As discovery progresses, the Mythbusters document should serve as a single source of truth for the team when deciding where to focus research and where there are still significant gaps in knowledge which lead to critical uncertainties.

Part of every research synthesis phase, such as collecting the findings from a round of user interviews, should also include the step of updating your Mythbusters document. If a hypothesis was addressed in the research (and honestly, at least some of them should have been) then their status should be updated. Did the research indicate that the hypothesis was true? Move the X one step to the right - if the indication is that it was false, move it to the left.

Leveraging this methodology, we start to get a more organic overview of how our knowledge and certainty is increasing.

In many cases, your research will lead to more questions and hypotheses - that’s great! Just add them to the list and keep the process going for them.

While many business stakeholders usually delight in spreadsheets, it often makes sense to present the ongoing results in a more visual way. This is something we are experimenting with here at MVP Factory and currently use something like this:

The hypotheses are sized by quantity and then moved along the axis (the numbers below refer to the IDs in the spreadsheet). Maintaining clear and open communication helps manage the Stakeholder Stress Gap.

The Stakeholder Stress Gap

Product Design is often a chaotic, non-linear process particularly in the early days of discovery and prototyping. Teams tend to limit the visibility into this chaos (intentionally or unintentionally) to outside stakeholders until there is greater certainty about outcomes which they can report with greater clarity and confidence.

These outside stakeholders can see that something is being done (after all, the team is working hard, and presumably charging for their time) - and presume that the progress is fairly linear in relation to time.

Thus, the design process feels different for outside stakeholder and leads to the Stakeholder Stress Gap:

(I will be writing in more detail about the Stakeholder Stress Gap in a future post, so make sure you subscribe to stay in the loop)

In order to reduce the Stress Gap, product teams can use regular check-ins using visual and concrete frameworks such as the Mythbuster to help adjust expectations and visibility into actual progress. And at the end of the day - reducing stress for everyone should always be a desirable design outcome.

Thanks to Mason Adair https://www.linkedin.com/in/mason-adair/ for showing me the basic Mythbusters concept and to Leland Peterson https://www.linkedin.com/in/leland-peterson-7630b19/ for introducing me to the Stakeholder Stress Gap.