Knowing when to launch: what is the balance between research vs release-and-learn?

You could spend until eternity validating an offering or feature. But there is a tipping point where this becomes detrimental. So what is the balance between building confidence vs getting something out there quickly? I was asked to run one of MOO’s Research Guild sessions on this topic while I was on the team and this is an extended summary. I know there’s loads of info out there on this topic, from people more qualified than me! Would love to hear your thoughts.

Examples of failures

  • No adoption / purchase

Keep in mind there’s a difference between:

  1. Creating something, it not working, learning fast and then able to make changes to respond.
Image for post
Image for post
The product team and I monitoring some key metrics following a feature release. I joke… Creative commons image:

Potential causes of failure

  • Not solving the right thing or just no need

If research was done:

  • Not tested enough

Here’s an interesting analysis of why Starbucks failed in Australia:

Costs of failure

  • Cost of building

But there’s also a cost in launching something too late:

  • Cost of building

There are more serious forms of failure

Which I’ve not focussed on in this write-up. Many of us have the peace of mind of working on products where failure doesn’t impose a safety risk on people. The high-risk end of the spectrum includes emergency and medical products: the cost of failure is too high.

The 2002 Überlingen mid-air collision was a huge tragedy. New anti-collision technology was introduced and it worked accurately in a technical sense. It didn’t work in its broader context, due to a number of factors (including conflicting orders from the technology vs air traffic control, training, workload of air traffic control staff) causing two planes to collide with each other.

I’m assuming the software itself was tested to a very high grade. But the error was not with the software’s accuracy. Could more research within the broader context have helped with this to recommend more training prior to the crash?

Building confidence and lowering risk

Image for post
Image for post
PRO TIP: Add in memes and gifs to your article to identify yourself as a millennial, dilute the seriousness of your topic and contribute further to our shortening attention spans. Did I lose you?

Everything we do prior to launch is a way to build confidence and some of these activities can only build so much confidence e.g. user testing is a great way to make sure things are working right, but it won’t be the same as releasing a feature live. It also tells you about usability, but won’t tell you about demand. Hopefully you’re amongst a culture that understands that building confidence is not the same as a contract that promises 100% success.

So each of these can build different types of confidence (and you need to be clear on what each is for) but at some point, you gotta put it out there to really see how it does (or kill it if you do have enough confidence it’s not working).

Building confidence:
More appropriate for new propositions / new offerings

  • Talk to customers you want, not just those you have

Building confidence:
More appropriate for new features or products once proposition is validated

  • Knowing before launch what success and failure measures are e.g. thresholds on A/B test, adoption metrics, etc.
“In the complex domain, cause and effect are only correlated in retrospect, and we cannot predict outcomes. We can see them and understand them in retrospect, but the complex domain is the domain of discovery and innovation. Expect the unexpected! Because of this, whatever we do in this space must be safe-to-fail. In this talk, we look at some different thinking tools which can help us to create these experiments, or probes… and to help us spot when we’re not doing so and probably should!”
  • Are your stakeholders on board with the agile process so that if you do learn and need to respond to that, they understand this is what sometimes happens with the leaner approach and that you can’t necessarily just move onto another feature? This is a whole other topic in itself about organisational culture and team structure…

It’s all a balance

Risks of releasing too quickly:

  • Teething issues

Risks of not releasing quickly enough:

  • Falling behind market

You’re never going to have 100% confidence when you launch.

What if you experience a massive fail?

I’m not experienced in this area, I’ve not been through or been accountable for a large-scale business failure. Unfortunately the times when it’s hardest to face what’s happened are when it is most important to face what’s happened.

This probably doesn’t happen as often due to organisational culture, politics, sensitivity and people leaving (knowledge lost).

More info

  • There’s a great organisation called Fuck up Nights that celebrates failure of any kind (I think the lovely Jane Austin introduced me to their work). They have meetups where people share failure stories and a free to download Fuckup Book that encourages you to fail, to face the fear and learn e.g. try and get fired from your job.

In illuminating and instructive business examples, you’ll see organizations with distinctively new operating principles: shifting from managing outputs to what the authors call “outcome-focused management”; forming self-guided teams that can read and react to a fast-changing environment; creating a learning-all-the-time culture that can understand and respond to new customer behaviors and the data they generate; and finally, developing in everyone at the company the new universal skills of customer listening, assessment, and response.

In summary, some things to think about are…

  • Risks in releasing too early vs too late

IRONY: Am I ready to publish? Not 100% ready but realised I am ready enough that I want to get this out there and see what feedback I get. My release plan for an article is normally: get feedback from people kind enough to read my draft via the draft link, then soft-launch the article and then if I feel ok about it share on social media.

What have I missed? What experiences have you had? It’d be great to hear from you.

Head of Design & Research at smava. Based in Berlin. Previously in Sydney & London.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store