Product development is all about iteration over and refinement of ideas and eventually turning them into profiltable businesses.
A lot of us start machine learning projects by coding, and that is not the right place to start. Some of us, start from an idea that we think is brilliant, that’s definitely better, but not a good place to start either. The reality, however, is that most successful teams start with many ideas and go through a systematic process of eliminating bad ones, until they’re left with better ones.
Usually better ideas are the ones that involved solving a specific problem for a specific user, therefore getting that user’s feedback is extremely important.
The next step is to grab the better ideas, create mockups for them, and see how the users react to those. This will weed out another subset of ideas.
Then, out of the remaining ones, you want to create prototypes to answer a few questions like “how would the user react to a more realistic looking version of your idea”, “would the technical parts that are most risky work”, or importantly “would someone buy this” where you try to sell the most basic version of your idea and see if anyone buys it.
Again, after invalidating more of your bad ideas, you start coding the more promising ones, and that’s the real validation of your idea being technically feasible. Once you have the most basic version of your end to end software ready, you’re ready to start validating other product aspects, like how can you bring users to it, can they use it, would they continue using it, etc.
Finally you get to the point that the real big user/solution fit question comes up, do they want it enough to pay for it enough to justify the cost? In an ideal world, they even want it enough that you can have a profit and grow over years to come.
In order to ensure that you can move along your journey as fast possible, and with as little risk as possible, you have to make sure you spend just enough time on each stage. For example, you should be able to come up with lots of ideas in span of hours, and you should be able to invalidate some of them with a mockup in a day or two. And putting the alpha version of your product in the hands of user shouldn’t really take more than a month. This is extremely important because this journey is not as linear as I said, it’s very likely that you hit roadblocks in every stage that forces you to go back all the way to the drawing board where you have to come up with new ideas and restart the process all over again.
Maybe one of the the most famous examples of this is Slack, where they went all the way to the business stage, and then had to abandon everything and go back to coming with new ideas to pivot to.
To put some labels on all these stages, and be able to refer to them later, the first few stages are discovery parts, then we go through short term validation, and finally through long term validation.
In the case that you can actually do all this for your core product feature, and make it to the end, you’ll have to repeat it for all other product features you will be adding
As you can see, a lot of what we are doing in product development is various types of validation, so let’s establish a framework for it, after all the science in data science should mean something! One point of caution that i would like to make is that, what we are trying to do here is in fact “invalidation” of bad ideas. It is very dangerous if you think your task is to prove that your idea is good, because we’re all human and prone to confirmation bias. We see what we want to see. Your task is in fact to identify ideas that won’t work as early as possible to avoid any long term investment that leads into disappointment
The framework that I’m about to lay out is fairly rigorous and can be used for all stages, discovery, short term validation, and long term validation
The first and foremost is writing down what you want to do. It’s amazing how many times I have met people who are well into their projects but can’t still articulate what it is that they are doing
Next, you need to know what assumptions you are making. For that you can do some exercise that I will lay out in more detail shortly. Those exercises will help you articulate what it is you are doing, why to do it all, and why to do it now, and eventually how you think it will happen. This will easily tease out a lot of assumptions that you are making about what you are doing
Next you need to come up with a way to validate these. This usually involved doing some research, either googling or talking to your potential users. Hopefully, you’ll learn a lot about the domain you are working in through these, and invalidate some of your assumptions. In all likelihood, you end up repeating this a few times, every time equipped by learnings that will refine what you want to do, why, and how.
For the validation plan, often you need to design experiments and create artifacts for them. Artifacts can be things like low finesse mockups, to the basic version of your application, all the way to an engineered version of your application at different stages. And you use these artifacts to obtain, qualitative, quantitative, and large scale data about how and why users interact with your app.
This framework can be applied to validate your massive transformative purpose all the way down to what color a particular button has to be
The “so what” framework, although it might sound dismissive at first sight, but it’s one of the most illuminating frameworks that i frequently use. It can really be used everywhere, but most importantly I use it when I want to articulate what it is I’m doing and why. When I want to identify the assumptions I’m making about a problem or a solution. And when I want to understand fundamentally why I want to do something to see why that’s a priority.
The way it works is the following: 1. You write down what you want to do 2. You read that and ask yourself “so what”, and write down the answer 3. Then you read that answer, and ask yourself “so what” again, and then answer 4. And you repeat 3-5 times before you get to the most fundamental reason
Now a few things might happen, you might go through this with no problem, which either means that you are doing it wrong, or you really know the domain of your problem. More likely, you might really struggle to do this beyond the second or third iteration, and that’s a good sign: it means that you have identified things that you don’t know which you have to think about more, or research more Remember one of our principles, if you are stuck and can’t move forward, you have to research more, and mostly likely, you need to talk to your users for that. In all likelihood, once you get to the 5th iteration, where you started might not sound the most reasonable solution any more. So, you might have to go back and modify that and repeat the exercise until everything flows through nice and smoothly.
A very similar framework to “so what” is “why now”. Here your attempt is to map out your assumptions about the urgency of you work with the same pattern as we did previously. Very similar to what we did in "so what" exercise, we repeatedly get to the bottom of why we want to do something right now.
But let me make a few points: