Covers: theory of Long-term Product Validation
Estimated time needed to finish: 10 minutes
Questions this item addresses:
  • How do you transition your best ideas to a product that can be used long-term?
How to use this item?

After your alpha release hopefully you are getting some feedback that should inform your next steps. At this point there are 2 possible outcomes, either some of your fundamental assumptions are being challenged in which case you might end up with a pivot, or if you are lucky and have done your homework right up to this point, and only some of your more minor assumptions are being challenged in which case you would just go ahead and improve things to prepare for the long-term use of your product.

Product Pivots

You do a pivot when you were pretty much completely wrong about a fundamental aspect of your problem, or maybe based on the initial feedback all of a sudden you identify a huge opportunity in a different aspect of the problem statement you are working on. More concretely, a pivot is a change in a specific but important aspect of your strategy to get you closer to your vision.

You should always pivot to a better idea, rather than away from a bad idea. In other words, it must be based on data and rigorous observations, and learning, and not frustration. To put it more bluntly, doing a random pivot is a good way of not getting anywhere, but a carefully calculated and well informed pivot is the foundation of successful product development. In fact, since strategy is just a vehicle to help you get to your vision, it’s very common and expected to have a few pivots along the way.

Now, let’s say you have done any pivots you need, and finally are getting favorable feedback, and ready to move closer to your beta release. In that case you have moved on to the second scenario I mentioned earlier.

Just to remind you, with all learnings, iterations, pivots, leading up to your alpha release and after that you are at a point that, hopefully, you are solving the users’ problem. This might not be the best solution to the users’ problems, but at least you have laid the foundations for rapid improvements As you move on ahead for a long term validation and use of your solution, there are many more aspects that you have to start taking into account beyond your bare minimum pipeline.

Data Related Improvements

For the starter, you have to think about the data you are using and your data collection practices.

  • Does your data meet your quality requirements for a long term solution? do you have a good way in place to monitor your data quality over time?
  • Are you collecting and using the data in an ethical and secure way, and is the privacy of your user respected?
  • What license is the data you are using released under? What implication does that have for your use?
  • Finally, in today’s world, data is a big part of your competitive advantage. Are you collecting data in a way that will set you apart from your competition? If you are using open source or purchased data, does your competition have access to the same data as you? How can you make sure your data is unique and will set you up for future success?

Model Related Improvemnts

Next moving on to your model:

  • Is there a justification for using more complex models? Do you expect a significant gain in performance and user experience?
  • Most probably you didn’t develop the algorithm you are using, make sure the package or api you are using allows you to use it the way you are from a licencing and intellectual property point of view. Whether your use is commercial, or open source, check out the terms of use of the corresponding license
  • Does the typical use cases for your application require interpretability of the results? Does your model support that? What post-hoc model interpretation techniques can you use?
  • What is the risk associated with using this particular model? How can that be quantified? How can that be mitigated?

Building and Packaging Related Improvements

  • Are you using the appropriate and sufficient testing to ensure that your whole pipeline doesn't break every time you change something?
  • If you need to use multiple models in production, say one model for each customer segment, or say one model for high level prediction and one for fine grained predictions, how are you managing those models?
  • If you have multiple container images for different scenarios, how are all those managed?

Deployment Related Improvements

  • If your application requires multiple models, how do you switch models back and forth between the production env and your model management solution?
  • Are you sure your deployed solution doesn't leave you vulnerable to attacks from bad actors?
  • Does your service need to be available 24/7? Can it handle spikes in usage? How often does it need to handle those?
  • Do you need multiple access levels? How are those set up?
  • How do you monitor and adjust the performance of your solution from software and model points of view post deployment and on an ongoing basis?

Delivery Related Improvements

  • If your solution has a UI, is the user experience and journey optimized for your target metrics like usage or sales? Is the user experience as simple as it can be?
  • If your solution required integration into other services and technologies, what is required for the integration? Do you have your APIs documented? Do you have your data transfer contracts between all services negotiated and documented?
  • If your solution sits in a larger business context, what’s expected of your solution to integrate into the business workflow? Is the user expected to behave differently as a result of the introduction of your solution? Is that communicated and properly planned? Do they need to be trained to use your solution?

User Feedback Collection

  • Are you collecting all the implicit and explicit user feedback that you need to improve the current feature and enable future features?
Fail to play? Open the link directly:
Author(s) / creator(s) / reference(s)
Amir Feizpour
0 comment

Preparing your Product for Long-term Operations and Use

Total time needed: ~10 minutes
This recipe goes through necessary steps for refactoring and setting up your code for long term use, and integrating into necessary tools and platforms for long term operations
Potential Use Cases
Product Operations and General Availability
Who is This For ?
INTERMEDIATEProduct Owners, Data Scientists, Machine Learning Engineers
Click on each of the following annotated items to see details.
Resource Asset2/2
VIDEO 1. Long-term Validation
  • How do you transition your best ideas to a product that can be used long-term?
10 minutes
RECIPE 2. What Is MLOps?
10 minutes

Concepts Covered

0 comment