The amount of data generated by people and machines increases exponentially every year. Cloud computing power to process data is more affordable and accessible than ever. As a result, machine learning technology has matured and become a key technology component in many digital products. Despite of the success and the hype around machine learning, the reality is that product management for machine learning is not simple. Bringing to market a feature or a product that relies on machine learning is more challenging than what it takes for traditional digital products. In fact, projects often fail because underestimate complexity and effort. Thus, strategies for machine learning products should weigh technical risk against other types of product risks such as value-creation, viability and usability.
This post presents 9 steps to do product management for machine learning. In particular, it focuses on how to bring to market products that use machine learning, so that technical risk is properly accounted against other types of risk. As a word of caution, this post does not constitute an management advice. Every product is different and different teams can bring products to market with a different approach. However, the steps described in this post can be considered as a checklist of factors to consider, among others, when bringing machine learning products to market.
What is Machine Learning
Before describing product management for machine learning, let us first understand what machine learning is with some application examples. Machine learning consists of developing and using decision-making systems that learn patterns from data. Due to its data-driven nature, machine learning is quite versatile and has different applications. For example, just to name a few:
- Computer vision to recognize objects from low quality photos or videos
- Chatbots to support customers in-real-time and anytime
- Recommender systems to personalize customer experience
- Predictive analytics to streamline asset maintenance
The key ingredient for a machine learning system is domain-specific data. In fact, as a general rule, the more the data the better a machine learning algorithm can outperform any other programmed algorithm.
The 9 steps to bring machine learning products to market
The 9 steps are the following.
- Identify target customer jobs
- Validate customer jobs
- Define the business case and address ethical concerns
- Establish a performance objective
- Test for feasibility
- Launch a minimum viable product
- Pivot or persevere
- Develop the machine learning model
- Launch the product and improve iteratively
Let us look at them one by one.
1. Identify target customers jobs
The first aspect to consider in product management for machine learning is to start with the customer. In fact, identifying the target customer is the first step to launch any product feature. This comes down to answering 4 questions:
- “who are the customers?”
- “what pains do customers have?”
- “what gains do they aspire to achieve?”
- “what jobs do customers perform?”
For value propositions that use machine learning, the “jobs” question is particularly important. This is because machine learning mainly aims to substitute and outperform humans in executing some jobs. Here some examples of jobs that machine learning usually help with:
- Discarded jobs. Jobs that users avoid just because they do not like to perform them. Some people, for example, do not like to search and choose the right investment for their savings. So, they either postpone it, rely on advise, or go with whatever their institution offer as a default option.
- Costly jobs. Jobs that people do not mind to do, but they cost too much or take too long to perform. For example, searching products or holiday destination online is not necessarily annoying, but it may take time.
- Undesirable duties. Jobs that people cannot avoid, because someone has to perform them, although people would prefer to do something else. For example, some types of customer support activities are tedious and repetitive. So, people would not mind to delegate them to an intelligent machine.
- Inaccessible jobs. Jobs that people would like to do but are not capable of. As an example, consider the job of recognizing defects in manufacturing. Defects are usually detected offline during the final production stage. Recognizing them online, directly on the production line, would mean quicker production and savings on resources.
- Expert jobs. Jobs that require some kind of expertise, knowledge or tooling that are scarce in a given context. A typical “expert” job is early diagnosis of diseases in geographically remote areas, where medical skills are not immediately available. Without early diagnosis, in these areas health risk drastically increases.
2. Validate customer jobs
Once target customers and their jobs are identified, it is not time to start developing any machine learning model yet. It is rather important to validate customer jobs with actual users in the target market to ensure that any model really adds value to customers.
Customer interviews are usually the best way to get fast and early insight on customers’ problems and wishes. Beyond confirming or disproving the existence if customer jobs, pains and gains, customer feedback can tell a lot about what other jobs are most important and possibly underserved for that customer. Ultimately, this will inform the product team on how good any machine-learning model should be or if it is even necessary.
The question this customer validation step should answer are:
- “Does the job we want to support really exist?”
- “Are target customers unsatisfied with the job we want to support?”
- “How can our product address the needs of our customers?”
- “If we address the needs, will customers buy our solution?”
- “What else do our customer need to get the value they do not currently get?”
Beyond customers interviews there are other methods that, depending on the customer representatives or time and resources available, can help to test a value proposition. Another method is to mock up the product feature’s front-end. This method is often called “fake-door testing” and was described in a previous post.
3. Define the business case and address ethical concerns
Regardless of what industry the product will target, or whether the company is a for-profit or no-profit, a business case will address the question on whether the product feature will have any return on investment. So, knowing the size of the market you are targeting with the new feature or product is crucial. At this stage the questions to answer are:
- “How many customers there are with the profile we have identified?”
- “How often are these customers unsatisfied and feel the need of a solution like ours?”
- “What price are these customers willing to pay for our solution?”
- “How much will it cost to serve this customers with our solution?”
These are very complex questions to answer. At this stage, there is no need to get much detail. Ballpark estimates are sufficient. For details on how to quantify the business case, this previous post should help.
Beyond the business case, there is also a qualitative, yet important question to answer at an early stage:
- “Should we build it?”
Ethical questions particularly apply to high-tech products such as those using machine learning. Indeed, as machine learning (and artificial intelligence in general) is getting used for higher impact decisions, ethical implications are more relevant. For example, a crucial aspect with machine learning is data bias, as a machine-learning algorithm is as smart as the data that trained it. Thus, if training data contains bias or prejudice, the algorithm will produces results that are systemically biased, producing results that can negatively impact some people. So, evaluating how to avoid bias early enough is crucial to make sure all the right precautions, such as proper data sampling, are taken early enough. In the worst case, the outcome might be that it is not possible to produce an unbiased algorithm.
4. Establish a performance objective
Once the business case is defined and ethical concerns addressed, the next step is to set a few critical business objectives and related machine-learning technical performance targets.
A business objective allows to identify what business performance a product or product feature should reach in order to be successful. For example, let us consider a chat-bot product to help patients find answers to medical questions, before reaching out to the family doctor. Although healthcare chatbots are an advanced application of machine learning, they are certainly not science fiction. For such a kind of chatbots, a business objective can be the number of cases correctly diagnosed or correctly identified as in need to see a doctor. In other words, the more cases the algorithm gets right, the better the algorithm is. The team could choose 95% as a target, since it is comparable with the rate of diagnosis performance experienced in outpatient settings in US, according to this study.
5. Test for feasibility
Once performance objective and capabilities are out of the way, how do we know it will be possible to build a model with the required performance? Indeed, before showing any solution or mockup to potential customers, ensuring that the core component of the product, the machine learning algorithm, works, is crucial. After all, developing a machine learning algorithm is not just traditional software. To check for feasibility, creating a proof of concept is one way. However, while it is sufficient to train a machine learning model on low-cost, easily-accessible data, when it comes to tuning the model and testing their actual performance, it is important to do it using data similar to what the model will see in production.
For example, let us say a product wants to identify food plates from pictures taken by smart phones. The product team can easily find images of plates with ingredients from the web in order to train an algorithm quickly. This is a great way to prove that it is possible to identify ingredients from food images. However, this does not prove that a product can identify ingredients from pictures of real users. The difference is subtle but important. Indeed, food images on the web mostly come from professional photographers. This means, that by training the model on such data, the algorithm becomes biased towards this kind of “high quality” pictures. Finding such “low quality” images tagged with the names of the ingredients or at least the name of the food might take time and resources. However, t is crucial to test that the product will work well with real data.
6. Launch a minimum viable product
Once you know the problem exists and you have proven the product feasibility, it is time to set aside technical feasibility and make a step forward towards customer value creation and usability. Indeed, before creating any actual product, validating that customers like the product and would buy it is crucial. Validation can consist of building an MVP (minimum viable product) or introducing an MVF (minimum viable feature) if the product already exists.
When it comes to machine learning, the “wizard of Oz” is a great strategy for executing an MVP or MVF. Strangely enough, the machine-learning algorithm is superfluous at this stage. All that is needed is real people in the back of the product doing the job that the machine-learning algorithm would do. Although using humans instead of machine learning sounds inefficient, it is a cost-effective way to quickly bring the product to market without building the most expensive and resource-angry part of it, the machine learning model.
7. Pivot or persevere
One of the possible results from the MVP can be that the product is not attractive enough. In this case, it is important to pivot to some other product by retaining as much as possible of what discovered so far. The need for pivoting can be hard to accept but is essential to avoid letting the product run on a dead end track. As mentioned in the previous post, when we invest money and resources on a project it is hard to shut it down when the costs outweigh the benefits.
If after a couple of pivots, there is finally a positive feedback from the MVP, congratulations! Now it is time to start setting up the product for your platform. Before coding the machine learning algorithm, it is important to avoid the “not invented here” syndrome. In other words, you do not want to rebuild components that can be acquired with little or no investment, unless you can create a competitive advantage out of them. Indeed, there are many open-source or licensable machine learning models that can speed up time-to-market because they work out-of-the box on general tasks, such as, for example, detecting people on images. They can then be trained on limited and specific data-sets in order to perform particular applications, such as recognizing happy people. Moreover, the cloud offers all the storage and computing power needed to train and test machine learning models without incurring capital investments.
8. Develop the machine learning model
When working on the machine learning model, it is tempting to get as much clean data as possible before starting to train and test any model. However, there are many possibilities to improve a machine learning model. Getting more data is only one way and sometimes it may be very expensive. Trying different algorithms or analyzing the data to get some useful clues on how to transform the data are other, sometimes cheaper, ways to do it.
In order to improve technical performance, having a feedback loop that improves the model on the basis of the previous iteration’s results is key to product development, especially in machine learning. Every loop should have an error analysis phase, where any error found during testing is analyzed so that the model can be improved. When a certain plateau is reached in terms of performance, gaining another bit of improvement may not be worth the amount of effort spent in getting more data on improving the algorithm. So, it is important to consider how the benefits-over-costs ratio changes while performance gets better.
Inspecting and understanding what type of errors the algorithm is making at each loop can be tedious, but gives much better results and less waste of resources than trying to improve a machine learning model just blindly.
Although more data is generally better for a machine learning algorithm, there is a point where there is diminishing return to data. In other words, by increasing data, the performance increases much less than at lower amounts of data.
9. Launch the product and improve iteratively
Knowing when a post-MVP product is ready for launch is almost an art. When doing product management for machine learning, there is not exception. Engineers will likely try to bring to production the best performing machine learning algorithm possible. On the other hand, sales and marketing will want to reach market as soon as possible. The role of product management in this case is to decide when “enough is enough”. In other words, the product manager should decide and sometimes reach consensus, on when the product is ready for launch, considering the constraints from the different stakeholders.
Regardless of when the product is launched, continuous improvement is crucial. When the product is live, there are many reasons why performance can degrade. The model will se degrade performance when in production, simply because seeing new data that the algorithm has never seen before will possibly cause your algorithm to perform worse. This can be for many reasons, from data quality to a different context than seen during development, or simply by evolving user preferences. So, it is important to monitor performance and trigger relevant changes of performance while in production.
Conclusion
There is no theory on how to bring machine learning products to the market. Product management for machine learning is indeed an iterative process that requires equal attention on the side of value creation and technical risk management.
To share your experience or any questions regarding product management for machine learning, feel free to comment below.