Currently, only the tech giants are making the most from AI solutions. Plenty of other companies still need to demonstrate the value of their AI investment. All the costs for the cloud services, licenses, finding and retaining the best talent, and time to develop and launch the solution. And what does that x% improvement means to the organisation?

TL;DR;

  • Calculate your costs. Add all the cloud resources and everything that is involved in the project.
  • Start the project with business metrics and map it to the model target before development.
  • Estimate the return on unit percentage improvement.
  • Agree on acceptance criteria. (SLA: service level agreement)

Here, I’ll present my views on the topic, underlining the potential reasons and sharing a strategy to address this problem. But first, let me introduce two terms, business objective and model objective. The business objective is a metric that a business wants to improve, while the model objective is the target of any AI system. And usually, these are different. There is typically an assumption that links these two metrics, and the devil is in the details.

Based on my experience, the following factors significantly limit the value estimation of AI solutions.

  • Isolated development of projects away from the business goal.
  • Missing benchmark, as most business processes were never evaluated.

The first part is more challenging as it requires a change in perspective from the leadership. While the second is more of an operational effort. However, there is a more straightforward approach to addressing these two aspects collectively. I define the following two strategies for developing any AI model.

  1. Bottom-up approach: We start with model development with a target, and when it reaches some maturity, we try to map the model target to the business. Usually, this mapping happens during the experimentation stage. (the most common phrase you’ll hear is, “that’s not what we want”)
  2. Top-down approach: We break down the business goal into measurable units and map which AI solutions can help achieve these results. (easier said than done as it requires extensive experience in the business domain with diverse knowledge of the AI models)

We work with an example from the retail industry for a customer-facing problem.

Recommendation example from business objective

The image above illustrates the intent of products being “relevant and easy to find”. Let’s take the top-down approach and break this into measurable goals of increased engagement and reduced returns (try to answer what will happen if we reach this goal?). An engagement improvement can further minimise churn and boost loyalty and profitable segments. While reduced return will help save on transportation and storage costs and lower the environmental impact.

Here we can employ recommendation techniques to achieve different smaller targets. (the solutions enable a prioritised idea, eliminating the isolated development scenario). In this case, one of the model objectives could be CTR (click-through rate) or view-to-bucket time. After some development and evaluation, we can improve CTR over historical values and mark it a success for the model. But more effort is needed to provide acceptable business value because the end goal is not CTR but the assumption that it will lead to more sales (for example).

Furthermore, we need to understand the impact of an AI solution on the complete value chain comprising loyalty, churn, and profitable segments. Hence, a recommendation project (or any other complex AI product) should measure the ripple effects to estimate its contribution. And this can serve as the business value (but here is the devil). Typically, these effects include other factors such as loyalty programs, offers and discounts, making it harder to estimate the role of one single model. Therefore, we need to clearly state the assumptions, such that engagement positively correlates with loyalty and profit segments (not hard to imagine). We can observe these effects via experimentation but should guarantee that benchmark estimates exist before the start.

Integrated Approach

The above image demonstrates the top-down approach directly coupling the model and business objectives. With this technique, we secure the model target to reflect the business need and quantify smaller goals collectively contributing to the product value. But why is the value estimation needed? Didn’t we already establish that recommendation is necessary? Yes, we did, but only for the business that has proved it made a profit using these AI solutions.

Every company has a unique customer base and product offerings. There are industries where most customer engagement comes from a handful of products. So do we need an extensive AI team to work on recommendations? We can only answer this by estimating the value of AI products. Value estimation is imperative for all existing companies with the AI function or new businesses looking to start their AI journey.

Since you are still here, let me bring two crucial points to your attention.

  1. Let the ROI (return on investment) be the north star to guide all your AI investments. It will help you prioritise the correct use cases and assist in answering the “build vs buy” question.
  2. Follow an integrated approach for all AI projects (ecosystem). The above example shows how the recommendations, personalisation, churn prediction and customer value model comprehensively affect the customer journey. Similarly, other solutions collectively can provide more value than individually.