How to Evaluate a Machine Learning Project: 7 Proven Steps

Now imagine that after having invested thousands of dollars in your new AI-powered software solution, you've found out after several months of use that your software generates inaccurate results, operates with prohibitively high expenses and customers are not willing to pay for it.

And that's a situation worth taking into account before spending money on your machine learning initiative. Evaluating a machine learning project early can help identify potential problems before they become expensive.

Since companies are now looking for ways to automate their processes, predict some events using AI, detect fraud, recommend some products using recommendation engines and generate content with help of generative models, selecting the right machine learning initiative is becoming increasingly important.

Even though you may have the most technically sophisticated project, it doesn't necessarily mean that your machine learning initiative will be commercially viable. A machine learning project must deliver business value as well as technical performance.

How do you estimate the feasibility of your machine learning project then?

There are several things you need to check in addition to your algorithm itself. A structured evaluation of a machine learning project helps businesses make more informed investment decisions.

In this article, I'm going to provide a list of seven steps for evaluating your machine learning project, estimating ROI, checking warning signs and taking final decisions on what should be done next.

Why Machine Learning Projects Need a Different Investment Evaluation

The uncertainty in machine learning ventures is something that cannot be fully captured by the usual software ventures. It relies on factors like data quality, the outcomes of the experiment conducted, actual circumstances, and the capability to ensure consistent performance of the model once deployed. These factors make evaluating a machine learning project different from evaluating conventional software investments.

An ordinary software venture would have a somewhat straightforward process in development. In machine learning, there could be multiple experiments undertaken by the developers before determining the viability of the solution.

As noted in Google's feasibility guide for machine learning projects, data availability, difficulty of the problem, prediction quality, technical demands, and costs will determine the viability of the project.

Here are three differences that really matter.

It is not a finished product. A model which works well on selected examples can have problems with working on partial data, weird inputs or altered customers' behavior. Practical usage involves even more conditions such as integration, security, monitoring and maintenance.

Development cost is not all the cost involved. In addition to developing the model itself, you will have to consider costs of computing resources, data preparation, software integration, expert labor and performance monitoring.

Performance is distinct from profitability. Having the right model does not guarantee that it is profitable to pay for the expenses incurred. 

Step 1: Evaluate the Business Problem and Market Opportunity

Before looking at the technology, raise the most important question of all: is this problem worth solving?

The business requirement must be determined first, not the adoption of machine learning just because competitors do so. This helps establish whether a machine learning project addresses a genuine business need.

Define the problem in terms of measurements

Does a company lose money due to bad recommendations for its products? Do people spend hundreds of hours on invoice processing manually? Is fraud causing unnecessary money losses?

Measure the problem in numbers

If a retailer has to invest 500 employee hours per month in transaction analysis and prevention of fraud, then it is obvious that an automated system that will reduce this manual labor without raising the rate of mistakes will have its place in the organization.

The assessment will determine:

• The costs of the problem as it stands now.

• The frequency of its occurrence.

• Its consequences.

• The level of improvement which is needed.

• The measurements for measuring the performance of the system.

Without a measurement, it is not possible to assess if the system will be useful for the firm. This measure can also serve as a basis for evaluating a machine learning project once it has been implemented.

Investigate the market and customer demand

If you’re considering an investment in a machine learning venture or commercial offering, consider its market opportunity.

Who is your customer? What is their problem? Are they willing to pay to solve it?

Consider your competition, alternatives, market dynamics, and possible adoption challenges.

A big market is not necessarily a good investment opportunity. It’s important if the project can actually reach the customer and offer a sufficient value proposition to get the business. A machine learning project should therefore be evaluated against realistic customer demand rather than market size alone.

Determine whether machine learning is necessary

All business challenges don’t necessarily call for machine learning algorithms.

A rules-based system, regular software solution, or workflow automation application may work out cheaper and better to solve the issue.

For example, consider a company’s need to classify customer support requests into some specific categories. If such a rules-based workflow solution works fine, then building an advanced algorithm will make things complicated.

It’s only where there is a pattern, prediction, or decision that is hard to solve through alternative solutions that machine learning comes handy.

Step 2: Assess Data Quality, Availability, and Ownership

Even a great design may lead to bad results when used with inappropriate training data – either incomplete, wrong, out-of-date or representative. Data readiness is therefore a critical consideration when evaluating a machine learning project.

Ensure whether the project has enough data for building and maintaining the proposed system before you start your investment.

Assess data availability and quantity of necessary data

Request the team to present data available, data source, and determine relevance of data in relation to the environment where the model will operate.

The key aspects include:

• Completeness: Are there any missing records and fields?

• Accuracy: Is the data relevant to what really happened?

• Consistency: Can the different systems work with the same formats and definitions?

• Representativity: Is the dataset representative to the target audience and the actual cases?

• Labeling: Are there any labelings for examples, and how good are they?

• Freshness: How often should the data be updated?

Depending on the case, model and technique, the required amount of data may vary.

The question here isn't only about the existence of data. The more important question is whether the data can ensure the needed level of performance.

Verify data access, ownership, and usage rights

It doesn’t follow from the fact that the data is there in the firm that it could be used in whatever manner.

These may include matters such as ownership, licensing, consent, privacy policies, and contractual obligations. The thing to note here is that in case the project that you are working on uses third party data, it should be sustainable and protected.

Also, you have to learn about how sensitive data will be stored, managed, protected, and maintained.

A project that cannot legally access the required data simply won’t work regardless of technology involved.

Data preparation expenses calculation

Unsorted and unlabeled data very rarely comes ready-made for machine learning.

Sorting and labeling, database integration, inconsistencies fixing, and overall data pipeline design might take time and efforts of a professional.

Ask about resource estimates for the data preparation and maintenance.

You should wonder if your data set creates a sustainable competitive advantage for your business. Proprietary data can increase competitiveness of your firm, but only in case it is relevant, legal, unique and boosts performance.

Investment criteria: A list of data, sample evaluation, proof of data usability and data preparation process.

Step 3: Validate Technical Feasibility and Model Performance

After establishing the possibility that the problem and data can be solved by machine learning, check on the viability of using machine learning to solve the problem. This is a central step in determining whether a machine learning project can deliver its intended results.

A great showcase isn’t sufficient. What’s needed here is reproducible proof that the system works well in realistic settings.

Evaluate the technical solution proposed

Consult technical personnel for information on what model/architecture is to be used, why this model/architecture is appropriate, and other models/architectures that have been thought of.

Look out if there are any solutions to the problem previously provided. This will cut development time and any uncertainty that may arise from it.

Review benchmarks and testing methods

Ask for the results of tests on representative samples which have not been used to train or fine-tune the model.

The team should have a relevant baseline against which to compare their solution – for instance, the current process within the business, a simple rule-based system, or another model.

This will depend on the task at hand.

For example:

• Precision is the proportion of positive predictions which are true positives.

• Recall is the proportion of positive cases found by the model.

• Mean absolute error is the average magnitude of prediction errors in numerical tasks.

• Latency refers to the time that the system requires in giving a response.

For the application of AI in the area of generation, one should check whether the result is correct, relevant, consistent, using the source materials correctly, and whether it produces any harmful outputs.

Connect model performance to business value

High accuracy does not mean automatic usefulness of the model.

In the case of a fraud detection system, false negatives might lead to losses, but false positives would irritate customers and cause additional expenses for investigating them.

All of this is dependent on the value of the trade-off between these two types of mistakes.

The minimum performance required from the model should be decided before developing the model. Setting this threshold in advance helps determine whether a machine learning project meets its technical and business objectives.

For more details, see Google's framework for a successful machine learning project, where they distinguish business metrics and model-performance metrics.

Investment test: Replicable results, applicable benchmarks, proper performance criteria, and enhancement of company’s performance indicators through the proposed system.

Step 4: Calculate Total Cost, ROI, and Payback Period

Machine learning could be considered an effective project in technical terms, but still a bad financial decision. The financial assessment will give you the idea of whether the advantages of the project will outweigh its costs. 

Identify the total cost of ownership

Do not restrict yourself with the budget of the model development.

Separate costs into three types.

1. Costs of initial development

It can include:

• Market research and testing of feasibility.

• Collection, preparation, and labeling of the data.

• Development and customizing of the model.

• Programming and integration into the existing system.

2. Costs of regular operation

This part includes cloud services, computing resources, payments for the API calls, data storage, monitoring, maintenance, updates of models, and human review.

3. Organizational costs

It involves hiring and training of personnel, re-designing workflows, compliance, change management, and necessary time for personnel to use the system.

Determine these costs for a certain period of time, for example, three years.

Estimate the expected business benefits

The potential advantages will be higher sales, less mistakes, decrease in fraud loss, quicker processing, decreased operating cost, and enhanced customer retention.

It is important to separate the cash savings from productivity savings.

For instance, an annual saving of 100 working hours per month for the employees may not always translate to a reduction in payroll cost. This depends on how the organization saves through the saved hour.

This should be reflected in your prediction of the cost savings.

Calculate ROI and payback period

Use the following formulas to establish a consistent financial baseline.

Return on investment (ROI):

ROI = (Total Benefits − Total Costs) / Total Costs × 100

Payback period:

Payback Period = Initial Investment / Net Periodic Cash Benefit

Assumption 2: Net Cash Benefits are reasonably stable. If not, then estimate by period rather than estimating net cash benefits in aggregate.

For instance, assume that the cost of developing and implementing the project is $100,000, annual financial benefits amount to $70,000 per year, while operating costs are $20,000 annually.

The annual net benefit will be $50,000.

If such values are relatively stable, the cost of the investment will be recovered after two years.

During the three-year period, the total benefit would reach $210,000, while the costs would equal $160,000, which gives us the overall ROI of 31.25%, excluding the financing, taxes, and discounting.

It is a fictional example and doesn’t reflect any concrete forecast for any particular project.

Test various financial scenarios

Build at least three different forecasts:

• The pessimistic scenario assumes the slow adoption, higher cost or lower performance of the project.

• The base case – the project’s performance corresponds to normal expectations based on factual data.

Step 5: Evaluate the Team, Execution Plan, and Delivery Readiness

You have to see whether the developers of the project possess enough knowledge to make this model into a good business product. The capabilities of the team matter a lot when dealing with a machine learning project.

Evaluate technical and business expertise

Examine the level of experience in machine learning, data engineering, software engineering, delivery, and business field of the team.

Knowledge of the customer’s workflow, possible risks associated with inaccurate modeling and the value that the product would add is essential for the team.

Review the roadmap and milestones

• Discovering business opportunities and feasibility analysis.

• Data collection and initial experimentation.

• Prototyping and testing of the prototype.

• Pilot phase and collection of user feedback.

• Release into production and monitoring.

Every phase should be budgeted, responsible person defined, scheduled and deliverable results set up.

Development of machine learning solutions is a trial process and hence the schedule needs to take into account the possible failures and iterations. To assume that every technical question will translate into development tasks is to make unrealistically optimistic estimations.

User adoption and change management

The solution would be of value if it was implemented effectively or there was some automation of processes involved. 

Step 6: Assess Scalability, Integration, and Long-Term Maintenance

Assess system integration and implementation

Think about how the model is going to integrate into current databases, applications, APIs and business processes.

Check for compatibility, data flow, security mechanisms, time performance and necessary infrastructure. Assess whether the solution should run on demand or it can process data batch by batch.

This issue is especially relevant when predictive analytics results determine real actions. The organization requires robust mechanisms to manage failure situations, validate results and switch to an alternative approach.

Evaluate the economics of scaling

Determine how the cost will vary with usage.

The cost associated with the use of generative AI could increase due to extra API costs for the increased number of request submissions. Increased computation power could be needed for a computer vision system as a result of increased number of images.

Calculate how much a single transaction, prediction, document or completed task cost. Compare that amount with the income or cost savings of a single activity. This calculation helps determine whether a machine learning project can remain financially viable as demand grows.

Unit economics of a project becomes better with an increasing demand if unit economics stay sustainable.

Think about model drift and decline in performance

Data changes over time. Customers' preferences change, fraud patterns become different, new processes appear.

All these changes may cause deterioration in the model performance regardless of initial positive evaluation results.

Find out what is the plan for tracking the quality of data, detecting performance degradation, failure investigation and model updating.

Google's productionizing guide covers data feed monitoring, model outputs, latency and other aspects.

Investigate vendor dependence

In case there is any dependence of the project on an external provider of models, API, or cloud computing service, consider consequences of price changes, service failures, license limitations, and feature removals.

Consider the possibility of replacement of the provider or usage of another approach with reasonable expenses. Vendor flexibility can help protect a machine learning project from unnecessary long-term costs and operational disruptions.

Investment test: Demand for credible architecture of deployment and maintenance of the project with guarantees of reliable operation of the project at its planned scale.

Step 7: Identify Investment Risks and Establish Safeguards

Evaluate technical and operational risks

Identify the case where the model gives a wrong prediction. While in some instances, an error could simply be a bother; in other instances, it could have dire consequences.

Possible actions for such situations may include human interference, confidence intervals, tests, and contingency plans among others.

Consider commercial and competitive risks

This project may become unattractive if it becomes easy for its competitors to duplicate the functionalities of this project using the available technology.

Identify the reason that gave the project a competitive edge. The reasons could be the data, process, relationship, knowledge, distribution or execution of the project.

Determine whether there is any customer concentration, pricing threat, market acceptance or other technologies which reduce the attractiveness of this project..

Establish investment gates and exit criteria

It is necessary to avoid making all the funding available prior to testing the critical assumptions behind the project.

Funding should be phased based on evidence.

For instance, a feasibility phase can involve ensuring that the data access and model performance criteria have been met. A pilot phase can involve acceptable cost of operations and user adoption. The full scale implementation phase can involve the business case and risk management.

When should the project proceed, be modified, stopped or killed?

Build a Machine Learning Project Investment Scorecard

In case of un-weighted score, add up all seven scores to get a score ranging from 7 to 35. The score weightings can also be modified according to the investment priorities.

The sum cannot make the investment decision alone. It will be impossible to compensate for such important issues as data misuse, unrealistic technical requirements, and safety risks with a good total score.

Evaluation of the score results:

• Strong investment opportunity: There are reasonable arguments in support of basic assumptions; reasonable returns; appropriate management of material risks.

• Conditional investment opportunity: Good investment potential, but there are some doubts that should be clarified through testing or pilot.

• Don't invest in this project now: The assumptions are unsubstantiated, the costs are unreasonable, or material risks cannot be managed.

The scorecard should only assist in decision making. Your criteria for scoring and minimum standards should be developed prior to evaluation of individual projects.

Red Flags to Watch for Before Funding an AI Project

There are a number of red flags to be aware of during the due diligence process.

Be concerned when the project:

• Loses sight of metrics and instead makes vague claims about the use of AI.

• Is not capable of showing performance against an established benchmark.

• Has uncertainties in relation to data availability, ownership, quality, or use rights.

• Makes financial projections that don't include integration, maintenance, or human management.

• The prototype only works because of hand-picked examples.

• Has no defined owner responsible for operation and maintenance.

Is high on operating cost that increases as use increases but has not done any calculation of unit economics.

Machine Learning Investment Due Diligence Checklist

Review this list prior to allocating funds for the machine learning initiative.

• The business problem and the current baseline are well-defined.

• There is proof for customer demand and market opportunity.

• Data quality, availability, ownership, and usage rights have been confirmed.

• Technical performance is validated against a proper baseline.

• Costs related to development, deployment, and operation have been estimated.

• Assumptions, expectations and risk scenarios have been defined regarding ROI.

• There is sufficient technical and commercial expertise within the team.

• Issues of integration, scalability, monitoring, and maintenance are covered.

The issues surrounding legality, privacy, security, ethics and competitiveness have been reviewed.

There is a set of clear milestones and criteria for stopping the program should the milestones not be met.

Document all factual statements.

In instances where there is no evidence, validation must become the criteria for future funding rather than the assumption being stated as a fact.

Invest in Evidence, Not AI Hype

The knowledge of how to evaluate a machine learning venture prior to investing in it will allow you to save money and discover a more commercially viable opportunity.

A venture with great potential will have a business challenge worth solving, solid data, validated technology, sound financials, proper execution, sustainability and low risk.

You don’t need to have absolute certainty regarding the enterprise to invest in it. What you need is a methodical process of reducing uncertainty through measurement and evaluation. 

Your starting point should be business validation, followed by assessment of data availability and technical feasibility.. Measure costs of ownership, validate the financial assumptions and prove the solution's effectiveness through a prudent pilot project.

The ideal machine learning venture to invest in will be not the one with the most sophisticated model, but the one that will solve a problem, deliver results and sustain itself on an acceptable level of risk.