The Strategic Guide to Moving AI prototype to production

We have all come across those scary statistics about the extremely high failure rates of AI projects. According to different reports by different industries, between 60 and 80 percent of machine learning projects fail because they do not leave the lab.

Creating an impressive, very accurate AI prototype on the local Jupyter notebook feels like a big celebration. You gather around the screen with your team watching your model generate amazing predictions and believe the hardest bit is already over. In actuality, creating the first version of your model is usually the easy part.

If you are a tech or product leader, then you will definitely know the feeling I am talking about. You have got your model working perfectly on your clean data with excellent accuracy scores. It seems absolutely absurd trying to fit this frail bit of coding into your active program and situate it in your architecture.

That is not always the situation. If you know how to achieve all of these transitions seamlessly, you will be able to do so and convert your fragile prototype project into a profitable business solution through successful machine learning deployment.

In this detailed guide on moving an AI prototype to production, we will explain how to do just that. You will get a clear roadmap on how to avoid the pitfalls that lead to the most common mistakes in deployment and gain a clear understanding of how to close the infamous gap between your data science and software engineering teams and start reaping the AI business ROI from your AI solutions.

Why Moving AI From Prototype to Production is So Difficult

In order to solve a problem, one must first understand its origin. The basic gap that exists in the process of machine learning is the understanding that there is a great difference between the prototype and machine learning in production. The former is made for accuracy in a perfect vacuum whereas the latter is made for robustness in the chaos of reality.

The "Works on My Machine" Trap

A data scientist prototyping their work is working in an environment which is highly controlled and very lenient. This is because they would spend hours cleaning the data, removing the missing values, and formatting it properly.

They train the model using static and historical datasets that provide a picture at one specific point in time. Due to the controlled environment, the performance of the model is superb.

In stark contrast to the highly-controlled environment above is the complete chaos of the live data environment. In real life, consumers introduce typos in their data inputs, sensors go down, and even the database schema keeps changing at random intervals.

Apart from that, the behavior of consumers keeps changing all the time. There arises what we call the “data drift” phenomenon – it is basically the case when the real data starts drifting away from the data used for training. 

Data drift causes your highly accurate prototype to give you some really awful predictions and that too with complete mathematical conviction, highlighting the need for data drift monitoring.

The Hidden Costs of an AI Production Environment

Executing a machine learning algorithm locally on your laptop is essentially free. You press the button, fans start spinning, and here comes your result. Scaling AI applications at the enterprise level is a completely different story from a financial perspective.

Costs of cloud computing, GPU allocation, fast storage, and even outgoing network connections can suck away an entire budget of the department incredibly fast.

In addition to direct expenses, the maintenance costs of an AI system are extremely huge. The deployment process of conventional software is usually performed once and maybe with some occasional bug fixes further.

With machine learning in production, it is an ongoing process involving constant training, monitoring, and other actions. An AI model is a living organism and if not taken care of, it will degrade. Financial planning of moving an AI algorithm into production includes these ongoing costs as well.

The Cross-Functional Silo Problem

There is the human factor, which tends to be the greatest source of delay. Both data scientists and software engineers are very bright people; their languages are completely different. The former knows statistics, probability theory, optimization of algorithms, and Python programming. The latter know microservices, scalability, latency, uptime, and system architecture.

If a data scientist throws an unstructured, raw Python code over the metaphorical wall and tells the engineers "deploy this", disaster will follow. They won’t know how to optimize the code from the math perspective, and the data scientists will not know how to containerize the app. Inability to collaborate on the technical level is the primary non-technical reason why machine learning deployment fails.

Step 1: Assess Your AI Prototype's Production Readiness

Before anything else, such as starting to set up the cloud servers to deploy AI models, you will have to pause and do a very honest evaluation of the prototype you are currently working on. The reason why this is so important is that you wouldn’t want to scale a process that is inherently broken.

Define Clear Business Objectives and ROI

You have to make yourself and your team go through the process of asking difficult and unpleasant questions. Does this AI model solve a truly existing business issue which is measurable or is it a cool application of modern technologies? In case you cannot define precisely what value this model adds to the business, you have no right to use it.

What you have to learn here is to be able to connect the data science metrics to the business metrics. For example, the data scientist can be proud of the fact that he or she has managed to build the model with the 94% F1 score or reduced the mean squared error to a certain number.

The problem is that neither the company's CEO nor the board of directors will be interested in such technical details. What they need is customer retention, expenses reduction, increased efficiency, and income. You have to explain how that 94% accuracy saves or makes the company money.

Evaluate Data Pipeline Scalability

It’s as simple as Garbage In - Garbage Out when it comes to models. Intensively work on the data ingestion. You won’t be close to going into production with your prototype if you use static data in a CSV file, which was manually exported from a CRM, preprocessed in Excel, and uploaded into a notebook.

You need to meet with your data engineers and figure out exactly how you are going to get data, which is going to flow in real-time from your end users through your app backend and to the database and to your model.

This means setting up robust pipelines, which are capable of handling missing data on the fly, validating data types before the data gets to your model, and scaling up immediately, in case of an influx of traffic into your application.

Establish Baseline Metrics for Success

The toughest thing that data scientists have to grasp is that 99% accuracy is not only unnecessary, but in most cases, it is simply unattainable in a production environment.

The task is to get the team to establish a benchmark on how to consider their model good enough to be deployed with their MVP as part of your AI deployment strategy. In some cases, a model which is 80% accurate and makes decisions in milliseconds can be far more useful than a model that is 95% accurate and requires ten seconds to run.

Bring in operational metrics that should go hand in hand with the statistical metrics. At what speed should the predictions be made to guarantee the best experience of using it by users? That is your latency metric. How many requests per second should your system process during the peak loads? It is your throughput metric.

Step 2: Transition from Notebooks to Scalable Code

Jupyter notebooks are great tools for exploratory analysis and visualizing the results but a complete nightmare when it comes to machine learning in production. They promote messy, sequential and stateful programming. It’s time to refactor our code if we want to build a good AI application.

Modularizing Your Machine Learning Code

The first technical challenge with your AI deployment strategy would be to split those long scripts that are executed from a single notebook into modular and testable object-oriented Python code.

That means that there should be no long script loading data, training the model, and performing predictions. There needs to be a distinct script for data pre-processing, for model inference, and for post-processing the results.

At this stage, it is essential to hammer home the significance of version control tools like Git for your data science team.

It will be critical in the production environment to know precisely which part of the code was modified by whom and when and why. More importantly, it will allow rolling back to the previous stable state of the model if a new version of a model accidentally breaks the application.

Implementing Proper Error Handling

In a local prototype, whenever the user happens to input a strange data type—such as typing "ten" instead of "10"—the program is immediately terminated by throwing a fatal error. When this happens in the production environment, it means a crashed web page, a failed transaction, and an angry user.

It is up to you to ensure that the code being written by your developers is well-defended. This includes using a try-catch statement when calling any function of your model. It includes writing defensive code that is able to handle edge cases, graceful handling of any missing JSON field, and above all, a code that returns a fallback value for the user rather than a raw system error message.

Choosing the Right Tech Stack to Deploy AI Models

After that, you have to choose the right machine learning tech stack to deploy your code in the cloud. The most common practice to make sure that the machine learning model will work seamlessly across all the devices and systems including the developer’s laptop, test server, or huge AWS cluster is to use containers.

Once the model, its exact dependencies, and necessary OS libraries are wrapped in a Docker container, the so-called "works on my machine" issue disappears forever. Docker images behave absolutely identically no matter where they are used.

To successfully deploy AI models, you have to create the interface to expose your model so that your software could interact with it. This is achievable by building a REST API using simple and lightweight Python frameworks, such as FastAPI. 

At the enterprise level, there are specialized serving frameworks for performance purposes, including NVIDIA Triton, TensorFlow Serving, and TorchServe, together with an artifact management solution like MLflow. Wrapping your model in an API allows for your web or mobile app to pass data to some endpoint and get a prediction back.

Step 3: Implement MLOps Best Practices

In order for you to be able to deploy your artificial intelligence applications in a scalable manner while ensuring that your engineering team is not severely burned out, you will have to make use of MLOps. Machine learning operations are what will make the machine learning deployment of your AI application successful.

What is MLOps and Why Do You Need It?

In simple terms, MLOps can be defined as the implementation of classical DevOps concepts such as continuous integration, testing, and delivery but with respect to machine learning applications.

The reason you need MLOps is that while traditional software code does not change much once written, say for example a piece of code which sorts a list written today would be able to sort that same list after five years from now, machine learning applications on the other hand are dynamic in nature.

They decay over time due to the changing dynamics of the environment around them. For example, a fraud detection application developed in 2019 would be completely irrelevant in 2021.

Setting Up Continuous Integration and Deployment (CI/CD)

The true ML pipeline begins with building ML CI/CD pipelines for AI development. Once your data scientist makes necessary modifications to the model and submits the new version of the code to the repository, automated testing should start.

This testing should be more than your regular testing process; it needs to cover data validation and model testing processes. Your pipeline needs to automatically confirm that the code compiles and the API reacts in a way you expect it to do.

The automated testing needs to confirm that the new version of the model at least performs the same as the previous versions of the model on the holdout test dataset.

After all these tests run successfully, comes the stage of deployment. You can use the help of orchestration frameworks like Kubeflow, Apache Airflow, or AWS SageMaker Pipelines for creating your automatic DAGs.

Your CI/CD pipeline will then package the model into a container and deploy it to the staging environment for one last sanity check from QA engineers and product owners before production replacement.

Automating the Model Retraining Pipeline

The ultimate state of the AI lifecycle is when you realize that the model needs to be automatically retrained each time its performance starts falling, because you cannot have human engineers doing this work every single time. And in order to make sure that your AI app stays smart and relevant, you need to implement an automated retraining pipeline.

You will have to implement a trigger mechanism within your automated infrastructure that is based on specific criteria. For example, you could write down the following one: "Retrain the model automatically each Sunday at 2:00 AM using the data of the last 30 days."

Here is an example of implementing a trigger based on performance: “In case the level of prediction confidence drops under 85% for three days in a row or 10,000 new unique rows were added into the database, start cloud computing for retraining and deliver output to CI/CD pipeline.”

Step 4: Monitor, Manage, and Mitigate Risks

For any AI product, deployment is not an endpoint but a beginning point. Your model goes online and interacts with end users, and from then onwards, you are in the most crucial stage of the machine learning cycle. You will have to keep monitoring it closely.

Tracking Model Drift and Performance Degradation

Model drift was already mentioned in previous paragraphs, but monitoring it in a production environment is unbelievably difficult, making robust data drift monitoring essential. In most cases, you cannot understand whether the model’s prediction was correct or not straight away.

If your AI suggests a certain movie to users, the only way to determine if it was a successful suggestion is when the user clicks the movie and watches it. In case if your model predicts that certain people will be unable to repay a loan, then you will have to wait for several months or even years to see if the model’s prediction was accurate or not.

In many cases, you will not get instant “true” answers (ground truth). This is why you should use some proxy measures of how much the model's predictions drift.

Suppose you have built an engine of recommendations for your e-commerce website. Suddenly you notice that the click-through rate and average cart value of your clients dropped dramatically – it means that the model’s predictions do not work anymore.

That is why you should create dashboards that will alert you about any anomalies in the model’s predictions through Slack or email messages.

Ensuring Data Privacy and Security Compliance

While in the prototype stage, data scientists are working with heavily scrubbed or perhaps even fully synthetic datasets, but production systems have to work with actual, raw user data.

The transition of an AI prototype to production introduces a lot of heavy, non-negotiable responsibilities regarding compliance with laws, including GDPR in Europe, CCPA in California, and other data security guidelines of the industry.

You need to make sure that your AI endpoints do not accidentally expose any PII via their output. In addition to that, you need to make sure that your AI endpoints are as heavily secured as your payment gateways.

Machine learning algorithms, in particular, are especially susceptible to certain cybersecurity threats, like data poisoning (when people feed your machine learning algorithm with malicious data to destroy its performance) or model inversion attack (when people try to deduce your training dataset based on your model's predictions).

Building a Fallback Mechanism

What might be the most important question you could pose to your engineering team would be: what exactly happens when the application is faced with an outage, a lengthy response time, or complete confusion from the AI when faced with an edge case?

Fails happen. APIs timeout. Providers face outages. It is essential that you recommend that your architecture team implement the application in a manner in which there is "graceful degradation".

Should the AI fail to return a result within the 200 millisecond requirement, the application should not crash nor present the user with an eternal loading spinner. Rather, it should instantly switch into a rule-based system which has been hardcoded.

If by some chance the AI chatbot is a customer support bot and encounters something it can’t figure out, then the interaction should instantly be transferred to a human operator.

Step 5: Foster a Collaborative AI Culture

We have talked extensively about code, containers, and pipelines. At the end of the day, it is not tools, cloud infrastructure, and scripts that make a successful product.

Scaling an AI application calls for an enormous cultural change in your company. You must intentionally destroy silos between the three major players – data scientists, software engineers, and product managers.

You must foster an environment in which these three groups use the same language and understand what is the definition of success for all of them. It might be a good idea to organize your team into AI squads, rather than leaving them in isolated silos. When a data scientist knows the latency limitations of the software engineer and vice versa, everything gets simpler and faster.

Scaling AI Applications for Long-Term Success

It goes without saying that the process of taking an AI prototype to production is quite challenging. As we have mentioned in this article, you need to be ready to assess your operational preparedness, refactoring your messy codebase into an extremely scalable one, embracing the power of automated workflows in MLOps, and monitoring your models continuously once they go live.

This is a challenging task on many levels, but the rewards from having a working, intelligent, and autonomous application in production are huge. Your business will get a great opportunity to change its unit economics, improve customer experience dramatically, and create a competitive advantage for itself in the market. It is time to stop prototyping and start building!