The Ultimate Guide to Data Preparation Steps for Machine Learning Projects

Did you know that data scientists dedicate up to 80% of their time simply trying to find, clean, and organize data? It is an alarming statistic to consider in light of the huge amounts of money spent on artificial intelligence. 

If you are an executive, this is precisely what has been holding your AI efforts back. You already have your data, and you certainly have your business problems to address, but crossing the enormous distance from an untidy database to a functioning, profitable predictive model can seem daunting.

Many businesses rush into algorithms without the foundation in place first. By understanding the essential data preprocessing process involved in a machine learning project, you will be able to turn your untidy business data into valuable training resources.

Missing this important step will lead you into trouble with your AI project faster than anything else. If you skip this part, then you'll end up developing models that fail, burning money from your engineering budget, and making faulty business forecasts that can actually damage your company financially. 

You need a proper process for preparing your business data for machine learning. Here is how you should do it.

Why High-Quality Data is the Lifeline of Machine Learning

Machine learning algorithms are extremely smart; They are not smart on their own. They lack intuition and common sense. They learn based on past trends. They can only be as intelligent, objective, and accurate as the input that is provided to them.

The "Garbage In, Garbage Out" Principle

You are likely familiar with the old computing maxim: 'Garbage In, Garbage Out' (GIGO]. From today's business perspective, this maxim holds the ultimate truth for the process of machine learning. 

If one supplies an algorithm with biased, inaccurate or dirty data, one can be sure that he/she will receive a totally wrong result.

Imagine asking a new hire to forecast sales using torn financial ledgers filled with crossed-out figures and upside-down graphs. The machine learning equivalent is feeding an algorithm dirty data—it simply cannot output the truth. The situation with a machine learning model is similar since an algorithm will perceive a mistake as the truth.

How Poor Data Preparation Costs Businesses Money

Here’s an example of how not cleansing your data before machine learning can cause financial losses for your business. For example, suppose you have a major regional chain store trying to forecast its requirements for the coming winter season. 

It inputs its historic sales data into a brand new algorithm. But nobody took care of unifying the date formats. Some stores use MM/DD/YYYY while others use DD/MM/YYYY format.

Because of this little mistake, the machine learning algorithm detects a surge in winter coat sales in the summer. And then it recommends buying tons of parkas in July. What’s the result? In the summer, you get warehouses stuffed with winter clothes and empty racks in the winter months, plus losing hundreds of thousands of dollars. 

Step 1: Define Your Business Objective and Data Requirements

Aligning ML Goals with Business Outcomes

The machine learning algorithm needs to have some sort of end goal associated with it. Do you want to predict customer churn for your retention team to take action? Or do you want to predict quarterly sales to help you with your logistics process? Or is there a recommendation engine being created for increasing the average order value?

All of these scenarios need a completely different strategy in order to get the dataset ready for the machine learning algorithm to learn from it. Everything starts with the alignment of all those people who will be involved in the whole process. Figure out what your target variable is, and everything else will follow.

Identifying the Right Data Sources

Now that you know what your prediction target is, you have to audit the business data you already have to determine whether or not you even have access to the necessary data for making the prediction. You need to identify the variables that impact your selected business outcome.

If you want to predict customer churn, you will need to gather data from your CRM system, which will provide you with the history of communications, your billing system, which will inform you about unpaid bills, and the product analytics system, which will show you any reductions in product usage.

Step 2: Data Collection and Aggregation

Since you have now figured out what you are supposed to look for, it is now time for you to go ahead and collect all the data. This stage involves centralization since it will bring all your data together in one place.

Gathering Internal vs. External Data

Your exclusive internal data – your sales data, website hits, customer service ticketing system – is your greatest asset due to its exclusivity. But internal data seldom reveals the full picture.

You have to augment your internal dataset with external data so that the machine learning algorithm would receive additional insight. If you are a coffee shop chain owner interested in predicting daily foot traffic, your internal sales data comes handy. Yet if you add external weather data (such as rain/snow forecast) or local economy data, you would greatly boost your model’s performance.

Overcoming Data Silos in Your Organization

The key challenge in preparing business data is managing data silos. Your marketing team has its software, while the sales team has its software. Your finance team has its own legacy software that no one else in the organization uses.

In order to create a machine learning algorithm, you can’t keep data in silos. What you have to do is combine all these streams into a single location, such as a data warehouse or a data lake. Setting up data pipelines that will bring all of this data into a single place will allow you to avoid looking for CSV files from different departments.

Step 3: Data Exploration and Profiling (EDA)

Having brought your data into one place, you might want to go ahead and begin doing aggressive cleaning. Wait! First, you have to familiarize yourself with your data. That step is called Exploratory Data Analysis, or EDA.

Characterizing Your Dataset

EDA is the process of profiling your data, understanding its nature and how it behaves. You are looking at some of the descriptive statistics. What is the mean of your sales variable? What is the median of the age of your customers? How is the distribution of data?

As you do this profiling, you start getting the stories behind your data.You find out which variables are having missing values in abundance, which variables are having high ranges, and which features are having high correlation. This information is necessary for you to be able to make good choices in later steps.

Spotting Anomalies and Missing Values Early

EDA is also your first warning sign. By performing initial diagnostics and visualizations on the data, you can discover some obvious mistakes before you get started.

Maybe you see that the column labeled “Customer Age” has many readings marked as 999. Perhaps a device in your manufacturing facility suddenly stopped capturing any information for three weeks in the month of October, creating a huge gap in your timeline. 

Identifying such anomalies is going to save you a lot of time and effort in the long run and will prevent you from developing a model based on a flawed premise.

Step 4: The Data Cleaning Process

This is where actual manual work kicks off. Machine learning data cleaning is an intensive process of dealing with data which is either incorrect, corrupted, incorrectly formatted, duplicate or incomplete.

Missing Values & Nulls

Any data in reality will most certainly include some missing values. The way you deal with such blanks will define how good your model performs.

You basically have two ways to approach the issue. You can simply drop these blanks, by deleting the entire row or column. It takes no time, you take a very big risk as you can throw away important information, particularly, when working with small datasets.

You can use imputation. It involves filling the blanks with certain values. For example, it can be the mean or median of that particular column. If you're more experienced, you can create a separate machine learning model for prediction of that value.

Removing Duplicates and Irrelevant Observations

Duplication of records silently kills machine learning accuracy. If the purchase made by one client is duplicated three times in the database, the machine will think that this kind of purchase is made three times more often than it really is. It totally distorts the model's weighting.

You need to get rid of exact duplicates systematically. You should eliminate irrelevant records. In case you want to make predictions about sales within the USA, the observations with data from your European divisions are totally irrelevant. 

Correcting Inconsistent Formatting and Typos

Human error in data entry is inevitable. In one column specifying the country of a client, you can find such variations as "USA", "U.S.", "United States", "US" and even misspellings "United States".

For the person, they all have the same meaning. But for the machine, they are different categories. You need to standardize the text data. It means that you should convert all data into lower case letters, correct possible typos and unify names' formats.

Step 5: Data Formatting and Type Conversion

Even when the data is clean and accurate, it may not necessarily be in a language understood by the algorithm for the machine learning. Machine learning algorithms are very picky about the data formats that they receive.

Standardizing Data Formats

Consistency is vital in this case especially when it comes to dates and currencies. This was previously discussed concerning the problem of having both MM/DD/YYYY and DD/MM/YYYY formats in one dataset. The solution to this would be to format all the date and time in ISO 8601 standard format (YYYY-MM-DD).

The same applies when getting data from various global departments for financial purposes. It is impossible to have Euros, dollars, and Yen together in one column since it needs to be converted to one base currency.

Converting Data Types for Algorithm Compatibility

This is one basic truth about machine learning. Machines understand numbers and not words. In case you have a column full of words (also called string), then the algorithm does not know how to process it.

It is important for you to make sure that your data types are correctly defined. The numerical values should be integer and float. If for instance some system has imported a column that consists of numbers in string format (e.g., 100 in the form of “100”), then it should be converted back into numbers.

Step 6: Handling Outliers and Anomalies

Outliers are points of data that are markedly different from other observations. Such points are located away from the normal distribution pattern of your data. The way to deal with such points depends on the business environment.

How to Handle Them or Discard

Not all outliers are bad, and just discarding them without further consideration is a wrong step to take.

For example, if you notice a sudden explosion in traffic on your website for some random Tuesday, it can be a bot attack (which you should discard). It can also be the day when an influential blogger talks about your product on social media. 

It is a real-life incident that has value for your business. By discarding such an outlier, you prevent your model from learning about the sales spikes. You have to examine the reason for the anomaly. If it was due to any typo, such as adding an extra zero in the price, you should discard it.

Techniques for Outlier Treatment

Should there be a need to cap or reduce any extreme values in your figures that seem to be causing distortions, yet you cannot afford to delete the entire row, then capping is one way out.

You establish the upper and lower limits of your figures. Any figure that exceeds the upper limit is converted to that limit. This way, you limit the gravitational effect of outlying figures.

Step 7: Feature Engineering and Selection

And this is often the stage when data science becomes more of an art than just a scientific approach. Feature engineering is the key to successful machine learning.

Creating New Features to Improve Accuracy of Models

"A feature" is just another name for a column in your database. Feature engineering is the procedure of using your columns to make new ones and reveal the hidden patterns better.

In your database, there should be a column called "Date of Birth." The algorithm may get this information but does not have any understanding of what "aging" means. So, by using "Date of Birth" and creating a new column named "Current Age," you give the model more useful numerical information.

Selecting the Most Relevant Variables

The first myth that exists in AI is the idea that feeding the algorithm with more data will make it "smarter." Loading too many useless columns to an algorithm will make it confused and decrease the speed of the training. It is referred to as the "curse of dimensionality."

The feature selection is a technique of aggressively reducing columns which have nothing to do with your dependent variable. If you want to predict the probability of purchasing a software subscription by a customer, then his/her eye color is not important at all. Remove the noise so the model will calculate only relevant columns.

Step 8: Data Transformation and Scaling

But there is one final mathematics obstacle that you need to overcome before your data is really ready for the algorithm.

Normalization vs. Standardization

The machine learning algorithm relies on mathematical formulas to calculate distances and relationships between data points. Having very different measurement scales for your variables will create an issue of prioritizing one variable over another regardless of its relevance.

Take a dataset with Age of a customer, which is measured between 18 and 80 years, and the Annual Income of a customer ranging between $30,000 and $200,000. Because of the very high numbers, the algorithm may conclude that the income variable is immensely more important than age because of its higher value.

This problem can be solved using Normalization or Standardization. Normalization is squashing data to fit it into the scale from 0 to 1; Standardization centers the data with a mean of zero. Both approaches give a fair chance to Age and Income variables to be compared without being affected by their raw values.

Encoding Categorical Variables

As was discussed above, the algorithm is only capable of processing numbers. Thus, what is the solution when the critical information is contained in categorical form, say "New York," "London," or "Tokyo" for the location of the client?

The solution lies in the use of encoding techniques such as One-Hot Encoding. Using this technique, a new column is created for each category. If the client is located in New York, then he gets 1 in the "New York" column and 0 in the "London" and "Tokyo" columns.

Step 9: Splitting Your Dataset for Training and Testing

You are all set with manipulating the data. The very last thing that you have to do before beginning the creation of your AI is to partition your pristine data into two different sets.

The Importance of the Train-Test Split

It is impossible to test a machine learning algorithm on the same data it learned from. It would be similar to giving an examinee the answer key before his/her exam. The examinee will get 100% marks but you won't really know whether he/she has learned the concepts or has just memorized the answers.

The general practice is that you should split your dataset in the ratio of 80:20 where 80% of your data will train the algorithm while the rest of 20% will be reserved for testing purposes.

Avoiding Data Leakage

In dividing up your data, you need to be hyper-cautious about data leakage. Data leakage is a scenario where information from sources other than the training data set is used inadvertently while building the model.

When you normalize all of your data prior to division into training and test sets, the training data has effectively been affected by averages in the test data. This small leak makes you falsely believe that your model is highly accurate, which in turn means that your model is bound to fail miserably when applied in real-life situations.

Your Next Steps in Machine Learning Readiness

Data preparation is seldom sexy. It isn’t going to win you any media attention, and it takes immense amounts of hard work and diligence. But, it is the silent hero behind every successful application of AI technology in the world today. 

By knowing how to prepare your data for machine learning projects, you can be assured that your algorithms have a solid base in the truth, not errors.

Don’t let poor data hold back your innovations. Start analyzing your own company’s data hygiene practices today. Check out your CRMs, your databases, and your data pipelines. Do you have what it takes to turn your messy data into one of your greatest assets?

For more information on how to align your data architecture with your AI goals, check out all of our other enterprise AI strategy guides right here at Gigmint.AI.