Learn Advanced ETL and ML Pre-Processing to Avoid Garbage

October 2, 2017 — by Kathrin Melcher

Today we look at a dataset that supposedly is already clean, joined with the right additional information, and in the right shape and we want to use it to train a prediction model. Unfortunately, a quick glance at the dataset reveals that it still has tons of missing values, it is not normalized, and contains too many too similar features.

This means that any algorithm would have a really hard time to train a good prediction model on it. Most likely it would produce a great candidate for a garbage-in-garbage-out type of a model.

So, before we can start with the fun part and train the model, we need to run some pre-processing. Because we know that the quality of the model can be only as good as the quality of the input data.

To improve the quality of the dataset we proceed with different pre-processing steps. We delete outliers, create new features from the raw data, impute missing values, reduce dimensionality, and much more including a number of automatic and machine learning approaches. It is all described in the video below.

Summarizing, this video is an overview of the pre-processing techniques needed before training a model and of the native KNIME nodes suitable implement them.


You Might Also Like

Migrating from Excel to KNIME Analytics Platform

In surveys about the most-used tool for data analysis Excel always comes in as one of the most commonly used tools . It is taught in schools and used by countle...

February 19, 2016 – by Iris Adä

What are you looking for?