The data science dilemma: Automation, APIs, or custom data science?

As companies place an increasing premium on data science, there is some debate about which approach is best to adopt — and there is no straight up, one-size-fits-all answer. It really depends on your organization’s needs and what you hope to accomplish.

There are three main approaches that have been discussed over the past couple of years; it’s worth taking a look at the merits and limitations of each as well as the human element involved. After all, knowing the capabilities of your team and who you’re attempting to serve with data science influences heavily how to implement it.

Authors: Paolo Tamagnini & Christian Dietz

Where To Get Answers to Your Data Science Questions?

When I start a new data science project with KNIME Analytics Platform, there are always a few questions I need to ask myself before I even pull in a single node to my blank workbench.

• “Can I train this kind of a model in KNIME?”
• “Which KNIME nodes will I need for this task?”
• “Has anyone else put together a use case like this with KNIME before?”
• “Can I download any KNIME workflows as inspiration?” 

To answer all these questions, all I need to do is ask the KNIME Hub. The KNIME Hub has been available at hub.knime.com since March 2019 but many new features have now been added with the release of KNIME Analytics Platform 4.0.

In this blog series we’ll be experimenting with the most interesting blends of data and tools. Whether it’s mixing traditional sources with modern data lakes, open-source devops on the cloud with protected internal legacy tools, SQL with noSQL, web-wisdom-of-the-crowd with in-house handwritten notes, or IoT sensor data with idle chatting, we’re curious to find out: will they blend? Want to find out what happens when IBM Watson meets Google News, Hadoop Hive meets Excel, R meets Python, or MS Word meets MongoDB?

Authors: Maarit Widmann (KNIME) and Alfredo Roccato (Data Science Trainer and Consultant)

Wheeling like a hamster in the data science cycle? Don’t know when to stop training your model?

Model evaluation is an important part of a data science project and it’s exactly this part that quantifies how good your model is, how much it has improved from the previous version, how much better it is than your colleague’s model, and how much room for improvement there still is.

In this series of blog posts, we review different scoring metrics: for classification, numeric prediction, unbalanced datasets, and other similar more or less challenging model evaluation problems.

In this blog series we’ll be experimenting with the most interesting blends of data and tools. Whether it’s mixing traditional sources with modern data lakes, open-source devops on the cloud with protected internal legacy tools, SQL with noSQL, web-wisdom-of-the-crowd with in-house handwritten notes, or IoT sensor data with idle chatting, we’re curious to find out: will they blend? Want to find out what happens when IBM Watson meets Google News, Hadoop Hive meets Excel, R meets Python, or MS Word meets MongoDB?

Author: Brendan Doherty, Seagate Technology

In this blog post I will discuss some of the processes and steps that were taken on the journey to embed KNIME in a High Volume Manufacturing Environment within Seagate Technology.

Seagate Technology are one of the world's largest manufacturers of electronic data storage technologies and solutions. Seagate Technology creates products and services that include network attached storage, high performance computing, data protection appliances, internal hard drives, backup and recovery services, flash storage, and related solutions. They are a vertically integrated company and have manufacturing plants based in many locations worldwide. The read/write heads for the HDDs are manufactured in two locations, one of which is in Derry City, Northern Ireland. These devices are highly complex, have a long manufacturing cycle time, and generate a lot of data during their fabrication. The plant has many different groups located at the site all of which use data from a wide variety of sources on a daily basis.

Author: Maarit Widmann

Wheeling like a hamster in the data science cycle? Don’t know when to stop training your model?

Model evaluation is an important part of a data science project and it’s exactly this part that quantifies how good your model is, how much it has improved from the previous version, how much better it is than your colleague’s model, and how much room for improvement there still is.

In this series of blog posts, we review different scoring metrics: for classification, numeric prediction, unbalanced datasets, and other similar more or less challenging model evaluation problems.

Author: Jeany Prinz

One of the great advantages of KNIME Analytics Platform is its ability to analyze diverse types of data. Today we want to move to the outer edge of the alphabet and look into data from the Zika virus. Our analysis is inspired by the Microreact project¹ “Zika virus in the Americas”² and is a nice use case for the exploration of epidemiological data with KNIME Analytics Platform. Epidemiology is the study of the distribution and determinants of health-related states or events³. In this post, therefore, we will answer the question: What routes did the Zika virus take as it spread across the globe, and how did its genetic makeup change the way it did? To this end, we will investigate and visualize both geolocational and phylogenetic data from the Zika virus. Using generic JavaScript nodes, we will create our own dynamic views and wrap them into a composite interactive view.

Even if you deal with very different data on a day-to-day basis, this blog post is still of high value, as we show how to increase the flexibility of your analysis using interactive generic JavaScript views.

Authors: Paolo Tamagnini, Simon Schmid, and Christian Dietz

Implementing a Web-based Blueprint for Semi-automated Machine Learning, using KNIME Analytics Platform

This article is a follow-up to our introductory article on the topic, “How to automate machine learning.” In this second post, we describe in more detail the techniques and algorithms happening behind the scenes during the execution of the web browser application, proposing a blueprint solution for the automation of the machine learning lifecycle.

The price to pay for automated machine learning (aka AutoML) is the loss of control to a black box kind of model. While such a price might be acceptable for circumscribed data science problems on well-defined domains, it might prove a limitation for more complex problems on a wider variety of domains. In these cases, a certain amount of interaction with the end users is actually desirable. This softer approach to machine learning automation — the approach we take at KNIME — is obtained via guided automation, a special instance of guided analytics.