KNIME Blog: general

KNIME general.

Will They Blend? Experiments in Data & Tool Blending. Today: Google Big Query meets SQLite. The Business of Baseball Games

Mon, 11/13/2017 - 10:16 admin

In this blog series we’ll be experimenting with the most interesting blends of data and tools. Whether it’s mixing traditional sources with modern data lakes, open-source devops on the cloud with protected internal legacy tools, SQL with noSQL, web-wisdom-of-the-crowd with in-house handwritten notes, or IoT sensor data with idle chatting, we’re curious to find out: will they blend? Want to find out what happens when IBM Watson meets Google News, Hadoop Hive meets Excel, R meets Python, or MS Word meets MongoDB?

Follow us here and send us your ideas for the next data blending challenge you’d like to see at willtheyblend@knime.com.

Today: Google Big Query meets SQLite. The Business of Baseball Games

Author: Dorottya Kiss, EPAM

The Challenge

They say if you want to know American society, first you have to learn baseball. As reported in a New York Times article, America had baseball even in times of war and depression, and it still reflects American society. Whether it is playing, watching, or betting on the games, baseball is in some way always connected to the lives of Americans.

According to Accuweather, different weather conditions play a significant role in determining the outcome of a baseball game. Air temperature influences the trajectory of the baseball; air density has an impact on the distance covered by the ball; temperature influences the pitcher’s grip; cloud coverage affects the visibility of the ball; and wind conditions - and weather in general - have various degrees of influence on the physical wellbeing of the players.

Another interesting article on Crowdhitter describes the fans’ attendance of the games and how this affects the home team’s success. Fan attendance at baseball games is indeed a key factor, in terms of both emotional and monetary support. So, what are the key factors determining attendance? On a pleasant day are they more likely to show up in the evening or during the day, or does it all just depend on the opposing team?

Some time ago we downloaded the data about attendance at baseball games for the 2016 season from Google’s Big Query Public data set and stored them on our own Google Big Query database. For the purpose of this blending experiment we also downloaded data about the weather during games from Weather Underground and stored these data on a SQLite database.

The goal of this blending experiment is to merge attendance data at baseball games from Google Big Query with weather data from SQLite. Since we have only data about one baseball season, it will be hard to train a model for reliable predictions of attendance. However, we have enough data for a multivariate visualization of the various factors influencing attendance.

Topic. Multivariate visual investigation of weather influence on attendance of baseball games.

Challenge. Blend attendance data from Google Big Query and weather data from SQLite.

Access Mode. Database Connector node with Simba 4.2 JDBC driver compatible with access to Google Big Query and dedicated SQLite Connector node.

Last Call. Your Flight is Boarding Now!

Mon, 11/06/2017 - 10:14 rs

Unless it is delayed, in which case, you can relax and read this vlog post.

How many flights are delayed each year?

How many flights are delayed at departure and how many are delayed at arrival?

Are some carriers more often delayed than others?

Are flights leaving on Thursdays more likely to be delayed than flights leaving on Sundays?

Are flights leaving Chicago airport more often delayed than flights leaving San Josè airport?

Could we use KNIME to interactively and graphically explore the airline data set and answer all - or at least most of - these questions?

Before we start with any kind of model training for more accurate predictions, it is always useful to examine the status quo and explore the kind of problem we are dealing with. This is where graphical interactive exploration comes in handy. Sunburst charts, box plots, line plots, stacked plots, scatter plots, network graphs, and other visualization techniques can offer some insights into the dataset and particularly into our delayed flights problem.

What about Data Access?

Mon, 10/30/2017 - 08:50 rs
  • Can KNIME connect to MySQL databases?
  • Sure! KNIME Analytics Platform has dedicated connectors for a number of databases and MySQL is one of them. We also have a generic connector for many other databases. Provided the JDBC driver file, KNIME can connect to most databases through this generic database connector node.
     
  • What about Microsoft SQL Server?
  • Sure! KNIME Analytics Platform has dedicated connectors for a number of databases, including MS SQL Server. Also, provided the JDBC driver file, KNIME can connect to other databases through a generic database connector node.
     
  • What about Oracle?
  • Sure! Provided the JDBC driver file, KNIME can connect to an Oracle database through the generic database connector node.
     
  • What about MongoDB?
  • Sure! KNIME Analytics Platform has a dedicated connector for MongoDB.
     

You don't have to choose! Blending KNIME and Python

Mon, 10/23/2017 - 10:35 greglandrum

This is going to be a bit different from our normal KNIME blog posts: instead of focusing on some interesting way of using KNIME or describing an example of doing data blending, I’m going to provide a personal perspective on why I think it’s useful to combine two particular tools: KNIME and Python. This came about because I keep getting questions like: “But you know Python really well, why would you use KNIME?” or “Now that you work at KNIME you aren’t really using Python anymore, right?”.

When to use one or the other?

So should you use Python or should you use KNIME?

Fortunately you don’t need to make this hard choice; it’s perfectly straightforward and, I think, quite productive to use both. It’s easy to take advantage of either tool from the other. I’ll spend most of the rest of this post looking at that. But there are areas where I think one tool or the other particularly shines.

KNIME Analytics: A Review

Mon, 10/16/2017 - 13:28 Kathrin

Let’s start this post with a question. How many different algorithms do you know that can solve classification problems? There are lots! Decision Tree, Random Forest, Deep Learning, Logistic Regression, just to name a few options. How to choose? It is hard to say in advance. 

Will They Blend? Experiments in Data & Tool Blending. Today: Finnish meets Italian and Portuguese through the Google Translate API. Preventing weather from getting lost in translation

Mon, 10/09/2017 - 11:18 admin

In this blog series we’ll be experimenting with the most interesting blends of data and tools. Whether it’s mixing traditional sources with modern data lakes, open-source devops on the cloud with protected internal legacy tools, SQL with noSQL, web-wisdom-of-the-crowd with in-house handwritten notes, or IoT sensor data with idle chatting, we’re curious to find out: will they blend? Want to find out what happens when IBM Watson meets Google News, Hadoop Hive meets Excel, R meets Python, or MS Word meets MongoDB?

Follow us here and send us your ideas for the next data blending challenge you’d like to see at willtheyblend@knime.com.

Today: Finnish meets Italian and Portuguese through the Google Translate API. Preventing weather from getting lost in translation

Advanced ETL Functionality and Machine Learning Pre-Processing

Mon, 10/02/2017 - 09:00 Kathrin

Today we look at a dataset that supposedly is already clean, joined with the right additional information, and in the right shape and we want to use it to train a prediction model. Unfortunately, a quick glance at the dataset reveals that it still has tons of missing values, it is not normalized, and contains too many too similar features.

Will They Blend? Experiments in Data & Tool Blending. Today: SugarCRM meets Salesforce. Crossing Accounts and Opportunities

Mon, 09/25/2017 - 10:30 RolandBurger

In this blog series we’ll be experimenting with the most interesting blends of data and tools. Whether it’s mixing traditional sources with modern data lakes, open-source devops on the cloud with protected internal legacy tools, SQL with noSQL, web-wisdom-of-the-crowd with in-house handwritten notes, or IoT sensor data with idle chatting, we’re curious to find out: will they blend? Want to find out what happens when IBM Watson meets Google News, Hadoop Hive meets Excel, R meets Python, or MS Word meets MongoDB?

Follow us here and send us your ideas for the next data blending challenge you’d like to see at willtheyblend@knime.com.

Today: SugarCRM meets Salesforce. Crossing Accounts and Opportunities

The Challenge

Businesses use Customer Relationship Management (CRM) systems to keep track of all their customer related activities – creating leads and opportunities, managing contacts and accounts, sending quotes and invoices, etc. As long as it is somehow related to the stream of revenue, it is (or at least should be) stored in a CRM system.

Since there is more than one CRM solution on the market, there is a distinct chance that your organization uses multiple CRM platforms. While there might be sound reasons for this, it also poses a significant challenge: How do you combine data from several platforms? How do you generate a single, consolidated report that shows you how well the sales activities of your whole company are going?

One option is to export some tables, fire up your spreadsheet software of choice, and paste the stuff together. Then do the same thing next week. And the week after. And the week after that one (you get the point). Doesn’t sound too enticing? Fear not! This is KNIME, and one of our specialties is to save you the frustration of doing things manually. Fortunately, both SugarCRM and Salesforce allow their users to access their services via REST API, and that is exactly what we are going to do in this blog post.

There are a couple of prerequisites here. First of all, you obviously need accounts for SugarCRM and Salesforce. If you don’t have them but still want to try this yourself, you’ll be happy to see that both companies offer free trial licenses:

https://info.sugarcrm.com/trial-crm-software.html?utm_source=crmsoftware&utm_medium=referral&utm_campaign=crmsoftware-review

https://developer.salesforce.com/signup

You can learn more about how to use the REST APIs of SugarCRM and Salesforce here:

http://support.sugarcrm.com/Documentation/Sugar_Developer/Sugar_Developer_Guide_7.9/Integration/Web_Services/v10/

https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/intro_what_is_rest_api.htm

Topic. Get a consolidated view of all customer data from two separate platforms

Challenge. Query data from SugarCRM and Salesforce via their APIs

Access Mode. KNIME REST Web Services

The Seven Steps to Model Management

Mon, 09/18/2017 - 14:20 berthold

We all know that just building a model is not the end of the line. However, deploying the model to put it into production is often also not the end of the story, although a complex one in itself (see our previous Blog Post on “The 7 Ways of Deployment”). Data scientists are increasingly often also tasked with the challenge to regularly monitor, fine tune, update, retrain, replace, and jump-start models - and sometimes even hundreds or thousands of models together.

Learning Deep Learning. A tutorial on KNIME Deeplearning4J Integration

Mon, 09/11/2017 - 09:10 jonfuller

The aim of this blog post is to highlight some of the key features of the KNIME Deeplearning4J (DL4J) integration, and help newcomers to either Deep Learning or KNIME to be able to take their first steps with Deep Learning in KNIME Analytics Platform.

Introduction

Useful Links

If you’re new to KNIME, here is a link to get familiar with the KNIME Analytics Platform:
https://www.knime.com/knime-online-self-training

If you’re new to Deep Learning, there are plenty of resources on the web, but these two worked well for me:
https://deeplearning4j.org/neuralnet-overview
http://playground.tensorflow.org/

If you are new to the KNIME nodes for deep learning, you can read more in the relevant section of the Node Guide:
https://www.knime.com/nodeguide/analytics/deep-learning

With a little bit of patience, you can run the example provided in this blog post on your laptop, since it uses a small dataset and only a few neural net layers. However, Deep Learning is a poster child for using GPUs to accelerate expensive computations. Fortunately DL4J includes GPU acceleration, which can be enabled within the KNIME Analytics Platform.

If you don’t happen to have a good GPU available, a particularly easy way to get access to one is to use a GPU-enabled KNIME Cloud Analytics Platform, which is the cloud version of KNIME Analytics Platform.

In the addendum at the end of this post we explain how to enable KNIME Analytics Platform to run deep learning on GPUs either on your machine or on the cloud for better performance.

Subscribe to KNIME Blog: general