What's New in KNIME Analytics Platform 4.2 and KNIME Server 4.11.0

KNIME-Software-Release-July-2020-Hero

KNIME Hub: Spaces

On KNIME Hub you can now create multiple Spaces, name them as you like, and define whether they should be public or private.

Here are tschmidt’s Spaces. He's created public Spaces for different projects that he’s working on with other people: BIRT reporting, Clustering Analysis, and a Churn Prediction Analysis project. He's also created a private Space, My Private Analysis, for an individual project, which is indicated by the lock icon.

Note: you need to sign in to see your Spaces.

You can now also show your appreciation for content provided by other people. In addition to adding comments, you can now like workflows, nodes, components, extensions, and Spaces. This can also all be done from within KNIME Analytics Platform.

KNIME Hub: User Profile Pages & Community

Introducing the new user profile page: A central spot to find your own Spaces and extensions as well as those of others. Your user profile page has three sections:

  • Overview: your own statistics
  • Extensions: if you’ve developed extensions, these will appear here
  • Spaces: the various Spaces you've created

Here is tschmidt’s profile page. He's published 21 workflows, 6 components, and has received 5 likes!

KNIME Hub Profile Page

The Extensions section shows any extensions that you have uploaded. Here, for example, is Redfield’s page. Redfield is a KNIME Partner, who has developed different KNIME Extensions.

KNIME Hub User Profile Extensions

New Connector Nodes and File Handling Capabilities

 

Salesforce

KNIME Labs now provides a set of nodes to connect to Salesforce - the Salesforce Integration - to read data from one of the most popular CRM systems. Authentication can be done via OAuth2 interactive web authentication or by providing username and password. Using the Salesforce Object Query Language (SOQL) the user can construct simple but powerful query strings to define the data to read.

Install Integration from KNIME Hub

SharePoint

One of the newly integrated file systems in our new File Handling Framework is Microsoft SharePoint. Use the new SharePoint Connector node to manage, read, and write your files on SharePoint from within KNIME Analytics Platform.

Install Extension from KNIME Hub

Amazon DynamoDB Extension (Labs)

The Amazon DynamoDB nodes allow you to utilize the key-value and document database from within KNIME Analytics Platform. The extension comes with nodes that allow you to seamlessly access and manipulate your data and tables in Amazon DynamoDB.

Example workflow on KNIME Hub

SAP Reader (Theobald)

We’ve partnered with Theobald Software, one of the world’s leading experts in SAP integration, to develop a KNIME node that allows you to extract data from various SAP systems. The new SAP Reader (Theobald) node connects to the commercial Theobald Xtract Universal Server to extract data from SAP S/4HANA, SAP BW, SAP ERP/ECC, SAP R/3, mySAP, etc. into KNIME. Note that the SAP Reader can only be used with a Theobald Xtract Universal Server license that includes the generic web service destinations.

Example workflow on KNIME Hub

New File Handling Framework (Labs)

Here is the first set of nodes in the new File Handling Framework. This will make it much easier to work with files within KNIME whether local, on KNIME Server, on an on-premise shared driver, or in cloud storage. 

We’ve also revised many of the existing reader and writer nodes to be compatible with the different file systems. We’ve improved their usability and added new features such as a file preview or reading multiple files in directories.

When sharing workflows you no longer need to write KNIME URLs when referencing files and folders in a KNIME Server repository, workflow repository, or the workflow itself. Instead, select the starting point and browse for the files you want. KNIME workflows now provide a dedicated data area, which will make it easier to share data with your workflow.

The new nodes are in the KNIME Labs -> File Handling (Labs) category. More nodes and file systems will come soon. So give it a try and provide us with lots of feedback!

 

TensorFlow 2 Integration

The new TensorFlow 2 integration enables the use of the TensorFlow 2 Python package with a few dedicated nodes and the DL Python nodes.

The TensorFlow 2 Network Reader node can read networks from the SavedModel format and the H5 format from an arbitrary source. TensorFlow 2 networks can be easily assembled from TensorFlow Keras layers or carefully designed to perform complex functions with custom layer implementations and powerful TensorFlow-Hub layers (e.g. BERT). After fitting the network to the training data with the DL Python Network Learner, the TensorFlow 2 Network Executor can be used to run the prediction for unseen data. The trained network can be saved with the TensorFlow 2 Network Writer node to be used in other workflows or applications.

Find out more and download the nodes from the KNIME Hub. And view these example workflows to see how they are used.

 

 

Performance Improvements

Python Speed-up

Performance improvements mean that executing Python nodes is now much snappier, which is particularly relevant for situations where multiple Python nodes follow each other, are inside a loop, or in the case of node executions that don’t involve lots of data. Startup time has been noticeably reduced when executing any Python scripting nodes, the deep learning Python scripting nodes, Keras nodes, and the MDF Reader node.

This has been realized by two key changes: (1) KNIME now initializes and maintains a pool of Python processes in the background for use by individual Python nodes; and (2) for the many users employing the popular Anaconda or Miniconda Python distributions, KNIME now invokes Python directly without the costly delay of first running a configuration script.

Simple File Reader

The Simple File Reader is a viable alternative to the File Reader. It does have a slightly smaller set of options available, but performs better. The node is on average three times faster and, for certain inputs, can achieve tenfold speedups over the more generic File Reader node.

Example workflow on KNIME Hub

Joiner Node

A new Joiner node has been released in Labs. Benchmarks have shown speedups for in-memory joins of around 2x-6x compared to the previous implementation. It also provides new functionality:

  • Split output: The joiner can be configured to output the matched and unmatched rows (left and right) in separate output ports and makes checking both data quality and joiner configuration easier.
  • Merge join columns: When using outer joins, the joiner can be configured to merge join column pairs into single columns, resulting in more concise output. This eliminates the need for downstream column merge nodes and also avoids a common pitfall with the default configuration of the previous Joiner node when doing outer joins.
View node on KNIME Hub

UX Improvements

In this release, we’ve added new nodes and worked on improving the user experience with a number of existing nodes, for example:

Table Difference Finder

This new node allows you to compare the structure and data of two KNIME data tables and to output their differences.

Example workflow on KNIME Hub

String Manipulation (Multiple Column)

Similar to the Math Formula (Multi Column), new String Manipulation (Multiple Column) node allows you to automatically perform a string manipulation on multiple columns.

Example workflow on KNIME Hub

Dynamic Ports

The Column Appender and Merge Variables nodes now allow for dynamic addition of input ports.

Tableau

This Tableau integration builds on the new Tableau Hyper API and ensures better compatibility and effortless installation.

Install integration on KNIME Hub

PowerBI

Buidling on the integration released in December 2019, the dialog in the PowerBI Extension automatically only shows compliant dataset, i.e., push datasets and shows an information if the workspace contains non-compliant datasets.

Install extension on KNIME Hub

R

All R nodes now allow you to set an R environment different to the default environment via the “Advanced” tab.

Database nodes

Various improvements have been implemented around our Database nodes, which were requested by our users. Improvements include bulk copy support for Microsoft SQL Server when using the DB Loader node, support for boolean columns in the DB Row Filter node, and allow users to open the dialog of most database nodes without a valid database connection or when working with the Remote Workflow Editor.

Download the free and open source KNIME Analytics Platform 4.2 and try out these new features!

Integrated Deployment

The new Integrated Deployment Extension allows you to capture the parts of your workflow needed for running in a production environment. These captured subsets hold all the relevant settings and transformations. They can be combined with one another, executed, saved as local workflows, and deployed to a KNIME Server. Integrated Deployment enables you to:

  • Automate your workflow design and deployment
  • Reduce risks of error associated with moving from model creation to deploying production processes
  • Update your production workflows automatically

Download this example workflow, which features the new nodes for Integrated Deployment from the KNIME Hub.

Find out more on the Integrated Deployment web page, or read the blog articles:

 

 

Elastic and Hybrid Execution

KNIME Executors actually run workflows. For example when a user starts a workflow on KNIME Server, via a REST API, scheduling, or the WebPortal. With the release of KNIME Server 4.11, the capabilities of KNIME Executors have been extended. The new features include Executor Groups and Executor Reservation, AutoScaling on AWS, and Hybrid Installation. This allows us to meet the requirements of corporations looking to serve large numbers of diverse users.

Executor Groups

Executor Groups are a new feature that enhances KNIME Executors, enabling users to create dedicated groups of executors for different departments or business units. Workflows get automatically routed to the correct group, which has the appropriate level of resources for that specific business unit. Each group can contain one or more executors.

Executor Groups simplify the scaling process. It's now possible to have your entire organization on a single KNIME Server installation, with a distinct Executor Group for each organizational entity. See the graphic below showing Executor Groups for the Marketing, Finance, and Engineering divisions, for example.

KNIME-Server-Executor-Groups

Executor Reservation

With Executor Reservation, you can ensure that jobs with special requirements are sent to a suitable executor. This goes beyond the already existing workflow pinning, since executors can now refuse to accept jobs unless certain requirements are met. This way, you can ensure that special resources are not blocked by workflows that don’t actually need them. For example there is no need to block a powerful GPU executor with basic ETL jobs.

Auto Scaling on AWS (BYOL & PAYG) with KNIME Server Large

KNIME Executors have now been added to the AWS Marketplace as bring-your-own-license (BYOL) and pay-as-you-go (PAYG) allowing you to automatically and dynamically start up new Executors. Using AWS Auto Scaling, you can detect if new KNIME Executors need to be started, e.g. in the case of heavy system load (scale out). Conversely, Executors are shut down again once load decreases (scale in). In the case of PAYG, dynamic scaling makes it possible to save money on both instance and license costs.

Hybrid Deployments

The new marketplace offerings also introduce the possibility of running a hybrid on-prem/cloud setup. You can install KNIME Server on premise and start up Executors in the cloud (on-demand PAYG) as needed. This allows you to start special-purpose Executors (e.g. GPU) without having to maintain specialized hardware year round.

KNIME Server Large BYOL on AWS

KNIME Server Large has been added to the AWS Marketplace as a BYOL service. This can be used as a base image to which on-demand pay-as-you-go executors can be added for elastic scaling.

Improved Monitoring

We have improved logging to better monitor levels of activity among KNIME Executors and Executor Groups.

 

Workflow Summary

To support governance, compliance and reporting requirements, the Workflow Summary feature extracts and exports all the details about your workflows out of KNIME - from execution information, workflow metadata and node settings through models all the way to details about data sources and fields used and created. Workflow Summary can be extracted from the pulldown menu or, by involving KNIME Server you can automate extraction of the workflow summaries and generate regular reports. When integrating the KNIME WebPortal, your report creation can be interactive.

A workflow summary can be exported as a single JSON or XML document. Highlights of the information extracted include: 

  • Runtime environment (installed plugins, system properties)
  • Workflow metadata
  • Workflow annotations
  • List of all contained nodes with properties like
    • Name, type, state, …
    • List of node outputs and their spec
    • List of node successors (by node id)
  • A summary of the output data (if desired and available) - but not the data itself
    • It can be generated either
      • Via File-menu entry
      • Server REST-endpoint to generate the summary for jobs
KNIME Workflow Summary

Examples on the KNIME Hub include a REST API automation example as well as a reporting and an interactive example, which use a component to extract information from the summary. These can be modified and extended to suit the needs of your organization. Download example workflows from the KNIME Hub.

Guided Analytics

The KNIME WebPortal is KNIME’s main end user application for Guided Analytics and has now been newly implemented and redesigned with a focus on user interface and user experience. A large majority of Widget nodes have now been rewritten to give them a fresh look and feel. Colors, fonts, and logos can easily be adjusted with the new theming capabilities. This allows for consistent branding and styling of the new WebPortal and widgets. Note that existing workflows are backwards compatible with the new WebPortal and widgets. Also: the previous WebPortal (v4.10 ) is still available, but you can start using the new WebPortal alongside it!

Learn more

IT Administrators

OAuth

In December 2019, we first introduced the ability to use Single Sign-On (SSO) by authenticating users with an Identity Provider with OAuth/OpenID Connect capability. KNIME Server 4.11 now adds a great number of features to the available capabilities. For example:

  • Automatic OIDC endpoint discovery
  • Claim to group mapping (a configurable claim from the userinfo endpoint can now be configured to be used as group information)
  • Landing page for OIDC login

Apache Tomcat

In previous releases, the application server component of KNIME Server was based on Apache TomEE - the Java Enterprise Edition of Apache Tomcat. With this release, TomEE is replaced by the standard Apache Tomcat. In line with the switch to the standard Apache Tomcat, the use of the old EJB mountpoints to connect to KNIME Server is now discouraged in favor of the newer REST implementation. We've made migrating to a REST mountpoint very simple. Just log in with an existing EJB mountpoint and there'll be a prompt from which you can switch to REST with a single click. REST provides many benefits in comparison to EJB - particularly advantages in terms of performance and stability.

Switch to Qpid

Previous releases of KNIME Server used RMI to establish a connection between an application server and executor. RMI has now been replaced by an embedded message queue based on Apache Qpid. Rather than having direct communication between application server and executor, events such as requests for job execution are passed on via the message queue. Qpid technology comes bundled with the KNIME Server installer, so no additional setup is necessary.

 

Community and Partner Extensions

Here are the new Community and Partner Extensions available in KNIME Analytics Platform 4.2:

Conformal Prediction Extension by Redfield

The Conformal Prediction extension developed by our partner Redfield enables KNIME Analytics Platform users to apply conformal prediction techniques in their workflows. Learn more about Conformal Prediction and this extension in this webinar recording.

Install extension from KNIME Hub

Cumulocity Connector extension by tarent

The Cumulocity Connector extension developed by our partner tarent allows you to connect to the Cumulocity IoT platform by Software AG from KNIME Analytics Platform. To learn more about this extension take a look at the blog article: Cumulocity, IoT, and KNIME Analytics Platform.

Install extension from KNIME Hub

Neo4J by Redfield

The Neo4J extension, developed by our partner Redfield, allows you to connect to the Neo4J graph database, extract and write data from and to the database.

Install extension from KNIME Hub

Genentech Nodes by Genentech

The Chemalot extension developed by Genentech enables users to build workflows from command line tools, running on remote infrastructure. The extension generates nodes based on a user supplied xml file.

Install extension from KNIME Hub

General Release Notes

Changes to Java Runtime and KNIME Analytics Platform Upgrade Procedure

KNIME Analytics Platform 4.2 is distributed with an updated version of Java (1.8.0.252-b09) and Eclipse (2020-03 / 4.15). Due to these changes, updates from previous versions of KNIME Analytics Platform via the KNIME update manager are not supported. A full new installation is required.

Changed or Removed Functionality

  • Windows 32bit support discontinued due to the Eclipse update (see above).
  • Using an external (pre-installed) Chrome to view JavaScript views is no longer available due to API changes between different versions of Chrome. Instead, use the Chromium binaries provided via the KNIME update site (pre-installed on Linux and Windows).
  • Chromium binaries are no longer pre-installed on MacOS due to Apple’s notarization process. They can be installed separately using “File” -> “Install KNIME Extensions”
  • Spotfire (Labs) extension was removed. The functionality is now available through the the partner extension provided by Tibco, which can be found under “KNIME Partner Extensions”

API Changes

KNIME Analytics Platform 4.2 no longer uses “Buddy Classloading” to find nodes and types in third party extensions. Nodes and types need to be properly registered through the respective extension point. For an exemplary fix of this issue, see this code change. Failing to do so will likely cause (old) workflows containing those nodes to load these nodes using a “missing node placeholder”. 

The node extension API itself has remained stable. However, changes due to the update of Eclipse may trickle down into third party extensions (e.g. dependencies to Apache Batik or similar). In most cases, a simple fix of the dependency declaration fixes the problem. If these problems are more elaborate, then please post them in the developer forum.

Explore more

Changelog

Read the detailed changelog for this release of KNIME Software.

Read changelog

knime_icons_rz Download KNIME

Download the latest version of KNIME and try out the new release features.

Download now