KNIME® Cluster Executor provides a slim connection layer between KNIME Analytics Platform and your high-performance computing cluster, allowing every node and application integrated in a KNIME workflow to be distributed across the cluster.
Clusters often sit idle because of the enormous effort required to operate, maintain, and support applications on the grid, but distribution of compute-intensive tasks frees local resources and maximizes your IT infrastructure. KNIME Cluster Executor simplifies interaction with the cluster and reduces costs by moving compute-intensive or license-restricted computations from the data scientist’s machine to the distributed computing environment.
Cluster Executor Concept
Workflows created with KNIME Analytics Platform can specify local or distributed execution for each node. Select your cluster execution engine of choice from within the preference settings and use node-specific settings for more refined cluster use.
Data may be split into multiple subsets (see picture) and sent to the master server in the cluster. The master then coordinates task execution. Third-party nodes not normally offering cluster support may also be distributed in this way.
Parallel nodes in a single workflow can be executed on separate resources in the cluster. When remote execution of all tasks is finished, KNIME Analytics Platform collects the results from the remotely-executed nodes and completes workflow execution locally.
KNIME Cluster Executor is used by power-users across industries to provide high-speed calculation when mission critical results are time sensitive.
Workflows that process large data sets or require compute-intensive calculations are ideal candidates for KNIME Cluster Executor. This extension has been used successfully for 3-D modelling calculations and virtual docking experiments in pharmaceutical research.
Another area of use is the analysis and mining of huge data volumes for customer relationship management, telecom, and financial databases.
KNIME Software Integration
KNIME Cluster Executor is a commercial extension of KNIME Analytics Platform.
KNIME Server complements Cluster Executor by adding shared repositories, advanced access management, enhanced execution (scheduling), and web enablement (web portal and REST services). On completion, results may be downloaded in popular document formats including PDF, Word, Excel, or PowerPoint.
KNIME Cluster Executor is fully compatible with other KNIME Collaboration Extensions, including Server Lite, WebPortal, and TeamSpace.
Example workflow using KNIME Cluster Executor
KNIME Cluster Executor is compatible with Grid Engine derivatives supporting DRMAA (including Sun/Oracle/Univa Grid Engine). It requires Linux running on both Submit Clients and Cluster Slaves.