What's new in
File Handling Framework
The File Handling Framework is now out of Labs and ready to use in production. We've taken a big leap in making data access and transformation more powerful and user friendly.
- Easy to use different file systems and migrate from one system/cloud to another
- Managing of various file systems within the same workflow
- Powerful framework that allows us to integrate more and more file systems
- Consistent user experience across all nodes and file systems
- Performance improvements
KNIME has always supported the mixing and matching of different file systems within the same workflow. Now, with increasing numbers of users working with KNIME in cloud and hybrid environments, the File Handling Framework has been rewritten to give KNIME users a consistent experience across all nodes and file systems.
Different file systems: The migrated reader and writer nodes can work with all supported file systems. Whether your files live in the cloud, in your data center, on your local hard drive, or any mixture of these, KNIME provides you with a unified way to access your data.
Dedicated connector nodes: Using the dedicated connector nodes it's easy to migrate from one file system to another by simply changing the file connection.
Maximizing performance: Whenever we migrate an existing reader or writer node we leave no stone unturned to maximize its performance. For example, the new Excel Reader node, as well as many other migrated nodes, now supports streaming execution and reads the same data much faster than its predecessor.
Consistent UX and Features for Readers and Writers
- Read multiple files from a folder into a single KNIME data table
- Visual column transformations, e.g. filtering, sorting, renaming, and type mapping - all available in a responsive and easy to use interface that instantly shows changes in a preview window
This new framework provides a consistent way of creating a KNIME data table from a single file or multiple files. The screenshot below, for example, displays the column transformation changes seen instantly in the preview window.
If you want to apply all these transformations on any number of existing KNIME data tables, you can use the new Table Manipulator node.
Columnar Table Backend for Fast Tables (KNIME Labs)
- Improved performance
- Memory efficiency and in-memory processing
Supporting type richness and flexibility in KNIME Analytics Platform has always been a priority but brings overhead with it: suboptimal use of main memory as cell elements in a table are represented by Java objects. Therefore we have reviewed and optimized the underlying data representation.
The new Columnar Table Backend extension uses a different underlying data layer based on a columnar representation. Read more about the new extension in the blog article Improved Performance with New Table Backend Extension.
- Easier to share and productionize workflows and components containing Python Scripting nodes; Conda Environment Propagation node captures and replicates required dependencies alongside your workflows or components
- Multiple input and output of Python Scripting nodes
- Integrate functionality from Jupyter notebooks directly into KNIME workflows / Invoke KNIME workflows from Python or Jupyter notebooks
Users commonly need to juggle different Python environments and share different pieces of code from various parties. The new Python enhancements enable this.
You can customize Python environments locally and then duplicate and maintain those environments on KNIME Server so that your workflow’s Python dependencies are always satisfied whether running jobs locally in KNIME or remotely on a KNIME Server. Check out an example workflow on the KNIME Hub, Environment Propagation and Python Script Example.
Multiple and dynamic input/output of Python script. Our core Python Scripting nodes have been updated so that you can now add as many ports as you need.
knimepy + KNIME Server REST API: The new Python knimepy package adds the ability to execute remote workflows on a KNIME Server. Local data can be used as input; output is received back from the workflow as pandas DataFrames. Try out an example workflow on the KNIME Hub, Run Workflow with knimepy.
Extensions and Integrations
A number of extensions and integrations are now moving out of KNIME Labs. These include:
- The KNIME Deep Learning Extensions based on Keras and Tensorflow
- Our integrations with H2O, Tableau, and Salesforce
The Salesforce integration has taken a particularly fast track since its initial release in July 2020. Thanks to frequent feedback this integration now also comes with a Salesforce Simple Query node that significantly simplifies reading data from Salesforce. No knowledge of the Salesforce query language required!
Building Components - Dialog Layout
Components let you bundle functionality for sharing and reusing. In this release we've simplified building components by enabling the component author to (vertically) lay out elements in the dialog. Learn more about Components here.
Verified Components: Latest Additions and Highlights
Components can be designed to work just like KNIME nodes. They are built in a KNIME workflow, enabling you to easily bundle, reuse, and share functionality. Verified Components are developed and verified by KNIME and regularly released on the KNIME Hub.
Visit our Verified Components webpage to see the most recent additions and links to all Component categories on the KNIME Hub. Some highlights include:
- Interpretability: XAI View
- Life Sciences: Molecular Properties Filter
- Visualizations: Automated Visualization
- Guided Analytics: Generic File Upload
- Automation: Analyze Workflow Summary
- Greater flexibility when creating spaces and editing metadata from the browser
- Profile page benefits - status and liked items
- KNIME Hub now integrated into KNIME Analytics Platform
The KNIME Hub provides more convenience and flexibility by enabling browser functionality. You can create, rename, and delete your spaces from the browser. You can also edit your space metadata i.e. change the space title and description via a browser.
Your profile page - your place on the KNIME Hub to share information and collaborate - now includes stats and lists of the items on the KNIME Hub you have liked as well as the likes you have received for your own work.
The KNIME Hub is now also integrated in KNIME Analytics Platform. You can access the Hub to explore resources and best practices while you continue working in the Analytics Platform. Explore and install new nodes and extensions via drag&drop from the KNIME Hub directly into your installation of KNIME Analytics Platform.
KNIME Server: Monitoring Portal
- New monitoring portal with quick access, wherever you are, to overviews of jobs, schedules, executors, and executor groups
- Revised log file download now delivers precisely the information you need
- Change KNIME Server configurations conveniently from your browser
KNIME Server brings new monitoring capabilities ensuring that you have quick access to jobs, schedules, and executors wherever you are.
Jobs: Users can access a comprehensive overview of their jobs, including a newly revised table with searching and filtering to find what they are looking for - even on Servers running large numbers of jobs.
Schedules: Display all schedules and their associated jobs on a single page and also enable/disable schedules directly from your browser.
Executors: An overview of all KNIME Executors and Executor Groups allows easily monitoring of the status and health of your infrastructure. Check on which Executor a particular job was run - especially helpful if you have multiple Executors with different reservation rules.
Log file download: The revised log file download from the WebPortal enables you to retrieve log files for a particular date range to deliver only the information you need.
Administration portal for easier Server configuration: The admin portal has undergone a redesign to match the new WebPortal design. You can now change most KNIME Server configuration settings from your browser as well.
KNIME Server: RabbitMQ for High Availability
- New RabbitMQ High Availability functionality makes KNIME Server better suited for applications where high availability is crucial
This new functionality allows KNIME Server and KNIME Executors to connect to a new RabbitMQ instance should the original instance crash. Both KNIME Server and KNIME Executors can switch over to a new queue if the original queue goes offline.
KNIME Server Large and Executors on Azure
KNIME Server Large and KNIME Executors are now available in the Azure Marketplace. KNIME Executors work in conjunction with KNIME Server Large to execute workflows. These are available in two editions:
- Bring Your Own License (BYOL) and
- Pay As You Go (PAYG)
The PAYG Executors are purchased through the Azure Marketplace and support elastic scaling - enabling you to consume the compute capacity you need when you need it.