From version 1.5.0 onwards, the Vernalis plugin for KNIME contains a number of 'timing' nodes in the flow control section. These are mainly 'Timed Loop' nodes, which loop until a time setting causes them to end, or the complete input table has been processed, and 'Delay' nodes, which simple wait until the time has passed. This page describes their function in more detail, and also provides some examples of how these nodes might be used.
General points
- The loop start nodes can be paired with conventional loop end nodes, or with the special timed loop end nodes.
- The loop start nodes emulate the corresponding KNIME core Chunked Loop and Table Row To Variable Loop starts
- The timed loop end nodes provide an additional port with any unprocessed rows from the loop start node
- The timed loop end nodes provide the last iteration counter, and the time at which loop execution ended as flow variables. (The delay nodes also provide this latter flow variable)
- It is important to node that it is the loop start node which indicates when the end time has passed. It is only checked once at the start of each loop iteration, so a loop processing a large number of rows or carrying out a very slow calculation can 'end' significantly after the designated end time.
- If the input table to a loop is completely processed before the end time is reached, then the loop execution will complete 'early' (i.e. it will not wait doing nothing until the end time) and any downstream nodes will execute.
- The Wait-to/for-time nodes have been deprecated as we donated them to the KNIME core product, where they have been replaced by a single 'Wait...' node
- The delay nodes can be inserted between any pair of nodes (or before any node(s), as the input port is optional) to pause workflow execution, using the normally hidden 'Mickey Mouse Ear' flow variable ports:
'For-time' or 'To-time'?
All the timed loop start nodes and the delay nodes have a 'For-time' and a 'To-time' variant, in which the timing behaviour is slightly different.
'For-time'
The 'for-time' variant means that the loop or delay will continue for a specified period of time, regardless of the starting time. The user enters the time as a number of days, hours, minutes and seconds in the 'Expiry Time' panel of the node dialog.
In the case of loops, this also contains a 'Start value of iteration counter' - allowing the user to generation a continuous 'iteration id' column at the loop end from successive runs of the workflow. It is important to note that this setting only changes the value at which the 'Current Iteration' flow variable starts, and does not influence in any way the processing of the input rows. The 'lastIteration' flow variable from the timed loop end can be saved and reloaded to populate this field using the Read/Write variables nodes.
'To-time'
In the 'To-time' variant, the user enters a time of day (in HH:mm) format, along with the behaviour to be expected if the time entered is before the time at which node or loop execution should start on the current day, in the 'Expiry Time' panel of the node dialog.
- 'Run to tomorrow' means that the node will run until the next occurence of the end time - i.e. on the following day, if that time has already expired today at the start of node or loop execution. If the time has not yet passed on the current day, then the node will only run until the specified time 'today', regardless of this setting
- 'Run through weekend' means that a node which would normally end it's execution on a Saturday or Sunday, based on its other settings (e.g. a node set to end at 18:45 which was started on a Saturday at 13:30, without 'run to tomorrow' set, or a node set to end at 07:00, with 'run to tomorrow' set, and execution started on a Friday) will continue to run until the indicated time on the following Monday
- Loop start nodes also have the 'Start value of iteration counter' as described in the 'For-time' section above.
Examples
The examples server contains an example worklow (099_Community-->11_Vernalis-->05_TimedLoops) showing three related example implementations, which are also described here for reference. The examples use 3-D conformer generation of molecules as an example of a time-consuming and processor-intensive process, but they are equally applicable to any other calculation.
In the first, an sd-file of 2-D molecules is read in, and then the workflow pauses until 18:00 hours before the processor-intensive step of conformer generation starts. (We could actually wait until 18:00 before reading the file by moving the 'Wait-to-time' node before the SDF Reader). The Chunk Loop Run-to-time Loop Start processed 1 entry at a time (we could chose more!), and ensures that the last molecule to be processed, regardless of whether the table is empty (if it is, then the loop will end at that point - it will not wait until the end time) is not started after 8am when the user wants their PC back.
The Timed Loop End (upto 4 ports) shows a new feature in the Vernalis loop end nodes - optional ports. These do not need to be connected. Their corresponding outputs either contain very empty (i.e. rowless and columnless) tables, or are inactive branches, according to the user setting (in this case, they are inactive branches, which serves to remind the user not to connect to them!) Two outputs are collected from the loop body (in this case we only actually do anything with the first one - which we save to an output file for later use; we could therefore disconnect the second table, as it is connected to an optional port). Finally, the last output port on the Timed Loop end (there is always one more output port than input ports on the timed loop end nodes) contains any rows from the input table to the loop start which were not processed because time ran out. In this case, these are saved, over-writing the input sdf-file which the SDF reader read, which means the user can simply reset and re-run the workflow to process further entries.
In the second example, this saving and manual reset/restart process is automated, by encasing the timed loop process above within a KNIME Recursive loop:
In this case, the Recursive loop End node passed back the unprocessed rows to the Recursive Loop Start. The workflow now waits until 18:00 before starting again to process the unprocessed rows from the previous night. The outer loop will now continue each night until all the input entries are processed.
Below shows modification to the start and end of the inner loop using the Read/Write variables nodes and a Flow Variable IF Switch (Flow Variable Value) to keep the iteration counter on the inner loop running continuously:
and
