There is a new KNIME forum. You can still browse and read content from our old forum but if you want to create new posts or join ongoing discussions, please visit our new KNIME forum: https://forum.knime.com

Increase rule based row filter speed

Member for

2 years 5 months ActionAndi

Hi,

I am wondering if there is a better/faster way to filter my data.

The data-table consists of a campaign index, measurement index and some measurement data. Within each measurement campaign index there are hundreds of rows of measurement results .

Measurement Campaign Measurement Index Result
1 0 1
1 1 2
1 2 3
1 3 4
2 0 5
2 1 6
2 2 7
2 3 8
3 0 9

 

To do some mathematics for each measurement campaign I created a table with the measurement indexes, which I use for a table to variable loop. Inside the loop I use this variable to filter out the Measurement Results using the row filter node.

To use the variable inport in the row filter node, I had to change the measurement campaign index to string type.

As you can see this is a quite complex workflow. And because of the size (+500.000 rows) it takes quite a long time to do the filtering. Do you have some hints for me to increase the speed?

Andi

 

 

 

Comments
Fri, 03/02/2018 - 05:44

Member for

2 years 9 months

daria.goldmann

Hi Andi,

There are several things I'm a bit confused. Could you please attach a sample workflow to illustrate the problem?

From the data I see, I would recommend to use GroupBy node. In Groups Tab you could group data by Measurement Campaign (or Measurement Index or both). In the Aggregation Tab you would aggregate the values present in the result columns with a mathematical or string operation of your choice (Sum, Count, List, etc).

Best,

Daria