The first meta-node, named "Read all Data", loops over all files to read and concatenate their content. The output of that meta-node is a massive data table of 176 millions rows containing all available raw data. In order to read all those rows from 6 files, the loop takes only half an hour. Which is really astonishing considering the amount of data!

The second meta-node, named "String to Datetime", converts all date/ time values (days, months, year, hours, and minutes) from the original proprietary format into a KNIME DateTime object. The last node of the sub-flow contained in this meta-node is a "Sorter" node to sort all rows in ascending order by time. All date/time transformation nodes were executed relatively quickly, using all together up to 2 hours. The "Sorter" node on the opposite represented the bottle neck of the whole sub-workflow, using almost 5 hours.

Finally, the aggregations. Two parallel meta-nodes aggregate the data rows into a daily, monthly, weekly, yearly time series and into an hourly time series respectively and calculate the amount of energy used during week-end vs. businees days, during each day of the week, and during morning, afternoon, evening, night, early morning, and late afternoon in average. These two meta-nodes also calculate new time series for each meter ID on a daily, monthly, weekly, yearly, and hourly scale. Both nodes are massively time consuming, due to all aggregations performed in terms of "GroupBy" and "Pivoting" nodes.

The "GroupBy", "Sorter", and "Pivoting" all share the same sorting algorithm and might subtract resources from each other, if running at the same time. This effect is not noticeable for smaller data, even for 50 millions rows. But with 176 millions rows it
quickly becomes a problem. In this case, you need to execute one node after the other, either manually or setting artificial dependencies by means of flow variable connections
across nodes. Still, even after inserting dependencies, these two meta-nodes took almost 3 days in total  to run.

All calculated values are then joined together and a few additional percentages are also inserted. The execution of this meta-node, named "% values", adds just a few hours to the total execution of the workflow.

The good news is that KNIME does not break. If you have enough time and patience you can still run your workflow on 176 millions rows, slowly but without major issues. In other projects using very large amounts of data, for example 50 millions rows, KNIME has never crashed and the execution times on the laptop above have always been acceptable.

The same Workflow using "RushAccelerator for KNIME"
However, for this project I had only one week available to present some kind of results. Three days of execution time were not really possible. For the first time, then I decided to approach the KNIME solution for Big Data, named RushAccelerator for KNIME and produced by Actian (http://bigdata.pervasive.com/Products/RushAccelerator-for-KNIME.aspx).
You can download a free trial or buy a license from their web site. Actually I downloaded what is now an old version of the "RushAccelerator for KNIME" software, when the company was still called Pervasive. Nowadays there is a new version with many more nodes available and a brand new company name "Actian".

"RushAccelerator for KNIME" installs a new category in your KNIME "Node Repository", named "Pervasive DataRush" that contains a few specialized nodes to run on a big data platform (see picture on the left).
With those, I rebuilt my workflow using the RushAccelerator for KNIME nodes for datetime manipulation, values calculation/extraction, and most of all aggregations.

On the plus side, you can mix most of KNIME nodes and RushAccelerator nodes without problems and some nodes have a similar GUI as their corresponding KNIME nodes.

On the minus side, the configuration window of some RushAccelerator nodes is quite different with respect to the configuration window of the corresponding KNIME nodes, like for example the "Rows to Columns" node which is supposed to perform the same task as the "Pivoting" node. However, the highly time consuming tasks were only a few, repeated many times. In addition, RushAccelerator is not for free, but license costs are affordable.

RushAccelerator nodes also have a new tab in the configuration window, named "Job Manager Selection". Here you can set the engine you intend to run the workflow on: either the default engine or the DataRush executor.

The new workflow built with RushAcceletor nodes is shown in the figure below. The structure remained the same as the original workflow and most of the nodes contained in the meta-nodes actually mirror the original KNIME nodes.

The new Execution Time
And now about the execution time!
The result was quite impressive. The execution time of the whole workflow went down from circa 3 days to one hour and 16 minutes!
That was amazing and moved my attitude towards Big Data from full skepticism into cautiously positive.
I did find an example here where Big Data really made the difference between the success of the project and a total failure!

This does not mean that all workflows need to run on some Big Data platform. However, depending on the amount of data, available time and machine, sometimes you might wish to speed up execution times enough to get results in a reasonable time!

And here the same example but implemented with a "Java Snippet" node.

f you have data files on the server (we will see in another post how to place data files on the server), you should see them in the KNIME Explorer when you log in.
Now, if you drag and drop any of the data files into your workflow editor, you will automatically get a "File Reader" node configured to read this file directly from the server. Notice that the URL used in the "File Reader" node does not use the "file://" protocol anymore, but the new "knime://" protocol which refers to a KNIME server. You should get something like:    
       knime://knime-server-demo/KNIME_Server/data/Sentiment_Rating.csv

However, if you want to read the file on the server in a more dynamic way, you can use the "Explorer Browser" to browse your server and look for the directory containing your file. Below, you can see the configuration window of the "Explorer Browser" node. The "Browse" button allows you to browse the workflows and files on the server. The "Explorer Browser" node outputs a flow variable containing the defined path. This path can then be used as setting in a "File Reader" node.

Once you have your data, at the end of your workflow execution, you might want to save them again on the server. To do that, you need:
1. to create a temporary local folder with the "Create Temp Dir" node
2. to transform the folder path into a file path by adding a filename (I used here a "Java Edit Variable" node)
3. to write the data into your just defined local filepath via a "CSV Writer" node
4. to upload the temporary file to its final destination on the server using an "Explorer Writer" node.
Below is the configuration window of the "Explorer Writer" node. It requires:
- the filepath of the file to upload in the form of a flow variable
- the target location on the server

And below you can see the full workflow that I used to read and write data from and back to the KNIME server.

Reading the Data
The data consists of 6 files. Reading the full data set means to loop across all 6 files and collect the read data. Reading a subset of the data means reading only one of the files.
The input then comes from a "List Files" node. The "List Files" node produces the list of files available in a selected location on your machine (see picture below).
The selected location is inserted in the configuration window of the "List Files" node and the node produces a data table where each row contains the file path to one of the files in the selected location.

At this point, a "TableRow To Variable Loop Start" node translates each file path into a variable and starts a loop across them, where a "File Reader" node reads the file, and the "Loop End" node collects the results. This would be one branch of the switch.

How to connect a File Reader to a switch node?
The other branch just reads one of the possible file. The problem here is: how do I force a "File Reader" node to follow the enabling/disabling of the switch node?
The switch node outputs a data table and all connected nodes will be disabled/enabled depending on the switch node status. However, the "File Reader" node does not take any input, so it cannot be connected to the switch node.

The solution to this problem is to make the connection via a Flow Variable. The data table with all paths, output by the switch node, is then converted into a flow variable with a "TableRow To Variable" node. The output flow variable, containing only the first path of the switch output data table, is passed to the "File Reader" node and used to set the file path. In this way, if the switch output port is disabled, the whole branch, including the "File Reader" node, is disabled.

Finishing the workflow
The switch block is then closed by and "END IF" node to collect the resulting data.

A "Java Edit variable" node is used to transform the initial selection between "full" and "partial" and to control the switch node output ports.
The final workflow is shown in the figure below.

This little workflow implements a selective reading.
If we choose "full", all data are read by looping on all available files.
If we choose "partial", only the content of the first file is read and imported into KNIME.

The “Boosting Learner Loop End” node (see figure below) sets the maximum number of iterations, the target column, and the predicted column. The target column and the predicted column are used to:
    -          Identify the mis-classified patterns
    -          Calculate the model error
    -          Calculate the model weight
The “Boosting Learner Loop Start” node uses the model weight and the mis-classified patterns to alter the composition of the training set.

The loop body includes any supervised training algorithm node, like a “Decision Tree Learner” or a “Naïve Bayes Learner”, and its corresponding predictor node. The predictor node is necessary, even though this is a learning meta-node, because at each iteration the identification of correctly and incorrectly classified patterns and the model error calculation are needed.

For each iteration the boosting loop outputs the model, its error, and its weight.

The “Boosting Predictor” meta-node receives the model list from the learner node and the test set patterns. For each test pattern, it loops on all models and weighs their prediction result.
The “Boosting Predictor Loop Start” node starts the boosting predictor loop by identifying the weight column and the model column (see settings in its configuration window).

The “Boosting Predictor Loop End” node implements the majority vote on all model results and assigns the final value to the test pattern. Its configuration window requires the identification of the prediction column.

The loop body just includes the predictor of the mining model selected for the learning phase.

Below you can see our implementation of boosting in KNIME, with a decision tree on the cars-85.csv data set. The task here was to predict a car fuel system based on its number of doors, wheel base, and width, by using maximum 5 models. Boosting produced 5 models (the maximum number allowed) and 0.714 accuracy.

If we used a simple decision tree alone for the same task, we would have also scored 0.714 accuracy. This shows that boosting should be used only for complex problems or weak classifiers. For strong classifiers and simple problems, the additional models only model noise and outliers.

The upper input port receives the training set data, while the lower input port receives the test data.
The output port presents the results for all the trained models and the final result after the majority vote.


The sub-workflow of the “Bagging” meta-node
The “Bagging” meta-node consists of two steps: training a number X of models and applying the X models to the test set.

The training step
The goal of the training step is to create X partitions of the training data and to train X models on them.

    -    This step starts with shuffling the data. Shuffling is necessary, otherwise the X partitions of training data might only represent a subset of the whole data universe.
    -     The “Chunk Loop Start” node then divides the training data set in a number of X partitions and loops around them.
    -     At each iteration, a partition is used to train a model. In this case, the model is a decision tree, but it could be any other classifier scheme.
    -      Finally, the “Model Loop End” node collects all models resulting from the learner node at each iteration.

The training loop then produces a list of X models, each one trained on a different partition of the training set.

The test step
This is also a loop running across the X developed models.

     -        The “Model Loop Start” node starts a loop on the models listed in the input data. At each iteration, one model of the list is loaded.
     -        The predictor node applies the current model to the test set. Notice that the predictor node is connected to the lower input port of the “Bagging” meta-node. This is because it has to operate on the test set.
     -        The loop is closed by a “Voting Loop End” node. This node collects on different columns the results of the predictor node using different models. Therefore its output data table shows as many columns as many classification models. In addition, the node also performs the majority vote, that is it outputs the final classification result for each test data row on the basis of how many classifiers agree about a given output class.

The Results
Below the output data table is shown for a “Bagging” node using 10 decision trees.
The parameter defining the number of classifiers to use is the number of data chunks in the “Chunk Loop Start” node.

Parameterizing the “Bagging” meta-node

The “Bagging” node can be easily parameterized by using a Quickform “Integer Input” node. This Quickform node defines the number of models to be created and controls the number of data chunks in the “Chunk Loop Start” node (see sub-workflow below).

Below, the configuration window of the newly parameterized “Bagging” meta-node is shown.

Comparing with the simple version of the same classifier
In the following workflow, we compare the performance of our parameterized “Bagging” node using decision trees vs. a simple decision tree configured similarly to the decision trees in the “Bagging” node.
Notice that the “Bagging” node includes both training and testing phases, unlike the simple decision tree technique.
The “bagging” strategy display an accuracy of 0.835, slightly better than the accuracy of 0.821 displayed by the simple decision tree, as it was to be expected.