Partition based dataset implementation

We want to implement our dataset based on entire partition instead of key sets.

A main idea behind the partition based datasets is the classic MapReduce.

The most important advantage of the MapReduce is an ability to perform computations on a data distributed across the cluster without involving significant data transmissions over the network. This idea is adopted in the partition based datasets in the following way:

1. Every dataset consists of partitions.
2. Partitions consists of a context built on top of the Apache Ignite Cache and recoverable data stored locally on every node.
3. Computations needed to be performed on a dataset splits on Map operations which executes on every partition and Reduce operations which reduces results of Map operations into one final result.

Why partitions have been selected as a building block of dataset and learning contain instead of cluster node?

One of the fundamental ideas of Apache Ignite Cache is that partitions are atomic, which means that they cannot be splitted between multiply nodes. As result in case of rebalancing or node failure partition will be recovered on another node with the same data it contained on the previous node.

In case of machine learning algorithm it's very important because most of the ML algorithms are iterative and require some context maintained between iterations. This context cannot be split or merged and should be maintained in the consistent state during the whole learning process.

Another idea behind the partition based datasets is that we need to have data (in every partition) in BLAS-like format as much as it possible.

BLAS and CUDA makes machine learning 100x faster and more reliable than algorithms based on self-written linear algebra subroutines and it means that not using BLAS is a recipe for disaster. In other words we need to keep data in BLAS-like format at any price.