10 J. Hajewski and S. Oliveira
of nodes (i.e., for n nodes, and a dataset with cardinality equal to |D|,each
subset has cardinality equal to |D|/|n|) and train a different model in parallel
for each subset. During the inference/aggregation phase, each model generates
a prediction for the same data point, the mode of these predictions is computed
and used as the final prediction of the bagged model for the given data point. If
there are an equal number of votes for each class, the master can choose a class
at random or train a model of its own to use as a tie-breaker.
3 Related Work
Yu et al. [22] propose a multiagent SVM ensemble in a sequential compute envi-
ronment. Their agents are designed to favor diversity by using disjoint subsets
from the training set. This approach improves the generalization of their model
but inhibits its scalability. As the number of agents grows, the size of the train-
ing set per agent decreases, eventually resulting in an agent-based model that
has little resemblance to the actual data distribution. Chen et al. [7]proposea
sequential, bagged SVM ensemble in the small data setting. Claesen et al. [8]
introduce an ensemble SVM library called EnsembleSVM with a focus on sequen-
tial efficiency via avoiding data duplication as well as duplicate support vector
evaluations. While this library achieves very impressive results, it is designed for
computational efficiency in the multicore setting rather than the distributed, big
data setting.
3.1 Hadoop and Spark
Distributed SVM architectures are widely studied. Perhaps the most popular
approach is a MapReduce-based architecture, using either Hadoop [18] or Spark
[23]. A number of implementations use a MapReduce framework via Hadoop
[1,2,9,12]. While MapReduce works well for many big data applications, it has
two bottlenecks: (i) data is stored on disk during intermediate steps and (ii) a
shuffle stage where data is shuffled between servers in a distributed sort These
two issues lead to decreased performance when running the iterative style algo-
rithms common to many machine learning algorithms. Specifically, highly itera-
tive algorithms will go through several phases of reading data from disk, process-
ing, writing data to disk, and shuffling for each iteration. Other work utilizes the
MapReduce framework via Spark [13,15,20,21]. Spark is built on top of Hadoop
and attempts to keep its data in memory, which alleviates the disk IO bottle-
neck. However, Spark can still incur communication overhead during the shuffle
stage of MapReduce and, if the data is large enough or there are too few nodes,
Spark may spill excessive data to disk (when there is more data than available
RAM). While this prior work shows promising results, the use of a MapReduce
framework can be sub-optimal. Reyes-Ortiz et al. [17] show that MPI generally
outperforms Spark in the distributed SVM setting, seeing as much as a 50×
speedup.
我的内容管理
展开
