2019 INNS BDDL大会：大数据与深度学习前沿进展

需积分: 10 141 浏览量更新于2024-07-17 收藏 22.12MB PDF 举报

"《Recent Advances in Big Data and Deep Learning》是由Luca Oneto, Nicolò Navarin, Alessandro Sperduti和Davide Anguita编辑的一本论文集，收录了2019年国际神经网络学会(INNS)举办的大型数据与深度学习(INNS BDDL)会议上的39篇原始文章。这本书主要关注人工智能领域，特别是人工神经网络、大数据以及相关主题的研究进展。会议在意大利热那亚的塞斯特里莱万特举行，时间为2019年4月16日至18日。" 本书是国际神经网络学会系列出版物的第一卷，由Plamen Angelov和Robert Kozma担任系列编辑。该系列旨在发表关于神经网络基础原理和应用，以及与行为和大脑过程建模相关的研究贡献。涉及的主题包括新的发展、最先进的理论、方法和实际应用，涵盖了神经网络和仿生智能系统的所有方面。深度学习是现代人工智能中的一个关键分支，它通过模拟人脑神经元的工作方式来构建模型，能够在大量数据上进行训练，从而实现诸如图像识别、自然语言处理和自动驾驶等复杂任务。近年来，深度学习在计算机视觉、语音识别和推荐系统等领域取得了显著成就。例如，深度卷积神经网络（CNN）在图像分类和识别中表现出色，循环神经网络（RNN）和长短时记忆网络（LSTM）则在处理序列数据如文本和音频流方面具有优势。大数据则是另一个重要的研究领域，它涉及到如何收集、存储、处理和分析海量的数据。大数据技术如Hadoop和Spark提供了处理大规模数据集的框架，而NoSQL数据库适应了非结构化数据的管理。通过大数据分析，可以挖掘出隐藏的模式、趋势和关联，为决策制定提供依据。大数据与深度学习的结合，使得在海量数据中训练更复杂的模型成为可能，进一步推动了人工智能的发展。《Recent Advances in Big Data and Deep Learning》一书中的文章可能涵盖了大数据处理的新算法、深度学习模型的优化、模型解释性、计算效率提升、以及这两个领域的交叉应用，比如在医疗健康、金融风控、社交媒体分析等领域的实践案例。通过这些文章，读者可以了解到当前在大数据和深度学习领域的最新研究动态和技术趋势，对于研究人员和从业人员来说，这是一份宝贵的参考资料。

6 G. Gnecco and F. Nutarelli

4. Ruud, P.A.: An Introduction to Classical Econometric Theory, 1st edn. Oxford

University Press, Oxford (2000)

5. Varian, H.R.: Big data: new tricks for econometrics. J. Econ. Perspect. 28, 3–28

(2014)

6. Wilkinson, J.H.: The evaluation of the zeros of ill-conditioned polynomials. Part I.

Numer. Math. 1, 150–166 (1959)

Distributed SmSVM Ensemble Learning

Jeﬀ Hajewski

(

)

and Suely Oliveira

Department of Computer Science, University of Iowa, Iowa City, IA, USA

{jeffrey-hajewski,suely-oliveira}@uiowa.edu

Abstract. Traditional ensemble methods are typically performed with

models that are fast to construct and evaluate, such as random trees and

Naive Baye’s. More complex models frequently suﬀer from increased com-

putational load in both training and inference. In this work, we present

a distributed ensemble method using SmoothSVM, a fast support vector

machine (SVM) algorithm. We build and evaluate a large ensemble of

SVMs in parallel, with little overhead when compared to a single SVM.

The ensemble of SVMs trains in less time than a single SVM while main-

taining the same test accuracy and, in some cases, even exhibits improved

test accuracy. Our approach also has the added beneﬁt of trivially scaling

to much larger systems.

Keywords: Support vector machine

· Parallel ensemble learning ·

Distributed SVM · SmoothSVM

1 Introduction

Support Vector Machines (SVMs) are commonly known for their CPU intensive

workloads and large memory requirements. In the big data setting, where it is

common for data size to exceed available memory, these issues quickly become

major bottlenecks. Because of their memory requirements, it is desirable to par-

allelize the SVM algorithm for large datasets, allowing them to take advantage

of greater compute resources and a larger memory pool. In this work, we imple-

ment a distributed SmoothSVM (SmSVM) ensemble model. SmSVM [11]isan

SVM algorithm that uses a smooth approximation to the hinge-loss function and

an active-set approximation to the 

norm. It has two major advantages over

the standard SVM formulation: (1) the smoothness of the loss function permits

use of Newton’s method for optimization and (2) the active-set approximation of

the 

norm smoothes the loss function but not the 

penalty, yielding a sparse

solution. This reduces the computational requirements and memory footprint.

Eﬃcient communication patterns in large-scale distributed systems is a com-

plex and challenging problem in both design and implementation. In the dis-

tributed machine learning setting, unnecessary communication can consume

valuable network bandwidth and memory resources with little impact on training

time and test accuracy. Conversely, too little communication can result in poorly

performing models due to reduced information transfer from the dataset to the

 Springer Nature Switzerland AG 2020

L. Oneto et al. (Eds.): INNSBDDL 2019, INNS 1, pp. 7–16, 2020.

https://doi.org/10.1007/978-3-030-16841-4

8 J. Hajewski and S. Oliveira

models. We circumvent these issues by scaling the amount of data distributed

to each node inversely with the number of worker nodes and use a bootstrap

sample of the dataset rather than disjoint subsets. This approach improves the

training speed due to improved convergence during optimization while training

on fewer data points per model. Our method is able to maintain test accuracy via

the improved generalizability of the ensemble models (an artifact of the models

being trained on diﬀerent subsets of the original dataset) while decreasing train-

ing time. This work investigates the parallelization problem via a distributed

ensemble of SmSVM models, each trained on diﬀerent, but possibly overlap-

ping, subsets of the original dataset (known as bagging [5]). The advantages of

this technique in the distributed machine learning setting are:

– Improved generalization of the aggregated models

– Reduced network utilization via sub-sampling the data

– Reduced training time due to the concurrent construction of SVMs on sub-

sampled datasets

We parallelize SmSVM using message passing via MPI, rather than a

MapReduce-based framework such as Hadoop or Spark. This approach allows

our system to easily scale from running locally on multiple cores to running

on a large cluster without requiring reconﬁguration. Additionally, it avoids the

startup overhead commonly associated with MapReduce-based frameworks (see,

for example, [17]). The drawback of this approach is having to handle commu-

nication between nodes at a much lower level; however, in our experience this is

typically a worthwhile trade-oﬀ when performance is the primary goal.

2 Background

In this section we describe the SmSVM algorithm in detail. The standard SVM

loss function with both 

and 

regularizers is shown in (1).



i=1

max(0, 1 − y

ω · x

)+μ||ω||

||ω||

(1)

We use a diﬀerent formulation, called SmoothSVM (SmSVM) [11], which uses a

smooth approximation to the hinge-loss and maintains an active-set to approxi-

mate the 

regularizer. The hinge-loss function, ψ(x

, ω) = max(0, 1 − y

ω ·

), is non-smooth and thus not globally diﬀerentiable. Deﬁning u =1− yω · x,

the smoothed hinge-loss is given by Eq. (2).



(u)=

(u +





+ u

) (2)

The choice of  is determined via a relaxation method, where we choose a larger

value for  initially, ﬁnd an optimal solution ω, and then scale  by a relaxation

Distributed SmSVM Ensemble Learning 9

factor, β.Inpracticewefoundβ = 100 to work suﬃciently well. Using this

deﬁnition, and deﬁning φ(ω)asin(3)

φ(ω)=



i=1



(1 − y

(ω · x

)) + μ||ω||

||ω||

(3)

the problem we aim to solve is given by (4).

min

φ(ω) (4)

The advantage of this formulation is that the optimization problem is smooth,

which allows us to use Newton’s method, rather than solving the dual problem.

Solving (4) is equivalent to solving (5) under the requirement that the Hessian,

H(ω), is positive semi-deﬁnite.

∇

φ(ω) = 0 (5)

However, SmSVM does not require the entire Hessian matrix because the active

indices of ω are maintained via the active-set. Thus, we reduce the dimensionality

of H(ω) by restricting it to the non-zero components of ω. With this formulation

we can use Newton’s method [16], which is an iterative solver based on the

recurrence relation

n+1

= ω

− f



(ω

)

−1

f(ω

)

where f(ω)=∇φ(ω), f



(ω)=H(ω), and the weight update is given via

ω ← ω + sd

In the general Newton’s method implementation we typically have s = 1. Com-

puting H(ω) is expensive – we limit the cost of computing H(ω)byusingan

Armijo linesearch [3] algorithm to reduce the number of required steps. This

done by solving (6).

argmin

s≥0

ψ(ω + sd)+μ||ω + sd||

(6)

Replacing ψ(ω + sd) via a second-order approximation yields

ψ(ω + sd) ≈ as

+ bs + c + μ||ω + sd||

with s ≥ 0. For a ≥ 0, μ ≥ 0 this is a coercive convex function, which means

it has a global minimum. We ﬁnd this global minimum via binary search. The

binary search is computationally eﬃcient because it is with respect to s and thus

does not require any additional function or gradient evaluations.

Bootstrap aggregation (commonly referred to as bagging) is a statistical tech-

nique that improves a model’s ability to generalize to unseen data. Bagging

consists of a bootstrap phase and an aggregation phase. During the bootstrap

phase, data is sub-sampled uniformly with replacement from the original dataset.

In this work, we sample a subset that is inversely proportional to the number

10 J. Hajewski and S. Oliveira

of nodes (i.e., for n nodes, and a dataset with cardinality equal to |D|,each

subset has cardinality equal to |D|/|n|) and train a diﬀerent model in parallel

for each subset. During the inference/aggregation phase, each model generates

a prediction for the same data point, the mode of these predictions is computed

and used as the ﬁnal prediction of the bagged model for the given data point. If

there are an equal number of votes for each class, the master can choose a class

at random or train a model of its own to use as a tie-breaker.

3 Related Work

Yu et al. [22] propose a multiagent SVM ensemble in a sequential compute envi-

ronment. Their agents are designed to favor diversity by using disjoint subsets

from the training set. This approach improves the generalization of their model

but inhibits its scalability. As the number of agents grows, the size of the train-

ing set per agent decreases, eventually resulting in an agent-based model that

has little resemblance to the actual data distribution. Chen et al. [7]proposea

sequential, bagged SVM ensemble in the small data setting. Claesen et al. [8]

introduce an ensemble SVM library called EnsembleSVM with a focus on sequen-

tial eﬃciency via avoiding data duplication as well as duplicate support vector

evaluations. While this library achieves very impressive results, it is designed for

computational eﬃciency in the multicore setting rather than the distributed, big

data setting.

3.1 Hadoop and Spark

Distributed SVM architectures are widely studied. Perhaps the most popular

approach is a MapReduce-based architecture, using either Hadoop [18] or Spark

[23]. A number of implementations use a MapReduce framework via Hadoop

[1,2,9,12]. While MapReduce works well for many big data applications, it has

two bottlenecks: (i) data is stored on disk during intermediate steps and (ii) a

shuﬄe stage where data is shuﬄed between servers in a distributed sort These

two issues lead to decreased performance when running the iterative style algo-

rithms common to many machine learning algorithms. Speciﬁcally, highly itera-

tive algorithms will go through several phases of reading data from disk, process-

ing, writing data to disk, and shuﬄing for each iteration. Other work utilizes the

MapReduce framework via Spark [13,15,20,21]. Spark is built on top of Hadoop

and attempts to keep its data in memory, which alleviates the disk IO bottle-

neck. However, Spark can still incur communication overhead during the shuﬄe

stage of MapReduce and, if the data is large enough or there are too few nodes,

Spark may spill excessive data to disk (when there is more data than available

RAM). While this prior work shows promising results, the use of a MapReduce

framework can be sub-optimal. Reyes-Ortiz et al. [17] show that MPI generally

outperforms Spark in the distributed SVM setting, seeing as much as a 50×

speedup.

剩余401页未读，继续阅读

coolcnc

粉丝: 4
资源: 11

2019 INNS BDDL大会：大数据与深度学习前沿进展

Recent Advances in Deep Learning: An Overview

Big Data and Computational Intelligence in Networking-CRC(2018).pdf

【Network Architecture】: Delving into DCGAN and Its Variants: Exploring the Diversity and Potential...

Time Series Chaos Theory: Expert Insights and Applications for Predicting Complex Dynamics

Financial Model Optimization Using MATLAB's Genetic Algorithm: Strategy Analysis and Maximizing ...

Challenges and Solutions for Multi-Label Classification Problems: 5 Strategies to Help You Overcome ...

【Project Practicality】: New Horizons in Image Transformation: A Practical Guide to the Application...

Origin教程009所需练习数据

大模型的稀疏激活方法及其高效推理应用研究：基于dReLU激活函数

STM32F103+PWM+DMA精准控制输出脉冲的数量和频率 源程序

最新资源

STM32F103+PWM+DMA精准控制输出脉冲的数量和频率源程序