Neurocomputing 261 (2017) 164–170
Contents lists available at ScienceDirect
Neurocomputing
journal homepage: www.elsevier.com/locate/neucom
Distributed extreme learning machine with alternating direction
method of multiplier
Minnan Luo
a , ∗
, Lingling Zhang
a
, Jun Liu
a
, Jun Guo
b
, Qinghua Zheng
a
a
SPKLSTN Lab, Department of Computer Science, Xi’an Jiaotong University, Xi’an 710049, China
b
Hardware Department, School of Computer Science and Technology, Northwest University, Xi’an 710127, China
a r t i c l e i n f o
Article history:
Received 26 September 2015
Revised 16 March 2016
Accepted 22 March 2016
Available online 14 February 2017
Keywords:
Extreme learning machine
Neuron work
Alternating direction method of multiplier
a b s t r a c t
Extreme learning machine, as a generalized single-hidden-layer feedforward network, has achieved much
attention for its extremely fast learning speed and good generalization performance. However, big data
often makes a challenge in large scale learning of extreme learning machine due to the memory limi-
tation of single machine as well as the distributed manner of large scale data in many applications. For
the purpose of relieving the limitation of memory with big data, in this paper, we exploit a novel dis-
tributed model to implement the extreme learning machine algorithm in parallel for large-scale data set,
namely distributed extreme learning machine (DELM). A corresponding algorithm is developed on the ba-
sis of alternating direction method of multipliers which has shown its effectiveness in distributed convex
optimization. Finally, extensive experiments on some benchmark data sets are carried out to illustrate
the effectiveness and superiority of the proposed DELM method with an analysis on the performance of
speedup, scaleup and sizeup.
©2017 Elsevier B.V. All rights reserved.
1. Introduction
Extreme learning machine is a generalized single-hidden-layer
feedforward network, where the parameters of hidden layer fea-
ture mapping are generated randomly according to any continu-
ous probability distribution [1] instead of being tuned by gradient
descent based algorithms. As a result, extreme learning machine
achieves extremely fast learning speed and better performance of
generalization. The ELM technique performs effectively and have
been applied in many applications of machine learning such as
classification [2,3] , clustering [4] and regression [5] . C. W. Deng
and G. B. Huang highlighted the new trends of multi-layer learn-
ing with extreme learning machine [6] . Extreme learning machine
is also used in many real life applications, for example, S. Shaha-
boddin et al. use extreme learning machine to estimate the wind
speed distribution [7] ; Deng et al. proposed an efficient image
super-resolution approach based on extreme learning machine to
reconstruct the high-frequency components containing details [8] .
It is noteworthy that traditional extreme learning machine is of-
ten implemented on a single machine, and therefore it is inevitable
to suffer from the limitation of memory with large scale data set.
∗
Corresponding author.
E-mail addresses: minnluo@mail.xjtu.edu.cn (M. Luo), zhanglingling@stu.
xjtu.edu.cn (L. Zhang), liukeen@mail.xjtu.edu.cn (J. Liu), guojun@nwu.edu.cn
(J. Guo), qhzheng@mail.xjtu.edu.cn (Q. Zheng).
Especially in the era of big data, the data set scale is usually ex-
tremely large and the data is often very high-dimensional for de-
tailed information [9,10] . On the other hand, it is actually necessary
to deal with the data set in different machines due to the follow-
ing two reasons: (1) The data set is stored and collected in a dis-
tributed manner because of the large scale of applications; (2) It
is impossible to collect all of data together for the reason of con-
fidentiality and the data set can be only accessed on their own
machine. Based on the analysis above, how to implement extreme
learning machine with respect to the data set which located in dif-
ferent machines becomes a key problems.
In previous work, some parallel or distributed extreme learning
machine have been implemented to meet the challenge of large-
scale data set [11,12] . For example, Q. He et al. took advantages
of the distributed environment provided by MapReduce [13] and
propose an parallel extreme learning machine on the basis of
MapReduce via designing the proper key, value pairs [14] . X.
Wang et al. focused on the issue of parallel ELM and propose M
3
extreme learning machine on the basis of min-max modular net-
work, namely as [15] . This approach decomposes the classification
problem into several small subproblems and trains individual ELM
for each subproblem; in the end, M
3
-network is adopted to ensem-
ble the individual classifiers together. Additionally, A. Akusok et
al. exploited a complete approach which successfully utilize high-
performance extreme learning machine toolbox for big data [16] .
http://dx.doi.org/10.1016/j.neucom.2016.03.112
0925-2312/© 2017 Elsevier B.V. All rights reserved.