没有合适的资源?快使用搜索试试~ 我知道了~
首页利用Hadoop进行大规模分布式深度学习实战指南
利用Hadoop进行大规模分布式深度学习实战指南
需积分: 10 8 下载量 18 浏览量
更新于2024-07-18
收藏 6.57MB PDF 举报
"《Deep Learning with Hadoop》是一本专为数据科学家量身打造的英文书籍,旨在帮助读者在大规模数据集上构建、实现并扩展分布式深度学习模型。该书不仅涵盖了深度学习的基本概念,还提供了如何利用Hadoop YARN框架进行深度学习模型的实施和并行化的方法。 书中首先介绍了深度学习的基本原理,包括深度前馈网络、各种学习算法(如监督学习、无监督学习和半监督学习)以及深度学习领域的专业术语。作者强调了深度学习在人工智能领域的重要性,特别是针对大数据中所面临的挑战,如维度灾难和梯度消失问题,以及分布式表示的优势。 作者深入剖析了分布式深度学习在处理大规模数据时的挑战,包括数据量大、计算复杂性增加等问题,并提供了解决策略。具体实践部分,本书详细讲解了如何使用deeplearning4j库来实现卷积神经网络(CNN)、受限玻尔兹曼机(RBM)、循环神经网络(RNN)等常见深度学习模型。此外,书中还提供了数学解释和视觉示例,以便读者理解RNN和去噪自编码器的设计和实现。 对于实战应用,书中的例子涵盖了大规模视频处理、图像处理和自然语言处理在Hadoop环境下的实际操作,让读者能够掌握如何在分布式系统中部署各种深度神经网络。全书采用循序渐进的方式,从基础概念出发,逐步提升读者的技能水平,通过丰富的实例帮助读者理解和掌握技术。 《Deep Learning with Hadoop》是一本全面的教程,适合希望在Hadoop平台上进行深度学习的实践者,无论他们对机器学习基础知识有一定了解,还是希望通过这本书深入了解如何在大规模数据环境中克服挑战并有效运用深度学习技术。"
资源详情
资源推荐
![](https://csdnimg.cn/release/download_crawler_static/10598488/bg10.jpg)
Conventions
In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles
and an explanation of their meaning.
Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles
are shown as follows: "The .build() function is used to build the layer."
A block of code is set as follows:
public static final String DATA_URL =
"http://ai.stanford.edu/~amaas/data/sentiment/*";
When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:
MultiLayerNetwork model = new MultiLayerNetwork(getConfiguration());
Model.init();
New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the
text like this: "In simple words, any neural network with two or more layers (hidden) is defined as a deep feed-forward network or feed-forward
neural network."
Note
Warnings or important notes appear in a box like this.
Tip
Tips and tricks appear like this.
![](https://csdnimg.cn/release/download_crawler_static/10598488/bg11.jpg)
Reader feedback
Feedback from our readers is always welcome. Let us know what you think about this book-what you liked or disliked. Reader feedback is
important for us as it helps us develop titles that you will really get the most out of. To send us general feedback, simply e-
mail feedback@packtpub.com, and mention the book's title in the subject of your message. If there is a topic that you have expertise in and you are
interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.
![](https://csdnimg.cn/release/download_crawler_static/10598488/bg12.jpg)
Customer support
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
Downloading the example code
You can download the example code files for this book from your account at http://www.packtpub.com. If you purchased this book elsewhere, you
can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
You can download the code files by following these steps:
1. Log in or register to our website using your e-mail address and password.
2. Hover the mouse pointer on the SUPPORT tab at the top.
3. Click on Code Downloads & Errata.
4. Enter the name of the book in the Search box.
5. Select the book for which you're looking to download the code files.
6. Choose from the drop-down menu where you purchased this book from.
7. Click on Code Download.
Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:
WinRAR / 7-Zip for Windows
Zipeg / iZip / UnRarX for Mac
7-Zip / PeaZip for Linux
The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Deep-Learning-with-Hadoop. We also have other
code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
Downloading the color images of this book
We also provide you with a PDF file that has color images of the screenshots/diagrams used in this book. The color images will help you better
understand the changes in the output. You can download this file from
https://www.packtpub.com/sites/default/files/downloads/DeepLearningwithHadoop_ColorImages.pdf.
Errata
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books-maybe a
mistake in the text or the code-we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and
help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata,
selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your
submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that
title.
To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search
field. The required information will appear under the Errata section.
Piracy
Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and
licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address
or website name immediately so that we can pursue a remedy.
Please contact us at copyright@packtpub.com with a link to the suspected pirated material.
We appreciate your help in protecting our authors and our ability to bring you valuable content.
Questions
If you have a problem with any aspect of this book, you can contact us at questions@packtpub.com, and we will do our best to address the
problem.
![](https://csdnimg.cn/release/download_crawler_static/10598488/bg13.jpg)
Chapter 1. Introduction to Deep Learning
"By far the greatest danger of Artificial Intelligence is that people conclude too early that they understand it."
--Eliezer Yudkowsky
Ever thought, why it is often difficult to beat the computer in chess, even for the best players of the game? How Facebook is able to recognize
your face amid hundreds of millions of photos? How can your mobile phone recognize your voice, and redirect the call to the correct person, from
hundreds of contacts listed?
The primary goal of this book is to deal with many of those queries, and to provide detailed solutions to the readers. This book can be used for a
wide range of reasons by a variety of readers, however, we wrote the book with two main target audiences in mind. One of the primary target
audiences is undergraduate or graduate university students learning about deep learning and Artificial Intelligence; the second group of readers
are the software engineers who already have a knowledge of big data, deep learning, and statistical modeling, but want to rapidly gain knowledge
of how deep learning can be used for big data and vice versa.
This chapter will mainly try to set a foundation for the readers by providing the basic concepts, terminologies, characteristics, and the major
challenges of deep learning. The chapter will also put forward the classification of different deep network algorithms, which have been widely used
by researchers over the last decade. The following are the main topics that this chapter will cover:
Getting started with deep learning
Deep learning terminologies
Deep learning: A revolution in Artificial Intelligence
Classification of deep learning networks
Ever since the dawn of civilization, people have always dreamt of building artificial machines or robots which can behave and work exactly like
human beings. From the Greek mythological characters to the ancient Hindu epics, there are numerous such examples, which clearly suggest
people's interest and inclination towards creating and having an artificial life.
During the initial computer generations, people had always wondered if the computer could ever become as intelligent as a human being! Going
forward, even in medical science, the need of automated machines has become indispensable and almost unavoidable. With this need and
constant research in the same field, Artificial Intelligence (AI) has turned out to be a flourishing technology with various applications in several
domains, such as image processing, video processing, and many other diagnosis tools in medical science too.
Although there are many problems that are resolved by AI systems on a daily basis, nobody knows the specific rules for how an AI system is
programmed! A few of the intuitive problems are as follows:
Google search, which does a really good job of understanding what you type or speak
As mentioned earlier, Facebook is also somewhat good at recognizing your face, and hence, understanding your interests
Moreover, with the integration of various other fields, for example, probability, linear algebra, statistics, machine learning, deep learning, and so
on, AI has already gained a huge amount of popularity in the research field over the course of time.
One of the key reasons for the early success of AI could be that it basically dealt with fundamental problems for which the computer did not require
a vast amount of knowledge. For example, in 1997, IBM's Deep Blue chess-playing system was able to defeat the world champion Garry
Kasparov [1]. Although this kind of achievement at that time can be considered significant, it was definitely not a burdensome task to train the
computer with only the limited number of rules involved in chess! Training a system with a fixed and limited number of rules is termed as hard-
coded knowledge of the computer. Many Artificial Intelligence projects have undergone this hard-coded knowledge about the various aspects of
the world in many traditional languages. As time progresses, this hard-coded knowledge does not seem to work with systems dealing with huge
amounts of data. Moreover, the number of rules that the data was following also kept changing in a frequent manner. Therefore, most of the
projects following that system failed to stand up to the height of expectation.
The setbacks faced by this hard-coded knowledge implied that those artificial intelligence systems needed some way of generalizing patterns
and rules from the supplied raw data, without the need for external spoon-feeding. The proficiency of a system to do so is termed as machine
learning. There are various successful machine learning implementations which we use in our daily life. A few of the most common and important
implementations are as follows:
Spam detection: Given an e-mail in your inbox, the model can detect whether to put that e-mail in spam or in the inbox folder. A common
naive Bayes model can distinguish between such e-mails.
Credit card fraud detection: A model that can detect whether a number of transactions performed at a specific time interval are carried
out by the original customer or not.
One of the most popular machine learning models, given by Mor-Yosef et al in 1990, used logistic regression, which could recommend
whether caesarean delivery was needed for the patient or not!
There are many such models which have been implemented with the help of machine learning techniques.
Figure 1.1: The figure shows the example of different types of representation. Let's say we want to train the machine to detect some empty
spaces in between the jelly beans. In the image on the right side, we have sparse jelly beans, and it would be easier for the AI system to
determine the empty parts. However, in the image on the left side, we have extremely compact jelly beans, and hence, it will be an extremely
difficult task for the machine to find the empty spaces. Images sourced from USC-SIPI image database
A large portion of performance of the machine learning systems depends on the data fed to the system. This is called representation of the data.
All the information related to the representation is called the feature of the data. For example, if logistic regression is used to detect a brain tumor
in a patient, the AI system will not try to diagnose the patient directly! Rather, the concerned doctor will provide the necessary input to the systems
according to the common symptoms of that patient. The AI system will then match those inputs with the already received past inputs which were
used to train the system.
Based on the predictive analysis of the system, it will provide its decision regarding the disease. Although logistic regression can learn and
decide based on the features given, it cannot influence or modify the way features are defined. Logistic regression is a type of regression model
where the dependent variable has a limited number of possible values based on the independent variable, unlike linear regression. So, for
example, if that model was provided with a caesarean patient's report instead of the brain tumor patient's report, it would surely fail to predict the
![](https://csdnimg.cn/release/download_crawler_static/10598488/bg14.jpg)
correct outcome, as the given features would never match with the trained data.
These dependencies of the machine learning systems on the representation of the data are not really unknown to us! In fact, most of our computer
theory performs better based on how the data are represented. For example, the quality of a database is considered based on how the schema
is designed. The execution of any database query, even on a thousand or a million lines of data, becomes extremely fast if the table is indexed
properly. Therefore, the dependency of the data representation of the AI systems should not surprise us.
There are many such examples in daily life too, where the representation of the data decides our efficiency. To locate a person amidst 20 people
is obviously easier than to locate the same person in a crowd of 500 people. A visual representation of two different types of data representation
is shown in the preceding Figure 1.1.
Therefore, if the AI systems are fed with the appropriate featured data, even the hardest problems could be resolved. However, collecting and
feeding the desired data in the correct way to the system has been a serious impediment for the computer programmer.
There can be numerous real-time scenarios where extracting the features could be a cumbersome task. Therefore, the way the data
are represented decides the prime factors in the intelligence of the system.
Note
Finding cats amidst a group of humans and cats can be extremely complicated if the features are not appropriate. We know that cats have
tails; therefore, we might like to detect the presence of tails as a prominent feature. However, given the different tail shapes and sizes, it is
often difficult to describe exactly how a tail will look like in terms of pixel values! Moreover, tails could sometimes be confused with the hands
of humans. Also, overlapping of some objects could omit the presence of a cat's tail, making the image even more complicated.
From all the above discussions, it can be concluded that the success of AI systems depends mainly on how the data are represented. Also,
various representations can ensnare and cache the different explanatory factors of all the disparities behind the data.
Representation learning is one of the most popular and widely practiced learning approaches used to cope with these specific problems.
Learning the representations of the next layer from the existing representation of data can be defined as representation learning. Ideally, all
representation learning algorithms have this advantage of learning representations, which capture the underlying factors, a subset that might be
applicable for each particular sub-task. A simple illustration is given in the following Figure 1.2:
Figure 1.2: The figure illustrates representation learning. The middle layers are able to discover the explanatory factors (hidden layers, in blue
rectangular boxes). Some of the factors explain each task's target, whereas some explain the inputs
However, dealing with extracting some high-level data and features from a massive amount of raw data, which requires some sort of human-level
understanding, has shown its limitations. There can be many such examples:
Differentiating the cry of two similar age babies.
Identifying the image of a cat's eye at both day and night time. This becomes clumsy, because a cat's eyes glow at night unlike during the
daytime.
In all these preceding edge cases, representation learning does not appear to behave exceptionally, and shows deterrent behavior.
Deep learning, a sub-field of machine learning, can rectify this major problem of representation learning by building multiple levels of
representations or learning a hierarchy of features from a series of other simple representations and features [2] [8].
剩余116页未读,继续阅读
![](https://profile-avatar.csdnimg.cn/022a3320888b4f62a6d1da07df395bb7_vnetoolxw_87.jpg!1)
vnetoolxw_87
- 粉丝: 4
- 资源: 16
上传资源 快速赚钱
我的内容管理 展开
我的资源 快来上传第一个资源
我的收益
登录查看自己的收益我的积分 登录查看自己的积分
我的C币 登录后查看C币余额
我的收藏
我的下载
下载帮助
![](https://csdnimg.cn/release/wenkucmsfe/public/img/voice.245cc511.png)
最新资源
- 多模态联合稀疏表示在视频目标跟踪中的应用
- Kubernetes资源管控与Gardener开源软件实践解析
- MPI集群监控与负载平衡策略
- 自动化PHP安全漏洞检测:静态代码分析与数据流方法
- 青苔数据CEO程永:技术生态与阿里云开放创新
- 制造业转型: HyperX引领企业上云策略
- 赵维五分享:航空工业电子采购上云实战与运维策略
- 单片机控制的LED点阵显示屏设计及其实现
- 驻云科技李俊涛:AI驱动的云上服务新趋势与挑战
- 6LoWPAN物联网边界路由器:设计与实现
- 猩便利工程师仲小玉:Terraform云资源管理最佳实践与团队协作
- 类差分度改进的互信息特征选择提升文本分类性能
- VERITAS与阿里云合作的混合云转型与数据保护方案
- 云制造中的生产线仿真模型设计与虚拟化研究
- 汪洋在PostgresChina2018分享:高可用 PostgreSQL 工具与架构设计
- 2018 PostgresChina大会:阿里云时空引擎Ganos在PostgreSQL中的创新应用与多模型存储
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈
![](https://img-home.csdnimg.cn/images/20220527035711.png)
![](https://img-home.csdnimg.cn/images/20220527035711.png)
![](https://img-home.csdnimg.cn/images/20220527035111.png)
安全验证
文档复制为VIP权益,开通VIP直接复制
![](https://csdnimg.cn/release/wenkucmsfe/public/img/green-success.6a4acb44.png)