大数据分析：超越Hadoop探索

需积分: 9 177 浏览量更新于2024-07-18 收藏 2.18MB PDF 举报

"Big Data Analytics Beyond Hadoop" 是一本探讨大数据分析领域超越Hadoop技术的书籍，由Vijay Srinivas Agneeswaran撰写。这本书可能深入讨论了在处理大规模数据时，如何利用各种技术和工具来提升数据分析的效率和深度。在当前的大数据环境中，Hadoop虽然作为基础架构的重要组成部分，但随着数据量的爆炸性增长和需求的多样化，仅依赖Hadoop已无法满足所有需求。"Beyond Hadoop" 暗示本书将介绍包括但不限于Spark、Flink、HBase、NoSQL数据库、流处理、实时分析以及云服务等更广泛的解决方案。这些技术能够补充和扩展Hadoop的功能，提供更快的数据处理速度、更强的实时分析能力，以及更灵活的数据存储和查询机制。书中的内容可能涵盖以下几个关键知识点： 1. **大数据处理框架**：除了Hadoop MapReduce之外，可能会详细讲解Apache Spark，它以其内存计算和低延迟特性成为大数据实时分析的首选。Spark SQL、Spark Streaming和MLlib库等组件可能也会被提及，它们分别用于结构化数据处理、实时流处理和机器学习。 2. **NoSQL数据库**：Hadoop通常与HBase、Cassandra等NoSQL数据库结合使用，以实现高效的数据存储和检索。书中可能介绍这些数据库的特性和使用场景，以及如何与Hadoop生态系统集成。 3. **流处理技术**：如Apache Flink，它支持事件时间处理和状态管理，适合处理连续不断的数据流。Flink的强一致性模型和低延迟处理能力可能在书中得到详细解释。 4. **大数据分析工具**：除了编程语言（如Java、Python、Scala）之外，可能还会讨论Pig、Hive等高级查询语言，它们提供了更便捷的方式来查询和分析Hadoop集群中的数据。 5. **云计算平台**：如Amazon Web Services (AWS)的EMR、Google Cloud Dataproc或Microsoft Azure HDInsight，它们提供了一键部署和管理Hadoop及相关服务的能力。书中可能涉及如何利用这些云服务进行大数据分析。 6. **数据可视化和BI工具**：例如Tableau、Power BI等，这些工具可以帮助用户更直观地理解和解读大数据分析结果。 7. **数据安全与治理**：大数据分析中数据的安全性、隐私保护以及数据治理策略也可能是讨论的重点，包括访问控制、数据加密和审计等方面。 8. **案例研究**：书中可能会包含实际业务场景中的案例，展示如何利用上述技术解决特定的大数据问题，帮助读者理解理论知识在实际工作中的应用。通过阅读此书，读者可以了解到大数据分析领域的最新发展和趋势，以及如何选择和应用合适的工具和技术来应对复杂的大数据挑战。对于希望提升大数据分析技能的专业人士来说，这是一本宝贵的资源。

the “dwarf” terminology that was used to characterize fundamental computational tasks in the

super-computing literature (Asanovic et al. 2006). These are the seven “giants”:

1. Basic statistics: This category involves basic statistical operations such as computing the

mean, median, and variance, as well as things like order statistics and counting. The operations

are typically O(N) for N points and are typically embarrassingly parallel, so perfect for Hadoop.

2. Linear algebraic computations: These computations involve linear systems, eigenvalue

problems, inverses from problems such as linear regression, and Principal Component Analysis

(PCA). Linear regression is doable over Hadoop (Mahout has the implementation), whereas PCA

is not easy. Moreover, a formulation of multivariate statistics in matrix form is difficult to realize

over Hadoop. Examples of this type include kernel PCA and kernel regression.

3. Generalized N-body problems: These are problems that involve distances, kernels, or

other kinds of similarity between points or sets of points (tuples). Computational complexity is

typically O(N

) or even O(N

). The typical problems include range searches, nearest neighbor

search problems, and nonlinear dimension reduction methods. The simpler solutions of N-body

problems such as k-means clustering are solvable over Hadoop, but not the complex ones such

as kernel PCA, kernel Support Vector Machines (SVM), and kernel discriminant analysis.

4. Graph theoretic computations: Problems that involve graph as the data or that can be

modeled graphically fall into this category. The computations on graph data include centrality,

commute distances, and ranking. When the statistical model is a graph, graph search is

important, as are computing probabilities which are operations known as inference. Some graph

theoretic computations that can be posed as linear algebra problems can be solved over Hadoop,

within the limitations specified under giant 2. Euclidean graph problems are hard to realize over

Hadoop as they become generalized N-body problems. Moreover, major computational

challenges arise when you are dealing with large sparse graphs; partitioning them across a

cluster is hard.

5. Optimizations: Optimization problems involve minimizing (convex) or maximizing

(concave) a function that can be referred to as an objective, a loss, a cost, or an energy function.

These problems can be solved in various ways. Stochastic approaches are amenable to be

implemented in Hadoop. (Mahout has an implementation of stochastic gradient descent.)

Linear or quadratic programming approaches are harder to realize over Hadoop, because they

involve complex iterations and operations on large matrices, especially at high dimensions. One

approach to solve optimization problems has been shown to be solvable on Hadoop, but by

realizing a construct known as All-Reduce (Agarwal et al. 2011). However, this approach might

not be fault-tolerant and might not be generalizable. Conjugate gradient descent (CGD), due to

its iterative nature, is also hard to realize over Hadoop. The work of Stephen Boyd and his

colleagues from Stanford has precisely addressed this giant. Their paper (Boyd et al. 2011)

provides insights on how to combine dual decomposition and augmented Lagrangian into an

optimization algorithm known as Alternating Direction Method of Multipliers (ADMM). The

ADMM has been realized efficiently over Message Passing Interface (MPI), whereas the Hadoop

implementation would require several iterations and might not be so efficient.

6. Integrations: The mathematical operation of integration of functions is important in big

data analytics. They arise in Bayesian inference as well as in random effects models. Quadrature

approaches that are sufficient for low-dimensional integrals might be realizable on Hadoop, but

not those for high-dimensional integration which arise in Bayesian inference approach for big

data analytical problems. (Most recent applications of big data deal with high-dimensional

data—this is corroborated among others by Boyd et al. 2011.) For example, one common

approach for solving high-dimensional integrals is the Markov Chain Monte Carlo (MCMC)

(Andrieu 2003), which is hard to realize over Hadoop. MCMC is iterative in nature because the

chain must converge to a stationary distribution, which might happen after several iterations

only.

7. Alignment problems: The alignment problems are those that involve matching between

data objects or sets of objects. They occur in various domains—image de-duplication, matching

catalogs from different instruments in astronomy, multiple sequence alignments used in

computational biology, and so on. The simpler approaches in which the alignment problem can

be posed as a linear algebra problem can be realized over Hadoop. But the other forms might be

hard to realize over Hadoop—when either dynamic programming is used or Hidden Markov

Models (HMMs) are used. It must be noted that dynamic programming needs

iterations/recursions. The catalog cross-matching problem can be posed as a generalized

N-body problem, and the discussion outlined earlier in point 3 applies.

To summarize, giant 1 is perfect for Hadoop, and in all other giants, simpler problems or smaller

versions of the giants are doable in Hadoop—in fact, we can call them dwarfs, Hadoopable

problems/algorithms! The limitations of Hadoop and its lack of suitability for certain classes of

applications have motivated some researchers to come up with alternatives. Researchers at the

University of Berkeley have proposed “Spark” as one such alternative—in other words, Spark

could be seen as the next-generation data processing alternative to Hadoop in the big data space.

In the previous seven giants categorization, Spark would be efficient for

• Complex linear algebraic problems (giant 2)

• Generalized N-body problems (giant 3), such as kernel SVMs and kernel PCA

• Certain optimization problems (giant 4), for example, approaches involving CGD

An effort has been made to apply Spark for another giant, namely, graph theoretic computations

in GraphX (Xin et al. 2013). It would be an interesting area of further research to estimate the

efficiency of Spark for other classes of problems or other giants such as integrations and

alignment problems.

The key idea distinguishing Spark is its in-memory computation, allowing data to be cached in

memory across iterations/interactions. Initial performance studies have shown that Spark can

be 100 times faster than Hadoop for certain applications. This book explores Spark as well as

the other components of the Berkeley Data Analytics Stack (BDAS), a data processing

alternative to Hadoop, especially in the realm of big data analytics that involves realizing

machine learning (ML) algorithms. When using the term big data analytics, I refer to the

capability to ask questions on large data sets and answer them appropriately, possibly by using

ML techniques as the foundation. I will also discuss the alternatives to Spark in this

space—systems such as HaLoop and Twister.

The other dimension for which the beyond-Hadoop thinking is required is for real-time

analytics. It can be inferred that Hadoop is basically a batch processing system and is not well

suited for real-time computations. Consequently, if analytical algorithms are required to be run

in real time or near real time, Storm from Twitter has emerged as an interesting alternative in

this space, although there are other promising contenders, including S4 from Yahoo and Akka

from Typesafe. Storm has matured faster and has more production use cases than the others.

Thus, I will discuss Storm in more detail in the later chapters of this book—though I will also

attempt a comparison with the other alternatives for real-time analytics.

The third dimension where beyond-Hadoop thinking is required is when there are specific

complex data structures that need specialized processing—a graph is one such example. Twitter,

Facebook, and LinkedIn, as well as a host of other social networking sites, have such graphs.

They need to perform operations on the graphs, for example, searching for people you might

know on LinkedIn or a graph search in Facebook (Perry 2013). There have been some efforts to

use Hadoop for graph processing, such as Intel’s GraphBuilder. However, as outlined in the

GraphBuilder paper (Jain et al. 2013), it is targeted at construction and transformation and is

useful for building the initial graph from structured or unstructured data. GraphLab (Low et al.

2012) has emerged as an important alternative for processing graphs efficiently. By processing, I

mean running page ranking or other ML algorithms on the graph. GraphBuilder can be used for

constructing the graph, which can then be fed into GraphLab for processing. GraphLab is

focused on giant 4, graph theoretic computations. The use of GraphLab for any of the other

giants is an interesting topic of further research.

The emerging focus of big data analytics is to make traditional techniques, such as market

basket analysis, scale, and work on large data sets. This is reflected in the approach of SAS and

other traditional vendors to build Hadoop connectors. The other emerging approach for

analytics focuses on new algorithms or techniques from ML and data mining for solving

complex analytical problems, including those in video and real-time analytics. My perspective is

that Hadoop is just one such paradigm, with a whole new set of others that are emerging,

including Bulk Synchronous Parallel (BSP)-based paradigms and graph processing paradigms,

which are more suited to realize iterative ML algorithms. The following discussion should help

clarify the big data analytics spectrum, especially from an ML realization perspective. This

should help put in perspective some of the key aspects of the book and establish the

beyond-Hadoop thinking along the three dimensions of real-time analytics, graph computations,

and batch analytics that involve complex problems (giants 2 through 7).

Big Data Analytics: Evolution of Machine Learning Realizations

I will explain the different paradigms available for implementing ML algorithms, both from the

literature and from the open source community. First of all, here’s a view of the three

generations of ML tools available today:

1. The traditional ML tools for ML and statistical analysis, including SAS, SPSS from IBM, Weka,

and the R language. These allow deep analysis on smaller data sets—data sets that can fit the

memory of the node on which the tool runs.

2. Second-generation ML tools such as Mahout, Pentaho, and RapidMiner. These allow what I

call a shallow analysis of big data. Efforts to scale traditional tools over Hadoop, including the

work of Revolution Analytics (RHadoop) and SAS over Hadoop, would fall into the

second-generation category.

3. The third-generation tools such as Spark, Twister, HaLoop, Hama, and GraphLab. These

facilitate deeper analysis of big data. Recent efforts by traditional vendors such as SAS

in-memory analytics also fall into this category.

First-Generation ML Tools/Paradigms

The first-generation ML tools can facilitate deep analytics because they have a wide set of ML

algorithms. However, not all of them can work on large data sets—like terabytes or petabytes of

data—due to scalability limitations (limited by the nondistributed nature of the tool). In other

words, they are vertically scalable (you can increase the processing power of the node on which

the tool runs), but not horizontally scalable (not all of them can run on a cluster). The

first-generation tool vendors are addressing those limitations by building Hadoop connectors as

well as providing clustering options—meaning that the vendors have made efforts to reengineer

the tools such as R and SAS to scale horizontally. This would come under the

second-/third-generation tools and is covered subsequently.

Second-Generation ML Tools/Paradigms

The second-generation tools (we can now term the traditional ML tools such as SAS as

first-generation tools) such as Mahout (http://mahout.apache.org), Rapidminer, and Pentaho

provide the capability to scale to large data sets by implementing the algorithms over Hadoop,

the open source MR implementation. These tools are maturing fast and are open source

(especially Mahout). Mahout has a set of algorithms for clustering and classification, as well as a

very good recommendation algorithm (Konstan and Riedl 2012). Mahout can thus be said to

work on big data, with a number of production use cases, mainly for the recommendation

system. I have also used Mahout in a production system for realizing recommendation

algorithms in financial domain and found it to be scalable, though not without issues. (I had to

tweak the source significantly.) One observation about Mahout is that it implements only a

smaller subset of ML algorithms over Hadoop—only 25 algorithms are of production quality,

with only 8 or 9 usable over Hadoop, meaning scalable over large data sets. These include the

linear regression, linear SVM, the K-means clustering, and so forth. It does provide a fast

sequential implementation of the logistic regression, with parallelized training. However, as

several others have also noted (see Quora.com, for instance), it does not have implementations

of nonlinear SVMs or multivariate logistic regression (discrete choice model, as it is otherwise

known).

Overall, this book is not intended for Mahout bashing. However, my point is that it is quite hard

to implement certain ML algorithms including the kernel SVM and CGD (note that Mahout has

an implementation of stochastic gradient descent) over Hadoop. This has been pointed out by

several others as well—for instance, see the paper by Professor Srirama (Srirama et al. 2012).

This paper makes detailed comparisons between Hadoop and Twister MR (Ekanayake et al.

2010) with regard to iterative algorithms such as CGD and shows that the overheads can be

significant for Hadoop. What do I mean by iterative? A set of entities that perform a certain

computation, wait for results from neighbors or other entities, and start the next iteration. The

CGD is a perfect example of iterative ML algorithm—each CGD can be broken down into daxpy,

ddot, and matmul as the primitives. I will explain these three primitives: daxpy is an operation

that takes a vector x, multiplies it by a constant k, and adds another vector y to it; ddot

computes the dot product of two vectors x and y; matmul multiplies a matrix by a vector and

produces a vector output. This means 1 MR per primitive, leading to 6 MRs per iteration and

eventually 100s of MRs per CG computation, as well as a few gigabytes (GB)s of communication

even for small matrices. In essence, the setup cost per iteration (which includes reading from

HDFS into memory) overwhelms the computation for that iteration, leading to performance

degradation in Hadoop MR. In contrast, Twister distinguishes between static and variable data,

allowing data to be in memory across MR iterations, as well as a combine phase for collecting all

reduce phase outputs and, hence, performs significantly better.

The other second-generation tools are the traditional tools that have been scaled to work over

Hadoop. The choices in this space include the work done by Revolution Analytics, among others,

to scale R over Hadoop and the work to implement a scalable runtime over Hadoop for R

programs (Venkataraman et al. 2012). The SAS in-memory analytics, part of the High

Performance Analytics toolkit from SAS, is another attempt at scaling a traditional tool by using

a Hadoop cluster. However, the recently released version works over Greenplum/Teradata in

剩余174页未读，继续阅读

粉丝: 0
资源: 2

大数据分析：超越Hadoop探索

vSphere Big Data Extensions与Hadoop最佳实践

大数据分析：Hadoop实战指南

HCIA-Big-Data：Hadoop分布式存储与考试重点

%5B搬书匠%231617%5D%5BBig Data Analytics Beyond Hadoop%5D.2014.英文版1

Pro Hadoop Data Analytics

Pro Hadoop Data Analytics(Apress,2016)

Sams.Teach.Yourself.Big.Data.Analytics.with.Microsoft.HDInsight

Handbook of Big Data Technologies

Hadoop in Practice

Manning.Hadoop.in.Practice.2012

最新资源