Cloudera Spark官方文档：深入SparkGuide

5星 · 超过95%的资源需积分: 9 93 浏览量更新于2024-07-20 收藏 1.28MB PDF 举报

"cloudera-spark 官方文档" Cloudera Spark官方文档是一份详尽的指南，专为使用Spark在Cloudera平台上进行大数据处理的用户设计。该文档覆盖了Spark的主要功能、配置、操作以及最佳实践，旨在帮助开发者和数据工程师充分利用Spark的强大性能和灵活性。 Spark是Apache Software Foundation的开源项目，它是一个快速、通用且可扩展的数据处理引擎，支持批处理、交互式查询、实时流处理和机器学习等多种计算模式。Cloudera作为一家专注于企业级Hadoop解决方案的公司，提供了与Spark集成的平台，使得在企业环境中部署和管理Spark变得更加便捷。文档中的"SparkGuide"部分可能包含以下关键知识点： 1. **Spark架构**：Spark的核心设计理念是基于内存计算，通过将数据缓存到内存中，显著提高了数据处理的速度。文档可能会详细介绍Spark的RDD（弹性分布式数据集）、DataFrame和Dataset API，以及DAG（有向无环图）执行模型。 2. **Spark组件**：包括Spark SQL（用于结构化数据处理）、Spark Streaming（实时流处理）、MLlib（机器学习库）、GraphX（图形处理），以及Spark Core（基础组件）等。每个组件的功能、使用方法和应用场景都会有所阐述。 3. **部署和配置**：文档可能会涵盖如何在Cloudera Manager上配置和管理Spark集群，包括设置资源分配、优化参数、监控性能和故障排查等内容。 4. **开发环境**：文档会指导如何使用Scala、Java、Python或R语言进行Spark应用开发，并介绍IDE集成、测试框架和提交作业的流程。 5. **案例研究**：为了帮助用户更好地理解Spark的实际应用，文档可能包含多个实际业务场景的例子，如实时日志分析、推荐系统、图分析等。 6. **安全性与权限**：在Cloudera环境下，Spark的安全性是重要一环。文档会讲解如何配置Kerberos认证、数据加密和访问控制，确保数据安全。 7. **性能调优**：这部分可能涉及内存管理、磁盘I/O优化、网络通信优化等方面的策略，帮助用户提升Spark应用的性能。 8. **重要通知和版权信息**：文档开头的"Important Notice"部分强调了Cloudera及其产品和服务的商标权，提醒用户尊重知识产权，遵守相关法律。 Cloudera Spark官方文档是一份全面的参考资料，对于想要在Cloudera平台上使用Spark的开发者和数据科学家来说，是不可或缺的学习和工作手册。通过深入阅读和实践，可以有效地掌握Spark在大数据处理中的应用技巧和最佳实践。

Developing Spark Applications

When you are ready to move beyond running core Spark applications in an interactive shell you need best

practices for building, packaging, and configuring applications and using the more advanced APIs. This section

describes how to develop, package, and run Spark applications, aspects of using Spark APIs beyond core Spark,

how to access data stored in Amazon S3, and best practices in building and configuring Spark applications.

Developing and Running a Spark WordCount Application

This tutorial describes how to write, compile, and run a simple Spark word count application in three of the

languages supported by Spark: Scala, Python, and Java. TheScala and Java code was originally developed for a

Cloudera tutorial written by Sandy Ryza.

Writing the Application

The example application is an enhanced version of WordCount, the canonical MapReduce example. In this version

of WordCount, the goal is to learn the distribution of letters in the most popular words in a corpus. The application:

1. Creates a SparkConf and SparkContext. A Spark application corresponds to an instance of the SparkContext

class. When running a shell, the SparkContext is created for you.

2. Gets a word frequency threshold.

3. Reads an input set of text documents.

4. Counts the number of times each word appears.

5. Filters out all words that appear fewer times than the threshold.

6. For the remaining words, counts the number of times each letter occurs.

In MapReduce, this requires two MapReduce applications, as well as persisting the intermediate data to HDFS

between them. In Spark, this application requires about 90 percent fewer lines of code than one developed using

the MapReduce API.

Here are three versions of the program:

• Figure 1: Scala WordCount on page 9

• Figure 2: Python WordCount on page 10

• Figure 3: Java 7 WordCount on page 10

import org.apache.spark.SparkContext

import org.apache.spark.SparkContext._

import org.apache.spark.SparkConf

object SparkWordCount {

def main(args: Array[String]) {

// create Spark context with Spark configuration

val sc = new SparkContext(new SparkConf().setAppName("Spark Count"))

// get threshold

val threshold = args(1).toInt

// read in text file and split each document into words

val tokenized = sc.textFile(args(0)).flatMap(_.split(" "))

// count the occurrence of each word

val wordCounts = tokenized.map((_, 1)).reduceByKey(_ + _)

// filter out words with fewer than threshold occurrences

val filtered = wordCounts.filter(_._2 >= threshold)

// count characters

val charCounts = filtered.flatMap(_._1.toCharArray).map((_, 1)).reduceByKey(_ + _)

Spark Guide | 9

Developing Spark Applications

return new Tuple2(s, 1);

}

).reduceByKey(

new Function2() {

public Integer call(Integer i1, Integer i2) {

return i1 + i2;

}

);

// filter out words with fewer than threshold occurrences

JavaPairRDD<String, Integer> filtered = counts.filter(

new Function, Boolean>() {

public Boolean call(Tuple2 tup) {

return tup._2 >= threshold;

}

);

// count characters

JavaPairRDD<Character, Integer> charCounts = filtered.flatMap(

new FlatMapFunction<Tuple2<String, Integer>, Character>() {

@Override

public Iterable<Character> call(Tuple2<String, Integer> s) {

Collection<Character> chars = new ArrayList<Character>(s._1().length());

for (char c : s._1().toCharArray()) {

chars.add(c);

}

return chars;

}

).mapToPair(

new PairFunction<Character, Character, Integer>() {

@Override

public Tuple2<Character, Integer> call(Character c) {

return new Tuple2<Character, Integer>(c, 1);

}

).reduceByKey(

new Function2<Integer, Integer, Integer>() {

@Override

public Integer call(Integer i1, Integer i2) {

return i1 + i2;

}

);

System.out.println(charCounts.collect());

}

Figure 3: Java 7 WordCount

Because Java 7 does not support anonymous functions, this Java program is considerably more verbose than

Scala and Python, but still requires a fraction of the code needed in an equivalent MapReduce program. Java 8

supports anonymous functions and their use can further streamline the Java application.

Compiling and Packaging the Scala and Java Applications

The tutorial uses Maven to compile and package the Scala and Java programs. Excerpts of the tutorial pom.xml

are included below. For best practices using Maven to build Spark applications, see Building Spark Applications

on page 26.

To compile Scala, include the Scala tools plug-in:

<groupId>org.scala-tools</groupId>

<xrefrtifactId>maven-scala-plugin</artifactId>

Spark Guide | 11

Developing Spark Applications

剩余51页未读，继续阅读

kx4xrl

粉丝: 1
资源: 10

Cloudera Spark官方文档：深入SparkGuide

CDH运维详解：非root用户启动Cloudera Manager

Cloudera Spark：高性能分布式计算框架

CentOS 7.1上Cloudera Manager 5.6的部署与故障预防指南

Impala-Kudu-HBase-Spark安装文档

Cloudera-Manager-中文手册

cloudera-cli-scripts:适用于各种管理功能的Cloudera CLI脚本

cloudera-manager-centos7-cm5.10.0_x86_64.tar.gz

cloudera-homework:专题课程的作业，来自 Cloudera“Apache Hadoop – 本科生课程”

cloudera官方文档

ClouderaManager-中文手册-实战总结.pdf

最新资源