Cloudera 管理指南：商标与版权信息

需积分: 4 198 浏览量更新于2024-07-18 收藏 4.22MB PDF 举报

"Cloudera 管理指南" Cloudera 是一家专注于提供大数据管理和分析解决方案的公司，其产品和服务主要围绕着 Hadoop 生态系统。本文档可能是一份 Cloudera 管理的详细指南，涵盖了 Cloudera 的产品、服务以及与 Hadoop 相关的管理实践。在 Hadoop 环境中，Cloudera 提供了 Cloudera Manager 这一工具，它是一个全面的管理平台，用于部署、配置、监控和管理 Hadoop 集群。Cloudera Manager 支持多种 Hadoop 组件，包括 HDFS、YARN、MapReduce、Hive、HBase 等，使得集群的日常运维变得更加简单和高效。 Cloudera Impala 是 Cloudera 推出的一款 MPP（大规模并行处理）SQL 查询引擎，它可以实现实时分析存储在 Hadoop 分布式文件系统（HDFS）和 HBase 中的数据，提供了与传统数据仓库接近的查询性能。文档中提到了对商标和版权的尊重，强调 Cloudera、Cloudera 的 logo 和产品名称是其公司的商标，未经许可不得复制或使用。同样，Apache Software Foundation 的 Hadoop 和 Hadoop 大象 logo 也是受版权保护的，这表明文档内容涉及到的软件和技术都是受法律保护的。此外，文档还指出引用任何产品、服务、流程或其他信息并不意味着由该公司进行背书、赞助或推荐。这意味着尽管文档可能提及了其他第三方的产品或服务，但并不表示 Cloudera 对这些产品或服务的性能或质量给予保证。关于版权，文档要求用户必须遵守所有适用的版权法律，未经允许，不得复制、存储或以任何形式传输文档内容。这强调了尊重知识产权的重要性。 "ClouderaAdministration" 可能是一份详细介绍了如何使用 Cloudera Manager 进行 Hadoop 集群管理，以及涉及 Cloudera Impala 实时分析的文档，同时也强调了与商标、版权相关的法律问题。对于使用和管理基于 Hadoop 的 Cloudera 集群的 IT 专业人员来说，这份文档是宝贵的参考资料。

Impala

For the Impala Daemon, the memory consumption is 0 if YARN_For_ResourceManager is set. If the memory

limit is defined but not -1, its value is used verbatim. If it's defined but -1, the consumption is equal to the total

RAM on the Daemon host. If it is undefined, the consumption is (total RAM * 0.8).

MapReduce

See Step 3 Custom Rules for Static Service Pools Wizard on page 15.

Solr

For the Solr Server JVM direct memory segment, the consumption is equal to the value verbatim provided

solr.hdfs.blockcache.enable and solr.hdfs.blockcache.direct.memory.allocation are both true.

Otherwise, the consumption is 0.

Step 7 Custom Rules

HDFS

• NameNode JVM heaps are equalized. For every pair of NameNodes in an HDFS service with different heap

sizes, the larger heap size is reset to the smaller one.

• JournalNode JVM heaps are equalized. For every pair of JournalNodes in an HDFS service with different heap

sizes, the larger heap size is reset to the smaller one.

• NameNode and Secondary NameNode JVM heaps are equalized. For every {NameNode, Secondary NameNode}

pair in an HDFS service with different heap sizes, the larger heap size is reset to the smaller one.

HBase

Master JVM heaps are equalized. For every pair of Masters in an HBase service with different heap sizes, the

larger heap size is reset to the smaller one.

MapReduce

JobTracker JVM heaps are equalized. For every pair of JobTrackers in an MapReduce service with different heap

sizes, the larger heap size is reset to the smaller one.

Oozie

Oozie Server JVM heaps are equalized. For every pair of Oozie Servers in an Oozie service with different heap

sizes, the larger heap size is reset to the smaller one.

YARN

ResourceManager JVM heaps are equalized. For every pair of ResourceManagers in a YARN service with different

heap sizes, the larger heap size is reset to the smaller one.

ZooKeeper

ZooKeeper Server JVM heaps are equalized. For every pair of servers in a ZooKeeper service with different heap

sizes, the larger heap size is reset to the smaller one.

Impala

If an Impala service has YARN_For_ResourceManager set, every Impala Daemon memory limit is set to the

value of (yarn.nodemanager.resource.memory-mb * 1 GB) if there's a YARN NodeManager co-located with

the Impala Daemon.

16 | Cloudera Administration

Configuring CDH and Managed Services

General Rules

HDFS

• dfs.datanode.du.reserved - For each DataNode, set to min((total space of DataNode host largest

mountpoint) / 10, 10 GB).

• dfs.namenode.name.dir - For each NameNode, set to the first two mountpoints on the NameNode host

with /dfs/nn appended.

• dfs.namenode.checkpoint.dir - For each Secondary NameNode, set to the first mountpoint on the

Secondary NameNode host with /dfs/snn appended.

• dfs.datanode.data.dir - For each DataNode, set to all the mountpoints on the host with /dfs/dn

appended.

• dfs.journalnode.edits.dir - For each JournalNode, set to the first mountpoint on the JournalNode host

with /dfs/jn appended.

• dfs.datanode.failed.volumes.tolerated - For each DataNode, set to (number of mountpoints on

DataNode host) / 2.

• dfs.namenode.service.handler.count and dfs.namenode.handler.count - For each NameNode, set

to max(30, ln(number of DataNodes in this HDFS service) * 20).

• dfs.block.local-path-access.user - For each HDFS service, set to impala if there's an Impala service

in the cluster.

This rule is unscoped; it can fire even if the HDFS service is not under scope.

• dfs.datanode.hdfs-blocks-metadata.enabled - For each HDFS service, set to true if there's an Impala

service in the cluster.

This rule is unscoped; it can fire even if the HDFS service is not under scope.

• dfs.client.read.shortcircuit - For each HDFS service, set to true if there's an Impala service in the

cluster.

This rule is unscoped; it can fire even if the HDFS service is not under scope.

• dfs.datanode.data.dir.perm - For each DataNode, set to 755 if there's an Impala service in the cluster

and the cluster isn’t Kerberized.

This rule is unscoped; it can fire even if the HDFS service is not under scope.

• fs.trash.interval - For each HDFS service, set to 1.

MapReduce

• mapred.local.dir - For each JobTracker, set to the first mountpoint on the JobTracker host with /mapred/jt

appended.

• mapred.local.dir - For each TaskTracker, set to all the mountpoints on the host with /mapred/local

appended.

• mapred.reduce.tasks - For each MapReduce service, set to max(1, sum_over_all(TaskTracker number of

reduce tasks (determined via mapred.tasktracker.reduce.tasks.maximum for that TaskTracker, which

is configured separately)) / 2).

• mapred.job.tracker.handler.count - For each JobTracker, set to max(10, ln(number of TaskTrackers in

this MapReduce service) * 20).

• mapred.submit.replication - If there's an HDFS service in the cluster, for each MapReduce service, set

to max(1, sqrt(number of DataNodes in the HDFS service)).

• mapred.tasktracker.instrumentation - If there's a management service, for each MapReduce service,

set to org.apache.hadoop.mapred.TaskTrackerCmonInst.

This rule is unscoped; it can fire even if the

MapReduce service is not under scope.

HBase

• hbase.replication - For each HBase service, set to true if there's a Key-Value Store Indexer service in the

cluster.

This rule is unscoped; it can fire even if the HBase service is not under scope.

• replication.replicationsource.implementation - For each HBase service, set to

com.ngdata.sep.impl.SepReplicationSource if there's a Keystore Indexer service in the cluster.

This

rule is unscoped; it can fire even if the HBase service is not under scope.

Cloudera Administration | 17

Configuring CDH and Managed Services

YARN

• yarn.nodemanager.local-dirs - For each NodeManager, set to all the mountpoints on the NodeManager

host with /yarn/nm appended.

• yarn.nodemanager.resource.cpu-vcores - For each NodeManager, set to the number of cores (including

hyperthreads) on the NodeManager host.

• mapred.reduce.tasks - For each YARN service, set to max(1, sum_over_all(NodeManager number of

cores, including hyperthreads) / 2).

• yarn.resourcemanager.nodemanagers.heartbeat-interval-ms - For each NodeManager, set to max(100,

10 * (number of NodeManagers in this YARN service)).

• yarn.scheduler.maximum-allocation-vcores - For each ResourceManager, set to

max_over_all(NodeManager number of vcores (determined via yarn.nodemanager.resource.cpu-vcores

for that NodeManager, which is configured separately)).

• yarn.scheduler.maximum-allocation-mb - For each ResourceManager, set to max_over_all(NodeManager

amount of RAM (determined via yarn.nodemanager.resource.memory-mb for that NodeManager, which

is configured separately)).

• mapreduce.client.submit.file.replication - If there's an HDFS service in the cluster, for each YARN

service, set to max(1, sqrt(number of DataNodes in the HDFS service)).

Hue

• WebHDFS dependency - For each Hue service, set to either the first HttpFS role in the cluster, or, if there are

none, the first NameNode in the cluster.

• HBase Thrift Server dependency- For each Hue service in a CDH 4.4 or later cluster, set to the first HBase

Thrift Server in the cluster.

Impala

For each Impala service, set Enable Impala auditing for Navigator to true if there's a Cloudera Management

Service with a Navigator audit server role.

This rule is unscoped; it can fire even if the Impala service is not under

scope.

All Services

If a service dependency is unset, and a service with the desired type exists in the cluster, set the service

dependency to the first such target service. Applies to all service dependencies except

YARN_For_ResourceManager. Applies only to the Express and Add Cluster wizards.

Role-Host Placement

Cloudera Manager employs the same role-host placement rule regardless of wizard. The set of hosts considered

depends on the scope. If the scope is a cluster, all hosts in the cluster are included. If a service, all hosts in the

service's cluster are included. If the Cloudera Management Service, all hosts in the deployment are included. The

rules are as follows:

1. The hosts are sorted from most to least physical RAM. Ties are broken by sorting on hostname (ascending)

followed by host identifier (ascending).

2. The overall number of hosts is used to determine which arrangement to use. These arrangements are

hard-coded, each dictating for a given "master" role type, what index (or indexes) into the sorted host list in

step 1 to use.

3. Master role types are included based on several factors:

• Is this role type part of the service (or services) under scope?

• Does the service already have the right number of instances of this role type?

• Does the cluster's CDH version support this role type?

• Does the installed Cloudera Manager license allow for this role type to exist?

18 | Cloudera Administration

Configuring CDH and Managed Services

4. Master roles are placed on each host using the indexes and the sorted host list. If a host already has a given

master role, it is skipped.

5. An HDFS DataNode is placed on every host outside of the arrangement described in step 2, provided HDFS

is one of the services under scope.

6. Certain "worker" roles are placed on every host where an HDFS DataNode exists, either because it existed

there prior to the wizard, or because it was added in the previous step. The supported worker role types are:

• MapReduce TaskTrackers

• YARN NodeManagers

• HBase RegionServers

• Impala Daemons

• Spark Workers

7. Hive gateways are placed on every host, provided a Hive service is under scope and a gateway didn’t already

exist on a given host.

This rule merely dictates the

default

placement of roles; you are free to modify it before it is applied by the

wizard.

Custom Configuration

Required Role:

Cloudera Manager exposes properties that allow you to insert custom configuration text into XML configuration,

property, and text files, or into an environment. The naming convention for these properties is:

XXX

Advanced

Configuration Snippet (Safety Valve) for

YYY

XXX YYY

Advanced Configuration Snippet (Safety Valve), where

XXX

is a service or role and

YYY

is the target.

The values you enter into a configuration snippet must conform to the syntax of the target. For an XML

configuration file, the configuration snippet must contain valid XML property definitions. For a properties file,

the configuration snippet must contain valid property definitions. Some files simply require a list of host

addresses.

The configuration snippet mechanism is intended for use in cases where there is configuration setting that is

not exposed as a configuration property in Cloudera Manager. Configuration snippets generally override normal

configuration. Contact Cloudera Support if you are required to use a configuration snippet that is not explicitly

documented.

Service-wide configuration snippets apply to all roles in the service; a configuration snippet for a role group

applies to all instances of the role associated with that role group.

There are configuration snippets for servers and client configurations. In general after changing a server

configuration snippet you must restart the server, and after changing a client configuration snippet you must

redeploy the client configuration. Sometimes you can refresh instead of restart. In some cases however, you

must restart a dependent server after changing a client configuration. For example, changing a MapReduce

client configuration marks the dependent Hive server as stale, which must be restarted. The Admin Console

displays an indicator when a server must be restarted. In addition, the All Configuration Issues tab on the Home

page lists the actions you must perform to propagate the changes.

Configuration Snippet Types and Syntax

SyntaxDescriptionType

<name>property_name</name>

Set configuration properties in various

configuration files; the property name indicates

Configuration

<value>property_value</value>

</property>

into which configuration file the configuration

will be placed. Configuration files have the

extension .xml or .conf.

Cloudera Administration | 19

Configuring CDH and Managed Services

SyntaxDescriptionType

For example, there are several configuration

snippets for the Hive service. One Hive

For example, to specify a MySQL connector

library, put this property definition in that

configuration snippet:

configuration snippet property is called the

HiveServer2 Advanced Configuration Snippet

for hive-site.xml; configuration you enter here

is inserted verbatim into the hive-site.xml

file associated with the HiveServer2 role group.

<value>file:///usr/share/java/mysql-connector-java.jar</value>

</property>

To see a list of configuration snippets that

apply to a specific configuration file, enter the

configuration file name in the Search field in

the top navigation bar. For example, searching

for mapred-site.xml shows the configuration

snippets that have mapred-site.xml in their

name.

key=value

Specify key-value pairs for a service, role, or

client that are inserted into the respective

environment.

One example of using an environment

configuration snippet is to add a JAR to a

Environment

For example, to add JDBC connectors to a Hive

gateway classpath, add

AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar

AUX_CLASSPATH=/usr/share/java/*

classpath. Place JARs in a custom location such

as /opt/myjars and extend the classpath via

the appropriate service environment

to Gateway Client Advanced Configuration

Snippet for hive-env.sh.

configuration snippet. The value of a JAR

property must conform to the syntax supported

by its environment. See Setting the class path.

Do not place JARs inside locations such as

/opt/cloudera or

/usr/lib/{hadoop*,hbase*,hive*,etc.}

that are managed by Cloudera because they

are overwritten at upgrades.

key1=value1

key2=value2

Set properties in a log4j.properties file.Logging

For example:

max.log.file.size=200MB

max.log.file.backup.index=10

key1=value1

key2=value2

Set properties to configure Hadoop metrics in

a hadoop-metrics.properties or

hadoop-metrics2.properties file.

Metrics

For example:

*.sink.foo.class=org.apache.hadoop.metrics2.sink.FileSink

namenode.sink.foo.filename=/tmp/namenode-metrics.out

secondarynamenode.sink.foo.filename=/tmp/secondarynamenode-metrics.out

host1.domain1

host2.domain2

Specify a list of host addresses that are allowed

or disallowed from accessing a service.

White and

black lists

Setting an Advanced Configuration Snippet

1. Click a service.

2. Click the Configuration tab.

3. Expand a category group and click an Advanced subcategory.

20 | Cloudera Administration

Configuring CDH and Managed Services

剩余326页未读，继续阅读

tweety

粉丝: 0
资源: 8

Cloudera 管理指南：商标与版权信息

基于Hadoop集群的数据仓库解决方案：Hive数据仓库软件

CDH运维详解：非root用户启动Cloudera Manager

CDH6.3.2安装包下载及离线教程

Cloudera-CDP-Ansible：在AWS，GCP，Azure上设置Cloudera CDP的Ansible方法

Cloudera Administration Handbook

cloudera官方文档

Cloudera 5.4.x Documentation系列官方文档

Cloudera管理员指南：英文版

大规模Hadoop集群管理指南：Cloudera Administrator手册

构建与维护大型Hadoop集群的Cloudera管理指南

最新资源