Cloudera管理员指南：英文版

需积分: 9 164 浏览量更新于2024-07-18 收藏 8.5MB PDF 举报

"Cloudera Administration - Cloudera Manager Administrator Guide" 在IT行业中，Cloudera是一个专注于大数据管理和分析的公司，它提供了一个名为Cloudera Manager的管理工具，使得企业能够轻松地部署、管理和监控Apache Hadoop及其相关服务。本指南，"Cloudera Manager Administrator Guide"，是为那些需要对Cloudera集群进行有效管理的专业人士准备的，主要以英文形式呈现。 Cloudera Manager是Cloudera的核心组件之一，它允许管理员执行以下关键任务： 1. **集群部署**：Cloudera Manager简化了Hadoop集群的安装过程，允许用户通过向导式界面快速配置和启动多个服务。 2. **服务监控**：提供实时性能指标，帮助管理员识别和解决问题，确保集群的稳定运行。 3. **配置管理**：集中式的配置管理，可以方便地更新和应用配置到集群中的各个节点和服务。 4. **安全与访问控制**：支持身份验证、授权和审计，确保数据的安全性和合规性。 5. **维护与更新**：自动化软件升级、补丁应用和备份恢复，降低手动操作的风险和复杂性。 6. **资源调度**：通过YARN（Yet Another Resource Negotiator）进行资源分配和调度，优化集群资源利用率。 7. **故障诊断**：提供日志聚合、警报和故障排除工具，帮助快速定位问题。文档中提到的版权信息表明，所有内容由Cloudera公司及其供应商或许可方拥有，未经许可，不得复制或使用其商标。同时，代码部分遵循Apache License Version 2.0，这是一种广泛使用的开源软件许可证，允许自由使用、修改和分发代码，但需遵守一定的条款。此外，文档还提到了Apache Hadoop和Hadoop大象标志是Apache Software Foundation的商标，这表明Cloudera产品与Hadoop生态系统紧密相关，提供了与Hadoop兼容的解决方案。 "Cloudera Administration"涵盖了Cloudera Manager在管理和运维大数据环境中的全面功能，是IT专业人员管理基于Cloudera的大数据平台的重要参考资料。对于想要深入理解和掌握Cloudera集群管理的读者来说，这份指南无疑是一份宝贵的资源。

When a service is added to Cloudera Manager, either through the installation or upgrade wizard or with the Add

Services workflow, Cloudera Manager automatically sets the configuration properties, based on the needs of the service

and characteristics of the cluster in which it will run. These configuration properties include both service-wide

configuration properties, as well as specific properties for each role type associated with the service, managed through

role groups. A role group is a set of configuration properties for a role type, as well as a list of role instances associated

with that group. Cloudera Manager automatically creates a default role group named Role Type Default Group for

each role type. See Role Groups on page 48.

Changing the Configuration of a Service or Role Instance (Classic Layout)

1. Go to the service status page.

2. Click the Configuration tab.

3. Under the appropriate role group, select the category for the properties you want to change.

4. To search for a text string (such as "snippet"), in a property, value, or description, enter the text string in the

Search box at the top of the category list.

5. Moving the cursor over the value cell highlights the cell; click anywhere in the highlighted area to enable editing

of the value. Then type the new value in the field provided (or check or uncheck the box, as appropriate).

• To facilitate entering some types of values, you can specify not only the value, but also the units that apply

to the value. for example, to enter a setting that specifies bytes per second, you can choose to enter the

value in bytes (B), KiBs, MiBs, or GiBs—selected from a drop-down menu that appears when you edit the

value.

•

If the property allows a list of values, click the icon to the right of the edit field to add an additional field.

An example of this is the HDFS DataNode Data Directory property, which can have a comma-delimited list of

directories as its value. To remove an item from such a list, click the icon to the right of the field you want

to remove.

6. Click Save Changes to commit the changes. You can add a note that will be included with the change in the

Configuration History. This will change the setting for the role group, and will apply to all role instances associated

with that role group. Depending on the change you made, you may need to restart the service or roles associated

with the configuration you just changed. Or, you may need to redeploy your client configuration for the service.

You should see a message to that effect at the top of the Configuration page, and services will display an outdated

configuration (Restart Needed), (Refresh Needed), or outdated client configuration indicator. Click the

indicator to display the Stale Configurations on page 28 page.

Validation of Configuration Properties

Cloudera Manager validates the values you specify for configuration properties. If you specify a value that is outside

the recommended range of values or is invalid, Cloudera Manager displays a warning at the top of the Configuration

tab and in the text box after you click Save Changes. The warning is yellow if the value is outside the recommended

range of values and red if the value is invalid.

Overriding Configuration Properties

For role types that allow multiple instances, each role instance inherits its configuration properties from its associated

role group. While role groups provide a convenient way to provide alternate configuration properties for selected

groups of role instances, there may be situations where you want to make a one-off configuration change—for example

when a host has malfunctioned and you want to temporarily reconfigure it. In this case, you can override configuration

properties for a specific role instance:

1. Go to the Status page for the service whose role you want to change.

2. Click the Instances tab.

3. Click the role instance you want to change.

4. Click the Configuration tab.

5. Change the configuration values as appropriate.

6. Save your changes.

You will most likely need to restart your service or role to have your configuration changes take effect.

16 | Cloudera Administration

Managing CDH and Managed Services

Viewing and Editing Overridden Configuration Properties

To see a list of all role instances that have an override value for a particular configuration setting, go to the entry for

the configuration setting in the Status page, expand the Overridden by n instance(s) link in the value cell for the

overridden value.

To view the override values, and change them if appropriate, click the Edit Overrides link. This opens the Edit Overrides

page, and lists the role instances that have override properties for the selected configuration setting.

On the Edit Overrides page, you can do any of the following:

• View the list of role instances that have overridden the value specified in the role group. Use the selections on

the drop-down menu below the Value column header to view a list of instances that use the inherited value,

instances that use an override value, or all instances. This view is especially useful for finding inconsistent properties

in a cluster. You can also use the Host and Rack text boxes to filter the list.

• Change the override value for the role instances to the inherited value from the associated role group. To do so,

select the role instances you want to change, choose Inherited Value from the drop-down menu next to Change

value of selected instances to and click Apply.

• Change the override value for the role instances to a different value. To do so, select the role instances you want

to change, choose Other from the drop-down menu next to Change value of selected instances to. Enter the new

value in the text box and then click Apply.

Resetting Configuration Properties to the Default Value

To reset a property back to its default value, click the Reset to the default value link below the text box in the value

cell. The default value is inserted and both the text box and the Reset link disappear. Explicitly setting a configuration

to the same value as its default (inherited value) has the same effect as using the Reset to the default value link.

There is no mechanism for resetting to an autoconfigured value. However, you can use the configuration history and

rollback feature to revert any configuration changes.

Restarting Services and Instances after Configuration Changes

If you change the configuration properties after you start a service or instance, you may need to restart the service or

instance to have the configuration properties become active. If you change configuration properties at the service

level that affect a particular role only (such as all DataNodes but not the NameNodes), you can restart only that role;

you do not need to restart the entire service. If you changed the configuration for a particular role instance (such as

one of four DataNodes), you may need to restart only that instance.

Cloudera Administration | 17

Managing CDH and Managed Services

1. Follow the instructions in Restarting a Service on page 41 or Starting, Stopping, and Restarting Role Instances on

page 46.

2. If you see a Finished status, the service or role instances have restarted.

3. Go to the Home > Status tab. The service should show a Status of Started for all instances and a health status of

Good.

For further information, see Stale Configurations on page 28.

Autoconfiguration

Cloudera Manager provides several interactive wizards to automate common workflows:

• Installation - used to bootstrap a Cloudera Manager deployment

• Add Cluster - used when adding a new cluster

• Add Service - used when adding a new service

• Upgrade - used when upgrading to a new version of CDH

• Static Service Pools - used when configuring static service pools

• Import MapReduce - used when migrating from MapReduce to YARN

In some of these wizards, Cloudera Manager uses a set of rules to automatically configure certain settings to best suit

the characteristics of the deployment. For example, the number of hosts in the deployment drives the memory

requirements for certain monitoring daemons: the more hosts, the more memory is needed. Additionally, wizards that

are tasked with creating new roles will use a similar set of rules to determine an ideal host placement for those roles.

Scope

The following table shows, for each wizard, the scope of entities it affects during autoconfiguration and role-host

placement.

Role-Host Placement ScopeAutoconfiguration ScopeWizard

New cluster, Cloudera Management

Service

New cluster, Cloudera Management

Service

Installation

New clusterNew clusterAdd Cluster

New serviceNew serviceAdd Service

Cloudera Management ServiceCloudera Management ServiceUpgrade

N/AExisting clusterStatic Service Pools

N/AExisting YARN serviceImport MapReduce

Certain autoconfiguration rules are unscoped, that is, they configure settings belonging to entities that aren't necessarily

the entities under the wizard's scope. These exceptions are explicitly listed.

Autoconfiguration

Cloudera Manager employs several different rules to drive automatic configuration, with some variation from wizard

to wizard. These rules range from the simple to the complex.

Configuration Scope

One of the points of complexity in autoconfiguration is configuration scope. The configuration hierarchy as it applies

to services is as follows: configurations may be modified at the service level (affecting every role in the service), role

group level (affecting every role instance in the group), or role level (affecting one role instance). A configuration found

in a lower level takes precedence over a configuration found in a higher level.

With the exception of the Static Service Pools, and the Import MapReduce wizard, all Cloudera Manager wizards follow

a basic pattern:

1. Every role in scope is moved into its own, new, role group.

18 | Cloudera Administration

Managing CDH and Managed Services

2. This role group is the receptacle for the role's "idealized" configuration. Much of this configuration is driven by

properties of the role's host, which can vary from role to role.

3. Once autoconfiguration is complete, new role groups with common configurations are merged.

4. The end result is a smaller set of role groups, each with an "idealized" configuration for some subset of the roles

in scope. A subset can have any number of roles; perhaps all of them, perhaps just one, and so on.

The Static Service Pools and Import MapReduce wizards configure role groups directly and do not perform any merging.

Static Service Pools

Certain rules are only invoked in the context of the Static Service Pools wizard. Additionally, the wizard autoconfigures

cgroup settings for certain kinds of roles:

• HDFS DataNodes

• HBase RegionServers

• MapReduce TaskTrackers

• YARN NodeManagers

• Impala Daemons

• Solr Servers

• Spark Standalone Workers

• Accumulo Tablet Servers

• Add-on services

YARN

yarn.nodemanager.resource.cpu-vcores - For each NodeManager role group, set to number of cores,

including hyperthreads, on one NodeManager member's host * service percentage chosen in

wizard.

All Services

Cgroup cpu.shares - For each role group that supports cpu.shares, set to max(20, (service percentage

chosen in wizard) * 20).

Cgroup blkio.weight - For each role groupthat supports blkio.weight, set to max(100, (service percentage

chosen in wizard) * 10).

Data Directories

Several autoconfiguration rules work with data directories, and there's a common sub-rule used by all such rules to

determine, out of all the mountpoints present on a host, which are appropriate for data. The subrule works as follows:

• The initial set of mountpoints for a host includes all those that are disk-backed. Network-backed mountpoints are

excluded.

• Mountpoints beginning with /boot, /cdrom, /usr, /tmp, /home, or /dev are excluded.

• Mountpoints beginning with /media are excluded, unless the backing device's name contains /xvd somewhere

in it.

• Mountpoints beginning with /var are excluded, unless they are /var or /var/lib.

• The largest mount point (in terms of total space, not available space) is determined.

• Other mountpoints with less than 1% total space of the largest are excluded.

• Mountpoints beginning with /var or equal to / are excluded unless they’re the largest mount point.

• Remaining mountpoints are sorted lexicographically and retained for future use.

Memory

The rules used to autoconfigure memory reservations are perhaps the most complicated rules employed by Cloudera

Manager. When configuring memory, Cloudera Manager must take into consideration which roles are likely to enjoy

more memory, and must not over commit hosts if at all possible. To that end, it needs to consider each host as an

entire unit, partitioning its available RAM into segments, one segment for each role. To make matters worse, some

Cloudera Administration | 19

Managing CDH and Managed Services

roles have more than one memory segment. For example, a Solr server has two memory segments: a JVM heap used

for most memory allocation, and a JVM direct memory pool used for HDFS block caching. Here is the overall flow during

memory autoconfiguration:

1. The set of participants includes every host under scope as well as every {role, memory segment} pair on those

hosts. Some roles are under scope while others are not.

2. For each {role, segment} pair where the role is under scope, a rule is run to determine four different values for

that pair:

• Minimum memory configuration. Cloudera Manager must satisfy this minimum, possibly over-committing

the host if necessary.

• Minimum memory consumption. Like the above, but possibly scaled to account for inherent overhead. For

example, JVM memory values are multiplied by 1.3 to arrive at their consumption value.

• Ideal memory configuration. If RAM permits, Cloudera Manager will provide the pair with all of this memory.

• Ideal memory consumption. Like the above, but scaled if necessary.

3. For each {role, segment} pair where the role is not under scope, a rule is run to determine that pair's existing

memory consumption. Cloudera Manager will not configure this segment but will take it into consideration by

setting the pair's "minimum" and "ideal" to the memory consumption value.

4. For each host, the following steps are taken:

a. 20% of the host's available RAM is subtracted and reserved for the OS.

b. sum(minimum_consumption) and sum(ideal_consumption) are calculated.

c. An "availability ratio" is built by comparing the two sums against the host's available RAM.

a. If RAM < sum(minimum) ratio = 0

b. If RAM >= sum(ideal) ratio = 1

c. Otherwise, ratio is computed via: RAM - sum(minimum)) / (sum(ideal) - sum(minimum)

5. For each {role, segment} pair where the role is under scope, the segment is configured to be (minimum + ((ideal

- minimum) * (host availability ratio))). The value is rounded down to the nearest megabyte.

6. The {role, segment} pair is set with the value from the previous step. In the Static Service Pools wizard, the role

group is set just once (as opposed to each role).

7. Custom post-configuration rules are run.

Customization rules are applied in steps 2, 3 and 7. In step 2, there's a generic rule for most cases, as well as a series

of custom rules for certain {role, segment} pairs. Likewise, there's a generic rule to calculate memory consumption in

step 3 as well as some custom consumption functions for certain {role, segment} pairs.

Step 2 Generic Rule Excluding Static Service Pools Wizard

For every {role, segment} pair where the segment defines a default value, the pair's minimum is set to the segment's

minimum value (or 0 if undefined), and the ideal is set to the segment's default value.

Step 2 Custom Rules Excluding Static Service Pools Wizard

HDFS

For the NameNode and Secondary NameNode JVM heaps, the minimum is 50 MB and the ideal is max(1 GB,

sum_over_all(DataNode mountpoints’ available space) / 0.000008).

MapReduce

For the JobTracker JVM heap, the minimum is 50 MB and the ideal is max(1 GB, round((1 GB * 2.3717181092

* ln(number of TaskTrackers in MapReduce service)) - 2.6019933306)). Ifthereare<=5 TaskTrackers,

the ideal is 1 GB.

For the mapper JVM heaps, the minimum is 1 and the ideal is (number of cores, including hyperthreads, on the

TaskTracker host). Note that memory consumption is scaled by mapred_child_java_opts_max_heap (the size of

a given task's heap).

20 | Cloudera Administration

Managing CDH and Managed Services

剩余481页未读，继续阅读

weixin_43391099

粉丝: 0
资源: 4

Cloudera管理员指南：英文版

构建与维护大型Hadoop集群的Cloudera管理指南

大规模Hadoop集群管理指南：Cloudera Administrator手册

Cloudera Manager集成Apache Zeppelin CSD与包裹指南

Cloudera Administration Handbook

cloudera-administration

cloudera官方文档

Cloudera-CDP-Ansible：在AWS，GCP，Azure上设置Cloudera CDP的Ansible方法

Cloudera 5.4.x Documentation系列官方文档

深入Cloudera Manager API的操作与应用

go 生成基于 graphql 服务器库.zip

最新资源