使用Cloudera Manager监控和诊断CDH集群

hadoop

operation

需积分: 9 95 浏览量更新于2024-07-18 收藏 5.54MB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

"Cloudera 是一款用于管理与监控CDH（Cloudera Distributed Hadoop）安装的工具，尤其适合系统管理员使用。通过Cloudera Manager Admin Console，管理员可以检查集群健康状况、指标和使用情况，查看处理活动，以及浏览事件、日志和报告，以便诊断问题和确保合规性。本资料涵盖了Cloudera操作的重要注意事项，商标信息以及对Hadoop生态系统的尊重和承认。" 在Hadoop操作中，Cloudera Manager是一个关键组件，它提供了全面的集群管理和监控功能。以下是Cloudera Manager在操作CDH时的一些主要知识点： 1. **集群监控**：Cloudera Manager允许管理员实时查看整个集群的运行状态，包括节点健康、资源利用率（CPU、内存、磁盘和网络）、服务状态等。这有助于及时发现并解决问题，保持集群的稳定运行。 2. **性能指标**：系统管理员可以获取到各种性能指标，如数据读写速度、任务执行时间等，这些信息对于优化Hadoop作业性能至关重要。 3. **日志管理和事件查看**：Cloudera Manager集成了日志管理和事件查看功能，使得查找和分析问题变得更为便捷。通过统一的日志界面，可以快速定位到问题源，加速故障排除。 4. **处理活动可视化**：Cloudera Manager提供作业和任务的可视化视图，让管理员能够了解数据流经各个阶段的情况，这对于理解和优化工作流程非常有用。 5. **报警和通知**：当集群出现异常或达到预设阈值时，Cloudera Manager会触发报警，并可以通过邮件或其他方式通知管理员，以便及时响应。 6. **配置管理**：管理员可以通过Cloudera Manager集中管理集群的配置，确保所有节点的一致性和合规性，同时支持动态调整以适应不断变化的工作负载。 7. **安全与合规**：Cloudera Manager支持安全管理，包括用户权限、认证和加密等，确保数据的安全。此外，提供审计日志和报告功能，便于满足合规性要求。 8. **软件升级和维护**：Cloudera Manager简化了CDH组件的升级过程，可以自动化处理版本更新和补丁应用，减少人工干预的风险。 9. **供应商和商标信息**：文档中提及了Cloudera及其产品和服务的商标信息，强调了对开源社区的贡献，例如Hadoop是Apache Software Foundation的商标。 10. **版权和使用限制**：用户在使用Cloudera Manager及相关文档时，需遵守所有的版权法律，未经许可不得复制、存储或以任何形式传播文档内容。 Cloudera Manager是Hadoop操作中的强大工具，它提供了丰富的功能来帮助管理员有效管理和优化CDH集群，确保高效、安全的数据处理环境。

资源详情

资源推荐

2. Select the YARN service instance.

3. Click the Configuration tab.

4. Expand the Service-Wide category.

5. Click the Monitoring category.

6. Set the Visibility Settings properties for admin and non-admin users.

7. Click Save Changes to commit the changes. You can add a note that will be included with the change in the

Configuration History.

Configuring Impala Query Monitoring

You can configure the visibility of the Impala query results and the size of the storage allocated to Impala query

results.

Configuring Query Visibility

To configure whether admin and non-admin users can view all queries, only that user's queries, or no queries:

1. Click the Clusters tab.

2. Select the Impala service instance.

3. Click the Configuration tab.

4. Expand the Service-Wide category.

5. Click the Monitoring category.

6. Set the Visibility Settings properties for admin and non-admin users.

7. Click Save Changes to commit the changes.

Configuring Impala Query Data Store Maximum Size

The query store stores enough information to make the query searchable through the filter language.

1. Click the Clusters tab.

2. Do one of the following:

• Select Clusters > Cloudera Management Service > Cloudera Management Service.

• On the Status tab of the Home page, in Cloudera Management Service table, click the Cloudera Management

Service link.

3. Click the Configuration tab.

4. Click the Service Monitor Default Group category.

5. In the Impala Storage section, set the firehose_impala_storage_bytes property. The default is 1 GiB.

6. Click Save Changes to commit the changes.

7. Restart the Service Monitor.

The firehose_impala_storage_bytes property determines the approximate amount of disk space dedicated

to storing Impala query data. Once the store reaches its maximum size, older data is deleted to make room for

newer queries. The disk usage is approximate because data deletion begins only when the limit has been reached.

Configuring Alerts

The following topics describe how to configure when alerts are raised and how they are delivered:

• Enabling Activity Monitor Alerts on page 17

• Enabling Configuration Change Alerts on page 17

• Enabling HBase Alerts on page 17

• Configuring Health Alerts on page 17

• Configuring Log Alerts on page 18

• Configuring Alert Delivery on page 18

16 | Cloudera Operation

Monitoring and Diagnostics

1. Select Administration > Alerts.

Click to the right of Health Alert Threshold.

3. In the category Event Server Default Group, click the Health Alert Threshold value, and choose the Bad or

Concerning radio button.

4. Click Save Changes to commit the changes. You can add a note that will be included with the change in the

Configuration History.

Configuring Alerts Transitioning Out of Alerting Health Threshold

You can configure an alert when a service or role instance transitions from an alerting to a non-alerting health

threshold.

1. Select Administration > Alerts.

Click to the right of Alert on Transitions out of Alerting Health.

3. In the category Event Server Default Group, check the Alert on Transitions out of Alerting Health checkbox.

4. Click Save Changes to commit the changes. You can add a note that will be included with the change in the

Configuration History.

Configuring Log Alerts

You can configure an alert when a daemon emits a log message that matches a specified regular expression.

See Configuring Log Alerts on page 19.

Configuring Alert Delivery

You can configure alerts to be delivered by email or sent as SNMP traps. If you choose email delivery, you can

add to or modify the list of alert recipient email addresses. You can also send a test alert e-mail. See Managing

Alerts.

Note: If alerting is enabled for events, you will be able to search for and view alerts in the Events tab,

even if you do not have email notification configured.

Configuring Log Events

You can enable or disable the forwarding of selected log events to the Event Server. This is enabled by default,

and is a service-wide setting (Enable Log Event Capture) for each service for which monitoring is provided. You

can enable and disable event capture for CDH services or for the Cloudera Management Service.

Configuring Logs

1. Go to a service.

2. Click the Configuration tab.

3. Select

Role

Default Group > Logs.

4. Edit a log property.

5. Click Save Changes to commit the changes.

6. Restart the role.

Configuring Log Directories

1. Do one of the following:

• On the Home page, click a cluster name.1.

2. Select Configuration > All Log Directories.

3. Edit a

Role

Log Directory property.

• 1. Go to a service.

2. Click the Configuration tab.

3. Select

RoleName

Default Group > Logs.

18 | Cloudera Operation

Monitoring and Diagnostics

4. Edit the Log Directory property.

2. Click Save Changes to commit the changes.

3. Restart the role.

Enabling and Disabling Log Event Capture

1. Select Clusters >

Service

2. Click the Configuration tab.

3. Expand the Service-Wide category.

4. Click the Monitoring category.

5. Modify the Enable Log Event Capture setting.

6. Click Save Changes to commit the changes. You can add a note that will be included with the change in the

Configuration History.

You can also modify the rules that determine how log messages are turned into events. Editing these rules is

not recommended.

For each role, there are rules that govern how its log messages are turned into events by the custom log4j

appender for the role. These are defined in the Rules to Extract Events from Log Files property for each HDFS,

MapReduce and HBase role, and for ZooKeeper, Flume agent, and monitoring roles as well.

Configuring Which Log Messages Become Events

1. Select Clusters >

Service

2. Expand the Service-Wide category.

3. Click the Monitoring category.

4. Select the role group for the Role for which you want to configure log events, or search for "Rules to Extract

Events from Log Files". Note that for some roles there may be more than one role group, and you may need

to modify all of them. The easiest way to ensure that you have found all occurrences of the property you

need to modify is to search for the property by name. Cloudera Manager will show all copies of the property

that match the search filter.

5. Edit these rules as needed.

6. Click Save Changes to commit the changes.

A number of useful rules are defined by default, based on Cloudera's experience supporting Hadoop clusters.

For example:

• The line {"rate": 10, "threshold":"FATAL"}, means log entries with severity FATAL should be forwarded

as events, up to 10 a minute.

• The line {"rate": 0, "exceptiontype": "java.io.EOFException"}, means log entries with the

exception java.io.EOFException should always be forwarded as an event.

The syntax for these rules is defined in the Description field for this property: the syntax lets you create rules

that identify log messages based on log4j severity, message content matching, and/or the exception type. These

rules must result in valid JSON.

Note: Editing these rules is not recommended. Cloudera Manager provides a default set of rules that

should be sufficient for most users.

Configuring Log Alerts

You specify that a log event should generate an alert (by setting "alert":true in the rule). If you specify a

content match, the entire content must match — if you want to match on a partial string, you must provide

wildcards as appropriate to allow matching the entire string.

Cloudera Operation | 19

Monitoring and Diagnostics

Monitoring Clusters

There are several ways to monitor clusters.

The Clusters tab in the top navigation bar displays each cluster's services in its own section, with the Cloudera

Management Service separately below. You can select the following cluster-specific pages: hosts, reports,

activities, and resource management.

The Home page Status tab displays the clusters being managed by Cloudera Manager. Each cluster is displayed

either in summary form or in full form depending on the configuration of the Administration > Settings > Other >

Maximum Cluster Count Shown In Full property. When the number of clusters exceeds the value of the property,

only cluster summary information displays.

To display a cluster Status page, click the cluster name on the Home page Status tab. The cluster Status page

displays a table containing links to the Hosts page and the status pages of the services running in the cluster.

Each service row in the table has a menu of actions that you select by clicking and can contain one or more

of the following indicators:

DescriptionMeaningIndicator

Indicates that the service has at least one health issue. The indicator shows

the number of health issues at the highest severity level. If there are Bad

Health issue

health test results, the indicator is red. If there are no Bad health test results,

but Concerning test results exist, then the indicator is yellow. No indicator is

shown if there are no Bad or Concerning health test results.

Important: If there is one Bad health test result and two Concerning

health results, there will be three health issues, but the number will

be one.

Click the indicator to display the Health Issues pop-up dialog.

By default only Bad health test results are shown in the dialog. To display

Concerning health test results, click the Also show

concerning issue(s)

link.Click the link to display the Status page containing with details about the

health test result.

Indicates that the service has at least one configuration issue. The indicator

shows the number of configuration issues at the highest severity level. If there

Configuration

issue

are configuration errors, the indicator is red. If there are no errors but

configuration warnings exist, then the indicator is yellow. No indicator is shown

if there are no configuration notifications.

Important: If there is one configuration error and two configuration

warnings, there will be three configuration issues, but the number

will be one.

Click the indicator to display the Configuration Issues pop-up dialog.

By default only notifications at the Error severity level are listed, grouped by

service name are shown in the dialog. To display Warning notifications, click

the Also show

warning(s) link.Click the message associated with an error

or warning to be taken to the configuration property for which the notification

has been issued where you can address the issue.See Managing Services.

20 | Cloudera Operation

Monitoring and Diagnostics

剩余96页未读，继续阅读

tweety

粉丝: 0
资源: 8

使用Cloudera Manager监控和诊断CDH集群

cloudera-operation官方文档

cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz

Cloudera大数据管理员指南：Cloudera Manager的部署与配置

Failed to start cloudera-scm-agent.service: Unit not found.

cloudera-manager-centos7-cm5.16.2_x86_64.tar

如何确定命令systemctl start cloudera-scm-server成功启动

Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)

cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz

systemctl start cloudera-scm-agent Failed to start cloudera-scm-agent.service: Unit not found.

Unit cloudera-scm-server.service entered failed state

systemctl start cloudera-scm-agent

file /usr/lib/systemd/system/supervisord.service from install of cloudera-manager-agent-6.2.1-1426065.el7.x86_64 conflicts with file from package supervisor-3.4.0-1.el7.noarch

/var/lib/cloudera-scm-agent/active_parcels.json的内容及用途

cm启动不会创建cloudera-scm-server.log

./cloudera-scm-agent

cloudera-manager的作用

CMF_CONF_FIR 应该在哪配置

cloudera配置文件位置

hadoop配置yum

最新资源