AIX HACMP 6.1故障排除手册（v120）：新功能与全面指南

版权申诉

23 浏览量更新于2024-06-21 收藏 1.01MB PDF 举报

本指南是针对AIX环境下的High Availability Cluster Multi-Processing (HACMP) Version 6.1的故障排除手册，适用于后续的所有版本更新和修改，直至有新的版本发布时注明。版权归属于IBM Corporation，特别指出美国政府用户在使用、复制或披露时受到GSA ADP Schedule合同的限制。 "About this document"部分可能会介绍文档的目的、结构、预期读者以及可能的使用前提，比如在使用前需阅读"Notes"章节中的注意事项，了解可能存在的限制和兼容性信息。 "Highlighting"部分可能涉及文档中重要的标记或强调内容，例如关键术语的解释、故障排除流程中的关键步骤，或者与HACMP 6.1新特性相关的重点提示。 "Case-sensitivity in AIX"部分讲述了在AIX系统中，字符大小写的敏感性，这对于理解和操作HACMP尤其重要，因为错误的大小写可能导致配置问题或误识别。 "ISO 9000"提及的质量管理标准可能与HACMP的开发过程、测试或文档编写质量相关，表明遵循了国际质量管理体系的标准。 "HACMP publications"部分列出了与HACMP相关的其他文档集，包括可能的安装指南、用户手册、技术白皮书等，以供参考。 "HACMP/XD publications"可能指代特定于HACMP扩展模块（如XD）的额外文档，这些扩展功能可能增强了系统的可用性和性能。 "HACMPSmartAssist publications"这部分可能涉及到HACMP智能助手的文档，这是一种自动化工具，用于简化故障排查和维护任务。 "Troubleshooting guide"的核心内容是故障排除策略和步骤，这部分可能包括： 1. "What's new in Troubleshooting HACMP"：介绍了本版故障排除手册相对于前版本的新增内容和技术改进，帮助用户了解最新解决方法。 2. "Troubleshooting HACMP clusters"：详细阐述了如何诊断和修复HACMP集群中的各种问题，可能涵盖网络、硬件、软件故障、配置错误等多方面。 3. "Troubleshooting an HACMP cluster overview"：给出了故障排查的整体框架，包括基本的故障排查流程和检查点。 "Using the Problem Determination Tool"部分强调了问题确定工具的使用，这是HACMP中一种诊断辅助工具，用于快速定位和分析问题根源。本指南提供了深入且实用的HACMP 6.1在AIX平台上的故障排除方法，旨在帮助用户高效地管理和维护高可用性集群环境。

v View detailed HACMP log files

v Change or show HACMP log file parameters

v Change or show Cluster Manager log file parameters

v Change or show a cluster log file directory

v Change all Cluster Logs directory

v Collect cluster log files for problem reporting.

Related concepts:

“Using cluster log files” on page 14

These topics explain how to use the HACMP cluster log files to troubleshoot the cluster. Included also are

some sections on managing parameters for some of the logs.

Related information:

Testing an HACMP cluster

Recovering from HACMP script failure

Select this option from the Problem Determination Tools menu to recover from an HACMP script

failure.

For example, if a script failed because it was unable to set the hostname, the Cluster Manager reports the

event failure. Once you correct the problem by setting the hostname from the command line, you must

get the Cluster Manager to resume cluster processing.

The Recover From HACMP Script Failure menu option invokes the /usr/es/sbin/cluster/utilities/

clruncmd command, which sends a signal to the Cluster Manager daemon (clstrmgrES ) on the specified

node, causing it to stabilize. You must again run the script manually to continue processing.

Make sure that you fix the problem that caused the script failure. You need to manually complete the

remaining steps that followed the failure in the event script (see /var/hacmp/log/hacmp.out ). Then, to

resume clustering, complete the following steps to bring the HACMP event script state to EVENT

COMPLETED:

1. Enter smit hacmp

2. In SMIT, select Problem Determination Tools > Recover From HACMP Script Failure.

3. Select the IP label/address for the node on which you want to run the clruncmd command and press

Enter. The system prompts you to confirm the recovery attempt. The IP label is listed in the /etc/hosts

file and is the name assigned to the service IP address of the node on which the failure occurred.

4. Press Enter to continue. Another SMIT panel appears to confirm the success of the script recovery.

Restoring HACMP configuration database from an active configuration

If cluster services are up and you make changes to the configuration, those changes have modified the

default configuration directory (DCD). You may realize that the impact of those changes was not well

considered and you want to undo them. Because nothing was modified in the active configuration

directory (ACD), all that is needed to undo the modifications to the DCD is to restore the DCD from the

ACD.

Select this option from the Problem Determination Tools menu to automatically save any of your

changes in the Configuration Database as a snapshot with the path /usr/es/sbin/cluster/snapshots/

UserModifiedDB before restoring the Configuration Database with the values actively being used by the

Cluster Manager.

1. Enter smit hacmp

2. In SMIT, select Problem Determination Tools > Restore HACMP Configuration Database from

Active Configuration.

SMIT displays:

Are you Sure?

8 High Availability Cluster Multi-Processing for AIX: Troubleshooting guide

3. Press Enter.

The snapshot is saved.

Related information:

Saving and restoring cluster configurations

Release locks set by dynamic reconfiguration

You can release locks set by dynamic reconfiguration.

Related information:

Dynamic reconfiguration issues and synchronization

Clear SSA disk fence registers

Select this option from the menu only in an emergency, usually only when recommended by IBM

support. If SSA Disk Fencing is enabled, and a situation has occurred in which the physical disks are

inaccessible by a node or a group of nodes that need access to a disk, clearing the fence registers will

allow access. Once this is done, the SSA Disk Fencing algorithm will be disabled unless HACMP is

restarted from all nodes.

To clear SSA Disk Fence Registers take the following steps:

1. Enter smit hacmp

2. In SMIT, stop cluster services (unless you are sure no contention for the disk will occur), by selecting

System Management (C-SPOC) > Manage HACMP Services > Stop Cluster Services.

3. Select Problem Determination Tools > Clear SSA Disk Fence Registers.

4. Select the affected physical volume(s) and press Enter.

5. Restart cluster services to enable SSA disk fencing again.

Related information:

Starting and stopping cluster services

HACMP cluster test tool

HACMP includes the Cluster Test Tool to help you test the recovery procedures for a new cluster before

it becomes part of your production environment.

You can also use it to test configuration changes to an existing cluster, when the cluster is not in service.

Related information:

Testing an HACMP cluster

HACMP error notification

You can set up AIX and HACMP error notification.

Related information:

Types of error notification

Opening a SMIT session on a node

As a convenience while troubleshooting your cluster, you can open a SMIT session on a remote node

from within the Problem Determination Tool SMIT panel.

To open a SMIT session on a remote node:

1. Select the Problem Determination Tools > Open a SMIT Session on a Node option. SMIT displays a

list of available cluster nodes.

2. Select the node on which you wish to open the SMIT session and press Enter.

Troubleshooting guide 9

Configuring cluster performance tuning

Cluster nodes sometimes experience extreme performance problems such as large I/O transfers, excessive

error logging, or lack of memory. When this happens, the Cluster Manager can be starved for CPU time

and it may not reset the deadman switch within the time allotted. Misbehaved applications running at a

priority higher than the Cluster Manager can also cause this problem.

The deadman switch is the AIX kernel extension that halts a node when it enters a hung state that

extends beyond a certain time limit. This enables another node in the cluster to acquire the hung node's

resources in an orderly fashion, avoiding possible contention problems. If the deadman switch is not reset

in time, it can cause a system panic and dump under certain cluster conditions.

Setting the following tuning parameters correctly may avoid some of the performance problems noted

above. To prevent the possibility of having to change the HACMP Network Modules Failure Detection

Rate, you should consider setting the AIX syncd frequency rate.

Set the AIX parameters on each cluster node.

You may also set the following HACMP network tuning parameters for each type of network:

v Failure Detection Rate

v Grace Period.

You can configure these related parameters directly from HACMP SMIT.

Network module settings are propagated to all nodes when you set them on one node and then

synchronize the cluster topology.

Related information:

Performance management and tuning

Setting syncd frequency

The syncd setting determines the frequency with which the I/O disk-write buffers are flushed. Frequent

flushing of these buffers reduces the chance of deadman switch time-outs.

The AIX default value for syncd as set in /sbin/rc.boot is 60. It is recommended to change this value to

10.

To change the syncd frequency setting, do the following on each node:

1. Enter smit hacmp

2. In SMIT, select Extended Configuration > Extended Performance Tuning Parameters > Change/Show

syncd frequency and press Enter.

3. Configure the field values with the recommended syncd frequency:

Field name Description

syncd frequency (in seconds) 10 is recommended for most clusters. Possible values are 0 to 32767.

Changing the failure detection rate of a network module after the initial

configuration

If you want to change the failure detection rate of a network module, either change the tuning

parameters of a network module to predefined values of Fast, Normal and Slow, or set these attributes

to custom values.

Also, use the custom tuning parameters to change the baud rate for TTYs if you are using RS232

networks that might not handle the default baud rate of 38400.

10 High Availability Cluster Multi-Processing for AIX: Troubleshooting guide

Related information:

Managing the cluster topology

Resetting HACMP tunable values

You can change the settings for a list of tunable values that were altered during cluster maintenance and

reset them to their default settings, or installation-time cluster settings. The installation-time cluster

settings are equal to the values that appear in the cluster after installing HACMP from scratch.

Resetting cluster tunables is useful when an administrative change has not produced the desired results,

and you want to return to the default values. While this may not produce the optimum configuration, it

is very likely to produce a working one and allows you to make further changes from a known base.

Note: Resetting the tunable values does not change any other aspects of the configuration, while

installing HACMP removes all user-configured configuration information including nodes, networks, and

resources.

You can change and reset HACMP tunable values to their default values under the following conditions:

v Before resetting HACMP tunable values, HACMP takes a cluster snapshot. After the values have been

reset to defaults, if you want to go back to your customized cluster settings, you can restore them with

the cluster snapshot. HACMP saves snapshots of the last ten configurations in the default cluster

snapshot directory, /usr/es/sbin/cluster/snapshots, with the name active.x.odm, where x is a digit

between 0 and 9, with 0 being the most recent.

v Stop cluster services on all nodes before resetting tunable values. HACMP prevents you from resetting

tunable values in a running cluster.

In some cases, HACMP cannot differentiate between user-configured information and discovered

information, and does not reset such values. For example, you may enter a service label and HACMP

automatically discovers the IP address that corresponds to that label. In this case, HACMP does not

reset the service label or the IP address. The cluster verification utility detects if these values do not

match.

The clsnapshot.log file in the snapshot directory contains log messages for this utility. If any of the

following scenarios are run, then HACMP cannot revert to the previous configuration:

v cl_convert is run automatically

v cl_convert is run manually

Listing tunable values

You can change and reset several different tunable values.

This list includes the following:

v User-supplied information.

– Network module tuning parameters such as failure detection rate, grace period and heartbeat rate.

HACMP resets these parameters to their installation-time default values.

– Cluster event customizations such as all changes to cluster events. Note that resetting changes to

cluster events does not remove any files or scripts that the customization used, it only removes the

knowledge HACMP has of pre- and post-event scripts.

– Cluster event rule changes made to the event rules database are reset to the installation-time default

values.

– HACMP command customizations made to the default set of HACMP commands are reset to the

installation-time defaults.

v Automatically generated and discovered information, generally users cannot see this information.

HACMP rediscovers or regenerates this information when the cluster services are restarted or during

the next cluster synchronization.

HACMP resets the following:

Troubleshooting guide 11

– Local node names stored in the cluster definition database

– Netmasks for all cluster networks

– Netmasks, interface names and aliases for disk heartbeating (if configured) for all cluster interfaces

– Instance numbers and default log sizes for the RSCT subsystem.

Resetting HACMP tunable values using SMIT

You can reset cluster tunable values to default values using SMIT.

To reset cluster tunable values to default values:

1. Enter smit hacmp.

2. In SMIT, select Extended Configuration > Extended Topology Configuration > Configure an

HACMP Cluster > Reset Cluster Tunables and press Enter.

Use this option to reset all the tunables (customizations) made to the cluster. Using this option returns

all tunable values to their default values but does not change the cluster configuration. HACMP takes

a snapshot file before resetting. You can choose to have HACMP synchronize the cluster when this

operation is complete.

3. Select the options as follows and press Enter:

Field name Description

Synchronize Cluster Configuration If you set this option to yes, HACMP synchronizes the cluster after resetting the

cluster tunables.

4. HACMP asks:

"Are you sure?"

5. Press Enter.

HACMP resets all the tunable values to their original settings and removes those that should be removed

(such as the nodes' knowledge about customized pre- and post-event scripts).

Related reference:

“Listing tunable values” on page 11

You can change and reset several different tunable values.

Resetting HACMP tunable values using the command line

The clsnapshot -t command also resets the cluster tunables. However, this command is intended for use

by IBM support.

You should use the SMIT interface to reset the cluster tunable values. See the man page for more

information.

Sample custom scripts

These topics include some scenarios where it is useful to run sample custom scripts.

Making cron jobs highly available

To help maintain the HACMP environment, you need to have certain cron jobs execute only on the

cluster node that currently holds the resources.

If a cron job executes in conjunction with a resource or application, it is useful to have that cron entry

fallover along with the resource. It may also be necessary to remove that cron entry from the cron table if

the node no longer possesses the related resource or application.

The following example shows one way to use a customized script to do this:

12 High Availability Cluster Multi-Processing for AIX: Troubleshooting guide

剩余119页未读，继续阅读

weixin_40191861_zj

粉丝: 84
资源: 1万+

AIX HACMP 6.1故障排除手册（v120）：新功能与全面指南

HACMP 解决方案

HACMP6.1 Planning guide-228

HACMP6.1 Installation guide-152

HACMP6.1 Programming client applications-120

HACMP 6.1 Smart Assist for Oracle user's guide-48

HACMP6.1 Smart Assist for WebSphere User's Guide-46

HX-hacmp6.1-v1.2.docx

系统软件配置规范-HACMP6.1-V1.3.docx

ibm hacmp 6.1

AIX6.1下HACMP6.1双机实施文档.doc

最新资源