快速入门与详细配置：CEPH官方文档详解

需积分: 12 114 浏览量更新于2024-07-21 收藏 1.24MB PDF 举报

"CEPH官方文档是关于分布式存储系统Ceph的详尽指南，由张绍文翻译自英文版，主要面向Linux系统的用户。文档详细介绍了如何从快速入门到深入配置的各个步骤。首先，章节1引导读者进行快速起步，包括在Debian/Ubuntu系统上安装Ceph软件包，配置基本环境，如写入配置文件并部署集群，以及介绍RBD（块设备）和Ceph文件系统的基本操作。文档强调了加入社区交流的重要性。接着，文档进入安装部分，提供了硬件和操作系统推荐，包括对CPU、内存、数据存储、网络和故障域的要求，以及针对不同场景的最低硬件配置建议。Gentoo操作系统上的安装方法也有所涉及。核心内容集中在第3章，详细介绍RADOS对象存储的配置。这部分涵盖硬盘和文件系统的最佳实践，配置文件ceph.conf的各个设置，如元变量、公共设置、网络设置、监控器、OSD（对象存储单元）配置，以及日志和调试选项。每个配置项都有详细的解释和示例，方便用户理解和应用。此外，文档还介绍了消息传递机制，确保了集群之间的有效通信。整体而言，这份文档不仅适合初次接触Ceph的新手，也是经验丰富的管理员进行系统优化和故障排查的重要参考资料。通过阅读和实践，用户可以全面掌握Ceph的部署和管理技巧。"

Ceph OSDs rely heavily upon the stability and performance of the underlying file system.

ceph 的 OSD 依赖于底层文件系统的稳定性和性能。

Note We currently recommend XFS for production deployments. We recommend btrfs for testing,

development, and any non-critical deployments. We believe that btrfs has the correct feature set and

roadmap to serve Ceph in the long-term, but XFS and ext4 provide the necessary stability for today’s

deployments. btrfs development is proceeding rapidly: users should be comfortable installing the latest

released upstream kernels and be able to track development activity for critical bug fixes.

注意：当前，我们推荐部署生产系统时使用 xfs 文件系统，建议用 btrfs 做测试、开发和其他不太要紧的

部署。我们相信，长期来看 btrfs 适合 ceph 的功能需求和发展方向，但是 xfs 和 ext4 能提供当前部署所必

需的稳定性。btrfs 开发在迅速推进，用户应该有能力经常更新到最新内核发布，而且能跟踪严重 bug 的

修正进度。

Ceph OSDs depend on the Extended Attributes (XATTRs) of the underlying file system for various forms

of internal object state and metadata. The underlying file system must provide sufficient capacity for

XATTRs. btrfs does not bound the total xattr metadata stored with a file. XFS has a relatively large limit

(64 KB) that most deployments won’t encounter, but the ext4 is too small to be usable. To use these file

systems, you should add the following like to the [osd] section of your ceph.conf file.:

ceph 的 OSD 有赖于底层文件系统的扩展属性（XATTR）存储各种内部对象状态和元数据。底层文件系统

必须给 XATTR 提供足够容量，btrfs 没有限制随文件存储的 xattr 元数据；xfs 的限制相对大(64KB)，多数

部署都不会有瓶颈；ext4 的则太小而不可用。要用这些文件系统，你得把下面这行写入 ceph.conf 的

[osd]段里。

filestore xattr use omap = true

FS BACKGROUND INFO

文件系统背景知识

The XFS and btrfs file systems provide numerous advantages in highly scaled data storage environments

when compared to ext3 and ext4. Both XFS and btrfs are journaling file systems, which means that they

are more robust when recovering from crashes, power outages, etc. These filesystems journal all of the

changes they will make before performing writes.

xfs 和 btrfs 相比较 ext3/4 而言，在高伸缩性数据存储方面有几个优势，xfs 和 btrfs 都是日志文件系统，

这使得在崩溃、断电后恢复时更健壮，因为文件系统在写入数据前会先记录所有变更。

XFS was developed for Silicon Graphics, and is a mature and stable filesystem. By contrast, btrfs is a

relatively new file system that aims to address the long-standing wishes of system administrators

working with large scale data storage environments. btrfs has some unique features and advantages

compared to other Linux filesystems.

xfs 由 Silicon Graphics 开发，是一个成熟、稳定的文件系统。相反，btrfs 是相对年轻的文件系统，它致

力于实现系统管理员梦寐以求的大规模数据存储基础，和其他 Linux 文件系统相比它有独一无二的功能和

优势。

btrfs is a copy-on-write filesystem. It supports file creation timestamps and checksums that verify

metadata integrity, so it can detect bad copies of data and fix them with the good copies. The copy-on-

write capability means that btrfs can support snapshots that are writable. btrfs supports transparent

compression and other features.

btrfs 是写时复制（ copy-on-write, cow）文件系统，它支持文件创建时间戳和校验和（可校验元数据完整

性）功能，所以它能探测到数据坏副本，并且用好副本修复。写时复制功能是说 btrfs 支持可写文件系统

快照。btrfs 也支持透明压缩和其他功能。

btrfs also incorporates multi-device management into the file system, which enables you to support

heterogeneous disk storage infrastructure, data allocation policies. The community also aims to provide

fsck, deduplication, and data encryption support in the future. This compelling list of features makes

btrfs the ideal choice for Ceph clusters.

btrfs 也集成了多设备管理功能，据此可以在底层支持异质硬盘存储，和数据分配策略。未来开发社区还

会提供 fsck、拆分、数据加密功能，这些诱人的功能正是 ceph 集群的理想选择。

3.1.2 ceph 的配置

When you start the Ceph service, the initialization process activates a series of daemons that run in the

background. The hosts in a typical Ceph cluster run at least one of four daemons:

你启动 ceph 服务的时候，初始化进程会把一系列守护进程放到后台运行。典型的 ceph 集群中至少要运

行下面四种守护进程中的一种。

Object Storage Device (ceph-osd)

Monitor (ceph-mon)

Metadata Server (ceph-mds)

Ceph Gateway (radosgw)

For your convenience, each daemon has a series of default values (i.e., many are set by

ceph/src/common/config_opts.h). You may override these settings with a Ceph configuration file.

对象存储设备(ceph-osd)

监视器(ceph-mon)

元数据服务器(ceph-mds)

ceph 网关(radosgw)

为方便起见，每个守护进程都有一系列默认值（参见 ceph/src/common/config_opts.h），你可以用 ceph

配置文件覆盖这些设置。

3.1.2.1

配置文件

ceph.conf

When you start a Ceph cluster, each daemon looks for a ceph.conf file that provides its configuration

settings. For manual deployments, you need to create a ceph.conf file to configure your cluster. For third

party tools that create configuration files for you (e.g., Chef), you may use the information contained

herein as a reference. The ceph.conf file defines:

启动 ceph 集群时，每个守护进程都从 ceph.conf 里查找它自己的配置。手动配置时，你需要创建

ceph.conf 来配置集群，使用第三方工具（如 chef）创建配置文件时可以参考下面的信息。ceph.conf 文件

定义了：

Cluster membership

Host names

Host addresses

Paths to keyrings

Paths to journals

Paths to data

Other runtime options

集群成员

主机名

主机 IP 地址

密钥路径

日志路径

数据路径

其它运行时选项

The default ceph.conf locations in sequential order include:

默认的 ceph.conf 位置相继排列如下：

$CEPH_CONF (i.e., the path following the $CEPH_CONF environment variable)

-c path/path (i.e., the -c command line argument)

/etc/ceph/ceph.conf

~/.ceph/config

./ceph.conf (i.e., in the current working directory)

The ceph.conf file uses an ini style syntax. You can add comments to the ceph.conf file by preceding

comments with a semi-colon (;) or a pound sign (#). For example:

ceph 文件使用 ini 风格的语法，以分号(;)和井号(#)开始的行是注释，如下：

# <--A number (#) sign precedes a comment.

; A comment may be anything.

# Comments always follow a semi-colon (;) or a pound (#) on each line.

# The end of the line terminates a comment.

# We recommend that you provide comments in your configuration file(s).

3.1.2.2 ceph.conf

设置

The ceph.conf file can configure all daemons in a cluster, or all daemons of a particular type. To

configure a series of daemons, the settings must be included under the processes that will receive the

configuration as follows:

ceph.conf 可配置集群里的所有守护进程，或者一种特定类型的守护进程

[global]

全局选项

Description: Settings under [global] affect all daemons in a Ceph cluster.

描述：[global]下的配置影响 ceph 集群里的所有守护进程。

Example: auth supported = cephx

[osd]

Description: Settings under [osd] affect all ceph-osd daemons in the cluster.

描述：[osd]下的配置影响集群里的所有 ceph-osd 进程。

Example: osd journal size = 1000

[mon]

Description: Settings under [mon] affect all ceph-mon daemons in the cluster.

描述：[mon]下的配置影响集群里的所有 ceph-mon 进程。

Example: mon addr = 10.0.0.101:6789

[mds]

Description: Settings under [mds] affect all ceph-mds daemons in the cluster.

描述：[mds]下的配置影响集群里的所有 ceph-mds 进程。

Example: host = myserver01

Global settings affect all instances of all daemon in the cluster. Use the [global] setting for values that

are common for all daemons in the cluster. You can override each [global] setting by:

全局设置影响集群内所有守护进程的例程，但可以用下面的设置覆盖[global]设置：

Changing the setting in a particular process type (e.g., [osd], [mon], [mds] ).

在[osd]、[mon]、[mds]下设置某一类的进程。

Changing the setting in a particular process (e.g., [osd.1] )

修改特定进程的设置，如[osd.1]。

Overriding a global setting affects all child processes, except those that you specifically override.

覆盖全局设置会影响所有子进程，明确剔除的例外。

A typical global setting involves activating authentication. For example:

典型的全局设置包括激活认证，例如：

[global]

# Enable authentication between hosts within the cluster.

# 在集群内的主机间认证

auth supported = cephx

You can specify settings that apply to a particular type of daemon. When you specify settings under

[osd], [mon] or [mds] without specifying a particular instance, the setting will apply to all OSDs,

monitors or metadata daemons respectively.

你可以在[osd]、[mon]、[mds]下设置特定类型的守护进程，而无须指定特定例程，这些设置会分别影响

所有 OSD、监视器、元数据进程。

You may specify settings for particular instances of a daemon. You may specify an instance by entering

its type, delimited by a period (.) and by the instance ID. The instance ID for an OSD is always numeric,

but it may be alphanumeric for monitors and metadata servers.

你也可以设置一个守护进程的特定例程，一个例程由类型和它的例程编号（ID）确定，OSD 的例程 ID 只

能是数字，监视器和元数据服务器的 ID 可包含字母和数字。

[osd.1]

# settings affect osd.1 only.

[mon.a]

# settings affect mon.a only.

[mds.b]

# settings affect mds.b only.

3.1.2.3

元变量

METAVARIABLES

Metavariables simplify cluster configuration dramatically. When a metavariable is set in a configuration

value, Ceph expands the metavariable into a concrete value. Metavariables are very powerful when used

within the [global], [osd], [mon] or [mds] sections of your configuration file. Ceph metavariables are

similar to Bash shell expansion.

元变量大大简化了集群配置，ceph 会把配置的元变量展开为具体值；元变量功能很强大，可以用在

[global]、[osd]、[mon]、[mds]段里，类似于 bash 的 shell 扩展。

Ceph supports the following metavariables:

ceph 支持下面的元变量：

$cluster

Description: Expands to the cluster name. Useful when running multiple clusters on the same hardware.

描述：展开为集群名字，在同一套硬件上运行多个集群时有用。

Example: /etc/ceph/$cluster.keyring

Default: ceph

$type

Description: Expands to one of mds, osd, or mon, depending on the type of the current daemon.、

描述：展开为 mds、osd、mon 中的一个，有赖于当前守护进程的类型。

Example: /var/lib/ceph/$type

$id

Description: Expands to the daemon identifier. For osd.0, this would be 0; for mds.a, it would be a.

描述：展开为守护进程标识；对 osd.0 来说标识为 0，mds.a 的标识是 a。

Example: /var/lib/ceph/$type/$cluster-$id

$host

Description: Expands to the host name of the current daemon.

描述：展开为当前守护进程的主机名。

$name

Description: Expands to $type.$id.

Example: /var/run/ceph/$cluster-$name.asok

3.1.2.4

共有设置

COMMON SETTINGS

The Hardware Recommendations section provides some hardware guidelines for configuring the cluster.

It is possible for a single host to run multiple daemons. For example, a single host with multiple disks or

RAIDs may run one ceph-osd for each disk or RAID. Additionally, a host may run both a ceph-mon and

an ceph-osd daemon on the same host. Ideally, you will have a host for a particular type of process. For

example, one host may run ceph-osd daemons, another host may run a ceph-mds daemon, and other

hosts may run ceph-mon daemons.

硬件推荐段提供了一些配置集群的硬件指导。一台机器可以运行多个进程，例如一台机器有多个硬盘或

RAID，可以为每个硬盘和 RAID 配置一个守护进程。另外，在同一台主机上也可以同时运行 ceph-mon 和

ceph-osd，理想情况下一台主机应该只运行一类进程，例如：一台主机运行着 ceph-osd 进程，另一台主

机运行着 ceph-mds 进程，ceph-mon 进程又在另外一台主机上。

Each host has a name identified by the host setting. Monitors also specify a network address and port

(i.e., domain name or IP address) identified by the addr setting. A basic configuration file will typically

specify only minimal settings for each instance of a daemon. For example:

每个主机都有一个标识名称（系统配置），监视器可用 addr 选项指定网络地址和端口（如域名或 IP 地

址），基本配置可以只指定最小配置。例如：

[mon.a]

host = hostName

mon addr = 150.140.130.120:6789

[osd.0]

host = hostName

Important The host setting is the short name of the host (i.e., not an fqdn). It is NOT and IP address

either. Enter hostname -s on the command line to retrieve the name of the host. Also, this setting is ONLY

for mkcephfs and manual deployment. It MUST NOT be used with chef or ceph-deploy.

重要：主机名设置是主机的短名字，不是正式域名 FQDN，也不是 IP 地址；在执行 hostname -s 就可以得

到短名字。此设置只在 mkcephfs 和手动部署时用到，用 chef 部署 ceph 时不必要。

3.1.2.5

网络

NETWORKS

Monitors listen on port 6789 by default, while metadata servers and OSDs listen on the first available

port beginning at 6800. Ensure that you open port 6789 on hosts that run a monitor daemon, and open

one port beginning at port 6800 for each OSD or metadata server that runs on the host. Ports are host-

specific, so you don’t need to open any more ports open than the number of daemons running on that

剩余150页未读，继续阅读

baidu_26826507

粉丝: 0
资源: 1

快速入门与详细配置：CEPH官方文档详解

ceph优化参数

ceph-docs_zh-v2.0.rar

ceph-cognize-docs_zh-v1.3

07-Yuan.Zhang-Ceph_in_China_Unicom_Labs.pdf

10-Ceph_over_Flash_Storage_to_Get_the_Advertised_Performance

Red_Hat_Ceph_Storage-3-Architecture_Guide-en-US

Red_Hat_Ceph_Storage-2-Architecture_Guide-en-US

Red_Hat_Ceph_Storage-4-Administration_Guide-en-US.pdf

ceph-mgr ceph-mon ceph-osd ceph-mds ceph-rgw

ansible-role-ceph_common:Ceph通用安装的Ansible角色

最新资源