DRBD 9用户指南：全面理解与配置存储镜像技术

drbd

需积分: 9 48 浏览量更新于2024-07-16 收藏 4.33MB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

《DRBD 9用户指南》是一份详尽的文档，介绍了DRBD（Distributed Replicated Block Device）——一个基于软件的、无共享存储的解决方案，用于在主机之间镜像块设备（如硬盘、分区、逻辑卷等）。该指南首先强调了阅读指南的重要性，然后逐步深入到DRBD的核心概念和技术特性。第1章“DRBD基础知识”涵盖了以下几个关键点： 1.1 内核模块：DRBD作为内核模块运行，它提供了块级数据复制功能，允许数据在两台或多台服务器间实时同步。 1.2 用户空间管理工具：指南介绍了用于配置、监控和管理DRBD的用户界面工具，如drbdadm、drbdsetup等，这些工具帮助用户轻松地设置和维护复制关系。 1.3 资源管理：资源在DRBD中扮演重要角色，每个复制对都有主和从两个角色，它们共同组成一个资源。 1.4 资源角色：主次资源的角色可以根据需要动态转换，确保在出现故障时能自动切换到另一台主机上的镜像。第二章“DRBD特性”详细列出了DRBD的高级功能： 2.1 单一主模式：允许在没有备份的情况下操作，但安全性较低。 2.2 双主模式：在两台主机上同时保持活跃，提供高可用性。 2.3 复制模式：包括实时同步、异步复制等，根据应用需求选择不同的同步级别。 2.4 多重冗余：支持多对复制，提高系统的可靠性。 2.5 自动资源提升：在主节点故障时，能够自动提升从节点为新的主节点。 2.6 多种复制传输方式：支持多种网络技术，如TCP/IP或InfiniBand，适应不同环境。后续章节分别讨论了： - 数据同步的效率优化和暂停/恢复功能 - 在线设备验证以检测数据一致性 - 数据完整性检查，确保复制流量正确无误 - 分布式脑裂问题通知与自动恢复策略 - 支持磁盘缓存刷新和垃圾收集（trim/discard） - 多种错误处理策略 - 三向复制和堆叠技术 - 远程复制技术（DRBD Proxy） - 便携式复制（Truck-Based Replication） - 浮动节点，实现动态负载均衡和横向扩展 - DRBD客户端的使用 - 一致性组（Quorum）实现分布式系统决策 - VCS（Version Control System）集成最后，章节介绍了DRBD的安装和构建过程，确保用户能够成功部署和配置这个关键的存储解决方案。总体而言，这份指南为IT专业人士提供了全面的DRBD 9技术参考，无论是初学者还是高级用户都能从中获益。

资源详情

资源推荐

2.10. Replication traffic integrity checking

DRBD optionally performs end-to-end message integrity checking using cryptographic message digest algorithms

such as MD5, SHA-1, or CRC-32C.

These message digest algorithms are not provided by DRBD, but by the Linux kernel crypto API; DRBD merely uses

them. Thus, DRBD is capable of utilizing any message digest algorithm available in a particular system’s kernel

configuration.

With this feature enabled, DRBD generates a message digest of every data block it replicates to the peer, which the

peer then uses to verify the integrity of the replication packet. If the replicated block can not be verified against the

digest, the connection is dropped and immediately re-established; because of the bitmap the typical result is a

retransmission. Thus, DRBD replication is protected against several error sources, all of which, if unchecked, would

potentially lead to data corruption during the replication process:

• Bitwise errors ("bit flips") occurring on data in transit between main memory and the network interface on the

sending node (which goes undetected by TCP checksumming if it is offloaded to the network card, as is

common in recent implementations);

• Bit flips occurring on data in transit from the network interface to main memory on the receiving node (the

same considerations apply for TCP checksum offloading);

• Any form of corruption due to a race conditions or bugs in network interface firmware or drivers;

• Bit flips or random corruption injected by some reassembling network component between nodes (if not using

direct, back-to-back connections).

See Configuring replication traffic integrity checking for information on how to enable replication traffic integrity

checking.

2.11. Split brain notification and automatic recovery

Split brain is a situation where, due to temporary failure of all network links between cluster nodes, and possibly due to

intervention by a cluster management software or human error, both nodes switched to the Primary role while

disconnected. This is a potentially harmful state, as it implies that modifications to the data might have been made on

either node, without having been replicated to the peer. Thus, it is likely in this situation that two diverging sets of data

have been created, which cannot be trivially merged.

DRBD split brain is distinct from cluster split brain, which is the loss of all connectivity between hosts managed by a

distributed cluster management application such as Heartbeat. To avoid confusion, this guide uses the following

convention:

• Split brain refers to DRBD split brain as described in the paragraph above.

• Loss of all cluster connectivity is referred to as a cluster partition, an alternative term for cluster split brain.

DRBD allows for automatic operator notification (by email or other means) when it detects split brain. See Split brain

notification for details on how to configure this feature.

While the recommended course of action in this scenario is to manually resolve the split brain and then eliminate its

root cause, it may be desirable, in some cases, to automate the process. DRBD has several resolution algorithms

available for doing so:

• Discarding modifications made on the younger primary. In this mode, when the network connection is re-

established and split brain is discovered, DRBD will discard modifications made, in the meantime, on the node

which switched to the primary role last.

• Discarding modifications made on the older primary. In this mode, DRBD will discard modifications made, in

the meantime, on the node which switched to the primary role first.

• Discarding modifications on the primary with fewer changes. In this mode, DRBD will check which of the

two nodes has recorded fewer modifications, and will then discard all modifications made on that host.

• Graceful recovery from split brain if one host has had no intermediate changes. In this mode, if one of

The DRBD9 and LINSTOR User’s Guide: 2.10. Replication traffic integrity checking

the hosts has made no modifications at all during split brain, DRBD will simply recover gracefully and declare the

split brain resolved. Note that this is a fairly unlikely scenario. Even if both hosts only mounted the file system

on the DRBD block device (even read-only), the device contents typically would be modified (eg. by filesystem

journal replay), ruling out the possibility of automatic recovery.

Whether or not automatic split brain recovery is acceptable depends largely on the individual application. Consider the

example of DRBD hosting a database. The discard modifications from host with fewer changes approach may be fine

for a web application click-through database. By contrast, it may be totally unacceptable to automatically discard any

modifications made to a financial database, requiring manual recovery in any split brain event. Consider your

application’s requirements carefully before enabling automatic split brain recovery.

Refer to Automatic split brain recovery policies for details on configuring DRBD’s automatic split brain recovery

policies.

2.12. Support for disk flushes

When local block devices such as hard drives or RAID logical disks have write caching enabled, writes to these devices

are considered completed as soon as they have reached the volatile cache. Controller manufacturers typically refer to

this as write-back mode, the opposite being write-through. If a power outage occurs on a controller in write-back

mode, the last writes are never committed to the disk, potentially causing data loss.

To counteract this, DRBD makes use of disk flushes. A disk flush is a write operation that completes only when the

associated data has been committed to stable (non-volatile) storage — that is to say, it has effectively been written to

disk, rather than to the cache. DRBD uses disk flushes for write operations both to its replicated data set and to its

meta data. In effect, DRBD circumvents the write cache in situations it deems necessary, as in activity log updates or

enforcement of implicit write-after-write dependencies. This means additional reliability even in the face of power

failure.

It is important to understand that DRBD can use disk flushes only when layered on top of backing devices that support

them. Most reasonably recent kernels support disk flushes for most SCSI and SATA devices. Linux software RAID (md)

supports disk flushes for RAID-1 provided that all component devices support them too. The same is true for device-

mapper devices (LVM2, dm-raid, multipath).

Controllers with battery-backed write cache (BBWC) use a battery to back up their volatile storage. On such devices,

when power is restored after an outage, the controller flushes all pending writes out to disk from the battery-backed

cache, ensuring that all writes committed to the volatile cache are actually transferred to stable storage. When running

DRBD on top of such devices, it may be acceptable to disable disk flushes, thereby improving DRBD’s write

performance. See Disabling backing device flushes for details.

2.13. Trim/Discard support

Trim/Discard are two names for the same feature: a request to a storage system, telling it that some data range is not

being used anymore [2: For example, a deleted file’s data.] and can get recycled.

This call originates in Flash-based storages (SSDs, FusionIO cards, etc.), which cannot easily rewrite a sector but

instead have to erase and write the (new) data again (incurring some latency cost). For more details, see eg. the

[[https://en.wikipedia.org/wiki/Trim_%28computing%29,wikipedia page]].

Since 8.4.3 DRBD includes support for Trim/Discard. You don’t need to configure or enable anything; if DRBD detects

that the local (underlying) storage system allows using these commands, it will transparently enable them and pass

such requests through.

The effect is that eg. a recent-enough mkfs.ext4 on a multi-TB volume can shorten the initial sync time to a few

seconds to minutes - just by telling DRBD (which will relay that information to all connected nodes) that most/all of

the storage is now to be seen as invalidated.

Nodes that connect to that resource later on will not have seen the Trim/Discard requests, and will therefore start a

full resync; depending on kernel version and file system a call to fstrim might give the wanted result, though.

The DRBD9 and LINSTOR User’s Guide: 2.12. Support for disk flushes



even if you don’t have storage with Trim/Discard support, some virtual block devices will provide

you with the same feature, for example Thin LVM.

2.14. Disk error handling strategies

If a hard drive fails which is used as a backing block device for DRBD on one of the nodes, DRBD may either pass on

the I/O error to the upper layer (usually the file system) or it can mask I/O errors from upper layers.

Passing on I/O errors

If DRBD is configured to pass on I/O errors, any such errors occurring on the lower-level device are transparently

passed to upper I/O layers. Thus, it is left to upper layers to deal with such errors (this may result in a file system being

remounted read-only, for example). This strategy does not ensure service continuity, and is hence not recommended

for most users.

Masking I/O errors

If DRBD is configured to detach on lower-level I/O error, DRBD will do so, automatically, upon occurrence of the first

lower-level I/O error. The I/O error is masked from upper layers while DRBD transparently fetches the affected block

from a peer node, over the network. From then onwards, DRBD is said to operate in diskless mode, and carries out all

subsequent I/O operations, read and write, on the peer node(s) only. Performance in this mode will be reduced, but the

service continues without interruption, and can be moved to the peer node in a deliberate fashion at a convenient

time.

See Configuring I/O error handling strategies for information on configuring I/O error handling strategies for DRBD.

2.15. Strategies for dealing with outdated data

DRBD distinguishes between inconsistent and outdated data. Inconsistent data is data that cannot be expected to be

accessible and useful in any manner. The prime example for this is data on a node that is currently the target of an on-

going synchronization. Data on such a node is part obsolete, part up to date, and impossible to identify as either. Thus,

for example, if the device holds a filesystem (as is commonly the case), that filesystem would be unexpected to mount

or even pass an automatic filesystem check.

Outdated data, by contrast, is data on a secondary node that is consistent, but no longer in sync with the primary

node. This would occur in any interruption of the replication link, whether temporary or permanent. Data on an

outdated, disconnected secondary node is expected to be clean, but it reflects a state of the peer node some time

past. In order to avoid services using outdated data, DRBD disallows promoting a resource that is in the outdated state.

DRBD has interfaces that allow an external application to outdate a secondary node as soon as a network interruption

occurs. DRBD will then refuse to switch the node to the primary role, preventing applications from using the outdated

data. A complete implementation of this functionality exists for the Pacemaker cluster management framework

(where it uses a communication channel separate from the DRBD replication link). However, the interfaces are generic

and may be easily used by any other cluster management application.

Whenever an outdated resource has its replication link re-established, its outdated flag is automatically cleared. A

background synchronization then follows.

See the section about the DRBD outdate-peer daemon (dopd) for an example DRBD/Heartbeat/Pacemaker

configuration enabling protection against inadvertent use of outdated data.

2.16. Three-way replication via stacking



Available in DRBD version 8.3.0 and above; deprecated in DRBD version 9.x, as more nodes can be

implemented on a single level. See Defining network connections for details.

When using three-way replication, DRBD adds a third node to an existing 2-node cluster and replicates data to that

node, where it can be used for backup and disaster recovery purposes. This type of configuration generally involves

Long-distance replication via DRBD Proxy.

The DRBD9 and LINSTOR User’s Guide: 2.14. Disk error handling strategies

Three-way replication works by adding another, stacked DRBD resource on top of the existing resource holding your

production data, as seen in this illustration:

Primary

Backup

Secondary

Upper layer

Lower layer

Figure 3. DRBD resource stacking

The stacked resource is replicated using asynchronous replication (DRBD protocol A), whereas the production data

would usually make use of synchronous replication (DRBD protocol C).

Three-way replication can be used permanently, where the third node is continuously updated with data from the

production cluster. Alternatively, it may also be employed on demand, where the production cluster is normally

disconnected from the backup site, and site-to-site synchronization is performed on a regular basis, for example by

running a nightly cron job.

2.17. Long-distance replication via DRBD Proxy

DRBD’s protocol A is asynchronous, but the writing application will block as soon as the socket output buffer is full (see

the sndbuf-size option in the man page of drbd.conf). In that event, the writing application has to wait until

some of the data written runs off through a possibly small bandwidth network link.

The average write bandwidth is limited by available bandwidth of the network link. Write bursts can only be handled

gracefully if they fit into the limited socket output buffer.

You can mitigate this by DRBD Proxy’s buffering mechanism. DRBD Proxy will place changed data from the DRBD

device on the primary node into its buffers. DRBD Proxy’s buffer size is freely configurable, only limited by the address

room size and available physical RAM.

Optionally DRBD Proxy can be configured to compress and decompress the data it forwards. Compression and

decompression of DRBD’s data packets might slightly increase latency. However, when the bandwidth of the network

link is the limiting factor, the gain in shortening transmit time outweighs the compression and decompression

overhead.

Compression and decompression were implemented with multi core SMP systems in mind, and can utilize multiple CPU

cores.

The fact that most block I/O data compresses very well and therefore the effective bandwidth increases justifies the

use of the DRBD Proxy even with DRBD protocols B and C.

See Using DRBD Proxy for information on configuring DRBD Proxy.



DRBD Proxy is one of the few parts of the DRBD product family that is not published under an

open source license. Please contact sales@linbit.com or sales_us@linbit.com for an evaluation

license.

2.18. Truck based replication

Truck based replication, also known as disk shipping, is a means of preseeding a remote site with data to be replicated,

by physically shipping storage media to the remote site. This is particularly suited for situations where

The DRBD9 and LINSTOR User’s Guide: 2.17. Long-distance replication via DRBD Proxy

• the total amount of data to be replicated is fairly large (more than a few hundreds of gigabytes);

• the expected rate of change of the data to be replicated is less than enormous;

• the available network bandwidth between sites is limited.

In such situations, without truck based replication, DRBD would require a very long initial device synchronization (on

the order of weeks, months, or years). Truck based replication allows to ship a data seed to the remote site, and so

drastically reduces the initial synchronization time. See Using truck based replication for details on this use case.

2.19. Floating peers



This feature is available in DRBD versions 8.3.2 and above.

A somewhat special use case for DRBD is the floating peers configuration. In floating peer setups, DRBD peers are not

tied to specific named hosts (as in conventional configurations), but instead have the ability to float between several

hosts. In such a configuration, DRBD identifies peers by IP address, rather than by host name.

For more information about managing floating peer configurations, see Configuring DRBD to replicate between two

SAN-backed Pacemaker clusters.

2.20. Data rebalancing (horizontal storage scaling)

If your company’s policy says that 3-way redundancy is needed, you need at least 3 servers for your setup.

Now, as your storage demands grow, you will encounter the need for additional servers. Rather than having to buy 3

more servers at the same time, you can rebalance your data across a single additional node.

Storage Node 1

Storage Node 2

Storage Node 3

Storage Node 1

Storage Node 2

Storage Node 3

Storage Node 4

Figure 4. DRBD data rebalancing

In the figure above you can see the before and after states: from 3 nodes with three 25TiB volumes each (for a net

75TiB), to 4 nodes, with net 100TiB.

DRBD 9 makes it possible to do an online, live migration of the data; please see Data rebalancing for the exact steps

needed.

2.21. DRBD client

With the multiple-peer feature of DRBD a number of interesting use-cases have been added, for example the DRBD

client.

The DRBD9 and LINSTOR User’s Guide: 2.19. Floating peers

剩余220页未读，继续阅读

only火车头

粉丝: 21
资源: 2

DRBD 9用户指南：全面理解与配置存储镜像技术

drbd安装配置[收集].pdf

drbd-users-guide.pdf

drbd rpm安装

Job for drbd.service failed because the control process exited with error code. S

CENTOS7+DRBD+NFS+KEEPLIVED搭建的详细文档、

zabbix监控drbd9

查看DRBD9同步进度的命令

单台虚拟机安装drbd

lsmod | grep drbd 关闭进程

mysql drbd

modprobe: FATAL: Module drbd is in use怎么解决

启动iptablss

/dev/mapper/centos-root 不存在

查看DRBD同步进度的命令

gitlab+drbd更换成高可用架构

NFS+DRBD+KEEPLIVED搭建教程

DRBD用别的网卡来同步

nfs+drbd+keepalived

drbd和glusterfs

开源的版图设计lvs drc工具有哪些

最新资源