BFO：批量文件操作优化海量文件性能

183 浏览量更新于2024-08-27 收藏 2.96MB PDF 举报

"BFO: Batch-File Operations on Massive Files for Consistent Performance Improvement" 这篇研究论文主要探讨了在处理大量文件时，现有本地文件系统性能不佳的问题，并提出了一种新的批量文件访问方法，称为BFO（Batch-File Operations）。现有的文件系统通常设计用于支持单一文件访问模式，当面对批量文件操作，尤其是小文件操作时，性能会显著下降。这种单个文件访问模式导致批量文件的访问被逐一串行化，产生大量非顺序、随机且常常依赖的I/O操作，这在存储端的数据和元数据之间造成效率低下。论文首先通过实验分析了批量文件访问效率低下的根本原因，即传统的文件系统未能有效地并行处理批量操作，特别是对于小文件的处理，由于每次只能处理一个文件，导致了大量的随机I/O，这些I/O操作在文件数据和元数据之间频繁进行，极大地影响了性能。为了解决这个问题，研究人员提出了BFO，一套优化的批量文件操作策略。BFO旨在通过开发创新的文件系统内核机制和算法来改善批量文件的处理效率，包括可能的预读取、缓存管理和并行I/O调度等技术。这些技术的目的是减少不必要的等待时间和提高磁盘利用率，将原本串行的文件访问转换为并行操作，从而减少I/O冲突，提高整体性能。此外，BFO可能还包括对元数据管理的优化，例如，通过改进的索引结构和元数据缓存策略来加速元数据查找，减少元数据操作的延迟。同时，可能会引入一种智能的I/O调度策略，以更有效地利用存储系统的带宽，优先处理关键或依赖的I/O请求。通过BFO，论文的目标是提供一种一致性的性能提升，使得无论是大数据集还是包含大量小文件的场景，都能实现高效、稳定的文件操作。这不仅对大数据处理、云计算环境和分布式存储系统有重大意义，还可能对其他需要处理大量文件的应用场景（如数据分析、日志处理和媒体处理）带来显著的性能提升。 BFO是一个针对批量文件操作的优化解决方案，旨在克服现有文件系统在处理批量小文件时的性能瓶颈，通过并行化、预处理和优化的元数据管理来提高整体效率，从而在各种计算环境中提供更好的服务质量和用户体验。

cannot fully exploit the performance potentials of these new

hardware.

As a real-world actual example, a ﬁle set of the meteoro-

logical administration of Hubei Province of China, consists

of 8,639,303 weather sampling ﬁles (about 1.5TB in total)

collected from hundreds of locations in 5-years, and needs to

be migrated from a source hard disk with NTFS to a target

RAID array with ext4. As a result, it takes about two days to

duplicate all ﬁles via the USB3.0 interface. We also employed

conﬁgurable system-level optimizations such as large buffer,

prefetching, I/O scheduling, and hardware RAID with higher

bandwidth, however, to little avail. This motivates us to explore

the root cause of the inefﬁciency.

B. Problem Analysis

The single-ﬁle access pattern, using the standard POSIX

system calls, is universally applicable and effectively hides

sophistical internal implementation of ﬁle systems from the

applications. However, when accessing a batch of ﬁles, the

pattern needs to repeatedly pass through a full storage I/O

stack, and frequently read/write metadata and data on different

locations of the underlying storage device, resulting in many

non-sequential, random and often dependent I/Os. Therefore,

for batch-ﬁle access, this approach accumulates I/O overhead

of each ﬁle, potentially leading to very low efﬁciency.

256

1024

4096

16384

4KB 16KB 64KB 256KB 1MB 4MB

Exection time (s)

File size in different file sets

HDD_R

HDD_S

SSD_R

SSD_S

(a) Read

128

512

2048

8192

4KB 16KB 64KB 256KB 1MB 4MB

Exection time (s)

File size in different file sets

HDD_R

HDD_S

SSD_R

SSD_S

(b) Write

Fig. 1. The overall execution time of accessing different ﬁle sets on three

storage devices with different access orders. The y-axis is in log scale.

1) Inefﬁciency: In order to experimentally explore the inef-

ﬁciency of the single-ﬁle pattern in batch-ﬁle access situations,

we design a set of experiments to investigate the impact

of ﬁle size and access order on the overall performance.

We use Filebench [30] to generate multiple ﬁle sets with

the same total amount of data (i.e., 4GB) with different ﬁle

sizes (i.e., from 4KB to 4MB) and ﬁle counts on hard disk

and SSD under default ext4 conﬁguration. Every ﬁle set is

consecutively stored in the storage devices, which is an ideal

layout for sequential accesses. However, users are unaware of

the locations of all accessed ﬁles, and may access these ﬁles

in any order. Therefore, to simulate two extreme access cases,

we further read all ﬁles in each ﬁle set in totally sequential

and random manners, and collect their execution times, shown

in Figure 1(a). On the one hand, the execution time of the

random read for 4KB-sized ﬁles is up to 57.8× longer than

the sequential under the same read case, when using hard disk

as the underlying storage device. Even using SSD with higher

performance, for random access, there still exists about 2.6×

performance degradation compared to the sequential access

case for the ﬁle set. On the other hand, we also observe in

Figure 1(a) that the read performance of large-ﬁle set (i.e.,

4MB-sized ﬁles) gradually reaches the peak performance of

the storage devices, the performance of small-ﬁle set (i.e.,

below 1MB-sized ﬁles), however, is much lower than that of

large-ﬁle set in both access orders. For example, the sequential

case with small ﬁles (e.g. 4KB) is signiﬁcantly slower than

the same case with large ﬁles (e.g. 4MB) by about 5×. Notice

that they have the same consecutive ﬁle data layout, and it

takes about extra 28 seconds to access inodes of 4KB-sized

ﬁle set. Therefore, the consecutive ﬁle data is not fetched

sequentially. Likewise, the performances of updating (writing)

a batch of ﬁles under different conﬁgurations are illustrated in

Figure 1(b). The performance behaviors are still similar to the

previous read case.

In summary, the traditional single-ﬁle access approach is

very inefﬁcient for batch-ﬁle operations, especially for small

ﬁles (below 1MB) in a random manner, and can hardly make

full use of the underlying devices.

2) Storage Behavior: In order to better understand the I/O

behaviors under the single-ﬁle access pattern in typical ﬁle

systems, we employ blktrace [31] to capture I/O footprints

when accessing the Linux kernel source codes (ver 3.5.0) as

a real ﬁle set.

Figure 2 and Figure 3 illustrate the read and write be-

haviors respectively during accessing the ﬁle set with three

representative ﬁle systems, ext4 [9], Btrfs [10], and F2FS

[32]. The test ﬁle set is initially stored contiguously on the

storage device in the read case, and is totally buffered in

memory in the write case. Nevertheless, the expected large

and sequential I/Os for the ﬁle data are actually broken into

more, smaller, and potentially non-sequential read/write I/Os,

due to the interweaving between metadata and ﬁle data I/Os.

For the read operation, the underlying ﬁle systems ﬁrst

access ﬁle metadata to determine the location of each ﬁle

data, and then read the ﬁle data. Considering that the ﬁle data

and metadata are always stored in different disk locations,

each ﬁle read operation actually entails at least two I/Os to

access metadata and data respectively. On the other hand,

for these ﬁle systems, a ﬁle write operation ﬁrst modiﬁes

the ﬁle inode, and then update the global metadata (e.g.,

bitmap) to conﬁrm the allocated disk space, and ﬁnally writes

the data. For the journaling ﬁle systems like ext4 and XFS

[33], the write operation also invokes additional journaling

剩余12页未读，继续阅读

weixin_38613640

粉丝: 5
资源: 882

BFO：批量文件操作优化海量文件性能

matlab求导代码-BFO:BFO（Brute-ForceOptimizer），一种Matlab软件包，用于解决连续变量和/或离散变量和/或

obographviz:将OBO图转换为dotgraphviz

Si5351vfo_bfo:使用Si5351的VFO和BFO组合

Bacteria Foraging Optimization (BFO)：使用细菌觅食技术进行功能优化-matlab开发

Digital Image Watermarking Using Optimized DWT & DCT：此代码用于使用三种方法组合的隐形数字图像水印：DWT、DCT、BFO-matlab开发

BFO-PSO混合算法解决多目标问题

基于BFO-BP神经网络的储层预测研究.pdf

Matlab源码：BFO-XGBoost算法分类预测与优化

优化配送中心选址：BFO-AFSA算法的应用与优势

改进的BFO算法：基于Lévy飞行的优化策略

最新资源