CHCF：云上异构计算加速大规模图像检索

164 浏览量更新于2024-08-26 收藏 2.69MB PDF 举报

"CHCF是基于云的异构计算框架，专为大规模图像检索设计，旨在利用云计算和异构计算资源提高检索效率。该框架由一系列工具和技术构成，包括编译、优化和执行多媒体挖掘应用的方法。通过CHCF，开发者可以快速高效地构建多媒体应用，并在异构系统中实现最佳性能。" 在过去的十年里，多媒体内容和应用的急剧增长对计算资源的需求呈指数级上升。与此同时，高性能计算领域正朝着异构计算的方向发展，即由不同类型的处理器（如CPU、GPU、FPGA等）组成的混合系统。然而，充分利用这些异构系统来开发特定领域的应用程序，同时最大化系统效率，是一项具有挑战性的任务。 CHCF（Cloud-Based Heterogeneous Computing Framework）是一个创新解决方案，它针对大规模图像检索问题，提供了全面的工具集和技术。CHCF的核心在于其编译器和实用程序库，它们能够帮助用户简化开发过程，快速构建针对多媒体挖掘的应用程序，并确保在异构硬件上实现高效的执行。 CHCF框架采用了多种技术来优化性能，可能包括但不限于： 1. **编译器优化**：通过分析应用的计算密集型部分，编译器可以自动进行代码转换，以适应不同硬件架构的特点，例如，将适合GPU并行处理的部分进行并行化。 2. **任务调度**：智能调度算法决定何时何地运行各个任务，以减少数据传输延迟，最大限度地减少计算资源的空闲时间。 3. **数据管理**：高效的数据存储和访问策略，确保大数据集在多设备间快速有效地移动。 4. **负载均衡**：动态调整工作负载，避免单个组件过载，确保整体系统的稳定性和性能。 5. **通信优化**：减少异构系统中不同处理器之间的通信开销，提高整体系统效率。 6. **能耗管理**：考虑能源效率，根据工作负载调整硬件的功耗状态，平衡性能与能耗。通过CHCF，开发者无需深入理解底层硬件的复杂性，就能充分利用云计算的弹性扩展能力和异构硬件的并行计算能力。这使得CHCF成为应对大规模图像检索问题的理想选择，尤其是在需要处理海量图片库时，能显著提高检索速度和精度。总结来说，CHCF是一个综合性的框架，它将复杂的异构计算环境抽象化，提供了一种高效、易用的方式来开发和执行多媒体挖掘应用，特别是在大规模图像检索场景下。这一框架的出现，对于推动多媒体处理技术的发展，以及解决因数据量激增带来的计算挑战，具有重要的实践意义。

1902 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 25, NO. 12, DECEMBER 2015

applied to multimedia mining, and the results of k-means

clustering and background substraction have been presented.

Moise et al. [15] propose to employ MapReduce to accelerate

searching a large amount of images. MapReduce is used

to ﬁnd the nearest neighbors for the generation of image

tags in [16]. Compared with MapReduce that abstracts the

computations into map and reduce operations, the proposed

CHCF is more general in terms of the operations using ﬁlter

paradigm; moreover, the proposed CHCF employs adaptive

data partitioning and knowledge-based hierarchical scheduling

to address the challenge of load imbalance.

As far as the heterogeneous cluster is concerned, several

frameworks or paradigms have been designed in recent years.

In [17], StarPU is introduced for numerical kernel design

to execute parallel tasks on a shared-memory machine with

heterogeneous hardwares. Since the number of processors in

a shared-memory machine is limited, the predeﬁned schedul-

ing strategy employed by StarPU improves the performance

efﬁciently. However, the scheduler proposed by StarPU may

become ineffective when ported onto distributed systems. This

is the reason why we design the knowledge-based hierarchical

scheduler, which is hybrid in the sense of static and dynamic

scheduling policies and thus more suitable for distributed

systems. In [18], a high-level model for parallel programming,

compiler, and runtime, called merge, is proposed, which

is similar to StarPU. Merge also aims at shared-memory

machines with heterogeneous hardwares and employs the

MapReduce programming paradigm to simplify the interface

for users, which in turn constrains its applicable range. In [19],

an elastic computing framework is proposed, which uses an

adapter to hide the difference of various kinds of processors,

such as GPU, Intel Phi MIC, and even ﬁeld-programmable gate

array. However, the elastic computing framework focuses on

the design of elastic functions and has few discussions about

scheduling. As inspired by the elastic computing framework,

the proposed CHCF employs a similar technique called ﬁlter

interface to hide the difference of the underlying processors

and remain future extension to other processors. He et al. [20]

propose a GPU-based MapReduce framework Mars, which

evaluates the beneﬁts of cooperative usage of CPUs and GPUs

on a number of traditional applications such as string matching

and matrix multiplication. However, the scheduling policy of

Mars is not mentioned in [20].

In addition to performance and scalability, a number of

works have made efforts on programmability, which share

similarities with the proposed CHCF on the goal of facilitating

domain-speciﬁc application development. In [21], a compiler

is proposed to enable users to write hybrid CPU/GPU code

by utilizing the OpenMP [22] directives. Ghoting et al. [23]

offer a high-level declarative language SystemML for pro-

gramming machine learning applications that are compiled

and executed with MapReduce. Compared with SystemML,

the proposed CHCF compiler is hybrid in the sense that

C/C++ and Java can be utilized to develop ﬁlters. Ma and

Agrawal [24] propose a code generation system, referred

to as AUTO-GC, to translate data mining applications on

GPU clusters. The results of two popular data mining algo-

rithms, including k-means clustering and principle component

analysis, are reported to demonstrate that a good scalability

is accomplished without noticeable overheads of the trans-

lated code. Compared with AUTO-GC that is actually a

code generation system for reduction operations, the pro-

posed CHCF provides a more general programming para-

digm with ﬁlters. Lee et al. [25] develop a similar compiler

framework for translating standard OpenMP applications to

Compute Uniﬁed Device Architecture-based general purpose

GPU applications. However, this compiler is operating in a

source-to-source translation manner.

Regarding load imbalance, the centralized paradigm is

widely used by scheduling policies such as [26] and [27],

which employs a master node to take charge of scheduling.

Although this paradigm is simple and effective when the num-

ber of nodes of the system is relatively small, the limitations

are obvious. First, the effectiveness may be sharply degraded

when the number of nodes increases. Second, the scheduling

made by the master is one-time assignment, but the informa-

tion for making the scheduling decision is constantly changing.

To overcome the disadvantage of centralized scheduling,

a distributed load balancing approach, called distributed

adaptive scheduling (DAS), is proposed in [28] for scientiﬁc

applications expressed as iterative loops with dependencies.

DAS combines the information about the run queue with the

timing history of each computation worker to make schedul-

ing. It is veriﬁed that DAS can achieve signiﬁcant performance

improvements over centralized approaches. However, as high-

performance clusters become heterogeneous and the scale of

the clusters continues to increase, DAS meets some challenges.

Since the computation workers have to communicate with

each other to make scheduling decisions, the overheads for

network communication may hurt the overall performance and

the effectiveness of scheduling when the number of nodes is

large. Moreover, the efﬁciency of DAS may become degraded

when the nodes are equipped with multiple GPUs. In contrary

to DAS, the proposed knowledge-based hierarchical scheduler

is masterless, which means that scheduling decisions are made

by the master scheduler and node schedulers cooperatively.

In another word, the master scheduler coordinates the node

schedulers and the node schedulers make scheduling decisions.

Therefore, the communication overheads are reduced since the

scheduling efﬁciency is improved by distributing scheduling

computations from the master scheduler to node schedulers.

III. CHCF O

VERVIEW

A. Architecture

The overall architecture of the proposed CHCF is depicted

in Fig. 1. As aforementioned, CHCF aims at developing and

executing multimedia mining applications on hybrid systems.

To achieve this, CHCF provides a high-level programming

language coupled with a utility library, in which multimedia

mining algorithms can be implemented for running with CPUs

and/or GPUs, and a set of tools for compilation, execution, and

optimization are designed. In CHCF, the popular programming

paradigm, called ﬁlter stream [29], is adopted; therefore,

all computing units, either atomic or compound, are repre-

sented by ﬁlters and effectively communicate with each other

剩余13页未读，继续阅读

weixin_38643269

粉丝: 2
资源: 902

CHCF：云上异构计算加速大规模图像检索

辽宁省葫芦岛一中2015_2016学年高二化学上学期期中试题含解析

ATV312施耐德变频器参数设置-(简易)

大数据量计算全息相关滤波器

SL-ST 差速器3D模型 SL-ST 差速器

C#大型药品进销存管理系统源码数据库 Access源码类型 WinForm

JAVAKTV点歌系统源码数据库 MySQL源码类型 WinForm

树叶形状、分布与树枝结构关系及其质量估算模型研究

大数据1+x(蓝桥课堂实操231216）解析

阿里云的yum源，替换CentOS的yum源

基于JAVA+SpringBoot+MySQL的职称评审管理系统lw设计与实现.docx

最新资源