哈希到CNN：二进制权重网络的训练方法

需积分: 10 104 浏览量更新于2024-08-13 1 收藏 809KB PDF 举报

"这篇研究论文探讨了如何将哈希技术应用于深度卷积神经网络（CNNs）中，以训练二进制权重网络，从而降低内存和计算资源的需求，便于在移动设备上部署。作者提出了一个新的方法，名为BWNH（Binary Weight Networks via Hashing），并揭示了内积保持哈希与二进制权重网络之间的紧密联系，表明通过哈希可以有效地训练二进制权重网络。" 在计算机视觉领域，深度卷积神经网络（CNNs）已经在各种任务中表现出优异的性能，这促使人们尝试将其应用于现实世界。然而，大多数先进的CNN模型需要大量的内存和计算资源，这成为在移动设备上部署的一大障碍。为了解决这个问题，研究者们开始探索低比特权重表示的可能性，这种方法可以显著减少存储需求，并提高网络推理的效率。本文提出的BWNH方法是一种新颖的训练策略，它利用哈希技术来生成二进制权重。哈希函数通常用于将高维度数据映射到固定长度的二进制码，而这里的研究发现，这种映射过程可以保留内积特性，这对于保持权重网络的计算效果至关重要。二进制权重网络的权重不再是连续值，而是仅包含0和1的二进制数，这样大幅度降低了存储量，同时简化了计算步骤。在BWNH方法中，训练过程被设计为寻找一种哈希方案，使得经过哈希后的二进制权重尽可能地接近原始的浮点权重，同时保持网络的预测能力。通过这种方式，二进制权重网络可以在保持良好性能的同时，实现资源的高效利用。实验部分，作者会对比BWNH与其他低比特表示方法在图像分类、目标检测等任务上的性能，以证明其优势。此外，论文可能还会分析不同哈希策略对网络性能的影响，以及在实际硬件平台上的速度和功耗表现。这篇研究论文对深度学习社区具有重要的贡献，它提供了一种新的、基于哈希的二进制权重网络训练方法，有望推动CNN在资源受限的设备上的广泛应用。这种方法不仅有助于优化模型的存储需求，还可能加速模型的运行速度，是低功耗计算和嵌入式系统领域的前沿研究之一。

From Hashing to CNNs: Training

Binary Weight Networks via Hashing

Qinghao Hu, Peisong Wang, Jian Cheng

Institute of Automation, Chinese Academy of Sciences, Beijing, China

University of Chinese Academy of Sciences, Beijing, China

Center for Excellence in Brain Science and Intelligence Technology, CAS, Beijing, China

{qinghao.hu, peisong.wang, jcheng}@nlpr.ia.ac.cn

Abstract

Deep convolutional neural networks (CNNs) have shown ap-

pealing performance on various computer vision tasks in re-

cent years. This motivates people to deploy CNNs to real-

world applications. However, most of state-of-art CNNs re-

quire large memory and computational resources, which hin-

ders the deployment on mobile devices. Recent studies show

that low-bit weight representation can reduce much storage

and memory demand, and also can achieve efﬁcient network

inference. To achieve this goal, we propose a novel approach

named BWNH to train Binary Weight Networks via Hash-

ing. In this paper, we ﬁrst reveal the strong connection be-

tween inner-product preserving hashing and binary weight

networks, and show that training binary weight networks can

be intrinsically regarded as a hashing problem. Based on this

perspective, we propose an alternating optimization method

to learn the hash codes instead of directly learning binary

weights. Extensive experiments on CIFAR10, CIFAR100 and

ImageNet demonstrate that our proposed BWNH outper-

forms current state-of-art by a large margin.

Introduction

Since Alexnet (Krizhevsky, Sutskever, and Hinton 2012)

made a success in ILSVRC2012 (Russakovsky et al. 2015),

deep convolutional neural networks have become more and

more popular. After that, various CNN models have been

proposed such as VGGNet (Simonyan and Zisserman 2014),

Inception (Szegedy et al. 2016), ResNet (He et al. 2016) and

so on. Nowadays, these CNN models have been playing an

important role in many computer vision areas (Krizhevsky,

Sutskever, and Hinton 2012; Ren et al. 2015; Long, Shel-

hamer, and Darrell 2015).

Attracted by the great performance of CNN models, many

people try to deploy CNNs to real world applications. Yet

the huge computational complexity and large parameter size

make CNN models hard to deploy on resource limited de-

vices such as mobile phones and embedded devices. The

huge computational complexity of CNN models makes the

inference phase very slow, which is unacceptable for many

real-time applications. The large parameter size brings three

difﬁculties. First, the large parameter size means that de-

ploying CNN models will consume huge disk storage. Sec-

ond, much run-time memory is required, which is limited in

 2018, Association for the Advancement of Artiﬁcial

many mobile devices. Third, large parameter size will cause

heavy DRAM access, which consumes more energy. Since

battery power is very limited in many mobile devices, this

severely affects devices’ battery life.

To alleviate these problems, a variety of methods have

been proposed to reduce the parameter size or acceler-

ate the inference phase. These methods can be divided

into three main categories: low-rank decomposition based

methods, pruning-based methods, and quantization based

methods. Low-rank decomposition based methods (Den-

ton et al. 2014; Jaderberg, Vedaldi, and Zisserman 2014;

Zhang et al. 2015; Wang and Cheng 2016) decompose a

weight matrix (tensor) into several small weight matrices

(tensors). These methods achieve good speed-ups for large

convolutional kernels, but usually perform poorly for small

kernels. Besides, the compression ratio of parameters is kind

of low by using low-rank based methods. Network prun-

ing has a long history and is still a widely used technique

for CNN compression and acceleration (Han et al. 2015;

Liu et al. 2015). The main idea of these methods is to

remove low-saliency parameters or small-weight connec-

tions (Han et al. 2015). In general, after the pruning step,

k-means clustering and Huffman coding are also required

to make a good compression ratio. The k-means cluster-

ing and Huffman coding bring inconvenience for inference

phase since we have to decode the Huffman codes and

use lookup table for k-means dictionary. As a result, the

decoding and lookup table will bring extra memory and

computational overhead. Quantization based methods in-

clude codebook based quantization methods and low-bit

weight representation. Codebook based quantization meth-

ods mainly use vector quantization algorithms such as K-

means, Product Quantization and so on (Gong et al. 2014;

Wu et al. 2016) to quantize the weight kernels. These meth-

ods require lookup tables to store the dictionary, and is un-

friendly to cache memory since accessing lookup tables is

random and unordered. Low-bit weight representation meth-

ods (Lin, Talathi, and Annapureddy 2015; Gupta et al. 2015;

Rastegari et al. 2016; Dong et al. 2017) represent weights

as low-bit ﬁxed point even binary values. Low-bit weight

representation can reduce run-time memory and storage de-

mand as no decoding or lookup tables are required. As a

special case of low-bit weight representation, binary weight

can achieve about 32× compression ratio. In addition, since

The Thirty-Second AAAI Conference

on Artificial Intelligence (AAAI-18)

3247

下载后可阅读完整内容，剩余7页未读，立即下载

weixin_38657465

粉丝: 7
资源: 948

哈希到CNN：二进制权重网络的训练方法

用于快速图像检索的二进制哈希码的深度学习_ C++代码_下载

二进制比较与反二进制比较

基于深度哈希卷积神经网络的医学图像检索.pdf

深度学习有关哈希的论文学习

多哈希孪生网络：短视频相似度检测系统开发

特征工程从零开始：手把手教你打造世界级特征

揭秘算术运算在计算机科学中的15个应用：从基础到前沿

OpenCV图像ROI截取：从入门到精通的进阶教程，助你成为图像处理高手

【模型优化专家】：交互特征编码的全面指南，从基础到高级应用

字符转数字算法Java：从零基础到精通，解锁数据转换的秘密

最新资源