深度信念网络固定点运算精度评估

27 浏览量更新于2024-08-29 收藏 714KB PDF 举报

"这篇研究论文‘Accuracy Evaluation of Deep belief Networks with Fixed-Point Arithmetic’探讨了在使用固定点算术实现深度信念网络（DBNs）时对精度的影响。文章由Jingfei Jiang、Rongdong Hu、Mikel Lujan和Yong Dou撰写，发表在2014年的‘Computer Modelling & New Technologies’期刊第18卷第6期，页码7-14。" 深度信念网络（DBNs）是机器学习领域最先进的技术之一，同时也是无监督学习算法中的关键组成部分。由于DBN的训练过程计算量大，因此对于FPGA（现场可编程门阵列）加速的研究具有重要意义。FPGA能够通过硬件定制化来提升运算速度，而固定点算术则可以在FPGA上实现DBN，从而缩短执行时间。然而，固定点算术的应用可能对网络的精度产生影响。过去的研究所关注的是在特定位宽下加速器的表现，而本文的创新之处在于通过实验评估全面展示了位宽变化对各种DBN配置的精度影响。论文中进行了明确的性能分析，揭示了不同位宽设置如何影响DBN的准确度和效率。作者们可能通过比较浮点和固定点运算的结果，分析了位宽减少对网络权重表示精度、激活函数的精度以及总体分类性能的影响。他们可能还探讨了在保持相似计算速度的同时，如何选择合适的位宽以最小化精度损失。此外，论文可能还讨论了在实际应用中，如何平衡精度与硬件资源利用之间的关系，这对于嵌入式系统和资源受限的环境尤为重要。固定点算术的使用可以降低计算的复杂性和功耗，但必须谨慎处理，以防止精度过度下降导致的模型性能退化。通过这种详尽的实验评估，论文为DBN在FPGA上的高效且精度可控的实现提供了关键的理论基础和实践指导。这样的研究成果对于优化深度学习系统的硬件设计和进一步提升计算效率有着深远的影响。

COMPUTER MODELLING & NEW TECHNOLOGIES 2014 18(6) 7-14 Jiang Jingfei, Hu Rongdong, Mikel Lujan, Dou Yong

Mathematical and Computer Modelling

Accuracy evaluation of deep belief networks with fixed-point

arithmetic

Jingfei Jiang

, Rongdong Hu

, Lujάn Mikel

, Yong Dou

Science and Technology on Parallel and Distributed Processing Laboratory, National University of Defense Technology, ChangSha, Hunan 410073,

China

University of Manchester, Manchester, M13 9PL, UK

Received 12 June 2014, www.tsi.lv

Abstract

Deep Belief Networks (DBNs) are state-of-art Machine Learning techniques and one of the most important unsupervised learning

algorithms. Training DBNs is computationally intensive which naturally leads to investigate FPGA acceleration. Fixed-point

arithmetic can be used when implementing DBNs in FPGAs to reduce execution time, but it is not clear the implications for

accuracy. Previous studies have focused only on accelerators using some fixed bit-widths. A contribution of this paper is to

demonstrate the bit-width effect on various configurations of DBNs in a comprehensive way by experimental evaluation. Explicit

performance changing points are found using various bit-widths. The impact of sigmoid function approximation, required part of

DBNs, is evaluated. A solution of mixed bit-widths DBN is proposed, fitting the bit-widths of FPGA primitives and gaining similar

performance to the software implementation. Our results provide a guide to inform the design choices on bit-widths when

implementing DBNs in FPGAs documenting clearly the trade-off in accuracy.

Keywords: deep belief network, fixed-point arithmetic, bit-width, FPGA

* Corresponding author e-mail: jingfeijiang@nudt.edu.cn

1 Introduction

Deep neural networks have become a “hot topic” in the

Machine Learning community with successful results

demonstrated with Deep Belief Networks (DBNs) [1],

denoising autoencoder [2], sparse coding [3] and etc.

DBNs have been shown to be among the best neural

networks even for challenging recognition, mining and

synthesis tasks. A DBN is built on a subset of neural

networks known as Restricted Boltzmann Machine

(RBM). Running a DBN is a time-consuming task due to

its large scale and processing characteristics. Many

experiments have often reported taking weeks, to search

the large parameter space (numbers of layers and

neurons, learning rate, momentum and all kinds of

regulation terms) and calculate millions of parameters

(weights and biases). One good example is Quoc et al. [4]

who used a cluster in Google of 1,000 machines (16,000

cores) for a week to demonstrate the success of larger

scale unsupervised learning from internet images

recognition.

Reducing the execution time of the training phase and

prediction of a DBN is one critical barrier which has

restricted the mass adoption of DBNs. Interest in the

acceleration of DBNs has built up in recent years. FPGAs

are attractive platforms for accelerating DBNs. For

example, a RBM of 256x256 nodes was tested on a

platform of four Xilinx Virtex II FPGAs and gained a

speedup of 145-fold over an optimized C program

running on a 2.8-GHz Intel processor [5]. Using Altera

Stratix III FPGA, Kim et al. [6] also gained significant

speedup for a 256x1024 RBM. Multi-FPGA solutions

were discussed to determine the extensibility of RBM in

[7, 8].

Existing works on FPGA implementations of neural

networks often have vast and regular processing units to

map neurons partially or wholly at a time. Weights and

neuron values are stored in on-chip RAM during

processing and are swapped out to off-chip memory after

processing. It is too expensive to support a large number

of floating-point units on chip and store values using the

standard double precision floating-point representations

in on-chip RAMs. Many of the previous attempts with

FPGAs for neural networks implemented fixed bit-widths

(8 bits, 16 bits or 32 bits). Bit-widths with integral

multiple of bytes are convenient to align with other

components (such as IP cores and user interfaces) and

easier to design. Previous works have mainly analysed

the impact of bit-widths on accuracy and execution time

of old-style neural networks [9-11]. All reported RBM (a

building component of DBN) designs on FPGA selected

fixed-point arithmetic with a fixed bit-width as well, e.g.

16 bits in [6, 8] or 32 bits in [5] without analyzing in

depth the implications for accuracy. Thus, it is not clear

whether this kind of fixed bit-width is really the most

suitable and area efficient for DBNs.

Using bit-width unequal to the machine word-length

on a standard processor or GPU may rarely deliver any

下载后可阅读完整内容，剩余7页未读，立即下载

weixin_38656676

粉丝: 5
资源: 950

深度信念网络固定点运算精度评估

【5】Very deep convolutional networks for large-scale image recognition.pdf

Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition

Measurement accuracy verification of aspheric surface test with computer-generated hologram

deep-learning-with-python-notebooks中文版.docx

long-short-term-memory-networks-with-python

DEEP-LEARNING-WITH-TENSORFLOW:使用TensorFlow教程进行深度学习

Deep-Learning-with-PyTorch（经典英文原版教材）.pdf

SOUND SOURCE LOCALIZATION BASED ON DEEP NEURAL NETWORKS WITH DIRECTIONAL ACTIVATE FUNCTION EXPLOITING PHASE INFORMATION

Factors associated with the accuracy of self-reported high-school grades

Close-range optical measurement of aircraft's 3D attitude and accuracy evaluation

最新资源