深度学习协同算法、硬件与调度设计：突破性能瓶颈

需积分: 10 11 浏览量更新于2024-07-09 收藏 8.76MB PDF 举报

"《深度学习应用的算法、硬件与调度协同设计》是一篇由Qijing Huang撰写的博士论文，出自于加州大学伯克利分校电气工程与计算机科学系，报告编号为UCB/EECS-2021-202。该论文针对当前深度学习领域面临的挑战，提出了一个创新的解决方案，即算法、硬件和调度的联合设计（Co-design）。随着深度学习在计算机视觉、神经语言处理等领域取得突破性进展，对计算性能的需求日益增长。然而，传统的异构系统设计面临着硬件开发周期漫长、算法与硬件匹配度低以及编译器局限等问题。论文作者意识到，为了应对这些挑战，必须打破传统的界限，实现算法、硬件和调度的协同优化。论文的核心贡献包括：1) 开发一种自上而下的方法和工具，能够自动从高级抽象层生成定制化的加速器系统，显著缩短硬件开发周期；2) 应用机器学习和其他优化技术，改进加速器设计和编译流程，提升硬件效率；3) 推动算法与硬件的共同设计，发掘更多的性能优化潜力，尤其是在深度学习任务中。论文的焦点在于端到端的系统优化，通过硬件加速，促进前沿深度学习算法的广泛应用，从而影响和改变日常生活中的诸多方面。作者的目标是在深度学习的生态系统中，构建一个更高效、更具竞争力的基础设施，以满足未来智能设备日益增长的计算需求。这篇161页的PDF文档深入探讨了深度学习领域的技术革新，对于系统架构师、算法工程师和硬件开发者而言，提供了宝贵的理论支持和实践指导。通过这篇论文，读者可以了解到如何在硬件、软件和算法层面进行紧密合作，以推动深度学习技术的未来发展。"

List of Tables

2.1

Accelerator Performance (The workload size is represented as image_width

channel_depth) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.1 Macro-structure of DiracDeltaNet . . . . . . . . . . . . . . . . . . . . . . . . . . 37

3.2 ShuﬄeNetV2-1.0x vs. DiracDeltaNet . . . . . . . . . . . . . . . . . . . . . . . . 37

3.3 Quantization Result on DiracDeltaNet . . . . . . . . . . . . . . . . . . . . . . . 38

3.4 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

3.5 Resource Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

3.6 Performance comparison of Synetgy and the previous works. . . . . . . . . . . . 46

3.7 Frame Rate on Diﬀerent Batch Size . . . . . . . . . . . . . . . . . . . . . . . . . 46

3.8

Runtime Analysis for the First and Last DiracDeltaNet Blocks in Diﬀerent Oper-

ator Conﬁgurations (Batch=10) . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

3.9

Ablation study of operation choices for object detection on VOC and COCO. The

top half shows the baselines with various kernel sizes, from 3

3 to 9

9. The

bottom half shows the comparison of diﬀerent designs for deformable convolution. 55

3.10

Co-designed hardware performance comparison. The top half shows the perfor-

mance of codesigned hardware corresponding to each algorithmic changes to the

default 3

3 convolution. The bottom half shows the results for the depthwise

3×3 convolution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

3.11 Quantized CoDeNet on VOC object detection. . . . . . . . . . . . . . . . . . . . 63

3.12 Quantized CoDeNet on COCO object detection. . . . . . . . . . . . . . . . . . . 64

3.13 Performance comparison with prior works. . . . . . . . . . . . . . . . . . . . . . 65

3.14 FPGA resource utilization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

4.1 State-of-the-art DNN accelerator schedulers. . . . . . . . . . . . . . . . . . . . . 72

4.2 CoSA Notations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

4.3

Example binary matrix

representing a schedule. A checkmark in s, t indicates

spatial or temporal mapping. A checkmark in

, ..., O

indicates the rank for

loop permutation. In this schedule, the loop tile of size 3 from problem dimension

is allocated within the GlobalBuf at the innermost loop level, assigned for

temporal execution. Both loop tiles from K are mapped to spatial resources. . . 78

xiii

Acknowledgments

This thesis would not have been possible without the help, support, and guidance of many

wonderful individuals.

First and foremost, I want to thank my advisor, John Wawrzynek. My journey at Berkeley

would not have started without John, who recruited me to his research group. He has been

very kind and supportive and has taught me many important lessons in research and life. He

has frequently encouraged me to express my opinions and patiently listened to my immature

research thoughts in the early years. His patience makes it possible for me to mature as

an engineer and a researcher who can think critically and articulate ideas conﬁdently. He

allowed me to pursue my interests without questioning my ability to work on new topics. His

trust and support have made my Ph.D. incredibly enjoyable and rewarding. His fundamental

and insightful research advice has taught me how to view and approach novel problems

in hardware design and computer architecture. Besides research, he truly cares about my

well-being and makes active eﬀorts to create a friendly environment for me to thrive. I

learned to keep an easy mind in graduate school from the life wisdom he shared in various

conversations. I’m grateful to have John as my advisor and wish I could keep him as a

lifetime mentor and friend.

I would also like to express my gratitude to other faculty members at Berkeley, especially

Yakun Sophia Shao, Joseph Gonzalez, Kurt Keutzer, Krste Asanovic, James Demmel, Ion

Stoica, and Alan Mishchenko. Sophia has had numerous meetings with me to provide detailed

feedback and guidance for my research and career. She is an exceptional academic model

to me. Joey has always been very kind and resourceful. He has oﬀered me a lot of helpful

advice for academic career development. Kurt has given me insightful feedback from the

machine learning perspective and set up a welcoming environment for collaboration with

his group. Krste has generously shared his experience in computer architecture and circuit

design. Jim has given me his detailed and helpful advice on optimization problems. Ion has

helped us proofread papers and recommended me good career development strategies. Alan

has discussed many of his high-impact projects with me and has helped broaden my horizon.

I appreciate all their time and eﬀort in sharing their expertise with junior researchers like

me. Outside of school, I’m fortunate to have many amazing industrial mentors, who have

introduced me to challenging research problems, and generously shared their advice in various

areas of life and research: Aravind Kalaiah, Hamid Shojaei, Azade Nazi, Sat Chatterjee,

Shobha Vasudevan, Azalia Mirhoseini, Yuandong Tian, and Randy Huang. I want to thank

them for positively inﬂuencing me in various ways. I also want to give special thanks to

Aravind, Sophia, and Joey, for agreeing to be on my qualify and dissertation committee.

I’m thankful to have opportunities to collaborate with many bright and dedicated re-

searchers during graduate school time. I also would like to thank them for their contributions

to the projects:

On Algorithm-Hardware Co-design: Zhen Dong, Yizhao Gao, Bichen Wu, Yifan Yang,

Amir Golami, Dequan Wang, Zhewei Yao, Tianjun Zhang, Yaohui Cai, Tian Li. I

xiv

learned a great deal about the advancement in machine learning and computer vision

from them.

On High-Level Synthesis and Centrifuge: Christopher Yarp, Sagar Karandikar, Nathan

Pemberton, Benjamin Brock, Liang Ma, Guohao Dai, Robert Quitt, Shaoyi Cheng,

Tan Nguyen, Arya Reais-Parsi. They showed their exceptional engineering skills and

expertise in subjects including HLS, hardware, systems, and compilers.

On Machine Learning for Hardware and AutoPhase: Ameer Haj-Ali, William Moses,

John Xiang, Keertana Settaluri, Hasan Genc. It was a great pleasure exploring new

topics with these creative and productive individuals.

On integer linear programming and CoSA Scheduling: Grace Dinh, Josh Kang, Thomas

Norell, Charles Hong. I learned many optimization techniques from them.

My growth as a graduate student has been dependent on many of my colleagues who

generously share their compassion and technical expertise. I want to thank James Martin,

Angie Wang, David Biancolin, Howard Mao, Albert Magyar, Lisa Wu, Seah Kim, Adam

Izraelevitz, Dayeol Lee, Yang You, Marquita Ellis, Alon Amid, Albert Ou, Colin Schmidt,

Jerry Zhao, Gilbert Bernstein, and many others, for creating an inclusive lab and department

environment. I would like to express my gratitude to Kosta Ilov, who was always there to

help me with various infrastructure issues.

Outside of research, I gained tremendous support from my friends at Berkeley. I want to

thank Cecilia Zhang, Xin Wang, Tianjun Zhang, Zhewei Yao, Zixi Hu, Nathan Pemberton,

Ameer Haj-Ali, Sagar Karandikar, Paras Jain, Biye Jiang, Isla Yang, Alice Ye, Cindy Chen,

Jessica Zhou, Daniel Seita, Michael Chang, and Danyang Zhuo. I appreciate their company

in various activities, including the numerous boba trips. I also gain lots of care and love from

my IGSM friends: Sara Hong, Wenxia Lin, Jenny Chung, Grace Borja, and many others.

I’d also like to thank my friends from the University of Toronto: Angela Xu, Anqi Duan,

Joshua Chou, Chain Zhang, Jack Luo, Suya Liu, Jenny Ren, Jacky Liu, Lucia Xu, Dean

Xu, Xiang Chen. They showed extraordinary kindness to me, and I always think of them as

family. In addition, I want to thank my undergraduate research group, led by Prof. Jason

Anderson, for oﬀering research opportunities for undergraduate students and being incredibly

encouraging and supportive to us.

I am extremely thankful to have the opportunity to interact with many extraordinary

individuals and form a community together. I might not have mentioned all of you here,

but you have made my journey full of fun and inspired me in various ways. Through you, I

learned that we are all connected people and that something we do will aﬀect others. Moving

forward, I hope I could be like you and do my little things to enlighten the days of others.

Lastly, I cannot thank my family enough for their unceasing support and inspiration from

the beginning, in ways and depths that no words can express. They have taught me to work

hard, persevere, and seek truth in all circumstances.

Chapter 1

Introduction

I started my graduate study aiming to answer the following question:

How do we design the most eﬃcient hardware possible?

Such a question would not be meaningful without deﬁning the proper context. To start

the research, we ﬁrst identiﬁed four necessary contexts for understanding eﬃcient hardware

development.

Degree of Specializaton.

The hardware eﬃciency is limited by the inherent workload

properties, i.e., whether general function support is required. The more functions to support,

the more complex the logic needs to be. There is a inherent trade-oﬀ between generality and

eﬃciency in hardware architecture design. General-purpose processors, such as CPU and

GPU, are Turing-complete and ﬂexible enough to accomplish arbitrary functions. Accelerators,

contrarily, do not guarantee Turing completeness, and only support speciﬁc functions. The

beneﬁts of specialization include minimization of overhead in control logic, more aggressive

domain-speciﬁc optimizations, etc. Given the same technology node and physical constraints,

accelerators can achieve a signiﬁcant performance improvement compared to general-purpose

processors.

Development Cycle and Cost.

The second context to consider is the expected time-

to-market and costs. Generally speaking, the more human hours and resources spent in

development, the more optimized the hardware is likely to be. However, too long of a

development cycle can cause the products to be irrelevant by the time they go on the market

because many applications are rapidly evolving. In addition, there is an associated non-

recurring engineering (NRE) cost that restricts the resources we can dedicate to hardware

design and optimization. As a result, we cannot assume unlimited time and resources for

hardware development.

End-to-end Systems.

Hardware does not work in isolation. Its compatibility with

algorithms and software also signiﬁcantly impacts the overall task execution performance

and eﬃciency. The commonly used hardware speciﬁcations (e.g., FLOPs, TOPs) do not

reﬂect the actual performance for running speciﬁc algorithms. Depending on the operational

intensity of the algorithm design, the peak hardware performance can diﬀer. Similarly, we

剩余160页未读，继续阅读

努力+努力=幸运

粉丝: 2
资源: 136

深度学习协同算法、硬件与调度设计：突破性能瓶颈

Alsuwaiyel.-.Algorithms.Design.Techniques.and.Analysis.(World.Scientific,.1999)

Network Flows -- Theory, Algorithms, and Applicat.pdf

压缩感知算法python【学习资源】GitHub上的开源项目CS-Recovery-Algorithms

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

"Superpixel-based Deblurring Using DeblurGAN-v2"

matlab peaks

最新资源