没有合适的资源?快使用搜索试试~ 我知道了~
首页信息论视角解析深度神经网络黑箱
信息论视角解析深度神经网络黑箱
需积分: 12 4 下载量 157 浏览量
更新于2024-07-19
收藏 3.74MB PDF 举报
"《揭开深度神经网络的黑箱:信息论视角》是一篇深入研究深度学习理论的论文。作者们在文中提出了一个新颖的方法,即通过信息论的角度来理解深度神经网络(DNN)的学习过程及其内在组织。他们回顾了之前的研究,特别是Tishby和Zaslavsky(2015)的工作,这些研究者将DNN的学习过程映射到信息平面——一个由各层之间保留的输入和输出变量之间的互信息值构成的空间。 信息平面理论认为,深度神经网络的目标是逐步优化信息瓶颈(IB)原理,即在压缩输入信息与预测输出之间找到一个平衡。在这个过程中,每一层都在寻求最优的压缩和预测之间的权衡。压缩是指减少不必要的输入细节,而预测则是指利用这些信息进行准确的输出生成。 论文作者在这个基础上进一步探索,展示了信息平面可视化方法的有效性。通过这种可视化工具,研究人员可以直观地观察到网络内部的信息流动和决策过程,从而揭示了网络学习过程中哪些特征被保留,哪些被忽略,以及这些决策如何随网络层次加深而演变。这种理解有助于我们洞察网络的复杂行为,提高模型的可解释性和透明度,同时也有助于优化模型架构和训练策略,以提升整体性能。 这篇论文对深度学习的理论基础进行了深入剖析,强调了信息论在理解和设计深度神经网络中的核心作用。它不仅提供了一个新的分析框架,而且为未来的模型设计和优化提供了有价值的洞见。对于那些关心深度学习内部机制和可解释性的研究人员来说,这是一篇不可或缺的重要文献。"
资源详情
资源推荐
![](https://csdnimg.cn/release/download_crawler_static/10230688/bg4.jpg)
SCHWARTZ-ZIV AND TISHBY
2. Information Theory of Deep Learning
In supervised learning we are interested in good representations, T(X), of the input patterns x ∈ X,
that enable good predictions of the label y ∈ Y . Moreover, we want to efficiently learn such
representations from an empirical sample of the (unknown) joint distribution P (X, Y ), in a way
that provides good generalization.
DNNs and Deep Learning generate a Markov chain of such representations, the hidden layers,
by minimization of the empirical error over the weights of the network, layer by layer. This opti-
mization takes place via stochastic gradient descent (SGD), using a noisy estimate of the gradient
of the empirical error at each weight, through back-propagation.
Our first important insight is to treat the whole layer, T , as a single random variable, charac-
terized by its encoder, P (T |X), and decoder, P (Y |T ) distributions. As we are only interested in
the information that flows through the network, invertible transformations of the representations,
that preserve information, generate equivalent representations even if the individual neurons encode
entirely different features of the input. For this reason we quantify the representations by two num-
bers, or order parameters, that are invariant to any invertible re-parameterization of T , the mutual
information of T with the input X and the desired output Y .
Next, we quantify the quality of the layers by comparing them to the information theoretic
optimal representations, the Information Bottleneck representations, and then describe how Deep
Learning SGD can achieve these optimal representations.
2.1 Mutual Information
Given any two random variables, X and Y , with a joint distribution p(x, y), their Mutual Informa-
tion is defined as:
I(X; Y ) = D
KL
[p(x, y)||p(x)p(y)] =
X
x∈X,y∈Y
p(x, y) log
p (x, y)
p (x) p (y)
(1)
=
X
x∈X,y∈Y
p (x, y) log
p (x|y)
p (x)
= H(X) − H(X|Y ) , (2)
where D
KL
[p||q] is the Kullback-Liebler divergence of the distributions p and q, and H(X) and
H(X|Y ) are the entropy and conditional entropy of X and Y , respectively.
The mutual information quantifies the number of relevant bits that the input variable X contains
about the label Y , on average. The optimal learning problem can be cast as the construction of an
optimal encoder of that relevant information via an efficient representation - a minimal sufficient
statistic of X with respect to Y - if such can be found. A minimal sufficient statistic can enable
the decoding of the relevant information with the smallest number of binary questions (on average);
i.e., an optimal code. The connection between mutual information and minimal sufficient statistics
is discussed in 2.3.
Two properties of the mutual information are very important in the context of DNNs. The first
is its invariance to invertible transformations:
I (X; Y ) = I (ψ(X); φ(Y ))) (3)
for any invertible functions φ and ψ.
4
剩余18页未读,继续阅读
![pdf](https://img-home.csdnimg.cn/images/20210720083512.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://profile-avatar.csdnimg.cn/966e5154c7a84e3e8637231fd5bea193_mei86233824.jpg!1)
梅mmmmm
- 粉丝: 18
- 资源: 4
上传资源 快速赚钱
我的内容管理 展开
我的资源 快来上传第一个资源
我的收益
登录查看自己的收益我的积分 登录查看自己的积分
我的C币 登录后查看C币余额
我的收藏
我的下载
下载帮助
![](https://csdnimg.cn/release/wenkucmsfe/public/img/voice.245cc511.png)
最新资源
- Hadoop生态系统与MapReduce详解
- MDS系列三相整流桥模块技术规格与特性
- MFC编程:指针与句柄获取全面解析
- LM06:多模4G高速数据模块,支持GSM至TD-LTE
- 使用Gradle与Nexus构建私有仓库
- JAVA编程规范指南:命名规则与文件样式
- EMC VNX5500 存储系统日常维护指南
- 大数据驱动的互联网用户体验深度管理策略
- 改进型Booth算法:32位浮点阵列乘法器的高速设计与算法比较
- H3CNE网络认证重点知识整理
- Linux环境下MongoDB的详细安装教程
- 压缩文法的等价变换与多余规则删除
- BRMS入门指南:JBOSS安装与基础操作详解
- Win7环境下Android开发环境配置全攻略
- SHT10 C语言程序与LCD1602显示实例及精度校准
- 反垃圾邮件技术:现状与前景
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈
![](https://img-home.csdnimg.cn/images/20220527035711.png)
![](https://img-home.csdnimg.cn/images/20220527035711.png)
![](https://img-home.csdnimg.cn/images/20220527035111.png)
安全验证
文档复制为VIP权益,开通VIP直接复制
![](https://csdnimg.cn/release/wenkucmsfe/public/img/green-success.6a4acb44.png)