没有合适的资源?快使用搜索试试~ 我知道了~
首页Information Theory - Inference and Learning Algorithms, David MacKay
资源详情
资源推荐
![](https://csdnimg.cn/release/download_crawler_static/659168/bg1.jpg)
Information Theory, Inference, and Learning Algorithms
David J.C. MacKay
![](https://csdnimg.cn/release/download_crawler_static/659168/bg2.jpg)
Information Theory,
Inference,
and Learning Algorithms
David J.C. MacKay
mackay@mrao.cam.ac.uk
c
1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005
c
Cambridge University Press 2003
Version 7.2 (fourth printing) March 28, 2005
Please send feedback on this book via
http://www.inference.phy.cam.ac.uk/mackay/itila/
Version 6.0 of this book was published by C.U.P. in September 2003. It will
remain viewable on-screen on the above website, in postscript, djvu, and pdf
formats.
In the second printing (version 6.6) minor typos were corrected, and the book
design was slightly altered to modify the placement of section numbers.
In the third printing (version 7.0) minor typos were corrected, and chapter 8
was renamed ‘Dependent random variables’ (instead of ‘Correlated’).
In the fourth printing (version 7.2) minor typos were corrected.
(C.U.P. replace this page with their own page ii.)
![](https://csdnimg.cn/release/download_crawler_static/659168/bg3.jpg)
Contents
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
1 Introduction to Information Theory . . . . . . . . . . . . . 3
2 Probability, Entropy, and Inference . . . . . . . . . . . . . . 22
3 More about Inference . . . . . . . . . . . . . . . . . . . . . 48
I Data Compression . . . . . . . . . . . . . . . . . . . . . . 65
4 The Source Coding Theorem . . . . . . . . . . . . . . . . . 67
5 Symbol Codes . . . . . . . . . . . . . . . . . . . . . . . . . 91
6 Stream Codes . . . . . . . . . . . . . . . . . . . . . . . . . . 110
7 Codes for Integers . . . . . . . . . . . . . . . . . . . . . . . 132
II Noisy-Channel Coding . . . . . . . . . . . . . . . . . . . . 137
8 Dependent Random Variables . . . . . . . . . . . . . . . . . 138
9 Communication over a Noisy Channel . . . . . . . . . . . . 146
10 The Noisy-Channel Coding Theorem . . . . . . . . . . . . . 162
11 Error-Correcting Codes and Real Channels . . . . . . . . . 177
III Further Topics in Information Theory . . . . . . . . . . . . . 191
12 Hash Codes: Codes for Efficient Information Retrieval . . 193
13 Binary Codes . . . . . . . . . . . . . . . . . . . . . . . . . 206
14 Very Good Linear Codes Exist . . . . . . . . . . . . . . . . 229
15 Further Exercises on Information Theory . . . . . . . . . . 233
16 Message Passing . . . . . . . . . . . . . . . . . . . . . . . . 241
17 Communication over Constrained Noiseless Channels . . . 248
18 Crosswords and Codebreaking . . . . . . . . . . . . . . . . 260
19 Why have Sex? Information Acquisition and Evolution . . 269
IV Probabilities and Inference . . . . . . . . . . . . . . . . . . 281
20 An Example Inference Task: Clustering . . . . . . . . . . . 284
21 Exact Inference by Complete Enumeration . . . . . . . . . 293
22 Maximum Likelihood and Clustering . . . . . . . . . . . . . 300
23 Useful Probability Distributions . . . . . . . . . . . . . . . 311
24 Exact Marginalization . . . . . . . . . . . . . . . . . . . . . 319
25 Exact Marginalization in Trellises . . . . . . . . . . . . . . 324
26 Exact Marginalization in Graphs . . . . . . . . . . . . . . . 334
27 Laplace’s Method . . . . . . . . . . . . . . . . . . . . . . . 341
![](https://csdnimg.cn/release/download_crawler_static/659168/bg4.jpg)
28 Model Comparison and Occam’s Razor . . . . . . . . . . . 343
29 Monte Carlo Methods . . . . . . . . . . . . . . . . . . . . . 357
30 Efficient Monte Carlo Methods . . . . . . . . . . . . . . . . 387
31 Ising Models . . . . . . . . . . . . . . . . . . . . . . . . . . 400
32 Exact Monte Carlo Sampling . . . . . . . . . . . . . . . . . 413
33 Variational Methods . . . . . . . . . . . . . . . . . . . . . . 422
34 Independent Component Analysis and Latent Variable Mod-
elling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437
35 Random Inference Topics . . . . . . . . . . . . . . . . . . . 445
36 Decision Theory . . . . . . . . . . . . . . . . . . . . . . . . 451
37 Bayesian Inference and Sampling Theory . . . . . . . . . . 457
V Neural networks . . . . . . . . . . . . . . . . . . . . . . . . 467
38 Introduction to Neural Networks . . . . . . . . . . . . . . . 468
39 The Single Neuron as a Classifier . . . . . . . . . . . . . . . 471
40 Capacity of a Single Neuron . . . . . . . . . . . . . . . . . . 483
41 Learning as Inference . . . . . . . . . . . . . . . . . . . . . 492
42 Hopfield Networks . . . . . . . . . . . . . . . . . . . . . . . 505
43 Boltzmann Machines . . . . . . . . . . . . . . . . . . . . . . 522
44 Supervised Learning in Multilayer Networks . . . . . . . . . 527
45 Gaussian Processes . . . . . . . . . . . . . . . . . . . . . . 535
46 Deconvolution . . . . . . . . . . . . . . . . . . . . . . . . . 549
VI Sparse Graph Codes . . . . . . . . . . . . . . . . . . . . . 555
47 Low-Density Parity-Check Codes . . . . . . . . . . . . . . 557
48 Convolutional Codes and Turbo Codes . . . . . . . . . . . . 574
49 Repeat–Accumulate Codes . . . . . . . . . . . . . . . . . . 582
50 Digital Fountain Codes . . . . . . . . . . . . . . . . . . . . 589
VII Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . 597
A Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 598
B Some Physics . . . . . . . . . . . . . . . . . . . . . . . . . . 601
C Some Mathematics . . . . . . . . . . . . . . . . . . . . . . . 605
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 620
![](https://csdnimg.cn/release/download_crawler_static/659168/bg5.jpg)
Preface
This book is aimed at senior undergraduates and graduate students in Engi-
neering, Science, Mathematics, and Computing. It expects familiarity with
calculus, probability theory, and linear algebra as taught in a first- or second-
year undergraduate course on mathematics for scientists and engineers.
Conventional courses on information theory cover not only the beauti-
ful theoretical ideas of Shannon, but also practical solutions to communica-
tion problems. This book goes further, bringing in Bayesian data modelling,
Monte Carlo methods, variational methods, clustering algorithms, and neural
networks.
Why unify information theory and machine learning? Because they are
two sides of the same coin. In the 1960s, a single field, cybernetics, was
populated by information theorists, computer scientists, and neuroscientists,
all studying common problems. Information theory and machine learning still
belong together. Brains are the ultimate compression and communication
systems. And the state-of-the-art algorithms for both data compression and
error-correcting codes use the same tools as machine learning.
How to use this book
The essential dependencies between chapters are indicated in the figure on the
next page. An arrow from one chapter to another indicates that the second
chapter requires some of the first.
Within Parts I, II, IV, and V of this book, chapters on advanced or optional
topics are towards the end. All chapters of Part III are optional on a first
reading, except perhaps for Chapter 16 (Message Passing).
The same system sometimes applies within a chapter: the final sections of-
ten deal with advanced topics that can be skipped on a first reading. For exam-
ple in two key chapters – Chapter 4 (The Source Coding Theorem) and Chap-
ter 10 (The Noisy-Channel Coding Theorem) – the first-time reader should
detour at section 4.5 and section 10.4 respectively.
Pages vii–x show a few ways to use this book. First, I give the roadmap for
a course that I teach in Cambridge: ‘Information theory, pattern recognition,
and neural networks’. The book is also intended as a textbook for traditional
courses in information theory. The second roadmap shows the chapters for an
introductory information theory course and the third for a course aimed at an
understanding of state-of-the-art error-correcting codes. The fourth roadmap
shows how to use the text in a conventional course on machine learning.
v
剩余639页未读,继续阅读
![pdf](https://img-home.csdnimg.cn/images/20210720083512.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![txt](https://img-home.csdnimg.cn/images/20210720083642.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://profile-avatar.csdnimg.cn/2e497364f2444e07a01a219165b92914_pongba.jpg!1)
pongba
- 粉丝: 1w+
- 资源: 8
上传资源 快速赚钱
我的内容管理 收起
我的资源 快来上传第一个资源
我的收益
登录查看自己的收益我的积分 登录查看自己的积分
我的C币 登录后查看C币余额
我的收藏
我的下载
下载帮助
![](https://csdnimg.cn/release/wenkucmsfe/public/img/voice.245cc511.png)
会员权益专享
最新资源
- 基于Springboot的医院信管系统
- 基于Springboot的冬奥会科普平台
- 基于Springboot的社区医院管理服务系统
- 基于Springboot的实习管理系统
- TI-TCAN1146.pdf
- 基于Springboot的留守儿童爱心网站
- S32K3XXRM.pdf
- Ansible Automation Platform 快速安装指南 v3.8.1
- Ansible Tower 发行注记 v3.8.1-76页
- C语言笔记-考研版(进阶)
- Design_of_Analog_CMOS_Integrated_Circuit20200602-85440-9wt61m-with-cover-page-v2 (1).pdf
- Ansible Automation Platform 安装和参考指南 v3.8.1-59页
- 浅析5G技术在工业互联网领域的应用研究
- 查重17 岑彩谊-基于otn技术的本地承载网-二稿 .docx
- 自考计算机应用基础知识点.doc
- 数据库系统安全、技术操作规程.doc
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈
![](https://img-home.csdnimg.cn/images/20220527035711.png)
![](https://img-home.csdnimg.cn/images/20220527035711.png)
![](https://img-home.csdnimg.cn/images/20220527035111.png)
安全验证
文档复制为VIP权益,开通VIP直接复制
![](https://csdnimg.cn/release/wenkucmsfe/public/img/green-success.6a4acb44.png)