高斯信道中互信息与最小均方误差的关系

需积分: 4 189 浏览量更新于2025-01-03 收藏 576KB PDF 举报

" Mutual information and minimum mean squared error (MMSE)之间的关系在高斯信道中的应用" 这篇由Dongning Guo, Shlomo Shamai (Shitz), 和 Sergio Verdú合著的论文，发表在2005年4月的IEEE Transactions on Information Theory期刊上，探讨了在加性高斯白噪声信道中任意分布的有限功率输入信号的特性。该研究揭示了一个新的公式，这个公式连接了输入-输出互信息（mutual information）与基于输出的输入最佳估计的最小均方误差（Minimum Mean-Square Error，简称MMSE）。文章的核心发现是，无论输入信号的统计特性如何，互信息关于信噪比（Signal-to-Noise Ratio, SNR）的导数等于MMSE的一半。这个等式不仅适用于标量信号，也适用于向量信号，同时涵盖了离散时间和连续时间的非因果MMSE估计。在连续时间非线性估计的背景下，这一基础信息论结果带来了意外的后果：对于任何具有有限功率的输入信号，在给定SNR下，因果滤波MMSE等于通过一个“扩展”信道实现的非因果平滑MMSE的平均值。这里的“扩展”信道指的是一个可以提前获取未来噪声样本的理想化信道。互信息是衡量两个随机变量之间依赖程度的一种度量，而在通信系统中，它通常用来量化信息传输的效率。MMSE则是估计理论中的一个重要概念，用于评估估计器的性能，即在给定观测数据的情况下，输入信号的最佳估计与其真实值之间的平均平方差。论文的这一发现为高斯信道下的信息传输和信号处理提供了深刻的见解，对于优化通信系统的性能，特别是在低信噪比条件下的信号恢复和估计问题，具有重要的理论和实践意义。这种关系可能会影响无线通信、数据压缩、图像处理以及许多其他领域的算法设计和系统优化。通过理解并利用这种内在联系，工程师和研究人员能够更有效地设计和分析通信系统，以提高信息传输的准确性和效率。

GUO et al.: MUTUAL INFORMATION AND MINIMUM MEAN-SQUARE ERROR IN GAUSSIAN CHANNELS 1265

Fig. 3. A Gaussian pipe where noise is added gradually.

Taking the expectation over on both sides of (42) yields

(43)

which establishes (33) by (35) together with the fact that

(44)

Hence the proof of Theorem 1.

Underlying the incremental-channel proof of Theorem 1 is

the chain rule for information

(45)

When

is a Markov chain, (45) becomes

(46)

where we let

. This applies to a train of outputs

tapped from a Gaussian pipe where noise is added gradually

until the SNR vanishes, as depicted in Fig. 3. The sum in (46)

converges to an integral as

becomes a ﬁner and ﬁner se-

quence of Gaussian channel outputs. To see this, note from (43)

that each conditional mutual information in (46) corresponds to

a low-SNR channel and is essentially proportional to the MMSE

times the SNR increment. This viewpoint leads us to an equiv-

alent form of Theorem 1

(47)

Therefore, as is illustrated by the curves in Fig. 1, the mutual

information is equal to an accumulation of the MMSE as a func-

tion of the SNR due to the fact that an inﬁnitesimal increase in

the SNR adds to the total mutual information an increase pro-

portional to the MMSE.

The inﬁnite divisibility of Gaussian distributions, namely, the

fact that a Gaussian random variable can always be decomposed

as the sum of independent Gaussian random variables of smaller

variances, is crucial in establishing the incremental channel (or

the Markov chain). This property enables us to study the mutual

information increase due to an inﬁnitesimal increase in the SNR,

and thus obtain the differential (15) and (22) in Theorems 1

and 2.

The following corollaries are immediate from Theorem 1 to-

gether with the fact that

is monotone decreasing.

Corollary 1: The mutual information

is a concave

function in

Corollary 2: The mutual information can be bounded as

(48)

(49)

(50)

D. Applications and Discussions

1) Some Applications of Theorems 1 and 2: The newly

discovered relationship between the mutual information and

MMSE ﬁnds one of its ﬁrst uses in relating code-division

multiple-access (CDMA) channel spectral efﬁciencies (mutual

information per dimension) under joint and separate decoding

in the large-system limit [30], [31]. Under an arbitrary ﬁ-

nite-power input distribution, Theorem 1 is invoked in [30]

to show that the spectral efﬁciency under joint decoding is

equal to the integral of the spectral efﬁciency under separate

decoding as a function of the system load. The practical lesson

therein is the optimality in the large-system limit of successive

single-user decoding with cancellation of interference from

already decoded users, and an individually optimal detection

front end against yet undecoded users. This is a generalization

to arbitrary input signaling of previous results that successive

cancellation with a linear MMSE front end achieves the CDMA

channel capacity under Gaussian inputs [32]–[35].

Relationships between information theory and estimation

theory have been identiﬁed occasionally, yielding results in one

area taking advantage of known results from the other. This is

exempliﬁed by the classical capacity–rate distortion relations,

that have been used to develop lower bounds on estimation

errors [36]. The fact that the mutual information and the MMSE

determine each other by a simple formula also provides a new

means to calculate or bound one quantity using the other. An

upper (resp., lower) bound for the mutual information is imme-

diate by bounding the MMSE for all SNRs using a suboptimal

(resp., genie-aided) estimator. Lower bounds on the MMSE,

e.g., [37], lead to new lower bounds on the mutual information.

An important example of such relationships is the case of

Gaussian inputs. Under power constraints, Gaussian inputs are

most favorable for Gaussian channels in information-theoretic

sense (they maximize the mutual information); on the other

hand, they are least favorable in estimation-theoretic sense

(they maximize the MMSE). These well-known results are seen

to be immediately equivalent through Theorem 1 (or Theorem

2 for the vector case). This also points to a simple proof of the

result that Gaussian inputs achieve capacity by observing that

the linear estimation upper bound for MMSE is achieved for

Gaussian inputs.

Another application of the new results is in the analysis of

sparse-graph codes, where [38] has recently shown that the

so-called generalized extrinsic information transfer (GEXIT)

function plays a fundamental role. This function is deﬁned for

arbitrary codes and channels as minus the derivative of the

input–output mutual information per symbol with respect to a

channel quality parameter when the input is equiprobable on

the codebook. According to Theorem 2, in the special case of

the Gaussian channel the GEXIT function is equal to minus

one half of the average MMSE of individual input symbols

given the channel outputs. Moreover, [38] shows that (1) leads

to a simple interpretation of the “area property” for Gaussian

channels (cf. [39]). Inspired by Theorem 1, [40] also advocated

The observations here are also relevant to continuous-time Gaussian chan-

nels in view of results in Section III.

Authorized licensed use limited to: IEEE Xplore. Downloaded on February 26, 2009 at 11:40 from IEEE Xplore. Restrictions apply.

剩余21页未读，继续阅读

FSTseng

粉丝: 0
资源: 7

高斯信道中互信息与最小均方误差的关系

Mutual Information 算法结合 Powell 方法优化探讨

高效准确的Semi-Global Matching: 基于Mutual Information的立体匹配方法

"信息论基础：第二版英文原版与书签目录

Nonlinear Inverse Reinforcement Learning with Mutual Information and Gaussian Process

Use of Mutual Information and Transfer Entropy to Assess Interaction between Parasympathetic and Sympathetic Activities of Nervous System from HRV

Automatic Image Registration using (Normalized) Mutual Information for users of IP toolbox:Rigid (translation and rotation) automatic image Registration using normalized mutual information-matlab开发

Stereo Processing by Semiglobal Matching and Mutual Information

Mutual Information functions for C and MATLAB.zip

local Mutual information

ami.zip_mutual_mutual information_zip

最新资源