概率数值法：计算艺术中的机器学习探索

需积分: 10 182 浏览量更新于2024-06-27 收藏 3.65MB PDF 举报

《概率数值方法：计算作为机器学习》是一本由剑桥大学出版社出版的专业书籍，首次发布于2022年。该书由Philipp Hennig、Michael A Osborne和Hans P Kersting三位作者共同编撰，旨在探讨将概率论与数值计算相结合的全新方法，从而将机器学习的技术融入到传统数学和应用科学的计算领域。这本书将数学、概率论、线性代数等核心概念与实际问题解决结合起来，使计算程序具备了学习和适应的能力，体现了『机器学习』与『应用数学』之间的紧密联系。本书分为八章，内容丰富且深入，涵盖了多个关键主题： 1. **数学背景**：为理解概率数值方法提供必要的理论基础，包括概率论的基本概念和原理，以及它们在数值分析中的应用。 2. **整合**：讨论如何将概率推理技术整合到数值计算中，以构建更加灵活和高效的算法，强调了这种方法的创新性和实用性。 3. **线性代数**：线性代数是概率数值方法的重要工具，章节中可能涉及随机矩阵、奇异值分解和特征值分解等内容，这些对于解决实际问题如求解线性系统和优化问题至关重要。 4. **局部优化**：讲解如何使用概率模型进行局部搜索，优化计算过程中的局部最优解，这在求解复杂函数的最小化问题时尤为重要。 5. **全局优化**：探讨全局优化策略，如何通过概率方法寻找全局最优解，尤其是在存在多模态或高维度搜索空间的情况下。 6. **求解常微分方程**：展示了如何运用概率数值方法来求解连续时间系统的动力学问题，这对于模拟和控制论有重要意义。 7. **前沿**：介绍了最新的研究进展和技术趋势，展示了概率数值方法在不断演进的科研领域中的最新应用和挑战。 8. **习题答案**：为读者提供了丰富的实践机会，包括案例研究和习题，帮助读者巩固所学知识并提升技能。此书适合人工智能、计算机科学、统计学和应用数学的研究生阅读，它不仅是一本学术参考书，也是跨学科研究者探索计算方法新范式的实用指南。通过本书，读者将深入了解概率数值计算的核心理念，并将其应用于实际问题的求解过程中。

4 Introduction

image classiﬁer using active learning. This classiﬁer receives

labelled images, uses those to predict the labels of unlabelled

images, and then uses those predictions to decide which new

data should be acquired.

It is possible to treat a numerical algorithm as just such an

agent, as laid out diagrammatically in Figure 1. Traditionally,

a numerical method takes in data, in the form of evaluations

(e.g. of an integrand), and returns predictions, or estimates

(e.g. of the integral). A numerical method must also provide

a rule, perhaps an adaptive one, determining which compu-

tations are actually performed (e.g. which nodes to evaluate):

these are decisions. There is thus a feedback-loop: an agent

that decides itself which data to collect may be inefﬁcient by

collecting redundant data, or unreliable if it neglects to probe

crucially informative areas of the data-domain. Explicitly, pn

treats numerical algorithms just as machine learning often treats

its algorithms: as agents.

More precisely, pn is concerned with probabilistic agents. As

above, pn uses probability distributions to quantify uncertainty.

Quantiﬁed uncertainty is crucial to all agents, and numerical

agents are no exception. In particular, this understanding of

numerical solvers brings probabilistic decision theory

to numerics,

M. J. Kochenderfer. Decision Making Un-

der Uncertainty: Theory and Application.

2015.

yielding a range of advantages.

For one thing, a numerical agent must decide when to stop.

Unlike in the computation of analytic expressions, there is not

always an obvious end to a numerical procedure: the error in

the current estimate is unknown, so it can be difﬁcult to deter-

mine when to stop. Generic and encapsulated methods take a

cautious approach, usually aiming to satisfy high demands on

precision. Doing so requires many iterations, each further reduc-

ing uncertainty and improving precision, but each coming at a

cost. That is, this cautious approach consumes much computa-

tion. pn provides a new solution to this problem: if uncertainty

is well-quantiﬁed, we may be satisﬁed with (quantiﬁed) vague,

and thus cheap answers. pn hence provides a principled means

of stopping early, enabling fewer iterations, enabling a reduction

in computation.

Uncertainty also guides exploration. An intelligent agent,

making a decision, must weigh the uncertain consequences of

its actions, occasionally gambling by taking an uncertain ac-

tion, in order to explore and learn. Performing such exploration

is core to effective numerics. Predictive uncertainty unlocks a

fundamental means of quantifying the value of a numerical iter-

ation, and weighing it against the real cost of the computation it

The Deep Roots of Probabilistic Numerics 5

will consume. There are almost always choices for the character

of an iteration, such as where to evaluate an integrand or an

objective function to be optimised. Not all iterations are equal,

and it takes an intelligent agent to optimise the cost–beneﬁt

trade-off.

On a related note, a well-designed probabilistic numerical

agent gives a reliable estimate of its own uncertainty over their

result. This helps to reduce bias in subsequent computations. For

instance, in ODE inverse problems, we will see how simulating

the forward map with a probabilistic solver accounts for the

tendency of numerical ODE solvers to systemically over- or

underestimate solution curves. While this does not necessarily

give a more precise ODE estimate (in the inner loop), it helps

the inverse-problem solver to explore the parameter space more

efﬁciently (in the outer loop). As these examples highlight, pn

hence promises to make more effective use of computation.

The Deep Roots of Probabilistic Numerics

Probabilistic Numerics has a long history. Quite early in the his-

tory of numerical computation, people noted that its demands

closely matched what was provided by the professional pro-

cess of guesswork known as statistical inference. It seemed to

those people that probability, central to the process of inference,

might be a natural language in which to describe computation

as the gathering of information. In the ﬁrst chapter to his semi-

nal nineteenth-century Calcul des Probabilités,

Henri Poincaré

H. Poincaré. Calcul des Probabilités. 1896.

§I.7, pp. 30–31. Emphasis in the original.

mused about assigning probabilities to not-yet-computed num-

bers:

[Roughly:] The need for probability only

arises out of uncertainty: It has no place if

we are certain that we know all aspects of

a problem. But our lack of knowledge also

must not be complete, otherwise we would

have nothing to evaluate. There is thus a

spectrum of degrees of uncertainty.

While the probability for the sixth decimal

digit of a number in a table of logarithms to

equal 6 is 1/10 a priori, in reality, all aspects

of the corresponding problem are well deter-

mined, and, if we wanted to make the effort,

we could ﬁnd out its exact value. The same

holds for interpolation, for the integration

methods of Cotes or Gauss, etc. (Emphasis

in the op. cit.)

Une question de probabilités ne se pose que par suite de notre

ignorance: il n’y aurait place que pour la certitude si nous

connaissions toutes les données du problème. D’autre part, notre

ignorance ne doit pas être complète, sans quoi nous ne pourrions

rien évaluer. Une classiﬁcation s’opérerait donc suivant le plus

ou moins de profondeur de notre ignorance.

Ainsi la probabilité pour que la sixième décimale d’un nombre

dans une table de logarithmes soit égale à 6 est a priori de 1/10;

en réalité, toutes les données du problème sont bien déterminées,

et, si nous voulions nous en donner la peine, nous connaîtrions

exactement cette probabilité. De même, dans les interpolations,

dans le calcul des intégrales déﬁnies par la méthode de Cotes ou

celle de Gauss, etc.

6 Introduction

Although Poincaré found it natural to assign probabilities to

the value of determined but unknown numbers, it seems the

idea of uncertainty about a fully, formally determined quantity

did not sit well with a majority of mathematicians. Rather than

assigning degrees of certainty about properties of a problem, it

seemed more acceptable to formally state assumptions required

to be strictly true at the onset of a theorem. When considering a

particular numerical estimation rule, one can then analyse the

convergence of the estimate in an asymptotic fashion, thereby

formally proving that (and in which sense) the rule is admissible.

This approach leaves the estimation of the rule’s error after a

ﬁnite number of steps as a separate and often subordinate part

of the routine.

By contrast, probabilistic inference makes the formulation of

probability distributions, characterising possible error, primary.

These distributions will include an explicit prior on the latent

quantity to be inferred, and an equally explicit likelihood func-

tion capturing the relationship of computable numbers to that

latent quantity. This approach may be more or less restrictive

than the asymptotic analysis described above. It might well be

more cumbersome to state, subject to philosophical intricacies,

and requires care to not introduce new intractable tasks along

the way. Nonetheless, phrasing computation as probabilistic

inference offers substantial advantages. For one thing, the ap-

proach yields a posterior probability distribution for quantities

of interest that self-consistently combines estimate and uncer-

tainty, and can approximately track their effect through several

steps of the computation.

In the twentieth century, the idea of computation as inference

lingered in the margins. In the 1950s and 1960s – the golden

age of probability theory following the formal works of Kol-

mogorov – perhaps Sul’din should be mentioned as the ﬁrst to

return to address it in earnest.

He focused on the approxima-

Sul’din (1959); Sul’din (1960).

tion of functions, a task underlying many numerical methods.

In statistics, where this process is known as regression, it took

on a life of its own, leading to a deep probabilistic framework

studied extensively to this day, whose early probabilistic inter-

pretations where driven by people like Sard (1963), or Kimeldorf

and Wahba (1970). Parallel to Sul’din’s work in Russia, the task

of integration found the attention of Ajne and Dalenius (1960)

in Scandinavia. The English-speaking audience perhaps ﬁrst

heard of these connections from Larkin (1972), who went on to

write several pieces on the connection between inference and

computation. Anyone who missed his works might have had

Twenty-First-Century Probabilistic Numerics 7

to wait over a decade. By then, the plot had thickened and au-

thors in many communities became interested in Bayesian ideas

for numerical analysis. Among them Kadane and Wasilkowski

(1985), Diaconis (1988), and O’Hagan (1992). Skilling (1991) even

ventured boldly toward solving differential equations, display-

ing the physicist’s willingness to cast aside technicalities in the

name of progress. Exciting as these insights must have been

for their authors, they seem to have missed fertile ground. The

development also continued within mathematics, for example in

the advancement of information-based complexity

and average-

Traub, Wasilkowski, and Wo´zniakowski

(1983); Packel and Traub (1987); Novak

(2006).

case analysis.

But the wider academic community, in particular

Ritter (2000)

users in computer science, seem to have missed much of it. But

the advancements in computer science did pave the way for

the second of the central insights of pn: that numerics requires

thinking about agents.

Twenty-First-Century Probabilistic Numerics

The twenty-ﬁrst century brought the coming-of-age of machine

learning. This new ﬁeld raised new computational problems,

foremost among them the presence of big data sets in the com-

putational pipeline, and thus the necessity to sub-sample data,

creating a trade-off between computational cost and precision.

Numerics is inescapably important to machine learning, with

popular tracks at its major conferences, and large tracts of ma-

chine learning masters’ degrees, devoted to optimisation alone.

But machine learning also caused a shift in perspective on mod-

elling itself.

Modelling (or inference) used to be thought of as a passive

mathematical map, from data to estimate. But machine learning

often views a model as an agent in autonomous interaction with

its environment, most explicitly in reinforcement learning. This

view of algorithms as agents is, as above, central to pn.

Machine learning has been infused with the viewpoints of

physicists and other scientists, who are accustomed to limits on

precision and the necessity of assumptions. The Bayesian view-

point on inference soon played a prominent (albeit certainly not

the only leading) role in laying its theoretical foundations. Text-

books like those of Jaynes and Bretthorst (2003), MacKay (2003),

Bishop (2006), and Rasmussen and Williams (2006) taught a gen-

eration of new students – the authors amongst them – to think

in terms of generative models, priors, and posteriors. Machine

learning’s heavy emphasis on numerics couldn’t help but lead

some of those students to apply their hammer, of probabilis-

8 Introduction

tic inference, to the nails of the numerical problems that they

encountered. These students were bound to revive questions

from the history of pn. For instance, if computation is inference,

should it then not be possible to build numerical algorithms

that:

rather than taking in a logical description of their task and

returning a (ﬂoating-) point estimate of its solution, instead

take probability distributions as inputs and outputs?;

use an explicit likelihood to capture a richer, generative

description of the relation between the computed numbers

and the latent, intractable quantity in question?; and

3. treat the CPU or GPU as an interactive source of data?

In 2012, the authors of this text co-organised, with J. P. Cun-

ningham, a workshop at the Neural Information Processing Sys-

tems conference on the shores of Lake Tahoe to discuss these

questions. We were motivated by our own work on Bayesian

Optimisation and Bayesian Quadrature. At the time, the wider

issue seemed new and unexplored to us. By a stroke of luck, we

managed to convince Persi Diaconis to make the trip up from

Stanford and speak. His talk pointed us to a host of prior work.

In search of an inclusive, short title for the workshop, we had

chosen to call it Probabilistic Numerics (pn). In the years since,

this label has been increasingly used by a growing number of

researchers who, like us, feel that the time has come to more

clearly and extensively connect the notions of inference and

computation. The 2012 workshop also marked the beginning of

a fruitful collaboration between the machine learning commu-

nity and statisticians like M. Girolami

and C. J. Oates, as well as

Hennig, Osborne, and Girolami (2015)

applied mathematicians like T. J. Sullivan, I. Ipsen, H. Owhadi

and others. They ensured that existing knowledge in either com-

munity was not forgotten,

and their research groups have since

Owhadi and Scovel (2016); Oates and

Sullivan (2019).

laid an increasingly broad and deep foundation to the notion

of computation as probabilistic and also more narrowly, care-

fully deﬁned Bayesian inference.

They have also undertaken a

Cockayne et al. (2019b)

commendable effort to build a community for pn within the

mathematical ﬁelds.

Although numerous interesting insights have already been

reached, just as many questions are still scarcely explored. Many

of them emerge from new application areas, and new associated

computational problems, in the age of Big Data, GPUs, and

distributed, compartmental, computation.

剩余411页未读，继续阅读

ShowMeAI

粉丝: 6264
资源: 42

概率数值法：计算艺术中的机器学习探索

Probabilistic Machine Learning

Bayesian Reasoning and Machine Learning

Pattern Recognition and Machine Learning-01-Preface

《machine learning: a probabilistic perspective》 下载

kevin p. murphy. machine learning a probabilistic perspective. the mit press

robabilistic U-Net:

probabilistic reversal learning task

贝叶斯网络在哪本书里有讲解

欧式距离学习参考文献

最新资源

《machine learning: a probabilistic perspective》下载