xvi Preface to the Second Edition
deserve our d ee pest gratitude for this edition as well, which would not exist were it not
for their contributions to edi t i on number one. To that long li st we must add many others
who contributed specifically to the second edition. Our students over the many years that
we have taught this material contributed in countless ways: exposing errors, o↵ering fixes,
and—not the least—being confused in places where we could have explained things better.
We especially thank Martha Steenstrup for reading and providing detailed comments
throughout. The chapters on psychology and neuroscience could not have been writ te n
without t h e help of many experts in those fields. We thank John Moore for his patient
tutoring over many many years on animal learning experiments, theor y, and neuroscience,
and for his careful reading of multiple drafts of Chapters 14 and 15. We also thank Matt
Botvinick, Nathaniel Daw, Peter Dayan, and Yael Niv for their penetrating comments on
drafts of these chapters, their essential guidance through the massive literature, and their
interception of many of our errors in early drafts. Of course, the remaining errors in these
chapters—and there must st il l be some—are totall y our own. We thank Phil Thomas for
helping u s make these chapters accessible to non- ps y chologists and non-neuroscientists,
and we thank Peter Sterling for helping us improve the exposition. We are grateful to Jim
Houk for introduci ng us to the subject of information processing in the basal ganglia and
for alerting us to other relevant aspects of neuroscien ce . Jos´e Mart´ınez, Terry Sejnowski,
David Silver, Gerry Tesauro, Georgios Theocharous, and Phil Thomas generously helped
us understand details of their reinforcement learning applications for inclusi on in the
case-studies chapt er , and they provided helpful comments on drafts of these sections.
Special thanks are owed to David Silver for helping us better understand Monte Carlo
Tree Search and the DeepMind Go-playing pr ogram s . We thank George Konidaris for his
help wit h the section on the Fourier b asi s. Emilio Cartoni, Thomas Cederborg, S t ef an
Dernbach, Clemens Rosenbaum, Patrick Taylor, Thomas Colin, and Pierre-Luc Bacon
helped us in a number important ways for which we are most grateful.
Sutton would also like to thank the members of the Reinforcement Learning and
Artificial Intelligence laboratory at the University of Alberta for contributions to the
second edition. He owes a part i cu l ar debt to Rupam M ah mood for essential contributions
to the treatment of o↵-policy Monte Carlo methods in Chapter 5, to Hamid Maei for
helping develop th e perspective on o↵-policy learni ng presented in Chapter 11, to Eric
Graves for conducting the experiments in Chapter 13, to Shangtong Zhang for replicating
and thus verifying almost all the experimental results, to Kris De Asis for improving
the new technical content of Chapters 7 and 12, and to Harm van Seijen for insights
that led to the separation of n-step methods from eligibili ty tr ace s an d (al on g wit h Hado
van Hasselt) for the ideas involving exact equivalence of forward and backward views of
eligibility traces presented in Chapter 12. Sut t on also gratefully acknowledges the support
and freedom he was granted by the Government of Alberta and t h e National Science and
Engineering Research Council of Canada t hr ou ghou t the period during which the second
edition was con ce i ved and written. In particular, he would like to thank Randy Goebe l
for creating a supportive and far-sighted environment for research in Alberta. He would
also like to thank DeepMind their support i n the last six months of writing the book.
Finally, we owe thanks to the m any careful readers of drafts of the second edition t h at
we posted on the internet. They found many errors that we had missed and alert ed us to
potential points of confusion.
剩余547页未读，继续阅读
