谷歌Site Reliability团队经验：打造大规模软件系统的实战指南

5星 · 超过95%的资源需积分: 21 152 浏览量更新于2024-07-20 10 收藏 9.65MB PDF 举报

《Site Reliability Engineering: How Google Runs Production Systems》是由Betsy Beyer、Chris Jones、Jennifer Petoff和Niall Richard Murphy编辑的一本集合了Google Site Reliability Team成员撰写的论文和文章。这本书的核心关注点在于挑战传统的软件开发观念，即强调设计和实施大型计算系统的重要性，而是倡导将工程师的关注点扩展到整个软件生命周期，包括构建、部署、监控和维护。在当今的IT行业中，随着软件系统的大部分时间都在实际使用中度过，Google通过其Site Reliability Engineering（SRE）方法，分享了他们如何成功地应对大规模软件系统的需求。SRE团队的经验表明，关注系统的整体生命周期对于提高系统的可扩展性、可靠性和效率至关重要。书中涵盖的原则、实践和架构模式，是经过长期实践检验并能在业界产生深远影响的成果。作者们详细介绍了如何将持续交付的理念与世界级的可靠性相结合，尤其是在处理庞大规模的业务时。他们强调了在设计、开发、测试、部署和维护各个阶段如何采取措施，确保系统的稳定运行，这使得SRE的理念不再仅仅是大型企业的专属，而是对所有面临规模化挑战的组织都具有重要的参考价值。例如，书中可能会探讨以下关键知识点： 1. **全生命周期管理**：强调软件开发不应仅限于设计和编码，而应延伸到系统上线后的监控、故障排查和持续优化。 2. **持续交付与自动化**：如何通过自动化工具和流程实现快速迭代和部署，降低风险并提升效率。 3. **容错和恢复策略**：分享Google如何设计系统以容忍故障，以及如何快速从故障中恢复，确保服务的不间断性。 4. **监控和预警系统**：介绍Google是如何利用先进的监控技术来识别潜在问题，并在问题发生之前采取预防措施。 5. **文化变革**：如何在组织内部推行SRE文化，培养全员关注系统健康和性能的文化氛围。 6. **度量和指标**：书中可能会涉及如何量化和优化系统性能的关键指标，以便进行决策和改进。 7. **案例研究**：书中可能包含Google在实际项目中的成功案例，展示SRE方法在实战中的应用和效果。阅读这本书不仅有助于IT专业人士学习Google的最佳实践，还可以为解决自身组织面临的规模挑战提供有价值的灵感和工具。无论是对于初创公司还是成熟企业，都值得一试，因为它能推动组织在面对日益复杂的技术环境时，提升整体软件系统的稳定性与效能。

that reappear in several chapters. We see choices from different perspectives, and

know that they correlate to resolve competing interests. The articles are not rigorous,

academic pieces; they are personal accounts, written with pride, in a variety of per‐

sonal styles, and from the perspective of individual skill sets. They are written bravely,

and with an intellectual honesty that is refreshing and uncommon in industry litera‐

ture. Some claim “never do this, always do that,” others are more philosophical and

tentative, reflecting the variety of personalities within an IT culture, and how that too

plays a role in the story. We, in turn, read them with the humility of observers who

were not part of the journey, and do not have all the information about the myriad

conflicting challenges. Our many questions are the real legacy of the volume: Why

didn’t they do X? What if they’d done Y? How will we look back on this in years to

come? It is by comparing our own ideas to the reasoning here that we can measure

our own thoughts and experiences.

The most impressive thing of all about this book is its very existence. Today, we hear a

brazen culture of “just show me the code.” A culture of “ask no questions” has grown

up around open source, where community rather than expertise is championed. Goo‐

gle is a company that dared to think about the problems from first principles, and to

employ top talent with a high proportion of PhDs. Tools were only components in

processes, working alongside chains of software, people, and data. Nothing here tells

us how to solve problems universally, but that is the point. Stories like these are far

more valuable than the code or designs they resulted in. Implementations are ephem‐

eral, but the documented reasoning is priceless. Rarely do we have access to this kind

of insight.

This, then, is the story of how one company did it. The fact that it is many overlap‐

ping stories shows us that scaling is far more than just a photographic enlargement of

a textbook computer architecture. It is about scaling a business process, rather than

just the machinery. This lesson alone is worth its weight in electronic paper.

We do not engage much in self-critical review in the IT world; as such, there is much

reinvention and repetition. For many years, there was only the USENIX LISA confer‐

ence community discussing IT infrastructure, plus a few conferences about operating

systems. It is very different today, yet this book still feels like a rare offering: a detailed

documentation of Google’s step through a watershed epoch. The tale is not for copy‐

ing—though perhaps for emulating—but it can inspire the next step for all of us.

There is a unique intellectual honesty in these pages, expressing both leadership and

humility. These are stories of hopes, fears, successes, and failures. I salute the courage

of authors and editors in allowing such candor, so that we, who are not party to the

hands-on experiences, can also benefit from the lessons learned inside the cocoon.

— Mark Burgess

author of In Search of Certainty

Oslo, March 2016

xiv | Foreword

1 The very fact that there is such large variance in these estimates tells you something about software engineer‐

ing as a discipline, but see, e.g., [Gla02] for more details.

Preface

Software engineering has this in common with having children: the labor before the

birth is painful and difficult, but the labor

aer the birth is where you actually spend

most of your effort. Yet software engineering as a discipline spends much more time

talking about the first period as opposed to the second, despite estimates that 40–90%

of the total costs of a system are incurred after birth.

The popular industry model

that conceives of deployed, operational software as being “stabilized” in production,

and therefore needing much less attention from software engineers, is wrong.

Through this lens, then, we see that if software engineering tends to focus on design‐

ing and building software systems, there must be another discipline that focuses on

the whole lifecycle of software objects, from inception, through deployment and oper‐

ation, refinement, and eventual peaceful decommissioning. This discipline uses—and

needs to use—a wide range of skills, but has separate concerns from other kinds of

engineers. Today, our answer is the discipline Google calls Site Reliability Engineer‐

ing.

So what exactly is Site Reliability Engineering (SRE)? We admit that it’s not a particu‐

larly clear name for what we do—pretty much every site reliability engineer at Google

gets asked what exactly that is, and what they actually do, on a regular basis.

Unpacking the term a little, first and foremost, SREs are engineers. We apply the prin‐

ciples of computer science and engineering to the design and development of com‐

puting systems: generally, large distributed ones. Sometimes, our task is writing the

software for those systems alongside our product development counterparts; some‐

times, our task is building all the additional pieces those systems need, like backups

or load balancing, ideally so they can be reused across systems; and sometimes, our

task is figuring out how to apply existing solutions to new problems.

2 For our purposes, reliability is “The probability that [a system] will perform a required function without fail‐

ure under stated conditions for a stated period of time,” following the definition in [Oco12].

3 The software systems we’re concerned with are largely websites and similar services; we do not discuss the

reliability concerns that face software intended for nuclear power plants, aircraft, medical equipment, or other

safety-critical systems. We do, however, compare our approaches with those used in other industries in Chap‐

ter 33.

In this, we are distinct from the industry term DevOps, because although we definitely regard infrastructure

as code, we have reliability as our main focus. Additionally, we are strongly oriented toward removing the

necessity for operations—see Chapter 7 for more details.

Next, we focus on system reliability. Ben Treynor Sloss, Google’s VP for 24/7 Opera‐

tions, originator of the term SRE, claims that reliability is the most fundamental fea‐

ture of any product: a system isn’t very useful if nobody can use it! Because reliability

is so critical, SREs are focused on finding ways to improve the design and operation

of systems to make them more scalable, more reliable, and more efficient. However,

we expend effort in this direction only up to a point: when systems are “reliable

enough,” we instead invest our efforts in adding features or building new products.

Finally, SREs are focused on operating services built atop our distributed computing

systems, whether those services are planet-scale storage, email for hundreds of mil‐

lions of users, or where Google began, web search. The “site” in our name originally

referred to SRE’s role in keeping the google.com website running, though we now run

many more services, many of which aren’t themselves websites—from internal infra‐

structure such as Bigtable to products for external developers such as the Google

Cloud Platform.

Although we have represented SRE as a broad discipline, it is no surprise that it arose

in the fast-moving world of web services, and perhaps in origin owes something to

the peculiarities of our infrastructure. It is equally no surprise that of all the post-

deployment characteristics of software that we could choose to devote special atten‐

tion to, reliability is the one we regard as primary.

The domain of web services, both

because the process of improving and changing server-side software is comparatively

contained, and because managing change itself is so tightly coupled with failures of all

kinds, is a natural platform from which our approach might emerge.

Despite arising at Google, and in the web community more generally, we think that

this discipline has lessons applicable to other communities and other organizations.

This book is an attempt to explain how we do things: both so that other organizations

might make use of what we’ve learned, and so that we can better define the role and

what the term means. To that end, we have organized the book so that general princi‐

ples and more specific practices are separated where possible, and where it’s appropri‐

ate to discuss a particular topic with Google-specific information, we trust that the

reader will indulge us in this and will not be afraid to draw useful conclusions about

their own environment.

xvi | Preface

5 In addition to this great story, she also has a substantial claim to popularizing the term “software engineering.”

We have also provided some orienting material—a description of Google’s production

environment and a mapping between some of our internal software and publicly

available software—which should help to contextualize what we are saying and make

it more directly usable.

Ultimately, of course, more reliability-oriented software and systems engineering is

inherently good. However, we acknowledge that smaller organizations may be won‐

dering how they can best use the experience represented here: much like security, the

earlier you care about reliability, the better. This implies that even though a small

organization has many pressing concerns and the software choices you make may dif‐

fer from those Google made, it’s still worth putting lightweight reliability support in

place early on, because it’s less costly to expand a structure later on than it is to intro‐

duce one that is not present. Part IV contains a number of best practices for training,

communication, and meetings that we’ve found to work well for us, many of which

should be immediately usable by your organization.

But for sizes between a startup and a multinational, there probably already is some‐

one in your organization who is doing SRE work, without it necessarily being called

that name, or recognized as such. Another way to get started on the path to improv‐

ing reliability for your organization is to formally recognize that work, or to find

these people and foster what they do—reward it. They are people who stand on the

cusp between one way of looking at the world and another one: like Newton, who is

sometimes called not the world’s first physicist, but the world’s last alchemist.

And taking the historical view, who, then, looking back, might be the first SRE?

We like to think that Margaret Hamilton, working on the Apollo program on loan

from MIT, had all of the significant traits of the first SRE.

In her own words, “part of

the culture was to learn from everyone and everything, including from that which

one would least expect.”

A case in point was when her young daughter Lauren came to work with her one day,

while some of the team were running mission scenarios on the hybrid simulation

computer. As young children do, Lauren went exploring, and she caused a “mission”

to crash by selecting the DSKY keys in an unexpected way, alerting the team as to

what would happen if the prelaunch program, P01, were inadvertently selected by a

real astronaut during a real mission, during real midcourse. (Launching P01 inadver‐

tently on a real mission would be a major problem, because it wipes out navigation

data, and the computer was not equipped to pilot the craft with no navigation data.)

Preface | xvii

With an SRE’s instincts, Margaret submitted a program change request to add special

error checking code in the onboard flight software in case an astronaut should, by

accident, happen to select P01 during flight. But this move was considered unneces‐

sary by the “higher-ups” at NASA: of course, that could never happen! So instead of

adding error checking code, Margaret updated the mission specifications documenta‐

tion to say the equivalent of “Do not select P01 during flight.” (Apparently the update

was amusing to many on the project, who had been told many times that astronauts

would not make any mistakes—after all, they were trained to be perfect.)

Well, Margaret’s suggested safeguard was only considered unnecessary until the very

next mission, on Apollo 8, just days after the specifications update. During midcourse

on the fourth day of flight with the astronauts Jim Lovell, William Anders, and Frank

Borman on board, Jim Lovell selected P01 by mistake—as it happens, on Christmas

Day—creating much havoc for all involved. This was a critical problem, because in

the absence of a workaround, no navigation data meant the astronauts were never

coming home. Thankfully, the documentation update had explicitly called this possi‐

bility out, and was invaluable in figuring out how to upload usable data and recover

the mission, with not much time to spare.

As Margaret says, “a thorough understanding of how to operate the systems was not

enough to prevent human errors,” and the change request to add error detection and

recovery software to the prelaunch program P01 was approved shortly afterwards.

Although the Apollo 8 incident occurred decades ago, there is much in the preceding

paragraphs directly relevant to engineers’ lives today, and much that will continue to

be directly relevant in the future. Accordingly, for the systems you look after, for the

groups you work in, or for the organizations you’re building, please bear the SRE Way

in mind: thoroughness and dedication, belief in the value of preparation and docu‐

mentation, and an awareness of what could go wrong, coupled with a strong desire to

prevent it. Welcome to our emerging profession!

xviii | Preface

剩余549页未读，继续阅读

xinconan2

粉丝: 269
资源: 399

谷歌Site Reliability团队经验：打造大规模软件系统的实战指南

Site.Reliability.Engineering..pdf

谷歌SRESite Reliability Engineering_How Google Runs Production Systems

Site Reliability Engineering - How Google Runs Production Systems 中英文合集

Site Reliability Engineering

目标检测数据集：瓶子图像缺陷检测数据【VOC标注格式】

毕业设计论文SpringBoot健身馆网站.docx

基于MATLAB车牌识别代码界面版(1).zip

基于循环卷积神经网络RCNN的语音情感识别分类源码+数据集(识别语音输出情感).zip

毕业设计论文SpringBoot教师工作量计算系统.docx

三维点云分割-基于Pytorch实现的Open3D-PointNet算法-优质项目实战.zip

最新资源