通过实验建立对大规模分布式系统的信心：混沌工程

需积分: 10 6 浏览量更新于2024-07-19 收藏 1.32MB DOCX 举报

"Chaos Engineering 是一种新兴的软件工程方法，旨在通过实验来建立对大规模分布式系统行为的信心。随着快速部署和高度灵活的系统成为常态，混沌工程的重要性日益凸显。它帮助我们在系统出现异常之前识别并修复潜在的问题，确保在复杂的生产环境中提供稳定的服务。在混沌工程中，我们关注的是系统整体的韧性，而不仅仅是单个服务的性能。即使每个服务都能正常工作，它们之间的相互作用也可能导致意想不到的结果，尤其是在面对罕见但破坏性强的事件时。例如，不恰当的超时设置可能导致过多的重试操作，下游服务在接收大量流量时可能会崩溃，或者单点故障可能触发连锁反应。通过主动引入和管理这些混乱因素，我们可以提前发现并解决这些弱点，防止它们在生产环境中造成真实的影响。 Chaos Engineering 的核心是进行受控实验，模拟可能发生的故障场景，以此来测试系统的应对能力。这种方法允许我们在安全的环境中观察系统的行为，找出脆弱点，并采取措施增强其健壮性。通过这种方式，我们可以在保持快速迭代和高度复杂性的同时，构建出更可靠的分布式系统。这本书《Chaos Engineering》由 Casey Rosenthal、Lorin Hochstein、Aaron Blohowiak、Nora Jones 和 Ali Basiri 共同撰写，深入探讨了混沌工程的原理和实践，提供了如何在实际项目中应用混沌工程的方法。它强调了实验设计、故障注入、监控和恢复策略等方面，为软件工程师提供了一套系统的解决方案，以应对大规模分布式系统的挑战。混沌工程是一种创新的工程方法，它推动我们超越传统的质量保证和测试，向更积极主动的方向发展，以确保我们的系统能够在现实世界的各种复杂情况中保持稳定和高效。通过学习和实施混沌工程，开发者和运维人员可以更好地理解和提升他们的系统在面对不确定性时的弹性，从而提高用户体验和业务连续性。"

10!"

Engineering is finding a foothold as a discipline that improves com‐

plex systems.

=

To determine whether your organization is ready to start adopting

Chaos Engineering, you need to answer one question: Is your sys‐

tem resilient to real-world events such as service failures and net‐

work latency spikes?

If you know that the answer is “no,” then you have some work to do

before applying the principles in this book. Chaos Engineering is

great for exposing unknown weaknesses in your production system,

but if you are certain that a Chaos Engineering experiment will lead

to a significant problem with the system, there’s no sense in running

that experiment. Fix that weakness first. Then come back to Chaos

Engineering and it will either uncover other weaknesses that you

didn’t know about, or it will give you more confidence that your

sys‐ tem is in fact resilient.

Another essential element of Chaos Engineering is a monitoring

system that you can use to determine the current state of your sys‐

tem. Without visibility into your system’s behavior, you won’t be

able to draw conclusions from your experiments. Since every

system is

1 Julia Cation, “Flight control breakthrough could lead to safer air travel”, Engineering at

Illinois, 3/19/2015.

>#

At the University of Illinois at Urbana-Champaign, Naira Hova‐

kimyan and her research team brought Chaos Engineering to

jets.

The test team comprised two B-52 pilots, an F-16 pilot, two

flight test engineers, and two safety pilots. During flight, the jet

was injec‐ ted with seven different failure configurations. These

configurations included both shifts in center of gravity and

changes in aerody‐ namic parameters! It was challenging for the

team to reproduce lift‐ ing body aerodynamics and other

configurations that are highly likely to cause failure. After

developing their failure scenarios and putting them into action,

the team was able to confidently deem the system safe for low-

altitude flight.

=0

unique, we leave it as an exercise for the reader to determine how

best to do root cause analysis when Chaos Engineering surfaces a

systemic weakness.

&?

In late 2010, Netflix introduced Chaos Monkey to the world. The

streaming service started moving to the cloud a couple of years ear‐

lier. Vertically scaling in the datacenter had led to many single

points of failure, some of which caused massive interruptions in

DVD delivery. The cloud promised an opportunity to scale hori‐

zontally and move much of the undifferentiated heavy lifting of

running infrastructure to a reliable third party.

The datacenter was no stranger to failures, but the horizontally

scaled architecture in the cloud multiplied the number of instances

that run a given service. With thousands of instances running, it

was virtually guaranteed that one or more of these virtual machines

would fail and blink out of existence on a regular basis. A new

approach was needed to build services in a way that preserved the

benefits of horizontal scaling while staying resilient to instances

occasionally disappearing.

At Netflix, a mechanism doesn’t really exist to mandate that engi‐

neers build anything in any prescribed way. Instead, effective lead‐

ers create strong alignment among engineers and let them figure

out the best way to tackle problems in their own domains. In this

case of instances occasionally disappearing, we needed to create

strong alignment to build services that are resilient to sudden

instance termination and work coherently end-to-end.

Chaos Monkey pseudo-randomly selects a running instance in pro‐

duction and turns it off. It does this during business hours, and at a

much more frequent rate than we typically see instances disappear.

By taking a rare and potentially catastrophic event and making it

frequent, we give engineers a strong incentive to build their service

in such a way that this type of event doesn’t matter. Engineers are

forced to handle this type of failure early and often. Through auto‐

mation, redundancy, fallbacks, and other best practices of resilient

design, engineers quickly make the failure scenario irrelevant to the

operation of their service.

Over the years, Chaos Monkey has become more sophisticated in

the way it specifies termination groups and integrates with Spin‐

剩余63页未读，继续阅读

一川烟雨

粉丝: 1
资源: 30

通过实验建立对大规模分布式系统的信心：混沌工程

chaosblade-help-doc-master.zip

chaosblade-1.6.1-linux-amd64.tar.gz

chaosblade-box安装

chaosblade-box

chaosblade是啥

mysql chaosblade

chaosblade模拟mysql故障

chaosblade使用教程

chaosblade-tool-1.5.0 安装

chaosblade 在linux上下载安装

最新资源