Mesos应用开发：构建弹性、可扩展和分布式系统

5星 · 超过95%的资源需积分: 9 29 浏览量更新于2024-07-21 1 收藏 3.87MB PDF 举报

"Building Applications on Mesos 2015.12.pdf" 本书"Building Applications on Mesos"由David Greenberg撰写，旨在介绍如何利用Mesos构建可扩展、弹性和分布式系统。Mesos是一个开源的分布式系统内核，它允许在数据中心有效地管理和调度计算资源，为现代云原生应用提供了强大的基础。 Mesos的核心概念是资源调度和隔离，它将数据中心的硬件抽象化，提供了一个统一的资源池，使得应用程序可以跨多个机器无缝运行。Mesos的架构设计使得它具有高可用性、容错性和可扩展性，能够处理大规模集群中的复杂工作负载。书中详细介绍了如何构建 Mesos 应用程序，涵盖了以下几个关键知识点： 1. **Mesos 架构**：深入解析 Mesos 的三层架构——Mesos Master、Agent 和 Framework。Master 负责全局资源管理和调度，Agent 运行在每个节点上，负责本地资源管理和任务执行，而 Framework 是运行在 Mesos 上的应用程序模型，如 Marathon 和 Aurora。 2. **资源管理**：理解 Mesos 如何分配 CPU、内存、磁盘和网络资源，以及如何通过资源Offers和Resource Requests进行资源的请求与匹配。 3. **框架开发**：学习如何编写自定义 Mesos 框架，包括使用 Mesos C++ API 或者 Scala API，以及如何实现调度器和执行器。 4. **容错机制**：探讨 Mesos 的故障恢复策略，如自动重试、故障转移和数据持久化，确保服务的高可用性。 5. **容器化**：Mesos 支持多种容器技术，如 Docker 和 Mesos Containerizer，讲解如何利用容器技术进行应用打包和隔离。 6. **Mesos 并发模型**：了解 Mesos 如何处理并发任务，如何处理并行任务执行和资源竞争。 7. **Marathon 与 Aurora**：这两个流行框架的使用，Marathon 用于无状态服务的部署和管理，Aurora 用于有状态任务的调度和执行，以及它们与 Mesos 的集成。 8. **监控与日志管理**：如何集成监控工具（如Prometheus、Grafana）和日志收集系统（如Logstash、Fluentd），以实现 Mesos 集群的可观测性。 9. **安全与身份验证**：Mesos 的安全性包括角色权限、认证和授权，以及如何设置安全策略以保护资源和数据。 10. **实战案例**：书中可能包含实际项目部署示例，如部署微服务架构，处理大数据工作负载，以及如何优化 Mesos 集群性能。 "Building Applications on Mesos" 提供了全面的指导，帮助开发者和系统管理员充分利用 Mesos 构建和管理现代云应用。这本书不仅适合 Mesos 新手，也适合希望提升 Mesos 使用技巧的有经验的读者。

tems (like Supervisor and SystemD). Groups of tasks can be combined to describe a

“role” or “recipe,” which represents a higher-level concept, like setting up a database

or web server. Finally, these groups of tasks are applied to an “inventory,” or set of

hosts, which then configures them according to the specification.

These tools are wildly more useful than the Bash, Perl, and SSH mashups of old; how‐

ever, they suffer from fundamental limitations. On the one hand, they allow for con‐

figuration to be written in a structured, understandable form. They enable the use of

best practices from software development—source control, encapsulation, and code

reuse through libraries and plug-ins—to extend to system administration. On the

other hand, they are fundamentally designed to make a fleet of machines conform to

a static configuration. This is intentional: once you’ve run your Ansible or Puppet

configuration, you want the cluster to be in the same, known consistent state. If, how‐

ever, you want your cluster to dynamically reallocate resources or shift configurations

depending on various external factors, such as the current load, these tools can’t help

you.

In this sense, it can be useful to think of Mesos as a deployment system. Distributed

applications can be made into Mesos frameworks for use on a Mesos cluster. Rather

than forcing you to use a static configuration description, each framework is essen‐

tially a role or recipe: the framework contains all the logic needed to install, start,

monitor, and use a given application. Many frameworks are themselves platforms;

when many applications follow the same general deployment pattern (like web

servers), then a single framework can manage them all (like Marathon, a Mesos

framework for stateless services). The power of Mesos is that since the framework is a

live, running program, it can make decisions on the fly, reacting to changing work‐

loads and cluster conditions. For instance, a framework could observe how many

resources seem to be available, and change its execution strategy to better perform

given the current cluster conditions. Because the deployment system runs constantly,

it can notice and adapt to failures in real time, automatically starting up new servers

as previous ones fail. Mesos frameworks are strictly more powerful than static

descriptions used by more traditional deployment systems.

Thus, you can think of Mesos as supplanting much of the functionality that Ansible

or Chef previously provided. You’ll still need a traditional configuration manager to

bootstrap the Mesos cluster; however, Mesos provides a more powerful platform for

hosting applications: frameworks can automatically and dynamically adapt to failing

machines and changing workloads, giving operators greater peace of mind and time

to work on new initiatives.

Mesos as an Execution Platform

Another way to think of Mesos is as a platform for hosting applications. Today, you

may use Heroku to run your web services, or you may manage a Hadoop cluster to

4 | Chapter 1: Introduction to Mesos

execute MapReduce jobs. In these examples, Heroku and the Hadoop cluster are the

platforms for web services and MapReduce jobs. A Mesos cluster can also be a plat‐

form for higher-level applications. But if your system already works, why add Mesos

to the mix? Mesos offers flexibility and mitigates the risk of being outpaced by tech‐

nology development; with Mesos, launching a Spark cluster or switching to a newer

technology is as simple as launching the framework and watching it bootstrap itself.

Consider the Heroku/PaaS scenario: Marathon can reliably launch your application,

automatically starting up new instances when machines crash or go down for mainte‐

nance. Mesos-DNS and HAProxy can manage your load balancing for you (see Chap‐

ter 3). But once you’ve got Mesos running, why bother continuing to maintain your

Hadoop cluster? After all, you could launch Hadoop on Mesos via Myriad, and you

could even stop dealing with HDFS administration, since there’s a framework for that

(Mesos HDFS).

Thus, you can think of Mesos as an execution platform: rather than paying for third-

party PaaS solutions and creating bespoke clusters for each big data analysis technol‐

ogy, you can have a single Mesos cluster. With your Mesos cluster as the foundation,

you can easily launch whatever frameworks you require to provide whatever func‐

tionality is needed, now or as new requirements and technologies arise.

How This Book Is Structured

At this point, let’s take a look at how the rest of the book will be structured. First,

we’re going to learn about Mesos itself. We’ll cover the architecture of Mesos and its

frameworks, and we’ll learn how resources are allocated across the cluster. We’ll also

dive into configuration settings you’ll want to know, such as how to provide service-

level agreements (SLAs) to certain frameworks, and useful command-line settings to

tune the overall performance and workload of your cluster.

Then, we’ll learn about two existing open source Mesos frameworks—Marathon and

Chronos—that provide application hosting and job scheduling functionality. We’ll

explore how to configure them, how to interact with them, and how to build deeper

integrations. After learning about these frameworks, you will be able to host several

common application architectures, such as web services and data pipelines, on Mesos,

with high reliability and scalability.

In the later parts of this book, we’ll go into great detail about how to build a custom

framework for Mesos. We’ll learn about the APIs that it exposes through the process

of implementing a sample job scheduling framework in Java. In developing this

framework, we’ll delve into how to build on the core Mesos APIs, what some com‐

mon pitfalls are and how to avoid them, and how to decide whether to build a new

framework or adapt an existing system.

How This Book Is Structured | 5

When a scheduler wants to do some work, it launches an executor. The executor is

simply the scheduler’s worker: the scheduler then decides to send one or more tasks

to the executor, which will work on those tasks independently, sending status updates

to the scheduler until the tasks are complete. We’ll learn more about the distinction

between tasks and executors in “Tasks and Executors” on page 21.

Now that we’ve got an idea of what a framework is, let’s take a step down the stack to

learn about the Mesos cluster itself.

Masters and Slaves

A Mesos cluster is comprised of two components: the Mesos masters and the Mesos

slaves. The masters are the software that coordinates the cluster; the slaves are what

execute code in containers. You might be thinking, “That sounds awfully similar to

how frameworks work.”

In that case, you’d be 100% correct. The Mesos cluster mirrors the organization of the

frameworks that run on top of it, because the master/slave pattern is simple, flexible,

and effective. The cluster software needs to be separate, however, to ensure that

frameworks remain isolated and secure from one another. First, we’ll look at a few

key ideas about the Mesos masters; then, we’ll take a look at the slaves.

The Masters

The Mesos masters are the brain of the cluster. They have several responsibilities:

•

They are the central source of truth for running tasks.

• They fairly share the cluster between all the connected frameworks.

• They host the primary UI for the cluster.

•

They ensure high availability and efficient allocation of resources. Considered

together, they are a single point of failure, but Mesos automatically handles and

recovers from individual master failures.

Let’s unpack these responsibilities, and see what they mean for the design of the mas‐

ters and how they impact our clusters:

Masters are the central source of truth for all running tasks

In order for masters to be able to serve the UI and data about tasks with minimal

latency, all task metadata is kept in memory. This means that the nodes that run

the masters should have plenty of memory—on a cluster with thousands of con‐

current tasks, the masters could, for instance, consume 20 GB of RAM. Note that

this excludes completed tasks, which relieves the memory pressure on the mas‐

ters for some workloads. The masters make an effort to store the metadata

related to recently completed tasks, but in the interest of saving memory, they

8 | Chapter 2: Getting Started with Mesos

剩余162页未读，继续阅读

xinconan2

粉丝: 269
资源: 399

Mesos应用开发：构建弹性、可扩展和分布式系统

BuildingApplicationsForTheMacAppStore.pdf 英文原版

Mesos.in.Action.2016.5.pdf

mesos集群在centos部署文档.pdf

Mesos Container在广告系统的应用.pdf

高洪涛：当当基于 Mesos 的 DevOps 实践.pdf

Mesos container在360广告系统的应用.pdf

Learning Spark 2015版.pdf

Mesos在教育课件云中的应用-黄凯.pdf

基于Mesos Docker的Elasticsearch容器化私有云.pdf

藏经阁-基于Mesos_Docker构建数据处理平台.pdf

最新资源