Mesos之上的应用构建

需积分: 9 88 浏览量更新于2024-07-20 收藏 3.87MB PDF 举报

"Building Applications on Mesos - David Greenberg - 利用弹性、可扩展和分布式系统构建应用" 《Mesos上的应用构建》是David Greenberg的一部作品，它深入探讨了如何利用Mesos这一强大的分布式系统框架来构建具有高可用性、可扩展性和弹性的应用程序。这本书对于那些想要在大规模分布式环境中开发和部署应用程序的IT专业人士来说，是一份宝贵的指南。 Mesos是Apache软件基金会的一个开源项目，设计用于管理和调度数据中心的计算资源，提供了一种高效、灵活的方式来运行各种分布式应用和服务。Mesos的核心功能是将计算资源抽象化，使应用程序能够跨多台机器无缝地运行，同时确保资源的高效分配和利用率。书中可能涵盖了以下几个关键知识点： 1. **Mesos架构**：介绍Mesos的基本架构，包括Mesos Master、Agent节点以及框架的概念。Master节点负责管理整个集群的资源分配，而Agent节点则是在物理或虚拟机上运行任务的实际执行者。框架是运行在Mesos之上的应用，它们可以是数据处理系统（如Spark）、容器编排系统（如Kubernetes或 Marathon）等。 2. **资源调度**：详细讲解Mesos如何进行资源调度，包括如何分配CPU、内存、磁盘和网络资源，以及如何实现资源的公平共享和优先级调度。 3. **容错与弹性**：讨论Mesos如何通过复制和故障转移机制实现系统的高可用性。这可能涉及Mesos的自动故障恢复机制，以及如何设计能够容忍硬件或软件故障的应用。 4. **分布式应用开发**：指导读者如何设计和构建能在Mesos上运行的分布式应用，包括编写能够与Mesos API交互的框架，以及如何确保应用的可扩展性和容错性。 5. **容器化与Docker集成**：由于Mesos对容器的支持，书籍可能会详细阐述如何使用Docker容器来打包和运行应用，并且解释Mesos如何与Docker API集成以提供容器编排能力。 6. **Marathon和Kubernetes等框架**：Mesos支持多种框架，比如Marathon用于长期运行的服务，Kubernetes用于更全面的容器编排。书中的内容可能会涵盖这些框架的使用和配置，以及如何利用它们构建微服务架构。 7. **监控和日志管理**：介绍如何在Mesos集群中实现有效的监控和日志收集，以确保系统的健康和性能优化。 8. **实战案例**：通过实际案例分析，展示如何在生产环境中部署和管理Mesos集群，以及如何解决可能出现的问题。 9. **最佳实践**：提供关于优化Mesos集群性能、提高资源利用率和确保系统稳定性的最佳实践。《Mesos上的应用构建》为读者提供了一个全面了解和掌握Mesos平台的途径，有助于开发者和系统管理员构建和管理现代化、弹性化的云原生应用。通过学习这本书，读者将能更好地理解和利用Mesos的强大功能，以应对日益复杂和动态的分布式计算挑战。

tems (like Supervisor and SystemD). Groups of tasks can be combined to describe a

“role” or “recipe,” which represents a higher-level concept, like setting up a database

or web server. Finally, these groups of tasks are applied to an “inventory,” or set of

hosts, which then configures them according to the specification.

These tools are wildly more useful than the Bash, Perl, and SSH mashups of old; how‐

ever, they suffer from fundamental limitations. On the one hand, they allow for con‐

figuration to be written in a structured, understandable form. They enable the use of

best practices from software development—source control, encapsulation, and code

reuse through libraries and plug-ins—to extend to system administration. On the

other hand, they are fundamentally designed to make a fleet of machines conform to

a static configuration. This is intentional: once you’ve run your Ansible or Puppet

configuration, you want the cluster to be in the same, known consistent state. If, how‐

ever, you want your cluster to dynamically reallocate resources or shift configurations

depending on various external factors, such as the current load, these tools can’t help

you.

In this sense, it can be useful to think of Mesos as a deployment system. Distributed

applications can be made into Mesos frameworks for use on a Mesos cluster. Rather

than forcing you to use a static configuration description, each framework is essen‐

tially a role or recipe: the framework contains all the logic needed to install, start,

monitor, and use a given application. Many frameworks are themselves platforms;

when many applications follow the same general deployment pattern (like web

servers), then a single framework can manage them all (like Marathon, a Mesos

framework for stateless services). The power of Mesos is that since the framework is a

live, running program, it can make decisions on the fly, reacting to changing work‐

loads and cluster conditions. For instance, a framework could observe how many

resources seem to be available, and change its execution strategy to better perform

given the current cluster conditions. Because the deployment system runs constantly,

it can notice and adapt to failures in real time, automatically starting up new servers

as previous ones fail. Mesos frameworks are strictly more powerful than static

descriptions used by more traditional deployment systems.

Thus, you can think of Mesos as supplanting much of the functionality that Ansible

or Chef previously provided. You’ll still need a traditional configuration manager to

bootstrap the Mesos cluster; however, Mesos provides a more powerful platform for

hosting applications: frameworks can automatically and dynamically adapt to failing

machines and changing workloads, giving operators greater peace of mind and time

to work on new initiatives.

Mesos as an Execution Platform

Another way to think of Mesos is as a platform for hosting applications. Today, you

may use Heroku to run your web services, or you may manage a Hadoop cluster to

4 | Chapter 1: Introduction to Mesos

execute MapReduce jobs. In these examples, Heroku and the Hadoop cluster are the

platforms for web services and MapReduce jobs. A Mesos cluster can also be a plat‐

form for higher-level applications. But if your system already works, why add Mesos

to the mix? Mesos offers flexibility and mitigates the risk of being outpaced by tech‐

nology development; with Mesos, launching a Spark cluster or switching to a newer

technology is as simple as launching the framework and watching it bootstrap itself.

Consider the Heroku/PaaS scenario: Marathon can reliably launch your application,

automatically starting up new instances when machines crash or go down for mainte‐

nance. Mesos-DNS and HAProxy can manage your load balancing for you (see Chap‐

ter 3). But once you’ve got Mesos running, why bother continuing to maintain your

Hadoop cluster? After all, you could launch Hadoop on Mesos via Myriad, and you

could even stop dealing with HDFS administration, since there’s a framework for that

(Mesos HDFS).

Thus, you can think of Mesos as an execution platform: rather than paying for third-

party PaaS solutions and creating bespoke clusters for each big data analysis technol‐

ogy, you can have a single Mesos cluster. With your Mesos cluster as the foundation,

you can easily launch whatever frameworks you require to provide whatever func‐

tionality is needed, now or as new requirements and technologies arise.

How This Book Is Structured

At this point, let’s take a look at how the rest of the book will be structured. First,

we’re going to learn about Mesos itself. We’ll cover the architecture of Mesos and its

frameworks, and we’ll learn how resources are allocated across the cluster. We’ll also

dive into configuration settings you’ll want to know, such as how to provide service-

level agreements (SLAs) to certain frameworks, and useful command-line settings to

tune the overall performance and workload of your cluster.

Then, we’ll learn about two existing open source Mesos frameworks—Marathon and

Chronos—that provide application hosting and job scheduling functionality. We’ll

explore how to configure them, how to interact with them, and how to build deeper

integrations. After learning about these frameworks, you will be able to host several

common application architectures, such as web services and data pipelines, on Mesos,

with high reliability and scalability.

In the later parts of this book, we’ll go into great detail about how to build a custom

framework for Mesos. We’ll learn about the APIs that it exposes through the process

of implementing a sample job scheduling framework in Java. In developing this

framework, we’ll delve into how to build on the core Mesos APIs, what some com‐

mon pitfalls are and how to avoid them, and how to decide whether to build a new

framework or adapt an existing system.

How This Book Is Structured | 5

When a scheduler wants to do some work, it launches an executor. The executor is

simply the scheduler’s worker: the scheduler then decides to send one or more tasks

to the executor, which will work on those tasks independently, sending status updates

to the scheduler until the tasks are complete. We’ll learn more about the distinction

between tasks and executors in “Tasks and Executors” on page 21.

Now that we’ve got an idea of what a framework is, let’s take a step down the stack to

learn about the Mesos cluster itself.

Masters and Slaves

A Mesos cluster is comprised of two components: the Mesos masters and the Mesos

slaves. The masters are the software that coordinates the cluster; the slaves are what

execute code in containers. You might be thinking, “That sounds awfully similar to

how frameworks work.”

In that case, you’d be 100% correct. The Mesos cluster mirrors the organization of the

frameworks that run on top of it, because the master/slave pattern is simple, flexible,

and effective. The cluster software needs to be separate, however, to ensure that

frameworks remain isolated and secure from one another. First, we’ll look at a few

key ideas about the Mesos masters; then, we’ll take a look at the slaves.

The Masters

The Mesos masters are the brain of the cluster. They have several responsibilities:

•

They are the central source of truth for running tasks.

• They fairly share the cluster between all the connected frameworks.

• They host the primary UI for the cluster.

•

They ensure high availability and efficient allocation of resources. Considered

together, they are a single point of failure, but Mesos automatically handles and

recovers from individual master failures.

Let’s unpack these responsibilities, and see what they mean for the design of the mas‐

ters and how they impact our clusters:

Masters are the central source of truth for all running tasks

In order for masters to be able to serve the UI and data about tasks with minimal

latency, all task metadata is kept in memory. This means that the nodes that run

the masters should have plenty of memory—on a cluster with thousands of con‐

current tasks, the masters could, for instance, consume 20 GB of RAM. Note that

this excludes completed tasks, which relieves the memory pressure on the mas‐

ters for some workloads. The masters make an effort to store the metadata

related to recently completed tasks, but in the interest of saving memory, they

8 | Chapter 2: Getting Started with Mesos

剩余162页未读，继续阅读

libingchen

粉丝: 29
资源: 263

Mesos之上的应用构建

MES界面开发

Mastering Mesos

none:Anyy Command oN Mesos

藏经阁-Microservices on DC OS and Container Orchestration on Mesos.

闫航-Docker on Mesos: 云计算的革命

nix-aurora:Nis on Mesos上的Aurora

HDFS on Apache Mesos.zip

Paco Nathan：Spark on Elastic Mesos

mesos

mesos-on-docker

最新资源