分布式实时计算模式：Storm实战蓝图

5星 · 超过95%的资源需积分: 9 167 浏览量更新于2024-07-23 1 收藏 4.62MB PDF 举报

"Packt.Storm Blueprints.Patterns for Distributed Real-time Computation" 本书"Storm Blueprints: Patterns for Distributed Real-time Computation"由P. Taylor Goetz和Brian O'Neill撰写，是一本详细介绍如何利用Apache Storm进行分布式实时数据处理和分析的权威指南。Storm是一款流行的开源大数据处理框架，特别适用于构建实时流处理系统。书中详细阐述了各种Storm设计模式，旨在帮助读者理解和实施适用于现实世界案例的分布式实时计算解决方案。书中的内容涵盖了多个关键知识点： 1. **Apache Storm介绍**：首先，书中会详细介绍Apache Storm的基本概念，包括其工作原理、核心组件（如Spout和Bolt）以及它如何实现高可用性和容错性。 2. **实时数据处理**：书中会深入探讨实时数据处理的重要性，特别是在大数据分析和快速响应系统中的角色。读者将学习到如何利用Storm构建能够处理大量实时数据流的系统。 3. **分布式计算模式**：作者会分享一系列设计模式，这些模式在构建复杂的实时处理管道时非常有用，如窗口操作（Tumbling Windows, Sliding Windows等）、状态管理和容错机制。 4. **案例研究**：书中包含多个真实世界的案例，如社交媒体分析、实时交易监控和物联网(IoT)数据处理，通过这些案例，读者可以理解如何将理论应用于实际场景。 5. **开发与部署**：详细指导如何设置开发环境，编写Storm拓扑，并将其部署到生产环境中。这包括使用Storm UI进行监控和故障排查。 6. **性能优化**：讨论如何优化Storm拓扑以提高处理速度和资源利用率，包括并行度调整、数据本地化策略以及数据序列化技术。 7. **集成其他技术**：书中可能会涵盖Storm与其他大数据工具（如Hadoop、Cassandra、Kafka等）的集成，以展示如何构建端到端的数据处理解决方案。 8. **最佳实践和安全**：提供有关如何确保系统的健壮性和安全性的一些建议，包括数据加密、身份验证和访问控制。 "Storm Blueprints"是针对那些想要利用Apache Storm进行分布式实时计算的开发者和数据工程师的重要资源。通过学习这本书，读者不仅可以掌握Storm的基础知识，还能获得解决实际问题的实用技巧和经验。

Table of Contents

[ vii ]

Launching an EC2 instance manually 283

Logging in to the EC2 instance 285

Introducing Apache Whirr 285

Installing Whirr 286

Conguring a Storm cluster with Whirr 287

Launching the cluster 288

Introducing Whirr Storm 289

Setting up Whirr Storm 289

Cluster conguration 290

Customizing Storm's conguration 291

Customizing rewall rules 292

Introducing Vagrant 294

Installing Vagrant 294

Launching your rst virtual machine 294

The Vagrantle and shared lesystem 296

Vagrant provisioning 297

Conguring multimachine clusters with Vagrant 298

Creating Storm-provisioning scripts 299

ZooKeeper 299

Storm 299

Supervisord 301

The Storm Vagrantle 301

Launching the Storm cluster 304

Summary 305

Index 307

Preface

The demand for timely, actionable information is pushing software systems to

process an increasing amount of data in a decreasing amount of time. Additionally,

as the number of connected devices increases and as these devices are applied

to a broadening spectrum of industries, that demand is becoming increasingly

pervasive. Traditional enterprise operational systems are being forced to operate on

scales of data that were originally associated only with Internet-scale companies.

This monumental shift is forcing the collapse of more traditional architectures and

approaches that separated online transactional systems and ofine analysis. Instead,

people are reimagining what it means to extract information from data. Frameworks

and infrastructure are likewise evolving to accommodate this new vision.

Specically, data generation is now viewed as a series of discrete events. Those event

streams are associated with data ows, some operational and some analytical, but

processed by a common framework and infrastructure.

Storm is the most popular framework for real-time stream processing. It provides

the fundamental primitives and guarantees required for fault-tolerant distributed

computing in high-volume, mission-critical applications. It is both an integration

technology as well as a data ow and control mechanism. Many large companies

are using Storm as the backbone of their big data platforms.

Using design patterns from this book, you will learn to develop, deploy, and operate

data processing ows capable of processing billions of transactions per hour/day.

Storm Blueprints: Patterns for Distributed Real-time Computation covers a broad range

of distributed computing topics, including not only design and integration patterns

but also domains and applications to which the technology is immediately useful

and commonly applied. This book introduces the reader to Storm using real-world

examples, beginning with simple Storm topologies. The examples increase in

complexity, introducing advanced Storm concepts as well as more sophisticated

approaches to deployment and operational concerns.

Preface

[ 2 ]

What this book covers

Chapter 1, Distributed Word Count, introduces the core concepts of distributed stream

processing with Storm. The distributed word count example demonstrates many of

the structures, techniques, and patterns required for more complex computations.

In this chapter, we will gain a basic understanding of the structure of Storm

computations. We will set up a development environment and understand

the techniques used to debug and develop Storm applications.

Chapter 2, Conguring Storm Clusters, provides a deeper look into the Storm

technology stack and the process of setting up and deploying to a Storm cluster.

In this chapter, we will automate the installation and conguration of a multi-node

cluster using the Puppet provisioning tool.

Chapter 3, Trident Topologies and Sensor Data, covers Trident topologies. Trident

provides a higher-level abstraction on top of Storm that abstracts away the details

of transactional processing and state management. In this chapter, we will apply

the Trident framework to process, aggregate, and lter sensor data to detect a

disease outbreak.

Chapter 4, Real-time Trend Analysis, introduces trend analysis techniques using

Storm and Trident. Real-time trend analysis involves identifying patterns in data

streams. In this chapter, you will integrate with Apache Kafka and will implement

a sliding window to compute moving averages.

Chapter 5, Real-time Graph Analysis, covers graph analysis using Storm to persist data

to a graph database and query that data to discover relationships. Graph databases

are databases that store data as graph structures with vertices, edges, and properties

and focus primarily on relationships between entities. In this chapter, you will

integrate Storm with Titan, a popular graph database, using Twitter as a data source.

Chapter 6, Articial Intelligence, applies Storm to an articial intelligence algorithm

typically implemented using recursion. We expose some of the limitations of Storm,

and examine patterns to accommodate those limitations. In this chapter, using

Distributed Remote Procedure Call (DRPC), you will implement a Storm topology

capable of servicing synchronous queries to determine the next best move in

tic-tac-toe.

Chapter 7, Integrating Druid for Financial Analytics, demonstrates the complexities

of integrating Storm with non-transactional systems. To support such integrations,

the chapter presents a pattern that leverages ZooKeeper to manage the distributed

state. In this chapter, you will integrate Storm with Druid, which is an open source

infrastructure for exploratory analytics, to deliver a congurable real-time system

for analysis of nancial events.

剩余335页未读，继续阅读

唐小牛

粉丝: 0
资源: 6

分布式实时计算模式：Storm实战蓝图

Storm Blueprints- Patterns for Distributed Real-time Computation(PACKT,2014)

Storm Blueprints: Patterns for Distributed Real-time Computation

Packt.Android.Design.Patterns.and.Best.Practices.2017

Packt.Applied.Architecture.Patterns.on.the.Microsoft.Platform

高清彩版 Packt.Implementing.Cloud.Design.Patterns.for.AWS

Packt.Django.Project.Blueprints.2016

Packt.Arduino.Android.Blueprints.2014

Packt.Java.9.Programming.Blueprints

Packt.-.NET.Design.Patterns.2017.Retail.EPUB.eBook-BitBook

PacktPub.Storm.Real-time.Processing.Cookbook.Aug.2013.pdf

最新资源