构建行动洞察：Practical ELK Stack教程（2017）

需积分: 10 153 浏览量更新于2024-07-20 1 收藏 5.04MB PDF 举报

《Practical ELK Stack》是一本由Gurpreet S. Sachdeva编著的实战指南，旨在帮助开发者和DevOps工程师利用Elasticsearch、Logstash和Kibana（ELK）堆栈构建能够提供行动导向见解和业务指标的系统。这本书的核心内容涵盖了以下几个关键知识点： 1. **需求背景与挑战**：书中首先介绍了日志分析的重要性以及当前面临的挑战，帮助读者理解在大数据时代，实时数据分析和转化成可操作洞察的需求。 2. **实时数据处理与分析**：作者指导如何在流式数据上进行实时分析，通过Logstash的事件处理功能（如数据传输、过滤和解析），实现数据的高效抓取和预处理。 3. **Elasticsearch技术详解**：作为核心组件，Elasticsearch被深入讲解，包括索引数据、删除数据以及其在存储和检索大量数据方面的高效性。读者可以了解到如何构建强大的搜索和分析能力。 4. **Logstash插件开发**：书中强调了Logstash的定制化能力，教导读者如何创建自己的Logstash插件，以便根据具体业务场景扩展和优化数据管道。 5. **Kibana的应用与可视化**：Kibana是数据分析和可视化的重要工具，作者展示了如何使用Kibana的数据发现、可视化和仪表板功能，创建出吸引人的图表、表格和地图，以支持决策制定。 6. **实践案例分享**：书中涵盖了一系列实际应用案例，从社交媒体分析到大型机构（如CERN）的网格监控，展示了ELK堆栈在不同领域的广泛使用，以及在Facebook、GitHub和Netflix等公司中的应用。 7. **受众定位**：《Practical ELK Stack》适合希望通过ELK堆栈从各种数据源获取深刻见解和商业价值的专业人士，特别是开发人员和DevOps工程师。 8. **版权与出版信息**：本书具有国际版权保护，提供了纸质版和电子版的ISBN号，以及Library of Congress Control Number，表明其出版时间和版权归属。《Practical ELK Stack》是一本实用的教程，不仅提供理论知识，还通过丰富的实践示例和技术细节，让读者掌握如何利用ELK堆栈构建高效的数据分析和可视化系统，从而提升业务效率和决策质量。

xix

About the Technical

Reviewers

Shyam Seshadri is currently a senior engineer at

Amazon, and before that headed the engineering

for Hopscotch, an e-commerce company targeted at

moms, and founded Fundoo Solutions Private Limited,

a Tech startup specializing in JavaScript (AngularJS

& NodeJS) and BigData. He co-authored the book

on AngularJS for O’Reilly publications, and conducts

Hands-on AngularJS workshops across the globe.

A geek at heart, even an MBA from the Indian School of

Business couldn’t keep him away from the technology

space.

Amit Singh is the Director of Engineering with the

Analytics and Information Management Practice in

Cognizant Technology Solutions. He has a Bachelor’s

of Technology Degree in Computer Science from

MIET, Meerut and a Post Graduate Diploma in IT from

IIIT-Bangalore. He is a seasoned Big Data Architect

with expertise in Fast Data and Search based analytics.

His current area of focus is IoT Analytics and is also

currently pursuing PhD in Data Sciences from

IIIT-Bangalore.

xxi

Acknowledgments

Writing this book was a challenging task but it provided me an opportunity to explore

something new. As pages were created and added, I realized how difficult it is to fit all the

details of the ELK stack within the page limit. I hope I have done the topic justice.

This book would not have been possible without the support and encouragement of

quite a few people. First of all, a big thanks to the makers of Elasticsearch, Logstash, and

Kibana. This wonderful stack has kept me hooked and provided the motivation to write

this book.

My sincere gratitude goes to Apress, my publisher, for giving me the opportunity to

author this book. Many people at Apress provided their invaluable support to make this

book possible:

• Celestin Suresh John, Senior Manager, Editorial Acquisition: I met

Celestin at a conference in Bangalore and casually discussed the

idea of writing a book. Celestin was open to it and asked me to

share ideas. After circulating few ideas and discussing them, we

settled on this book. Thank you, Celestin, for having the confidence

in me and for setting up the Apress team to work with me.

• Sanchita Mandal, Coordinating Editor: I have had support from

Sanchita in so many ways: by providing valuable feedback on

draft chapters; helping to plan the book and the structure of the

chapters; encouraging me; guiding me on upcoming steps; and

helping me overcome bumps in the road.

• Laura Berendson, Development Editor: Laura has done an

awesome job reviewing the complete book contents in a limited

time and catching even the smallest errors in the book. She

played a major role in shaping the organization of individual

chapters and the overall book.

• Technical reviewers Amit Singh and Shyam Seshadri have done

a terrific job of reviewing the contents and sharing their valuable

feedback and comments. The reviewers' detailed feedback has

helped me to improve the book throughout the writing process.

• I remain obligated to my organization, Aricent, and my colleagues

for providing me opportunity to explore lots of interesting

technologies and experiment with them.

Finally, I would like to thank my wife, Ina, daughter, Jasmine, and son, Ishu. This

book would not have been possible without their unconditional love, support, and

encouragement.

xxiii

Introduction

“In God we trust. All others must bring data.”

—W. Edwards Deming, statistician, professor, and author.

We are living in an increasingly connected world, with the IoT and big data raising the

bar for the amount of data that can be comprehended. There is an increasing trend

of IT infrastructure moving to public clouds like Amazon Web Services and Microsoft

Azure, making log analytics platforms more and more critical. Isolating performance

issues becomes challenging due to factors like load fluctuation, dynamic number of

users, and change in environments. These issues cannot be monitored by traditional log

management systems. Similarly, almost all kinds of computing devices, systems, and

applications emit logs to indicate the state of these systems. Elasticsearch is uniquely

positioned to perform log management for both cloud scale systems and traditional

computing environments.

The ELK stack, Elasticsearch, Logstash, and Kibana, is a powerful combination of

tools to address log management and data analytics. Elasticsearch provides deep search

and data analytics capabilities. It is a distributed, multitenant-capable, full-text search

engine with an HTTP web interface and schema-free JSON documents. Logstash enables

centralized logging, log enrichment, and parsing. It facilitates centralized data processing

of all types, normalizing varying schema and formats. Kibana provides powerful and

beautiful data visualizations. It provides visualization capabilities on top of the content

indexed on an Elasticsearch cluster. In short, the ELK stack makes searching and

analyzing data easier than ever before.

What Is This Book About?

I was first introduced to the ELK stack three years back while working on a project

to centralize logs and provide analytics on top of it. I was impressed by the capabilities

of this suite of tools and realized its potential to skim though huge quantities of logs

and provide elegant visualizations for it. Through this book I want to showcase the

amazing capabilities of Elasticsearch, Logstash, and Kibana. Just as it has helped me in

troubleshooting many challenging situations, I hope the readers will also benefit in a

similar manner.

This book will first introduce the ELK (Elasticsearch, Logstash, and Kibana) stack,

starting with showing how to set up the stack by installing the tools and the basic

configuration. Then it will demonstrate building a basic data pipeline using the ELK

stack. Next, you’ll explore the key features of Logstash and its role in the ELK stack,

including creating Logstash plugins, which will enable you to use your own customized

plugins. The importance of Elasticsearch and Kibana in the ELK stack is also covered,

along with various types of advanced data analysis, and a variety of charts, tables, and

maps. This book will cover the practical cases where the ELK stack is being used to

■ IntroduCtIon

xxiv

provide actionable insights. Detailed coverage is given to production-related aspects and

scaling the stack.

How Is the Book Organized?

Chapter 1 emphasizes the importance of log analysis in today’s big data-crazy

world. It analyzes the challenges with log analysis. It presents the ELK stack as a thorough

solution for log analysis. Different components of the ELK stack (Elasticsearch, Logstash,

and Kibana) are introduced with a description of their functions and installation.

Chapter 2 gets you started with using Logstash for log generation, collection, and

filtering. It begins with introducing the configuration settings of Logstash. It then goes on

to illustrate how Logstash facilitates shipping of logs, filtering, and transforming any type

of data to a common format. This can further help in arriving at actionable insights.

Chapter 3 throws light on the internal organization of Logstash and its plugins.

Logstash has a diverse collection of input, filter, codec, and output plugins. An overview

of the common plugins is provided. It then shows you how to create and use your own

custom plugin.

Chapter 4 introduces data management using Elasticsearch. This chapter shows

how to add data, index it, update it, and delete it. It goes on to show how to work with

distributed document stores.

Chapter 5 explores the elaborate mechanism for searching for data available in

Elasticsearch. It also illustrates Query DSL and filters.

Chapter 6 examines how Elasticsearch maps data. It then goes on to show how to

map data for relevant analysis.

Chapter 7 explores the subject of aggregates. It provides a top-level view of the entire

set of documents. This is unlike queries, which just focus on a particular document. It

also covers grouping of documents into buckets.

Chapter 8 introduces Kibana. It explains basic concepts and key features.

Chapter 9 shows how to work with Kibana by illustrating its interface to filter

and visualize log messages gathered by Elasticsearch. It covers the main interface

components, and demonstrates how to create searches, visualizations, and dashboards.

Chapter 10 covers the last piece in the Kibana armor: the dashboard. Various

visualizations can be combined to give a holistic view using a dashboard. This serves as a

single area for visualizing and analyzing data in real time.

Chapter 11 provides guidance on scaling the ELK cluster. This enhances the

capability to handle more data, index many more datasets, and search data faster. In

these days of cloud computing and NoSQL databases, scaling is very important because

there are situations when it is required to process millions or even billions of documents.

It’s not always possible to support this kind of load with one instance of Elasticsearch.

Chapter 12 addresses the key aspects of running the ELK stack in a production

environment. Monitoring the different components and troubleshooting any problem is

quite important. Custom configurations are required for specific scenarios.

Chapter 13 highlights some of the real life stories of how the ELK stack is being used

for a diverse set of scenarios. The ELK stack has transcended from the realm of lab trials

to live multinode clusters. These success stories should encourage you to experiment

with the ELK stack for your data storage and search needs.

Who Should Read This Book?

This book is for anybody who is dealing with data. The ELK stack can help in

solving existing problems and open the way to new features that have yet to be rolled

in. This book is for beginners and experienced users alike. While the beginners will get

剩余317页未读，继续阅读

vanridin

粉丝: 108
资源: 1187

构建行动洞察：Practical ELK Stack教程（2017）

ops-elkstack:Elk Stack演示和部署

elk-stack-project1:ELK Stack配置

Learning ELK Stack

Elk-Stack-Project:Ansible，Docker，Elk Stack

ELK Stack.md

elk stack中文教程

ELK Stack 权威指南

ELKstack权威指南

ELKstack 中文指南

2023-04-06-项目笔记 - 第二百八十九阶段 - 4.4.2.287全局变量的作用域-287 -2025.10.17

最新资源