使用ELK堆栈构建实战业务洞察与指标

需积分: 9 45 浏览量更新于2024-07-20 收藏 5.01MB PDF 举报

"Practical ELK Stack: Build Actionable Insights and Business Metrics Using the Combined Power of Elasticsearch, Logstash, and Kibana" 《实用ELK栈》是一本详细介绍如何利用Elasticsearch、Logstash和Kibana这三款强大的开源工具构建系统，从而从各种数据源中获取有价值洞察和业务指标的指南。这本书旨在教你如何设置ELK栈，构建数据管道，并创建自定义插件，以满足特定的数据处理需求。 ELK栈由Elasticsearch（一个强大的搜索引擎和分布式实时数据分析引擎）、Logstash（用于收集、解析和转发日志数据的工具）和Kibana（提供数据可视化和交互式探索的界面）组成。作者Gurpreet S. Sachdeva在书中详细阐述了每个组件的关键特性及其在整体解决方案中的作用。 Logstash是ELK栈中的重要一环，它负责接收、过滤和解析事件数据。通过学习如何创建Logstash插件，读者可以扩展其功能，实现对自定义数据格式的支持。Elasticsearch则提供了高效的数据存储和检索能力，支持实时分析，而Kibana则以其直观的数据发现、可视化和仪表板功能，帮助用户将数据转化为可操作的见解。书中的章节涵盖了从日志分析的必要性到实时数据流分析，以及如何利用ELK栈进行数据索引、删除、搜索、映射和分析。此外，还介绍了如何利用聚合功能进行数据探索，以及如何利用Kibana创建引人入胜的可视化和仪表板。书中还分享了ELK栈在实际应用中的案例，如社交媒体分析和CERN的网格监控，以及Facebook、GitHub和Netflix等公司如何利用ELK栈。这本书面向的是希望通过ELK栈构建洞察力丰富和业务指标系统的开发者或DevOps工程师。无论你是初学者还是有经验的用户，都能从这本书中找到深入理解ELK栈和实践应用的宝贵知识。以下是书中的章节概览： 1. ELK栈介绍 2. 使用Logstash进行数据传输、过滤和解析 3. 扩展Logstash 4. 创建、索引和删除数据 5. 搜索数据 6. 映射与分析 7. 利用聚合进行数据探索 8. 探索Kibana 9. Kibana数据可视化 10. Kibana仪表板设计 11. 针对大规模设计 12. ELK栈在生产环境中的应用 13. 实际生活中的故事通过本书，读者不仅可以掌握ELK栈的基础知识，还能深入了解如何在实际项目中实施和优化，从而实现高效的数据管理和分析。

xix

About the Technical

Reviewers

Shyam Seshadri is currently a senior engineer at

Amazon, and before that headed the engineering

for Hopscotch, an e-commerce company targeted at

moms, and founded Fundoo Solutions Private Limited,

a Tech startup specializing in JavaScript (AngularJS

& NodeJS) and BigData. He co-authored the book

on AngularJS for O’Reilly publications, and conducts

Hands-on AngularJS workshops across the globe.

A geek at heart, even an MBA from the Indian School of

Business couldn’t keep him away from the technology

space.

Amit Singh is the Director of Engineering with the

Analytics and Information Management Practice in

Cognizant Technology Solutions. He has a Bachelor’s

of Technology Degree in Computer Science from

MIET, Meerut and a Post Graduate Diploma in IT from

IIIT-Bangalore. He is a seasoned Big Data Architect

with expertise in Fast Data and Search based analytics.

His current area of focus is IoT Analytics and is also

currently pursuing PhD in Data Sciences from

IIIT-Bangalore.

xxi

Acknowledgments

Writing this book was a challenging task but it provided me an opportunity to explore

something new. As pages were created and added, I realized how difficult it is to fit all the

details of the ELK stack within the page limit. I hope I have done the topic justice.

This book would not have been possible without the support and encouragement of

quite a few people. First of all, a big thanks to the makers of Elasticsearch, Logstash, and

Kibana. This wonderful stack has kept me hooked and provided the motivation to write

this book.

My sincere gratitude goes to Apress, my publisher, for giving me the opportunity to

author this book. Many people at Apress provided their invaluable support to make this

book possible:

• Celestin Suresh John, Senior Manager, Editorial Acquisition: I met

Celestin at a conference in Bangalore and casually discussed the

idea of writing a book. Celestin was open to it and asked me to

share ideas. After circulating few ideas and discussing them, we

settled on this book. Thank you, Celestin, for having the confidence

in me and for setting up the Apress team to work with me.

• Sanchita Mandal, Coordinating Editor: I have had support from

Sanchita in so many ways: by providing valuable feedback on

draft chapters; helping to plan the book and the structure of the

chapters; encouraging me; guiding me on upcoming steps; and

helping me overcome bumps in the road.

• Laura Berendson, Development Editor: Laura has done an

awesome job reviewing the complete book contents in a limited

time and catching even the smallest errors in the book. She

played a major role in shaping the organization of individual

chapters and the overall book.

• Technical reviewers Amit Singh and Shyam Seshadri have done

a terrific job of reviewing the contents and sharing their valuable

feedback and comments. The reviewers' detailed feedback has

helped me to improve the book throughout the writing process.

• I remain obligated to my organization, Aricent, and my colleagues

for providing me opportunity to explore lots of interesting

technologies and experiment with them.

Finally, I would like to thank my wife, Ina, daughter, Jasmine, and son, Ishu. This

book would not have been possible without their unconditional love, support, and

encouragement.

xxiii

Introduction

“In God we trust. All others must bring data.”

—W. Edwards Deming, statistician, professor, and author.

We are living in an increasingly connected world, with the IoT and big data raising the

bar for the amount of data that can be comprehended. There is an increasing trend

of IT infrastructure moving to public clouds like Amazon Web Services and Microsoft

Azure, making log analytics platforms more and more critical. Isolating performance

issues becomes challenging due to factors like load fluctuation, dynamic number of

users, and change in environments. These issues cannot be monitored by traditional log

management systems. Similarly, almost all kinds of computing devices, systems, and

applications emit logs to indicate the state of these systems. Elasticsearch is uniquely

positioned to perform log management for both cloud scale systems and traditional

computing environments.

The ELK stack, Elasticsearch, Logstash, and Kibana, is a powerful combination of

tools to address log management and data analytics. Elasticsearch provides deep search

and data analytics capabilities. It is a distributed, multitenant-capable, full-text search

engine with an HTTP web interface and schema-free JSON documents. Logstash enables

centralized logging, log enrichment, and parsing. It facilitates centralized data processing

of all types, normalizing varying schema and formats. Kibana provides powerful and

beautiful data visualizations. It provides visualization capabilities on top of the content

indexed on an Elasticsearch cluster. In short, the ELK stack makes searching and

analyzing data easier than ever before.

What Is This Book About?

I was first introduced to the ELK stack three years back while working on a project

to centralize logs and provide analytics on top of it. I was impressed by the capabilities

of this suite of tools and realized its potential to skim though huge quantities of logs

and provide elegant visualizations for it. Through this book I want to showcase the

amazing capabilities of Elasticsearch, Logstash, and Kibana. Just as it has helped me in

troubleshooting many challenging situations, I hope the readers will also benefit in a

similar manner.

This book will first introduce the ELK (Elasticsearch, Logstash, and Kibana) stack,

starting with showing how to set up the stack by installing the tools and the basic

configuration. Then it will demonstrate building a basic data pipeline using the ELK

stack. Next, you’ll explore the key features of Logstash and its role in the ELK stack,

including creating Logstash plugins, which will enable you to use your own customized

plugins. The importance of Elasticsearch and Kibana in the ELK stack is also covered,

along with various types of advanced data analysis, and a variety of charts, tables, and

maps. This book will cover the practical cases where the ELK stack is being used to

■ IntroduCtIon

xxiv

provide actionable insights. Detailed coverage is given to production-related aspects and

scaling the stack.

How Is the Book Organized?

Chapter 1 emphasizes the importance of log analysis in today’s big data-crazy

world. It analyzes the challenges with log analysis. It presents the ELK stack as a thorough

solution for log analysis. Different components of the ELK stack (Elasticsearch, Logstash,

and Kibana) are introduced with a description of their functions and installation.

Chapter 2 gets you started with using Logstash for log generation, collection, and

filtering. It begins with introducing the configuration settings of Logstash. It then goes on

to illustrate how Logstash facilitates shipping of logs, filtering, and transforming any type

of data to a common format. This can further help in arriving at actionable insights.

Chapter 3 throws light on the internal organization of Logstash and its plugins.

Logstash has a diverse collection of input, filter, codec, and output plugins. An overview

of the common plugins is provided. It then shows you how to create and use your own

custom plugin.

Chapter 4 introduces data management using Elasticsearch. This chapter shows

how to add data, index it, update it, and delete it. It goes on to show how to work with

distributed document stores.

Chapter 5 explores the elaborate mechanism for searching for data available in

Elasticsearch. It also illustrates Query DSL and filters.

Chapter 6 examines how Elasticsearch maps data. It then goes on to show how to

map data for relevant analysis.

Chapter 7 explores the subject of aggregates. It provides a top-level view of the entire

set of documents. This is unlike queries, which just focus on a particular document. It

also covers grouping of documents into buckets.

Chapter 8 introduces Kibana. It explains basic concepts and key features.

Chapter 9 shows how to work with Kibana by illustrating its interface to filter

and visualize log messages gathered by Elasticsearch. It covers the main interface

components, and demonstrates how to create searches, visualizations, and dashboards.

Chapter 10 covers the last piece in the Kibana armor: the dashboard. Various

visualizations can be combined to give a holistic view using a dashboard. This serves as a

single area for visualizing and analyzing data in real time.

Chapter 11 provides guidance on scaling the ELK cluster. This enhances the

capability to handle more data, index many more datasets, and search data faster. In

these days of cloud computing and NoSQL databases, scaling is very important because

there are situations when it is required to process millions or even billions of documents.

It’s not always possible to support this kind of load with one instance of Elasticsearch.

Chapter 12 addresses the key aspects of running the ELK stack in a production

environment. Monitoring the different components and troubleshooting any problem is

quite important. Custom configurations are required for specific scenarios.

Chapter 13 highlights some of the real life stories of how the ELK stack is being used

for a diverse set of scenarios. The ELK stack has transcended from the realm of lab trials

to live multinode clusters. These success stories should encourage you to experiment

with the ELK stack for your data storage and search needs.

Who Should Read This Book?

This book is for anybody who is dealing with data. The ELK stack can help in

solving existing problems and open the way to new features that have yet to be rolled

in. This book is for beginners and experienced users alike. While the beginners will get

剩余317页未读，继续阅读

ramissue

粉丝: 354
资源: 1487

使用ELK堆栈构建实战业务洞察与指标

Learning Elastic Stack 6.0-Packt Publishing(2017).pdf

Learning ELK Stack中文版

ELK Stack.md

帮我出十道elk面试题并给出答案

linux运维监控工具

ELK /. splunk.

推荐几篇关于ELK的文档 要资料全，质量高

创建elk用户并加属组elk

elk完整的docker-compose.yaml怎么写

电商日志模块有哪些技术可以实现

最新资源

推荐几篇关于ELK的文档要资料全，质量高