利用Splunk进行大数据分析：实战指南

需积分: 9 131 浏览量更新于2024-07-17 收藏 9.82MB PDF 举报

"Big Data Analytics Using Splunk" 是一本专为大数据分析爱好者和专业人员设计的电子书，由Apress出版社发布，提供了高清且是最新的英文版。本书涵盖了大数据领域的重要概念和技术，特别是如何利用Splunk这一强大的数据分析工具进行深入挖掘和处理。该书分为16个章节，从入门到进阶，确保读者逐步掌握 Splunk 的各项功能。首先，第1章"Big Data and Splunk" 引导读者理解大数据的基本概念以及它在实际场景中的应用，以及如何通过Splunk平台进行集成和管理。第二章"Getting Data into Splunk" 介绍了数据导入的方法，包括各种数据源的连接和配置，使读者能熟练地将海量数据导入到 Splunk 中进行后续分析。接下来的章节进一步深入，如第3章"Processing and Analyzing the Data"，讲述了数据清洗、预处理和基础统计分析的技巧。第4章"Visualizing the Results" 部分着重于数据可视化，如何使用 Splunk的各种图表和仪表板呈现复杂的数据模式，帮助用户更直观地理解数据背后的故事。定义警报和监控在第5章"Defining Alerts" 中讨论，读者可以学习如何设置自动化警报，及时发现异常情况。网络网站监控是第6章的主题，探讨如何通过Splunk监控网站性能和可用性，确保业务连续性。第7章"Using Log Files to Create Advanced Analytics" 提供了高级分析技术，比如使用日志文件进行深入洞察，对业务操作进行优化。 "Airline On-Time Performance Project" 在第8章中作为一个实战项目，展示了如何运用Splunk分析航班准时率，从而提升运营效率。第9至11章分别涉及获取航班数据、分析航班、机场和延误情况，以及对特定航班多年数据的长期趋势分析。社交媒体分析在第12章"Analyzing Tweets" 和第13章"Analyzing Foursquare Check-Ins" 中展开，展示了如何利用社交媒体数据进行消费者行为分析和地点定位。第14章"Sentiment Analysis" 则讲解情感分析，帮助企业理解客户反馈的情感倾向。远程数据收集在第15章中探讨，关注如何有效地搜集来自不同地理位置的数据。最后，第16章"Scaling and High Availability" 关注的是如何扩展和确保Splunk系统的高可用性，确保在处理大规模数据时的稳定性和可靠性。附录A提供性能管理和最佳实践，为读者提供了深入理解和优化Splunk性能的实用指南。这本书不仅适合初学者，也是经验丰富的数据分析师和 IT 专业人士提升技能、解决实际问题的宝贵资源。

CHAPTER 2 ■ GETTING DATA INTO SPLUNK

The dialog box that comes up gives you options to set the source type. By default Splunk parses the data files

provided as input and determines what the source type for the file is. It then loads or indexes the file according to

the source type characteristics. In this case, because we are loading an access log file, which has been identified as a

combined access log file, we accept the default option of “Use auto-detected source type” as seen in Figure 2-3 and

click the Continue button. In Chapter 3, we will explain in detail the format of an access combined log file and what

information is stored in the log entries. In Chapter 9, we will review other options such as defining the input of custom

data files, which needs to be processed differently.

What are typically known as records, or just plain lines of data, are referred to in Splunk as events, and every

event has a timestamp. Throughout the book we will be using, analyzing, and manipulating timestamps as they are

a key element of Splunk and big data analysis. The next screen in the data loading process presents the way Splunk

has broken down the log entries into different events along with associated timestamps. The default behavior is

Figure 2-2. Preview data

Figure 2-3. Set source type

CHAPTER 2 ■ GETTING DATA INTO SPLUNK

to break an event on the timestamp, but if Splunk cannot find the timestamp it will present one single event that

contains all the lines of the file. We can then customize where to break the entries into different events, which we

will learn in Chapter 9. In this case, Splunk breaks down very nicely the entries of the access log into events as can

be seen in Figure 2-4. Here you can see that each event has the information about a user activity that happened on

MyGizmoStore.com. The preview option also shows the number of events extracted from the log file, which in this case

contains 243 events.

Note ■ If you are familiar with relational database concepts, it may help to realize that each event broken down by

Splunk is conceptually equal to a single row in a relational database table.

Now that we have confirmed that the data looks good and has been processed correctly by Splunk, you can click

on the Continue button, which will take us to the “Add new” page, where we will customize some of the settings before

we get the data finally indexed into Splunk. Because this is a one-time file processing operation, we will select the

“Index a file once from this Splunk server” radio button under Source. Because our computer has a rather complex

name, we also chose to change the Host file value to BigDBook-Test.

In the previous step, we asked Splunk to automatically determine the sourcetype, so we will see the sourcetype

is set to automatic. By default, when data is loaded into Splunk it goes into the main index, which is the mechanism

used to store, process, and analyze data. Splunk also offers the ability to define and use other indexes, which can help

you better organize and manage your data, especially regarding data governance, such as access, protection, and

retention policies. For this example, we will use the main, but we will see how to create a new one and make use of it

in the next section. Click on the Save button. The next page shows a Success message if Splunk is able to process the

data completely.

Figure 2-4. Data preview for access.log

CHAPTER 2 ■ GETTING DATA INTO SPLUNK

What we have seen so far is how to get data into Splunk using one-time file indexing. In real-world use cases, we

would be seeing that log files are being continuously updated with new entries, and these expanded log files need

to be processed and analyzed on a continuous basis. This is where Splunk provides the ability to monitor certain

directories where files are being continuously updated. Splunk’s directory monitoring capability lets us specify the

directory that needs to be monitored and the files in that directory will be used as data input. Before we explore the

monitoring option, let us go back to MyGizmoStore.com one more time. We have used a static log file from the online

store to see how we can get that kind of data into Splunk. In the real-world MyGizmoStore.com would probably be

running on multiple web servers on different hosts and writing out log files which need to be monitored, processed,

and analyzed on a continuous basis.

To simulate this behavior we have come up with a test data generator to generate log files for MyGizmoStore.com.

We will review how the test data generator operates and how we can start generating the data.

Data Generator

The sample data generator is written using the Python scripting language. It is designed to run on both Microsoft

Windows and Linux operating systems. The sample data generator uses a random generator to create random IP

addresses, which point to the visitor of the web site; a predetermined set of product identification codes that look

like “CA-NY-99” and item identification codes in the form of HYD-19. The gizmos sold on the store can be categorized

as follows:

BLUE_GIZMOS•

RED_GADGETS•

WHITE_WIDGETS•

ORANGE_WATCHMACALLITS•

PURPLE_DOOHICKEYS•

BLACK_DOODADS•

The visitors to the web site can perform the following actions:

Purchase•

Add an item to the shopping cart (Addtocart)•

Remove an item from the shopping cart (Remove)•

View the catalog of gizmos (View)•

Change the quantity of an item in the shopping cart (Changequantity)•

The HTTP protocol used by web sites includes a return code that either signifies success in the operation or

describes a problem. The code 200 indicates a successful operation, whereas codes in the 400s and 500s indicate some

sort of a problem. The data generator creates a realistic percentage of error codes. Additionally, the generator includes

a random selection of user agents. These are a fancy name for the description of the combination of the browser and

operating system used by the visitors, which also include the version number of both.

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)•

Opera/9.01 (Windows NT 5.1; U; en)•

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-US) AppleWebKit/533.4 (KHTML, like •

Gecko) Chrome/5.0.375.38 Safari/533.4

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) •

剩余361页未读，继续阅读

J-10

粉丝: 18
资源: 483

利用Splunk进行大数据分析：实战指南

BigDataAnalyticsUsingSplunk.pdf 英文原版

Apress.Big.Data.Analytics.Using.Splunk.May.2013

big-data-analytics-using-splunk:Peter Zadrozny和Raghu Kodali的“使用Splunk进行大数据分析”的源代码-Big source code

Big Data Made Easy - A Working Guide To The Complete Hadoop Toolset

Splunk 2020PPT汇总（36份）.zip

实时大数据分析：Splunk实战

MiniGui业务开发基础培训-htk

com.harmonyos.exception.DiskReadWriteException(解决方案).md

网络分析-Wireshark数据包筛选技巧详解及应用实例

com.harmonyos.exception.BatteryOverheatException(解决方案).md

最新资源