大数据清洗策略与实战技巧

需积分: 41 132 浏览量更新于2024-07-18 收藏 12.95MB PDF 举报

"在大数据处理中，数据清洗是至关重要的步骤。这关乎到数据分析结果的准确性和可靠性。本文将分享一些关于如何在大数据环境中高效地进行数据清理的经验和策略。数据清洗，通常被称为数据预处理，是数据分析流程中的关键环节。它涉及到查找并处理数据集中的不一致、错误、缺失值以及异常值。以下是一些核心的数据清洗方法： 1. **缺失值处理**：数据集中经常会出现缺失值，这可能是由于记录时的疏忽或是数据源本身的问题。处理缺失值的方法有删除含有缺失值的记录（如果数量不大），使用平均值、中位数或众数填充，以及使用预测模型来估算缺失值。 2. **异常值检测与处理**：异常值是那些远离正常数据分布的数值，可能对统计分析产生严重影响。可以使用箱型图、Z-score或IQR方法识别异常值，并根据业务需求决定是删除还是替换。 3. **重复值处理**：数据集中可能存在完全相同的记录，这会影响分析结果。通过去重操作，可以确保每个观测值只被计算一次。 4. **一致性检查**：检查数据的一致性是数据清洗的重要部分，比如日期格式的一致、数值单位的一致等。不一致的数据可能导致错误的分析结果。 5. **数据类型转换**：确保数据类型正确是数据清洗的重要任务。例如，日期应该被存储为日期类型，而数值应该为数字类型，以方便后续的计算和分析。 6. **异常检测**：利用统计学方法和机器学习算法，可以发现数据集中的异常模式，这些异常可能是数据质量问题的标志。 7. **数据规范化**：对于具有不同量级或范围的特征，进行标准化或归一化可以使它们在分析时处于相同的基础水平。 8. **字符串处理**：包括去除不必要的空格、统一大小写、提取子串等，以提高数据的可读性和处理效率。 9. **数据整合**：当数据来自多个来源时，可能需要合并和匹配数据。这涉及到键值对齐、处理不同编码系统和解决命名冲突等问题。 10. **使用数据清洗工具**：市面上有许多数据清洗工具，如OpenRefine、Trifacta、DataWrangler等，可以帮助简化数据清洗过程，提高工作效率。在进行大数据清洗时，需要注意数据隐私和合规性问题，遵循相关的法律法规，确保数据处理的合法性和道德性。同时，数据清洗并非一次性工作，而是需要随着数据的更新和分析需求的变化持续进行的过程。通过有效的数据清洗，我们可以提高数据质量，从而得出更准确的洞察和决策。Megan Squire的《Clean Data》一书提供了更多关于数据清洗的实践策略和技术，是深入理解这个主题的宝贵资源。"

[ vii ]

Preface

"Pray, Mr. Babbage, if you put into the machine the wrong gures, will the right

answer come out?"

– Charles Babbage (1864)

"Garbage in, garbage out"

– The United States Internal Revenue Service (1963)

"There are no clean datasets."

– Josh Sullivan, Booz Allen VP in Fortune (2015)

In his 1864 collection of essays, Charles Babbage, the inventor of the rst calculating

machine, recollects being dumbfounded at the "confusion of ideas" that would

prompt someone to assume that a computer could calculate the correct answer

despite being given the wrong input. Fast-forward another 100 years, and the

tax bureaucracy started patiently explaining "garbage in, garbage out" to express

the idea that even for the all-powerful tax collector, computer processing is still

dependent on the quality of its input. Fast-forward another 50 years to 2015: a

seemingly magical age of machine learning, autocorrect, anticipatory interfaces, and

recommendation systems that know me better than I know myself. Yet, all of these

helpful algorithms still require high-quality data in order to learn properly in the

rst place, and we lament "there are no clean datasets".

Preface

[ viii ]

This book is for anyone who works with data on a regular basis, whether as a data

scientist, data journalist, software developer, or something else. The goal is to teach

practical strategies to quickly and easily bridge the gap between the data we want

and the data we have. We want high-quality, perfect data, but the reality is that most

often, our data falls far short. Whether we are plagued with missing data, data in

the wrong format, data in the wrong location, or anomalies in the data, the result is

often, to paraphrase rapper Notorious B.I.G., "more data, more problems".

Throughout the book, we will envision data cleaning as an important, worthwhile

step in the data science process: easily improved, never ignored. Our goal is to

reframe data cleaning away from being a dreaded, tedious task that we must slog

through in order to get to the real work. Instead, armed with a few tried-and-true

procedures and tools, we will learn that just like in a kitchen, if you wash your

vegetables rst, your food will look better, taste better, and be better for you. If you

learn a few proper knife skills, your meat will be more succulent and your vegetables

will be cooked more evenly. The same way that a great chef will have their favorite

knives and culinary traditions, a great data scientist will want to work with the very

best data possible and under the very best conditions.

What this book covers

Chapter 1, Why Do You Need Clean Data? motivates our quest for clean data by

showing the central role of data cleaning in the overall data science process. We

follow with a simple example showing some dirty data from a real-world dataset.

We weigh the pros and cons of each potential cleaning process, and then we describe

how to communicate our cleaning changes to others.

Chapter 2, Fundamentals – Formats, Types, and Encodings, sets up some foundational

knowledge about le formats, compression, and data types, including missing and

empty data and character encodings. Each section has its own examples taken from

real-world datasets. This chapter is important because we will rely on knowledge of

these basic concepts for the rest of the book.

Chapter 3, Workhorses of Clean Data – Spreadsheets and Text Editors, describes how to

get the most data cleaning utility out of two common tools: the text editor and the

spreadsheet. We will cover simple solutions to common problems, including how to

use functions, search and replace, and regular expressions to correct and transform

data. At the end of the chapter, we will put our skills to test using both of these tools

to clean some real-world data regarding universities.

Preface

[ ix ]

Chapter 4, Speaking the Lingua Franca – Data Conversions, focuses on converting data

from one format to another. This is one of the most important data cleaning tasks,

and it is useful to have a variety of tools at hand to easily complete this task. We

rst proceed through each of the different conversions step by step, including back

and forth between common formats such as comma-separated values (CSV), JSON,

and SQL. To put our new data conversion skills into practice, we complete a project

where we download a Facebook friend network and convert it into a few different

formats so that we can visualize its shape.

Chapter 5, Collecting and Cleaning Data from the Web, describes three different ways to

clean data found inside HTML pages. This chapter presents three popular tools to

pull data elements from within marked-up text, and it also provides the conceptual

foundation to understand other methods besides the specic tools shown here. As

our project for this chapter, we build a set of cleaning procedures to pull data from

web-based discussion forums.

Chapter 6, Cleaning Data in PDF Files, introduces several ways to meet this most

stubborn and all-too-common challenge for data cleaners: extracting data that has

been stored in Adobe's Portable Document Format (PDF) les. We rst examine

low-cost tools to accomplish this task, then we try a few low-barrier-to-entry tools,

and nally, we experiment with the Adobe non-free software itself. As always, we

use real-world data for our experiments, and this provides a wealth of experience

as we learn to work through problems as they arise.

Chapter 7, RDBMS Cleaning Techniques, uses a publicly available dataset of tweets to

demonstrate numerous strategies to clean data stored in a relational database. The

database shown is MySQL, but many of the concepts, including regular-expression

based text extraction and anomaly detection, are readily applicable to other storage

systems as well.

Chapter 8, Best Practices for Sharing Your Clean Data, describes some strategies to

make your hard work as easy for others to use as possible. Even if you never plan

to share your data with anyone else, the strategies in this chapter will help you stay

organized in your own work, saving you time in the future. This chapter describes

how to create the ideal data package in a variety of formats, how to document your

data, how to choose and attach a license to your data, and also how to publicize your

data so that it can live on if you choose.

Chapter 9, Stack Overow Project, guides you through a full-length project using a

real-world dataset. We start by posing a set of authentic questions that we can

answer about that dataset. In answering this set of questions, we will complete the

entire data science process introduced in Chapter 1, Why Do You Need Clean Data? and

we will put into practice many of the cleaning processes we learned in the previous

chapters. In addition, because this dataset is so large, we will introduce

a few new techniques to deal with the creation of test datasets.

Preface

[ x ]

Chapter 10, Twitter Project, is a full-length project that shows how to perform one of

the hottest and fastest-changing data collection and cleaning tasks out there right

now: mining Twitter. We will show how to nd and collect an existing archive

of publicly available tweets on a real-world current event while adhering to legal

restrictions on the usage of the Twitter service. We will answer a simple question

about the dataset while learning how to clean and extract data from JSON, the most

popular format in use right now with API-accessible web data. Finally, we will

design a simple data model for long-term storage of the extracted and cleaned data

and show how to generate some simple visualizations.

What you need for this book

To complete the projects in this book, you will need the following tools:

• A web browser, Internet access, and a modern operating system.

The browser and operating system should not matter, but access to a

command-line terminal window is ideal (for example, the Terminal

application in OS X). In Chapter 5, Collecting and Cleaning Data from the Web,

one of the three activities relies on a browser-based utility that runs in

the Chrome browser, so keep this in mind if you would like to complete

this activity.

• A text editor, such as Text Wrangler for Mac OSX or Notepad++ for

Windows. Some integrated development environments (IDEs, such as

Eclipse) can also be used as a text editor, but they typically have many

features you will not need.

• A spreadsheet application, such as Microsoft Excel or Google Spreadsheets.

When possible, generic examples are provided that can work on either of

these tools, but in some cases, one or the other is required.

• A Python development environment and the ability to install Python

libraries. I recommend the Enthought Canopy Python environment,

which is available here: https://www.enthought.com/products/canopy/.

• A MySQL 5.5+ server installed and running.

• A web server (running any server software) and PHP 5+ installed.

• A MySQL client interface, either the command-line interface, MySQL

Workbench, or phpMyAdmin (if you have PHP running).

Preface

[ xi ]

Who this book is for

If you are reading this book, I guess you are probably in one of two groups. One

group is the group of data scientists who already spend a lot of time cleaning data,

but you want to get better at it. You are probably frustrated with the tedium of data

cleaning, and you are looking for ways to speed it up, become more efcient, or just

use different tools to get the job done. In our kitchen metaphor, you are the chef who

just needs to brush up on a few knife skills.

The other group is made up of people doing the data science work but who never

really cared about data cleaning before. But now, you are starting to think that

maybe your results might actually get better if you had a cleaning process. Maybe

the old adage "garbage in, garbage out" is starting to feel a little too real. Maybe you

are interested in sharing your data with others, but you do not feel condent about

the quality of the datasets you are producing. With this book, you will gain enough

condence to "cook in public" by learning a few tricks and creating new habits that

will ensure a tidy, clean data science environment.

Either way, this book will help you reframe data cleaning away from being a symbol

of drudgery and toward being your hallmark of quality, good taste, style, and

efciency. You should probably have a bit of programming background, but you do

not have to be great at it. As with most data science projects, a willingness to learn

and experiment as well as a healthy sense of curiosity and a keen attention to detail

are all very important and valued.

Conventions

In this book, you will nd a number of text styles that distinguish between different

kinds of information. Here are some examples of these styles and an explanation of

their meaning.

Code words in text, database table names, folder names, lenames, le extensions,

pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "

The issue is that

open() is not prepared to handle UTF-8 characters."

A block of code is set as follows:

for tweet in stream:

encoded_tweet = tweet['text'].encode('ascii','ignore')

print counter, "-", encoded_tweet[0:10]

f.write(encoded_tweet)

剩余271页未读，继续阅读

weixin_42092222

粉丝: 0
资源: 37

大数据清洗策略与实战技巧

蘑菇ROM助手 v18.0.1703.02.zip

大数据处理技术网页数据清洗及分词

大数据分析--数据清洗和准备

大数据HDFS中数据备份策略与机制

旅游大数据中的数据挖掘与知识发现技术

数据预处理技术在大数据计算中的重要性

大数据HDFS中数据一致性模型与实现

数据索引技术在大数据搜索中的作用与优化

大数据时代放弃数据的精确性是什么意思

大数据中向一张表中增量插入数据全量插入数据和存量插入数据区别

最新资源