Python数据分析师必备指南：挖掘与分析

需积分: 9 68 浏览量更新于2024-07-25 1 收藏 11.58MB PDF 举报

《Python for Data Analysis》是由Wes McKinney编著的一本专著，专注于介绍如何利用Python进行数据处理和分析。本书是2013年出版的第一版，版权由Wes McKinney持有，适用于教育、商业或销售推广用途。它在O'Reilly Media公司发行，该公司位于美国加利福尼亚州塞巴斯托波尔，同时提供在线版本供读者访问。该书的核心内容覆盖了Python在数据分析领域的重要应用，包括但不限于数据清洗、数据转换、数据可视化、数据结构和算法、Pandas库的深入剖析，以及与NumPy、SciPy等其他科学计算库的集成。作者Wes McKinney以其深厚的专业背景和实践经验，为读者提供了一个全面而实用的指南，旨在帮助读者掌握如何利用Python进行高效的数据探索、处理和建模。书中强调了Python语言的易用性和灵活性，特别是其Pandas库，它是基于NumPy构建的，使得数据操作变得直观且强大。Pandas的功能涵盖了数据读取（如CSV、Excel、SQL数据库等）、数据清洗（处理缺失值、异常值）、数据重塑（如分组、聚合、透视表）和高级数据分析（如时间序列分析）等方面。此外，《Python for Data Analysis》还涉及到了数据挖掘的基本概念和技术，展示了如何通过Python实现数据预处理、特征工程、模型构建和评估。对于那些希望在大数据时代提升数据分析技能的读者，这本书不仅是一本技术手册，也是一本实践指南，提供了丰富的实例和实战案例，帮助读者逐步掌握Python在数据科学中的核心角色。在编写过程中，该书得到了Julie Steele和Meghan Blanchette两位编辑的专业支持，生产编辑Melanie Yarbrough确保了内容的流畅组织，而Teresa Exley担任了校对工作。封面设计由Karen Montgomery负责，内部设计则由David Futato呈现，插图出自Rebecca Demarest之手。本书的初版发布于2012年10月，并定期更新以反映最新的技术和工具发展。《Python for Data Analysis》是一本不可或缺的数据分析入门和进阶教材，无论你是Python新手还是经验丰富的数据科学家，都能从中获益匪浅，提升Python在实际项目中的数据处理和分析能力。

CHAPTER 1

Preliminaries

What Is This Book About?

This book is concerned with the nuts and bolts of manipulating, processing, cleaning,

and crunching data in Python. It is also a practical, modern introduction to scientific

computing in Python, tailored for data-intensive applications. This is a book about the

parts of the Python language and libraries you’ll need to effectively solve a broad set of

data analysis problems. This book is not an exposition on analytical methods using

Python as the implementation language.

When I say “data”, what am I referring to exactly? The primary focus is on structured

data, a deliberately vague term that encompasses many different common forms of

data, such as

• Multidimensional arrays (matrices)

• Tabular or spreadsheet-like data in which each column may be a different type

(string, numeric, date, or otherwise). This includes most kinds of data commonly

stored in relational databases or tab- or comma-delimited text files

• Multiple tables of data interrelated by key columns (what would be primary or

foreign keys for a SQL user)

• Evenly or unevenly spaced time series

This is by no means a complete list. Even though it may not always be obvious, a large

percentage of data sets can be transformed into a structured form that is more suitable

for analysis and modeling. If not, it may be possible to extract features from a data set

into a structured form. As an example, a collection of news articles could be processed

into a word frequency table which could then be used to perform sentiment analysis.

Most users of spreadsheet programs like Microsoft Excel, perhaps the most widely used

data analysis tool in the world, will not be strangers to these kinds of data.

Why Python for Data Analysis?

For many people (myself among them), the Python language is easy to fall in love with.

Since its first appearance in 1991, Python has become one of the most popular dynamic,

programming languages, along with Perl, Ruby, and others. Python and Ruby have

become especially popular in recent years for building websites using their numerous

web frameworks, like Rails (Ruby) and Django (Python). Such languages are often

called scripting languages as they can be used to write quick-and-dirty small programs,

or scripts. I don’t like the term “scripting language” as it carries a connotation that they

cannot be used for building mission-critical software. Among interpreted languages

Python is distinguished by its large and active scientific computing community. Adop-

tion of Python for scientific computing in both industry applications and academic

research has increased significantly since the early 2000s.

For data analysis and interactive, exploratory computing and data visualization, Python

will inevitably draw comparisons with the many other domain-specific open source

and commercial programming languages and tools in wide use, such as R, MATLAB,

SAS, Stata, and others. In recent years, Python’s improved library support (primarily

pandas) has made it a strong alternative for data manipulation tasks. Combined with

Python’s strength in general purpose programming, it is an excellent choice as a single

language for building data-centric applications.

Python as Glue

Part of Python’s success as a scientific computing platform is the ease of integrating C,

C++, and FORTRAN code. Most modern computing environments share a similar set

of legacy FORTRAN and C libraries for doing linear algebra, optimization, integration,

fast fourier transforms, and other such algorithms. The same story has held true for

many companies and national labs that have used Python to glue together 30 years’

worth of legacy software.

Most programs consist of small portions of code where most of the time is spent, with

large amounts of “glue code” that doesn’t run often. In many cases, the execution time

of the glue code is insignificant; effort is most fruitfully invested in optimizing the

computational bottlenecks, sometimes by moving the code to a lower-level language

like C.

In the last few years, the Cython project (http://cython.org) has become one of the

preferred ways of both creating fast compiled extensions for Python and also interfacing

with C and C++ code.

Solving the “Two-Language” Problem

In many organizations, it is common to research, prototype, and test new ideas using

a more domain-specific computing language like MATLAB or R then later port those

2 | Chapter 1: Preliminaries

ideas to be part of a larger production system written in, say, Java, C#, or C++. What

people are increasingly finding is that Python is a suitable language not only for doing

research and prototyping but also building the production systems, too. I believe that

more and more companies will go down this path as there are often significant organ-

izational benefits to having both scientists and technologists using the same set of pro-

grammatic tools.

Why Not Python?

While Python is an excellent environment for building computationally-intensive sci-

entific applications and building most kinds of general purpose systems, there are a

number of uses for which Python may be less suitable.

As Python is an interpreted programming language, in general most Python code will

run substantially slower than code written in a compiled language like Java or C++. As

programmer time is typically more valuable than CPU time, many are happy to make

this tradeoff. However, in an application with very low latency requirements (for ex-

ample, a high frequency trading system), the time spent programming in a lower-level,

lower-productivity language like C++ to achieve the maximum possible performance

might be time well spent.

Python is not an ideal language for highly concurrent, multithreaded applications, par-

ticularly applications with many CPU-bound threads. The reason for this is that it has

what is known as the global interpreter lock (GIL), a mechanism which prevents the

interpreter from executing more than one Python bytecode instruction at a time. The

technical reasons for why the GIL exists are beyond the scope of this book, but as of

this writing it does not seem likely that the GIL will disappear anytime soon. While it

is true that in many big data processing applications, a cluster of computers may be

required to process a data set in a reasonable amount of time, there are still situations

where a single-process, multithreaded system is desirable.

This is not to say that Python cannot execute truly multithreaded, parallel code; that

code just cannot be executed in a single Python process. As an example, the Cython

project features easy integration with OpenMP, a C framework for parallel computing,

in order to to parallelize loops and thus significantly speed up numerical algorithms.

Essential Python Libraries

For those who are less familiar with the scientific Python ecosystem and the libraries

used throughout the book, I present the following overview of each library.

Essential Python Libraries | 3

NumPy

NumPy, short for Numerical Python, is the foundational package for scientific com-

puting in Python. The majority of this book will be based on NumPy and libraries built

on top of NumPy. It provides, among other things:

• A fast and efficient multidimensional array object ndarray

• Functions for performing element-wise computations with arrays or mathematical

operations between arrays

• Tools for reading and writing array-based data sets to disk

• Linear algebra operations, Fourier transform, and random number generation

• Tools for integrating connecting C, C++, and Fortran code to Python

Beyond the fast array-processing capabilities that NumPy adds to Python, one of its

primary purposes with regards to data analysis is as the primary container for data to

be passed between algorithms. For numerical data, NumPy arrays are a much more

efficient way of storing and manipulating data than the other built-in Python data

structures. Also, libraries written in a lower-level language, such as C or Fortran, can

operate on the data stored in a NumPy array without copying any data.

pandas

pandas provides rich data structures and functions designed to make working with

structured data fast, easy, and expressive. It is, as you will see, one of the critical in-

gredients enabling Python to be a powerful and productive data analysis environment.

The primary object in pandas that will be used in this book is the DataFrame, a two-

dimensional tabular, column-oriented data structure with both row and column labels:

>>> frame

total_bill tip sex smoker day time size

1 16.99 1.01 Female No Sun Dinner 2

2 10.34 1.66 Male No Sun Dinner 3

3 21.01 3.5 Male No Sun Dinner 3

4 23.68 3.31 Male No Sun Dinner 2

5 24.59 3.61 Female No Sun Dinner 4

6 25.29 4.71 Male No Sun Dinner 4

7 8.77 2 Male No Sun Dinner 2

8 26.88 3.12 Male No Sun Dinner 4

9 15.04 1.96 Male No Sun Dinner 2

10 14.78 3.23 Male No Sun Dinner 2

pandas combines the high performance array-computing features of NumPy with the

flexible data manipulation capabilities of spreadsheets and relational databases (such

as SQL). It provides sophisticated indexing functionality to make it easy to reshape,

slice and dice, perform aggregations, and select subsets of data. pandas is the primary

tool that we will use in this book.

4 | Chapter 1: Preliminaries

For financial users, pandas features rich, high-performance time series functionality

and tools well-suited for working with financial data. In fact, I initially designed pandas

as an ideal tool for financial data analysis applications.

For users of the R language for statistical computing, the DataFrame name will be

familiar, as the object was named after the similar R data.frame object. They are not

the same, however; the functionality provided by data.frame in R is essentially a strict

subset of that provided by the pandas DataFrame. While this is a book about Python, I

will occasionally draw comparisons with R as it is one of the most widely-used open

source data analysis environments and will be familiar to many readers.

The pandas name itself is derived from panel data, an econometrics term for multidi-

mensional structured data sets, and Python data analysis itself.

matplotlib

matplotlib is the most popular Python library for producing plots and other 2D data

visualizations. It was originally created by John D. Hunter (JDH) and is now maintained

by a large team of developers. It is well-suited for creating plots suitable for publication.

It integrates well with IPython (see below), thus providing a comfortable interactive

environment for plotting and exploring data. The plots are also interactive; you can

zoom in on a section of the plot and pan around the plot using the toolbar in the plot

window.

IPython

IPython is the component in the standard scientific Python toolset that ties everything

together. It provides a robust and productive environment for interactive and explor-

atory computing. It is an enhanced Python shell designed to accelerate the writing,

testing, and debugging of Python code. It is particularly useful for interactively working

with data and visualizing data with matplotlib. IPython is usually involved with the

majority of my Python work, including running, debugging, and testing code.

Aside from the standard terminal-based IPython shell, the project also provides

• A Mathematica-like HTML notebook for connecting to IPython through a web

browser (more on this later).

• A Qt framework-based GUI console with inline plotting, multiline editing, and

syntax highlighting

• An infrastructure for interactive parallel and distributed computing

I will devote a chapter to IPython and how to get the most out of its features. I strongly

recommend using it while working through this book.

Essential Python Libraries | 5

剩余467页未读，继续阅读

wy51r

粉丝: 1
资源: 2

Python数据分析师必备指南：挖掘与分析

《Python For Data Analysis》数据集

python_for_data_analysis_2nd_chinese_version：《利用Python进行数据分析·第2版》

Python for Data Analysis-带书签目录超清文字版英文版

python for data analysis

python for data Analysis

Python for data analysis

Python For Data Analysis (2013)高清完整PDF版

python for data analysis(pdf)

Python for Data Analysis， 2nd Edition.pdf

Python for Data Analysis 第二版

最新资源