Python数据分析实战：Pandas、NumPy与IPython应用详解

需积分: 13 87 浏览量更新于2024-07-18 收藏 5.2MB PDF 举报

《Python for Data Analysis》是一本由Wes McKinney编著的权威指南，专为数据分析师提供深入学习Python在数据处理中的应用。该书针对的是第二版，出版日期为2017年9月，ISBN为978-1-491-95766-0。本书主要关注Python三大核心数据处理库：Pandas、NumPy和IPython，以及与之相关的其他科学计算库，如SciPy、scikit-learn和statsmodels。第1章概述了本书的目的和Python选择作为数据处理语言的理由。作者指出，Python凭借其简洁易读的语法、丰富的库支持和跨平台的优势，解决了数据分析师面临的“两种语言”问题，即通常需要同时掌握SQL和统计软件。此外，书中提到了几个关键Python库： 1. **NumPy**：基础的数值计算库，提供了强大的多维数组对象和数学函数。 2. **Pandas**：专为数据操作和分析设计，提供了DataFrame和Series等高效的数据结构，便于数据清洗、整理和探索。 3. **IPython**（现在更常用的是Jupyter Notebook）：交互式环境，提供了丰富的代码执行、数据可视化和文档字符串功能。 4. **SciPy**：科学计算库，包含优化、插值、信号处理等高级功能。 5. **scikit-learn**：机器学习库，提供各种监督和无监督学习算法。 6. **statsmodels**：统计模型库，专注于统计分析和预测。第1章还讨论了如何安装和配置Python环境，包括Windows、Apple OS X/macOS和GNU/Linux系统上的安装步骤，以及Python 2与Python 3版本的选择。此外，作者强调了社区支持和参与相关会议的重要性，以及如何使用代码示例和配套数据集。整本书的内容结构围绕Python语言基础、IPython和Jupyter Notebook的使用展开，通过实际案例帮助读者逐步掌握数据处理和分析的各个环节。如果你是数据分析师或对Python数据分析感兴趣，这本书将为你提供一套全面且实用的学习路径。

the computational bottlenecks, sometimes by moving the code to a lower-level lan‐

guage like C.

Solving the “Two-Language” Problem

In many organizations, it is common to research, prototype, and test new ideas using

a more specialized computing language like SAS or R and then later port those ideas

to be part of a larger production system written in, say, Java, C#, or C++. What people

are increasingly finding is that Python is a suitable language not only for doing

research and prototyping but also for building the production systems. Why main‐

tain two development environments when one will suffice? I believe that more and

more companies will go down this path, as there are often significant organizational

benefits to having both researchers and software engineers using the same set of pro‐

gramming tools.

Why Not Python?

While Python is an excellent environment for building many kinds of analytical

applications and general-purpose systems, there are a number of uses for which

Python may be less suitable.

As Python is an interpreted programming language, in general most Python code will

run substantially slower than code written in a compiled language like Java or C++.

As programmer time is often more valuable than CPU time, many are happy to make

this trade-off. However, in an application with very low latency or demanding

resource utilization requirements (e.g., a high-frequency trading system), the time

spent programming in a lower-level (but also lower-productivity) language like C++

to achieve the maximum possible performance might be time well spent.

Python can be a challenging language for building highly concurrent, multithreaded

applications, particularly applications with many CPU-bound threads. The reason for

this is that it has what is known as the global interpreter lock (GIL), a mechanism that

prevents the interpreter from executing more than one Python instruction at a time.

The technical reasons for why the GIL exists are beyond the scope of this book. While

it is true that in many big data processing applications, a cluster of computers may be

required to process a dataset in a reasonable amount of time, there are still situations

where a single-process, multithreaded system is desirable.

This is not to say that Python cannot execute truly multithreaded, parallel code.

Python C extensions that use native multithreading (in C or C++) can run code in

parallel without being impacted by the GIL, so long as they do not need to regularly

interact with Python objects.

1.2 Why Python for Data Analysis? | 3

1.3 Essential Python Libraries

For those who are less familiar with the Python data ecosystem and the libraries used

throughout the book, I will give a brief overview of some of them.

NumPy

NumPy, short for Numerical Python, has long been a cornerstone of numerical com‐

puting in Python. It provides the data structures, algorithms, and library glue needed

for most scientific applications involving numerical data in Python. NumPy contains,

among other things:

• A fast and efficient multidimensional array object ndarray

• Functions for performing element-wise computations with arrays or mathemati‐

cal operations between arrays

•

Tools for reading and writing array-based datasets to disk

• Linear algebra operations, Fourier transform, and random number generation

• A mature C API to enable Python extensions and native C or C++ code to access

NumPy’s data structures and computational facilities

Beyond the fast array-processing capabilities that NumPy adds to Python, one of its

primary uses in data analysis is as a container for data to be passed between algo‐

rithms and libraries. For numerical data, NumPy arrays are more efficient for storing

and manipulating data than the other built-in Python data structures. Also, libraries

written in a lower-level language, such as C or Fortran, can operate on the data stored

in a NumPy array without copying data into some other memory representation.

Thus, many numerical computing tools for Python either assume NumPy arrays as a

primary data structure or else target seamless interoperability with NumPy.

pandas

pandas provides high-level data structures and functions designed to make working

with structured or tabular data fast, easy, and expressive. Since its emergence in 2010,

it has helped enable Python to be a powerful and productive data analysis environ‐

ment. The primary objects in pandas that will be used in this book are the DataFrame,

a tabular, column-oriented data structure with both row and column labels, and the

Series, a one-dimensional labeled array object.

pandas blends the high-performance, array-computing ideas of NumPy with the flex‐

ible data manipulation capabilities of spreadsheets and relational databases (such as

SQL). It provides sophisticated indexing functionality to make it easy to reshape, slice

and dice, perform aggregations, and select subsets of data. Since data manipulation,

4 | Chapter 1: Preliminaries

preparation, and cleaning is such an important skill in data analysis, pandas is one of

the primary focuses of this book.

As a bit of background, I started building pandas in early 2008 during my tenure at

AQR Capital Management, a quantitative investment management firm. At the time,

I had a distinct set of requirements that were not well addressed by any single tool at

my disposal:

• Data structures with labeled axes supporting automatic or explicit data alignment

—this prevents common errors resulting from misaligned data and working with

differently indexed data coming from different sources

•

Integrated time series functionality

• The same data structures handle both time series data and non–time series data

• Arithmetic operations and reductions that preserve metadata

• Flexible handling of missing data

• Merge and other relational operations found in popular databases (SQL-based,

for example)

I wanted to be able to do all of these things in one place, preferably in a language well

suited to general-purpose software development. Python was a good candidate lan‐

guage for this, but at that time there was not an integrated set of data structures and

tools providing this functionality. As a result of having been built initially to solve

finance and business analytics problems, pandas features especially deep time series

functionality and tools well suited for working with time-indexed data generated by

business processes.

For users of the R language for statistical computing, the DataFrame name will be

familiar, as the object was named after the similar R data.frame object. Unlike

Python, data frames are built into the R programming language and its standard

library. As a result, many features found in pandas are typically either part of the R

core implementation or provided by add-on packages.

The pandas name itself is derived from panel data, an econometrics term for multidi‐

mensional structured datasets, and a play on the phrase Python data analysis itself.

matplotlib

matplotlib is the most popular Python library for producing plots and other two-

dimensional data visualizations. It was originally created by John D. Hunter and is

now maintained by a large team of developers. It is designed for creating plots suit‐

able for publication. While there are other visualization libraries available to Python

programmers, matplotlib is the most widely used and as such has generally good inte‐

1.3 Essential Python Libraries | 5

gration with the rest of the ecosystem. I think it is a safe choice as a default visualiza‐

tion tool.

IPython and Jupyter

The IPython project began in 2001 as Fernando Pérez’s side project to make a better

interactive Python interpreter. In the subsequent 16 years it has become one of the

most important tools in the modern Python data stack. While it does not provide any

computational or data analytical tools by itself, IPython is designed from the ground

up to maximize your productivity in both interactive computing and software devel‐

opment. It encourages an execute-explore workflow instead of the typical edit-compile-

run workflow of many other programming languages. It also provides easy access to

your operating system’s shell and filesystem. Since much of data analysis coding

involves exploration, trial and error, and iteration, IPython can help you get the job

done faster.

In 2014, Fernando and the IPython team announced the Jupyter project, a broader

initiative to design language-agnostic interactive computing tools. The IPython web

notebook became the Jupyter notebook, with support now for over 40 programming

languages. The IPython system can now be used as a kernel (a programming language

mode) for using Python with Jupyter.

IPython itself has become a component of the much broader Jupyter open source

project, which provides a productive environment for interactive and exploratory

computing. Its oldest and simplest “mode” is as an enhanced Python shell designed to

accelerate the writing, testing, and debugging of Python code. You can also use the

IPython system through the Jupyter Notebook, an interactive web-based code “note‐

book” offering support for dozens of programming languages. The IPython shell and

Jupyter notebooks are especially useful for data exploration and visualization.

The Jupyter notebook system also allows you to author content in Markdown and

HTML, providing you a means to create rich documents with code and text. Other

programming languages have also implemented kernels for Jupyter to enable you to

use languages other than Python in Jupyter.

For me personally, IPython is usually involved with the majority of my Python work,

including running, debugging, and testing code.

In the accompanying book materials, you will find Jupyter notebooks containing all

the code examples from each chapter.

SciPy

SciPy is a collection of packages addressing a number of different standard problem

domains in scientific computing. Here is a sampling of the packages included:

6 | Chapter 1: Preliminaries

剩余528页未读，继续阅读

LamperL

粉丝: 0
资源: 3

Python数据分析实战：Pandas、NumPy与IPython应用详解

数据分析python

python tutoral python入门指南

2048(python)

python数据分析与可视化python数据分析与可视化python数据分析与可视化python数据分析与可视化.txt

Python数据分析入门.rar_python 大数据_python 数据分析_python大数据_python数据分析_大数据

python数据分析-Python数据分析模块

Python数据分析与挖掘实战.zip_Python 数据挖掘_python_python 案例_python数据分析_挖掘实战

Python数据分析

【Python数据分析实例及教程】Python数据分析实例及教程

Python进行数据分析完整版.pdf_python_数据分析_

最新资源