数据科学：基础知识概览

需积分: 10 26 浏览量更新于2024-07-18 2 收藏 1.62MB PDF 举报

"data science" 数据科学是当前信息技术领域中的一个重要分支，它融合了统计学、计算机科学和领域知识，旨在从海量数据中提取有价值的信息并进行预测与决策。《MIT Press Essential Knowledge》系列图书中专门有一本关于数据科学的书籍，由John D. Kelleher和Brendan Tierney撰写，提供了对这一领域的入门级但深入的理解。数据科学家的工作通常包括数据清洗、数据挖掘、机器学习、可视化和解释。在描述中提到，我们生活在一个信息过剩的时代，快速获取信息变得容易，但理解其背后的原理却是一项挑战。数据科学书籍的目标就是为非专业人士提供一个了解复杂概念的入口，通过基础知识帮助读者建立对世界的深刻理解。在数据科学中，基础概念如数据预处理、大数据处理技术（例如云计算，如Nayan Ruparelia的《Cloud Computing》）、机器学习算法（如Ethem Alpaydin的《Machine Learning: The New AI》）以及人工智能（AI）的相关应用都是核心内容。数据科学不仅仅是技术，也涉及到道德和社会影响，如隐私问题和数据伦理，以及如何在现代社会中使用数据来推动企业和社会的进步。此外，数据科学中的元数据（ Metadata，如Jeffrey Pomerantz的《Metadata》）管理是确保数据质量、可搜索性和可重用性的重要方面。而数据的可视化（如信息可视化，可能与Michael Buckland的《Information and Society》相关）则使得复杂的数据集能够被有效地理解和传达。在实际应用中，数据科学广泛应用于推荐系统、市场趋势分析、公共卫生研究、社交媒体分析等。随着物联网（IoT，如Samuel Greengard的《The Internet of Things》）的发展，数据来源变得更加多样化，数据科学家的角色也在不断演变，他们需要处理来自各种设备和传感器的实时数据流。最后，数据科学的发展还与开放访问（Open Access，如Peter Suber的《OpenAccess》）的运动密切相关，这鼓励学术成果的公开共享，促进了研究的协作和知识的传播。数据科学是连接现实世界现象和决策制定的桥梁，它通过收集、分析和解释数据，帮助我们更好地理解世界，并依据这些理解做出更明智的选择。《MIT Press Essential Knowledge Series》的《Data Science》一书正是为那些希望进入这个领域或深化理解的人提供了一个宝贵的资源。

world vastly increased the amount of data we collected. The earliest

form of writing developed in Mesopotamia around 3200 BC and was

used for commercial record keeping. This type of record keeping

captures what is known as transactional data. Transactional data

include event information such as the sale of an item, the issuing of an

invoice, the delivery of goods, credit card payment, insurance claims,

and so on. Nontransactional data, such as demographic data, also

have a long history. The earliest-known censuses took place in

pharaonic Egypt around 3000 BC. The reason that early states put so

much effort and resources into large data-collection operations was

that these states needed to raise taxes and armies, thus proving

Benjamin Franklin’s claim that there are only two things certain in

life: death and taxes.

In the past 150 years, the development of the electronic sensor, the

digitization of data, and the invention of the computer have

contributed to a massive increase in the amount of data that are

collected and stored. A milestone in data collection and storage

occurred in 1970 when Edgar F. Codd published a paper explaining

the relational data model, which was revolutionary in terms of setting

out how data were (at the time) stored, indexed, and retrieved from

databases. The relational data model enabled users to extract data

from a database using simple queries that defined what data the user

wanted without requiring the user to worry about the underlying

structure of the data or where they were physically stored. Codd’s

paper provided the foundation for modern databases and the

development of structured query language (SQL), an international

standard for defining database queries. Relational databases store data

in tables with a structure of one row per instance and one column per

attribute. This structure is ideal for storing data because it can be

decomposed into natural attributes.

Databases are the natural technology to use for storing and

retrieving structured transactional or operational data (i.e., the type of

data generated by a company’s day-to-day operations). However, as

companies have become larger and more automated, the amount and

variety of data generated by different parts of these companies have

dramatically increased. In the 1990s, companies realized that although

they were accumulating tremendous amounts of data, they were

Download from finelybook 7450911@qq.com

repeatedly running into difficulties in analyzing those data. Part of the

problem was that the data were often stored in numerous separate

databases within the one organization. Another difficulty was that

databases were optimized for storage and retrieval of data, activities

characterized by high volumes of simple operations, such as SELECT,

INSERT, UPDATE, and DELETE. In order to analyze their data, these

companies needed technology that was able to bring together and

reconcile the data from disparate databases and that facilitated more

complex analytical data operations. This business challenge led to the

development of data warehouses. In a data warehouse, data are taken

from across the organization and integrated, thereby providing a more

comprehensive data set for analysis.

Over the past couple of decades, our devices have become mobile

and networked, and many of us now spend many hours online every

day using social technologies, computer games, media platforms, and

web search engines. These changes in technology and how we live have

had a dramatic impact on the amount of data collected. It is estimated

that the amount of data collected over the five millennia since the

invention of writing up to 2003 is about 5 exabytes. Since 2013,

humans generate and store this same amount of data every day.

However, it is not only the amount of data collected that has grown

dramatically but also the variety of data. Just consider the following

list of online data sources: emails, blogs, photos, tweets, likes, shares,

web searches, video uploads, online purchases, podcasts. And if we

consider the metadata (data describing the structure and properties of

the raw data) of these events, we can begin to understand the meaning

of the term big data. Big data are often defined in terms of the three

Vs: the extreme volume of data, the variety of the data types, and the

velocity at which the data must be processed.

The advent of big data has driven the development of a range of new

database technologies. This new generation of databases is often

referred to as “NoSQL databases.” They typically have a simpler data

model than traditional relational databases. A NoSQL database stores

data as objects with attributes, using an object notation language such

as the JavaScript Object Notation (JSON). The advantage of using an

object representation of data (in contrast to a relational table-based

model) is that the set of attributes for each object is encapsulated

Download from finelybook 7450911@qq.com

within the object, which results in a flexible representation. For

example, it may be that one of the objects in the database, compared to

other objects, has only a subset of attributes. By contrast, in the

standard tabular data structure used by a relational database, all the

data points should have the same set of attributes (i.e., columns). This

flexibility in object representation is important in contexts where the

data cannot (due to variety or type) naturally be decomposed into a set

of structured attributes. For example, it can be difficult to define the

set of attributes that should be used to represent free text (such as

tweets) or images. However, although this representational flexibility

allows us to capture and store data in a variety of formats, these data

still have to be extracted into a structured format before any analysis

can be performed on them.

The existence of big data has also led to the development of new

data-processing frameworks. When you are dealing with large volumes

of data at high speeds, it can be useful from a computational and speed

perspective to distribute the data across multiple servers, process

queries by calculating partial results of a query on each server, and

then merge these results to generate the response to the query. This is

the approach taken by the MapReduce framework on Hadoop. In the

MapReduce framework, the data and queries are mapped onto (or

distributed across) multiple servers, and the partial results calculated

on each server are then reduced (merged) together.

A History of Data Analysis

Statistics is the branch of science that deals with the collection and

analysis of data. The term statistics originally referred to the collection

and analysis of data about the state, such as demographics data or

economic data. However, over time the type of data that statistical

analysis was applied to broadened so that today statistics is used to

analyze all types of data. The simplest form of statistical analysis of

data is the summarization of a data set in terms of summary

(descriptive) statistics (including measures of a central tendency, such

as the arithmetic mean, or measures of variation, such as the range).

However, in the seventeenth and eighteenth centuries the work of

people such as Gerolamo Cardano, Blaise Pascal, Jakob Bernoulli,

Abraham de Moivre, Thomas Bayes, and Richard Price laid the

Download from finelybook 7450911@qq.com

foundations of probability theory, and through the nineteenth century

many statisticians began to use probability distributions as part of

their analytic tool kit. These new developments in mathematics

enabled statisticians to move beyond descriptive statistics and to start

doing statistical learning. Pierre Simon de Laplace and Carl Friedrich

Gauss are two of the most important and famous nineteenth-century

mathematicians, and both made important contributions to statistical

learning and modern data science. Laplace took the intuitions of

Thomas Bayes and Richard Price and developed them into the first

version of what we now call Bayes’ Rule. Gauss, in his search for the

missing dwarf planet Ceres, developed the method of least squares,

which enables us to find the best model that fits a data set such that

the error in the fit minimizes the sum of squared differences between

the data points in the data set and the model. The method of least

squares provided the foundation for statistical learning methods such

as linear regression and logistic regression as well as the development

of artificial neural network models in artificial intelligence (we will

return to least squares, regression analysis, and neural networks in

chapter 4).

Between 1780 and 1820, around the same time that Laplace and

Gauss were making their contributions to statistical learning, a

Scottish engineer named William Playfair was inventing statistical

graphics and laying the foundations for modern data visualization

and exploratory data analysis. Playfair invented the line chart and

area chart for time-series data, the bar chart to illustrate comparisons

between quantities of different categories, and the pie chart to

illustrate proportions within a set. The advantage of visualizing

quantitative data is that it allows us to use our powerful visual abilities

to summarize, compare, and interpret data. Admittedly, it is difficult

to visualize large (many data points) or complex (many attributes)

data sets, but data visualization is still an important part of data

science. In particular, it is useful in helping data scientists explore and

understand the data they are working with. Visualizations can also be

useful to communicate the results of a data science project. Since

Playfair’s time, the variety of data-visualization graphics has steadily

grown, and today there is research ongoing into the development of

novel approaches to visualize large, multidimensional data sets. A

Download from finelybook 7450911@qq.com

recent development is the t-distributed stochastic neighbor

embedding (t-SNE) algorithm, which is a useful technique for

reducing high-dimensional data down to two or three dimensions,

thereby facilitating the visualization of those data.

The developments in probability theory and statistics continued into

the twentieth century. Karl Pearson developed modern hypothesis

testing, and R. A. Fisher developed statistical methods for

multivariate analysis and introduced the idea of maximum likelihood

estimate into statistical inference as a method to draw conclusions

based on the relative probability of events. The work of Alan Turing in

the Second World War led to the invention of the electronic computer,

which had a dramatic impact on statistics because it enabled much

more complex statistical calculations. Throughout the 1940s and

subsequent decades, a number of important computational models

were developed that are still widely used in data science. In 1943,

Warren McCulloch and Walter Pitts proposed the first mathematical

model of a neural network. In 1948, Claude Shannon published “A

Mathematical Theory of Communication” and by doing so founded

information theory. In 1951, Evelyn Fix and Joseph Hodges proposed

a model for discriminatory analysis (what would now be called a

classification or pattern-recognition problem) that became the basis

for modern nearest-neighbor models. These postwar developments

culminated in 1956 in the establishment of the field of artificial

intelligence at a workshop in Dartmouth College. Even at this early

stage in the development of artificial intelligence, the term machine

learning was beginning to be used to describe programs that gave a

computer the ability to learn from data. In the mid-1960s, three

important contributions to ML were made. In 1965, Nils Nilsson’s

book titled Learning Machines showed how neural networks could be

used to learn linear models for classification. The following year, 1966,

Earl B. Hunt, Janet Marin, and Philip J. Stone developed the concept-

learning system framework, which was the progenitor of an important

family of ML algorithms that induced decision-tree models from data

in a top-down fashion. Around the same time, a number of

independent researchers developed and published early versions of

the k-means clustering algorithm, now the standard algorithm used

for data (customer) segmentation.

Download from finelybook 7450911@qq.com

剩余182页未读，继续阅读

luoye_ua2

粉丝: 0
资源: 15

数据科学：基础知识概览

EssentialAlgorithms.pdf 英文原版

Data Science Programming in Python 无水印pdf

Data Science with Java: Practical Methods for Scientists and Engineers

DataScience

Datascience

datascience

dataScience

DataScience365：DataScience365

DataScience:DataScience笔记和实践

DataScience:DataScience & DataAnalysis 学习记录&整理

最新资源