大数据2.0处理系统：一项综合调查

需积分: 9 148 浏览量更新于2024-07-19 收藏 4.73MB PDF 举报

"《Big Data 2.0》是由Sherif Sakr编著的一本关于大数据处理系统的综合性调查书籍，旨在全面概述大数据处理领域的各个方面。本书不仅关注某一特定研究领域或数据类型，而是讨论了大数据系统的研究与开发的广泛议题。书中内容既有描述性也有分析性，旨在为读者提供平衡的信息。" 在大数据（Big Data）这个迅速发展的领域中，"Big Data 2.0" 指的是随着技术进步和需求变化而出现的新一代大数据处理概念。这本书深入探讨了这些变化，包括如何更有效地处理和分析海量数据，以及如何利用这些数据驱动业务洞察和创新。书中的内容可能涵盖了以下几个关键知识点： 1. **大数据定义与特征**：介绍了大数据的“4V”特性——Volume（大量）、Velocity（高速）、Variety（多样）和Veracity（真实性），并可能扩展到新出现的第五个V——Value（价值），强调数据的潜在价值。 2. **大数据生态系统**：涵盖了Hadoop、Spark等开源框架，以及NoSQL数据库，如HBase、Cassandra，还有云存储解决方案，如Amazon S3和Google Cloud Storage，以及它们在大数据处理中的角色。 3. **实时与流处理**：讨论了Apache Kafka、Storm、Flink等实时数据处理工具，如何实现实时数据分析和快速响应。 4. **大数据分析与挖掘**：涵盖了机器学习算法，如深度学习、神经网络，以及预测分析、模式识别在大数据中的应用。 5. **数据可视化**：介绍了如何通过Tableau、D3.js等工具将复杂数据转化为易于理解的图形，以支持决策制定。 6. **数据安全与隐私**：讨论了大数据处理过程中的安全挑战，包括数据加密、匿名化技术，以及GDPR等法规对数据处理的影响。 7. **大数据架构**：介绍了分布式计算、微服务架构以及容器技术（如Docker和Kubernetes）如何支撑大数据平台的构建。 8. **案例研究**：书中可能包含实际案例，展示了不同行业如何利用大数据2.0技术解决问题，实现业务转型和增长。 9. **未来趋势**：展望了大数据领域的未来发展方向，可能包括边缘计算、量子计算以及AI在大数据处理中的进一步融合。通过阅读《Big Data 2.0》，读者不仅可以了解到大数据处理的最新技术和工具，还能理解大数据在各行各业中的应用，以及如何构建和优化大数据处理系统以应对不断变化的数据环境。此外，书中的分析内容有助于读者评估不同技术的优缺点，为实际项目选择合适的大数据解决方案。

2 1 Introduction

1 TB of trade information, and about 30 billion RFID tags are created. Add to this

mix the data generated by the hundreds of millions of GPS devices sold every year,

and the more than 30 million networked sensors currently in use (and growing at

a rate faster than 30 % per year). These data volumes are expected to double every

two years over the next decade. IBM reported that we are currently producing 2.5

quintillion bytes of data every day.

IDC predicts that the worldwide volume of data

will reach 40 zettabytes by 2020

where 85 % of all these data will be of new data

types and formats including server logs and other machine-generated data, data from

sensors, social media data, and many other data sources. All of these data will enable

us to do things we were not able to before and thereby create value for the world

economy. However, clearly, many application domains are facing major challenges

on processing such massive amounts of generated data from different sources and in

various formats. Therefore, almost all scientiﬁc funding and government agencies

introduced major strategies and plans to support Big Data research and applications.

In the enterprise world, many companies continuously gather massive datasets that

store customer interactions, product sales, and results from advertising campaigns

on the Web in addition to various types of other information [3]. In practice, a

company can generate up to petabytes of information over the course of a year: Web

pages, clickstreams, blogs, social media forums, search indices, email, documents,

instant messages, text messages, consumer demographics, sensor data from active

and passive systems, and more. By many estimates, as much as 80 % of these data are

semi-structured or unstructured. In practice, it is typical that companies are always

seeking to become more nimble in their operations and more innovative with their

data analysis and decision-making processes. They are realizing that time lost in these

processes can lead to missed business opportunities. The core of the data management

challenge is for companies to gain the ability to analyze and understand Internet-scale

information just as easily as they can now analyze and understand smaller volumes

of structured information.

The Big Data term has been coined under the tremendous and explosive growth

of the world’s digital data which are generated from various sources and in differ-

ent formats. In principle, the Big Data term is commonly described by 3 V main

attributes (Fig. 1.1): the Volume attribute describes the massive amount of data that

can be billions of rows and millions of columns, the V ariety attribute represents the

variety of formats, data sources, and structures, and the V elocity attribute reﬂects

the very high speed of data generation, ingestion, and near realtime analysis. In Jan-

uary 2007, Jim Gray, a database scholar, described the Big Data phenomenon as the

Fourth Paradigm [4] and called for a paradigm shift in the computing architecture

and large-scale data processing mechanisms. The ﬁrst three paradigms were experi-

mental, theoretical, and, more recently, computational science. Gray argued that the

only way to cope with this paradigm is to develop a new generation of computing

tools to manage, visualize, and analyze the data ﬂood. According to Gray, computer

architectures have become increasingly imbalanced where the latency gap between

http://www-01.ibm.com/software/data/bigdata/what-is-big-data.html.

http://www.emc.com/about/news/press/2012/20121211-01.htm.

4 1 Introduction

with the management of hardware and software resources. In fact, the discussion in

industry and academia has taken a while for them to be able to deﬁne the roadmap for

deﬁning what cloud computing actually means [9, 10, 11]. The US National Insti-

tute of Standards and Technology (NIST) has published a deﬁnition that reﬂects the

most commonly agreed features of cloud computing. This deﬁnition describes the

cloud computing technology as: “A model for enabling convenient, on-demand net-

work access to a shared pool of conﬁgurable computing resources (e.g., networks,

servers, storage, applications, and services) that can be rapidly provisioned and

released with minimal management effort or service provider interaction.” In prin-

ciple, one of the important features provided by cloud computing technology is that

computing hardware and software capabilities are made accessible via the network

and accessed through standard mechanisms that can be supported by heterogeneous

thin or fat client platforms ( e.g., laptops, mobile phones, and PDAs). In particular,

cloud computing provides a number of advantages for hosting the deployments of

data-intensive applications such as:

• Reduced time-to-market by removing or simplifying the time-consuming hardware

provisioning, purchasing, and deployment processes

• Reduced monetary cost by following a pay-as-you-go business model

• Unlimited (virtually) computing resources and scalability by adding resources as

the workload increases

Therefore, cloud computing has been considered as a signiﬁcant step towards

achieving the long-held dream of envisioning computing as a utility [12] where

the economies of scale principles effectively help to drive the cost of computing

infrastructure down. In practice, big players of the technology companies (e.g.,

Amazon, Microsoft, Google, IBM) have been quite active in establishing their own

data centers across the world to ensure reliability by providing redundancy for their

provided infrastructure, platforms, and applications to the end users. In principle,

cloud-based services offer several advantages such as: ﬂexibility and scalability of

storage, computing and application resources, optimal utilization of infrastructure,

and reduced costs. Hence, cloud computing provides a great chance to supply stor-

age, processing, and analytics resources for Big Data applications. A recent analysis

reported that 53 % of enterprises have deployed (28 %) or plan to deploy (25 %) their

Big Data Analytics (BDA) applications in the cloud.

In cloud computing, the provider’s computing resources are pooled to serve multi-

ple consumers using a multitenant model with various virtual and physical resources

dynamically assigned and reassigned based on the demand of the application work-

load. Therefore, it achieves the sense of location independence. Examples of such

http://research.gigaom.com/2014/11/big-data-analytics-in-the-cloud-the-enterprise-wants-it-

now/.

1.2 Big Data and Cloud Computing 5

shared computing resources include storage, memory, network bandwidth, process-

ing, virtual networks, and virtual machines. In practice, one of the main principles

for the data center technology is to exploit the virtualization technology to increase

the utilization of computing resources. Hence, it supplies the main ingredients of

computing resources such as CPUs, storage, and network bandwidth as a commod-

ity at low unit cost. Therefore, users of cloud services do not need to be concerned

about the problem of resource scalability because the provided resources can be vir-

tually considered as being inﬁnite. In particular, the business model of public cloud

providers relies on the mass acquisition of IT resources that are made available to

cloud consumers via various attractive pricing models and leasing packages. This

provides applications or enterprises with the opportunity to gain access to powerful

infrastructure without the need to purchase it.

1.3 Big Data Storage Systems

In general, relational database management systems (e.g., MySQL, PostgreSQL,

SQL Server, Oracle) have been considered as the one-size-ﬁts-all solution for data

persistence and retrieval for decades. They have matured after extensive research

and development efforts and very successfully created a large market and many

solutions in different business domains. However, the ever-increasing need for scal-

ability and new application requirements have created new challenges for traditional

RDBMS [13]. In particular, we are currently witnessing a continuous increase of

user-driven and user-generated data that results in a tremendous growth in the type

and volume of data that is produced, stored, and analyzed. For example, various newer

sets of source data generation technologies such as sensor technologies, automated

trackers, global positioning systems, and monitoring devices are producing massive

datasets. In addition to the speedy data growth, data have also become increasingly

sparse and semi-structured in nature. In particular, data structures can be classiﬁed

into four main types as follows:

• Structured data: Data with a deﬁned format and structure such as CSV ﬁles,

spreadsheets, traditional relational databases, and OLAP data cubes.

• Semi-structured data: Textual data ﬁles with a ﬂexible structure that can be

parsed. The popular example of such type of data is the Extensible Markup Lan-

guage (XML) data ﬁles with its self-describing information.

• Quasi-structured data: Textual data with erratic data formats such as Web click-

stream data that may contain inconsistencies in data values and formats.

• Unstructured data: Data that have no inherent structure such as text documents,

images, PDF ﬁles, and videos.

剩余110页未读，继续阅读

eeabcde

粉丝: 0
资源: 15

大数据2.0处理系统：一项综合调查

HCIA-Big_Data_V2.0_培训教材（完整书签版）.pdf

bigdata-2:大数据 二

华为HCIA-BigData 2.0实验指南：实战大数据组件与平台

eskimo:Eskimo是最先进的大数据基础架构和管理Web控制台，用于构建，管理和操作Big Data 2.0 Analytics集群。 这是Eskimo Community Edition的git存储库

华为HCIA-Big Data V2.0培训教材

HCIA-Big Data V2.0视频.zip

华为HCIA-Big Data V2.0 LVC公开课培训.rar

华为HCIA-Big Data V2.0 LVC公开课培训视频教程【共25集】.rar

HCIP-Big Data Developer V2.0.pdf

The Big Data of Enterprise Architecture2.0

最新资源

bigdata-2:大数据二

eskimo:Eskimo是最先进的大数据基础架构和管理Web控制台，用于构建，管理和操作Big Data 2.0 Analytics集群。这是Eskimo Community Edition的git存储库