大数据规划与处理解析

需积分: 10 161 浏览量更新于2024-07-24 收藏 3.59MB PDF 举报

"Planning_for_Big_Data" 大数据是信息技术领域中的一个重要概念，它涉及如何处理、存储和分析海量数据的问题。大数据的特点通常概括为“5V”：Volume（大量）、Velocity（高速）、Variety（多样）、Value（低价值密度）和Veracity（真实性）。随着互联网、物联网、社交媒体和传感器网络等技术的发展，数据量呈现指数级增长，大数据的重要性也日益凸显。在大数据的获取方面，企业通常利用各种数据源，如社交媒体平台、交易系统、物联网设备、公共数据集等，来收集结构化和非结构化的数据。例如，ACIO’s Handbook to the Changing Data Landscape 提到了企业使用SQL Server等数据库系统来处理关键业务应用，同时Hadoop作为开源大数据处理框架，被广泛用于处理非结构化和半结构化数据，如日志文件、文本、图像和视频等。大数据处理需要高效的数据管理系统，如Hadoop生态系统中的HDFS（Hadoop Distributed File System）用于分布式存储，MapReduce则提供并行计算的能力。此外，还有如Spark这样的实时处理框架，以提高数据处理速度和效率。例如，Tom White的《Hadoop: The Definitive Guide》深入介绍了Hadoop的原理和应用。在大数据分析方面，企业采用各种工具和技术，如数据挖掘、机器学习、预测建模等，从大数据中提取有价值的信息。书籍如《Machine Learning for Hackers》探讨了如何将这些方法应用于实际问题。同时，使用开源工具进行数据分析变得越来越普遍，如Philipp K. Janert的《Data Analysis with Open Source Tools》介绍了如何利用Python、R等工具进行数据清洗、可视化和统计分析。大数据的实施还需要考虑数据安全、隐私保护和合规性。随着数据量的增长，如何在保障数据安全的同时，实现数据的快速访问和分析，成为了一个挑战。此外，跨地域的数据处理也需要考虑法律法规的差异，如《Nov11–13, 2013 London, England》所示的国际会议，可能会讨论这些问题。规划大数据涉及到选择合适的技术栈，设计可扩展的架构，构建数据处理流程，以及培养具备数据科学技能的团队。大数据不仅仅是一种技术趋势，更是一种改变企业决策方式和商业模式的力量，它可以帮助企业洞察市场动态，优化运营，甚至创新产品和服务。通过学习和实践，企业和个人可以更好地应对大数据带来的机遇与挑战。

tion faster, incorporating what’s learned into the next iteration. Today, what

Boyd learned in a cockpit applies to nearly everything we do.

Data-Obese, Digital-Fast

In our always-on lives we’re flooded with cheap, abundant information. We

need to capture and analyze it well, separating digital wheat from digital chaff,

identifying meaningful undercurrents while ignoring meaningless social flot-

sam. Clay Johnson argues that we need to go on an information diet, and makes

a good case for conscious consumption. In an era of information obesity, we

need to eat better. There’s a reason they call it a feed, after all.

It’s not just an overabundance of data that makes Boyd’s insights vital. In the

last 20 years, much of human interaction has shifted from atoms to bits. When

interactions become digital, they become instantaneous, interactive, and easily

copied. It’s as easy to tell the world as to tell a friend, and a day’s shopping is

reduced to a few clicks.

The move from atoms to bits reduces the coefficient of friction of entire in-

dustries to zero. Teenagers shun e-mail as too slow, opting for instant mes-

sages. The digitization of our world means that trips around the OODA loop

happen faster than ever, and continue to accelerate.

We’re drowning in data. Bits are faster than atoms. Our jungle-surplus wet-

ware can’t keep up. At least, not without Boyd’s help. In a society where every

person, tethered to their smartphone, is both a sensor and an end node, we

need better ways to observe and orient, whether we’re at home or at work,

solving the world’s problems or planning a play date. And we need to be con-

stantly deciding, acting, and experimenting, feeding what we learn back into

future behavior.

We’re entering a feedback economy.

The Big Data Supply Chain

Consider how a company collects, analyzes, and acts on data.

2 | Chapter 1: The Feedback Economy

In big data, the collection is often challenging because of the sheer volume of

information, or the speed with which it arrives, both of which demand new

approaches and architectures.

Ingesting and cleaning

Once the data is collected, it must be ingested. In traditional business intelli-

gence (BI) parlance, this is known as Extract, Transform, and Load (ETL): the

act of putting the right information into the correct tables of a database schema

and manipulating certain fields to make them easier to work with.

One of the distinguishing characteristics of big data, however, is that the data

is often unstructured. That means we don’t know the inherent schema of the

information before we start to analyze it. We may still transform the informa-

tion — replacing an IP address with the name of a city, for example, or ano-

nymizing certain fields with a one-way hash function — but we may hold onto

the original data and only define its structure as we analyze it.

Hardware

The information we’ve ingested needs to be analyzed by people and machines.

That means hardware, in the form of computing, storage, and networks. Big

data doesn’t change this, but it does change how it’s used. Virtualization, for

example, allows operators to spin up many machines temporarily, then destroy

them once the processing is over.

Cloud computing is also a boon to big data. Paying by consumption destroys

the barriers to entry that would prohibit many organizations from playing with

large datasets, because there’s no up-front investment. In many ways, big data

gives clouds something to do.

Platforms

Where big data is new is in the platforms and frameworks we create to crunch

large amounts of information quickly. One way to speed up data analysis is to

break the data into chunks that can be analyzed in parallel. Another is to build

a pipeline of processing steps, each optimized for a particular task.

Big data is often about fast results, rather than simply crunching a large amount

of information. That’s important for two reasons:

1. Much of the big data work going on today is related to user interfaces and

the web. Suggesting what books someone will enjoy, or delivering search

results, or finding the best flight, requires an answer in the time it takes a

4 | Chapter 1: The Feedback Economy

page to load. The only way to accomplish this is to spread out the task,

which is one of the reasons why Google has nearly a million servers.

2. We analyze unstructured data iteratively. As we first explore a dataset, we

don’t know which dimensions matter. What if we segment by age? Filter

by country? Sort by purchase price? Split the results by gender? This kind

of “what if” analysis is exploratory in nature, and analysts are only as

productive as their ability to explore freely. Big data may be big. But if it’s

not fast, it’s unintelligible.

Much of the hype around big data companies today is a result of the retooling

of enterprise BI. For decades, companies have relied on structured relational

databases and data warehouses — many of them can’t handle the exploration,

lack of structure, speed, and massive sizes of big data applications.

Machine learning

One way to think about big data is that it’s “more data than you can go through

by hand.” For much of the data we want to analyze today, we need a machine’s

help.

Part of that help happens at ingestion. For example, natural language process-

ing tries to read unstructured text and deduce what it means: Was this Twitter

user happy or sad? Is this call center recording good, or was the customer

angry?

Machine learning is important elsewhere in the data supply chain. When we

analyze information, we’re trying to find signal within the noise, to discern

patterns. Humans can’t find signal well by themselves. Just as astronomers use

algorithms to scan the night’s sky for signals, then verify any promising

anomalies themselves, so too can data analysts use machines to find interesting

dimensions, groupings, or patterns within the data. Machines can work at a

lower signal-to-noise ratio than people.

Human exploration

While machine learning is an important tool to the data analyst, there’s no

substitute for human eyes and ears. Displaying the data in human-readable

form is hard work, stretching the limits of multi-dimensional visualization.

While most analysts work with spreadsheets or simple query languages today,

that’s changing.

Creve Maples, an early advocate of better computer interaction, designs sys-

tems that take dozens of independent, data sources and displays them in nav-

igable 3D environments, complete with sound and other cues. Maples’ studies

The Big Data Supply Chain | 5

剩余83页未读，继续阅读

jiangmintuo

粉丝: 0
资源: 2

大数据规划与处理解析

Planning for Big Data

海量数据规划

Thinking with Data

planning for big data

TrajectoryAnalyzer() = default; /** * @brief constructor * @param planning_published_trajectory trajectory data generated by * planning module */ TrajectoryAnalyzer( const planning::ADCTrajectory *planning_published_trajectory);

Big Data Analytics Tools and Technology for Effective Planning 无水印原版pdf

规划大数据：不断变化的数据格局的CIO手册Planning for Big Data: A CIO's Handbook to the Changing Data Landscape

Big data support of urban planning and management : the experience in China

The Enterprise Big Data Lake

Big Data Fundamentals(PrenticeHall,2016)

最新资源

TrajectoryAnalyzer() = default; /** * @brief constructor * @param planning_published_trajectory trajectory data generated by * planning module / TrajectoryAnalyzer( const planning::ADCTrajectory planning_published_trajectory);