大数据分析：高效规划的工具与技术

需积分: 4 71 浏览量更新于2024-07-18 收藏 28.76MB PDF 举报

"《大数据分析：有效规划的工具和技术》是一本探讨大数据分析各个方面的书籍，将涵盖工具、技术、应用、案例研究以及该领域的未来方向。本书的章节由来自知名大学和机构的研究人员、科学家和实践者贡献，旨在为读者提供有益的信息。本书是查普曼&霍尔/CRC的大数据系列之一，该系列旨在展示大数据领域的新研究成果和应用，以及正在发展的计算工具和技术。其涵盖范围广泛，包括但不限于社交网络、传感器网络、数据为中心的计算、天文学、基因组学、医疗数据分析、大规模电子商务等领域。" 在大数据分析领域，理解并利用工具和技术对于有效规划至关重要。以下是一些关键知识点： 1. **大数据定义**：大数据是指数据量巨大、类型多样、处理速度快且具有潜在价值的数据集合。它超越了传统数据处理能力的范畴，需要新型技术和方法来管理和分析。 2. **大数据工具**：Hadoop是大数据处理的基石，它包括HDFS（分布式文件系统）和MapReduce（分布式计算框架）。除此之外，还有Apache Spark，提供更快速的内存计算，适用于实时分析和流处理。其他工具如HBase（分布式数据库）、Cassandra（NoSQL数据库）、Pig（数据分析平台）和Hive（数据仓库工具）也常用于大数据环境。 3. **大数据技术**：包括数据挖掘、机器学习、深度学习和人工智能等。这些技术帮助从海量数据中提取模式、预测趋势和进行决策支持。例如，Apache Mahout和TensorFlow用于机器学习，而Apache Flink和Storm则处理实时数据流。 4. **大数据应用**：在多个行业中都有广泛应用，如金融领域的风险评估、零售业的客户行为分析、医疗保健的疾病预测、社交媒体的情感分析等。 5. **案例研究**：书中可能包含各种成功案例，如Netflix如何利用大数据推荐电影，或者亚马逊如何通过用户购物行为进行个性化营销。 6. **研究方向**：未来的研究可能涉及更高效的数据存储技术、实时分析优化、数据安全与隐私保护，以及跨领域的数据融合和集成。 7. **大数据对业务的影响**：大数据分析不仅提升了决策效率，还帮助企业发现新的商业模式，优化运营，并通过精准营销提高客户满意度。 8. **挑战与伦理**：大数据分析的同时，也需要考虑数据质量、数据治理、隐私问题以及算法公平性等挑战。通过《大数据分析：有效规划的工具和技术》，读者可以深入了解这个领域的最新进展，学习如何运用工具和技术解决实际问题，从而在大数据时代中取得竞争优势。

Challenges in Big Data

Pothireddy Venkata Lakshmi Narayana Rao,

Pothireddy Siva Abhilash, and PS Pavan Kumar

CONTENTS

Introduction .....................................................................................................................................2

Background .................................................................................................................................2

Goals and Challenges of Analyzing Big Data ........................................................................2

Paradigm Shifts ..........................................................................................................................3

Organization of This Paper ....................................................................................................... 4

Algorithms for Big Data Analytics...........................................................................................4

k-Means ...................................................................................................................................4

Classication Algorithms: k-NN .........................................................................................5

Application of Big Data: A Case Study ...................................................................................5

Economics and Finance ........................................................................................................5

Other Applications ................................................................................................................6

Salient Features of Big Data ........................................................................................................... 7

Heterogeneity..............................................................................................................................7

Noise Accumulation ..................................................................................................................8

Spurious Correlation ..................................................................................................................9

Coincidental Endogeneity ....................................................................................................... 11

Impact on Statistical Thinking ................................................................................................13

Independence Screening .........................................................................................................15

Dealing with Incidental Endogeneity ................................................................................... 16

Impact on Computing Infrastructure .................................................................................... 17

Literature Review ..........................................................................................................................19

MapReduce ...............................................................................................................................19

Cloud Computing ....................................................................................................................22

Impact on Computational Methods ......................................................................................22

First-Order Methods for Non-Smooth Optimization ..........................................................23

Dimension Reduction and Random Projection ...................................................................24

Future Perspectives and Conclusion .......................................................................................... 27

Existing Methods ......................................................................................................................27

Proposed Methods ........................................................................................................................29

Probabilistic Graphical Modeling ..........................................................................................29

Mining Twitter Data: From Content to Connections ..........................................................29

Late Work: Location-Specic Tweet Detection and Topic Summarization

inTwitter ..............................................................................................................................29

Tending to Big Data Challenges in Genome Sequencing and RNA Interaction

Prediction...................................................................................................................................30

Single-Cell Genome Sequencing .......................................................................................30

2 Big Data Analytics

Introduction

Enormous data guarantee new levels of investigative disclosure and nancial quality.

What is new about Big Data and how they vary from the conventional little or medium-

scale information? This paper outlines the open doors and difculties brought by Big Data,

with accentuation on the recognized elements of Big Data and measurable and computa-

tional technique and in addition registering engineering to manage them.

Background

We are entering the time of Big Data, a term that alludes to the blast of data now accessible.

Such a Big Data development is driven by the way that gigantic measures of high-dimensional

or unstructured information are consistently delivered and are presented in a much less

“ luxurious” format than they used to be. For instance, in genomics we have seen an enor-

mous drop in costs for sequencing of an entire genome [1]. This is likewise valid in many

different scientic areas, for example, online network examination, biomedical imaging,

high-recurrence money transactions, investigation of reconnaissance recordings, and retail

deals. The current pattern for these vast amounts of information to be delivered and stored in

an inexpensive manner is likely to keep up or even quicken in the future [2]. This pattern will

have a profound effect on science, designing, and business. For instance, logical advances are

turning out to be increasingly information driven, and specialists will increasingly consider

themselves customers of information. The monstrous measures of high-dimensional infor-

mation convey both open doors and new difculties to information examination. Substantial

measurable investigations for Big Data handling are turning out to be progressively essential.

Goals and Challenges of Analyzing Big Data

What are the purposes of violation depressed Big Data? As per Fan and Lu [3], two principal

objectives of high-dimensional information investigation are to create powerful strategies that

can precisely anticipate the future perceptions and in the meantime gain understanding into

the relationship between the elements and reactions for experimental purposes. In addition,

because of the extensive specimen size, Big Data offers an ascent to two more objectives: to

comprehend heterogeneity and shared traits across various subpopulations.

At the end of the day, Big Data gives guarantees for:

1. Investigating the shrouded structures of every subpopulation of the information,

which is generally not possible and may even be dealt with as “exceptions” when

the specimen size is small; and

2. Extricating imperative regular elements across numerous subpopulations not-

withstanding the expansive individual varieties of data.

RNA Structure and RNA–RNA Association Expectation ..............................................30

Identifying Qualitative Changes in Living Systems ...........................................................31

Acknowledgments ........................................................................................................................ 31

References ......................................................................................................................................31

Additional References for Researchers and Advanced Readers for Further Reading ........34

Key Terminology and Denitions ..............................................................................................34

3Challenges in Big Data

What are the difculties of investigating Big Data? Big Data is portrayed by high dimen-

sionality and substantial specimen size. These two elements raise three one-of-a-kind

difculties:

1. High dimensionality brings clamor gathering, spurious relationships, and coinci-

dental homogeneity;

2. High dimensionality consolidated with vast specimen size brings additional con-

siderations, for example, regarding substantial computational expense and algo-

rithmic imsiness;

3. The gigantic examples in Big Data are regularly totaled from various sources at

various times, utilizing distinctive advances. This creates issues regarding hetero-

geneity, trial varieties, and factual predispositions and obliges us to employ more

versatile and hardy methodologies.

Paradigm Shifts

To handle the troubles of Big Data, we require new quantiable derivation and computa-

tional techniques. As an example, various standard systems that perform well for moderate

test sizes don’t scale to enormous amounts of data. Basically, various truthful methodolo-

gies that perform well for low-dimensional data are going up against basic troubles in

separating high-dimensional data. To plot effective, truthful strategies for exploring and

anticipating Big Data, we need to address Big Data issues, for instance, heterogeneity, hul-

labaloo gathering, spurious connections, and fortuitous endogeneity, despite changing the

quantiable precision and computational protability.

With respect to exactness, estimation diminishment, and variable determination are crit-

ical parts in exploring high-dimensional data. We will address these disturbing, building

issues. As a case in point, in a high-dimensional portrayal, Fan and Fan [4] and Pittelkow

and Ghosh [5] reported that a standard course of action using all parts plays out no bet-

ter than any subjective guess, due to racket gathering. This induces new regularization

methods [6–10] and without question calls for self-sufciency screening [11–13]. In addi-

tion, high dimensionality presents spurious connections between responses and arbitrary

covariates, which may incite wrong truthful reasoning and false exploratory conclusions

[14]. High dimensionality also give rise to adventitious endogeneity, a wonder that various

irregular covariates may obviously be connected with the remaining tumults. The endoge-

neity makes true inclinations and causes model determination inconsistency that can lead

to wrong trial exposures [15,16]. In any case, most true techniques rely upon suspicious

exogenous suppositions that can’t be endorsed by data (see our discussion of unplanned

endogeneity region, below) [17]).

New quantiable frameworks in light of these issues are basically required. As for ef-

ciency, Big Data convinces the headway of new computational base and data stockpiling

procedures. Streamlining is consistently a mechanical assembly, not a target, for Big Data

examination. Such a perspective change has provoked colossal advances on upgrades of

speedy congurations that are versatile to handle huge data amounts with high dimen-

sionality. This fabricates cross-mediations for different elds, including bits of knowledge,

change, and applied mathematics. As a case in point, Donoho and Elad [18] showed that

the nondeterministic polynomial–time hard (NP-hard) best subset backslide can be recast

as an L1-standard rebuffed smallest-squares issue, which can be comprehended within a

point procedure.

4 Big Data Analytics

Elective gurings to animate this L1-standard rebuffed smallest-squares issues, for

instance, least edge backslide [19], edge incline dive [20], coordinate drop [21,22], and itera-

tive shrinkage-thresholding computations [23,24], are proposed. Other than limitless scale

upgrade counts, Big Data in a like manner stirs the progression of majorization–minimization

computations [25–27], “extensive-scale screening and little-scale streamlining” framework

[28], parallel guring strategies [29–31], and evaluated estimations that are versatile to tre-

mendous sample sizes.

Organization of This Paper

The next section focuses on analytics to handle the increases in Big Data [32] and outlines

the issue from the perspectives of science, urban planning, and social science. The salient

features of the Big Data portion of this chaper clear up some unique segments of Big Data

and their consequences for quantiable conclusions. Quantiable strategies that handle

these Big Data issues are discussed in the section on impact on truthful considering [33].

The impact on enrolling base section gives an outline on a exible guring base for Big

Data stockpiling and taking care of it. The section on the impact on computational meth-

ods covers the computational pieces of Big Data and introduces some recent advances.

Finally, we present our conclusions and anticipated future directions.

Algorithms for Big Data Analytics

k-Means

What does it do? k-Means implies k bunches from a group of articles arranged so that the

individuals from a gathering are more comparable. It is a prevalent bunch examination

system for investigating a data set. What is bunch investigation? Bunch investigation is a

group of calculations intended to shape gatherings such that the gathered individual data

are more comparative versus nonbunch individuals. Bunches and gatherings are synony-

mous in the realm of group examination. Is there a case for this? Certainly, assuming we

have a data set of patients. In group examination, these are called perceptions. We know

different things about every patient, like age, heartbeat, pulse, VO

(max), cholesterol, and

so forth [34]. This is a vector speaking to the patient.

You can essentially think about a vector as a rundown of numbers we consider about

the patient. This rundown can likewise be deciphered as directions in multidimensional

space. Heartbeat can be one measurement, pulse another measurement, etc.

You may ponder, given this arrangement of vectors, how would we group together

patients that have comparable age, beat, circulation strains, and so on? We need to know

the best part.

k-implies determines what number of bunches you need. k-implies can deal with the

rest. How does k-implies deal with the rest? k-implies has heaps of varieties to enhance for

specic sorts of information. At an abnormal state, they all accomplish something like this:

1. k-implies picks foci in multidimensional space to speak to each of the k groups.

These are called centroids.

2. Every patient will be nearest to one of these k centroids. They ideally will not all be

nearest to the same one, so they can shape a group around their closest centroid.

3. What we have then are k bunches, and every patient is considered an individual

from a group.

5Challenges in Big Data

4. k-implies then nds the inside for each of the k groups in light of its bunch indi-

viduals (correct, utilizing the patient vectors).

5. This focus turns into the new centroid for the bunch.

6. Since the centroid is in a better place now, patients may now be nearer to different

centroids. At the end of the day, they may change bunch enrollment.

7. Steps 2 to 6 are rehashed until the centroids change no more and the bunch enroll-

ments balance out. This is called meeting.

Is it safe to say whether this is managed or unsupervised? It depends, yet most would

group the k-implies as unsupervised. Other than determining the quantity of groups,

k-signies “takes in” the bunches all alone with no data about which group a percep-

tion has a place. k-means can be semidirected. Why use k-implies? I don’t think many

researchers will have an issue with this [35]. The key offering purpose of k-means is its

straightforwardness. Its straightforwardness means it is for the most part quicker and

more procient than other calculations, particularly over huge data sets. It shows signs of

improvement.

k-means can be utilized to prebunch an enormous data set after a more costly group

investigation on the subgroups. k-means can likewise be utilized to quickly “play” with k

and investigate whether there are disregarded examples or connections in the data set. It’s

not all smooth cruising.

Two key shortcomings of k-means are its vulnerability to anomalies and its vulnerabil-

ity to the underlying decision of centroids. One last thing to remember is that k-means

are intended to work on ceaseless information; one will have to run a few iterations to

motivate it to chip away at discrete information [36]. Where is it utilized? A huge amount

of executions for k-implies grouping are accessible online, through the programs Apache

Mahout, Julia, R, SciPy, Weka, MATLAB, and SAS.

If decision trees and clustering do not impress you, you are going to love the next

algorithm.

Classiﬁcation Algorithms: k-NN

The k-nearest neighbor (k-NN) classier is a standout among the most surely under-

stood techniques in information mining, on account of its viability and effortlessness.

Nonetheless, it does not have the versatility to oversee enormous data sets. The funda-

mental issues found for managing vast scales of information are runtimes and memory

utilization.

Application of Big Data: A Case Study

Economics and Finance

Over the previous decade, more undertakings accepted the data-driven approach

to management that was more centered around organizations, decreasing risks and

improving execution. The undertakings are executing specic data examination tasks

to accumulate, store, regulate, and separate tremendous data sets to the extent of

sources to perceive key business bits of learning that can be handled to support better

essential initiatives. As a case in point, available cash-related data sources join stock

costs, coin, and subordinate trades, trade records, high-repeat trades, unstructured

剩余413页未读，继续阅读

wang1062807258

粉丝: 13
资源: 272

大数据分析：高效规划的工具与技术

Big Data Analytics Tools and Technology for Effective Planning 无水印原版pdf

Big Data Analytics for Healthcare

大数据与云计算融合技术相关文献

Starting secondary namenodes [bigdata] bigdata: ssh: Could not resolve hostname bigdata: Name or service not known怎么解决

big data analytics with java 电子档

bin/kafka-console-consumer.sh --bootstrap-server bigdata02:9092 --topic test_ogg

hdfs://bigdata01:9000/wordcount/input 路径在哪

Could not resolve hostname bigdata1: Name or service not known

sudo chown bigdata：bigdata -R/opt/module命令报错

最新资源