Stream Cube：流式数据多维分析架构

需积分: 9 32 浏览量更新于2024-07-31 收藏 1.8MB PDF 举报

"Stream Cube: An Architecture for Multi-Dimensional Analysis of Data Streams" 文章“Stream Cube”由韩加炜教授等多位学者共同撰写，探讨了在处理流式数据时如何构建数据立方体以进行多维分析。数据立方体是数据仓库领域中的一个重要概念，它允许用户快速查询和分析大量数据，而流式数据立方体则专门针对持续生成的、无界的数据流。文章发表于《分布式与并行数据库》(Distributed and Parallel Databases)期刊，2005年，卷18，第173-197页。DOI号为10.1007/s10619-005-3296-1，这是一篇经过同行评审的学术论文，推荐人是Ahmed Elmagarmid。在实时监控系统、电信系统和其他动态环境中，产生的流式数据量巨大且可能无限。这些数据通常包含在较低层次的抽象级别上，而大多数分析师更关心的是较高层次的动态变化，如趋势和模式。因此，有效地处理和分析流式数据立方体成为了一个挑战。 Stream Cube架构提供了一种解决这个问题的方案。它旨在支持对数据流的实时多维分析，使得用户能够在数据流动的同时进行即时的洞察和决策。该架构可能包括以下几个关键组成部分： 1. **数据摄入（Data Ingestion）**：高效地收集和摄取流式数据，确保数据的实时性。 2. **数据存储（Data Storage）**：设计适合流数据的存储结构，可能涉及到分布式存储系统，以处理高吞吐量的数据流。 3. **立方体构建（Cube Construction）**：在数据流不断更新的同时，动态地生成和维护数据立方体。这可能涉及到在线聚合算法，使得计算可以在数据到达时即时完成。 4. **索引和查询优化（Indexing and Query Optimization）**：为了快速响应多维查询，需要有效的索引结构，并优化查询处理以减少延迟。 5. **容错和可扩展性（Fault Tolerance and Scalability）**：面对潜在的数据丢失或系统故障，Stream Cube需要具备恢复机制，并随着数据量的增长而扩展。 6. **资源管理（Resource Management）**：在有限的计算和存储资源下，平衡立方体的构建速度与查询性能。 7. **实时分析（Real-time Analytics）**：提供实时的统计和分析功能，以揭示数据流中的模式和趋势。 “Stream Cube”文章关注的是如何在大数据流的背景下，通过构建和维护数据立方体来实现高效的多维度分析，这对于实时监控、预测和决策制定具有重要意义。这一研究对于处理现代社会中不断增长的实时数据流问题提供了有价值的理论和技术支持。

STREAM CUBE: AN ARCHITECTURE FOR MULTI-DIMENSIONAL ANALYSIS 177

sibling of c

iff c

and c

have identical values in all dimensions except one dimension A

where c

[A] and c

[A] have the same parent in the dimension’s domain hierarchy. A cell

which has k non-

∗

values is called a k-d cell.(Weuse“∗” to indicate “all”, i.e., the highest

level on any dimension.)

A tuple c ∈ D is called a base cell. A base cell does not have any descendant. A cell c

is an aggregated cell iff it is an ancestor of some base cell. For each aggregated cell c, its

values on the measure attributes are derived from the complete set of descendant base cells

of c. An aggregated cell c is an iceberg cell iff its measure value satisﬁes a speciﬁed iceberg

condition, such as measure ≥val

. The data cube that consists of all and only the iceberg

cells satisfying a speciﬁed iceberg condition I is called the iceberg cube of a database D

under condition I.

Notice that in stream data analysis, besides the popularly used SQL aggregate-based

measures, such as

COUNT, SUM, MAX, MIN, and AVG, regression is a useful measure.

A stream data cell compression technique LCR (linearly compressed representation) is

developed in [10] to support efﬁcient on-line regression analysis of stream data in data

cubes. The study [10] shows that for linear and multiple linear regression analysis, only a

small number of regression measures rather than the complete stream of data need to be

registered. This holds for regression on both the time dimension and the other dimensions.

Since it takes a much smaller amount of space and time to handle regression measures in

a multi-dimensional space than handling the stream data itself, it is preferable to construct

regression (-measured) cubes by computing such regression measures.

A data stream is considered as a huge volume, inﬁnite ﬂow of data records, such as Web

click streams, telephone call logs, and on-line transactions. The data is collected at the

most detailed level in a multi-dimensional space, which may represent time, location, user,

theme, and other semantic information. Due to the huge amount of data and the transient

behavior of data streams, most of the computations will scan a data stream only once.

Moreover, the direct computation of measures at the most detailed level may generate a

huge number of results but may not be able to disclose the general characteristics and

trends of data streams. Thus data stream analysis will require to consider aggregations and

analysis at multi-dimensional and multi-level space.

Our task is to support efﬁcient, high-level, on-line, multi-dimensional analysis of such

data streams in order to ﬁnd unusual (exceptional) changes of trends, according to users’

interest, based on multi-dimensional numerical measures. This may involve construction

of a data cube, if feasible, to facilitate on-line, ﬂexible analysis.

3. Architecture for on-line analysis of data streams

To facilitate on-line, multi-dimensional analysis of data streams, we propose a stream

cube

architecture with the following features: (1) tilted time frame, (2) two critical layers: a

minimal interesting layer and an observation layer, and (3) partial computation of data

cubes by popular-path cubing. The stream data cubes so constructed are much smaller than

those constructed from the raw stream data but will still be effective for multi-dimensional

stream data analysis tasks.

剩余24页未读，继续阅读

rush_lee

粉丝: 0
资源: 1

Stream Cube：流式数据多维分析架构

streams工具

Cube Attacks on Tweakable Black Box Polynomials

CRC.zip_but2u1_morentt_stm32 crc_stm32cube_stm32cube crc

VS-Cube: Analyzing Variations of Multi-dimensional Patterns over Data Streams

TTS-Cube:递归神经网络的2端到2端语音合成

流动stream_VR游戏开发_天空盒子_Skybox_高清_16K_EXR

VS-Cube：多维度数据流模式变化分析

使用STM32Cube构建嵌入式网络服务器

STM32Cube中的DMA数据传输及其应用案例

streambuff stm32cubeide

最新资源