Teradata QueryGrid：连接Hadoop的高效数据库解决方案（15.0版）

需积分: 2 28 浏览量更新于2024-07-19 收藏 931KB PDF 举报

QueryGrid 是 Teradata 为 Hadoop 开发的一种解决方案，旨在实现大数据处理中不同数据仓库系统之间的高效集成。在本文档中，我们关注的是 Teradata Database（Teradata 数据库）与 Hadoop 的连接，版本为 Release 15.0 B035-1185-015K，发布日期为2015年1月。这个产品是 Teradata Corporation 及其关联公司的许可产品，强调了 Teradata 在数据仓库管理和企业智能领域的专业知识。 Teradata QueryGrid 允许用户在执行 SQL 查询时利用 Hadoop 集群的强大计算能力，尤其是对于大规模数据处理任务。它利用了 Hadoop 的 MapReduce 技术，将 SQL 查询转换为可以在 Hadoop 上运行的作业，从而实现了跨平台的数据分析。通过这种方式，Teradata 用户可以利用 Hadoop 的分布式存储和并行处理优势，同时保持对熟悉和优化的 SQL 语言的访问。 Teradata Database 提供了一系列高级功能，如 Active Data Warehousing、Active Enterprise Intelligence、Aprimo Marketing Studio、Aster、BYNET、Claraview、DecisionCast、Gridscale、MyCommerce、SQL-MapReduce、Teradata Decision Experts、Teradata Labs、Teradata Service Connect、Teradata Source Experts、Web Analyst 和 Xkoto，这些都是 Teradata 数据库生态系统的重要组成部分，反映了公司在大数据处理和商业智能领域的广泛技术实力。文档还提到了一些其他知名品牌的商标，如 Adaptec 和 SCSI Select，表明这些技术可能被集成或兼容在 QueryGrid 解决方案中。AMD的 Opteron 和 Apache 的 Hadoop，包括其标志性的黄色大象图标，都表明了与开源技术的紧密联系。此外，Apple 的 Mac 和 OS X 也可能是 QueryGrid 支持的硬件平台之一。总结来说，Teradata QueryGrid 是一种关键的工具，它将 Teradata 数据库与 Hadoop 结合起来，提供了一种灵活且强大的数据处理方式，使得企业能够在处理海量数据时实现高性能查询和分析，同时保持了对传统 SQL 语言的无缝支持。这种集成有助于企业优化数据管理流程，降低成本，提高数据分析效率。

Note: e Teradata-to-Hadoop connector does not support the use of the associated

credentials.

INVOKER

A keyword that indicates that the authorization describing the trusted remote user

credentials associated with the server being dened is present in the user database at the

time that the foreign server is accessed.

You can assign only one INVOKER authorization name per database.

You can specify either a DEFINER or an INVOKER, but not both.

Note: e Teradata-to-Hadoop connector does not support the use of the associated

credentials.

authorization_name

e name given to this authorization, which records the credentials to be used when

accessing the foreign server.

Note: e Teradata-to-Hadoop connector does not support the use of the associated

credentials.

USING Option

USING introduces the global name value pairs that provide the server

denition information

or the local name value pairs that contain table operator denition information. USING must

be followed by at least one name value pair.

name ('value ')

Attributes that dene a server object are called global name value pairs. Attributes that

contain table operator denition information are called local name value pairs.

• server

e DNS host name or IP address for the Hadoop name node. is attribute is required.

• port

e port for the name node service. is attribute is required.

• hiveport

e port for Hive access. is attribute is required.

• username

e Hadoop user’s credential. is attribute is required.

No password is associated with a Hadoop user. HDFS checks for a user name for access. If

HDFS is not congured for le permissions, then the user name is optional. If the user name

is set to auto generated, then the session user is used.

• dbname

e name of the user’s database.

You can use the dbname name value pair in the server object to limit access to specic

databases. e database name in the server object overrides users' selections.

• tablename

e name of the table to be imported or exported.

You can use the tablename name value pair in the server object to limit access to specic

databases. e table name in the server object overrides users' selections.

• hosttype

e type of host, for example, hadoop.

Chapter 2 Using Teradata QueryGrid: Teradata Database-to-Hadoop Syntax

CREATE FOREIGN SERVER

16 Teradata QueryGrid: Teradata Database-to-Hadoop

• columns

A column name list or an asterisk * to indicate that all of the columns should be imported.

• partitionlter

Limits the data to transfer to a particular Hadoop partition. Applies only to Hadoop-

partitioned columns. partitionlter is the only attribute that limits the data being read at the

source.

See Using the Teradata-to-Hadoop Connector in SELECT Statements.

• transformformatting

Indicates that array list data is formatted appropriately so that it can be cast directly into a

Teradata array column type based on data type.

• hiveserver

e location of the Hive server.

• default_string_size

Default size when it is not available through a query.

• default_col_size

Default size when it is not available through a query.

Examples of typical values for some server attributes are as follows:

• for a hosttype: hadoop

• for the server: a name or an IP address

• for port: a port number such as 8080

• hadoop_properties ('<property1=value>, <property3=value1,value2>')

Sets specic properties used to interact with Hadoop. If there are multiple arguments, you

must delimit them with angle brackets. If there is only one argument, you can omit the angle

brackets. For example, the syntax for the hadoop_properties clause for a High Availability

(HA) target supports an updated syntax where multiple values can be included. In this case

the properties must be enclosed by le and right angle brackets.

No spaces are allowed within or between arguments.

e High Availability hadoop properties are dened based on the name service that is dened

on your hadoop server. For example, you would replace the MYCOMPANY_HADOOP02 in

the following SQL with your own name service ID:

hadoop_properties('

<dfs.client.use.datanode.hostname=true>

,<dfs.datanode.usedatanode.hostname=true>

,<dfs.nameservices=MYCOMPANY_HADOOP02>

,<dfs.ha.namenodes.MYCOMPANY_HADOOP02=nn1,nn2>

,<dfs.namenode.rpc-address.MYCOMPANY_HADOOP02.nn1=hdp230-2:8020>

,<dfs.namenode.rpc-address.MYCOMPANY_HADOOP02.nn2=hdp230-3:8020>

,<dfs.client.failover.proxy.provider.MYCOMPANY_HADOOP02=org.apache.

hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider>')

In most cases, you should set the dfs.client.use.datanode.hostname property to true. If you

have a setup where your TPA nodes are on one BYNET, the Hadoop cluster is on another

BYNET, and they are communicating with one another via Ethernet, then you should also set

the dfs.datanode.usedatanode.hostname property to true.

• usenativequalication ('value')

When this ag is set to true for a foreign server, Teradata Database examines the two

conditions that follow:

Chapter 2 Using Teradata QueryGrid: Teradata Database-to-Hadoop Syntax

CREATE FOREIGN SERVER

Teradata QueryGrid: Teradata Database-to-Hadoop 17

• Hive table data size is large and there are qualifying predicates on non-partitioned

Hive columns. Large is dened as having a number of splits that is larger than the

number of Teradata Database nodes.

• e queried Hive object is a view.

When either of the two conditions are met, Teradata Database constructs a Hive query from

the Hive object name, the referenced columns, and the qualifying predicates. It then creates a

Hive staging table from the constructed query and retrieves data in the staging table. e

staging table is dropped aer all data has been retrieved.

Valid values are true and false.

For queries that involve joins between two HCatalog tables, Teradata Database brings the

data into Teradata spool and joins them in the database.

e join is not pushed into Hive. For

example, the join syntax in the following query requires the manual FOREIGN TABLE

SELECT to be accomplished in Hive:

SELECT h1.c1, h2.c2 FROM h1@hadoop1, h2@hadoop1 WHERE h1.id =

h2.id ;

For information about the local name value pairs that are used with the IMPORT and

EXPORT table operators, see LOAD_FROM_HCATALOG version and

LOAD_TO_HCATALOG version.

Import Operator

DO IMPORT WITH

Associates an IMPORT table operator with a foreign server.

Export Operator

DO EXPORT WITH

Associates an EXPORT table operator with a foreign server.

Operator Option

database_name.

e name of the database that contains the operator that you want to call. For

LOAD_TO_HCATALOG_HDPversion and LOAD_FROM_HCATALOG_HDPversion,

the database_name must be SYSLIB.

table_operator

e name of the table operator to use. Teradata supplies the following table operators for

use with the Teradata-to-Hadoop connector:

• LOAD_FROM_HCATALOG_HDP1_3_2 and

LOAD_FROM_HCATALOG_HDP2_1_2

• LOAD_TO_HCATALOG_HDP1_3_2 and LOAD_TO_HCATALOG_HDP2_1_2

Note: Be sure that you use the version number in the operator name when you create a

foreign server.

For information about the table operators, see Teradata QueryGrid: Teradata Database-

to-Hadoop Operators and Stored Procedures.

Chapter 2 Using Teradata QueryGrid: Teradata Database-to-Hadoop Syntax

CREATE FOREIGN SERVER

18 Teradata QueryGrid: Teradata Database-to-Hadoop

剩余103页未读，继续阅读

li12345bukeneng

粉丝: 0
资源: 14

Teradata QueryGrid：连接Hadoop的高效数据库解决方案（15.0版）

hadoop.dll & winutils.exe For hadoop-2.8.0

Apache Flume, Distributed Log Collection for Hadoop（第二版）

Aspose for Hadoop:该项目包含Aspose for Hadoop项目的源代码。-开源

basic-spring-for-hadoop:Hadoop 与 Spring for Hadoop 的比较

gp for hadoop

hadoop.dll for hadoop2

sentiment analysis for hadoop

Elasticsearch for Hadoop

Hadoop - IBM Data Engine for Hadoop and Spark

hadoop.dll winutils.exe for hadoop3.0.0

最新资源