Kylo 数据湖平台安装与配置指南 v0.8.3

需积分: 9 64 浏览量更新于2024-07-19 收藏 8.12MB PDF 举报

"Kylo v0.8.3 文档是一个详细指南，由ThinkBig（现为Teradata公司）发布，涵盖了关于Kylo平台的各个方面，包括其特性、常见问题解答、术语定义、版本更新、下载信息、安装步骤、配置、权限设置以及安全方面的内容。文档旨在帮助用户理解并部署基于Apache Hadoop和Spark的数据湖解决方案Kylo。" Kylo是一个全面的数据湖平台，它构建在Hadoop和Spark之上，提供了企业级的数据摄取、数据准备和数据发现功能。Kylo的目标是提供一个即开即用、对业务友好的数据湖解决方案，使得数据处理流程更加便捷。在Kylo v0.8.3的文档中，我们可以找到以下关键知识点： 1. **Kylo概述**：这部分可能包含Kylo的基本概念、设计理念和它如何与Hadoop和Spark生态系统集成。 2. **特性**：Kylo的特色功能，如数据治理、元数据管理、数据质量检查、工作流管理等，可能在此部分详细介绍。 3. **FAQ**：常见问题解答可以帮助用户解决在使用Kylo过程中可能遇到的问题，提高部署和使用的效率。 4. **术语**：定义了与Kylo相关的专业术语，有助于用户理解文档中的专业词汇。 5. **发布笔记**：记录了v0.8.3版本的更新内容和改进，包括新功能、修复的bug和性能优化。 6. **下载**：提供了获取Kylo软件的链接和不同组件的安装包信息。 7. **安装准备**：包括系统需求、依赖检查和安装前的准备工作。 8. **服务账户创建**：说明如何为Kylo创建和管理服务账户，这是安全操作的重要环节。 9. **离线TAR准备**：离线安装的步骤，适用于没有网络连接或需要在隔离环境中部署的情况。 10. **安装Kylo**：详细指导用户如何安装Kylo，包括基本配置和验证过程。 11. **安装额外组件**：可能包括如Ranger、Sentry这样的安全组件，或者其他的扩展功能。 12. **启用Kerberos**：Kerberos是一种安全认证协议，用于实现Hadoop环境的身份验证。 13. **额外配置**：可能涵盖更高级的设置，如内存调整、Java环境配置等。 14. **授予HDFS权限**：确保正确设置用户和角色对HDFS资源的访问权限。 15. **启动服务**：启动Kylo及其依赖服务的步骤。 16. **导入模板**：使用预定义模板加速数据处理流程。 17. **创建样本馈送**：通过示例数据来测试和验证Kylo的功能。 18. **配置验证**：检查安装和配置是否正确，确保系统能够正常运行。 19. **HDP 2.5 Kerberos/Ranger集群部署指南**：针对特定Hadoop发行版的安全配置。 20. **Yarn集群模式配置**：优化Yarn以支持Kylo在集群环境中的高效运行。 21. **Kylo Spark属性**：调整Kylo与Spark交互的配置参数，以优化性能。 22. **Postgres Metastore配置**：使用Postgres作为元数据存储的设置方法。 23. **加密配置属性**：确保敏感配置信息的安全。 24. **日志文件**：日志管理和监控的相关信息。 25. **身份验证**：涉及用户登录和权限验证的机制。 26. **Kylo Kerberos SPNEGO**：使用Kerberos进行无感知认证（SPNEGO）的设置。 27. **访问控制**：如何管理用户和角色的访问权限。 28. **Spark用户模拟配置**：允许Spark作业以其他用户的身份运行，增强安全性。 29. **启用Ranger授权**：使用Ranger进行细粒度的访问控制。 30. **启用Sentry授权**：Sentry是另一个提供Hadoop数据访问控制的服务。 31. **Kylo UI和SSL**：使用SSL加密Web界面通信，保护用户交互数据。 32. **NiFi和SSL**：NiFi是数据流管理工具，SSL配置确保其传输安全。 33. **认证**：整体认证策略和流程的概述。此文档详细阐述了Kylo的安装、配置和管理，对于想要利用Hadoop和Spark构建数据湖的企业和开发人员来说，是一份宝贵的参考资料。通过深入学习和实践，用户可以有效地利用Kylo来管理和分析大数据。

Kylo Documentation, Release 0.8.3

Tool Comparisons

Is Kylo similar to any commercial products?

Kylo has similar capabilities to Podium and Zaloni Bedrock. Kylo is an open-source option. One differentiator is

Kylo’s extensibility. Kylo provides a plug-in architecture with a variety of extensions available to developers, and the

use of NiFi templates provides incredible ﬂexibility for batch and streaming use cases.

Is Kylo’s operations dashboard similar to Cloudera Manager and Apache Ambari?

Kylo’s dashboard is feed-health centric. Health of a feed is determined by job completion status, service level agree-

ment violations, and rules that measure data quality. Kylo provides the ability to monitor feed performance and

troubleshoot issues with feed job failures.

Kylo monitors services in the cluster and external dependencies to provide a holistic view of services your data lake

depends on. Kylo provides a simple plugin for adding enterprise services to monitor. Kylo includes plugins for pulling

service status from Ambari and Cloudera Navigator. This is useful for correlating service issues with feed health

problems.

Is Kylo’s metadata server similar to Cloudera Navigator, Apache Atlas?

In some ways. Kylo is not trying to compete with these and could certainly imagine integration with these tools.

Kylo includes its own extensible metadata server. Navigator is a governance tool that comes as part of the Cloudera

Enterprise license. Among other features, it provides data lineage of your Hive SQL queries. We think this is useful but

only provides part of the picture. Kylo’s metadata framework is really the foundation of an entire data lake solution. It

captures both business and operational metadata. It tracks lineage at the feed-level. Kylo provides IT Operations with

a useful dashboard, providing the ability to track/enforce Service Level Agreements, and performance metrics. Kylo’s

REST APIs can be used to do metadata exchange with tools like Atlas and Navigator.

How does Kylo compare to traditional ETL tools like Talend, Informatica, Data

Stage?

Kylo uses Apache NiFi to orchestrate pipelines. NiFi can connect to many different sources and perform lightweight

transformations on the edge using 180+ built-in processors. Generally workload is delegated to the cluster where the

bulk of processing power is available. Kylo’s NiFi processor extensions can effectively invoke Spark, Sqoop, Hive,

and even invoke traditional ETL tools (for example: wrap 3rd party ETL jobs).

Many ETL (extract-transform-load) tools are focused on SQL transformations using their own proprietary technology.

Data warehouse style transformations tend to be focused on issues such as loading normalized relational schemas

such as a star or snowﬂake. Hadoop data patterns tend to follow ELT (extract and load raw data, then transform). In

Hadoop, source data is often stored in raw form, or ﬂat denormalized structures. Powerful transformation techniques

are available via Hadoop technologies, including Kylo’s leveraging of Spark. We don’t often see the need for expensive

and complicated ETL technologies for Hadoop.

Kylo provides a user interface for an end-user to conﬁgure new data feeds including schema, security, validation, and

cleansing. Kylo provides the ability to wrangle and prepare visual data transformations using Spark as an engine.

What is Kylo’s value-add over plain Apache NiFi?

NiFi acts as Kylo’s pipeline orchestration engine, but NiFi itself does not provide all of the tooling required for a data

lake solution. Some of Kylo’s distinct beneﬁts over vanilla NiFi and Hadoop:

10 Chapter 2. FAQ

Kylo Documentation, Release 0.8.3

• Write-once, use many times. NiFi is a powerful IT tool for designing pipelines, but most data lake feeds utilize

just a small number of unique ﬂows or “patterns”. Kylo allows IT the ﬂexibility to design and register a NiFi

template as a data processing model for feeds. This enables non-technical business users to conﬁgure dozens,

or even hundreds of new feeds through Kylo’s simple, guided stepper-UI. In other words, our UI allows users to

setup feeds without having to code them in NiFi. As long as the basic ingestion pattern is the same, there is no

need for new coding. Business users will be able to bring in new data sources, perform standard transformations,

and publish to target systems.

• Operations Dashboard UI can be used for monitoring data feeds. It provides centralized health monitoring of

feeds and related infrastructure services, Service Level Agreements, data quality metrics reporting, and alerts.

• Web modules offer key data lake features such as metadata search, data discovery, data wrangling, data browse,

and event-based feed execution (to chain together ﬂows).

• Rich metadata model with integrated governance and best practices.

• Kylo adds a set of data lake speciﬁc NiFi extensions around Data Proﬁle, Data Cleanse, Data Validate,

Merge/Dedupe, High-water. In addition, general Spark and Hive processors not yet available with vanilla NiFi.

• Pre-built templates that implement data lake best practices: Data Ingest, ILM, and Data Processing.

Scheduler

How does Kylo manage job priority?

Kylo exposes the ability to control which yarn queue a task executes on. Typically scheduling this is done through the

scheduler. There are some advanced techniques in NiFi that allow further prioritization for shared resources.

Can Kylo support complicated ETL scheduling?

Kylo supports cron-based scheduling, but also timer-based, or event-based using JMS and an internal Kylo ruleset.

NiFi embeds the Quartz.

What’s the difference between “timer” and “cron” schedule strategies?

Timer is ﬁxed interval, “every 5 minutes or 10 seconds”. Cron can be conﬁgured to do that as well, but can handle

more complex cases like “every tues at 8AM and 4PM”.

Does Kylo support 3rd party schedulers

Yes, feeds can be triggered via JMS or REST.

Does Kylo support chaining feeds? One data feed consumed by another data feed?

Kylo supports event-based triggering of feeds based on preconditions or rules. One can deﬁne rules in the UI that

determine when to run a feed, such as “run when data has been processed by feed a and feed b and wait up to an hour

before running anyway”. We support simple rules up to very complicated rules requiring use of our API.

2.7. Scheduler 11

Kylo Documentation, Release 0.8.3

Security

Does Kylo support roles?

Kylo supports the deﬁnition of roles (or groups), and the speciﬁc permissions a user with that role can perform, down

to the function level.

What authentication methods are available?

Kylo uses Spring Security. Using pluggable login-modules, it can integrate with Active Directory, Kerberos, LDAP,

or most any authentication provider. See Developer Getting Started Guide.

What security features does Kylo support?

Kylo provides plugins that integrate with Apache Ranger or Apache Sentry, depending on the distribution that you

are running. These can be used to conﬁgure feed-based security and impersonating users properly to enforce user

permissions. Kylo fully supports Kerberized clusters and built-in features, such as HDFS encryption.

Is Kylo PCI compliant?

Kylo can be conﬁgured to use TLSv1.2 for all network communication it uses internally or externally. We are testing

running NiFi repositories on encrypted disk with a client. v0.8 will include some improvements required for full PCI

compliance.

Data Ingest

What is Kylo’s standard batch ingest workﬂow?

Kylo includes a sample pipeline template that implements many best practices around data ingest, mostly utilizing

Spark. Kylo makes it very simple for a business user to conﬁgure ingest of new source ﬁles and RDMBS tables into

Hive. Data can be read from a ﬁlesystem attached to the edge node, or directly using Kylo’s sqoop processor into

Hadoop. Original data is archived into a distinct location. Small ﬁles are optionally merged and headers stripped, if

needed. Data is cleansed, standardized, and validated based on user-deﬁned policies. Invalid records are binned into

a separate table for later inspection. Valid records are inserted into a ﬁnal Hive table with options such as (append,

snapshot, merge with dedupe, upsert, etc). Target format can differ from the raw source, contain custom partitions,

and group-based security. Finally each batch of valid data is automatically proﬁled.

Does Kylo support batch and streaming?

Yes, either types of pipelines can conﬁgured with Kylo. Kylo tracks performance statistics of streaming-style feeds in

activity over units of time. Kylo tracks performance of batch feeds in jobs and steps.

Which raw formats does Kylo support?

Kylo has a pluggable architecture for adding support for new types. Currently Kylo supports delimited-text formats

(for example: csv, tab, pipe) and all Hadoop formats, such as ORC, Parquet, RCFile, AVRO, and JSON.

12 Chapter 2. FAQ

剩余350页未读，继续阅读

养喵少年

粉丝: 54
资源: 3

Kylo 数据湖平台安装与配置指南 v0.8.3

Kylo调研总结

docker-kylo:Kylo docker镜像从官方来源扩展而来

kylo-ren-app

Kylo-Accelerator:Kylo专用的回购

kylo+nifi安装教程

Kylo为企业级的数据湖赋能

Fortnite Kylo Ren HD Wallpapers-crx插件

Kylo Ren Star Wars Wallpapers New Tab-crx插件

Kylo：企业级数据湖管理平台深度解析

[ro.build.display.id]: [ES6_kylo_stage_1.43_20230604]如何使用正则表达式匹配“ES6_kylo_stage_1.43_20230604”？

最新资源