深入探索：开源ETL工具Pentaho Kettle实战

需积分: 35 138 浏览量更新于2024-07-22 1 收藏 15.68MB PDF 举报

"该资源是一本关于开源ETL工具Pentaho Kettle的深度解析书籍，名为《Pentaho Kettle Solutions: Building Open Source ETL Solutions with Pentaho Data Integration》。书中详细介绍了如何利用Pentaho Kettle构建开源ETL解决方案，由Wiley Publishing, Inc.出版。" Pentaho Kettle，也称为Kettle或PDI，是一个强大的、全面的数据集成平台，用于提取（Extract）、转换（Transform）和加载（Load）数据，是ETL过程的核心组成部分。它提供了一个图形化的用户界面（GUI），让用户无需编写代码就能设计复杂的ETL流程。Kettle的特点在于其灵活性和可扩展性，支持各种数据源，包括数据库、文件、Web服务、甚至是其他ETL工具。本书《Pentaho Kettle Solutions》详细探讨了以下关键知识点： 1. **Kettle的基础概念**：涵盖ETL的基本原理，Kettle的工作机制，以及如何通过Kettle的Job和Transformation来组织数据处理任务。 2. **Kettle的安装与配置**：介绍如何在不同操作系统上安装和设置Kettle环境，包括数据集成服务器的部署和配置。 3. **Kettle的图形化工作流设计**：详细讲解如何使用 Spoon 工具创建、编辑和执行Job和Transformation，包括步骤的添加、连接、参数配置等。 4. **数据提取与加载**：讨论如何连接到各种数据源，如关系型数据库、CSV文件、XML文档等，以及如何从这些源中提取数据和加载到目标系统。 5. **数据转换与清洗**：介绍Kettle的转换步骤，如数据类型转换、过滤、聚合、清洗、排序等，以及如何进行数据质量控制。 6. **高级特性与最佳实践**：涵盖了工作流调度、错误处理、日志记录、性能优化等方面，提供实际项目中的解决方案和案例研究。 7. **分布式与并行处理**：讲解如何利用Kettle的并行处理能力，如Pan和Kitchen的使用，以及在集群环境中运行Kettle作业。 8. **Pentaho生态系统集成**：介绍Pentaho BI Suite的其他组件，如Pentaho Report Designer、Pentaho Analysis ( Mondrian ) 和Pentaho Dashboard，展示Kettle与其他组件的协同工作。 9. **实战示例与项目**：书中包含多个实际的ETL项目案例，帮助读者将理论知识应用到实践中，提升解决实际问题的能力。通过阅读这本书，读者不仅可以深入了解Pentaho Kettle的功能和用法，还能学习到如何设计和实施高效、可靠的ETL流程，对于想要在开源ETL领域深化技能的IT专业人员来说是一份宝贵的资源。

Pentaho Kettle Solutions

Building Open Source ETL Solutions with Pentaho Data Integration

writing to databases, and it had a versatile JavaScript step that allowed you to get by

most tough problems. Most of all, it was very flexible and easy to use. This was, after

all, a business intelligence tool, not a Java project.

However, it was clear that at some point it needed to become a lot better. So help

arrived in the early years in the form of a friend, Wim De Clercq, the co-owner of ixor

(www.ixor.be) and a senior enterprise Java architect. He explained the basics of core

Java concepts such as packages and exception handling. Time was spent reading up

on design patterns such as singletons to simplify the code.

Listening to that advice meant performing massive amounts of code changes. As a

consequence, it was not unusual back then for Matt to spend weekends doing nothing

but re-factoring code in Eclipse, rewriting tens of thousands of lines of code. But, bit

by bit, over the course of many weeks and months, things kept going in the right

direction.

Kettle Gets Some Traction

These initial positive results were shared with peers, colleagues, and other senior BI

consultants to hear what they thought of Kettle. That’s how the news spread around

slowly, and that’s how in 2004, Kettle got deployed at the Flemish Traffic Center

(www.verkeerscentrum.be) where billions of rows of data had to be integrated from

thousands of data sources all over Belgium. There was no time to write new code and

no money to buy a big name ETL tool—so Kettle entered the picture. The specific

tasks that had to be performed at the traffic center led to many improvements that

could be implemented in a full time capacity for the first time. Consequently, Kettle

improved very fast in that period. For example, the database drivers improved

dramatically because now there were really diverse test cases. It was also around that

time that messages got out to the world to let people know they could download a

gratis (free of charge, not open source) copy of Kettle for their own use.

Reactions were few but mostly positive. The most interesting response came from a

nice person named Jens Bleuel in Germany who asked if it was possible to integrate

third-party software into Kettle, more specifically an SAP/R3 connector. Kettle

version 1.2 was just deployed at the Traffic Center and it would certainly be possible

to put code in there, but it didn’t have a plugin architecture. Jens’ request to integrate

existing SAP/R3 code into Kettle was the main reason to develop the plugin system,

and this became version 2.0 of Kettle. Ultimately, this effort took until the end of

2004. It was a fairly complete release with support for slowly changing dimensions,

junk dimensions, 28 steps and 13 databases. It was then that the real potential of the

tool started to show. This in turn led to the creation by Jens Bleuel of the very first

Kettle plugin, ProSAPCON, used to read data from an SAP/R3 server.

Kettle Goes Open Source

There was a lot of excitement during that period, and Matt and Jens agreed to start

promoting the sale of Kettle from the kettle.be website and via the newfound partner

Proratio (www.proratio.de), the company where Jens was working at that time.

Pentaho Kettle Solutions

Building Open Source ETL Solutions with Pentaho Data Integration

Improvements kept coming and evaluation requests and interest mounted. However,

doing development and sales for a complete ETL tool is too big a task for any single

person. Matt discovered also that working on Kettle was fun, but selling it was not.

He had to find a way to concentrate on the fun part of Kettle development. So by late

summer 2005, the decision was made to go open source. This would let Kettle sell

itself as well as attract contributions from external developers.

When the code and free downloads of version 2.2 were first published in December

2005, the response was massive. The download package that was put up on JavaForge

got downloaded around 35,000 times in the first week alone. The news spread all over

the world pretty quickly.

Given the large number of open source projects that had turned into

“abandon-ware,” it was important to build a community around Kettle as fast as

possible. That meant answering (literally) thousands of e-mails and forum posts in the

next few years. Fortunately, help quickly arrived in the form of an open source

business intelligence company called Pentaho (www.pentaho.com), which acquired

the rights to the source code and employed Matt as lead developer of Kettle. Later,

Kettle was re-branded as Pentaho Data Integration.

Help also arrived in the form of many developers, translators, reviewers, doc

writers, and thousands of bug reports without whose help Kettle would not be where it

is today.

About This Book

The beginnings of Pentaho Kettle Solutions go back to August 2009, around the time

when the first book by Roland and Jos, Pentaho Solutions, was released. Just like

anyone who has run his or her first marathon, they proclaimed to each other “never

again.” But, as they saw the enthusiastic responses to the first book, the tone of the

conversation gradually changed and soon they were saying that if there were to be

another book, it had to be about Kettle and data integration. When Bob Elliot, the

publisher of the first Pentaho book, approached them about the possibility of writing a

Kettle book, the topics and table of contents were already well underway. To their

relief (and surprise), both their spouses encouraged them to go ahead. The good news

kept coming; after consulting Matt Casters about helping out with the review process,

he offered to become one of the main authors, an offer that was gladly accepted, of

course.

The same motivation that spurred the writing of Pentaho Solutions still holds today:

There’s an ongoing and increasing interest in open source and free software solutions,

combined with a growing recognition that business intelligence (BI) solutions are

essential in measuring and improving an organization’s performance. These BI

solutions require an integrated collection of data that is prepared in such a way that it

is directly usable for analysis, reporting, and dashboarding. This is the key reason why

most BI projects start with a data integration effort, and why this book is invaluable in

assisting you with this.

Pentaho Kettle Solutions

Building Open Source ETL Solutions with Pentaho Data Integration

Over the past decade, open source variants of more and more types of software

have become commonly accepted and respected alternatives to their more costly and

less flexible proprietary counterparts. The fact that software is open source is often

mistaken for being free of cost, and although that might be true if you only look at the

license costs, a BI solution cannot (and never will) be free of cost. There are costs

associated with hardware, implementation, maintenance, training, and migration, and

when this is all added up it turns out that licenses make up only a small portion of the

total lifecycle cost of any software solution. Open source, however, is much more

than a cheaper way of acquiring software. The fact that the source code is freely

available to anyone ensures better code quality because it is more likely that bugs are

found when more people have access to the source than just the core developers. The

fact that open source software is built on open standards using standard programming

languages (mostly Java) makes it extremely flexible and extensible. And the fact that

most open source software is not tied to a particular operating system extends this

flexibility and freedom even further.

What is usually lacking, however, is a good set of documentation and manuals.

Most open source projects provide excellent quality software, but developers usually

care more about getting great software out than delivering proper documentation. And

although you can find many good sources of information about Kettle, we felt there

was a need for a single source of information to help an ETL developer on his or her

way in discovering the Kettle toolset and building robust data integration solutions.

That is exactly what this book is for—to help you to build data integration solutions

using Kettle.

Who Should Read This Book

This book is meant for anyone who wants to know how to deliver ETL solutions

using Kettle. Maybe you’re an IT manager looking for a cost-efficient ETL solution,

an IT professional looking to broaden your skill set, or a BI or data warehouse

consultant responsible for developing ETL solutions in your organization. Maybe

you’re a software developer with a lot of experience building open source solutions

but still new to the world of data integration. And maybe you’re already an

experienced ETL developer with deep knowledge of one or more of the existing

proprietary tools. In any case, we assume you have a hands-on mentality because this

is a hands-on book. We do expect some familiarity with using computers to deliver

information, installing software, and working with databases, but most of the topics

will be explained right from the start. Of course, the data integration concepts are

explained as well, but the primary focus is on how to transform these concepts into a

working solution. That is exactly why the book is called Pentaho Kettle Solutions.

What You Will Need to Use This Book

In order to use this book, you need only two things: a computer and an Internet

connection. All the software we discuss and use in this book is freely available over

the Internet for download and use. The system requirements for the computer you will

xii

Pentaho Kettle Solutions

Building Open Source ETL Solutions with Pentaho Data Integration

need are fairly moderate; in fact, any computer that is less than four years old will do

the job just fine, as long as you have at least 1 gigabyte of RAM installed and 2

gigabytes of free disk space available for downloading and installing software.

The various chapters contain URLs where you can find and download the software

being used and the accompanying installation instructions. As for Pentaho, there are,

apart from the actual source code of course, four versions of the software that you can

use:

• GA (General Availability) releases: These are the stable builds of the

software, usually not the most recent ones but surely the most reliable.

• Release candidates: The “almost ready” next versions of the software,

possibly with a few minor bugs still in them.

• Milestone releases: These are created more frequently and allow you to

work with recent versions introducing new features.

• Nightly builds: The most up-to-date versions of the software, but also the

least stable ones.

When writing this book, we mostly worked with the nightly builds that generally

precede the GA releases by three months or more. At the time of writing, the GA

version of Kettle 4.0 has just been released. This means that when you read this book,

the software used in this book will have already been put through its paces and most

initial bugs will be fixed. This allows you to work through the material using a stable,

bug-free product, and you can concentrate on building solutions, not fixing bugs.

The complete list with download options is available online at

http://wiki.pentaho.com/display/COM/Community+Edition+Downloads.

What You Will Learn from This Book

This book will teach you:

• What data integration is, and why you need it

• The concepts that form the foundation of the Kettle solution

• How to install and configure Kettle, both on a single computer and a

client/server environment

• How to build a complete end-to-end ETL solution for the MySQL Sakila

demo database

• What the 34 subsystems of ETL are and how they translate to the Kettle

toolkit

• How Kettle can be used for data extraction, cleansing and conforming,

handling dimension tables, loading fact tables, and working with OLAP cubes

• What the Kettle development lifecycle looks like

• How to take advantage of Pentaho’s Agile BI tools from within the Kettle

development environment

• How to schedule and monitor jobs and transformations

xiii

Pentaho Kettle Solutions

Building Open Source ETL Solutions with Pentaho Data Integration

• How to work with multiple developers and manage different versions of an

ETL solution

• What data lineage, impact analysis, and auditing is, and how Kettle supports

these concepts

• How to increase the performance and throughput of Kettle using partitioning,

parallelization, and dynamic clustering

• How to use complex files, web services, and web APIs

• How to use Kettle to load an enterprise data warehouse designed according to

the Data Vault principles

• How to integrate Kettle with other solutions and how to extend Kettle by

developing you own plugins

How This Book Is Organized

This book explains ETL concepts, technologies, and solutions. Rather than using a

single example, we use several scenarios to illustrate the various concepts, although

the MySQL Sakila example database is heavily used throughout the book. When the

example relies on a database, we have taken care to ensure the sample code is

compatible with the popular and ubiquitous MySQL database (version 5.1).

These samples provide the technical details necessary to understand how you can

build ETL solutions for real-world situations. The scope of these ETL solutions

ranges from the level of the departmental data mart to the enterprise data warehouse.

Part I: Getting Started

Part I of this book focuses on gaining a quick and high-level understanding of the

Kettle software, its architecture, and its capabilities. This part consists of the

following chapters:

Chapter 1: ETL Primer—Introduces the main concepts and challenges found in

data-integration projects. We explain what the difference between transaction and

analytical systems is, where ETL fits in, and how the various components of an ETL

solution work to solve data-integration problems.

Chapter 2: Kettle Concepts—Provides an overview of the design principles used

as the foundations for Kettle and the underlying architecture of the software. We

explain the basic building blocks, consisting of jobs, transformations, steps, and

hops, and how they interact with each other. You’ll also learn how Kettle interacts

with databases and how this can be influenced by setting database-specific options.

This chapter also contains a hands-on mini tutorial to quickly walk you through

Kettle’s user interface.

Chapter 3: Installation and Configuration—Explains how and where to obtain

the Kettle software and how to install it. We explain which programs make up

Kettle, and how these different programs relate to each other and to building ETL

xiv

剩余722页未读，继续阅读

yuruiqiang

粉丝: 3
资源: 17

深入探索：开源ETL工具Pentaho Kettle实战

kettle 4.0.0 jar 包

common-etl:利用spring的通用ETL框架。-开源

TASKCTL 5.1 安装与初始化.pdf

ETL工具Kettle用户手册5.0,开源etl工具kettle,Java

开源ETL工具kettle系列之常见问题

ETL工具Kettle用户手册5.0,开源etl工具kettle,Java源码.zip

开源ETL工具kettle系列之增量更新设计技巧

开源ETL工具kettle系列之增量更新设计技巧参照.pdf

开源ETL工具Kettle的相关文档

开源ETL工具kettle实战gbase8s数据迁移

最新资源