DataStage开发指南：Java调度与性能优化秘籍

4星 · 超过85%的资源需积分: 50 195 浏览量更新于2024-07-27 收藏 9.51MB PDF 举报

"本文档是关于WebSphere® DataStage and QualityStage的Parallel Job Advanced Developer Guide的版本8.0.1，由Ascential Software Corporation和International Business Machines Corporation联合提供。这份指南主要针对Java调度DataStage的开发者，包含了关键的开发文档，旨在帮助用户深入了解和优化DataStage的设计与开发实践。第1章"Introduction"（介绍）部分，会概述DataStage在数据处理中的作用和架构，以及术语解释，让读者对整个系统有一个基础的认识。这包括但不限于对数据流图、作业（Job）、阶段（Stage）、连接器（Connectors）等概念的定义和重要性。第2章"Job design tips"（作业设计技巧）详述了如何高效地设计DataStage作业。首先，介绍了WebSphere DataStage Designer界面的操作指南，让开发者熟悉其工具特性，以便更好地进行可视化设计。章节中还重点讨论了处理大量数据的方法，包括但不限于优化数据读取和写入策略，提高性能。 "Processing large volumes of data"部分可能会提供一些性能优化建议，如使用并行处理、分块读取、缓存技术等，以减少I/O操作对系统的影响。同时，还可能提到如何利用集群资源来扩展处理能力。 "Modular development"强调模块化设计的重要性，鼓励开发者将复杂任务拆分成可复用的小模块，便于维护和测试。这有助于提高代码的可读性和可维护性。 "Designing for good performance"部分可能会涉及最佳实践，如选择合适的连接器、优化数据传输和处理逻辑，以及如何监控和调试性能瓶颈。 "Combining data"和"Sorting data"部分则分别讲述了如何有效地合并和排序数据，这两个操作在数据处理流程中至关重要，可能涉及到数据清洗和预处理的优化。 "Default and explicit type conversions"讨论了类型转换的策略，包括何时使用默认转换，何时选择显式转换，以及如何避免潜在的数据丢失或错误。最后，"Using Tran"可能是关于Transaction Manager的使用指导，确保在处理敏感数据时具有事务支持，保证数据的一致性和完整性。这份文档为DataStage开发者提供了丰富的技术细节和实用建议，旨在提升开发效率和生产环境中的性能，是Java环境下DataStage项目开发不可或缺的参考资料。"

xiv Parallel Job Advanced Developer Guide

Chapter 1. Introduction

This manual is intended for the IBM

WebSphere

DataStage

user who has mastered the basics of

parallel job design and now wants to progress further.

The manual covers the following topics:

v Job Design Tips. This chapter contains miscellaneous tips about designing parallel jobs, from use of the

WebSphere DataStage Designer interface to handling large volumes of data.

v Improving Performance. This chapter describes methods by which you can evaluate the performance of

your parallel job designs and come up with strategies for improving them.

v Link Buffering. This chapter contains an in-depth description of when and how WebSphere DataStage

buffers data within a job, and how you can change the automatic settings if required.

v Specifying Your Own Parallel Stages. This chapter describe the interface WebSphere DataStage provides

for defining your own parallel job stage types.

v Environment Variables. This chapter list all the environment variables that are available for affecting

the set up and operation of parallel jobs.

v Operators. A number of chapters describe the operators that underlie the parallel job stages.

v WebSphere DataStage Development Kit (Job Control Interfaces). This chapter lists the various interfaces

that enable you to run and control WebSphere DataStage jobs without using the WebSphere DataStage

Director client.

Terminology

Because of the technical nature of some of the descriptions in this manual, it sometimes talks about

details of the engine that drives parallel jobs. This involves the use of terms that may be unfamiliar to

ordinary parallel job users.

v Operators. These underlie the stages in a WebSphere DataStage job. A single stage may correspond to a

single operator, or a number of operators, depending on the properties you have set, and whether you

have chosen to partition or collect or sort data on the input link to a stage. At compilation, WebSphere

DataStage evaluates your job design and will sometimes optimize operators out if they are judged to

be superfluous, or insert other operators if they are needed for the logic of the job.

v OSH. This is the scripting language used internally by the WebSphere DataStage parallel engine.

v Players. Players are the workhorse processes in a parallel job. There is generally a player for each

operator on each node. Players are the children of section leaders; there is one section leader per

processing node. Section leaders are started by the conductor process running on the conductor node

(the conductor node is defined in the configuration file).

Chapter 2. Job design tips

This chapter gives some hints and tips for the good design of parallel jobs.

WebSphere DataStage Designer interface

The following are some tips for smooth use of the WebSphere DataStage Designer when actually laying

out your job on the canvas.

v To re-arrange an existing job design, or insert new stage types into an existing job flow, first disconnect

the links from the stage to be changed, then the links will retain any meta data associated with them.

v A Lookup stage can only have one input stream, one output stream, and, optionally, one reject stream.

Depending on the type of lookup, it can have several reference links. To change the use of particular

Lookup links in an existing job flow, disconnect the links from the Lookup stage and then right-click to

change the link type, for example, Stream to Reference.

v The Copy stage is a good placeholder between stages if you anticipate that new stages or logic will be

needed in the future without damaging existing properties and derivations. When inserting a new

stage, simply drag the input and output links from the Copy placeholder to the new stage. Unless the

Force property is set in the Copy stage, WebSphere DataStage optimizes the actual copy out at runtime.

Processing large volumes of data

The ability to process large volumes of data in a short period of time depends on all aspects of the flow

and the environment being optimized for maximum throughput and performance. Performance tuning

and optimization are iterative processes that begin with job design and unit tests, proceed through

integration and volume testing, and continue throughout the production life cycle of the application. Here

are some performance pointers:

v When writing intermediate results that will only be shared between parallel jobs, always write to

persistent data sets (using Data Set stages). You should ensure that the data is partitioned, and that the

partitions, and sort order, are retained at every stage. Avoid format conversion or serial I/O.

v Data Set stages should be used to create restart points in the event that a job or sequence needs to be

rerun. But, because data sets are platform and configuration specific, they should not be used for

long-term backup and recovery of source data.

v Depending on available system resources, it may be possible to optimize overall processing time at run

time by allowing smaller jobs to run concurrently. However, care must be taken to plan for scenarios

when source files arrive later than expected, or need to be reprocessed in the event of a failure.

v Parallel configuration files allow the degree of parallelism and resources used by parallel jobs to be set

dynamically at runtime. Multiple configuration files should be used to optimize overall throughput

and to match job characteristics to available hardware resources in development, test, and production

modes.

The proper configuration of scratch and resource disks and the underlying filesystem and physical

hardware architecture can significantly affect overall job performance.

Within clustered ETL and database environments, resource-pool naming can be used to limit processing

to specific nodes, including database nodes when appropriate.

Modular development

You should aim to use modular development techniques in your job designs in order to maximize the

reuse of parallel jobs and components and save yourself time.

v Use job parameters in your design and supply values at run time. This allows a single job design to

process different data in different circumstances, rather than producing multiple copies of the same job

with slightly different arguments.

v Using job parameters allows you to exploit the WebSphere DataStage Director’s multiple invocation

capability. You can run several invocations of a job at the same time with different runtime arguments.

v Use shared containers to share common logic across a number of jobs. Remember that shared

containers are inserted when a job is compiled. If the shared container is changed, the jobs using it will

need recompiling.

Designing for good performance

Here are some tips for designing good performance into your job from the outset.

Avoid unnecessary type conversions.

Be careful to use proper source data types, especially from Oracle. You can set the

OSH_PRINT_SCHEMAS environment variable to verify that runtime schemas match the job design

column definitions.

If you are using stage variables on a Transformer stage, ensure that their data types match the expected

result types.

Use Transformer stages sparingly and wisely

Do not have multiple stages where the functionality could be incorporated into a single stage, and use

other stage types to perform simple transformation operations (see ″Using Transformer Stages″ for more

guidance).

Increase sort performance where possible

Careful job design can improve the performance of sort operations, both in standalone Sort stages and in

on-link sorts specified in the Inputs page Partitioning tab of other stage types. See ″Sorting Data″ for

guidance.

Remove unneeded columns

Remove unneeded columns as early as possible within the job flow. Every additional unused column

requires additional buffer memory, which can impact performance and make each row transfer from one

stage to the next more expensive. If possible, when reading from databases, use a select list to read just

the columns required, rather than the entire table.

Avoid reading from sequential files using the Same partitioning method.

Unless you have specified more than one source file, this will result in the entire file being read into a

single partition, making the entire downstream flow run sequentially unless you explicitly repartition (see

″Using Sequential File Stages″ for more tips on using Sequential file stages).

Combining data

The two major ways of combining data in a WebSphere DataStage job are via a Lookup stage or a Join

stage. How do you decide which one to use?

Lookup and Join stages perform equivalent operations: combining two or more input data sets based on

one or more specified keys. When one unsorted input is very large or sorting is not feasible, Lookup is

preferred. When all inputs are of manageable size or are pre-sorted, Join is the preferred solution.

4 Parallel Job Advanced Developer Guide

剩余932页未读，继续阅读

chen756430226

粉丝: 0

DataStage开发指南：Java调度与性能优化秘籍

DataStage开发全攻略：新手入门必备总结

DataStage Job与Stage详解：商业智能开发必备指南

DataStage EE开发全攻略：从基础到高级应用

DataStage学习版文档V0.67

DataStage作业开发指南.pdf

IBM DataStage 中文学习指南入门必备

DataStage安装与升级指南

DataStage安装与配置实战指南

Proton-WMS数据迁移必备攻略：实现从旧系统到新平台的无缝过渡

【电磁】基于matlab GUI FDTD时域有限差分的变电站暂态电磁计算【含Matlab源码 11057期】.zip

最新资源