精通Spring Batch：批量处理与优化指南

需积分: 16 76 浏览量更新于2024-07-24 收藏 11.9MB PDF 举报

"Pro.Spring.Batch.Jul.2011" 本书深入浅出地讲解了Spring Batch框架及其在批处理中的应用。Spring Batch是Spring生态体系中的一个模块，专门用于处理批量数据处理任务，它提供了全面的批处理功能，包括作业管理、步骤控制、数据读取、处理与写入，以及测试和调优。 1. **批处理和Spring** - 批处理是一种处理大量数据的方式，通常一次性处理多条记录，而非单个处理。 - Spring框架以其模块化和灵活性，使得集成批处理功能变得简单，Spring Batch就是这样的一个模块，它为批处理提供了强大的支持。 2. **Spring Batch 101** - 这一章介绍了Spring Batch的基本概念，包括核心组件、架构和工作流程。 - 学习如何初始化Spring Batch应用，理解其依赖注入和配置机制。 3. **样本作业** - 通过实例展示了如何创建一个基本的Spring Batch作业，包括定义作业、步骤和任务执行逻辑。 4. **理解和使用作业与步骤** - 作业（Job）是Spring Batch中的一组步骤，定义了批处理的整体流程。 - 步骤（Step）是作业的基本单元，实际执行数据处理操作，可以包含读取、处理和写入等操作。 5. **作业仓库和元数据** - 作业仓库用于存储作业和步骤的运行状态及结果，确保批处理的可恢复性和事务性。 - 元数据记录了作业运行时的信息，对于监控和故障排查至关重要。 6. **运行作业** - 介绍如何启动和管理作业，包括命令行接口、程序控制以及通过Spring Boot等现代方式。 7. **读者（Readers）** - 读者组件负责从数据源读取数据，可以是数据库、文件、网络流等。 - 讲解了如何自定义和配置各种类型的读者。 8. **项处理器（Item Processors）** - 项处理器接收读者读取的数据，并进行转换或业务逻辑处理。 - 深入探讨如何实现和优化项处理器的性能。 9. **项写入器（Item Writers）** - 项写入器将处理后的数据写入目标，如数据库、文件或其他系统。 - 探讨不同类型的写入策略和并发写入技术。 10. **样本应用** - 提供了一个完整的Spring Batch应用示例，帮助读者将理论知识应用到实践中。 11. **扩展与调优** - 讨论如何根据需求扩展Spring Batch，包括并行处理、分片和分布式批处理。 - 提供了关于性能调优和资源管理的建议。 12. **测试批处理过程** - 强调了批处理过程的测试重要性，包括单元测试、集成测试和端到端测试。 - 教授如何构建测试框架以确保批处理代码的稳定性和可靠性。书中的索引方便读者快速定位所需信息，作者和审阅者的专业背景为内容的权威性提供了保证。通过学习这本书，读者能够掌握Spring Batch的核心功能，并具备构建高效、可维护的批处理系统的能力。

CHAPTER 2  SPRING BATCH 101

• org.springframework.batch.core.Step

Like the job, represents the step as

configured in the XML as well as

provides the ability to execute a step.

• org.springframework.batch.item.ItemReader<T>

A strategy interface that provides the

ability to input items.

• org.springframework.batch.item.ItemProcessor<T>

A facility to apply business logic to an

individual item as provided.

• org.springframework.batch.item.ItemWriter<T>

A strategy interface that provides the

ability to output a list of items.

One of the advantages of the way Spring has structured a job is that it decouples each step into its

own independent processor. Each step is responsible for obtaining its own data, applying the required

business logic to it, and then writing the data to the appropriate location. This decoupling provides a

number of features:

• Flexibility: The ability to alter the order of processing with nothing more than an

XML change is something many frameworks talk about yet very few deliver. Spring

Batch is one that does deliver. Thinking about the earlier bank account example.,

If you wanted to apply the debits before the credits, the only change required

would be to reorder the steps in the job XML (Chapter 4 shows an example). You

can also skip a step, execute a step conditionally based on the results of a previous

step, or even run multiple steps in parallel by doing nothing more than tweaking

the XML.

• Maintainability: With the code for each step decoupled from the steps before and

after it, steps are easy to unit-test, debug, and update with virtually no impact on

other steps. Decoupled steps also make it possible to reuse steps in multiple jobs.

As you’ll see in upcoming chapters, steps are nothing more than Spring beans and

can be reused just like any other bean in Spring.

• Scalability: Decoupling steps in a job provides a number of options to scale your

jobs. You can execute steps in parallel. You can divide the work within a step

across threads and execute the code of a single step in parallel (you see a bit more

about this later in the chapter). Any of these abilities lets you meet the scalability

needs of your business with minimum direct impact on your code.

• Reliability: By decoupling each step and each piece within a step, you can

structure jobs such that they can be restarted at a given point in the process. If a

job fails after processing 50,000 records out of 10 million in step 3 out of 7, you can

restart it right where it left off.

Job Execution

Let’s look at what happens with the components and their relationships when a job is run. Notice in

Figure 2-2 that the piece most of the components share is JobRepository. This is a datastore (in memory

or a database) that is used to persist information about the job and step executions. A JobExecution or

CHAPTER 2  SPRING BATCH 101

StepExecution is information about a single run of the job or step. You see more detail about what is in

the executions and the repository later in this chapter and in Chapter 5.

Job

JobRepository

Step

ItemReader

JobLauncher

ItemProcessor

ItemWriter

Figure 2-2. The job components and their relationships

Running a job begins with the JobLauncher. The JobLauncher verifies whether the job has been run

before by checking the JobRepository, validates the parameters being passed into the job, and, finally,

executes the job.

The processing of a job and a step are very similar. A job goes through the list of steps it has been

configured to run, executing each one. As a chunk of items completes, Spring Batch updates the

JobExecution or StepExecution in the repository with the results of the execution. A step goes through a

list of items as read in by the ItemReader. As the step processes each chunk of items, the StepExecution

in the repository is updated with where it is in the step. Things like current commit count, start and end

times, and other information are stored in the repository. When a job or step is complete, the related

execution is updated in the repository with the final status.

One of the things that changed in Spring Batch from version 1 to 2 was the addition of chunked

processing. In version 1, records were read in, processed, and written out one at a time. The issue with

this is that it doesn’t take advantage of the ability to batch-write that Java’s file and database I/O

provides (buffered writing and batch updates). In version 2 and beyond of Spring Batch, the framework

has been updated. Reading and processing is still a singular operation; there is no reason to load a large

amount of data into memory if it can’t be processed. But now, the write only occurs once a commit

count interval occurs. This allows for more performant writing of records as well as a more capable

rollback mechanism.

Parallelization

A simple batch process’s architecture consists of a single-threaded process that executes a job’s steps in

order from start to finish. However, Spring Batch provides a number of parallelization options that you

should be aware of as you move forward. (Chapter 11 covers these options in detail.) There are four

different ways to parallelize your work: dividing work via multithreaded steps, parallel execution of full

steps, remote chunking, and partitioning.

CHAPTER 2  SPRING BATCH 101

Remote Chunking

The last two approaches to parallelization allow you to spread processing across multiple JVMs. In all

cases previously, the processing was performed in a single JVM, which can seriously hinder the

scalability options. When you can scale any part of your process horizontally across multiple JVMs, the

ability to keep up with large demands increases.

The first remote-processing option is remote chunking. In this approach, input is performed using a

standard ItemReader in a master node; the input is then sent via a form of durable communication (JMS

for example) to a remote slave ItemProcessor that is configured as a message driven POJO. When the

processing is complete, the slave sends the updated item back to the master for writing. Because this

approach reads the data at the master, processes it at the slave, and then sends it back, it’s important to

note that it can be very network intensive. This approach is good for scenarios where the cost of I/O is

small compared to the actual processing.

Partitioning

The final method for parallelization within Spring Batch is partitioning, shown in Figure 2-5. Again, you

use a master/slave configuration; but this time you don’t need a durable method of communication, and

the master serves only as a controller for a collection of slave steps. In this case, each of your slave steps

is self-contained and configured the same as if it was locally deployed. The only difference is that the

slave steps receive their work from the master node instead of the job itself. When all the slaves have

completed their work, the master step is considered complete. This configuration doesn’t require

durable communication with guaranteed delivery because the JobRepository guarantees that no work is

duplicated and all work is completed—unlike the remote-chunking approach, in which the

JobRepository has no knowledge of the state of the distributed work.

Step 1

Step 2

Slave

Step 2

Slave

Step 2

Slave

Step 2

Master

Step 1

Figure 2-5. Partitioning work

Batch Administration

Any enterprise system must be able to start and stop processes, monitor their current state, and even

view results. With web applications, this is easy: in the web application, you see the results of each action

you request, and tools like Google Analytics provide various metrics on how your application is being

used and is performing.

However, in the batch world, you may have a single Java process running on a server for eight hours

with no output other than log files and the database the process is working on. This is hardly a

manageable situation. For this reason, Spring has developed a web application called Spring Batch

Admin that lets you start and stop jobs and also provides details about each job execution.

CHAPTER 2  SPRING BATCH 101

Documentation

One of the strengths of Spring Batch is that real developers wrote it who have experience developing

batch processes in various enterprises. From this experience comes not only a comprehensive

framework but also a wealth of documentation to boot. The Spring Batch web site contains one of the

best collections of documentation for an open source project I’ve ever worked with. Along with the

formal documentation, the JavaDoc is also useful for API specifics. Finally, Spring Batch provides 19

different sample jobs for you to reference as you develop your own batch applications (see Table 2-2).

Table 2-2. Sample Batch Jobs

Batch Job Description

adhocLoopJob An infinite loop used to demonstrate the exposing of elements via

JMX and the running of the job in a background thread (instead of

the main JobLauncher thread).

beanWrapperMapperSampleJob A job with two steps that is used to demonstrate the mapping of

file fields to domain objects as well as validation of file-based

input.

compositeItemWriterSampleJob A step can have only one reader and writer. The CompositeWriter

is the way around this. This sample job demonstrates how.

customerFilterJob Uses an ItemProcessor to filter out customers that aren’t valid.

This job also updates the filter count field of the step execution.

delegatingJob Using the ItemReaderAdapter, delegates the reading of input to a

configured method of a POJO.

footballJob A football statistics job. After loading two input files, one with

player data and one with game data, the job generates a selection

of summary statistics for the players and games and writes them

to the log file.

groovyJob Uses Groovy (a dynamic JVM language) to script the unzipping

and zipping of a file.

headerFooterSample Using callbacks, adds the ability to render a header and footer on

the output.

hibernateJob Spring Batch readers and writers don’t use Hibernate by default.

This job shows how to integrate Hibernate into your job.

infiniteLoopJob Just a job with an infinite loop, used to demonstrate stop and

restart scenarios

剩余497页未读，继续阅读

神码IT

粉丝: 0
资源: 1

精通Spring Batch：批量处理与优化指南

Pro Spring Batch

Apress.Pro.Spring.4th.Edition

[Pro.Spring.3(2012)].Clarence.Ho.文字版

Spring batch学习资料

spring batch简单使用示例

spring.kafka.producer.batch-size

基于spring batch，写一段示例代码，可以通过指定的param传入job和step中进行使用，step采取reader和wirter的读法

SpringBatch 面试题

Spring Batch技术相关的面试, 一般会提问哪些问题

tf.train.batch与tf.data.Dataset.batch有什么不同

最新资源