A job that validates input data

Prerequisite

As explained in How to proceed with the tutorial, it will include implementation of the jobs for A job that inputs/outputs data by accessing database and A job that inputs/outputs data by accessing a file.
However, it must be noted that the description is for the case where the implementation is added to the job which accesses the database.

Overview

Create a job that validates input data (hereafter, referred as input check).

Note that, since this section is based on TERASOLUNA Batch 5.x Development guideline, refer Input check, for details.

Background, process overview and business specifications of Description of the application to be created are given below.

Background

Some mass retail stores issue point cards for members.
Membership types include "Gold member", "Normal member" and the services are provided based on the membership types.
As a part of the service, 100 points are added for "gold members" and 10 points are added for "normal members" at the end of the month, for the members who have purchased a product during that month.

Process overview

TERASOLUNA Batch 5.x will be using in an application as a monthly batch process which adds points based on the membership type.
Implement a process which verifies validity of whether the data exceeding upper limit value of points exists in input data.

Business specifications

Business specifications are as given below.

Check that points of input data do not exceed 1,000,000 points
- When an error occurs during the check, the process terminates abnormally (No exception handling)
When the product purchasing flag is "1"(process target), points are added based on membership type
- Add 100 points when membership type is "G"(Gold member) and add 10 points when membership type is "N"(Normal member)
Product purchasing flag is updated to "0" (initial status) after adding points
Upper limit of points is 1,000,000 points
If the points exceed 1,000,000 after adding the points, they are adjusted to 1,000,000 points

Table specifications

Specifications of member information table which serve as an input and output resource are as shown below.
Since it acts as an explanation for a job which accesses the database as per Prerequisite, refer File specifications for resource specifications of input and output for a job accessing a file.

Member information table (member_info)
No	Attribute name	Column name	PK	Data type	Number of digits	Explanation
1	Member id	id		CHAR	8	Represents a fixed 8 digit number which uniquely identifies a member.
2	Membership type	type	-	CHAR	1	Membership types are shown as below. "G"(Gold member), "N"(Normal member)
3	Product purchasing flag	status	-	CHAR	1	It shows whether you have purchased a product in the month. It is updated to "1" (process target) when the product is purchased and to "0" (initial status) during monthly batch process.
4	Point	point	-	INT	7	It shows the points retained by the members. Initial value is 0.

Job overview

The process flow and process sequence are shown below to understand the overview of the job which performs input check created here.

Since it acts as an explanation for the job which accesses a database as per Prerequisite, it must be noted that it is likely to be different from the process flow and process sequence for the job which access the file.

Input check is classified into unit item check and correlated item check etc., however only unit item checks are handled here.
Use Bean Validation for unit item check. For details, refer Input check classification.

Process flow overview: Overview of process flow is shown below.

Process flow of job which verifies validity of input data

Process sequence in case of a Chunk model: Process sequence in case of a Chunk model is explained.
Since this job is explained by assuming the usage of abnormal data, the sequence diagram shows occurrence of error (abnormal end) during input check.
When the input check is successful, process sequence after input check is same as sequence diagram of database access (Refer Job overview.

In case of Chunk model, input check is performed within the timing when data is passed to ItemProcessor.

Orange object represents a class to be implemented at this time.

ProcessSequence of Validation Job by ChunkModel

Sequence diagram of Chunk model

Explanation of sequence diagram

A step is executed from the job.
Step opens the resource
MyBatisCursorItemReader fetches all the member information from member_info table (issue select statement).
- Repeat subsequent process until input data is exhausted.
- Start a framework transaction with the chunk units.
- Repeat steps from 4 to 12 until the chunk size is achieved.
Step fetches one record of input data from MyBatisCursorItemReader.
MyBatisCursorItemReader fetches one record of input data from member_info table.
member_info table returns input data to MyBatisCursorItemReader.
MyBatisCursorItemReader returns the input data to step.
Step processes input data by PointAddItemProcessor.
PointAddItemProcessor requests SpringValidator for input check.
SpringValidator performs input check based on input check rules and throws an exception (ValidationException) in case an error occurs during the check.
PointAddItemProcessor reads input data and adds points.
PointAddItemProcessor returns process results to the step.
Step outputs chunk size data by MyBatisBatchItemWriter.
MyBatisBatchItemWriter updates member information for member_info table (issue update statement).

When an exception occurs in the processes from 4 to 14, perform the subsequent process.

Step rolls back framework transaction.
Step returns an exit code (here, abnormal end: 255) to the job.

Process sequence for a Tasklet model: Process sequence for the Tasklet model is explained.
Since this job is explained assuming the usage of abnormal data, the sequence diagram shows occurrence of error (abnormal end) during input check.
When the input check is successful, process sequence after the input check is same as sequence diagram of database access (Refer Job overview).

In case of a Tasklet model, input check is performed within any timing in Tasklet#execute().
Here, it is done immediately after fetching the data.

Orange object represents a class to be implemented at this time.

ProcessSequence of Validation Job by TaskletModel

Sequence diagram of Tasklet model

Explanation of sequence diagram

A step is executed from the job.
- Step starts a framework transaction.
Step executes PointAddTasklet.
PointAddTasklet opens a resource.
MyBatisCursorItemReader fetches all the member information from member_info table (issue select statement).
- Repeat steps from 5 to 13 until the input data is exhausted.
- Repeat processes from 5 to 11 until a fixed number of records is achieved.
PointAddTasklet fetches one record of input data from MyBatisCursorItemReader.
MyBatisCursorItemReader fetches 1 record of input data from member_info table.
member_info table returns input data to MyBatisCursorItemReader.
MyBatisCursorItemReader returns input data to tasklet.
PointAddTasklet requests SpringValidator for input check.
SpringValidator performs input check based on input check rules and throws an exception (ValidationException) in case an error occurs during the check.
PointAddTasklet reads input data and adds points.
PointAddTasklet` outputs data of fixed records by MyBatisBatchItemWriter``.
MyBatisBatchItemWriter updates member information for member_info table (issue update statement).

When an exception occurs in the processes from 2 to 13, perform the subsequent process.

PointAddTasklet throws an exception (here, ValidationException)in the step.
Step rolls back the framework transaction.
Step returns an exit code (here, abnormal end: 255) to the job.

Setting for implementing input check process

Use Hibernate Validator in input check. Although it is configured already in TERASOLUNA Batch 5.x, Hibernate Validator and Bean must be defined in the dependency relation of the library.

Configuration example of dependent library (pom.xml)

<dependency>
    <groupId>org.hibernate</groupId>
    <artifactId>hibernate-validator</artifactId>
</dependency>

src/main/resources/META-INF/spring/launch-context.xml

<bean id="validator" class="org.springframework.batch.item.validator.SpringValidator"
      p:validator-ref="beanValidator"/>

<bean id="beanValidator"
      class="org.springframework.validation.beanvalidation.LocalValidatorFactoryBean" />

How to implement a Chunk model and a Tasklet model are explained subsequently.

Implementation in Chunk model
Implementation in Tasklet model

Implementation in Chunk model

Execute the job which performs input check in the Chunk model, from its creation to execution using following procedure.

Defining input check rules
Implementation of input check process
Job execution

Defining input check rules

Assign Bean Validation annotation to the field for checking DTO class in order to perform input check.
For input check annotation, refer Bean Validation check rules and Hibernate Validator check rules of TERASOLUNA Server 5.x Development Guideline.

Since it is used in common for Chunk model / Tasklet model, it can be skipped if it has already been implemented.

Here, define a check rule to check whether points exceed 1,000,000 (upper limit value).

org.terasoluna.batch.tutorial.common.dto.MemberInfoDTO

package org.terasoluna.batch.tutorial.common.dto;

import javax.validation.constraints.Max;

public class MemberInfoDto {
    private String id;

    private String type;

    private String status;

    @Max(1000000) // (1)
    private int point;

    // Getter and setter are omitted.
}

Explanation
Sr. No.	Explanation
(1)	Assign @Max annotation which indicates that the target field is less than or equal to the specified value.

Implementation of input check process

Implement input check process in business logic class which adds the points.

Add implementation of input check process to already implemented PointAddItemProcessor class.
Since it acts as an explanation for a job which accesses the database as per Prerequisite, add only (1)~(3) from below to the implementation for the job accessing the file.

org.terasoluna.batch.tutorial.dbaccess.chunk.PointAddItemProcessor

// Package and the other import are omitted.

import javax.inject.Inject;

@Component
public class PointAddItemProcessor implements ItemProcessor<MemberInfoDto, MemberInfoDto> {
    // Definition of constants are omitted.

    @Inject // (1)
    Validator<MemberInfoDto> validator; // (2)

    @Override
    public MemberInfoDto process(MemberInfoDto item) throws Exception {
        validator.validate(item); // (3)

        // The other codes of business logic are omitted.
    }
}

Explanation
Sr. No.	Explanation
(1)	Inject an instance of `SpringValidator`.
(2)	Set DTO to be fetched through `ItemReader`in the type argument of `org.springframework.batch.item.validator.Validator.
(3)	Execute `Validator#validate()` by using DTO fetched through `ItemReader` as an argument. Originally, try-catch was implemented to handle input check error and catch the exception while executing `validate()`, however, since exception handling using try-catch is explained in Job which performs exception handling by try-catch, exception handling is not implemented here.

Job execution

Execute the created job on IDE and verify the results.

Execute the job from execution configuration

Execute the job from execution configuration which is created in advance.

Here, the job is executed by using abnormal data.
Since how to change input data varies according to resource (database or file) which handles a job performing input check, execute as below.

When input check is implemented for the job which inputs or outputs data by accessing database: Execute job by using execution configuration created in Execute job from execution configuration of the job which inputs or outputs data by accessing database.

In order to use abnormal data, comment out script of normal data and remove comment out of abnormal data by Database Initialize of batch-application.properties.

src/main/resources/batch-application.properties

# Database Initialize
tutorial.create-table.script=file:sqls/create-member-info-table.sql
#tutorial.insert-data.script=file:sqls/insert-member-info-data.sql
tutorial.insert-data.script=file:sqls/insert-member-info-error-data.sql

When input check is implemented for the job which inputs or outputs data by accessing a file: Execute job by using execution configuration created in Execute job from execution configuration of the job which inputs or outputs data by accessing a file.

In order to use abnormal data, change path of input file (inputFile) from normal data (insert-member-info-data.csv) to abnormal data (insert-member-info-error-data.csv), from the arguments set in execution configuration.

Verify console log

Open Console View and verify that log of the following details is output.
Here, verify that the process ends abnormally (FAILED) and org.springframework.batch.item.validator.ValidationException has occurred.

Example of console log output

[2017/08/28 10:53:21] [main] [o.s.b.c.s.AbstractStep] [ERROR] Encountered an error executing step jobPointAddChunk.step01 in job jobPointAddChunk
org.springframework.batch.item.validator.ValidationException: Validation failed for org.terasoluna.batch.tutorial.common.dto.MemberInfoDto@1fde4f40:
Field error in object 'item' on field 'point': rejected value [1000001]; codes [Max.item.point,Max.point,Max.int,Max]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [item.point,point]; arguments []; default message [point],1000000]; default message [must be less than or equal to 1000000]
        at org.springframework.batch.item.validator.SpringValidator.validate(SpringValidator.java:54)

        (.. omitted)

Caused by: org.springframework.validation.BindException: org.springframework.validation.BeanPropertyBindingResult: 1 errors
Field error in object 'item' on field 'point': rejected value [1000001]; codes [Max.item.point,Max.point,Max.int,Max]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [item.point,point]; arguments []; default message [point],1000000]; default message [must be less than or equal to 1000000]
        ... 29 common frames omitted
[2017/08/28 10:53:21] [main] [o.s.b.c.l.s.SimpleJobLauncher] [INFO ] Job: [FlowJob: [name=jobPointAddChunk]] completed with the following parameters: [{jsr_batch_run_id=408}] and the following status: [FAILED]
[2017/08/28 10:53:21] [main] [o.s.c.s.ClassPathXmlApplicationContext] [INFO ] Closing org.springframework.context.support.ClassPathXmlApplicationContext@2145433b: startup date [Mon Aug 28 10:53:18 JST 2017]; root of context hierarchy

Verify exit code

Verify that the process has terminated abnormally, by exit code.
For verification process, refer Job execution and results verification. Verify that exit code (exit value) is 255 (abnormal end).

Confirm the Exit Code of ValidationJob for ChunkModel

Verify exit code

Verify output resource

Verify output resource (database or file) using the job which implements input check.

In case of a Chunk model, since an intermediate commit method is adopted, verify that update is determined upto chunk just prior to the error location part.

Verify member information table

Use Data Source Explorer and verify member information table.
Compare the contents of member information table before and after the update, and verify that the contents are in accordance with verification details.
For verification procedure, refer Refer database by using Data Source Explorer.

Verification details

Regarding records from 1 to 10 (Records with member id from "00000001" to "00000010")
- status column
  - Records with "0"(initial status) record should not exist
- point column
  - Points should be added according to membership type
    
    100 points when type column is "G" (Gold member)
    
    10 points when type column is "N"(Normal member)
Regarding records from 11 to 15 (Records with member id from "00000011" to "00000015")
- Should not be updated

Contents of the member information table before and after update are shown below.

Contents of the member information table before and after update

Verify member information file

Compare input and output contents of member information file and verify that the contents are in accordance with the verification details.

Verification details

Member information file is output in output directory
- Output file: files/output/output-member-info-data.csv
Output records should contain only the records from 1 to 10 (records with member id from "00000001" to "00000010")
Regarding output records
- status column
  - Records with "0"(initial status) should not exist
- point column
  - Points should be added according to membership type, for adding points
    
    100 points when type column is "G"(Gold member)
    
    10 points when type column is "N"(Normal member)

Input and output details of member information file are shown below.
File fields are output in the sequence of id(member ID), type(membership type), status(product purchasing flag) and point(Points).

Input/Output details of member information file

Implementation in Tasklet model

Operations from creation to execution of a job which performs input check in Tasklet model are implemented by following procedure.

Defining input check rules
Implementation of input check process
Job execution

Defining input check rules

In order to perform input check, assign a Bean Validation annotation to the field for checking DTO class.
For input check annotation, refer Bean Validation check rules and Hibernate Validator check rules of TERASOLUNA Server 5.x Development Guideline.

Since it is used in common for Chunk model / Tasklet model, it can be skipped if it has already been implemented.

Here, define check rules to check whether points have exceeded 1,000,000 (upper limit).

org.terasoluna.batch.tutorial.common.dto.MemberInfoDTO

package org.terasoluna.batch.tutorial.common.dto;

import javax.validation.constraints.Max;

public class MemberInfoDto {
    private String id;

    private String type;

    private String status;

    @Max(1000000) // (1)
    private int point;

    // Getter and setter are omitted.
}

Explanation
Sr. No.	Explanation
(1)	Assign @Max annotation which indicates that target field is less than or equal to the specified numeric value.

Implementation of input check process

Implement input check processing in business logic class which performs point addition.

Add implementation of input check process in already implemented PointAddTasklet class.
Since it acts as an explanation of job which accesses the database as per Prerequisite, only (1)~(3) below are added for the implementation in case of job accessing the file.

org.terasoluna.batch.tutorial.dbaccess.tasklet.PointAddTasklet

// Package and the other import are omitted.

import javax.inject.Inject;

@Component
public class PointAddTasklet implements Tasklet {
    // Definition of constant, ItemStreamReader and ItemWriter are omitted.

    @Inject // (1)
    Validator<MemberInfoDto> validator; // (2)

    @Override
    public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
        MemberInfoDto item = null;

        List<MemberInfoDto> items = new ArrayList<>(CHUNK_SIZE);

        try {
            reader.open(chunkContext.getStepContext().getStepExecution().getExecutionContext());

            while ((item = reader.read()) != null) {
                validator.validate(item); // (3)

                // The other codes of business logic are omitted.
            }

            writer.write(items);
        } finally {
            reader.close();
        }

        return RepeatStatus.FINISHED;
    }
}

Explanation
Sr. No.	Explanation
(1)	Inject an instance of `SpringValidator`.
(2)	Set a DTO to be fetched through `ItemReader` in type argument of `org.springframework.batch.item.validator.Validator`.
(3)	Execute `Validator#validate()` using DTO fetched through `ItemReader` as an argument. Originally, try-catch was implemented to handle input check error and catch the exception while executing `validate()`, however since exception handling using try-catch is explained in Job which performs exception handling by try-catch, exception handling is not implemented here.

Job execution

Execute the created job on IDE and verify results.

Execute job from execution configuration

Execute the job from already created execution configuration.

Here, execute the job by using abnormal data.
Since how to change input data varies based on the resource (database or file) which handles a job implementing input check, implement as below.

When input check is implemented for the job which inputs or outputs data by accessing a database: Execute job by using execution configuration created in Execute job from execution configuration of job which inputs or outputs data by accessing database.

In order to use abnormal data, comment out script of normal data and remove comment out of script of abnormal data by Database Initialize of batch-application.properties.

src/main/resources/batch-application.properties

# Database Initialize
tutorial.create-table.script=file:sqls/create-member-info-table.sql
#tutorial.insert-data.script=file:sqls/insert-member-info-data.sql
tutorial.insert-data.script=file:sqls/insert-member-info-error-data.sql

When input check is implemented for the job which inputs or outputs data by accessing a file: Execute job by using execution configuration created in Execute job from execution configuration of the job which inputs or outputs data by accessing a file.

In order to use abnormal data, change the path of input file (inputFile) from normal data (insert-member-info-data.csv) to abnormal data (insert-member-info-error-data.csv), from the arguments set in execution configuration.

Verify console log

Open Console View and verify whether the log for following details is output.
Here, verify that process ends abnormally (FAILED) and org.springframework.batch.item.validator.ValidationException has occurred.

Example of console log output

[2017/09/12 10:18:49] [main] [o.s.b.c.s.AbstractStep] [ERROR] Encountered an error executing step jobPointAddTasklet.step01 in job jobPointAddTasklet
org.springframework.batch.item.validator.ValidationException: Validation failed for org.terasoluna.batch.tutorial.common.dto.MemberInfoDto@2c383e33:
Field error in object 'item' on field 'point': rejected value [1000001]; codes [Max.item.point,Max.point,Max.int,Max]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [item.point,point]; arguments []; default message [point],1000000]; default message [must be less than or equal to 1000000]
        at org.springframework.batch.item.validator.SpringValidator.validate(SpringValidator.java:54)

(.. omitted)

Caused by: org.springframework.validation.BindException: org.springframework.validation.BeanPropertyBindingResult: 1 errors
Field error in object 'item' on field 'point': rejected value [1000001]; codes [Max.item.point,Max.point,Max.int,Max]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [item.point,point]; arguments []; default message [point],1000000]; default message [must be less than or equal to 1000000]
        ... 24 common frames omitted
[2017/09/12 10:18:49] [main] [o.s.b.c.l.s.SimpleJobLauncher] [INFO ] Job: [FlowJob: [name=jobPointAddTasklet]] completed with the following parameters: [{jsr_batch_run_id=476}] and the following status: [FAILED]
[2017/09/12 10:18:49] [main] [o.s.c.s.ClassPathXmlApplicationContext] [INFO ] Closing org.springframework.context.support.ClassPathXmlApplicationContext@735f7ae5: startup date [Tue Sep 12 10:18:47 JST 2017]; root of context hierarchy

Verify exit code

Verify that the process has terminated abnormally by using exit code.
For verification procedure, refer Job execution and results verification. Verify that exit code (exit value) is 255(abnormal end).

Confirm the Exit Code of ValidationJob for TaskletModel

Verify exit code

Verify output resource

Verify output resource (database or file) by job which implements input check.

Since batch commit method is used in case of a Tasklet model, it must be confirmed that it has not been updated at all in case an error occurs.

Verify member information table

Use Data Source Explorer and verify member information table.
Compare contents of member information table before and after update, and verify whether the contents are in accordance with the verification details.
For verification procedure, refer Refer database by using Data Source Explorer.

Verification details

Data should not be updated for all the records

Contents of the member information table in the initial state

Verify member information file

Compare input and output details of member information file and verify that the contents are in accordance with the verification details.

Verification details

Member information file is output as blank file in the output directory
- Output file: files/output/output-member-info-data.csv