Asynchronous execution type job

Premise

As explained in How to proceed with the tutorial, it is assumed that asynchronous execution is performed for already created jobs. Note that, asynchronous execution method includes Method which uses DB polling and Method which uses Web container.
However, it must be noted that the description is for explanation of asynchronous execution of job using DB polling.

Overview

Execute job asynchronously using DB polling.

Note that, since this section is based on TERASOLUNA Batch 5.x Development guideline, refer Asynchronous execution (DB polling) for details.
Background, process overview and business specifications of the application to be created are omitted as jobs are already created as per Premise.

Asynchronous execution of the job by using DB polling is subsequently explained by the following procedure.

Preparation
Start asynchronous batch daemon
Register job information in job request table
Job execution and results verification
Stopping asynchronous batch daemon
Verifying job execution status

Preparation

Implement preparation to perform asynchronous execution (DB polling).

Operations to be implemented are as given below.

Polling process setting
Job configuration
Input resource setting

Polling process setting

Configure settings required for asynchronous execution by batch-application.properties.
Since TERASOLUNA Batch 5.x is already configured, a detail explanation is omitted. For the explanation of each item, refer polling process settings of Various settings.

src/main/resources/batch-application.properties

# TERASOLUNA AsyncBatchDaemon settings.
async-batch-daemon.scheduler.size=1
async-batch-daemon.schema.script=classpath:org/terasoluna/batch/async/db/schema-h2.sql
async-batch-daemon.job-concurrency-num=3
# (1)
async-batch-daemon.polling-interval=5000
async-batch-daemon.polling-initial-delay=1000
# (2)
async-batch-daemon.polling-stop-file-path=/tmp/stop-async-batch-daemon

Explanation
Sr.No.	Explanation
(1)	Set the polling cycle (milliseconds). Here, specify 5000 milliseconds (5 seconds).
(2)	Configure exit file path to stop asynchronous batch daemon. Since this tutorial is based on the assumption that it is implemented in Windows environment, stop-async-batch-daemon file is placed under C:\tmp in this configuration.

Job configuration

Job to be executed asynchronously is set to automaticJobRegistrar of async-batch-daemon.xml.

As an example, configuration which specify Job which inputs or outputs data by accessing database(chunk model) is shown below.

src/main/resources/META-INF/spring/async-batch-daemon.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:jdbc="http://www.springframework.org/schema/jdbc"
       xmlns:c="http://www.springframework.org/schema/c"
       xmlns:p="http://www.springframework.org/schema/p"
       xmlns:task="http://www.springframework.org/schema/task"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
            http://www.springframework.org/schema/jdbc http://www.springframework.org/schema/jdbc/spring-jdbc.xsd
            http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task.xsd
            http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">

    <!-- omitted -->

    <bean id="automaticJobRegistrar" class="org.springframework.batch.core.configuration.support.AutomaticJobRegistrar">
        <property name="applicationContextFactories">
            <bean class="org.springframework.batch.core.configuration.support.ClasspathXmlApplicationContextsFactoryBean"
                p:resources="classpath:/META-INF/jobs/dbaccess/jobPointAddChunk.xml" /> <!-- (1) -->
        </property>
        <property name="jobLoader">
            <bean class="org.springframework.batch.core.configuration.support.DefaultJobLoader"
                p:jobRegistry-ref="jobRegistry" />
        </property>
    </bean>

    <!-- omitted -->

</beans>

Explanation
Sr. No.	Explanation
(1)	Specify a Bean definition file of job targeted for asynchronous execution. Wild cards (*/) can also be used. Please refer to precautions in Job settings while specifying a job.

Considerations while designing a job

Since it is possible to execute the same job in parallel as a characteristic of asynchronous execution (DB polling), it is necessary to prevent the impact of the same job while executing the jobs in parallel.

In this tutorial, the same job ID is used for database access job and file access job. Although these jobs are not executed in parallel, in this tutorial, it should be kept in mind while designing a job since an error is likely to occur if multiple jobs with the same job ID are specified.

Input resource setting

Set an input resource (database or file) while executing a job asynchronously.
Here, a job that uses normal data is executed.

Set an input resource for the job to access database and the job to access file.

In case of a job which accesses database: Set Database Initialize script of batch-application.proeprties as shown below.

src/main/resources/batch-application.proeprties

# Database Initialize
tutorial.create-table.script=file:sqls/create-member-info-table.sql
tutorial.insert-data.script=file:sqls/insert-member-info-data.sql
#tutorial.insert-data.script=file:sqls/insert-member-info-error-data.sql

In case of a job which accesses a file

It must be verified in advance that the input file is deployed and output directory exists.

Input file
- files/input/input-member-info-data.csv
Output directory
- files/output/

Regarding preparation of data for input resource in this tutorial

In case of a job which accesses database, execute INSERT SQL while starting asynchronous batch daemon (ApplicationContext generation) and prepare the data in database.

In case of a job which accesses file, place input file in the directory, and specify path of input/output file as the parameter part of the job information while registering job information in the job request table.

Start asynchronous batch daemon

Start AsyncBatchDaemon provided by TERASOLUNA Batch 5.x.

Create the execution configuration as shown below and start asynchronous batch daemon.
Refer Create Run Configuration (Execution configuration) for the creation procedure.

Value to be set in Main tab of Run Configurations
Item name	Value
Name	Run Job With AsyncBatchDaemon (Set any value)
Project	terasoluna-batch-tutorial
Main class	org.terasoluna.batch.async.db.AsyncBatchDaemon

When asynchronous batch daemon is started, polling process is executed within 5 seconds interval (milliseconds specified in async-batch-daemon.polling-interval of batch-application.properties).
Output example of log is shown below.
This log shows that the polling process was executed three times.

Output example of console log

[2017/09/06 18:53:23][main] [o.t.b.a.d.AsyncBatchDaemon] [INFO ] Async Batch Daemon start.

(.. omitted)

[2017/09/06 18:53:27] [main] [o.t.b.a.d.AsyncBatchDaemon] [INFO ] Async Batch Daemon will start watching the creation of a polling stop file. [Path:\tmp\stop-async-batch-daemon]
[2017/09/06 18:53:27] [daemonTaskScheduler-1] [o.t.b.a.d.JobRequestPollTask] [INFO ] Polling processing.
[2017/09/06 18:53:33] [daemonTaskScheduler-1] [o.t.b.a.d.JobRequestPollTask] [INFO ] Polling processing.
[2017/09/06 18:53:38] [daemonTaskScheduler-1] [o.t.b.a.d.JobRequestPollTask] [INFO ] Polling processing.

Register job information in job request table

Issue a SQL (INSERT statement) to register information for executing the job in job request table (batch_job_request).

For table specifications of job request table, refer job request table structure of Table for polling.

How to execute SQL on STS is shown below.

SQL execution procedure

Display Data Source Explorer View
For how to display View, refer Preparation to refer database from STS.

Data Source Explorer View

Open SQL Scrapbook
Right-click datasource and click "Open SQL Scrapbook".

SQL Scrapbook

Describe SQL
SQL for executing a job which accesses a database and a job which accesses a file are shown below using an example of chunk model.

In case of a job which accesses database: SQL to be described is shown below.

SQL for execution request of job which accesses database

INSERT INTO batch_job_request(job_name,job_parameter,polling_status,create_date)
VALUES ('jobPointAddChunk', '', 'INIT', current_timestamp);

In case of a job which accesses a file: SQL to be described is shown below.

SQL for execution request of a job which accesses a file

INSERT INTO batch_job_request(job_name,job_parameter,polling_status,create_date)
VALUES ('jobPointAddChunk', 'inputFile=files/input/input-member-info-data.csv,outputFile=files/output/output-member_info_out.csv', 'INIT', current_timestamp);

Image after SQL description is shown below.
Here, SQL for execution request of a job which accesses database is described.

SQL input

Execute SQL
As shown below, set Connection Profile of SQL Scrapbook, right-click on the margin and click → [Execute All].
Connection Profile contents are based on contents set in Preparation which refers a database from STS.

Execute SQL

Verify execution results of SQL
As shown in the diagram below, Status of SQL executed by SQL Results View is Succeeded.

Verify SQL execution results

Verify a job request table
As shown in the figure below, verify that information for executing the job is registered in the job request table.
POLLING_STATUS is registered by INIT, however, if the polling is executed already, POLLING_STATUS becomes POLLED or EXECUTED.
For details of POLLING_STATUS, refer Transition pattern of polling status (polling_status).

Verify job request table

Job execution and results verification

Verify execution results of a job for asynchronous execution.

Verifying console log

Open Console View and verify that log of following details is output.
Here, verify that the processing is completed (COMPLETED) and no exceptions have occurred.

Console log output example

(.. omitted)

[2017/09/06 18:59:50] [daemonTaskScheduler-1] [o.t.b.a.d.JobRequestPollTask] [INFO ] Polling processing.
[2017/09/06 18:59:55] [daemonTaskScheduler-1] [o.t.b.a.d.JobRequestPollTask] [INFO ] Polling processing.
[2017/09/06 18:59:55] [daemonTaskExecutor-1] [o.s.b.c.l.s.SimpleJobOperator] [INFO ] Checking status of job with name=jobPointAddChunk
[2017/09/06 18:59:55] [daemonTaskExecutor-1] [o.s.b.c.l.s.SimpleJobOperator] [INFO ] Attempting to launch job with name=jobPointAddChunk and parameters=
[2017/09/06 18:59:55] [daemonTaskExecutor-1] [o.s.b.c.l.s.SimpleJobLauncher] [INFO ] Job: [FlowJob: [name=jobPointAddChunk]] launched with the following parameters: [{jsr_batch_run_id=117}]
[2017/09/06 18:59:55] [daemonTaskExecutor-1] [o.s.b.c.j.SimpleStepHandler] [INFO ] Executing step: [jobPointAddChunk.step01]
[2017/09/06 18:59:55] [daemonTaskExecutor-1] [o.s.b.c.l.s.SimpleJobLauncher] [INFO ] Job: [FlowJob: [name=jobPointAddChunk]] completed with the following parameters: [{jsr_batch_run_id=117}] and the following status: [COMPLETED]
[2017/09/06 19:00:00] [daemonTaskScheduler-1] [o.t.b.a.d.JobRequestPollTask] [INFO ] Polling processing.
[2017/09/06 19:00:05] [daemonTaskScheduler-1] [o.t.b.a.d.JobRequestPollTask] [INFO ] Polling processing

Verifying exit codes

In the case of asynchronous execution, it is not possible to check exit code if the job is been executed on STS.
Check job execution status with Verifying job execution status.

Verifying output resource

Verify output resource (database or file) according to the executed job.

In case of a job which accesses database

Compare contents of member information table before and after update and verify that the contents are in accordance with verification details.
For verification procedure, refer Refer database by using Data Source Explorer.

Verification details

status column
- Records with "0"(initial status) should not exist
point column
- Points should be added according to membership type, for point addition
  - 100 points when type column is "G"(gold member)
  - 10 points when type column is "N"(normal member)
- Records exceeding 1,000,000 points (upper limit value) should not exist

Contents of member information table before and after update are shown below.

Contents of member information table before and after update

In case of a job which accesses a file

Compare input and output details of member information file and verify that the contents are in accordance with verification details.

Verification details

Member information file should be output in the output directory
- Output file: files/output/output-member-info-data.csv
status field
- Records with "0"(initial status) should not exist
point field
- Points should be added according to membership type, for point addition
  - 100 points when type field is "G"(gold member)
  - 10 points when type field is "N"(normal member)
- Records with points exceeding 1,000,000(upper limit value) should not exist

Input and output details of member information file are shown below.
File fields are output in the sequence of id(member id, type(membership type), status(product purchasing flag) and point(point).

Input and output details of member information file

Stopping asynchronous batch daemon

Create an exit file and stop asynchronous batch daemon.

Create stop-async-batch-daemon file(blank file) in C:tmp as instructed in polling process setting.

Create exit file

Verify that asynchronous batch daemon is stopped as shown below in STS console.

Verify termination of asynchronous batch daemon

(.. omitted)

[2017/09/08 21:41:41] [daemonTaskScheduler-1] [o.t.b.a.d.JobRequestPollTask] [INFO ] Polling processing.
[2017/09/08 21:41:44] [main] [o.t.b.a.d.AsyncBatchDaemon] [INFO ] Async Batch Daemon has detected the polling stop file, and then shutdown now!
[2017/09/08 21:41:44] [main] [o.s.c.s.ClassPathXmlApplicationContext] [INFO ] Closing org.springframework.context.support.ClassPathXmlApplicationContext@6b2fad11: startup date [Fri Sep 08 21:41:01 JST 2017]; root of context hierarchy
[2017/09/08 21:41:44] [main] [o.s.b.c.c.s.GenericApplicationContextFactory$ResourceXmlApplicationContext] [INFO ] Closing ResourceXmlApplicationContext:file:/C:/dev/workspace/tutorial/terasoluna-batch-tutorial/target/classes/META-INF/jobs/dbaccess/jobPointAddChunk.xml
[2017/09/08 21:41:44] [main] [o.s.c.s.DefaultLifecycleProcessor] [INFO ] Stopping beans in phase 0
[2017/09/08 21:41:44] [main] [o.t.b.a.d.JobRequestPollTask] [INFO ] JobRequestPollTask is called shutdown.
[2017/09/08 21:41:44] [main] [o.s.s.c.ThreadPoolTaskScheduler] [INFO ] Shutting down ExecutorService 'daemonTaskScheduler'
[2017/09/08 21:41:44] [main] [o.s.s.c.ThreadPoolTaskExecutor] [INFO ] Shutting down ExecutorService
[2017/09/08 21:41:44] [main] [o.t.b.a.d.AsyncBatchDaemon] [INFO ] Async Batch Daemon stopped after all jobs completed.

Verifying job execution status

Verify the job status and execution result in meta data table of JobRepository. Here, refer to batch_job_execution.

SQL to verify job status is shown below.

SQL for checking job status

SELECT job_execution_id,start_time,end_time,exit_code FROM batch_job_execution WHERE job_execution_id =
(SELECT max(job_execution_id) FROM batch_job_request WHERE job_execution_id IS NOT NULL);

In this SQL, execution status of the last job is fetched.

SQL execution results can be verified in SQL Results View displayed after execution of SQL on STS.
Verify that EXIT_CODE is COMPLETED as shown in the diagram below.

Verify job status