TERASOLUNA Batch Framework for Java (5.x) Development Guideline - version 5.0.1.RELEASE, 2017-9-27
> INDEX

Overview

Exclusive control is a process performed to maintain consistency of data when update processing is performed simultaneously for the same resource from multiple transactions. In the case where there is a possibility that updating processing is performed simultaneously for the same resource from multiple transactions, it is basically necessary to perform exclusive control.

Multiple transaction means following in this chapter.

  • Transaction at the time of simultaneous execution of multiple jobs

  • Transaction at the time of simultaneous execution with online processing

Exclusive control of multiple jobs

When multiple jobs are executed at the same time, it is fundamental to design jobs so that exclusive control is not required. This means that it is basic to divide the resources to be accessed and the processing target for each job.

Since the concept of exclusive control is the same as online processing, please refer to Exclusive Control in TERASOLUNA Server 5.x Development Guideline

Here, focus on the part not described in TERASOLUNA Server 5.x.

The usage method of this function is same in the chunk model as well as tasklet model.

Necessity of Exclusive Control

For the necessity of exclusive control, please refer to Necessity of Exclusive Control in TERASOLUNA Server 5.x Development Guideline.

Exclusive Control for File

Exclusive control for file is generally implemented by file locking.

File Locking

File locking is a mechanism for restricting reading and writing from other programs while using files with a certain program. The outline of file lock processing is as follows.

Scenario
  • The batch process A acquires the lock of the file and starts the file updating process.

  • Batch process B attempts to update the same file and attempts to acquire the file lock fails.

  • The batch process A ends the processing and unlocks the file

ExclusiveControl_File_Senario
Overview of File Lock Processing
  1. The Batch Process A tries to acquire the lock of the Target File.

  2. The Batch Process A succeeds in acquiring the lock of the Target File.

  3. The Batch Process B tries to acquire the lock of the Target File.

  4. The Batch Process A writes the Target File.

  5. Since the Batch Process A is locked the Target File, the Batch Process B fails to acquire the lock of the Target File.

  6. The Batch Process B performs processing of file update failure.

  7. The Batch Process A releases the lock of the Target File.

Prevention of Deadlock

Even in a file,in the same as a database, when acquiring a lock on multiple files, deadlock may occur in some cases. Therefore, it is important to make a rule the update order of files.
The prevention of deadlock is similar to prevention of deadlock between tables in the database. For details, refer to Prevention of deadlock in TERASOLUNA Server 5.x Development Guideline.

Exclusive Control of Database

For details About Exclusive Control of Database, refer to Exclusive control using database locking in TERASOLUNA Server 5.x Development Guideline.

Choose Exclusive Control Scheme

Explain the locking scheme and suitable situation for TERASOLUNA Batch 5.x.

Choose exclusive control scheme
Lock scheme Suitable situation

Optimistic locking

The case where process can be continued with the update result of another transaction being out of processing targets in a transaction at the time of concurrent execution.

Pessimistic locking

Process in which the processing time is long and it is difficult to redo because the situation of the target data has changed during processing
Process requiring exclusive control for files

Relationship between Exclusive Control and Components

The relationship between each component provided by TERASOLUNA Batch 5.x and exclusive control is as follows.

Optimistic lock
Relationship between exclusive control and components
Processing model Component File Database

Chunk

ItemReader

-

Acquires data including a column that can confirm that the same data is obtained at the time of acquiring and updating such as Version column.

ItemProcessor

-

Exclusive control is unnecessary.

ItemWriter

-

Check the difference between acquisition and update, confirm that it is not updated by other processing, then update.

Tasklet

Tasklet

-

When acquiring data, execute the processing described in the ItemReader section, and when updating the data, the processing described in ItemWriter section.
The concept is the same when using the Mapper interface directly.

Optimistic lock on files

Because of the characteristic of the file, do not apply optimistic lock on files.

Pessimistic lock
Relationship between exclusive control and components
Processing model Component File Database

Chunk

ItemReader

-

Use FOR UPDATE of SQL statement.

ItemProcessor

Since it is fundamental to handle locked data, in principle, exclusive control is not performed here.

ItemWriter

-

Update data without conscious of exclusion.

Tasklet

Tasklet

Get a file lock right after opening a file with ItemStreamReader.
Release the file lock just before closing ItemStreamWriter.

When acquiring data, execute the processing described in the ItemReader section, and when updating the data, the processing described in ItemWriter section.
The concept is the same when using the Mapper interface directly.

Pessimistic lock on file

Pessimistic lock on files should be implemented in the tasklet model. In the chunk model, due to its structure, there is a period that can not be excluded in the gap of chunk processing. Also, it is assumed that file access is done by Injecting ItemStreamReader / ItemStreamWriter.

Waiting time due to Pessimistic lock in database

When pessimistic locking is performed, the wait time for processing due to contention may be prolonged. In that case, it is reasonable to use the pessimistic lock by specifying the NO WAIT option and the timeout time.

How to use

Explain how to use exclusive control by resource.

Exclusive control of file

Exclusive control of file with TERASOLUNA Batch 5.x is realized by implementing Tasklet. As a means of achieving exclusion, exclusive control is performed by file lock acquisition using the java.nio.channels.FileChannel class.

Details of the FileChannel class

For details and how to use FileChannel class, refer to Javadoc.

Show an example of using FileChannel class to get a file lock.

Tasklet implementation
@Component
@Scope("step")
public class FileExclusiveTasklet implements Tasklet {

    private String targetPath = null; // (1)

    @Inject
    ItemStreamReader<SalesPlanDetail> reader;

    @Inject
    ItemStreamWriter<SalesPlanDetailWithProcessName> writer;

    @Override
    public RepeatStatus execute(StepContribution contribution,
            ChunkContext chunkContext) throws Exception {

        // omitted.

        FileChannel fc = null;
        FileLock fileLock = null;

        try {
            try {
                File file = new File(targetPath);
                fc = FileChannel.open(file.toPath(), StandardOpenOption.WRITE,
                        StandardOpenOption.CREATE,
                        StandardOpenOption.APPEND); // (2)
                fileLock = fc.tryLock();  // (3)
            } catch (IOException e) {
                logger.error("Failure other than lock acquisition", e);
                throw new FailedOtherAcquireLockException(
                        "Failure other than lock acquisition", e);
            }
            if (fileLock == null) {
                logger.error("Failed to acquire lock. [processName={}]", processName);
                throw new FailedAcquireLockException("Failed to acquire lock");
            }

            reader.open(executionConetxt);
            writer.open(executionConetxt);  // (4)

            // (5)
            SalesPlanDetail item;
            List<SalesPlanDetailWithProcessName> items = new ArrayList<>();
            while ((item = reader.read()) != null) {

                // omitted.

                items.add(item);
                if (items.size() >= 10) {
                    writer.write(items);
                    items.clear();
                }
            }
            if (items.size() > 0) {
                writer.write(items);
            }

        } finally {
            if (fileLock != null) {
                try {
                    fileLock.release();  // (6)
                } catch (IOException e) {
                    logger.warn("Lock release failed.", e);
                }
            }
            if (fc != null) {
                try {
                    fc.close();
                } catch (IOException e) {
                    // do nothing.
                }
            }
            try {
                writer.close(); // (7)
            } catch (ItemStreamException e) {
                // ignore
            }
            try {
                reader.close();
            } catch (ItemStreamException e) {
                // ignore
            }
        }
        return RepeatStatus.FINISHED;
    }

    // (8)
    @Value("#{jobParameters['outputFile']}")
    public void setTargetPath(String targetPath) {
        this.targetPath = targetPath;
    }
}
Description
Sr. No. Description

(1)

File path to be exclusively controlled.

(2)

Get file channel.
In this example, channels for new creation, addition and writing of files are obtained.

(3)

Get file lock.

(4)

Open file to be locked if the file lock is fetched successfully.

(5)

Execute business logic with file output.

(6)

Release file lock.

(7)

Close file to be exclusively controlled.

(8)

Set file path.
In this example, it receives from the job parameter.

About the method of FileChannel used for lock acquisition

It is recommended to use the tryLock() method which is not waiting because the lock() method waits until the lock is released if the target file is locked. Note that trylock() can select shared lock and exclusive lock, but in batch processing, exclusive lock is normally used.

Exclusive control between threads in the same VM

Attention must be paid to exclusive control between threads in the same VM. When processing files between threads in the same VM, the lock function using the FileChannel class can not determine whether a file is locked by another thread’s processing.
Therefore, exclusive control between threads does not function. In order to avoid this, exclusive control between threads can be performed by performing synchronization processing in the part where writing to the file is performed.
However, synchronizing reduces the merit of parallel processing, and it is not different from processing with a single thread. As a result, since it is not suitable to perform exclusive control with different threads on the same file and process it, do not design and implement such processing.

About appendAllowed property of FlatFileItemWriter

When creating (overwriting) a file, exclusive control can be realized by setting the appendAllowed property to false (default). This is because FileChannel is controlled inside FlatFileItemWriter. However, if the file is appended (appendAllowed property is true), developers need to implement exclusive control with FileChannel.

Exclusive Control of Database

Explain exclusive control of database in TERASOLUNA Batch 5.x.

The exclusive control implementation of the database is basically How to implement while using MyBatis3 in TERASOLUNA Server 5.x Development Guideline. In this guideline, Explain it on the premise that How to implement while using MyBatis3 is done.

As shown in Relationship between Exclusive Control and Components, there are variations due to combination of processing model and component.

Variation of exclusive control of database

Exclusive control scheme

Processing model

Component

Optimistic lock

Chunk model

ItemReader/ItemWriter

Tasklet model

ItemReader/ItemWriter

Mapper interface

Pessimistic lock

Chunk model

ItemReader/ItemWriter

Tasklet model

ItemReader/ItemWriter

Mapper interface

When using the Mapper interface in tasklet model, the explanation is omitted. Refer to How to implement while using MyBatis3.

When using ItemReader/ItemWriter in tasklet model, the calling part in the Mapper interface is replaced by ItemReader/ItemWriter, so the explanation is also omitted.

Therefore, exclusive control of chunk model will be explained here.

Optimistic Lock

Explain Optimistic lock in chunk model.

Since the behavior of the job changes according to the setting of the assertUpdates property of MyBatisBatchItemWriter, it is necessary to set it appropriately according to the business requirements.

Show the job definition for optimistic lock.

job definition
<!-- (1) -->
<bean id="reader"
      class="org.mybatis.spring.batch.MyBatisCursorItemReader" scope="step"
      p:queryId="org.terasoluna.batch.functionaltest.ch05.exclusivecontrol.repository.ExclusiveControlRepository.branchFindOne"
      p:sqlSessionFactory-ref="jobSqlSessionFactory"/>
    <property name="parameterValues">
        <map>
            <entry key="branchId" value="#{jobParameters['branchId']}"/>
        </map>
    </property>
</bean>

<!-- (2) --->
<bean id="writer"
      class="org.mybatis.spring.batch.MyBatisBatchItemWriter" scope="step"
      p:statementId="org.terasoluna.batch.functionaltest.ch05.exclusivecontrol.repository.ExclusiveControlRepository.branchExclusiveUpdate"
      p:sqlSessionTemplate-ref="batchModeSqlSessionTemplate"
      p:assertUpdates="true" />  <!-- (3) -->

<batch:job id="chunkOptimisticLockCheckJob" job-repository="jobRepository">
    <batch:step id="chunkOptimisticLockCheckJob.step01">
        <batch:tasklet transaction-manager="jobTransactionManager">
            <batch:chunk reader="reader" processor="branchEditItemProcessor"
                         writer="writer" commit-interval="10" />
        </batch:tasklet>
    </batch:step>
</batch:job>
Description
Sr. No. Description

(1)

Set SQLID of data acquisition by optimistic lock.

(2)

Set SQLID of data update by optimistic lock.

(3)

Set whether to check the number of batch updates.
If set to true (default), throw an exception if the number of updates is 0.
if set to false, perform normal processing even if the number of updates is 0.

Pessimistic Lock

Explain pessimistic lock in chunk model.

Show the job definition for pessimistic lock.

job definition
<!-- (1) -->
<bean id="reader"
      class="org.mybatis.spring.batch.MyBatisCursorItemReader" scope="step"
      p:queryId="org.terasoluna.batch.functionaltest.ch05.exclusivecontrol.repository.ExclusiveControlRepository.branchFindOneWithNoWaitLock"
      p:sqlSessionFactory-ref="jobSqlSessionFactory">
    <property name="parameterValues">
        <map>
            <entry key="branchId" value="#{jobParameters['branchId']}"/>
        </map>
    </property>
</bean>

<!-- (2) -->
<bean id="writer"
      class="org.mybatis.spring.batch.MyBatisBatchItemWriter" scope="step"
      p:statementId="org.terasoluna.batch.functionaltest.ch05.exclusivecontrol.repository.ExclusiveControlRepository.branchUpdate"
      p:sqlSessionTemplate-ref="batchModeSqlSessionTemplate"
      p:assertUpdates="#{new Boolean(jobParameters['assertUpdates'])}" />

<batch:job id="chunkPessimisticLockCheckJob" job-repository="jobRepository">
    <batch:step id="chunkPessimisticLockCheckJob.step01">
        <batch:tasklet transaction-manager="jobTransactionManager">
            <batch:chunk reader="reader" processor="branchEditItemProcessor"
                         writer="writer" commit-interval="10" />
        </batch:tasklet>
    </batch:step>
    <batch:listeners>
        <batch:listener ref="jobExecutionLoggingListener"/>
    </batch:listeners>
</batch:job>
Description
Sr. No. Description

(1)

Set SQLID of data acquisition by pessimistic lock.

(2)

Set the same SQLID as SQL of data update without exclusive control.

Behavior during exclusive control

If performing pessimistic lock by setting NO WAIT or timeout, when excluded by another transaction, an exception is thrown in the doOpen() method of MyBatisCursorItemReader.

TERASOLUNA Batch Framework for Java (5.x) Development Guideline - version 5.0.1.RELEASE, 2017-9-27