Exclusive Control

Overview

Exclusive control is a process performed to maintain consistency of data when update processing is performed simultaneously for the same resource from multiple transactions. In the case where there is a possibility that updating processing is performed simultaneously for the same resource from multiple transactions, it is basically necessary to perform exclusive control.

Here, multiple transactions include the following transactions.

Transaction at the time of simultaneous execution of multiple jobs
Transaction at the time of simultaneous execution with online processing

Exclusive control of multiple jobs

When multiple jobs are executed at the same time, it is fundamental to design jobs so that exclusive control is not required. This means that it is basic to divide the resources to be accessed and the processing target for each job.

Since the concept of exclusive control is same as online processing, please refer to Exclusive Control in TERASOLUNA Server 5.x Development Guideline

Here, we will focus on the part not explained in TERASOLUNA Server 5.x.

The usage method of this function is same in the chunk model as well as tasklet model.

Necessity of Exclusive Control

For the necessity of exclusive control, please refer to Necessity of Exclusive Control in TERASOLUNA Server 5.x Development Guideline.

Exclusive Control for File

Exclusive control for file is generally implemented by file locking.

File Locking: File locking is a mechanism for restricting reading and writing from other programs while using files with a certain program. The outline of file lock processing is as follows.

Scenario

The batch process A acquires the lock of the file and starts the file updating process.
Batch process B attempts to update the same file and fails the attempt to acquire the file lock.
The batch process A ends the processing and unlocks the file

Overview of File Lock Processing

The Batch Process A tries to acquire the lock of the Target File.
The Batch Process A succeeds in acquiring the lock of the Target File.
The Batch Process B tries to acquire the lock of the Target File.
The Batch Process A writes the Target File.
Since the Batch Process A has locked the Target File, the Batch Process B fails to acquire the lock of the Target File.
The Batch Process B performs processing of file update failure.
The Batch Process A releases the lock of the Target File.

Prevention of Deadlock

Even for the files, when a lock is to be fetched for multiple files similar to database, a deadlock may occur. Therefore, it is important to create a rule for update order of files.
The prevention of deadlock is similar to prevention of deadlock between tables in the database. For details, refer to Prevention of deadlock in TERASOLUNA Server 5.x Development Guideline.

Exclusive Control of Database

For details about Exclusive Control of Database, refer to Exclusive control using database locking in TERASOLUNA Server 5.x Development Guideline.

Choose Exclusive Control Scheme

Explain the locking scheme and suitable situation for TERASOLUNA Batch 5.x.

Choose exclusive control scheme
Lock scheme	Suitable situation
Optimistic locking	In a concurrent transaction, when the update results of another transaction can be considered out of scope of processing and the process can be continued
Pessimistic locking	Process wherein the processing time is long and carrying out process again due to change in status of the data to be processed is difficult. Process requiring exclusive control for files

Relationship between Exclusive Control and Components

The relationship between each component provided by TERASOLUNA Batch 5.x and exclusive control is as follows.

Optimistic lock

Relationship between exclusive control and components
Processing model	Component	File	Database
Chunk	ItemReader	-	Acquires data including a column that can confirm that the same data is obtained at the time of acquiring and updating such as Version column.
	ItemProcessor	-	Exclusive control is unnecessary.
	ItemWriter	-	Check the difference between an acquisition and update, confirm that it is not updated by other processing, then update.
Tasklet	Tasklet	-	When acquiring data, execute the processing described in the ItemReader section, and when updating the data, the processing described in ItemWriter section. The concept is the same when using the Mapper interface directly.

Optimistic lock on files

Because of the characteristic of the file, do not apply optimistic lock on files.

Pessimistic lock

Relationship between exclusive control and components
Processing model	Component	File	Database
Chunk	ItemReader	-	Issue SELECT statement without using pessimistic lock. Exclusive control is not performed in ItemReader as connection is different from ItemProcessor and ItemWriter. Performance is improved by using minimum required data (key information) as the condition to fetch data in ItemProcessor by SELECT.
	ItemProcessor	-	Using Mapper interface, issue SELECT FOR UPDATE in SQL statement by using the data (key information) fetched in ItemReader as the condition.
	ItemWriter	-	Update data without considering exclusive control in ItemWriter since there is same transaction as ItemProcessor having pessimistic lock.
Tasklet	Tasklet	Get a file lock right after opening a file with ItemStreamReader. Release the file lock just before closing ItemStreamWriter.	When fetching data, directly use ItemReader or Mapper interface to issue SELECT FOR UPDATE statement. When updating data, implement the process explained in ItemWriter. The concept is the same when using the Mapper interface directly.

Precautions due to pessimistic lock in database of chunk model

The data (key information) fetched in ItemReader is not exclusively controlled while it is passed to ItemProcessor and the original data may have been updated by other transaction. Therefore, the condition of fetching data by ItemProcessor should include the condition to fetch the data (key information) same as ItemReader.
When the data cannot be fetched in ItemProcessor, there is a need to consider and implement continuation or interruption of process considering the possibility that data is updated by other transaction.

Pessimistic lock on file

Pessimistic lock on files should be implemented in the tasklet model. In the chunk model, due to its structure, there is a period that can not be excluded in the gap of chunk processing. Also, it is assumed that file access is done by Injecting ItemStreamReader / ItemStreamWriter.

Waiting time due to Pessimistic lock in database

When pessimistic locking is performed, the wait time for processing due to contention may be prolonged. In that case, it is reasonable to use the pessimistic lock by specifying the NO WAIT option and the timeout time.

How to use

Explain how to use exclusive control by resource.

Exclusive Control of file
Exclusive Control of Database

Exclusive Control of file

Exclusive control of file with TERASOLUNA Batch 5.x is realized by implementing Tasklet. As the means of achieving exclusion, exclusive control is performed by file lock acquisition using the java.nio.channels.FileChannel class.

Details of the FileChannel class

For details and how to use FileChannel class, refer to Javadoc.

Show an example of using FileChannel class to get a file lock.

Tasklet implementation

@Component
@Scope("step")
public class FileExclusiveTasklet implements Tasklet {

    private String targetPath = null; // (1)

    @Inject
    ItemStreamReader<SalesPlanDetail> reader;

    @Inject
    ItemStreamWriter<SalesPlanDetailWithProcessName> writer;

    @Override
    public RepeatStatus execute(StepContribution contribution,
            ChunkContext chunkContext) throws Exception {

        // omitted.

        FileChannel fc = null;
        FileLock fileLock = null;

        try {
            try {
                File file = new File(targetPath);
                fc = FileChannel.open(file.toPath(), StandardOpenOption.WRITE,
                        StandardOpenOption.CREATE,
                        StandardOpenOption.APPEND); // (2)
                fileLock = fc.tryLock();  // (3)
            } catch (IOException e) {
                logger.error("Failure other than lock acquisition", e);
                throw new FailedOtherAcquireLockException(
                        "Failure other than lock acquisition", e);
            }
            if (fileLock == null) {
                logger.error("Failed to acquire lock. [processName={}]", processName);
                throw new FailedAcquireLockException("Failed to acquire lock");
            }

            reader.open(executionContext);
            writer.open(executionContext);  // (4)

            // (5)
            SalesPlanDetail item;
            List<SalesPlanDetailWithProcessName> items = new ArrayList<>();
            while ((item = reader.read()) != null) {

                // omitted.

                items.add(item);
                if (items.size() >= 10) {
                    writer.write(items);
                    items.clear();
                }
            }
            if (items.size() > 0) {
                writer.write(items);
            }

        } finally {
            if (fileLock != null) {
                try {
                    fileLock.release();  // (6)
                } catch (IOException e) {
                    logger.warn("Lock release failed.", e);
                }
            }
            if (fc != null) {
                try {
                    fc.close();
                } catch (IOException e) {
                    // do nothing.
                }
            }
            try {
                writer.close(); // (7)
            } catch (ItemStreamException e) {
                // ignore
            }
            try {
                reader.close();
            } catch (ItemStreamException e) {
                // ignore
            }
        }
        return RepeatStatus.FINISHED;
    }

    // (8)
    @Value("#{jobParameters['outputFile']}")
    public void setTargetPath(String targetPath) {
        this.targetPath = targetPath;
    }
}

Description
Sr. No.	Description
(1)	The file path to be exclusively controlled.
(2)	Get file channel. In this example, channels for new creation, addition and writing of files are obtained.
(3)	Get file lock.
(4)	Open file to be locked if the file lock is fetched successfully.
(5)	Execute business logic with file output.
(6)	Release file lock.
(7)	Close the file to be exclusively controlled.
(8)	Set file path. In this example, it receives from the job parameter.

About the method of FileChannel used for lock acquisition

It is recommended to use the tryLock() method which is not waiting because the lock() method waits until the lock is released if the target file is locked. Note that trylock() can select shared lock and exclusive lock, but in batch processing, exclusive lock is normally used.

Exclusive control between threads in the same VM

Attention must be paid to exclusive control between threads in the same VM. When processing files between threads in the same VM, the lock function using the FileChannel class cannot determine whether a file is locked by processing of another thread.
Therefore, exclusive control between threads does not function. In order to avoid this, exclusive control between threads can be performed by performing synchronization processing in the part where writing to the file is performed.
However, synchronizing reduces the merit of parallel processing, and it is not different from processing with a single thread. As a result, since it is not suitable to perform exclusive control with different threads for the same file and continue processing, such a process should not be designed and implemented.

About appendAllowed property of FlatFileItemWriter

When creating (overwriting) a file, exclusive control can be realized by setting the appendAllowed property to false (default). This is because FileChannel is controlled inside FlatFileItemWriter. However, if the file is appended (appendAllowed property is true), developers need to implement exclusive control with FileChannel.

Exclusive Control of Database

Explain exclusive control of database in TERASOLUNA Batch 5.x.

The exclusive control implementation of the database is basically How to implement while using MyBatis3 in TERASOLUNA Server 5.x Development Guideline. In this guideline, explanation is given assuming that the implementation method while using How to implement while using MyBatis3 is ready.

As shown in Relationship between Exclusive Control and Components, there are variations due to combination of processing model and component.

Variation of exclusive control of database
Exclusive control scheme	Processing model	Component
Optimistic lock	Chunk model	ItemReader/ItemWriter
	Tasklet model	ItemReader/ItemWriter
	Tasklet model	Mapper interface
Pessimistic lock	Chunk model	ItemReader/ItemWriter
	Tasklet model	ItemReader/ItemWriter
	Tasklet model	Mapper interface

When using the Mapper interface in tasklet model, the explanation is omitted. Refer to How to implement while using MyBatis3.

When using ItemReader/ItemWriter in tasklet model, the calling part in the Mapper interface is replaced by ItemReader/ItemWriter, so the explanation is also omitted.

Therefore, exclusive control of chunk model will be explained here.

Optimistic Lock

Explain optimistic lock in chunk model.

Since the behavior of the job changes according to the setting of the assertUpdates property of MyBatisBatchItemWriter, it is necessary to set it appropriately according to the business requirements.

Show the job definition for optimistic lock.

job definition

<!-- (1) -->
<bean id="reader"
      class="org.mybatis.spring.batch.MyBatisCursorItemReader" scope="step"
      p:queryId="org.terasoluna.batch.functionaltest.ch05.exclusivecontrol.repository.ExclusiveControlRepository.branchFindOne"
      p:sqlSessionFactory-ref="jobSqlSessionFactory">
    <property name="parameterValues">
        <map>
            <entry key="branchId" value="#{jobParameters['branchId']}"/>
        </map>
    </property>
</bean>

<!-- (2) --->
<bean id="writer"
      class="org.mybatis.spring.batch.MyBatisBatchItemWriter" scope="step"
      p:statementId="org.terasoluna.batch.functionaltest.ch05.exclusivecontrol.repository.ExclusiveControlRepository.branchExclusiveUpdate"
      p:sqlSessionTemplate-ref="batchModeSqlSessionTemplate"
      p:assertUpdates="true" />  <!-- (3) -->

<batch:job id="chunkOptimisticLockCheckJob" job-repository="jobRepository">
    <batch:step id="chunkOptimisticLockCheckJob.step01">
        <batch:tasklet transaction-manager="jobTransactionManager">
            <batch:chunk reader="reader" processor="branchEditItemProcessor"
                         writer="writer" commit-interval="10" />
        </batch:tasklet>
    </batch:step>
</batch:job>

Description
Sr. No.	Description
(1)	Set SQLID of data acquisition by optimistic lock.
(2)	Set SQLID of data update by optimistic lock.
(3)	Set whether to check the number of batch updates. If set to `true` (default), throw an exception if the number of updates is 0. If set to `false`, perform normal processing even if the number of updates is 0.

Pessimistic Lock

Explain pessimistic lock in chunk model.

Show the job definition for pessimistic lock.

job definition

<!-- (1) -->
<mybatis:scan
      base-package="org.terasoluna.batch.functionaltest.ch05.exclusivecontrol.repository"
      template-ref="batchModeSqlSessionTemplate"/>

<!-- (2) -->
<bean id="reader" class="org.mybatis.spring.batch.MyBatisCursorItemReader" scope="step"
      p:queryId="org.terasoluna.batch.functionaltest.ch05.exclusivecontrol.repository.ExclusiveControlRepository.branchIdFindByName"
      p:sqlSessionFactory-ref="jobSqlSessionFactory">
    <property name="parameterValues">
        <map>
            <entry key="branchName" value="#{jobParameters['branchName']}"/>
        </map>
    </property>
</bean>

<!-- (3) -->
<bean id="writer" class="org.mybatis.spring.batch.MyBatisBatchItemWriter" scope="step"
      p:statementId="org.terasoluna.batch.functionaltest.ch05.exclusivecontrol.repository.ExclusiveControlRepository.branchUpdate"
      p:sqlSessionTemplate-ref="batchModeSqlSessionTemplate"
      p:assertUpdates="#{new Boolean(jobParameters['assertUpdates'])}"/>

<batch:job id="chunkPessimisticLockCheckJob" job-repository="jobRepository">
    <batch:step id="chunkPessimisticLockCheckJob.step01">
        <batch:tasklet transaction-manager="jobTransactionManager">
            <!-- (4) -->
            <batch:chunk reader="reader" processor="branchEditWithkPessimisticLockItemProcessor"
                         writer="writer" commit-interval="3"/>
        </batch:tasklet>
    </batch:step>
</batch:job>

ItemProcessor performing pessimistic lock

@Component
@Scope("step")
public class BranchEditWithkPessimisticLockItemProcessor implements ItemProcessor<String, ExclusiveBranch> {

    // (5)
    @Inject
    ExclusiveControlRepository exclusiveControlRepository;

    // (6)
    @Value("#{jobParameters['branchName']}")
    private String branchName;

        //omitted.

    @Override
    public ExclusiveBranch process(String item) throws Exception {

      // (7)
        Branch branch = exclusiveControlRepository.branchFindOneByNameWithNowWaitLock(item, branchName);

        if (branch != null) {
            ExclusiveBranch updatedBranch = new ExclusiveBranch();

            updatedBranch.setBranchId(branch.getBranchId());
            updatedBranch.setBranchName(branch.getBranchName() + " - " + identifier);
            updatedBranch.setBranchAddress(branch.getBranchAddress() + " - " + identifier);
            updatedBranch.setBranchTel(branch.getBranchTel());
            updatedBranch.setCreateDate(branch.getUpdateDate());
            updatedBranch.setUpdateDate(new Timestamp(clock.millis()));
            updatedBranch.setOldBranchName(branch.getBranchName());

            return updatedBranch;
        } else {
            // (8)
            logger.warn("An update by another user occurred. [branchId: {}]", item);
            return null;
        }
    }
}

Explanation
Sr. No.	Explanation
(1)	Set `batchModeSqlSessionTemplate` such that Mapper interface is in the update mode as ItemWriter.
(2)	Set SQLID for data fetch without pessimistic lock. Set `branchName` from job start parameter as extraction condition. Performance can be improved by narrowing down the items fetched by this SQL to minimum required in order to uniquely identify the data in (6).
(3)	Set SQLID same as SQL for data update without exclusive control.
(4)	Set ItemProcessor for data fetch by pessimistic lock.
(5)	Inject Mapper interface for data fetch by pessimistic lock.
(6)	Set `branchName` from job start parameter as extraction condition of pessimistic lock.
(7)	Call method for data fetch by pessimistic lock. Since same conditions as the extraction conditions of (2) are set, job start parameter `branchName` is passed as an argument in addition to key information (id). When pessimistic lock is performed by setting NO WAIT and timeout and exclusion is performed by other transaction, an exception occurs here.
(8)	When the target data is updated first by another transaction and it cannot be fetched, the method for data fetch by pessimistic lock returns null. When the method for data fetch by pessimistic lock returns null, an exception occurs and it is required to handle it as per business requirements such as interrupting the process. Here, the subsequent process continues by ouput of WARN log and returning null.

Regarding components performing pessimistic lock in tasklet model

Use ItemReader that issues SQL for performing pessimistic lock in tasklet model.It is same when using Mapper interface directly.