Setting:Currently pipeSize is a variable in the TransferJobRequest that allows the API to determine how many read() calls to a single write() in any reader and writer implemented.
Problem: We need to change this variable dynamically for all steps that are processing. The reason is pipeSize effectively determines how much we hit the network/disk using consecutive calls which can be thought to similarly as Sequential IO for a segment of a file. This in itself needs to be dynamic as network conditions can dramatically change and pipSize directly impacts how much IO we do.
Solution: Spring Batch provides a CompletionPolicy.java interface where we can define when a chunk is deemed fully read in/produced. We can create an impl that will determine how many times are read through a variable that gets changed via the Rest API. This will allow for dynamic control of the commit-interval/pipeSize of the running job.
Setting:Currently pipeSize is a variable in the TransferJobRequest that allows the API to determine how many read() calls to a single write() in any reader and writer implemented.
Problem: We need to change this variable dynamically for all steps that are processing. The reason is pipeSize effectively determines how much we hit the network/disk using consecutive calls which can be thought to similarly as Sequential IO for a segment of a file. This in itself needs to be dynamic as network conditions can dramatically change and pipSize directly impacts how much IO we do.
Solution: Spring Batch provides a CompletionPolicy.java interface where we can define when a chunk is deemed fully read in/produced. We can create an impl that will determine how many times are read through a variable that gets changed via the Rest API. This will allow for dynamic control of the commit-interval/pipeSize of the running job.