Policy-Based Task orchestration
Last updated
Last updated
Manages and coordinates data movement tasks based on predefined policies or rules. This allows the management of an unlimited number of tasks, including their execution, but also monitoring and reporting aspects.
Workflow Manager allows the creation of different data movement tasks:
Copy
Move
Erase
Scan
Different type of policies are available:
Data Migration
Data Archiving (Active Archive)
Data Protection (Offline Archive)
Currently, the two workflows Data Archiving and Data Protection are available in the , when the Data Migration is available in the .
Each task includes the definition of the source storage, with its dataset and the destination storage.
It is possible to define multiple tasks, they are executed in parallel mode in a multi-threaded system.
Each task will execute the movement of a list of files (1 or n) and segment the processing into 3 main parts:
Preparation phase
Execution phase
Finalization phase
For each task execution process, specific jobs are created. The default segmentation is defined to optimize the data processing (example of a default setting: create a job of 1 GB or 100 files maximum).
Each job will be sent into a different queue system. A dispatching engine will coordinate the processing of each job in regard to the defined priority.
Then, different workers are created to do the copy of files and to do the reporting to Finalization process about the status of the copy.
The workflow manager can execute these 3 phases in two main different modes:
sequential processing
parallel processing
Each running task can be paused or stopped. If a task is paused, it can always be restarted. Pressing the stop button will stop the task. Once the stop button has been pressed, it may take a few seconds or minutes before the task is stopped. This is because the task always finishes the current job under processing.
An unlimited data set selector brings granularity to the data you have to select. Movement can be executed between any type of storage from storage where hot data are stored to cold storage.
For any type of data movement, the Workflow Manager always maintains the original path where the contents were originally stored.
Business applications are configured to access the same folders, even if data have been moved. It is why it is very important that when data are retrieved, they are recopied into their original folder.
Meanwhile, Nodeum offers different options to change this behavior and provides solutions to relocate the contents into another folder structure.
Each task finishes with a status, each task status determines the result of the processing that has been executed. This following table describes each type of status.
