# Data Mover Management

The centralized dashboard enables users to monitor task progress with real-time updates on data status. Users can define unlimited tasks simultaneously to move, archive, and protect data across various storage tiers. 

Task management features provide control and visibility.

To start, first create a task

 and choose a workflow 

1. <mark style="color:blue;">Active Archive</mark>
2. <mark style="color:blue;">Offline Archive</mark>
3. <mark style="color:blue;">Data Exchange</mark>
4. <mark style="color:blue;">Maintenance</mark>

Each workflow has its own settings:

* [ ] <mark style="color:green;">Actions</mark>
* [ ] <mark style="color:green;">Source</mark>
* [ ] <mark style="color:green;">Destination</mark>
* [ ] <mark style="color:green;">Granular File Selection</mark>
* [ ] <mark style="color:green;">Scheduler</mark>
* [ ] <mark style="color:green;">Filter</mark>&#x20;
* [ ] <mark style="color:green;">Advanced Setting : Priority, Metadata Management, …</mark>

### Task Filtering

Each workflow task allows for the configuration of various filtering options. There are two available modes for filter configuration:

* Basic&#x20;
* Advanced

#### **Basic Mode** <a href="#filtering-basicmode" id="filtering-basicmode"></a>

This mode includes the capability to define expressions directly in the Console based on the following types:

<table><thead><tr><th width="206.33333333333331">Type of Filter</th><th width="215">Operator</th><th>Description</th></tr></thead><tbody><tr><td>Creation Date</td><td>Older Than<br>Earlier Than<br>Before<br>After </td><td><p>Filter the selection on when the file has been created. </p><p>It allows to select only files that have been created before or after a specific date.</p></td></tr><tr><td>File Name</td><td>Matches<br>Doesn't Match</td><td><p>It allows to filter the selection based on the name of the file.</p><p>Examples :<br>'.' for any type of character <br>'*' from 0 to infinite</p><p><em>Regex expressions are also supported.</em></p></td></tr><tr><td>File extension</td><td>Equal<br>Doesn't Match</td><td><p>It allows to filter your selection based on the file extension. </p><p>File extension name has to be defined. Example: pdf</p></td></tr><tr><td>File Path</td><td>Matches<br>Doesn't Match</td><td><p>It allows to filter your selection based on the folder(s) path.</p><p><em>Regex expressions are also supported.</em></p></td></tr><tr><td>Last access date</td><td>Older Than<br>Earlier Than<br>Before<br>After </td><td>Same but with a trigger on when the file has been accessed for the last time.</td></tr><tr><td>Last modification date</td><td>Older Than<br>Earlier Than<br>Before<br>After </td><td><p>Filter the selection on when the file has been modified for the last time. </p><p>It allows to select only files that have been modified before or after a specific date.</p></td></tr><tr><td>Size</td><td>More than<br>Less than</td><td>Filter the file selection based on the files size</td></tr></tbody></table>

#### &#x20;

#### **Advanced Mode** <a href="#filtering-advancedmode" id="filtering-advancedmode"></a>

This mode allows the usage of an online scripting editor to define advances filter which cannot be defined in using the simple graphical mode.

The scripting Language is "LUA" , which a known language , with online documentation available.

The scripting language supports :

* Logical Operator such as 'and' , 'or'
* Defined Functions

**Example of a LUA script**

```bash
--[[
This function is auto-generated.
By editing it, you might lose your filter configuration
when going back to "Basic" mode
--]]
function test ( file )
return
       filter_size(file, ">", 2344000)
       or
       filter_name(file, "\\b(\\w*archive\\w*)\\b")
       or
       filter_path(file, "\\b(\\w*archive\\w*)\\b")
       or
       filter_relative_date(file, "C", ">", 20, "D")
       or
       filter_relative_date(file, "M", ">", 20, "D")
       or
       filter_relative_date(file, "A", "<", 20, "D")
end
```

&#x20;

#### Standard filtering methods are available :

**Filtering by the size of the file**&#x20;

**Example of LUA code**

```bash
...
filter_size(file, ">", 2344000)
..
```

<table><thead><tr><th width="212">Result</th><th>The workflow will only process files with a size which is bigger than 234000KB</th></tr></thead><tbody><tr><td>file</td><td>Match the process file</td></tr><tr><td>">"</td><td>Comparison symbol</td></tr><tr><td>234000</td><td>File size</td></tr></tbody></table>

&#x20;

**Filtering by the name of the file**&#x20;

**Example of LUA code**

```bash
...
filter_path(file, "\\b(\\w*archive\\w*)\\b")
..
```

| **Results**                     | **The workflow will only process files which are located in folder with a name which contains "archive"** |
| ------------------------------- | --------------------------------------------------------------------------------------------------------- |
| file                            | Match the process file                                                                                    |
| "\\\b(\\\w\*archive\\\w\*)\\\b" | Regex expression                                                                                          |

&#x20;

**Filtering by the creation date**&#x20;

**Example of LUA code**

```bash
...
filter_relative_date(file, "C", ">", 20, "D")
...
```

&#x20;

| **Result** | **The workflow will only process files with a creation date older or earlier than**     |
| ---------- | --------------------------------------------------------------------------------------- |
| file       | Match the process file                                                                  |
| "C"        | Relative to the Creation Date                                                           |
| ">"        | ">", "<" are available for (older than, earlier than)                                   |
| 20         | Number of Year, Month, Day, Hour, Minute, Second                                        |
| "D"        | "Y", "M", "D", "H", "M", "S" are available for (Year, Month, Day, Hour, Minute, Second) |

&#x20;

**Filtering by the modification date**

**Example of LUA code**

```bash
...
filter_relative_date(file, "C", ">", 20, "M")
...
```

| **Result** | **The workflow will only process files with an access date older or earlier than**      |
| ---------- | --------------------------------------------------------------------------------------- |
| file       | Match the process file                                                                  |
| "C"        | Relative to the Last Access Date                                                        |
| ">"        | ">", "<" are available for (older than, earlier than)                                   |
| 20         | Number of Year, Month, Day, Hour, Minute, Second                                        |
| "D"        | "Y", "M", "D", "H", "M", "S" are available for (Year, Month, Day, Hour, Minute, Second) |

Some addition information which can be helpful for using the feature :

* LUA WebSite : <https://www.lua.org/>
* LUA Online Script validation <https://www.tutorialspoint.com/execute_lua_online.php>
* REGEX Online validation : <https://regexr.com/>

### Data Protection / Offline Archive&#x20;

#### Definition

Data Protection or Offline Archiving task secures selected data on a dedicated storage tier. Data accessibility is managed by the administrator behind the workflow. It allows only the data to be read or retrieved online.&#x20;

Depending on the medium chosen, data can be physically moved off-site. This requires human intervention to retrieve the data online.&#x20;

Users have the option of running an offline workflow that will copy or move the data to a secondary storage pool.

#### <mark style="background-color:green;">Why to use this workflow?</mark>

1. Protect stored content that is online (online means "modifiable" by users or applications). Workflow allows these data to be copied according to defined criteria.
2. Archive data on a disconnected media, with the possibility of get it online - offsite.

#### Traceability

A complete traceability is kept. &#x20;

User/administrator have an overview of their data mobility even if the data has been secure offline.

In case of tape library, data can even be move out of the library (off site).  All contents are stored in open format, under common standards.

#### Available Criteria

<table><thead><tr><th width="226">Source</th><th width="255">Destination</th><th>Type of action  Mover </th></tr></thead><tbody><tr><td><p>Primary Storage :</p><ul><li>Nas</li><li>Object Storage</li></ul></td><td><p>Secondary Storage :</p><ul><li>Tape Pools</li><li>Cloud Pools</li><li>Nas Pools</li></ul></td><td><p></p><ol><li>Copy</li><li><a data-footnote-ref href="#user-content-fn-1">Move</a></li><li>Erase</li></ol><p></p></td></tr><tr><td>Container </td><td><p>Secondary Storage :</p><ul><li>Tape Pools</li><li>Cloud Pools</li><li>Nas Pools</li></ul></td><td><ol><li>Copy</li><li>Move</li><li>Erase</li></ol></td></tr><tr><td>Restore Source</td><td>Destination </td><td><ol><li>Move </li></ol></td></tr><tr><td>Tape Pools</td><td><ul><li>Container (cache)</li><li>Primary Storage (NAS or Object Storage)</li></ul></td><td><ol><li>Copy</li><li>Move</li></ol></td></tr><tr><td>Cloud Pools</td><td><ul><li>Container (cache)</li><li>Primary Storage (NAS or Object Storage)</li><li>Cloud Pool</li></ul></td><td><ol><li>Copy</li><li>Move</li></ol></td></tr><tr><td>Nas Pools</td><td><ul><li>Container (cache)</li><li>Primary Storage (NAS or Object Storage)</li><li>Nas Pool</li></ul></td><td><ol><li>Copy</li><li>Move</li></ol></td></tr><tr><td>Task</td><td><ul><li>Container (cache)</li><li>Primary Storage (NAS or Object Storage)</li></ul></td><td><ol><li>Copy</li><li>Move</li></ol></td></tr><tr><td>From Reference File</td><td><ul><li>Container (cache)</li><li>Primary Storage (NAS or Object Storage)</li></ul></td><td><ol><li>Copy</li><li>Move</li></ol></td></tr></tbody></table>

#### Specific Option

| Option                       | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Remove also selected folders | <p></p><p>This option lets you delete the original folder structure after a <strong>Move</strong> operation.</p><ul><li>It is only available in an <strong>Offline Archive – Move</strong> workflow from <strong>primary storage to tape storage</strong>.</li><li>By default, folders are kept in place, but their contents are moved (leaving them empty).</li><li>If you enable <strong>“Remove also selected folders”</strong>, both the files <strong>and</strong> their original folders will be removed from primary storage.</li></ul> |

### Data Archiving / Active Archive

{% hint style="info" %}
Before to use this workflow, make sure to configure the container and cache.  Learn more to [Container configuration section](broken://pages/e65MpRrigCgw9EoW4wYz)
{% endhint %}

#### Definition

To archive a collection of data that can be moved from an expensive primary storage tier to a secondary storage tier. The data may or may not be removed from primary storage and must remain accessible from time to time. Data archiving allows data to be stored that does not require intensive reading and writing, but can be modified or read from time to time.

#### <mark style="background-color:green;">Why to use this workflow?</mark>

Data is stored on less expensive storage media, such as object storage, tape storage or even low-cost disks. An active archive includes functionality to seamlessly move data between different storage tiers (disk - tape - cloud) so that applications or users can access it directly.

{% hint style="info" %}
This will provide movement of files from where files are stored to the destination.

* If files are in the Cache, the files are directly copied into the destination.
* If files are in another “Active Archive” Secondary Storage, the files are recovered and copied into the cache and then copied into the final destination.
  {% endhint %}

For a file directory structure, there is no change on the file presentation. Each file stay in the same folder.  Nodeum uses a Virtual File System and an internal mechanism stubbing. This means that the files always pass through the caching disk.

Data can be directly copied into the Container (SMB/NFS/S3) and then defined workflow will execute task movement linked to this container.

{% hint style="success" %}
You can migrate data efficiently by using an Active Archive Workflow, moving it from Primary Storage
{% endhint %}

## Task Monitoring

To aid in monitoring task execution, the Console offers a comprehensive monitoring feature. This feature provides a global overview of all task activities.

Three different sections are available :&#x20;

#### Upcoming task

This section shows the task(s) scheduled.  For each task, you can see detailed information including the date - time when the task will be executed.&#x20;

It is possible to skip an execution in clicking in the right cross icon.

If a task hasn't been executed properly, then you will an information saying that "task execution missed for one or multiple occurence".

<figure><img src="/files/h6TaTWFA9P3WVfczj0uM" alt=""><figcaption></figcaption></figure>

#### Running task

This section provides the list of the task(s) which are currently running. &#x20;

For each of these tasks, you can

1. See the detail and the status.
2. Pause the task execution.
3. Resume a paused task .
4. Stop the task execution

#### History

In this section you can find all the tasks that have been completed.

<figure><img src="/files/K5K2PNJIYSz1zyjol0p5" alt=""><figcaption><p>List of tasks history</p></figcaption></figure>

## Task Report&#x20;

For each task running or finished, you can open to see the global view about the task.

To see the full informations click on the arrow beside the task:

<figure><img src="/files/ecFkdKPgXQcFY56zFlYh" alt=""><figcaption><p>List of Nodeum's tasks</p></figcaption></figure>

And have a look on the task information:

<figure><img src="/files/HfnSbNyXrNcyvdtKols3" alt=""><figcaption><p>Screenshot about a task execution</p></figcaption></figure>

Definitions of task informations :&#x20;

| Informations    | Definition                                                                                                                                     |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Timing          | <ol><li>Task starts the x date and at x hours</li><li>Elapsed time for the task</li><li>Estimated remaining time to process the task</li></ol> |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| Type            | <ol><li>Offline Archive</li><li>Active Archive</li><li>Data Exchange</li></ol>                                                                 | <p>Move or Copy to a destination:</p><p><img src="https://mt-c-storage.atlassian.net/wiki/download/attachments/1007681564/image2018-3-19_10-50-42.png?version=1&#x26;modificationDate=1547218570840&#x26;cacheVersion=1&#x26;api=v2" alt=""> Network Shared Folder named Container</p><p><img src="https://mt-c-storage.atlassian.net/wiki/download/attachments/1007681564/image2018-3-19_10-51-57.png?version=1&#x26;modificationDate=1547218571533&#x26;cacheVersion=1&#x26;api=v2" alt=""> Nas</p><p><img src="https://mt-c-storage.atlassian.net/wiki/download/attachments/1007681564/image2018-3-19_10-51-17.png?version=1&#x26;modificationDate=1547218571188&#x26;cacheVersion=1&#x26;api=v2" alt=""> Tape</p> |
| Processed size  | size of data processed                                                                                                                         |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| Processed items | Number of items of data processed                                                                                                              | To see the good process click on the tab processed items\*                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| Logs            | Details of task's log                                                                                                                          |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

{% hint style="success" %}
Each task report  contains all files processed during the task, a downloaded copy can be exported.
{% endhint %}

### Task Log

Below is a table listing messages found in the logs, mostly arranged in the

| Message                                                                                          | Alternative value                                                                             | Definition                                                                                                                                                                                                                                                       |
| ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Task is starting                                                                                 | n/a                                                                                           | The task starts.                                                                                                                                                                                                                                                 |
| Parallel Processing mode is Enabled                                                              | Parallel Processing mode is Disabled                                                          | In Enabled mode, all processes run concurrently, unlike in Disabled mode where processes run sequentially. Disabled mode is useful                                                                                                                               |
| Initialization Process is starting                                                               | n/a                                                                                           | The process in charged of the inventory is started.                                                                                                                                                                                                              |
| License is valid                                                                                 | License is not valid (`'description of the error'`), task cannot be executed                  | It checks the validity of the Nodeum licence is valid.                                                                                                                                                                                                           |
| Task Job Limits : 100 files/job - 2 parallel job(s) - 200 parallel DBFile/job - 1000000000 B/job | <p># parallel job(s)<br># parallel DBFile/job<br># B/job</p>                                  | It defines the number of files to be processed per batch. This example means that the task will included 2 jobs; each of them will contains maximum 200 files or a size of maximum 1GB. When one of these values is reached, then another batch will be defined. |
| Incremental Mode is Configured. The process will select only new content                         | Full Mode is Configured.                                                                      | The process will select only new content or will select all files to be processed.                                                                                                                                                                               |
| Processing Order is Configured. (Ascending)                                                      | Processing Order is Configured. (Descending)                                                  | This manages the order of files processing. Best performance is Ascending.                                                                                                                                                                                       |
| Check integrity of data transfers: MD5                                                           | <p>Check integrity of data transfers: CRC32<br>Check integrity of data transfers: XXHAS64</p> | The type of integrity check is displayed.                                                                                                                                                                                                                        |
| Pool `pool name`: Tape `tape barcode`: VALID                                                     | n/a                                                                                           | This verifies if the associated pool contains a valid tape.                                                                                                                                                                                                      |
| Pool `pool name:`1 valid tape(s)                                                                 | n/a                                                                                           | This summarizes the number of valid tapes in the associated pool.                                                                                                                                                                                                |
| Main Process is starting                                                                         | n/a                                                                                           | The copy process starts.                                                                                                                                                                                                                                         |
| Finalization Process is starting                                                                 | n/a                                                                                           | The process responsible of managing the catalog changes starts.                                                                                                                                                                                                  |
| Tape `tape barcode`: is Mounted on the drive `drive s/n`                                         | n/a                                                                                           | Indicates a request to mount a tape in a tape drive.                                                                                                                                                                                                             |
| Tape `tape barcode`loaded on the drive `drive s/n` is Dismounted                                 | n/a                                                                                           | Indicates a request to dismount a tape in a tape drive.                                                                                                                                                                                                          |
| LTFS File System of the Tape `tape barcode`loaded on the drive `drive s/n` is Mounted            | n/a                                                                                           | The LTFS partition is mounted for the specified tape in the specified tape drive.                                                                                                                                                                                |
| LTFS File System of the Tape `tape barcode`loaded on the drive `drive s/n`is Umounted            | n/a                                                                                           | The LTFS partition is unmounted for the specified tape in the specified tape drive.                                                                                                                                                                              |
| Unable to Mount LTFS File System of the Tape `tape barcode`loaded on the drive `drive s/n`       | n/a                                                                                           | The tape cannot mount its LTFS file system properly.                                                                                                                                                                                                             |
| Unable to Copy a File. (Read-Only LTFS FileSystem, Try to mount it again)                        | n/a                                                                                           | <p>Indicates that the tape is mounted with Read-Only LTFS file system and can't copy the file. The task will remount another tape.<br><mark style="background-color:red;">It is highly recommended to check the tape manually.</mark></p>                        |
| Total of items/size parsed: `xxxx`(`yyy`TB)                                                      | n/a                                                                                           | This list the result of the Initialization process and list the number of files and size to be processed by the task.                                                                                                                                            |
| Total of items/size already up to date on the destination: `xxxx`(`yyy`TB)                       | n/a                                                                                           | This is the result of the verification of the number of files and the size already available at the destination.                                                                                                                                                 |
| Total of items/size to process on this schedule: `xxxx`(`yyy`TB)                                 | n/a                                                                                           | This is the summary of what will be processed during this task execution.                                                                                                                                                                                        |
| Initialization Process is finished                                                               | n/a                                                                                           | This confirms that the Initialization process is finished and then files inventory is available.                                                                                                                                                                 |
| All Files were processed on this Rule                                                            | n/a                                                                                           | This confirms that all files have been processed.                                                                                                                                                                                                                |
| Main Process is finished                                                                         | n/a                                                                                           | The Main process is finalized.                                                                                                                                                                                                                                   |
| Retry made on volume `tape barcode`: `xx`                                                        | n/a                                                                                           | Shows the number of mount retries for all the defined tape barcode.                                                                                                                                                                                              |
| Finalization Process is finished                                                                 | n/a                                                                                           | The Finalization process is finalized.                                                                                                                                                                                                                           |
| Task is finished with Success                                                                    | n/a                                                                                           | The Task is finalized.                                                                                                                                                                                                                                           |

[^1]: “Remove also selected folder” option is available for removing also the file structure after a move operation.

    This option is only available for an Offline Archive - **Move** workflow from primary storage to tape storage


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.nodeum.io/data-archiving/console/data-mover-management.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.