/config.json) [$ND_CONFIG]
--config-dir value, -C value path to configuration folder (default: "/home/valery/.config/.nd") [$ND_CONFIG_DIR]
--alias value alias in configuration file for authentication (default: "default") [$ND_ALIAS]
--url value URL of Nodeum [$ND_URL]
--auth-type value Auth type (empty or ldap) [$ND_AUTH_TYPE]
--access-token value for API authentication (1st authentication method) [$ND_ACCESS_TOKEN]
--refresh-token value for API authentication (1st authentication method, not saved in config) [$ND_REFRESH_TOKEN]
--authorization-endpoint value for Device Authorization Flow (2nd authentication method)
--token-endpoint value for Device Authorization Flow (2nd authentication method)
--client-id value for Device Authorization Flow (2nd authentication method)
--scopes value for Device Authorization Flow (2nd authentication method)
--persist-session persist Device Authorization session on disk for 1 hour (default: true)
--persist-session-renew if persist session is enabled, renew the token (default: false)
--username value for API authentication (3rd authentication method) [$ND_USERNAME]
--password value for API authentication (3rd authentication method) [$ND_PASSWORD]
--anonymous no login (default: false)
--help, -h show help
--version, -v print the version
COPYRIGHT:
nd is a property of Nodeum and its subsidiaries, if any. The intellectual and
technical concepts contained herein are proprietary to Nodeum and its
subsidiaries and may be covered by Belgium and Foreign Patents, patents in
process, and are protected by trade secret or copyright law. Dissemination of
this information or reproduction of this material is strictly forbidden unless
prior written permission is obtained from Nodeum.
```
The following table lists **nd** commands:
| Name | Shortcut | Description |
| ------ | -------- | ------------------------------------------------------------------ |
| admin | | Access the administration commands |
| config | | Configure the Nodeum client |
| copy | cp | Create a task to copy data between two storages |
| move | mv | Create a task to move data between two storages |
| task | | List detailed information about created task |
| pool | | list, destroy available pools(for admin only) |
| help | h | Displays a summary of command usage and parameters on the terminal |
### Parameters
#### **Syntax**
The **nd** client provides a Bash completion mechanism to facilitate the search of commands.
Metadata key can't include an = character.
#### **Global Parameters**
| Name | Shortcut | Description | Default |
| ------------------------------ | -------- | ----------------------------------------------------------------------- | -------------------------------------- |
| --json | | Output as JSON | false |
| --debug | | Enable verbose logging (shows INFO and ERROR level logs) | false |
| --config value | -C value | Path to configuration file | /home/nodeum01/.config/.nd/config.json |
| --config-dir value | -C value | Path to configuration folder (default: "/home/nodeum01/.config/.nd") | |
| --alias value | | Alias in configuration file for authentication | default |
| --url value | | URL of Nodeum | |
| --access-token value | | for API authentication (1st authentication method) | |
| --refresh-token value | | for API authentication (1st authentication method, not saved in config) | |
| --authorization-endpoint value | | for Device Authorization Flow (2nd authentication method) | |
| --token-endpoint value | | for Device Authorization Flow (2nd authentication method) | |
| --client-id value | | for Device Authorization Flow (2nd authentication method) | |
| --scopes value | | for Device Authorization Flow (2nd authentication method) | |
| --persist-session | | persist Device Authorization session on disk for 1 hour | true |
| --persist-session-renew | | if persist session is enabled, renew the token | false |
| --username value | | for API authentication (3rd authentication method) | |
| --password value | | for API authentication (3rd authentication method) | |
| --anonymous | | no login | false |
| --help | -h | show help | false |
| --version | -v | print the version | false |
**Mandatory Parameters for OpenID integration**
Parameters are available for each data movement task.
| Name | Description | Value |
| --------------------------- | ------------------------------------------------- | ------ |
| --md project\_name= | Name of project defined in OpenStack | string |
| --md project\_domain\_name= | Name of the project's domain defined in OpenStack | string |
| --md user\_domain\_name= | Name of the project's user defined in OpenStack | string |
| --md region\_name= | Name of the region's project defined in OpenStack | string |
**Standard Parameters**
Parameters are available for each data movement task.
| Name | Shortcut | Description | Value | Default value |
| ----------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------- | ---------------- | ----------------------- |
| --help | -h | Show help | | false |
| --no-run | | Create a task and don't launch the task directly | | false |
| --name value | -n | Name of task | string | automatically generated |
| --comment value | | Comment of task | string | empty |
| --overwrite value | | Overwrite all identical files already stored at destination | true - false | false |
| --priority value | | Priority of the task, between 0 and 9 (0 is the highest priority) | 0 - 9 | 0 |
| --recursive | -R | Execute a recursive copy of the folder. If subfolders are present, the service will also copy the contents of each subfolder | | false |
| --working-dir value | --wd | Defines the workpath to be kept at destination | . - .. - path | 0 |
| --ignore-hidden value | | Task will not handle hidden file(s) | true - false | false |
| --progress value | | Display live progress when running the task | true - false | true |
| --processed-nodes value | | Display the processed nodes when running a task when the flag--progress is set | none, error, all | error |
**Advanced Parameters**
Parameters are available for each data movement task.
| Name | Shortcut | Description | Value | Default value |
| ---------------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------- | -------------- | ------------- |
| --context-uid value | --uid | Define the User ID which will handle the movement | integer | unset |
| --context-gid value | --gid | Define the Group ID which will handle the movement | integer | unset |
| --defer | | when requesting the run of the task, will defer it for later with an unique ID | | false |
| --parallel value | | Define the number of mover which will handle the movement. Maximum value is determined by the deployed implementation. | 1-20 | 1 |
| --callback type | | Add callback. Format is type:./path/to/file | ./path/to/file | |
| --trigger-md key=value | --md key=value | Set metadata on the trigger. Format is key=value. Accepts multiple inputs | key=value | |
| --task-md key=value | | Set metadata on the task. Format is key=value. Accepts multiple inputs | key=value | |
| --files-md key=value | | Set metadata on the files. Format is key=value. Accepts multiple inputs | key=value | |
| --run-as-user=username | | Run the task as a different user | | unset |
### Filter Parameters
Filters allow you to include or exclude files based on various criteria during copy/move operations. Filters are executed on the mover side before processing files, reducing unnecessary transfers.
#### **Filter Logic**
* **Include filters** of the same type are **OR'd** (match any)
* **Exclude filters** and different filter types are **AND'd** (all must pass)
#### **Available Filter Flags**
| Category | Flags | Description |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- |
| **Path/Name** | --include-path=REGEXP
--exclude-path=REGEXP
--include-filename=REGEXP
--exclude-filename=REGEXP
| Filter by full path or filename using regular expressions |
| **Extension** | --include-ext=".pdf"
--exclude-ext=EXT
| Filter by file extension (e.g., `.pdf`, `.tmp`) |
| **Size** | --size-more-than=SIZE
--size-less-than=SIZE
| Filter by file size (supports KB, MB, GB, KiB, MiB, GiB) |
| **Modification Time** | --mtime-older-than=DURATION
--mtime-earlier-than=DURATION
--mtime-before=DATE
--mtime-after=DATE
| Filter by modification date |
| **Change Time** | --ctime-older-than=DURATION
--ctime-earlier-than=DURATION
--ctime-before=DATE
--ctime-after=DATE
| Filter by change date |
| **Access Time** | --atime-older-than=DURATION
--atime-earlier-than=DURATION
--atime-before=DATE
--atime-after=DATE
| Filter by access date |
| **Metadata** | --include-metadata=KEY=REGEXP
--exclude-metadata=KEY=REGEXP
| Filter by file metadata/xattrs |
**Duration Units**
* `h` or `hour` - Hours
* `d` or `day` - Days
* `w` or `week` - Weeks
* `M` or `month` - Months
* `Y` or `year` - Years
**Filter Examples**
**Copy PDFs larger than 10MB, modified more than 30 days ago:**
```
nd copy \
--include-ext=".pdf" \
--size-more-than="10MB" \
--mtime-older-than="30d" \
--recursive \
/data/archive /backup
```
**Exclude temporary and log files:**
```
nd copy \
--exclude-ext=".tmp" \
--exclude-ext=".log" \
--exclude-path=".*node_modules.*" \
--recursive \
/data/project /backup
```
**Copy only images modified in the last week:**
```
nd copy \
--include-ext=".doc" \
--include-ext=".docx" \
--size-more-than="1MB" \
--mtime-before="2024-01-01" \
--exclude-path=".*draft.*" \
--recursive \
/data /archive
```
**Complex filter combining multiple criteria:**
**TO DO**
### Configuration
**nd** uses a JSON formatted configuration file used for storing certain kinds of information, such as the authentication and authorization options. By default, this configuration file is unique by user. It is stored in its home directory.
For Linux and macOS, the default configuration file location is `.config/.nd/config.json` which is store in the `$HOME`. For Windows, the configuration file is stored in `$AppData$`.
You can display the configuration file location in using the command `nd --help`:
```
[nodeum01@]$ nd --help
NAME:
nd - Nodeum CLI
....
--config value path to configuration file (default: /config.json) [$ND_CONFIG]
--config-dir value, -C value path to configuration folder (default: "/home/nodeum01/.config/.nd") [$ND_CONFIG_DIR]
```
You can use the `--config value` where value is the path to a JSON formatted configuration file that **nd** uses for storing data. The `ND_CONFIG` environment variable can be used to set the value.
Store the configuration file in a 'central' directory to allow each user to get the same **nd** client configuration. For this, the `--config-dir value` is available.
**Command**
| Command | Description |
| -------------------- | -------------------------------------------------------------------------- |
| `--config value` | this option specifies the JSON filename where the configuration is stored |
| `--config-dir value` | this option specifies the directory where the JSON configuration is stored |
#### SSL
**nd** client allows SSL configuration to communicate with the Data Mover service which listen in HTTPS. It is required to add the public certificate generated with the server.
The public certificate file has to be stored in this `config-dir` folder: `.config/.nd/certs/CAs/`.
The certificates on the server side have been generated following this command:
```
openssl11 req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
\-keyout ./nodeum_certs/private.key \
\-out ./nodeum_certs/public.crt \
\-subj "/CN=nodeum-qualif.mt-c.local" \
\-addext "subjectAltName=DNS:nodeum.domain.local,DNS:localhost,IP:127.0.0.1,IP:1.1.1.1"
```
Where `1.1.1.1` is the ip of the nginx interface and `nodeum.domain.local` is its hostname including domain name.
#### Authentication and Authorization
**Description**
Three authentifcation options are available:
* Username / Password
* IDP with OpenID
* authenticate via Auth Service if configured
**Command**
| Command | Description |
| ------------------------- | --------------------------------------------------------------- |
| `nd config save` | this command save the **nd** configuration for authentication |
| `nd config review` | this command display the information related to the credentials |
| `nd config clear-session` | clear persisted session |
**Username / Password configuration**
The **nd** client provides a basic method of authentication in using Username / Password credentials.
```
\[nodeum01@nodeum\]$ nd config save \
\--url=http:// \
\--username= \
\--password=
```
| Option | Description |
| ------------ | ----------------------------------------------------------------------------------------- |
| `--url` | this refer to the node which hosts the following service `DATA MANAGEMENT WEB SERVICES` |
| `--username` | this is the username to grant authorization to the service |
| `--password` | this is the password associated to the username that will grant the access to the service |
**OpenID configuration**
The **nd** client provides an OpenID authentication mechanism. In this case, **nd** Client has to be configured with the appropriated IDP to handle proper token management.
The basic configuration is the following one:
```
\[nodeum01@nodeum\]$ nd config save \
\--url=http:// \
\--authorization-endpoint= \
\--token-endpoint= \
\--client-id= \
\--scopes="openid,profile,offline_access,email"
```
| Option | Description |
| -------------------------- | --------------------------------------------------------------------------------------- |
| `--url` | this refer to the node which hosts the following service `DATA MANAGEMENT WEB SERVICES` |
| `--authorization-endpoint` | this is the endpoint url to grant authorization to the service |
| `--token-endpoint` | this url is used to programmatically request tokens |
| `--client-id` | this refer to the client identifier which is provided by the OpenID provider |
| `--persist-session` | persist Device Authorization session on disk for 1 hour (default: true) |
| `--persist-session-renew` | if persist session is enabled, renew the token (default: false) |
The standard behavior is to request a token, which is automatic if there is no token available. The token will be stored in a cache during 15 minutes. The `--persist-session-renew` option can be defined to true to force a request token process any time the user has to interface with the **nd** client.
The token renewal is automatic based on the `renewal token`.
#### Authentication via Auth Service
If the **Auth Service** is configured, authentication will be handled automatically. The Auth Service runs as a `systemd` service and must be properly set up by a privileged/admin user. Once configured, normal users can authenticate via the Auth Service through a Unix socket, without having to manage authentication themselves.
The Auth Service communicates with the Nodeum server to create new Linux users or update existing passwords as needed.
#### Admin/Privileged User Setup
* If the Auth Service is installed via **`.rpm` or `.deb` package**:
* The service file is automatically installed and the service will be restarted.
* The only manual step required is to set the configuration directory: `export ND_CONFIG_DIR=/etc/nodeum/config`
* Then authenticate to Nodeum using **username/password** or another supported method (e.g., **OpenID**). with above method mentioned
* You may also define an alias for the Auth Service (or leave it empty to use the default).
* If the Auth Service is **not installed via package**:
* The admin must manually create the service file and configure everything.
#### Normal User Workflow
* A user simply runs commands like:
`nd cp ...` or `nd mv ...`
* The CLI checks whether the Auth Service is running.
* If it is, the CLI communicates with the Auth Service over the Unix socket to automatically authenticate the user (by creating/updating their username and password).
#### Multiple Clusters / Aliases
* If the Auth Service is configured with multiple clusters or aliases, users will be prompted to choose which alias to connect with.
* Example error when no default alias is set:
```
INFO Auth service is running
FATA No default alias found in auth service.
Please specify an alias using one of the following methods:
• nd --alias=
• nd config --alias=
Available aliases: cluster1, cluster2, cluster3
```
* Once a user sets an alias, they are authenticated for that cluster.
* To switch to another cluster, simply pass a different alias.
* If the Auth Service has a **default alias** configured, users will automatically be authenticated with that alias without being prompted.
#### **LDAP Server Configuration**
**nd server-config ldap**
Configure LDAP settings on the Nodeum server.
**Commands:**
| Command | Description |
| --------------------------- | ------------------------------ |
| `nd server-config ldap get` | Get current LDAP configuration |
| `nd server-config ldap set` | Set LDAP configuration |
**Note:** Requires admin privileges.
**Example:**
```
# Get LDAP configuration
nd server-config ldap get
# Set LDAP configuration
nd server-config ldap set [options]
```
### Alias & default flags
**Description**
Alias & default flags allow structuration of different group of settings. Different alias can be defined in the configuration file.
**Definition of alias & default flags**
Alias and flags are declared in the configuration file `~/.config/.nd/config.json"`. Default flags are defined for each available parameter command. Flags can be overwritted in the **nd** command.
Example in config file:
```
{
// ...
// default: name of alias used by default
"default": {
// ...
"default_flags": {
// this is the name of the command, example: nd copy
"copy": {
// this is the flag, example: nd copy --md project_domain_name=MYPROJECTDOMAIN --md user_domain_name=MYUSERDOMAIN
"md": ["project_domain_name=MYPROJECTDOMAIN", "user_domain_name=MYUSERDOMAIN"]
}
}
},
// myorganisation: name of another alias
"myorganisation": {
// ...
}
}
```
**Alias usage**
The nd command allows the usage of alias, example : `nd --alias myorganisation copy`
### Data Mover Service Status
#### Command
| Command | Description |
| ----------------- | ----------------------------------------------------------------------- |
| `nd admin status` | retrieve the status and the health of each services part of the cluster |
| `nd admin logs` | retrieve all Logs and return their contents |
#### Data Management Monitoring Services
`nd admin status` command requests the Data Management Monitoring service to retrieve the status and health of each services part of the cluster. This command returns list of services. The following information are displayed:
* Service Status
* Service version
* Host where the service is deployed
* Its uptime
* Consumed memory
**Output where all services are reachable**
```
╭───┬───────────────────┬────────────────────────────┬─────────────────┬────────────┬───────────╮
│ │ NAME │ VERSION │ ADDRESS │ UPTIME │ MEMORY │
├───┼───────────────────┼────────────────────────────┼─────────────────┼────────────┼───────────┤
│ ● │ micro.http.broker │ ff.http.broadcast │ 10.2.2.83:50050 │ N/A │ N/A │
│ ● │ nodeum.dispatcher │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.84:50010 │ 172h11m8s │ 34.92 MB │
│ ● │ nodeum.dispatcher │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50010 │ 172h13m32s │ 22.61 MB │
│ ● │ nodeum.front │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50000 │ N/A │ N/A │
│ ● │ nodeum.monitoring │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.85:50060 │ 171h2m47s │ 17.20 MB │
│ ● │ nodeum.mover │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.84:50020 │ 172h4m48s │ 25.23 MB │
│ ● │ nodeum.mover │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.85:50020 │ 172h4m48s │ 25.63 MB │
│ ● │ nodeum.scheduler │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50040 │ 172h18m6s │ 6.91 MB │
│ ● │ mongo │ 4.4.17 │ 10.2.2.83:27017 │ 268h0m44s │ 212.86 MB │
│ ● │ mongo │ 4.4.17 │ 10.2.2.84:27017 │ 268h0m44s │ 212.86 MB │
│ ● │ mongo │ 4.4.17 │ 10.2.2.85:27017 │ 268h0m44s │ 212.86 MB │
│ ● │ mysql │ 10.6.11-MariaDB │ 10.2.2.84:3306 │ 150h21m0s │ N/A │
│ ● │ redis-sentinel │ 6.2.6 │ 10.2.2.83:26379 │ 150h16m52s │ N/A │
│ ● │ redis-sentinel │ 6.2.6 │ 10.2.2.84:26379 │ 150h16m52s │ N/A │
│ ● │ redis-sentinel │ 6.2.6 │ 10.2.2.85:26379 │ 150h16m52s │ N/A │
│ ● │ redis │ 6.2.6 │ 10.2.2.83:6379 │ 285h37m50s │ N/A │
│ ● │ redis │ 6.2.6 │ 10.2.2.84:6379 │ 169h59m59s │ N/A │
│ ● │ redis │ 6.2.6 │ 10.2.2.85:6379 │ 169h59m59s │ N/A │
│ ● │ etcd │ 3.4.13 │ 10.2.2.83:2379 │ N/A │ N/A │
│ ● │ etcd │ 3.4.13 │ 10.2.2.84:2379 │ N/A │ N/A │
│ ● │ etcd │ 3.4.13 │ 10.2.2.85:2379 │ N/A │ N/A │
╰───┴───────────────────┴────────────────────────────┴─────────────────┴────────────┴───────────╯
```
**Output where some services are not available**
```
╭───┬───────────────────┬────────────────────────────┬─────────────────┬────────────┬───────────╮
│ │ NAME │ VERSION │ ADDRESS │ UPTIME │ MEMORY │
├───┼───────────────────┼────────────────────────────┼─────────────────┼────────────┼───────────┤
│ ● │ micro.http.broker │ ff.http.broadcast │ 10.2.2.83:50050 │ N/A │ N/A │
│ ● │ nodeum.dispatcher │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.84:50010 │ 172h11m8s │ 34.92 MB │
│ ● │ nodeum.dispatcher │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50010 │ 172h13m32s │ 22.61 MB │
│ ● │ nodeum.front │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50000 │ N/A │ N/A │
│ ● │ nodeum.monitoring │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.85:50060 │ 171h2m47s │ 17.20 MB │
│ ● │ nodeum.mover │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.84:50020 │ 172h4m48s │ 25.23 MB │
│ ● │ nodeum.mover │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.85:50020 │ 172h4m48s │ 25.63 MB │
│ ● │ nodeum.scheduler │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50040 │ 172h18m6s │ 6.91 MB │
│ ● │ mongo │ 4.4.17 │ 10.2.2.83:27017 │ 268h0m44s │ 212.86 MB │
│ ● │ mongo │ 4.4.17 │ 10.2.2.84:27017 │ 268h0m44s │ 212.86 MB │
│ ● │ mongo │ 4.4.17 │ 10.2.2.85:27017 │ 268h0m44s │ 212.86 MB │
│ ● │ mysql │ 10.6.11-MariaDB │ 10.2.2.84:3306 │ 150h21m0s │ N/A │
│ ● │ redis-sentinel │ 6.2.6 │ 10.2.2.83:26379 │ 150h16m52s │ N/A │
│ ● │ redis-sentinel │ 6.2.6 │ 10.2.2.84:26379 │ 150h16m52s │ N/A │
│ ● │ redis-sentinel │ 6.2.6 │ 10.2.2.85:26379 │ 150h16m52s │ N/A │
│ ● │ redis │ 6.2.6 │ 10.2.2.83:6379 │ 285h37m50s │ N/A │
│ ● │ redis │ 6.2.6 │ 10.2.2.84:6379 │ 169h59m59s │ N/A │
│ ● │ redis │ 6.2.6 │ 10.2.2.85:6379 │ 169h59m59s │ N/A │
│ ● │ etcd │ 3.4.13 │ 10.2.2.83:2379 │ N/A │ N/A │
│ ● │ etcd │ 3.4.13 │ 10.2.2.84:2379 │ N/A │ N/A │
│ ● │ etcd │ 3.4.13 │ 10.2.2.85:2379 │ N/A │ N/A │
│ ○ │ nodeum.mover │ │ │ │ │
╰───┴───────────────────┴────────────────────────────┴─────────────────┴────────────┴───────────╯
```
#### Data Management Log Management Services
`nd admin logs` command requests the Data Management Log Management Services to retrieve all Logs and return their contents.
```
[nodeum01@ ~]$ nd admin logs --help
NAME:
nd admin logs - display application logs
USAGE:
nd admin logs [command options] [arguments...]
OPTIONS:
--since value, -S value show logs not older than the specified date
--until value, -U value show logs not newer than the specified date
--tag value, -u value [ --tag value, -u value ] filter by tags
--level value minimum level of logs, one of [trace debug info warn error fatal]
--help, -h show help (default: false)
```
This command allows different parameters:
| Option | Description |
| ------ | ------------------------------------------------------------------------------------------ |
| since | to only show logs not older than a specified date |
| until | to only show logs not newer than a specified date |
| tag | to filter the logs per type of services. Example nodeum.monitoring |
| level | to define the minimum level of logs, one of \[trace - debug - info - warn - error - fatal] |
**Export**
The Logs can be exported in using the standard OS mechanism of exportation
```
./nd admin logs > 'nodeum_site-name_log.txt'
```
Where `nodeum_site-name_log.txt` is the file name and site represents the name of the site.
### Data Mover Task Management
Follow these steps to create a task:
1. Use Mandatory Parameters
2. Define the data source
3. Define the destination
4. Apply others parameters if needed
#### Task Creation
The `nd copy` command send a copy request to the data mover service from a storage A (nod://posix\_storage/) to a storage B ([nod-cloud://cloud\_storage/](https://nod-cloud/cloud_storage/)).
**Command with minimal parameters**
`nd copy \ --md project_name= \ --md project_domain_name= \ --md user_domain_name= \ nod://posix_storage/path/subpath/ \ nod-cloud://cloud_storage/container`
**Detailed syntax**
```
nd copy \
--md project_name= \
--md project_domain_name= \
--md user_domain_name= \
nod://posix_storage/path/subpath/ \
nod-cloud://cloud_storage/container
```
**Example of creation task with additional parameters**
`nd copy \ --md project_name= \ --md project_domain_name= \ --md user_domain_name= \ --working-dir nod://largedata2_pool/storagetestdata/ \ --recursive \ nod://posix_storage/path/subpath/ \ nod-cloud://cloud_storage/container`
**Detailed syntax**
```
nd copy \
--md project_name= \
--md project_domain_name= \
--md user_domain_name= \
--working-dir nod://posix_storage/path/ \
--recursive \
nod://posix_storage/path/subpath/ \
nod-cloud://cloud_storage/container
```
**Available Parameters**
| Option | Description |
| -------------------------------- | ----------------------------------------------------------------------------------------------- |
| `--callback type:./path/to/file` | add callback. Format is type:./path/to/file (accepts multiple inputs) |
| `--defer` | when requesting the run of the task, will defer it for later with an unique ID (default: false) |
| `--no-run` | just create the task, don't run it (default: false) |
| `--progress` | when running the task, display live progress (default: true) |
| `--recursive` | copy directories recursively (default: false) |
| `--ignore-hidden` | ignore hidden files and folders, starting with (default: false) |
| `--overwrite` | overwrite existing entries (default: false) |
| `--priority value` | task priority \[0..9] (default: 0) |
| `--working-dir value` | set working directory |
| `--remove-root-folder` | remove root folder (default: false) |
#### **Task creation from directly Absolute and Relative path.**
The ND client supports using mounted filesystem paths in addition to `nod://` URIs. The client automatically resolves these paths against configured NAS shares.
**Supported Path Formats:**
* Legacy format: `nod://pool-name/path/to/file`
* Mounted path: `/mnt/nas/path/to/file`
* Relative path: `./relative/path/to/file`
**How it works:**
1. Client fetches NAS share configurations from Nodeum server
2. Matches input path against share mount points using longest-prefix matching
3. Converts to `nod://poolname/relative/path` format internally
4. Absolute paths must match a configured share (error if not found)
5. Relative path to work file system have to be mounted where nd client runs to get absolute path of current working directory. and find a match against nas\_share path.
*NB: The Nodeum pool extractor needs to identify which storage pool a file belongs to by matching the file path against NAS share paths stored in the database. When passing **absolute paths or relative path** to the nd client, the NAS share must be mounted at the **exact same path** on the client machine as it's stored in the database - for example, if the database has `/p/projectspace`, you must mount it to `/p/projectspace` (not `/mnt/projectspace` or anything else). This way, when you access a file like `/p/projectspace/file.txt`, the extractor can match `/p/projectspace` to the database entry, determine the pool name, and correctly build the `nod://poolname/file.txt` URI scheme. If the mount point differs from the database path, the extractor cannot match them and fails to identify the pool.*
```
# by passing absolute path
nd copy /mnt/nas/data/file.txt /mnt/backup/archive/
# this only works if file is mounted which will get absolute path then c
nd copy ./data/file.txt /mnt/backup/
nd copy nod://mypool/data/file.txt nod://backup/archive/
```
**Working Directory Explanation**
The definition of a working directory allows to define where the files will be stored at the destination. Different options are available, they are described in the following definition.
With `--wd=.`
| Source | Destination | Result |
| ------------------------------ | ------------------------ | -------------------------------------- |
| `nod://source/folder/FILE.txt` | `nod://dest/directory/` | `nod://dest/directory/FILE.txt` |
| `nod://source/folder/FILE.txt` | `nod://dest/RENAMED.txt` | `nod://dest/RENAMED.txt` |
| `nod://source/folder/` | `nod://dest/directory/` | `nod://dest/directory/FILE.txt` |
| `nod://source/folder/` | `nod://dest/directory` | `nod://dest/directory/FILE.txt` |
| `nod://source/folder` | `nod://dest/directory/` | `nod://dest/directory/folder/FILE.txt` |
| `nod://source/folder` | `nod://dest/directory` | `nod://dest/directory/FILE.txt` |
With `--wd=..`
| Source | Destination | Result |
| ------------------------------ | ------------------------ | --------------------------------------------- |
| `nod://source/folder/FILE.txt` | `nod://dest/directory/` | `nod://dest/directory/folder/FILE.txt` |
| `nod://source/folder/FILE.txt` | `nod://dest/RENAMED.txt` | `nod://dest/RENAMED.txt` |
| `nod://source/folder/` | `nod://dest/directory/` | `nod://dest/directory/folder/FILE.txt` |
| `nod://source/folder/` | `nod://dest/directory` | `nod://dest/directory/FILE.txt` |
| `nod://source/folder` | `nod://dest/directory/` | `nod://dest/directory/source/folder/FILE.txt` |
| `nod://source/folder` | `nod://dest/directory` | `nod://dest/directory/FILE.txt` |
#### Examples of Tasks Creation
**Execute a task copy from Posix to Swift**
`nd copy \ --md project_name= \ --md project_domain_name= \ --md user_domain_name= \ --working-dir nod://posix_storage/path/ \ --recursive \ --ignore-hidden \ nod://posix_storage/path/subpath/ \ nod-cloud://cloud_storage/container`
**Detailed syntax**
```
nd copy \
--md project_name= \
--md project_domain_name= \
--md user_domain_name= \
--working-dir nod://posix_storage/path/ \
--recursive \
--ignore-hidden \
nod://posix_storage/path/subpath/ \
nod-cloud://cloud_storage/container
```
**Execute a task copy from Swift to Posix**
`nd copy \ --md project_name= \ --md project_domain_name= \ --md user_domain_name= \ --working-dir nod://posix_storage/path/ \ --recursive \ --ignore-hidden \ nod-cloud://cloud_storage/container/path/ \ nod://posix_storage/path/`
**Detailed syntax**
```
nd copy \
--md project_name= \
--md project_domain_name= \
--md user_domain_name= \
--working-dir nod-cloud://cloud_storage/container/ \
--recursive \
--ignore-hidden \
nod-cloud://cloud_storage/container/path/ \
nod://posix_storage/path/
```
**Execute a task move from Posix to Swift**
`nd move \ --md project_name= \ --md project_domain_name= \ --md user_domain_name= \ --working-dir nod://posix_storage/path/ \ --recursive \ --ignore-hidden \ nod://posix_storage/path/subpath/ \ nod-cloud://cloud_storage/container`
**Detailed syntax**
```
nd move \
--md project_name= \
--md project_domain_name= \
--md user_domain_name= \
--working-dir nod://posix_storage/path/ \
--recursive \
--ignore-hidden \
nod://posix_storage/path/subpath/ \
nod-cloud://cloud_storage/container
```
**Execute a task move from Swift to Posix**
`nd move \ --md project_name= \ --md project_domain_name= \ --md user_domain_name= \ --working-dir nod-cloud://cloud_storage/container/ \ --recursive \ --ignore-hidden \ nod-cloud://cloud_storage/container/path/ \ nod://posix_storage/path/`
**Detailed syntax**
```
nd move \
--md project_name= \
--md project_domain_name= \
--md user_domain_name= \
--working-dir nod-cloud://swift_connector/my_new_container/ \
--recursive \
--ignore-hidden \
nod-cloud://cloud_storage/container/path/ \
nod://posix_storage/path/
```
**Execute a defer task**
The objective of defer task is to create the task and already initiate the authentication process but defers its execution. Unique IDs will be returned.
```
$ nd --access-token copy --defer-run
{“defer”: “a42e5461-1410-4f17-9a2a-d7a60e308b24"}
```
#### List created tasks
**Command**
`nd task list`
**Description**
This command list all tasks created by the user in the data mover service.
The columns describe:
| Output | Description |
| ---------- | -------------------------------------------- |
| TASK ID | ID of the task |
| TASK NAME | Name of the task defined during the creation |
| COMMENT | Associated comment |
| CREATED BY | User who has created the task |
**Output**
```
+--------------------------+------------------------------------------------------------+---------+-------------------+-----------------------+
| TASK ID | TASK NAME | COMMENT | CREATED BY | LAST EXECUTION STATUS |
+--------------------------+------------------------------------------------------------+---------+-------------------+-----------------------+
| 6287a774a91db0b194e97d8d | From /largedata2_pool/storagetestdata/data5 to my_pool1 | | Valery Guilleaume | done |
| 628b45f5a91db04211739dc6 | From /largedata2_pool/storagetestdata/data5 to my_pool1 | | Valery Guilleaume | done |
| 628b468ea91db04211739dc9 | From /largedata2_pool/storagetestdata/data5 to my_pool1 | | Valery Guilleaume | done |
| 6331d52ba91db02d6797e6ae | From nod://largedata2_pool/storagetestdata/ to vg--1598 | | Valery Guilleaume | stopped_by_user |
| 6331d5aca91db02d6797e6b1 | From nod://largedata2_pool/storagetestdata/ to vg--1598 | | Valery Guilleaume | stopped_by_user |
| 6331d5c5a91db02d6797e6b4 | From nod://largedata2_pool/storagetestdata/ to vg--1590 | | Valery Guilleaume | stopped_by_user |
| 6331d677a91db02d6797e6b7 | From nod://largedata2_pool/storagetestdata/ to vg--1598 | | Valery Guilleaume | done |
| 6331d692a91db02d6797e6ba | From nod://largedata2_pool/storagetestdata/ to vg--1598 | | Valery Guilleaume | stopped_by_user |
| 6331d948a91db02d6797e6c6 | From nod://largedata2_pool/storagetestdata/ to vg--1598 | | Valery Guilleaume | stopped_by_user |
| 6333ffdda91db091264b68a5 | From nod://largedata2_pool/storagetestdata/ to vg--1500 | | Valery Guilleaume | finished with warning |
| 6333fff6a91db092d94b68a2 | From nod://largedata2_pool/storagetestdata/ to vg--1500 | | Valery Guilleaume | finished with warning |
| 63355214a91db0397f128dbf | From nod://largedata2_pool/storagetestdata/ to vg--1500 | | Valery Guilleaume | done |
| 63355218a91db0397f128dc2 | From nod://largedata2_pool/storagetestdata/ to vg--1500 | | Valery Guilleaume | done |
| 63355220a91db0a6ec22073e | From nod://largedata2_pool/storagetestdata/ to vg--1501 | | Valery Guilleaume | done |
| 6335549aa91db0397f128dc5 | From nod://largedata2_pool/storagetestdata/ to vg--1500 | | Valery Guilleaume | done |
| 6335549da91db0397f128dc8 | From nod://largedata2_pool/storagetestdata/ to vg--1502 | | Valery Guilleaume | done |
| 633554a1a91db0a6ec220741 | From nod://largedata2_pool/storagetestdata/ to vg--1506 | | Valery Guilleaume | done |
| 63358216a91db0397f128dcb | From nod://largedata2_pool/storagetestdata/ to vg--1500 | | Valery Guilleaume | finished with warning |
| 6335822da91db0397f128dce | From nod://largedata2_pool/storagetestdata/ to vg--1502 | | Valery Guilleaume | done |
| 633584d5a91db0397f128dd1 | From nod://largedata2_pool/storagetestdata/ to vg--1502 | | Valery Guilleaume | finished with warning |
| 6336b341a91db0397f128dd4 | From nod://largedata2_pool/storagetestdata/ to vg--1500 | | Valery Guilleaume | done |
| 6336b35da91db0c2985ff420 | From nod://largedata2_pool/storagetestdata/ to vg--1500 | | Valery Guilleaume | done |
+--------------------------+------------------------------------------------------------+---------+-------------------+-----------------------+
| NUMBER OF TASK(S) 22 | | | |
+--------------------------+------------------------------------------------------------+---------+-------------------+-----------------------+
```
#### Tasks status
**Description**
At the end of each task execution, the task result is displayed if the `--progress` parameter is set at true.
```
INFO Connecting with device flow...
INFO Connected with user Valery Guilleaume
Processed size ... done! [8.17MB in 39s]
Processed items ... done! [471 in 39s]
ID: 633ecc74975854bbc799a424
Task ID: 633ecc74a91db0f38f7abc2e
Name: From nod://largedata2_pool/storagetestdata/ to vguilleaume
Comment:
Created by: Valery Guilleaume
Nodes: 471 / 471
Size: 8.17 MB / 8.17 MB
Status: done
```
**Get the status of a task**
**Description**
The `nd task status` command allow to display the status any task. The default command execution display a summary of the task status including number of files copied, size copied, overall status, ....
Additional parameters are available to get more insights about the task.
**Additional Parameters**
| Parameters | Description |
| ------------------------ | ------------------------------------------------------------------------- |
| `--progress` | display live progress (default: false) |
| `--processed-node value` | display the processed nodes. One of (none, error, all) (default: "error") |
**Example of Command**
`# nd task status 633ecc74a91db0f38f7abc2e`
where `633ecc74a91db0f38f7abc2e` is the id of the task
**Output**
```
INFO Connecting with device flow...
INFO Connected with user Valery Guilleaume
ID: 633ecc74975854bbc799a424
Task ID: 633ecc74a91db0f38f7abc2e
Name: From nod://largedata2_pool/storagetestdata/ to vguilleaume
Comment:
Created by: Valery Guilleaume
Nodes: 471 / 471
Size: 8.17 MB / 8.17 MB
Status: done
```
#### List the status of all executed tasks
**Command**
`nd task list-exec 6389c04605e7b8ff6df35cc4`
**Description**
This command list all tasks executed by the user in the data mover service. The columns describe:
| Output | Description |
| ----------- | ----------------------------------------------------------- |
| ID | ID of the executed task |
| STARTED AT | Date when the task has been started |
| FINISHED AT | Date when the task has been finished |
| NODES | Number of files copied / Total number of files to be copied |
| SIZE | Size of files copied / Total size of files to be copied |
| STATUS | Status of the executed task |
**Output**
```
+--------------------------+---------------------+---------------------+---------+---------------------+-----------------+
| ID | STARTED AT | FINISHED AT | NODES | SIZE | STATUS |
+--------------------------+---------------------+---------------------+---------+---------------------+-----------------+
| 6389c105be32a3507d90a6ea | 02 Dec 22 10:10 CET | 02 Dec 22 10:27 CET | 86 / 86 | 5.89 GB / 5.89 GB | done |
| 6389e5e191c3044a236f7baf | 02 Dec 22 12:47 CET | 02 Dec 22 12:49 CET | 3 / 86 | 108.38 MB / 5.89 GB | stopped_by_user |
| 6389e64391c3044a236f7c09 | 02 Dec 22 12:49 CET | 02 Dec 22 12:50 CET | 2 / 86 | 19.52 kB / 5.89 GB | stopped_by_user |
| 6389f33d91c3044a236f7c63 | 02 Dec 22 13:44 CET | 02 Dec 22 14:00 CET | 22 / 86 | 1.76 GB / 5.89 GB | stopped_by_user |
| 6389f704c4df1940f3253444 | 02 Dec 22 14:00 CET | 02 Dec 22 14:03 CET | 5 / 86 | 108.41 MB / 5.89 GB | stopped_by_user |
| 6389f7bac4df1940f325349e | 02 Dec 22 14:03 CET | 02 Dec 22 14:04 CET | 2 / 86 | 19.52 kB / 5.89 GB | stopped_by_user |
| 6389f812c4df1940f32534f8 | 02 Dec 22 14:05 CET | 02 Dec 22 14:06 CET | 2 / 86 | 19.52 kB / 5.89 GB | stopped_by_user |
| 6389fc3cc4df1940f3253552 | 02 Dec 22 14:23 CET | 02 Dec 22 16:11 CET | 0 / 0 | 0.00 B / 0.00 B | stopped_by_user |
| 638a188cfbd1fa439d567ae2 | 02 Dec 22 16:23 CET | 02 Dec 22 16:25 CET | 2 / 86 | 19.52 kB / 5.89 GB | stopped_by_user |
| 638a1916fbd1fa439d567b3c | 02 Dec 22 16:26 CET | 02 Dec 22 16:27 CET | 2 / 86 | 19.52 kB / 5.89 GB | stopped_by_user |
| 638a1986fbd1fa439d567b96 | 02 Dec 22 16:28 CET | 02 Dec 22 16:30 CET | 2 / 86 | 22.02 kB / 5.89 GB | stopped_by_user |
| 638a21c5fbd1fa439d567bf0 | 02 Dec 22 17:03 CET | 05 Dec 22 11:18 CET | 86 / 86 | 5.89 GB / 5.89 GB | done |
+--------------------------+---------------------+---------------------+---------+---------------------+-----------------+
```
#### **Task Control Commands**
New commands to control running tasks:
| Command | Description |
| ------------------- | --------------------------- |
| `nd task pause` | Pause a running task |
| `nd task resume` | Resume a paused task |
| `nd task stop` | Stop a running task |
| `nd task processed` | Get list of processed nodes |
**nd task pause**
Pause a running task execution.
**Usage:**
```
nd task pause EXECUTIONID
nd task pause TASKID
```
**Example:**
`nd task pause 507f1f77bcf86cd799439011`
**nd task resume**
Resume a paused task execution.
**Usage:**
```
nd task resume EXECUTIONID
nd task resume TASKID
```
**Example:**
```
nd task resume 507f1f77bcf86cd799439011
```
**nd task stop**
Stop a running task execution.
**Flags:**
* `--force`, `-f` - Force stop the task
**Usage:**
```
nd task stop EXECUTIONID
nd task stop --force TASKID
```
**nd task processed**
Get detailed list of processed nodes for a task execution.
**Flags:**
* `--processed-nodes=FILTER` - Filter processed nodes: `error` or `all` (default: all)
**Usage:**
```
nd task processed EXECUTIONID
nd task processed --processed-nodes=error TASKID
```
**Output includes:**
* File path
* Processing status
* Dispatcher ID
* Mover ID
* Errors (if any)
**Example:**
```
# Show all processed files
nd task processed 507f1f77bcf86cd799439011
# Show only files with errors
nd task processed --processed-nodes=error 507f1f77bcf86cd799439011
```
#### **Pool Management**
**nd pool list**
List pools with optional filtering.
**Flags:**
* `--content=TYPE` - Filter by content type (cloud, nas, tape)
* `--type=TYPE` - Filter by pool type (primary, active\_archive, offline\_archive, etc.)
* `--name=NAME` - Filter by pool name
* `--comment=TEXT` - Filter by pool comment
* `--id=ID` - Filter by pool ID
**Example:**
```
# List all available
nd pool list
# List all NAS pools
nd pool list --content=nas
# List all primary pools
nd pool list --type=primary
# List specific pool by name
nd pool list --name=mypool
```
**Output**
```
╭───────────────────┬────────────┬───────────────────────────────────┬─────────┬─────────┬─────────────────┬────────────┬───────────────╮
│ POOL ID │ NAME │ COMMENT │ TYPE │ CONTENT │ PATH │ STORAGE ID │ WRITE PROTECT │
├───────────────────┼────────────┼───────────────────────────────────┼─────────┼─────────┼─────────────────┼────────────┼───────────────┤
│ 2 │ nfsv3 │ NFS V3 Primary │ primary │ nas │ srv/nfsshare-v3 │ │ │
│ 3 │ nfsv4 │ NFS V4 Primary │ primary │ nas │ srv/nfsshare-v4 │ │ │
│ 7 │ source │ Primary pool for source container │ primary │ nas │ │ 1 │ │
│ 8 │ dest │ Primary pool for dest container │ primary │ nas │ │ 1 │ │
│ 16 │ Samba │ Samba Primary │ primary │ nas │ /mnt/samba │ │ │
│ 17 │ containers │ Container │ primary │ nas │ mnt/containers │ │ │
├───────────────────┼────────────┼───────────────────────────────────┼─────────┼─────────┼─────────────────┼────────────┼───────────────┤
│ NUMBER OF POOL(S) │ 6 │ │ │ │ │ │ │
╰───────────────────┴────────────┴───────────────────────────────────┴─────────┴─────────┴─────────────────┴────────────┴───────────────╯
```