ND Client

Install, configure, and use the `nd` CLI to create and monitor Data Mover tasks.

The Nodeum Client nd command line tool provides a modern set of commands to execute data movement operations with Nodeum. The nd command line tool is built for compatibility with the Nodeum data Mover for expected functionality and behavior.

nd has the following syntax:

nd [GLOBALFLAGS] COMMAND --help

See Command Quick Reference for a list of supported commands.

Copyright
nd is a property of Nodeum and its subsidaries, if any. The intellectual and technical concepts contained herein are proprietary to Nodeum and its subsidaries and may be covered by Belgium and Foreign Patents, patents in process, and are protected by trade secret or copyright law. Dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Nodeum.

Related Version: 2.0.13

Here is a list of ND client packages:

Architecture

x86

aarch64

RPM

DEB

Binary Linux

Binary Windows

Binary MacOS

Quickstart

Install nd

Install the nd command line tool onto the host machine. Go in the section that corresponds to the host machine operating system or environment:

Instructions for Linux Users

To add a temporary extension to your system PATH for running the nd bash client, use the commands below. For permanent PATH modifications, refer to your operating system's guidance.

Alternatively, execute nd by navigating to the parent folder and running ./nd --help

# Download the binary
curl https://get.nodeum.io/public/nd-2.0.13-linux-amd64 \
  --create-dirs \
  -o $HOME/nodeum-binaries/nd

# Make it executable
chmod +x $HOME/nodeum-binaries/nd

# Add to PATH (temporary for current session)
export PATH=$PATH:$HOME/nodeum-binaries/

# Test it
nd --help

Instructions for macOS Users

# Download the binary
curl https://get.nodeum.io/public/nd-2.0.13-darwin-amd64 \
  --create-dirs \
  -o $HOME/nodeum-binaries/nd

# Make it executable
chmod +x $HOME/nodeum-binaries/nd

# Add to PATH (temporary for current session)
export PATH=$PATH:$HOME/nodeum-binaries/

# Test it
nd --help

Instructions for Windows Users

Open the following file in a browser:

https://get.nodeum.io/public/nd-2.0.13-windows-amd64

Once download, rename the file to nd.exe and execute the file by double clicking on it, or by running the following in the command prompt or powershell:

\path\to\nd.exe --help

ND client installation (RPM example)

# Download the RPM

wget https://get.nodeum.io/public/nd-2.0.13-x86_64.rpm

# Install the RPM
dnf install ./nd-2.0.13-x86_64.rpm

Access to the interface

[nodeum01@]$ nd 
NAME:
   nd - Nodeum CLI

USAGE:
   nd [global options] command [command options] [arguments...]

VERSION:
   2.0.13

COMMANDS:
   admin
   auth-service   Service that manages users based on Linux users
   config         configure the Nodeum Client
   copy, cp       create copy task
   move, mv       create move task
   pool           manage pools
   server-config
   settings       manage user preferences (synced with the web UI)
   task
   help, h        Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --json                          output as JSON (default: false)
   --debug                         enable verbose logging (default: false) [$ND_DEBUG]
   --config value                  path to configuration file (default: <config-dir>/config.json) [$ND_CONFIG]
   --config-dir value, -C value    path to configuration folder (default: "/home/valery/.config/.nd") [$ND_CONFIG_DIR]
   --alias value                   alias in configuration file for authentication (default: "default") [$ND_ALIAS]
   --url value                     URL of Nodeum [$ND_URL]
   --auth-type value               Auth type (empty or ldap) [$ND_AUTH_TYPE]
   --access-token value            for API authentication (1st authentication method) [$ND_ACCESS_TOKEN]
   --refresh-token value           for API authentication (1st authentication method, not saved in config) [$ND_REFRESH_TOKEN]
   --authorization-endpoint value  for Device Authorization Flow (2nd authentication method)
   --token-endpoint value          for Device Authorization Flow (2nd authentication method)
   --client-id value               for Device Authorization Flow (2nd authentication method)
   --scopes value                  for Device Authorization Flow (2nd authentication method)
   --persist-session               persist Device Authorization session on disk for 1 hour (default: true)
   --persist-session-renew         if persist session is enabled, renew the token (default: false)
   --username value                for API authentication (3rd authentication method) [$ND_USERNAME]
   --password value                for API authentication (3rd authentication method) [$ND_PASSWORD]
   --anonymous                     no login (default: false)
   --help, -h                      show help
   --version, -v                   print the version

COPYRIGHT:
   nd is a property of Nodeum and its subsidiaries, if any. The intellectual and
   technical concepts contained herein are proprietary to Nodeum and its
   subsidiaries and may be covered by Belgium and Foreign Patents, patents in
   process, and are protected by trade secret or copyright law. Dissemination of
   this information or reproduction of this material is strictly forbidden unless
   prior written permission is obtained from Nodeum.

The following table lists nd commands:

Name

Shortcut

Description

admin

Access the administration commands

config

Configure the Nodeum client

copy

Create a task to copy data between two storages

move

Create a task to move data between two storages

task

List detailed information about created task

pool

list, destroy available pools(for admin only)

help

Displays a summary of command usage and parameters on the terminal

Parameters

Syntax

The nd client provides a Bash completion mechanism to facilitate the search of commands.

Metadata key can't include an = character.

Global Parameters

Name

Shortcut

Description

Default

--json

Output as JSON

false

--debug

Enable verbose logging (shows INFO and ERROR level logs)

false

--config value

-C value

Path to configuration file

/home/nodeum01/.config/.nd/config.json

--config-dir value

-C value

Path to configuration folder (default: "/home/nodeum01/.config/.nd")

--alias value

Alias in configuration file for authentication

default

--url value

URL of Nodeum

--access-token value

for API authentication (1st authentication method)

--refresh-token value

for API authentication (1st authentication method, not saved in config)

--authorization-endpoint value

for Device Authorization Flow (2nd authentication method)

--token-endpoint value

for Device Authorization Flow (2nd authentication method)

--client-id value

for Device Authorization Flow (2nd authentication method)

--scopes value

for Device Authorization Flow (2nd authentication method)

--persist-session

persist Device Authorization session on disk for 1 hour

true

--persist-session-renew

if persist session is enabled, renew the token

false

--username value

for API authentication (3rd authentication method)

--password value

for API authentication (3rd authentication method)

--anonymous

no login

false

--help

-h

show help

false

--version

-v

print the version

false

Mandatory Parameters for OpenID integration

Parameters are available for each data movement task.

Name

Description

Value

--md project_name=

Name of project defined in OpenStack

string

--md project_domain_name=

Name of the project's domain defined in OpenStack

string

--md user_domain_name=

Name of the project's user defined in OpenStack

string

--md region_name=

Name of the region's project defined in OpenStack

string

Standard Parameters

Parameters are available for each data movement task.

Name

Shortcut

Description

Value

Default value

--help

-h

Show help

false

--no-run

Create a task and don't launch the task directly

false

--name value

-n

Name of task

string

automatically generated

--comment value

Comment of task

string

empty

--overwrite value

Overwrite all identical files already stored at destination

true - false

false

--priority value

Priority of the task, between 0 and 9 (0 is the highest priority)

0 - 9

--recursive

-R

Execute a recursive copy of the folder. If subfolders are present, the service will also copy the contents of each subfolder

false

--working-dir value

--wd

Defines the workpath to be kept at destination

. - .. - path

--ignore-hidden value

Task will not handle hidden file(s)

true - false

false

--progress value

Display live progress when running the task

true - false

true

--processed-nodes value

Display the processed nodes when running a task when the flag--progress is set

none, error, all

error

Advanced Parameters

Parameters are available for each data movement task.

Name

Shortcut

Description

Value

Default value

--context-uid value

--uid

Define the User ID which will handle the movement

integer

unset

--context-gid value

--gid

Define the Group ID which will handle the movement

integer

unset

--defer

when requesting the run of the task, will defer it for later with an unique ID

false

--parallel value

Define the number of mover which will handle the movement. Maximum value is determined by the deployed implementation.

1-20

--callback type

Add callback. Format is type:./path/to/file

./path/to/file

--trigger-md key=value

--md key=value

Set metadata on the trigger. Format is key=value. Accepts multiple inputs

key=value

--task-md key=value

Set metadata on the task. Format is key=value. Accepts multiple inputs

key=value

--files-md key=value

Set metadata on the files. Format is key=value. Accepts multiple inputs

key=value

--run-as-user=username

Run the task as a different user

unset

Filter Parameters

Filters allow you to include or exclude files based on various criteria during copy/move operations. Filters are executed on the mover side before processing files, reducing unnecessary transfers.

Filter Logic

Include filters of the same type are OR'd (match any)
Exclude filters and different filter types are AND'd (all must pass)

Available Filter Flags

Configuration

nd uses a JSON formatted configuration file used for storing certain kinds of information, such as the authentication and authorization options. By default, this configuration file is unique by user. It is stored in its home directory.

For Linux and macOS, the default configuration file location is .config/.nd/config.json which is store in the $HOME. For Windows, the configuration file is stored in $AppData$ .

You can display the configuration file location in using the command nd --help:

[nodeum01@]$ nd --help 
NAME:
   nd - Nodeum CLI
....
   --config value                  path to configuration file (default: <config-dir>/config.json) [$ND_CONFIG]
   --config-dir value, -C value    path to configuration folder (default: "/home/nodeum01/.config/.nd") [$ND_CONFIG_DIR]

You can use the --config value where value is the path to a JSON formatted configuration file that nd uses for storing data. The ND_CONFIG environment variable can be used to set the value.

Store the configuration file in a 'central' directory to allow each user to get the same nd client configuration. For this, the --config-dir value is available.

Command

Command

Description

--config value

this option specifies the JSON filename where the configuration is stored

--config-dir value

this option specifies the directory where the JSON configuration is stored

SSL

nd client allows SSL configuration to communicate with the Data Mover service which listen in HTTPS. It is required to add the public certificate generated with the server.

The public certificate file has to be stored in this config-dir folder: .config/.nd/certs/CAs/.

The certificates on the server side have been generated following this command:

openssl11 req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
\-keyout ./nodeum_certs/private.key \
\-out ./nodeum_certs/public.crt \
\-subj "/CN=nodeum-qualif.mt-c.local" \
\-addext "subjectAltName=DNS:nodeum.domain.local,DNS:localhost,IP:127.0.0.1,IP:1.1.1.1"

Where 1.1.1.1 is the ip of the nginx interface and nodeum.domain.local is its hostname including domain name.

Authentication and Authorization

Description

Three authentifcation options are available:

Username / Password
IDP with OpenID
authenticate via Auth Service if configured

Command

Command

Description

nd config save

this command save the nd configuration for authentication

nd config review

this command display the information related to the credentials

nd config clear-session

clear persisted session

Username / Password configuration

The nd client provides a basic method of authentication in using Username / Password credentials.

\[nodeum01@nodeum\]$ nd config save \
\--url=http:// \
\--username= \
\--password=

Option

Description

--url

this refer to the node which hosts the following service DATA MANAGEMENT WEB SERVICES

--username

this is the username to grant authorization to the service

--password

this is the password associated to the username that will grant the access to the service

OpenID configuration

The nd client provides an OpenID authentication mechanism. In this case, nd Client has to be configured with the appropriated IDP to handle proper token management.

The basic configuration is the following one:

\[nodeum01@nodeum\]$ nd config save \
\--url=http:// \
\--authorization-endpoint= \
\--token-endpoint= \
\--client-id= \
\--scopes="openid,profile,offline_access,email"

Option

Description

--url

this refer to the node which hosts the following service DATA MANAGEMENT WEB SERVICES

--authorization-endpoint

this is the endpoint url to grant authorization to the service

--token-endpoint

this url is used to programmatically request tokens

--client-id

this refer to the client identifier which is provided by the OpenID provider

--persist-session

persist Device Authorization session on disk for 1 hour (default: true)

--persist-session-renew

if persist session is enabled, renew the token (default: false)

The standard behavior is to request a token, which is automatic if there is no token available. The token will be stored in a cache during 15 minutes. The --persist-session-renew option can be defined to true to force a request token process any time the user has to interface with the nd client.

The token renewal is automatic based on the renewal token.

Authentication via Auth Service

If the Auth Service is configured, authentication will be handled automatically. The Auth Service runs as a systemd service and must be properly set up by a privileged/admin user. Once configured, normal users can authenticate via the Auth Service through a Unix socket, without having to manage authentication themselves.

The Auth Service communicates with the Nodeum server to create new Linux users or update existing passwords as needed.

Admin/Privileged User Setup

If the Auth Service is installed via .rpm or .deb package:
- The service file is automatically installed and the service will be restarted.
- The only manual step required is to set the configuration directory: export ND_CONFIG_DIR=/etc/nodeum/config
- Then authenticate to Nodeum using username/password or another supported method (e.g., OpenID). with above method mentioned
- You may also define an alias for the Auth Service (or leave it empty to use the default).
If the Auth Service is not installed via package:
- The admin must manually create the service file and configure everything.

Normal User Workflow

A user simply runs commands like:
nd cp ... or nd mv ...
The CLI checks whether the Auth Service is running.
If it is, the CLI communicates with the Auth Service over the Unix socket to automatically authenticate the user (by creating/updating their username and password).

Multiple Clusters / Aliases

If the Auth Service is configured with multiple clusters or aliases, users will be prompted to choose which alias to connect with.

Example error when no default alias is set:

INFO  Auth service is running
FATA  No default alias found in auth service.
Please specify an alias using one of the following methods:
  • nd --alias=<alias> <command>
  • nd config <command> --alias=<alias>
Available aliases: cluster1, cluster2, cluster3

Once a user sets an alias, they are authenticated for that cluster.
To switch to another cluster, simply pass a different alias.
If the Auth Service has a default alias configured, users will automatically be authenticated with that alias without being prompted.

LDAP Server Configuration

nd server-config ldap

Configure LDAP settings on the Nodeum server.

Commands:

Command

Description

nd server-config ldap get

Get current LDAP configuration

nd server-config ldap set

Set LDAP configuration

Note: Requires admin privileges.

Example:

# Get LDAP configuration
nd server-config ldap get

# Set LDAP configuration
nd server-config ldap set [options]

Alias & default flags

Description

Alias & default flags allow structuration of different group of settings. Different alias can be defined in the configuration file.

Definition of alias & default flags

Alias and flags are declared in the configuration file ~/.config/.nd/config.json". Default flags are defined for each available parameter command. Flags can be overwritted in the nd command.

Example in config file:

{
  // ...
  // default: name of alias used by default
  "default": {          
   // ...
   "default_flags": {
    // this is the name of the command, example: nd copy
    "copy": {
       // this is the flag, example: nd copy --md project_domain_name=MYPROJECTDOMAIN --md user_domain_name=MYUSERDOMAIN
      "md": ["project_domain_name=MYPROJECTDOMAIN", "user_domain_name=MYUSERDOMAIN"]
    }
   }
  },
  // myorganisation: name of another alias
  "myorganisation": {  
     // ...
  }
}

Alias usage

The nd command allows the usage of alias, example : nd --alias myorganisation copy

Data Mover Service Status

Command

Description

nd admin status

retrieve the status and the health of each services part of the cluster

nd admin logs

retrieve all Logs and return their contents

Data Management Monitoring Services

nd admin status command requests the Data Management Monitoring service to retrieve the status and health of each services part of the cluster. This command returns list of services. The following information are displayed:

Service Status
Service version
Host where the service is deployed
Its uptime
Consumed memory

Output where all services are reachable

╭───┬───────────────────┬────────────────────────────┬─────────────────┬────────────┬───────────╮
│   │ NAME              │ VERSION                    │ ADDRESS         │ UPTIME     │ MEMORY    │
├───┼───────────────────┼────────────────────────────┼─────────────────┼────────────┼───────────┤
│ ● │ micro.http.broker │ ff.http.broadcast          │ 10.2.2.83:50050 │ N/A        │ N/A       │
│ ● │ nodeum.dispatcher │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.84:50010 │ 172h11m8s  │ 34.92 MB  │
│ ● │ nodeum.dispatcher │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50010 │ 172h13m32s │ 22.61 MB  │
│ ● │ nodeum.front      │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50000 │ N/A        │ N/A       │
│ ● │ nodeum.monitoring │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.85:50060 │ 171h2m47s  │ 17.20 MB  │
│ ● │ nodeum.mover      │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.84:50020 │ 172h4m48s  │ 25.23 MB  │
│ ● │ nodeum.mover      │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.85:50020 │ 172h4m48s  │ 25.63 MB  │
│ ● │ nodeum.scheduler  │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50040 │ 172h18m6s  │ 6.91 MB   │
│ ● │ mongo             │ 4.4.17                     │ 10.2.2.83:27017 │ 268h0m44s  │ 212.86 MB │
│ ● │ mongo             │ 4.4.17                     │ 10.2.2.84:27017 │ 268h0m44s  │ 212.86 MB │
│ ● │ mongo             │ 4.4.17                     │ 10.2.2.85:27017 │ 268h0m44s  │ 212.86 MB │
│ ● │ mysql             │ 10.6.11-MariaDB            │ 10.2.2.84:3306  │ 150h21m0s  │ N/A       │
│ ● │ redis-sentinel    │ 6.2.6                      │ 10.2.2.83:26379 │ 150h16m52s │ N/A       │
│ ● │ redis-sentinel    │ 6.2.6                      │ 10.2.2.84:26379 │ 150h16m52s │ N/A       │
│ ● │ redis-sentinel    │ 6.2.6                      │ 10.2.2.85:26379 │ 150h16m52s │ N/A       │
│ ● │ redis             │ 6.2.6                      │ 10.2.2.83:6379  │ 285h37m50s │ N/A       │
│ ● │ redis             │ 6.2.6                      │ 10.2.2.84:6379  │ 169h59m59s │ N/A       │
│ ● │ redis             │ 6.2.6                      │ 10.2.2.85:6379  │ 169h59m59s │ N/A       │
│ ● │ etcd              │ 3.4.13                     │ 10.2.2.83:2379  │ N/A        │ N/A       │
│ ● │ etcd              │ 3.4.13                     │ 10.2.2.84:2379  │ N/A        │ N/A       │
│ ● │ etcd              │ 3.4.13                     │ 10.2.2.85:2379  │ N/A        │ N/A       │
╰───┴───────────────────┴────────────────────────────┴─────────────────┴────────────┴───────────╯

Output where some services are not available

╭───┬───────────────────┬────────────────────────────┬─────────────────┬────────────┬───────────╮
│   │ NAME              │ VERSION                    │ ADDRESS         │ UPTIME     │ MEMORY    │
├───┼───────────────────┼────────────────────────────┼─────────────────┼────────────┼───────────┤
│ ● │ micro.http.broker │ ff.http.broadcast          │ 10.2.2.83:50050 │ N/A        │ N/A       │
│ ● │ nodeum.dispatcher │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.84:50010 │ 172h11m8s  │ 34.92 MB  │
│ ● │ nodeum.dispatcher │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50010 │ 172h13m32s │ 22.61 MB  │
│ ● │ nodeum.front      │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50000 │ N/A        │ N/A       │
│ ● │ nodeum.monitoring │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.85:50060 │ 171h2m47s  │ 17.20 MB  │
│ ● │ nodeum.mover      │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.84:50020 │ 172h4m48s  │ 25.23 MB  │
│ ● │ nodeum.mover      │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.85:50020 │ 172h4m48s  │ 25.63 MB  │
│ ● │ nodeum.scheduler  │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50040 │ 172h18m6s  │ 6.91 MB   │
│ ● │ mongo             │ 4.4.17                     │ 10.2.2.83:27017 │ 268h0m44s  │ 212.86 MB │
│ ● │ mongo             │ 4.4.17                     │ 10.2.2.84:27017 │ 268h0m44s  │ 212.86 MB │
│ ● │ mongo             │ 4.4.17                     │ 10.2.2.85:27017 │ 268h0m44s  │ 212.86 MB │
│ ● │ mysql             │ 10.6.11-MariaDB            │ 10.2.2.84:3306  │ 150h21m0s  │ N/A       │
│ ● │ redis-sentinel    │ 6.2.6                      │ 10.2.2.83:26379 │ 150h16m52s │ N/A       │
│ ● │ redis-sentinel    │ 6.2.6                      │ 10.2.2.84:26379 │ 150h16m52s │ N/A       │
│ ● │ redis-sentinel    │ 6.2.6                      │ 10.2.2.85:26379 │ 150h16m52s │ N/A       │
│ ● │ redis             │ 6.2.6                      │ 10.2.2.83:6379  │ 285h37m50s │ N/A       │
│ ● │ redis             │ 6.2.6                      │ 10.2.2.84:6379  │ 169h59m59s │ N/A       │
│ ● │ redis             │ 6.2.6                      │ 10.2.2.85:6379  │ 169h59m59s │ N/A       │
│ ● │ etcd              │ 3.4.13                     │ 10.2.2.83:2379  │ N/A        │ N/A       │
│ ● │ etcd              │ 3.4.13                     │ 10.2.2.84:2379  │ N/A        │ N/A       │
│ ● │ etcd              │ 3.4.13                     │ 10.2.2.85:2379  │ N/A        │ N/A       │
│ ○ │ nodeum.mover      │                            │                 │            │           │
╰───┴───────────────────┴────────────────────────────┴─────────────────┴────────────┴───────────╯

Data Management Log Management Services

nd admin logs command requests the Data Management Log Management Services to retrieve all Logs and return their contents.

[nodeum01@ ~]$ nd admin logs --help
NAME:
   nd admin logs - display application logs

USAGE:
   nd admin logs [command options] [arguments...]

OPTIONS:
   --since value, -S value                          show logs not older than the specified date
   --until value, -U value                          show logs not newer than the specified date
   --tag value, -u value [ --tag value, -u value ]  filter by tags
   --level value                                    minimum level of logs, one of [trace debug info warn error fatal]
   --help, -h                                       show help (default: false)

This command allows different parameters:

Option

Description

since

to only show logs not older than a specified date

until

to only show logs not newer than a specified date

tag

to filter the logs per type of services. Example nodeum.monitoring

level

to define the minimum level of logs, one of [trace - debug - info - warn - error - fatal]

Export

The Logs can be exported in using the standard OS mechanism of exportation

./nd admin logs > 'nodeum_site-name_log.txt'

Where nodeum_site-name_log.txt is the file name and site represents the name of the site.

Data Mover Task Management

Follow these steps to create a task:

Use Mandatory Parameters
Define the data source
Define the destination
Apply others parameters if needed

Task Creation

The nd copy command send a copy request to the data mover service from a storage A (nod://posix_storage/) to a storage B (nod-cloud://cloud_storage/).

Command with minimal parameters

nd copy \ --md project_name=<my project name> \ --md project_domain_name=<my projectdomain name> \ --md user_domain_name=<my user domain name> \ nod://posix_storage/path/subpath/ \ nod-cloud://cloud_storage/container

Detailed syntax

nd copy \
--md project_name=<my project name> \
--md project_domain_name=<my projectdomain name> \
--md user_domain_name=<my user domain name> \
nod://posix_storage/path/subpath/ \
nod-cloud://cloud_storage/container

Example of creation task with additional parameters

nd copy \ --md project_name=<my project name> \ --md project_domain_name=<my project domain name> \ --md user_domain_name=<my user domain name> \ --working-dir nod://largedata2_pool/storagetestdata/ \ --recursive \ nod://posix_storage/path/subpath/ \ nod-cloud://cloud_storage/container

Detailed syntax

nd copy \
--md project_name=<my project name> \
--md project_domain_name=<my project domain name> \
--md user_domain_name=<my user domain name> \
--working-dir nod://posix_storage/path/ \
--recursive \
nod://posix_storage/path/subpath/ \
nod-cloud://cloud_storage/container

Available Parameters

Option

Description

--callback type:./path/to/file

add callback. Format is type:./path/to/file (accepts multiple inputs)

--defer

when requesting the run of the task, will defer it for later with an unique ID (default: false)

--no-run

just create the task, don't run it (default: false)

--progress

when running the task, display live progress (default: true)

--recursive

copy directories recursively (default: false)

--ignore-hidden

ignore hidden files and folders, starting with (default: false)

--overwrite

overwrite existing entries (default: false)

--priority value

task priority [0..9] (default: 0)

--working-dir value

set working directory

--remove-root-folder

remove root folder (default: false)

Task creation from directly Absolute and Relative path.

The ND client supports using mounted filesystem paths in addition to nod:// URIs. The client automatically resolves these paths against configured NAS shares.

Supported Path Formats:

Legacy format: nod://pool-name/path/to/file
Mounted path: /mnt/nas/path/to/file
Relative path: ./relative/path/to/file

How it works:

Client fetches NAS share configurations from Nodeum server
Matches input path against share mount points using longest-prefix matching
Converts to nod://poolname/relative/path format internally
Absolute paths must match a configured share (error if not found)
Relative path to work file system have to be mounted where nd client runs to get absolute path of current working directory. and find a match against nas_share path.

NB: The Nodeum pool extractor needs to identify which storage pool a file belongs to by matching the file path against NAS share paths stored in the database. When passing absolute paths or relative path to the nd client, the NAS share must be mounted at the exact same path on the client machine as it's stored in the database - for example, if the database has /p/projectspace, you must mount it to /p/projectspace (not /mnt/projectspace or anything else). This way, when you access a file like /p/projectspace/file.txt, the extractor can match /p/projectspace to the database entry, determine the pool name, and correctly build the nod://poolname/file.txt URI scheme. If the mount point differs from the database path, the extractor cannot match them and fails to identify the pool.

# by passing absolute path
nd copy /mnt/nas/data/file.txt /mnt/backup/archive/

# this only works if file is mounted which will get absolute path then c
nd copy ./data/file.txt /mnt/backup/


nd copy nod://mypool/data/file.txt nod://backup/archive/

Working Directory Explanation

The definition of a working directory allows to define where the files will be stored at the destination. Different options are available, they are described in the following definition.

With --wd=.

Source

Destination

Result

nod://source/folder/FILE.txt

nod://dest/directory/

nod://dest/directory/FILE.txt

nod://source/folder/FILE.txt

nod://dest/RENAMED.txt

nod://source/folder/

nod://dest/directory/

nod://dest/directory/FILE.txt

nod://source/folder/

nod://dest/directory

nod://dest/directory/FILE.txt

nod://source/folder

nod://dest/directory/

nod://dest/directory/folder/FILE.txt

nod://source/folder

nod://dest/directory

nod://dest/directory/FILE.txt

With --wd=..

Source

Destination

Result

nod://source/folder/FILE.txt

nod://dest/directory/

nod://dest/directory/folder/FILE.txt

nod://source/folder/FILE.txt

nod://dest/RENAMED.txt

nod://source/folder/

nod://dest/directory/

nod://dest/directory/folder/FILE.txt

nod://source/folder/

nod://dest/directory

nod://dest/directory/FILE.txt

nod://source/folder

nod://dest/directory/

nod://dest/directory/source/folder/FILE.txt

nod://source/folder

nod://dest/directory

nod://dest/directory/FILE.txt

Examples of Tasks Creation

Execute a task copy from Posix to Swift

nd copy \ --md project_name=<my project name> \ --md project_domain_name=<my project domain name> \ --md user_domain_name=<my user domain name> \ --working-dir nod://posix_storage/path/ \ --recursive \ --ignore-hidden \ nod://posix_storage/path/subpath/ \ nod-cloud://cloud_storage/container

Detailed syntax

nd copy \
--md project_name=<my project name> \
--md project_domain_name=<my project domain name> \
--md user_domain_name=<my user domain name> \
--working-dir nod://posix_storage/path/ \
--recursive \
--ignore-hidden \
nod://posix_storage/path/subpath/ \
nod-cloud://cloud_storage/container

Execute a task copy from Swift to Posix

nd copy \ --md project_name=<my project name> \ --md project_domain_name=<my project domain name> \ --md user_domain_name=<my user domain name> \ --working-dir nod://posix_storage/path/ \ --recursive \ --ignore-hidden \ nod-cloud://cloud_storage/container/path/ \ nod://posix_storage/path/

Detailed syntax

nd copy \
--md project_name=<my project name> \
--md project_domain_name=<my project domain name> \
--md user_domain_name=<my user domain name> \
--working-dir nod-cloud://cloud_storage/container/ \
--recursive \
--ignore-hidden \
nod-cloud://cloud_storage/container/path/ \
nod://posix_storage/path/

Execute a task move from Posix to Swift

nd move \ --md project_name=<my project name> \ --md project_domain_name=<my project domain name> \ --md user_domain_name=<my user domain name> \ --working-dir nod://posix_storage/path/ \ --recursive \ --ignore-hidden \ nod://posix_storage/path/subpath/ \ nod-cloud://cloud_storage/container

Detailed syntax

nd move \
--md project_name=<my project name> \
--md project_domain_name=<my project domain name> \
--md user_domain_name=<my user domain name> \
--working-dir nod://posix_storage/path/ \
--recursive \
--ignore-hidden \
nod://posix_storage/path/subpath/ \
nod-cloud://cloud_storage/container

Execute a task move from Swift to Posix

nd move \ --md project_name=<my project name> \ --md project_domain_name=<my project domain name> \ --md user_domain_name=<my user domain name> \ --working-dir nod-cloud://cloud_storage/container/ \ --recursive \ --ignore-hidden \ nod-cloud://cloud_storage/container/path/ \ nod://posix_storage/path/

Detailed syntax

nd move \
--md project_name=<my project name> \
--md project_domain_name=<my project domain name> \
--md user_domain_name=<my user domain name> \
--working-dir nod-cloud://swift_connector/my_new_container/ \
--recursive \
--ignore-hidden \
nod-cloud://cloud_storage/container/path/ \
nod://posix_storage/path/

Execute a defer task

The objective of defer task is to create the task and already initiate the authentication process but defers its execution. Unique IDs will be returned.

$ nd --access-token copy <options...> --defer-run 
{“defer”: “a42e5461-1410-4f17-9a2a-d7a60e308b24"}

List created tasks

Command

nd task list

Description

This command list all tasks created by the user in the data mover service.

The columns describe:

Output

Description

TASK ID

ID of the task

TASK NAME

Name of the task defined during the creation

COMMENT

Associated comment

CREATED BY

User who has created the task

Output

+--------------------------+------------------------------------------------------------+---------+-------------------+-----------------------+
| TASK ID                  | TASK NAME                                                  | COMMENT | CREATED BY        | LAST EXECUTION STATUS |
+--------------------------+------------------------------------------------------------+---------+-------------------+-----------------------+
| 6287a774a91db0b194e97d8d | From /largedata2_pool/storagetestdata/data5 to my_pool1    |         | Valery Guilleaume | done                  |
| 628b45f5a91db04211739dc6 | From /largedata2_pool/storagetestdata/data5 to my_pool1    |         | Valery Guilleaume | done                  |
| 628b468ea91db04211739dc9 | From /largedata2_pool/storagetestdata/data5 to my_pool1    |         | Valery Guilleaume | done                  |
| 6331d52ba91db02d6797e6ae | From nod://largedata2_pool/storagetestdata/ to vg--1598    |         | Valery Guilleaume | stopped_by_user       |
| 6331d5aca91db02d6797e6b1 | From nod://largedata2_pool/storagetestdata/ to vg--1598    |         | Valery Guilleaume | stopped_by_user       |
| 6331d5c5a91db02d6797e6b4 | From nod://largedata2_pool/storagetestdata/ to vg--1590    |         | Valery Guilleaume | stopped_by_user       |
| 6331d677a91db02d6797e6b7 | From nod://largedata2_pool/storagetestdata/ to vg--1598    |         | Valery Guilleaume | done                  |
| 6331d692a91db02d6797e6ba | From nod://largedata2_pool/storagetestdata/ to vg--1598    |         | Valery Guilleaume | stopped_by_user       |
| 6331d948a91db02d6797e6c6 | From nod://largedata2_pool/storagetestdata/ to vg--1598    |         | Valery Guilleaume | stopped_by_user       |
| 6333ffdda91db091264b68a5 | From nod://largedata2_pool/storagetestdata/ to vg--1500    |         | Valery Guilleaume | finished with warning |
| 6333fff6a91db092d94b68a2 | From nod://largedata2_pool/storagetestdata/ to vg--1500    |         | Valery Guilleaume | finished with warning |
| 63355214a91db0397f128dbf | From nod://largedata2_pool/storagetestdata/ to vg--1500    |         | Valery Guilleaume | done                  |
| 63355218a91db0397f128dc2 | From nod://largedata2_pool/storagetestdata/ to vg--1500    |         | Valery Guilleaume | done                  |
| 63355220a91db0a6ec22073e | From nod://largedata2_pool/storagetestdata/ to vg--1501    |         | Valery Guilleaume | done                  |
| 6335549aa91db0397f128dc5 | From nod://largedata2_pool/storagetestdata/ to vg--1500    |         | Valery Guilleaume | done                  |
| 6335549da91db0397f128dc8 | From nod://largedata2_pool/storagetestdata/ to vg--1502    |         | Valery Guilleaume | done                  |
| 633554a1a91db0a6ec220741 | From nod://largedata2_pool/storagetestdata/ to vg--1506    |         | Valery Guilleaume | done                  |
| 63358216a91db0397f128dcb | From nod://largedata2_pool/storagetestdata/ to vg--1500    |         | Valery Guilleaume | finished with warning |
| 6335822da91db0397f128dce | From nod://largedata2_pool/storagetestdata/ to vg--1502    |         | Valery Guilleaume | done                  |
| 633584d5a91db0397f128dd1 | From nod://largedata2_pool/storagetestdata/ to vg--1502    |         | Valery Guilleaume | finished with warning |
| 6336b341a91db0397f128dd4 | From nod://largedata2_pool/storagetestdata/ to vg--1500    |         | Valery Guilleaume | done                  |
| 6336b35da91db0c2985ff420 | From nod://largedata2_pool/storagetestdata/ to vg--1500    |         | Valery Guilleaume | done                  |
+--------------------------+------------------------------------------------------------+---------+-------------------+-----------------------+
| NUMBER OF TASK(S)       22                                                            |         |                   |                       |
+--------------------------+------------------------------------------------------------+---------+-------------------+-----------------------+

Tasks status

Description

At the end of each task execution, the task result is displayed if the --progress parameter is set at true.

INFO Connecting with device flow...
INFO Connected with user Valery Guilleaume
Processed size ... done! [8.17MB in 39s]
Processed items ... done! [471 in 39s]
          ID: 633ecc74975854bbc799a424
     Task ID: 633ecc74a91db0f38f7abc2e
        Name: From nod://largedata2_pool/storagetestdata/ to vguilleaume
     Comment:
  Created by: Valery Guilleaume
       Nodes: 471 / 471
        Size: 8.17 MB / 8.17 MB
      Status: done

Get the status of a task

Description

The nd task status command allow to display the status any task. The default command execution display a summary of the task status including number of files copied, size copied, overall status, ....

Additional parameters are available to get more insights about the task.

Additional Parameters

Parameters

Description

--progress

display live progress (default: false)

--processed-node value

display the processed nodes. One of (none, error, all) (default: "error")

Example of Command

# nd task status 633ecc74a91db0f38f7abc2e

where 633ecc74a91db0f38f7abc2e is the id of the task

Output

INFO Connecting with device flow...
INFO Connected with user Valery Guilleaume
          ID: 633ecc74975854bbc799a424
     Task ID: 633ecc74a91db0f38f7abc2e
        Name: From nod://largedata2_pool/storagetestdata/ to vguilleaume
     Comment:
  Created by: Valery Guilleaume
       Nodes: 471 / 471
        Size: 8.17 MB / 8.17 MB
      Status: done

List the status of all executed tasks

Command

nd task list-exec 6389c04605e7b8ff6df35cc4

Description

This command list all tasks executed by the user in the data mover service. The columns describe:

Output

Description

ID of the executed task

STARTED AT

Date when the task has been started

FINISHED AT

Date when the task has been finished

NODES

Number of files copied / Total number of files to be copied

SIZE

Size of files copied / Total size of files to be copied

STATUS

Status of the executed task

Output

+--------------------------+---------------------+---------------------+---------+---------------------+-----------------+
| ID                       | STARTED AT          | FINISHED AT         | NODES   | SIZE                | STATUS          |
+--------------------------+---------------------+---------------------+---------+---------------------+-----------------+
| 6389c105be32a3507d90a6ea | 02 Dec 22 10:10 CET | 02 Dec 22 10:27 CET | 86 / 86 | 5.89 GB / 5.89 GB   | done            |
| 6389e5e191c3044a236f7baf | 02 Dec 22 12:47 CET | 02 Dec 22 12:49 CET | 3 / 86  | 108.38 MB / 5.89 GB | stopped_by_user |
| 6389e64391c3044a236f7c09 | 02 Dec 22 12:49 CET | 02 Dec 22 12:50 CET | 2 / 86  | 19.52 kB / 5.89 GB  | stopped_by_user |
| 6389f33d91c3044a236f7c63 | 02 Dec 22 13:44 CET | 02 Dec 22 14:00 CET | 22 / 86 | 1.76 GB / 5.89 GB   | stopped_by_user |
| 6389f704c4df1940f3253444 | 02 Dec 22 14:00 CET | 02 Dec 22 14:03 CET | 5 / 86  | 108.41 MB / 5.89 GB | stopped_by_user |
| 6389f7bac4df1940f325349e | 02 Dec 22 14:03 CET | 02 Dec 22 14:04 CET | 2 / 86  | 19.52 kB / 5.89 GB  | stopped_by_user |
| 6389f812c4df1940f32534f8 | 02 Dec 22 14:05 CET | 02 Dec 22 14:06 CET | 2 / 86  | 19.52 kB / 5.89 GB  | stopped_by_user |
| 6389fc3cc4df1940f3253552 | 02 Dec 22 14:23 CET | 02 Dec 22 16:11 CET | 0 / 0   | 0.00 B / 0.00 B     | stopped_by_user |
| 638a188cfbd1fa439d567ae2 | 02 Dec 22 16:23 CET | 02 Dec 22 16:25 CET | 2 / 86  | 19.52 kB / 5.89 GB  | stopped_by_user |
| 638a1916fbd1fa439d567b3c | 02 Dec 22 16:26 CET | 02 Dec 22 16:27 CET | 2 / 86  | 19.52 kB / 5.89 GB  | stopped_by_user |
| 638a1986fbd1fa439d567b96 | 02 Dec 22 16:28 CET | 02 Dec 22 16:30 CET | 2 / 86  | 22.02 kB / 5.89 GB  | stopped_by_user |
| 638a21c5fbd1fa439d567bf0 | 02 Dec 22 17:03 CET | 05 Dec 22 11:18 CET | 86 / 86 | 5.89 GB / 5.89 GB   | done            |
+--------------------------+---------------------+---------------------+---------+---------------------+-----------------+

Task Control Commands

New commands to control running tasks:

Command

Description

nd task pause

Pause a running task

nd task resume

Resume a paused task

nd task stop

Stop a running task

nd task processed

Get list of processed nodes

nd task pause

Pause a running task execution.

Usage:

nd task pause EXECUTIONID
nd task pause TASKID

Example:

nd task pause 507f1f77bcf86cd799439011

nd task resume

Resume a paused task execution.

Usage:

nd task resume EXECUTIONID
nd task resume TASKID

Example:

nd task resume 507f1f77bcf86cd799439011

nd task stop

Stop a running task execution.

Flags:

--force, -f - Force stop the task

Usage:

nd task stop EXECUTIONID
nd task stop --force TASKID

nd task processed

Get detailed list of processed nodes for a task execution.

Flags:

--processed-nodes=FILTER - Filter processed nodes: error or all (default: all)

Usage:

nd task processed EXECUTIONID
nd task processed --processed-nodes=error TASKID

Output includes:

File path
Processing status
Dispatcher ID
Mover ID
Errors (if any)

Example:

# Show all processed files 
nd task processed 507f1f77bcf86cd799439011  
# Show only files with errors 
nd task processed --processed-nodes=error 507f1f77bcf86cd799439011

Pool Management

nd pool list

List pools with optional filtering.

Flags:

--content=TYPE - Filter by content type (cloud, nas, tape)
--type=TYPE - Filter by pool type (primary, active_archive, offline_archive, etc.)
--name=NAME - Filter by pool name
--comment=TEXT - Filter by pool comment
--id=ID - Filter by pool ID

Example:

# List all available
nd pool list

# List all NAS pools
nd pool list --content=nas

# List all primary pools
nd pool list --type=primary

# List specific pool by name
nd pool list --name=mypool

Output

╭───────────────────┬────────────┬───────────────────────────────────┬─────────┬─────────┬─────────────────┬────────────┬───────────────╮
│           POOL ID │ NAME       │ COMMENT                           │ TYPE    │ CONTENT │ PATH            │ STORAGE ID │ WRITE PROTECT │
├───────────────────┼────────────┼───────────────────────────────────┼─────────┼─────────┼─────────────────┼────────────┼───────────────┤
│                 2 │ nfsv3      │ NFS V3 Primary                    │ primary │ nas     │ srv/nfsshare-v3 │            │               │
│                 3 │ nfsv4      │ NFS V4 Primary                    │ primary │ nas     │ srv/nfsshare-v4 │            │               │
│                 7 │ source     │ Primary pool for source container │ primary │ nas     │                 │ 1          │               │
│                 8 │ dest       │ Primary pool for dest container   │ primary │ nas     │                 │ 1          │               │
│                16 │ Samba      │ Samba Primary                     │ primary │ nas     │ /mnt/samba      │            │               │
│                17 │ containers │ Container                         │ primary │ nas     │ mnt/containers  │            │               │
├───────────────────┼────────────┼───────────────────────────────────┼─────────┼─────────┼─────────────────┼────────────┼───────────────┤
│ NUMBER OF POOL(S) │ 6          │                                   │         │         │                 │            │               │
╰───────────────────┴────────────┴───────────────────────────────────┴─────────┴─────────┴─────────────────┴────────────┴───────────────╯

Last updated 5 hours ago

Was this helpful?

hashtagQuickstart

hashtagInstall nd

hashtagInstructions for Linux Users

hashtagInstructions for macOS Users

hashtagInstructions for Windows Users

hashtagND client installation (RPM example)

hashtagAccess to the interface

hashtagParameters

hashtagSyntax

hashtagGlobal Parameters

hashtagFilter Parameters

hashtagFilter Logic

hashtagAvailable Filter Flags

hashtagConfiguration

hashtagSSL

hashtagAuthentication and Authorization

hashtagAuthentication via Auth Service

hashtagAdmin/Privileged User Setup

hashtagNormal User Workflow

hashtagMultiple Clusters / Aliases

hashtagLDAP Server Configuration

hashtagAlias & default flags

hashtagData Mover Service Status

hashtagCommand

hashtagData Management Monitoring Services

hashtagData Management Log Management Services

hashtagData Mover Task Management

hashtagTask Creation

hashtagTask creation from directly Absolute and Relative path.

hashtagExamples of Tasks Creation

hashtagList created tasks

hashtagTasks status

hashtagList the status of all executed tasks

hashtagTask Control Commands

hashtagPool Management

Quickstart

Install nd

Instructions for Linux Users

Instructions for macOS Users

Instructions for Windows Users

ND client installation (RPM example)

Access to the interface

Parameters

Syntax

Global Parameters

Filter Parameters

Filter Logic

Available Filter Flags

Configuration

SSL

Authentication and Authorization

Authentication via Auth Service

Admin/Privileged User Setup

Normal User Workflow

Multiple Clusters / Aliases

LDAP Server Configuration

Alias & default flags

Data Mover Service Status

Command

Data Management Monitoring Services

Data Management Log Management Services

Data Mover Task Management

Task Creation

Task creation from directly Absolute and Relative path.

Examples of Tasks Creation

List created tasks

Tasks status

List the status of all executed tasks

Task Control Commands

Pool Management