Links

ND Client

The Nodeum Client nd command line tool provides a modern set of commands to execute data movement operations with Nodeum. The nd command line tool is built for compatibility with the Nodeum v2 for expected functionality and behavior.
nd has the following syntax:
nd [GLOBALFLAGS] COMMAND --help
See Command Quick Reference for a list of supported commands.
Copyright
nd is a property of Nodeum and its subsidaries, if any. The intellectual and technical concepts contained herein are proprietary to Nodeum and its subsidaries and may be covered by Belgium and Foreign Patents, patents in process, and are protected by trade secret or copyright law. Dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Nodeum.
Related Version: ND-2.0.8

Quickstart

Install nd

Install the nd command line tool onto the host machine. Go in the section that corresponds to the host machine operating system or environment:

Linux

The following commands add a temporary extension to your system PATH for running the nd bash client. Defer to your operating system instructions for making permanent modifications to your system PATH.
Alternatively, execute nd by navigating to the parent folder and running ./nd --help
$ curl https://get.nodeum.io/public/nd-2.x.y-linux-amd64 \
\--create-dirs \
\-o $HOME/nodeum-binaries/nd
$ chmod +x $HOME/nodeum-binaries/nd export PATH=$`PATH:`$HOME/nodeum-binaries/
nd --help

macOS

$ curl https://get.nodeum.io/public/nd-2.x.y-darwin-amd64 \
\--create-dirs \
\-o $HOME/nodeum-binaries/nd
$ chmod +x $`HOME/nodeum-binaries/nd export PATH=`$`PATH:`$HOME/nodeum-binaries/
nd --help

Windows

Open the following file in a browser:
https://get.nodeum.io/public/nd-2.0.7-windows-amd64
Once download, rename the file to nd.exe and execute the file by double clicking on it, or by running the following in the command prompt or powershell:
\path\to\nd.exe --help

ND client installation (RPM example)

$ wget https://get.nodeum.io/public/nd-2.x.y-x86_64.rpm
$ dnf install ./nd-2.x.y-x86_64.rpm

1. Access to the interface

$ nd
NAME:
nd - Nodeum CLI
USAGE:
nd [global options] command [command options] [arguments...]
VERSION:
2.0.8
COMMANDS:
admin
config configure the Nodeum Client
copy, cp create copy task
move, mv create move task
task
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--json output as JSON (default: false)
--config value path to configuration file (default: <config-dir>/
config.json) [$ND_CONFIG]
--config-dir value, -C value path to configuration folder (default: "/home/
nodeum01/.config/.nd") [$ND_CONFIG_DIR]
--alias value alias in configuration file for authentication
(default: "default") [$ND_ALIAS]
--url value URL of Nodeum [$ND_URL]
--access-token value for API authentication (1st authentication method)
[$ND_ACCESS_TOKEN]
--refresh-token value for API authentication (1st authentication method,
not saved in config) [$ND_REFRESH_TOKEN]
--authorization-endpoint value for Device Authorization Flow (2nd authentication
method)
--token-endpoint value for Device Authorization Flow (2nd authentication
method)
--client-id value for Device Authorization Flow (2nd authentication
method)
--scopes value for Device Authorization Flow (2nd authentication
method)
--persist-session persist Device Authorization session on disk for 1
hour (default: true)
--persist-session-renew if persist session is enabled, renew the token
(default: false)
--username value for API authentication (3rd authentication method)
[$ND_USERNAME]
--password value for API authentication (3rd authentication method)
[$ND_PASSWORD]
--anonymous no login (default: false)
--help, -h show help
--version, -v print the version
COPYRIGHT:
nd is a property of Nodeum and its subsidiaries, if any. The intellectual and
technical concepts contained herein are proprietary to Nodeum and its
subsidiaries and may be covered by Belgium and Foreign Patents, patents in
process, and are protected by trade secret or copyright law. Dissemination of
this information or reproduction of this material is strictly forbidden unless
prior written permission is obtained from Nodeum.

2. Command Quick Reference

The following table lists nd commands:
Name
Shortcut
Description
admin
Access the administration commands
config
Configure the Nodeum client
copy
cp
Create a task to copy data between two storages
move
mv
Create a task to move data between two storages
task
List detailed information about created task
help
h
Displays a summary of command usage and parameters on the terminal

Parameters

Syntax

The nd client provides a Bash completion mechanism to facilitate the search of commands.
Metadata key can't include an = character.

Global Parameters

Name
Shortcut
Description
Default
--json
Output as JSON
false
--config value
-C value
Path to configuration file
/home/nodeum01/
.config/.nd/
config.json
--config-dir value
-C value
Path to configuration folder (default: "/home/nodeum01/.config/.nd")
--alias value
Alias in configuration file for authentication
default
--url value
URL of Nodeum
--access-token value
for API authentication (1st authentication method)
--refresh-token value
for API authentication (1st authentication method, not saved in config)
--authorization-endpoint value
for Device Authorization Flow (2nd authentication method)
--token-endpoint value
for Device Authorization Flow (2nd authentication method)
--client-id value
for Device Authorization Flow (2nd authentication method)
--scopes value
for Device Authorization Flow (2nd authentication method)
--persist-session
persist Device Authorization session on disk for 1 hour
true
--persist-session-renew
if persist session is enabled, renew the token
false
--username value
for API authentication (3rd authentication method)
--password value
for API authentication (3rd authentication method)
--anonymous
no login
false
--help
-h
show help
false
--version
-v
print the version
false

Mandatory Parameters for OpenID integration

Parameters are available for each data movement task.
Name
Description
Value
--md project_name=
Name of project defined in OpenStack
string
--md project_domain_name=
Name of the project's domain defined in OpenStack
string
--md user_domain_name=
Name of the project's user defined in OpenStack
string
--md region_name=
Name of the region's project defined in OpenStack
string

Standard Parameters

Parameters are available for each data movement task.
Name
Shortcut
Description
Value
Default value
--help
-h
Show help
false
--no-run
Create a task and don't launch the task directly
false
--name value
-n
Name of task
string
automatically generated
--comment value
Comment of task
string
empty
--overwrite value
Overwrite all identical files already stored at destination
true - false
false
--priority value
Priority of the task, between 0 and 9 (0 is the highest priority)
0 - 9
0
--recursive
-R
Execute a recursive copy of the folder. If subfolders are present, the service will also copy the contents of each subfolder
false
--working-dir value
--wd
Defines the workpath to be kept at destination
. - .. - path
0
--ignore-hidden value
Task will not handle hidden file(s)
true - false
false
--progress value
Display live progress when running the task
true - false
true
--processed-nodes value
Display the processed nodes when running a task when the flag--progress is set
none, error, all
error

Advanced Parameters

Parameters are available for each data movement task.
Name
Shortcut
Description
Value
Default value
--context-uid value
--uid
Define the User ID which will handle the movement
integer
unset
--context-gid value
--gid
Define the Group ID which will handle the movement
integer
unset
--defer
when requesting the run of the task, will defer it for later with an unique ID
false
--parallel value
Define the number of mover which will handle the movement. Maximum value is determined by the deployed implementation.
1-20
1
--callback type
Add callback. Format is type:./path/to/file
./path/to/file
--trigger-md key=value
--md key=value
Set metadata on the trigger. Format is key=value. Accepts multiple inputs
key=value
--task-md key=value
Set metadata on the task. Format is key=value. Accepts multiple inputs
key=value
--files-md key=value
Set metadata on the files. Format is key=value. Accepts multiple inputs
key=value

3. Configuration

nd uses a JSON formatted configuration file used for storing certain kinds of information, such as the authentication and authorization options. By default, this configuration file is unique by user. It is stored in its home directory.
For Linux and macOS, the default configuration file location is .config/.nd/config.json which is store in the $HOME. For Windows, the configuration file is stored in $AppData$.
You can display the configuration file location in using the command nd --help:
$ nd --help
NAME:
nd - Nodeum CLI
....
--config value path to configuration file (default: <config-dir>/
config.json) [$ND_CONFIG]
--config-dir value, -C value path to configuration folder (default: "/home/
nodeum01/.config/.nd") [$ND_CONFIG_DIR]
You can use the --config value where value is the path to a JSON formatted configuration file that nd uses for storing data. The ND_CONFIG environment variable can be used to set the value.
Store the configuration file in a 'central' directory to allow each user to get the same nd client configuration. For this, the --config-dir value is available.

Command

Command
Description
--config value
this option specifies the JSON filename where the configuration is stored
--config-dir value
this option specifies the directory where the JSON configuration is stored

SSL

nd client allows SSL configuration to communicate with the Data Mover service which listen in HTTPS. It is required to add the public certificate generated with the server.
The public certificate file has to be stored in this config-dir folder: .config/.nd/certs/CAs/.
The certificates on the server side have been generated following this command:
$ openssl11 req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
\-keyout ./nodeum_certs/private.key \
\-out ./nodeum_certs/public.crt \
\-subj "/CN=nodeum-qualif.mt-c.local" \
\-addext "subjectAltName=DNS:nodeum.domain.local,DNS:localhost,IP:127.0.0.1,
IP:1.1.1.1"
Where 1.1.1.1 is the ip of the nginx interface and nodeum.domain.local is its hostname including domain name.

Authentication and Authorization

Description

Two authentifcation options are available:
  • Username / Password
  • IDP with OpenID

Command

Command
Description
nd config save
this command save the nd configuration for authentication
nd config review
this command display the information related to the credentials
nd config clear-session
clear persisted session

Username / Password configuration

The nd client provides a basic method of authentication in using Username / Password credentials.
$ nd config save \
\--url=http:// \
\--username= \
\--password=
Option
Description
--url
this refer to the node which hosts the following service DATA MANAGEMENT WEB SERVICES
--username
this is the username to grant authorization to the service
--password
this is the password associated to the username that will grant the access to the service

OpenID configuration

The nd client provides an OpenID authentication mechanism. In this case, nd Client has to be configured with the appropriated IDP to handle proper token management.
The basic configuration is the following one:
$ nd config save \
\--url=http:// \
\--authorization-endpoint= \
\--token-endpoint= \
\--client-id= \
\--scopes="openid,profile,offline_access,email"
Option
Description
--url
this refer to the node which hosts the following service DATA MANAGEMENT WEB SERVICES
--authorization-endpoint
this is the endpoint url to grant authorization to the service
--token-endpoint
this url is used to programmatically request tokens
--client-id
this refer to the client identifier which is provided by the OpenID provider
--persist-session
persist Device Authorization session on disk for 1 hour (default: true)
--persist-session-renew
if persist session is enabled, renew the token (default: false)
The standard behavior is to request a token, which is automatic if there is no token available. The token will be stored in a cache during 15 minutes. The --persist-session-renew option can be defined to true to force a request token process any time the user has to interface with the nd client.
The token renewal is automatic based on the renewal token.

Alias & default flags

Description

Alias & default flags allow structuration of different group of settings. Different alias can be defined in the configuration file.

Definition of alias & default flags

Alias and flags are declared in the configuration file ~/.config/.nd/config.json". Default flags are defined for each available parameter command. Flags can be overwritted in the nd command.
Example in config file:
{
// ...
// default: name of alias used by default
"default": {
// ...
"default_flags": {
// this is the name of the command, example: nd copy
"copy": {
// this is the flag, example: nd copy --md project_domain_name=MYPROJECTDOMAIN
--md user_domain_name=MYUSERDOMAIN
"md": ["project_domain_name=MYPROJECTDOMAIN", "user_domain_name=MYUSERDOMAIN"]
}
}
},
// myorganisation: name of another alias
"myorganisation": {
// ...
}
}

Alias usage

The nd command allows the usage of alias, example : nd --alias myorganisation copy

4. Data Mover Service Status

Command

Command
Description
nd admin status
retrieve the status and the health of each services part of the cluster
nd admin logs
retrieve all Logs and return their contents

Data Management Monitoring Services

nd admin status command requests the Data Management Monitoring service to retrieve the status and health of each services part of the cluster. This command returns list of services. The following information are displayed:
  • Service Status
  • Service version
  • Host where the service is deployed
  • Its uptime
  • Consumed memory

Output where all services are reachable

╭───┬───────────────────┬────────────────────────────┬─────────────────┬────────────┬───────────╮
│ │ NAME │ VERSION │ ADDRESS │ UPTIME │ MEMORY │
├───┼───────────────────┼────────────────────────────┼─────────────────┼────────────┼───────────┤
│ ● │ micro.http.broker │ ff.http.broadcast │ 10.2.2.83:50050 │ N/A │ N/A │
│ ● │ nodeum.dispatcher │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.84:50010 │ 172h11m8s │ 34.92 MB │
│ ● │ nodeum.dispatcher │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50010 │ 172h13m32s │ 22.61 MB │
│ ● │ nodeum.front │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50000 │ N/A │ N/A │
│ ● │ nodeum.monitoring │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.85:50060 │ 171h2m47s │ 17.20 MB │
│ ● │ nodeum.mover │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.84:50020 │ 172h4m48s │ 25.23 MB │
│ ● │ nodeum.mover │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.85:50020 │ 172h4m48s │ 25.63 MB │
│ ● │ nodeum.scheduler │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50040 │ 172h18m6s │ 6.91 MB │
│ ● │ mongo │ 4.4.17 │ 10.2.2.83:27017 │ 268h0m44s │ 212.86 MB │
│ ● │ mongo │ 4.4.17 │ 10.2.2.84:27017 │ 268h0m44s │ 212.86 MB │
│ ● │ mongo │ 4.4.17 │ 10.2.2.85:27017 │ 268h0m44s │ 212.86 MB │
│ ● │ mysql │ 10.6.11-MariaDB │ 10.2.2.84:3306 │ 150h21m0s │ N/A │
│ ● │ redis-sentinel │ 6.2.6 │ 10.2.2.83:26379 │ 150h16m52s │ N/A │
│ ● │ redis-sentinel │ 6.2.6 │ 10.2.2.84:26379 │ 150h16m52s │ N/A │
│ ● │ redis-sentinel │ 6.2.6 │ 10.2.2.85:26379 │ 150h16m52s │ N/A │
│ ● │ redis │ 6.2.6 │ 10.2.2.83:6379 │ 285h37m50s │ N/A │
│ ● │ redis │ 6.2.6 │ 10.2.2.84:6379 │ 169h59m59s │ N/A │
│ ● │ redis │ 6.2.6 │ 10.2.2.85:6379 │ 169h59m59s │ N/A │
│ ● │ etcd │ 3.4.13 │ 10.2.2.83:2379 │ N/A │ N/A │
│ ● │ etcd │ 3.4.13 │ 10.2.2.84:2379 │ N/A │ N/A │
│ ● │ etcd │ 3.4.13 │ 10.2.2.85:2379 │ N/A │ N/A │
╰───┴───────────────────┴────────────────────────────┴─────────────────┴────────────┴───────────╯

Output where some services are not available

╭───┬───────────────────┬────────────────────────────┬─────────────────┬────────────┬───────────╮
│ │ NAME │ VERSION │ ADDRESS │ UPTIME │ MEMORY │
├───┼───────────────────┼────────────────────────────┼─────────────────┼────────────┼───────────┤
│ ● │ micro.http.broker │ ff.http.broadcast │ 10.2.2.83:50050 │ N/A │ N/A │
│ ● │ nodeum.dispatcher │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.84:50010 │ 172h11m8s │ 34.92 MB │
│ ● │ nodeum.dispatcher │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50010 │ 172h13m32s │ 22.61 MB │
│ ● │ nodeum.front │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50000 │ N/A │ N/A │
│ ● │ nodeum.monitoring │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.85:50060 │ 171h2m47s │ 17.20 MB │
│ ● │ nodeum.mover │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.84:50020 │ 172h4m48s │ 25.23 MB │
│ ● │ nodeum.mover │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.85:50020 │ 172h4m48s │ 25.63 MB │
│ ● │ nodeum.scheduler │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50040 │ 172h18m6s │ 6.91 MB │
│ ● │ mongo │ 4.4.17 │ 10.2.2.83:27017 │ 268h0m44s │ 212.86 MB │
│ ● │ mongo │ 4.4.17 │ 10.2.2.84:27017 │ 268h0m44s │ 212.86 MB │
│ ● │ mongo │ 4.4.17 │ 10.2.2.85:27017 │ 268h0m44s │ 212.86 MB │
│ ● │ mysql │ 10.6.11-MariaDB │ 10.2.2.84:3306 │ 150h21m0s │ N/A │
│ ● │ redis-sentinel │ 6.2.6 │ 10.2.2.83:26379 │ 150h16m52s │ N/A │
│ ● │ redis-sentinel │ 6.2.6 │ 10.2.2.84:26379 │ 150h16m52s │ N/A │
│ ● │ redis-sentinel │ 6.2.6 │ 10.2.2.85:26379 │ 150h16m52s │ N/A │
│ ● │ redis │ 6.2.6 │ 10.2.2.83:6379 │ 285h37m50s │ N/A │
│ ● │ redis │ 6.2.6 │ 10.2.2.84:6379 │ 169h59m59s │ N/A │
│ ● │ redis │ 6.2.6 │ 10.2.2.85:6379 │ 169h59m59s │ N/A │
│ ● │ etcd │ 3.4.13 │ 10.2.2.83:2379 │ N/A │ N/A │
│ ● │ etcd │ 3.4.13 │ 10.2.2.84:2379 │ N/A │ N/A │
│ ● │ etcd │ 3.4.13 │ 10.2.2.85:2379 │ N/A │ N/A │
│ ○ │ nodeum.mover │ │ │ │ │
╰───┴───────────────────┴────────────────────────────┴─────────────────┴────────────┴───────────╯

Data Management Log Management Services

nd admin logs command requests the Data Management Log Management Services to retrieve all Logs and return their contents.
$ nd admin logs --help
NAME:
nd admin logs - display application logs
USAGE:
nd admin logs [command options] [arguments...]
OPTIONS:
--since value, -S value show logs not older than the specified date
--until value, -U value show logs not newer than the specified date
--tag value, -u value [ --tag value, -u value ] filter by tags
--level value minimum level of logs, one of [trace debug info warn
error fatal]
--help, -h show help (default: false)
This command allows different parameters:
Option
Description
since
to only show logs not older than a specified date
until
to only show logs not newer than a specified date
tag
to filter the logs per type of services. Example nodeum.monitoring
level
to define the minimum level of logs, one of [trace - debug - info - warn - error - fatal]

Export

The Logs can be exported in using the standard OS mechanism of exportation
$ ./nd admin logs > 'nodeum_site-name_log.txt'
Where nodeum_site-name_log.txt is the file name and site represents the name of the site.

5. Data Mover Task Management

Follow these steps to create a task:
  1. 1.
    Use Mandatory Parameters
  2. 2.
    Define the data source
  3. 3.
    Define the destination
  4. 4.
    Apply others parameters if needed

Task Creation

The nd copy command send a copy request to the data mover service from a storage A (nod://posix_storage/) to a storage B (nod-cloud://cloud_storage/).

Command with minimal parameters

nd copy \ --md project_name=<my project name> \ --md project_domain_name=<my projectdomain name> \ --md user_domain_name=<my user domain name> \ nod://posix_storage/path/subpath/ \ nod-cloud://cloud_storage/container
Detailed syntax
$ nd copy \
--md project_name=<my project name> \
--md project_domain_name=<my projectdomain name> \
--md user_domain_name=<my user domain name> \
nod://posix_storage/path/subpath/ \
nod-cloud://cloud_storage/container

Example of creation task with additional parameters

nd copy \ --md project_name=<my project name> \ --md project_domain_name=<my project domain name> \ --md user_domain_name=<my user domain name> \ --working-dir nod://largedata2_pool/storagetestdata/ \ --recursive \ nod://posix_storage/path/subpath/ \ nod-cloud://cloud_storage/container
Detailed syntax
$ nd copy \
--md project_name=<my project name> \
--md project_domain_name=<my project domain name> \
--md user_domain_name=<my user domain name> \
--working-dir nod://posix_storage/path/ \
--recursive \
nod://posix_storage/path/subpath/ \
nod-cloud://cloud_storage/container

Available Parameters

Option
Description
--callback type:./path/to/file
add callback. Format is type:./path/to/file (accepts multiple inputs)
--defer
when requesting the run of the task, will defer it for later with an unique ID (default: false)
--no-run
just create the task, don't run it (default: false)
--progress
when running the task, display live progress (default: true)
--recursive
copy directories recursively (default: false)
--ignore-hidden
ignore hidden files and folders, starting with (default: false)
--overwrite
overwrite existing entries (default: false)
--priority value
task priority [0..9] (default: 0)
--working-dir value
set working directory

Working Directory Explanation

The definition of a working directory allows to define where the files will be stored at the destination. Different options are available, they are described in the following definition.
With --wd=.
Source
Destination
Result
nod://source/folder/FILE.txt
nod://dest/directory/
nod://dest/directory/FILE.txt
nod://source/folder/FILE.txt
nod://dest/RENAMED.txt
nod://dest/RENAMED.txt
nod://source/folder/
nod://dest/directory/
nod://dest/directory/FILE.txt
nod://source/folder/
nod://dest/directory
nod://dest/directory/FILE.txt
nod://source/folder
nod://dest/directory/
nod://dest/directory/folder/FILE.txt
nod://source/folder
nod://dest/directory
nod://dest/directory/FILE.txt
With --wd=..
Source
Destination
Result
nod://source/folder/FILE.txt
nod://dest/directory/
nod://dest/directory/folder/FILE.txt
nod://source/folder/FILE.txt
nod://dest/RENAMED.txt
nod://dest/RENAMED.txt
nod://source/folder/
nod://dest/directory/
nod://dest/directory/folder/FILE.txt
nod://source/folder/
nod://dest/directory
nod://dest/directory/FILE.txt
nod://source/folder
nod://dest/directory/
nod://dest/directory/source/folder/FILE.txt
nod://source/folder
nod://dest/directory
nod://dest/directory/FILE.txt

Examples of Tasks Creation

Execute a task copy from Posix to Swift

nd copy \ --md project_name=<my project name> \ --md project_domain_name=<my project domain name> \ --md user_domain_name=<my user domain name> \ --working-dir nod://posix_storage/path/ \ --recursive \ --ignore-hidden \ nod://posix_storage/path/subpath/ \ nod-cloud://cloud_storage/container
Detailed syntax
$ nd copy \
--md project_name=<my project name> \
--md project_domain_name=<my project domain name> \
--md user_domain_name=<my user domain name> \
--working-dir nod://posix_storage/path/ \
--recursive \
--ignore-hidden \
nod://posix_storage/path/subpath/ \
nod-cloud://cloud_storage/container

Execute a task copy from Swift to Posix

nd copy \ --md project_name=<my project name> \ --md project_domain_name=<my project domain name> \ --md user_domain_name=<my user domain name> \ --working-dir nod://posix_storage/path/ \ --recursive \ --ignore-hidden \ nod-cloud://cloud_storage/container/path/ \ nod://posix_storage/path/
Detailed syntax
$ nd copy \
--md project_name=<my project name> \
--md project_domain_name=<my project domain name> \
--md user_domain_name=<my user domain name> \
--working-dir nod-cloud://cloud_storage/container/ \
--recursive \
--ignore-hidden \
nod-cloud://cloud_storage/container/path/ \
nod://posix_storage/path/

Execute a task move from Posix to Swift

nd move \ --md project_name=<my project name> \ --md project_domain_name=<my project domain name> \ --md user_domain_name=<my user domain name> \ --working-dir nod://posix_storage/path/ \ --recursive \ --ignore-hidden \ nod://posix_storage/path/subpath/ \ nod-cloud://cloud_storage/container
Detailed syntax
$ nd move \
--md project_name=<my project name> \
--md project_domain_name=<my project domain name> \