| Ansible based installation | /spaces/gzbGsZqZH8Ro8zxRYgwn/pages/k0O7BcvIaVggywyjw80n |
| Catalog | Workflow Management | Trend Analytic | |
|---|---|---|---|
| Standard End-User | yes | no | no |
| Administrator User | yes | yes | yes |
System User Management Settings
| Barcode | Description |
|---|---|
| To prepare the tape for use, format the LTFS partition as it is currently unrecognized | |
| The tape is ready and can be used | |
| The tape has a write protection enabled | |
| The tape's capacity is full, no space left |
| Protect | Description |
|---|---|
| No tape protection is set | |
| The tape is protected by Nodeum software in using the lock feature | |
| The tape's physical write protection lock is enabled | |
| Nodeum software and physical write protection are both enabled |
| Status | Description |
|---|---|
| The tape is in good condition and ready | |
| The tape is flagged for review; a tape verification task must be performed |
#### Information Description
| Value | Description |
|---|---|
| Path | Path of the folder or file |
| Creation Date |
|
| Last Modification Date | Last modification date of files or folfer and its size |
| Type of Task (Offline , active Copy/move) | Ex. : it is the date of last offline copy done You can see the following details :
|
| In cache | The file or folder are save in Active archive workflows. The data is in cache save on a specfic container. |
Basket
| Type | Description | Example |
|---|---|---|
| Administrator see everything | if it is enable : allow all "administrator accounts" to see all contents which are in the catalog. If it is disable, the "administrator accounts" will not see the files they are not granted to see. | |
| Default Filter | to hide in the catalog the hidden files (.*) | -file_name:(.*) |
| Source | Destination | Type of action Mover |
|---|---|---|
Primary Storage :
| Secondary Storage :
|
|
| Container | Secondary Storage :
|
|
| Restore Source | Destination |
|
| Tape Pools |
|
|
| Cloud Pools |
|
|
| Nas Pools |
|
|
| Task |
|
|
| From Reference File |
|
|
This option lets you delete the original folder structure after a Move operation.
List of tasks history
List of Nodeum's tasks
Screenshot about a task execution
Move or Copy to a destination:
Network Shared Folder named Container
Nas
Tape
# parallel job(s)
# parallel DBFile/job
# B/job
Check integrity of data transfers: CRC32
Check integrity of data transfers: XXHAS64
Indicates that the tape is mounted with Read-Only LTFS file system and can't copy the file. The task will remount another tape.
It is highly recommended to check the tape manually.
| Log | Definition | Action |
|---|---|---|
| Drive %s is in Maintenance Mode. Set it Online to Start the Execution of this Task | Drive N°xx is on maintenance | Select another drive to run the task |
| Error during allocation of a scratch tape to the pool | impossible to get a tape from de scratch pool | Check if tape are in the scratch pool |
| File is not copied on any Tape on this pool | When we will reload a file and the files doesn't exist on a tape in the pool choosen | |
| Invalid cloud pool | Cloud pool doesn't exist | create the cloud pool before to run the task |
| Invalid Destination or Task destination is invalid | The destination choosen doesn't exist or the connection failed | Check if the destination exist |
| Invalid NAS pool | NAS pool doesn't exist | create the NAS pool before to run the task |
| Invalid Pool | Pool choosen doesn't exist | Check if the pool exist. If not create it |
| Pool does not have a valid tape and scratch pool is empty | No tape is in the pool and no tape in scratch pool | Insert a tape in the pool. Don't forget to format it before to run the task |
| Pool does not have a valid tape and unable to get a scratch volume for this task | No tape is in the pool and impossible to get a tape in scratch pool | Insert a tape in the pool. Don't forget to format it before to run the task |
| Tape %s is not in the library | Tape Nbre xx is not in the tape library | Put the tape in the library |
| Tape destination is invalid | barcode doesn't exist. The tape haven't mount and is out of tape library | |
| Tape pool destination is invalid | tape pool doesn't exist | create the tape pool before to run the task |
| Status | Definition | Actions |
|---|---|---|
| EACCESS | Permission Denied on File (in Source or Destination). | Check the privilege. |
| EINVAL | Invalid Argument. The device is not managed, or the device is not well configured. | Check the Configuration. |
| ENOSPC | Not enough space on the device. | Move Data from Cache to Secondary Storage |
| EROFS | The problem of writing in the file system. | Perhaps the file system is in Read-Only |
| FAILED | Some special characters in the name of the file could block the process of this file. The number of characters (in the file's name) is too long. | Rename the file with appropriated character(s) and restart the task. Check in the Primary Storage if the file is available. |
| FATALERR | The problem of DB update in the LTFS format. | |
| GETCACHERR | The file is removed from secondary storage. | |
| LTFSERR | LTFS error. Mount LTFS ERROR. | Make a tape Consistency. |
| MLERROR | Error during a data enrichment. | |
| NOTEMPTY | Impossible to delete a folder empty. | |
| NOFILE |
This situation is due because the file(s) are not on the source or the destination. | Check the connectivity of the Storage |
| OFFLINE |
|
|
| OK | Files copy or move with Success. | |
| PARTIAL | Copy did with success on a part of files | |
| TAPEERR | Tape drive or Tape Library on Error. | Check if the tape library is ready. |
| UPTODATE | The same file(s) already exist in the destination. |
| Type of Filter | Operator | Description |
|---|---|---|
| Creation Date | Older Than Earlier Than Before After | Filter the selection on when the file has been created. It allows to select only files that have been created before or after a specific date. |
| File Name | Matches Doesn't Match | It allows to filter the selection based on the name of the file. Examples : Regex expressions are also supported. |
| File extension | Equal Doesn't Match | It allows to filter your selection based on the file extension. File extension name has to be defined. Example: pdf |
| File Path | Matches Doesn't Match | It allows to filter your selection based on the folder(s) path. Regex expressions are also supported. |
| Last access date | Older Than Earlier Than Before After | Same but with a trigger on when the file has been accessed for the last time. |
| Last modification date | Older Than Earlier Than Before After | Filter the selection on when the file has been modified for the last time. It allows to select only files that have been modified before or after a specific date. |
| Size | More than Less than | Filter the file selection based on the files size |
| Result | The workflow will only process files with a size which is bigger than 234000KB |
|---|---|
| file | Match the process file |
| ">" | Comparison symbol |
| 234000 | File size |
List of available Containers
Form to create a container
| Configuration | Description | |
|---|---|---|
| Container Name | It is the name of the Container. | The name is fixed at creation and cannot be altered. If you are looking to use it as S3 target please refer to the Specifications to follow the naming conventions. |
| Comment | This allows adding a description or comment. | |
| Advanced Settings -> Protocol Activation | The 3 protocols (NFS - SMB - S3) are automatically available. | Security is applied specifically to each section per protocol. |
Automatic Push Data
Open the option icon and select Privileges
Schema of pool management
Tape Overview
Off Premise Storage Cost Calculator
On Premise Storage Cost Calculator
Nodeum System Information
System configuration metadata
System Configuration API Security
System Configuration API Cors
| 
|
| **Nodeum Web Console** | | |
| **Nodeum scheduler** | System Date & Time
System Service
| Name | Description | Mandatory |
|---|---|---|
| Notification Manager | Service managing all Nodeum services supervision and inter-process communication handler | yes |
| Core Manager | Workflow Manager task orchestration and storage connection handler | yes |
| Tape Library Manager | Tape Library and Drive management | yes |
| Data Mining | Data Collector and handling system statistics analyses | yes |
| File System Virtualization | Provide global file system virtualization across storage node. | no |
| Watchdog | Service Monitoring Manager which executes the automatic restart of failing service | yes |
| Ref. File Parsing | Service which handles the xml/json file parsing | no |
| Scheduler | Task Scheduling service which manages execution of task at selected time. | yes |
| File Listing Processing | Service that manages the Primary Storage mounting and its content parsing and inventory. | yes |
| Indexation Engine | Service that sync the database with the system and index the contents | no |
| Name | Description | Mandatory |
|---|---|---|
| Solr | Content Indexation Engine which provides fast access at each content | yes |
| NGINX | HTTP Web Service | yes |
| Delayed: Job | It executes tasks as a background process in Rails environment. It increases page rendering speed when asynchronous jobs are processed in the background. | yes |
| MySQL | Database Management | yes |
| MongoDB | Database Management | yes |
| MinIO | Data Transport - S3 Frontend Service | no |
| NFS | Data Transport - NFS Service | yes |
| SMB | Data Transport - SMB Service | yes |
| SNMP | Monitoring - SNMP Service | yes |
| SLAPD | Authentication - SLAPD Service | yes |
| NSLCD | Authentication - NSLCD Service | yes |
System Network
| Parameters | Definition |
|---|---|
| Name | Complete the name of the storage |
| Comment | |
| Protocol | Choose the protocol SMB or NFS |
| Version | Choose the version of the protocol For SMB : Version 1 - 2.1 and 3.0 For NFS : Version 3 - 4 |
| Host | Complete the Hostname |
Nas Target
| Parameters | Definition |
|---|---|
| Name | Complete the name of the storage |
| Comment | |
| Protocol | Choose the protocol SMB or NFS |
| Version | Choose the version of the protocol For SMB : Version 1 - 2.1 and 3.0 For NFS : Version 3 - 4 |
| Host | Complete the Hostname |
| Parameters | Definition |
|---|---|
| Name* | Complete the name you define in the toml (ansible playbook) |
| Comment | |
| Protocol | Choose unspecified (+++ Mandatory) |
| Host | Complete the Hostname |
| Parameters | Definition |
|---|---|
| NAS Name* | Select the NAS |
| Name* | Give a name for your target |
| Target | Use the target Path |
| Mount Options | |
| Username | |
| Password |
| Status | Definition | Comments |
|---|---|---|
 | NAS is Online | |
 | NAS is Offline | To reconnect the NAS, just push on the button to refresh the connection |
| Parameters | Definition |
| Name | Complete the name of the storage |
| Provider | Choose a provider |
| Region | Complete the region* |
| URL | Complete the url |
| Backend | Select the backend : Rclone or S3FS |
| Mount Options | |
| Access Key ID | Complete* |
| Secret access key | Complete* |
Tape overview
Tape Library Configuration - step1
Tape Library Configuration - Step2
Tape Library Configuration - Step3
| Mandatory Information | Description |
|---|---|
| Name | Complete a name |
| Emulation | Select the supported emulation for your Tape Library |
Tape Library Configuration - Step4
Tape drive Configuration - Step 1
Tape drive Configuration - Step 2
Tape drive Configuration - Step 3
Tape drive Configuration - Step 3
| Status | Description |
|---|---|
| Configured | Tape drive is configured |
| Not configured | Tape drive is not configured |
| Online | Tape drive is connected and operational |
| Offline | Tape drive is connected but in maintenance mode |
Tape Library Synchronisation - Step 1
Tape Library Synchronisation - Step 2
Tape Drive synchronisation - Step 1
Tape Drive synchronisation - Step 2
Tape Drive synchronisation - Step 3
System Audits Data Access
System Audit Data management
| Request type | Description |
|---|---|
GET | Access one or more resources and return the result as JSON. |
POST | Return 201 Created if the resource is successfully created and return the newly created resource as JSON. |
GET / PUT | Return 200 OK if the resource is accessed or modified successfully. The (modified) result is returned as JSON. |
DELETE | Returns 204 No Content if the resource was deleted successfully or 202 Accepted if the resource is scheduled to be deleted. |
| Return values | Description |
|---|---|
200 OK | The GET, PUT or DELETE request was successful, and the resource itself is returned as JSON. |
202 Accepted | The GET, PUT or DELETE request was successful, and the resource is scheduled for processing. |
204 No Content | The server has successfully fulfilled the request, and there is no additional content to send in the response payload body. |
201 Created | The POST request was successful, and the resource is returned as JSON. |
304 Not Modified | The resource hasn’t been modified since the last request. |
400 Bad Request | A required attribute of the API request is missing. For example, the title of an issue is not given. |
401 Unauthorized | The user isn’t authenticated. A valid user token is necessary. |
403 Forbidden | The request isn’t allowed. For example, the user isn’t allowed to delete a project. |
404 Not Found | A resource couldn’t be accessed. For example, an ID for a resource couldn’t be found. |
405 Method Not Allowed | The request isn’t supported. |
409 Conflict | A conflicting resource already exists. For example, creating a project with a name that already exists. |
422 Unprocessable | The entity couldn’t be processed. |
429 Too Many Requests | The user exceeded the application rate limits. |
500 Server Error | While handling the request, something went wrong on the server. |
| Parameter | Description |
|---|---|
page | Page number (default: 1). |
per_page | Number of items to list per page (default: 20, max: 100). |
$ curl -X GET -H "Accept: application/json" -H "Authorization: Bearer hv2ipsrv7y-JScJpV8h-x9w" "https://nodeum.local/api/v2/tapes?limit=10"
Verify an API endpoint
| Options | Default Value | Description |
|---|---|---|
| connect_timeout | "300" (seconds) | Time to wait for connection before giving up |
Default_acl
| “private” | By default, it is the "private" ACL which set the default canned ACL to apply to all written s3 objects.
The following URL lists these options: |
| ecs | disable | This option instructs a query to the ECS container credential metadata address instead of the instance metadata address. |
| enable_content_md5 | disable | Allow check data integrity of uploads via the MD5 checksum. This can add CPU overhead to transfers. |
| iam_role | no IAM role | This option requires the IAM role name or "auto". If you specify "auto", it will automatically use the IAM role names that are set to an instance. If you specify this option without any argument, it is the same as that you have specified the "auto". |
| list_object_max_keys | “1000” | This specify the maximum number of keys returned by S3 list object API. |
| max_stat_cache_size | "100,000" entries (about 40MB)) | maximum number of entries in the stat cache, and this maximum is also treated as the number of symbolic link cache. |
| multipart_size | 10 | Part size, in MB, for each multipart request. The minimum value is 5 MB and the maximum value is 5 GB. |
| multireq_max | 20 | This is about the maximum number of parallel requests for listing objects. |
| no_check_certificate | The server certificate won't be checked against the available certificate authorities. This is very useful if there is self-signed certificate. | |
| nomultipart | Disable multipart uploads | |
| parallel_count | 5 | Number of parallel requests for uploading big objects. by multipart post request and sends parallel requests. It is necessary to set this value depending on a CPU and a network band. |
| readwrite_timeout | "120" (seconds) | Time to wait between read/write activity before giving up |
| retries | 5 | Number of times the system will do a retry when an S3 transaction failed. |
singlepart_copy_limit | 512 | Maximum size, in MB, of a single-part copy before trying multipart copy. |
| ssl_verify_hostname | 2 | When 0, do not verify the SSL certificate against the hostname. |
| storage_class | "standard" | This option has to be supported by the storage vendor; it allows to store the objects with specified storage class. Possible values are standard, standard_ia, onezone_ia, reduced_redundancy and intelligent_tiering. |
Task maintenance
List Container's user
User Creation
User Creation Page
Group Creation
Group Creation Page
Container User Management
Edit Container
Define the Container's Priviledges
User detail
/mnt/FUSE *(rw,all_squash,anonuid=1000,anongid=2000,subtree_check,fsid=0)
System Notifications
| Hostname | relay.edpnet.be |
| From Email address | any type of email address |
| User Name | empty if your SMTP server doesn’t required any authentication |
| Password | empty if your SMTP server doesn’t required any authentication |
| Hostname | smtp.office365.com |
| From Email address | this has to be an email address which is recognized into your Office 365 account. |
| User Name | This has to be an authorized email address from your Office 365 organization. |
| Password | This is the password linked to the Username specified below. |
| Hostname | smtp.gmail.com:587 |
| From Email address | this has to be an email address which is recognized into your Google Mail account. |
| User Name | This has to be an authorized email address in your Google Mail organization. |
| Password | For passwords, it is required to create an App password in Google portal. This will allows to by pass the 2FA mechanism. |
SNMP
| Version | SNMP version you want to configure. |
| Community | The community string you want to use; by default public should be used. |
| System Location | not mandatory but can be useful to define the solution location. |
| System Contact | not mandatory but can be useful to define the solution point of contact. |
| Destination address | Hostname or IP address of the server where the trap has to be sent. |
| Community | The community string which is configured in your trap handler software. |
| Version | SNMP version which is supported by your trap handler software. |
| Notification level | Define the level of severity that will trigger the SNMP trap messages. (see point 3) |
| 0 | Message | Action Notification: Level 0 Activation Required To receive notifications regarding task completion, please activate Level 0. |
| 1 | Information | Severity Level: Informational This message serves as an informational notice, indicating that no error has occurred. No response is necessary. It may signal ongoing operations or the successful completion of a task. |
| 2 | Warning | This message indicates a potential error where the software defaults to supplanted input due to missing data. The operation's outcome is assumed to align with the user's intent. |
| 3 | Error | Error Detection and Recovery Notice An error has been detected. While automatic recovery procedures were likely applied, and processing continued, a default input may have been substituted. Consequently, the operation's results might be invalid. The function may only be partially complete; for instance, while some items in a list may be processed correctly, others might fail. |
| 4 | Severe | Error Handling Notification Issue: A critical error occurred that cannot be automatically resolved. Default settings cannot be applied. Implications:
Outcome: The operation results are invalid. |
| 5 | Abnormal End of Program or Function | The operation concluded, either due to invalid data handling issues or a user termination. |
| 6 | Abnormal End of Task | The task was either not initiated or did not complete successfully. Potential issues include a routing step failing to start or ending unexpectedly, a job-level function not being executed as required, or the overall job being terminated. |
| 7 | System Status | This type of urgent message is sent exclusively to the system operator, providing either a status update or a warning concerning a device, subsystem, or the entire system. |
| 8 | Device Integrity | A message of this severity is sent solely to the system operator. It indicates a device malfunction or that a device is no longer operational. You might restore system operation yourself, or you may need the help of a service representative. |
| 9 | System Alert | A message of this severity is issued only to the system operator. It warns of a condition that, although not severe enough to stop the system now, could become more severe unless preventive measures are taken. |
| 10 | System Integrity | A severe message is issued solely to the system operator. It details a condition that makes a subsystem or the entire system non-functional. |
| File | Description |
|---|---|
| All | This gather all below informations |
| Data Mining | Statistics about product usage. |
| File System Virtualization | This is about the container usage feature. |
| Library Manager | This is about the Tape Library and Tape Drive Management. |
| Manager | This is about the the Workflow Manager and help to solve issue which happened during tasks executions. |
| Watchdog | This is about the watchdog and how services are monitored |
| System | Anything about the system itself. |
| Interface | Anything about the Console |
Event Logs
| Operating System | Version | Architecture | |
|---|---|---|---|
| Linux | RHEL | 8 - 9 | x86-64 |
| Rocky | 8 - 9 | x86-64 | |
| Almalinux | 8 - 9 | x86-64 | |
| Ubuntu | 22.04 LTS | x86-64 | |
| Ubuntu | 24.04 LTS | x86-64 |
| Up to 10 Million Files | Up to 100 Million Files | Up to 1 Billion Files | |
|---|---|---|---|
| Memory | 32 GB | 128GB | Upon Request |
| Processors | 6 | 2 x 12 | Upon Request |
| System Disk | 110 GB | 900 GB | Upon Request |
| Client Operating system | Version | Access Container | Access Console |
|---|---|---|---|
| Windows | Windows 10 - 11
Windows Server 2008, 2012, 2016 | SMB / NFS | HTTP/HTTPS |
| MacOS | MacOS | SMB / NFS | HTTP/HTTPS |
| Linux | Ubuntu
Debian Fedora CentOS RHEL | SMB / NFS | HTTP/HTTPS |
| Type | Definition |
|---|---|
| Full Backup | Backup everything every time |
| Incremental Backup | Backup only the difference since the last backup |
| Option | Definition | Comments |
|---|---|---|
| Timezone | The schedule can be defined based on a different timezone | Defined by default, it will be current system timezone |
| Recurrence | Once | Defined by default , there is no recurrence selected |
| Every | Minute - Day - Month - Year | |
| Expands | Expand the scope of the recurrence | |
| Count Option | Number of backup |
| Components | Description | |
|---|---|---|
| All databases | This includes SQL and NoSQL information: file catalog, metadata, task execution history. | |
| S3 - Swift API Keys | Authentication Key Credentials of S3 and Swift storage provider. | |
| Data enrichment API Keys | Authentication Key Credentials of Machine Learning provider. | |
| Tasks execution log reports | History of each files processed per task. | |
| Licence Key | Nodeum Encrypted License Key |
| Nodeum Services | Resiliency Level |
|---|---|
| Notification Manager | Failover |
| Core Manager | Failover |
| Tape Library Manager | Failover |
| Data Mining | Failover |
| File System Virtualization | Failover |
| Watchdog | Failover |
| Ref. File Parsing | Failover |
| Scheduler | Failover |
| File Listing Processing | Failover |
| Indexation Engine | Failover |
| System Services | RESILIENCY LEVEL |
|---|---|
| CACHE Disk | Cluster |
| Solr | Cluster |
| NGINX | Cluster |
| MariaDB | Cluster |
| MongoDB | Cluster |
| SMB | Cluster |
| NFS | Cluster |
| MinIO | Not yet available |
| Ansible Based Installation | /pages/FPZIQ2Woi6IFUDEClO6q | |||
| Docker Based Deployement | /pages/j6DaXbHP5pjfBNh7Um31 | |||
| Slurm Integration | /pages/0aBGj2ZGotTlvfVYh6kh |
/etc/nodeum/mover.toml
--
...
[[storages.options]]
# Applying additional options to connect to POSIX file system
type = "nas-share"
parent_name = "data1"
[storages.options.handler]
# will always be considered mounted in this directory, no mounting will be
attempted
always_mounted = true
path = "/mnt/data1"
...
$ yum install wget
$ wget get.nodeum.io/public/nodeum-docker-compose-v2-x-y-zzz.tar.gz
$ tar -xvzf /root/nodeum-docker-compose-v2-x-y-zzz.tar.gz
$ cd v2-x-y
$ docker image pull mariadb:10.6
$ docker image pull mongo:4.4
$ docker image pull redis:6.2.6
$ docker image pull quay.io/coreos/etcd:v3.5.8
$ docker image save mariadb:10.6 mongo:4.4 redis:6.2.6 quay.io/coreos/etcd:v3.5.8 |
gzip > services.tgz
| Catalog | Workflow Management | Trend Analytic | |
|---|---|---|---|
| Standard End-User | yes | no | no |
| Administrator User | yes | yes | yes |
List of tasks history
List of Nodeum's tasks
Screenshot about a task execution
Move or Copy to a destination:
Nas
Tape