> For the complete documentation index, see [llms.txt](https://docs.nodeum.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.nodeum.io/data-archiving/monitoring.md).

# Monitoring

## NRPE Monitoring

### What is available

NRPE (Nagios Remote Plugin Executor) is a small daemon that runs on the monitored host and executes local check scripts on behalf of a remote monitoring server, returning a standard Nagios-style status: `OK`, `WARNING`, `CRITICAL`, or `UNKNOWN`.

Three Nodeum-specific plugins are provided, all Python 3 scripts:

| Plugin                     | Checks                                                                                                                              |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `check_nodeum_services.py` | That the core Nodeum services (and their dependencies: Solr, nginx, MariaDB, MongoDB) are active via `systemctl`                    |
| `check_nodeum_scsi.py`     | That the tape drives/media changers known to Nodeum (from the `drives` DB table) are actually visible as SCSI devices (`lsscsi -g`) |
| `check_nodeum_tasks.py`    | That recent task executions (archive/rehydration/etc. rules) haven't failed, errored, or stopped within a given time window         |

#### check\_nodeum\_services

Runs `systemctl is-active` against two lists of services:

* **Critical**: `nodeum_manager`, `nodeum_scheduler`, `nodeum_indexation_engine`, `nodeum_dm`, `nodeum_file_system_virt`, `solr`, `nginx`, `mariadb`, `mongod`.
* **Optional**: `nodeum_watchdog`, `nodeum_notifications`.

Arguments:

* `-c/--critical` (default `1`): number of failed critical services that triggers `CRITICAL`.
* `-w/--warning` (default `1`): number of failed optional services that triggers `WARNING`.

Exit codes: `CRITICAL` (2) if failed critical services ≥ `-c`; else `WARNING` (1) if failed optional services ≥ `-w`; else `OK` (0). `UNKNOWN` (3) if thresholds are negative or `-w` > `-c`.

#### check\_nodeum\_scsi

Cross-checks two sources:

1. The Nodeum database `drives` table (vendor, product, device path, sg device) — i.e. what Nodeum *expects* to find.
2. The live output of `lsscsi -g` — i.e. what the OS actually *sees*.

It fails if `lsscsi` can't be parsed, if the DB lists drives but no `mediumx`-type device exists at all (the media changer), or if a specific expected drive (matched by vendor + product name) can't be found among the `tape`-type SCSI devices.

Arguments: `-d/--debug` only (verbose output). No thresholds — it's a binary "all expected drives present" check. Exit codes: `CRITICAL` (2) on any mismatch or parse failure, `OK` (0) otherwise.

#### check\_nodeum\_tasks

Queries `stats_info_exec_rule` for task executions that finished, within a `[start, end)` time window, with status `3` (error), `6` (stopped\_by\_system), or `8` (finished\_with\_warnings). `3`/`6` count as critical, `8` counts as warning.

Arguments:

* `-s/--start-time`, `-e/--end-time` (**required**): Unix timestamp or `YYYY-MM-DD HH:MM:SS` / `YYYY-MM-DD HH:MM` / `YYYY-MM-DD`.
* `-w/--warning` (default `5`), `-c/--critical` (default `10`): thresholds on the number of warning/critical executions found.
* `-d/--debug`.

Exit codes: `CRITICAL` (2) if critical count ≥ `-c`; else `WARNING` (1) if warning count ≥ `-w`; else `OK` (0). `UNKNOWN` (3) on bad thresholds or a DB error.

This check is meant to be called with a **rolling window** (e.g. "the last hour") computed by the monitoring server/scheduler, not a fixed date — see the example in section 3.

### How the plugins can be deployed

#### Ansible

The `nodeum.nrpe` role automates everything below, and is part of the standard Nodeum installation playbook (`install.yml`). It runs on every host where `nrpe_enabled: true`.

To enable it on a given server, edit that host's inventory file (`inventory/host_vars/srv01`) ; this section is already present there, commented out / disabled by default:

```yaml
###! NRPE Configuration
# If using ssl is enabled, the certificate files are expected to be in:
# /etc/nodeum/certs/CAs/public.crt
# and /etc/nodeum/certs/private.key
# This can be changed post installation in: /etc/nrpe.d/nodeum_nrpe.cfg
nrpe_enabled: true
nrpe_use_ssl: true
# NRPE additional allowed hosts (comma-separated, no leading comma needed, use 0.0.0.0 to allow any IP to connect to the nrpe server)
nrpe_additional_allowed_hosts: 10.0.0.50
```

then re-run the installation playbook as usual (e.g. `ansible-playbook -i inventory install.yml`). There's no need to invoke the role directly — it's already wired into `install.yml` behind the `nrpe_enabled` gate.

Once enabled, the role installs the `nrpe`/Nagios-plugin OS packages, installs `pymysql`, deploys the three plugin scripts to `/usr/lib/nagios/plugins/`, renders the NRPE command config, sets up the sudoers rule, configures SSL, starts the service, and runs a smoke test.&#x20;

**Verify** from the NRPE host itself: `check_nrpe -H 127.0.0.1 -c check_nodeum_services -a 1 1`.

### How this integrates through NRPE

#### Server-side (the monitored Nodeum server)

NRPE needs a command definition per check, telling it which script to run and how to map the `$ARGn$` placeholders the monitoring server will send. These live in a config file (template: templates/nodeum\_nrpe.cfg.j2), typically dropped into `/etc/nrpe.d/nodeum_nrpe.cfg`:

```
command[check_nodeum_services]=sudo /usr/lib/nagios/plugins/check_nodeum_services.py -c $ARG1$ -w $ARG2$ $ARG3$
command[check_nodeum_scsi]=sudo /usr/lib/nagios/plugins/check_nodeum_scsi.py $ARG1$
command[check_nodeum_tasks]=sudo /usr/lib/nagios/plugins/check_nodeum_tasks.py -w $ARG1$ -c $ARG2$ -s "$ARG3$" -e "$ARG4$" $ARG5$

# Required because these commands take arguments
dont_blame_nrpe=1

server_port=5666
server_address=0.0.0.0
allowed_hosts=127.0.0.1,::1,<your monitoring server IP(s)>

# Optional, recommended
ssl_cert_file=/etc/nodeum/certs/CAs/public.crt
ssl_privatekey_file=/etc/nodeum/certs/private.key
```

Key points:

* `allowed_hosts` is an IP allow-list — the monitoring server's address **must** be added here (via `nrpe_additional_allowed_hosts` in Ansible), or NRPE will refuse the connection.
* `dont_blame_nrpe=1` is required for any command using `$ARGn$`; without it, NRPE ignores arguments sent by the client for security reasons.
* SSL is on by default (TLS between the monitoring server's `check_nrpe` and the local `nrpe` daemon). If disabled (`nrpe_use_ssl: false`), both the daemon and the `check_nrpe` client must be called with `-n`.
* NRPE listens on TCP port `5666` — make sure it's open between the monitoring server and the Nodeum node (firewall/security group).

#### Client-side (the monitoring server: Nagios / Icinga)

On the monitoring server, define a `check_nrpe` command and one service check per plugin, passing the same arguments the local `command[...]` definitions expect, in order:

```
# check_nodeum_services: -c <critical threshold> -w <warning threshold>
check_nrpe -H <nodeum-srv> -c check_nodeum_services -a 1 1

# check_nodeum_scsi: no required arguments
check_nrpe -H <nodeum-srv> -c check_nodeum_scsi

# check_nodeum_tasks: -w <warn> -c <crit> -s <start> -e <end>
# (compute a rolling window, e.g. "last hour", on the monitoring side)
check_nrpe -H <nodeum-srv> -c check_nodeum_tasks \
  -a 5 10 "$(date -u -d '-1 hour' '+%Y-%m-%d %H:%M:%S')" "$(date -u '+%Y-%m-%d %H:%M:%S')"
```

Example Nagios `service` object definitions:

```
define service {
    host_name               nodeum-srv-01
    service_description     Nodeum Services
    check_command           check_nrpe!check_nodeum_services!1!1
    use                     generic-service
}

define service {
    host_name               nodeum-srv-01
    service_description     Nodeum Tape Drives
    check_command           check_nrpe!check_nodeum_scsi
    use                     generic-service
}

define service {
    host_name               nodeum-srv-01
    service_description     Nodeum Failed Tasks (last hour)
    check_command           check_nrpe!check_nodeum_tasks!5!10!"$LASTHOURSTART$"!"$LASTHOURENDTIME$"
    use                     generic-service
}
```

(`$LASTHOURSTART$`/`$LASTHOURENDTIME$` are placeholders — compute the actual rolling window with your monitoring system's macro/templating mechanism, or wrap the check in an event handler/script that fills in the current time range.)

If SSL is disabled on the node, add `-n` to every `check_nrpe` invocation above.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.nodeum.io/data-archiving/monitoring.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
