bjoerrrn / storj-system-health.sh Goto Github PK

Add delay of 10 days, before showing a new storj version.

Then, display message just once a day.

[sn2] : hdd 25.30%
.. new version : 1.74.1 > 1.73.4

Version comparison needs to be enriched. Is too stupid currently.

update example screenshots on GitHub

Add a config option which excludes single, already full nodes from "low upload rate" warning.

In case of pending audits: run script again automatically in order to verify that audits are working well.

Send a warning message only in the case, the "verification run" of the script still warns about pending audits (not anymore immediately).

• create config entries for all 3 score types
• alert only after the configured time frame
• in case zero (0) is configured, alert will only appear in -d mode

Add daily earnings status
Add monthly earnings status
Add daily upload data
Add daily download data

Fire just one error in case "other" and "severe" error log selections are equal in order to avoid double quoting and alerting for the same error messages.

A simple string compare should handle most of the cases.

• Low audit success rate (<95%)
• Low repair success rate (<95%). Risk of getting disqualified.
• Low customer download success rate (<90%). No disqualification risk.
• Low upload success rate (<90%). No disqualification risk.
• No upload or download activity for quite some time.
• Storagenode not running
• Storagenode unable to checkin including pingback error.
• ...

may this can be a help: https://forum.storj.io/t/guide-to-debug-my-storage-node-uplink-s3-gateway-satellite/1372

.. instead of using the delay configured in credo (10,800).

In order to speed up and allow a more accurate selection

/api/sno/

"diskSpace":{
      "used":1374306238338,
      "available":9500000000000,
      "trash":58233200156,
      "overused":0
   },

New log behavior of failed alerts is really annoying. Alerting needs to be further reduced.

.. and do not handle it as a general "ignore", but as a specific case

And consequently remove variable settings within the script

To let the operator know, which message belongs to which node.

Besides audit issues, also repeat error reports, if an error was found.

reuse satellite status from this script?
https://github.com/ReneSmeekes/storj_earnings

There needs to be a logic to not or much less send the same error log extract again and again.

Usually, older error messages should disappear after 24h, as the log selection is limited. Anyway, one e-mail for each new error found should be fine.

Now static 1 day - could be a time indicator and/or symbol to select the complete history

Is your feature request related to a problem? Please describe.
integrate ntfy.sh push message support

Describe the solution you'd like
https://docs.ntfy.sh/install/

Is your feature request related to a problem? Please describe.
a time lag of audit requests is not logged into the storage node log file.

issues on that will result more or less quick into disqualification without any warning to the storage node operator (SNO).

Describe the solution you'd like
analyse the log extract selected by minlog and check for long time lags between audit requests

example select statement:

cat /mnt/WD1003/logs/sn1.log | grep 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE | grep -E "GET_AUDIT" | jq -R '. | split("\t") | (.[4] | fromjson) as $body | {SatelliteID: $body."Satellite ID", ($body."Piece ID"): {(.[0]): .[3]}}' | jq -s 'reduce .[] as $item ({}; . * $item)'

example result from the command:

{
  "SatelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",
  "NG6KAUMU7TP22DNGROKBU2MRRNV675QYEOJC3X2BXH4OCML6BPNQ": {
    "2022-06-28T21:24:39.646Z": "download started",
    "2022-06-28T21:24:40.002Z": "downloaded"
  },
  "IADTQX62PCZQEJRRYPCKNWX3QSPG7A3U53IBWPQRSX6ZMH6I45UQ": {
    "2022-06-28T21:32:40.597Z": "download started",
    "2022-06-28T21:32:40.893Z": "downloaded",
    "2022-07-09T20:00:10.698Z": "download started",
    "2022-07-09T20:00:10.995Z": "downloaded"
  },
  "MZEPH4JSGSAJZ72QQV4YOYYVGLER7KOQPBUB2VEANL4MPNSZDBTA": {
    "2022-06-28T21:58:56.184Z": "download started",
    "2022-06-28T21:58:56.454Z": "downloaded"
  },
  "GFATHGO2WFBZNAOQJKXYNHTFKH2T5T4OXK3BEL7U62FNK5ZRR6OQ": {
    "2022-06-28T22:08:49.765Z": "download started",
    "2022-06-28T22:08:50.089Z": "downloaded"
  },
...

Additional context

• forum discussion
• issue created in storj project