mindflavor / prometheus_folder_size_exporter Goto Github PK

View Code? Open in Web Editor NEW

18.0 5.0 4.0 50 KB

A Rust only folder size exporter for Prometheus (Grafana)

License: MIT License

Rust 100.00%

rust prometheus-exporter grafana folder-size hyper async

prometheus_folder_size_exporter's Introduction

Prometheus Folder size Exporter

Intro

A Rust Prometheus exporter for folder size. This tool exports the folder size information (optionally including every subdir) in a format that Prometheus can understand.

Prerequisites

You need Rust to compile this code. Simply follow the instructions on Rust's website to install the toolchain. If you get wierd errors while compiling please try and update your Rust version first (I have developed it on rustc 1.53.0-nightly (f82664191 2021-03-21)).

Compilation

To compile the latest master version:

git clone https://github.com/MindFlavor/prometheus_folder_size_exporter.git
cd prometheus_folder_size_exporter
cargo install --path .

If you want the latest release you can simply use:

cargo install prometheus_folder_size_exporter

Usage

Start the binary with -h to get the complete syntax. The parameters are:

Parameter	Mandatory	Valid values	Default	Description
`-v`	no			Enable verbose mode.
`-p`	no	any valid port number	9974	Specify the serivce port. This is the port your Prometheus instance should point to.
`-i`	yes	a valid config json file	-	The configuration file. This json is detailed below (you can find an example here: example.json).
`-b`	no	Any number > 0	Off	Enables the async storage calculation. The value specifies how often (in seconds) the calculation will be done. If not specified, the values will be calculated synchronously at each HTTP Get.

Once started, the tool will listen on the specified port (or the default one, 9974, if not specified) and return a Prometheus valid response at the url /metrics. So to check if the tool is working properly simply browse the http://localhost:9974 (or whichever port you choose).

JSON configuration file

Name	Valid values	Description
`path`	Any valid path	The starting analysis path.
`explode_depth`	Any positive number or -1	This setting controls how deep the folder explosion will go. -1 means no limit. 0 means no explosion.
`sum_remaining_subfolders`	`true` or `false`	This setting specifies if the last exploded folder size should include the subfolders.

So, for example, to monitor a single folder set explode_depth = 0 and pick sum_remaining_subfolders based on is you want the total folder + subfolder size or not. To monitor a folder and its first level subfolders you specify explode_depth = 1.

Systemd service file

Now add the exporter to the Prometheus exporters as usual. I recommend to start it as a service. My systemd service file is like this one:

[Unit]
Description=Prometheus Folder size Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/prometheus_folder_size_exporter -i /etc/prometheus_folder_size_exporter.json -p 9974

[Install]
WantedBy=multi-user.target

prometheus_folder_size_exporter's People

Contributors

Stargazers

Watchers

Forkers

brandonmcclure devops-phani rubennl crom-ap

prometheus_folder_size_exporter's Issues

sparse files size issue

First, I'd like to thank you for the prometheus_folder_size_exporter. It's been a valuable tool for our monitoring needs.

I've noticed that the current implementation reports the apparent (or nominal) size of files. While this might be useful in some contexts, it can lead to misleading metrics when dealing with sparse files. As you might know, the apparent size of a sparse file can be much larger than the actual disk space it uses.

To make the exporter more accurate for environments where sparse files are common, would it be possible to add an option or feature to report the actual disk space used by files, rather than their apparent size?

This would provide a more accurate picture of disk usage, especially when monitoring systems or applications that make use of sparse files.

Thank you for considering this feature request.

rename folder_size

Hi,
Looking for metrics in other products I see metrics like go_memstats_heap_alloc_bytes.
Don't you think it worth renaming it to something like folder_size_bytes to reflect actual data size?
Also, it's often _total is used but I don't know any common habits \ rules about it.

Best,
Andrey.

Dont fail on permission denied

Hi There,

Is there a way to skip error on permission denied? at the moment if its encounters a file that it cannot read it exits

user: None })
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }', src/main.rs:133:64
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Install fail

Hi, I tried installation both with repo and compiling but every time I have this error

error[E0277]: the trait bound ServerOptions: From<([{integer}; 4], u16)>is not satisfied --> src/main.rs:143:16 | 143 | let addr = ([0, 0, 0, 0], bind).into(); | ^^^^^^^^^^^^^^^^^^^^ ---- required by a bound introduced by this call | | | the traitFrom<([{integer}; 4], u16)>is not implemented forServerOptions| = note: required for([{integer}; 4], u16)to implementInto`

error[E0277]: ServerOptions doesn't implement std::fmt::Display
--> src/main.rs:145:43
|
145 | log::info!("starting exporter on {}", addr);
| ^^^^ ServerOptions cannot be formatted with the default formatter
|
= help: the trait std::fmt::Display is not implemented for ServerOptions
= note: in format strings you may be able to use {:?} (or {:#?} for pretty-print) instead
= note: this error originates in the macro format_args which comes from the expansion of the macro log::info (in Nightly builds, run with -Z macro-backtrace for more info)

For more information about this error, try rustc --explain E0277.
error: could not compile prometheus_folder_size_exporter due to 2 previous errors
error: failed to compile prometheus_folder_size_exporter v0.5.0 (/home/XXXXXX/prometheus_folder_size_exporter), intermediate artifacts can be found at /home/XXXXXXX/prometheus_folder_size_exporter/target
`

On Debian with
cargo 1.67.0 (8ecd4f20a 2023-01-10)
rustc 1.67.0 (fc594f156 2023-01-24)

Gist for generate conf.json

https://gist.github.com/scatenag/43a066cdbddb4b55288316546baa094a

recursive doesn't work

Hi,
Thank you that you wrote this very useful tool but looks like recursive doesn't work or I understand it wrong.

In response, I have got only two folders which I specify but I was expecting to see all folders in my home directory.
Please, can you check what might be wrong?

[
  { "path": "/var/log/", "recursive": false },
  { "path": "/home/andrey/", "recursive": true }
]

prometheus_folder_size_exporter -i config.json
[2020-11-28T23:30:26Z INFO  prometheus_folder_size_exporter] using options: Options { folders_file: "config.json", verbose: false, folders_to_scan: FolderScanner { folders: [FolderToScan { path: "/var/log/", recursive: false, user: None }, FolderToScan { path: "/home/andrey/", recursive: true, user: None }] } }
[2020-11-28T23:30:26Z INFO  prometheus_folder_size_exporter] starting exporter on 0.0.0.0:9974

folder_size{path="/var/log/",recursive="false"} 7160401
folder_size{path="/home/andrey/",recursive="true"} 98957688368

Why don't you change a 'folder_size_bytes' to gauge type?

Directory size can be increased or decreased over time. It seems the gauge type is more suitable for directory size.

cargo install fails

Hello,

I discovered this project this morning an tried to get it running, but I'm hitting a wall.
I tried to install as explained in the readme.

Fact: I'm new to rust, be gentle 😊

First attempt

cargo install prometheus_folder_size_exporter.

luilegeant@laptop : ~  > cargo install prometheus_folder_size_exporter
    Updating crates.io index
  Installing prometheus_folder_size_exporter v0.4.1
   Compiling autocfg v1.0.1
   Compiling libc v0.2.91
   Compiling cfg-if v1.0.0
   Compiling log v0.4.14
   Compiling proc-macro2 v1.0.24
   Compiling unicode-xid v0.2.1
   Compiling syn v1.0.64
   Compiling pin-project-lite v0.2.6
   Compiling cc v1.0.67
   Compiling memchr v2.3.4
   Compiling fnv v1.0.7
   Compiling lazy_static v1.4.0
   Compiling slab v0.4.2
   Compiling once_cell v1.7.2
   Compiling untrusted v0.7.1
   Compiling futures-core v0.3.13
   Compiling spin v0.5.2
   Compiling proc-macro-hack v0.5.19
   Compiling itoa v0.4.7
   Compiling proc-macro-nested v0.1.7
   Compiling bytes v1.0.1
   Compiling cfg-if v0.1.10
   Compiling httparse v1.3.5
   Compiling futures-task v0.3.13
   Compiling pin-utils v0.1.0
   Compiling bytes v0.5.6
   Compiling base64 v0.13.0
   Compiling try-lock v0.2.3
   Compiling adler v1.0.2
   Compiling serde v1.0.125
   Compiling gimli v0.23.0
   Compiling bitflags v1.2.1
   Compiling regex-syntax v0.6.23
   Compiling tower-service v0.3.1
   Compiling ryu v1.0.5
   Compiling failure_derive v0.1.8
   Compiling httpdate v0.3.2
   Compiling quick-error v1.2.3
   Compiling pin-project-lite v0.1.12
   Compiling termcolor v1.1.2
   Compiling serde_json v1.0.64
   Compiling object v0.23.0
   Compiling unicode-width v0.1.8
   Compiling rustc-demangle v0.1.18
   Compiling futures-sink v0.3.13
   Compiling hashbrown v0.9.1
   Compiling openssl-probe v0.1.2
   Compiling ansi_term v0.11.0
   Compiling strsim v0.8.0
   Compiling serde_derive v1.0.125
   Compiling vec_map v0.8.2
   Compiling num-traits v0.2.14
   Compiling num-integer v0.1.44
   Compiling tokio v1.4.0
   Compiling miniz_oxide v0.4.4
   Compiling num-bigint v0.2.6
   Compiling num-iter v0.1.42
   Compiling num-rational v0.2.4
   Compiling indexmap v1.6.2
   Compiling num-complex v0.2.4
   Compiling tracing-core v0.1.17
   Compiling futures-channel v0.3.13
   Compiling http v0.2.3
   Compiling ring v0.16.20
   Compiling humantime v1.3.0
   Compiling addr2line v0.14.1
   Compiling textwrap v0.11.0
   Compiling http-body v0.4.1
   Compiling http-body v0.3.1
   Compiling tracing v0.1.25
   Compiling want v0.3.0
   Compiling net2 v0.2.37
   Compiling iovec v0.1.4
   Compiling mio v0.7.11
   Compiling signal-hook-registry v1.3.0
   Compiling socket2 v0.3.19
   Compiling num_cpus v1.13.0
   Compiling atty v0.2.14
   Compiling quote v1.0.9
   Compiling aho-corasick v0.7.15
   Compiling mio v0.6.23
   Compiling clap v2.33.3
   Compiling backtrace v0.3.56
   Compiling regex v1.4.5
   Compiling mio-uds v0.6.8
   Compiling sct v0.6.0
   Compiling webpki v0.21.4
   Compiling synstructure v0.12.4
   Compiling env_logger v0.7.1
   Compiling env_logger v0.6.2
   Compiling pin-project-internal v1.0.5
   Compiling futures-macro v0.3.13
   Compiling tokio-macros v0.2.6
   Compiling ct-logs v0.8.0
   Compiling rustls v0.19.0
   Compiling num v0.2.1
   Compiling pin-project v1.0.5
   Compiling futures-util v0.3.13
   Compiling tokio v0.2.25
   Compiling rustls-native-certs v0.5.0
   Compiling tokio-rustls v0.22.0
   Compiling failure v0.1.8
   Compiling tracing-futures v0.2.5
   Compiling hyper v0.14.4
   Compiling tokio-util v0.3.1
   Compiling h2 v0.2.7
   Compiling hyper-rustls v0.22.1
   Compiling prometheus_exporter_base v1.1.2
   Compiling hyper v0.13.10
   Compiling prometheus_folder_size_exporter v0.4.1
error[E0308]: mismatched types
   --> /home/luilegeant/.cargo/registry/src/github.com-1ecc6299db9ec823/prometheus_folder_size_exporter-0.4.1/src/main.rs:148:34
    |
148 |         Box::pin(perform_request(request, state))
    |                                  ^^^^^^^ expected struct `Body`, found struct `hyper::body::body::Body`
    |
    = note: expected struct `Request<Body>`
               found struct `Request<hyper::body::body::Body>`
    = note: perhaps two different versions of crate `hyper` are being used?

error: aborting due to previous error

For more information about this error, try `rustc --explain E0308`.
error: failed to compile `prometheus_folder_size_exporter v0.4.1`, intermediate artifacts can be found at `/tmp/cargo-installNVEaiF`

Caused by:
  could not compile `prometheus_folder_size_exporter`

To learn more, run the command again with --verbose.

Second attempt: via git clone

If i do a git clone, cd in folder and cargo install --path . I get 2 errors:

Compiling prometheus_folder_size_exporter v0.4.1 (/home/luilegeant/Documents/repositories/test/prometheus_folder_size_exporter)
error[E0308]: mismatched types
   --> src/main.rs:148:34
    |
148 |         Box::pin(perform_request(request, state))
    |                                  ^^^^^^^ expected struct `Body`, found struct `hyper::body::body::Body`
    |
    = note: expected struct `Request<Body>`
               found struct `Request<hyper::body::body::Body>`
    = note: perhaps two different versions of crate `hyper` are being used?

error[E0271]: type mismatch resolving `<Pin<Box<impl std::future::Future>> as std::future::Future>::Output == std::result::Result<std::string::String, Box<(dyn StdError + Send + Sync + 'static)>>`
   --> src/main.rs:147:5
    |
147 |     prometheus_exporter_base::render_prometheus(addr, state, |request, state| {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `Box`, found struct `failure::Error`
    | 
   ::: /home/luilegeant/.cargo/registry/src/github.com-1ecc6299db9ec823/prometheus_exporter_base-1.2.0/src/lib.rs:200:17
    |
200 |     Fut: Future<Output = Result<String, Box<dyn Error + Send + Sync>>> + Send + 'static,
    |                 ----------------------------------------------------- required by this bound in `render_prometheus`
    |
    = note: expected enum `std::result::Result<_, Box<(dyn StdError + Send + Sync + 'static)>>`
               found enum `std::result::Result<_, failure::Error>`

error: aborting due to 2 previous errors

Some errors have detailed explanations: E0271, E0308.
For more information about an error, try `rustc --explain E0271`.
error: failed to compile `prometheus_folder_size_exporter v0.4.1 (/home/luilegeant/Documents/repositories/test/prometheus_folder_size_exporter)`, intermediate artifacts can be found at `/home/luilegeant/Documents/repositories/test/prometheus_folder_size_exporter/target`

Caused by:
  could not compile `prometheus_folder_size_exporter`

To learn more, run the command again with --verbose.

Details

cargo --version: cargo 1.50.0 (f04e7fab7 2021-02-04)
cargo install prometheus_folder_size_exporter => was done on 2021-03-23
cargo install --path . (after git clone) => was done on commit id: 4740050

Let me know if I shall provide more details.

limit recursive depth for Explode mode

Hope I'm not asking too much :)
As an example, I want to monitor /home/ folder itself and all subfolders there but not more.
For that case, depth would be 1.

I hope it will let me monitor / folder and not be that precise with settings.