rust-cli / confy Goto Github PK

I use confy to store float values in toml. It was working fine until recently (a month ago?) instead of storing something like
value = 0.0
it is stored like
value = --0.0
and of course after restart confy returns an error on this value.
I understand that this is probably a bug in the toml crate (?) which has likely got a minor update which breaks confy, but I am not sure how to report the issue there.

confy depends on a quite old version of directories.

In Cargo.toml the directories version is specified as

 directories = "^2.0"

This is equivalent to "2.0" which means >=2.0.0, <3.0.0

Please change it to "4.0"

confy::load("name") crashes withBadConfigDirectoryStr

Version 0.4.0

Confy 0.3.1

Steps:

1. remove the closing single quote from a value in your toml file, so that it becomes invalid

key: 'value
otherkey: 'value'

2. run your program, which uses this code:

let cfg: Config = match confy::load("appname") {
        Ok(_) => {
            println!("Done!");
            std::process::exit(exitcode::OK);
        }
        Err(e) => {
            eprintln!("Error: {}", e);
            std::process::exit(exitcode::CONFIG);
        }
    }

Expected:

The program exits cleanly with exitcode::CONFIG error code, after printing Error: etc.

Actual Result:

`thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { inner: ErrorInner { kind: NewlineInString, line: Some(0), col: 14, message: "", key: [] } }', libcore/result.rs:1009:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::_print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::continue_panic_fmt
   6: rust_begin_unwind
   7: core::panicking::panic_fmt
   8: core::result::unwrap_failed
   9: confy::load
  10: ren::main
  11: std::rt::lang_start::{{closure}}
  12: std::panicking::try::do_call
  13: __rust_maybe_catch_panic
  14: std::rt::lang_start_internal
  15: main
  16: __libc_start_main
  17: _start

For a crate like this, I think it's super important to lay out goals and non-goals. Let's figure out what they are!

Some sources of inspiration for goals and non-goals:

• configure which has a configuration source trait
• viper (go): wide ranging, covers env vars and command line args, etcd and consul, and others

Just spotted in the documentation for the store functionality:

let my_cfg = MyConf {};
confy::store(my_cfg)?;

I reckon should be:

let my_cfg = MyConf {};
confy::store("my-app-name", my_cfg)?;

Just a small detail, but certainly worth clearing up!

Should it be possible to have a global config that is overwritten by a local/ user config, etc. And how would be the best way to implement this?

Suggested by #2 (comment)

Confy seems to work fine as long as you're adding information to the config file, but as soon as you remove anything from the config, confy breaks. For example, consider the following code:

extern crate confy;
#[macro_use] extern crate serde_derive;
use std::default::{Default};

#[derive(Serialize, Deserialize)]
struct Config {
    lines: Vec<i32>
}

impl Default for Config {
    fn default() -> Self {
        Self {
            lines: vec![1, 2, 3]
        }
    }
}

fn main() -> Result<(), std::io::Error> {
    let mut config: Config = confy::load("confy-bug")?;
    confy::store("confy-bug", config)
}

If you run the above code you will produce the following output as expected:

lines = [
    1,
    2,
    3,
]

however if you add the following:

 fn main() -> Result<(), std::io::Error> {
     let mut config: Config = confy::load("confy-bug")?;
+    config.lines.clear();
     confy::store("confy-bug", config)
 }

You will instead end up with this syntactically incorrect file:

lines = []
   1,
    2,
    3,
]

It seems that confy is just writing on top of the old file without cleaning up any excess lines left over from the last write.

I was able to fix the issue locally by making the following changes:

diff --git a/src/lib.rs b/src/lib.rs
index f468e89..cb3e03b 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -143,7 +143,7 @@ pub fn store<T: Serialize>(name: &str, cfg: T) -> Result<(), IoError> {
     ].iter()
         .collect();

-    let mut f = OpenOptions::new().write(true).create(true).open(path)?;
+    let mut f = OpenOptions::new().write(true).create(true).truncate(true).open(path)?;
     let s = toml::to_string_pretty(&cfg).unwrap();
     f.write_all(s.as_bytes())?;
     Ok(())

It feels weird that I should need to implement a feature with my CLI application to display the path to the configuration file, when all I need to know is the default location for that file to delete it locally for testing. I get it, help oneself, but you could save a lot of people a lot of searching / unnecessary code by just making a small table in the readme for the main OSes supported in Rust (windows, macos, linux, wasm?).

Note this isn't a judgement on the quality of this project, so please realize I just want it to be more accessible to folks newer to using Rust for CLI development.

Comments can make config files more user-friendly by giving hints about possible values and where more info can be found.
When config files get large, sections are useful to break it up so that things are easier to find.

Thanks for making this, confy made my life much much easier today.

Result is None when test.toml is blank. I wish to get as below, how can I do it?

Some(
    Test {
        sub: Some(
            [
                Data {
                    id: 2,
                    name: "3",
                },
                Data {
                    id: 4,
                    name: "5",
                },
            ],
        ),
    },
)

my code is here:

testconfig.rs:

use serde::{Deserialize, Serialize};

#[derive(Debug, Serialize, Deserialize)]
pub struct Data {
    pub id: usize,
    pub name: String,
}
impl Default for Data {
    fn default() -> Self {
        Self {
            id: 1,
            name: "test1".to_string(),
        }
    }
}
#[derive(Debug, Serialize, Deserialize)]
pub struct Test {
    #[allow(dead_code)]
    pub sub: Option<Vec<Data>>,
}
impl Default for Test {
    fn default() -> Self {
        Self { sub: Some(vec![Data{id:2,name: "3".to_string()}, Data{id:4, name: "5".to_string()}]) }
    }
}
#[derive(Debug, Serialize, Deserialize)]
pub struct TestGlobalConfig {
    #[allow(dead_code)]
    pub test: Option<Test>,
}
impl Default for TestGlobalConfig {
    fn default() -> Self {
        Self { test: Some(Test::default()) }
    }
}

main.rs:

use std::sync::RwLock;
pub mod testconfig;
use crate::testconfig::TestGlobalConfig;
lazy_static::lazy_static! {
    static ref GLOBAL: RwLock<TestGlobalConfig> ={
        let path = concat!(env!("CARGO_MANIFEST_DIR"),"/test.toml");
        if let Ok(global_config) = confy::load_path::<TestGlobalConfig>(path) { 
            RwLock::new(global_config)
        } else {
            panic!("load config file {path:?} error!");
        }
    };
}
fn main() {
    let binding = GLOBAL.read().unwrap();
    let settings = &binding.test;
    println!("{:#?}", settings);
}

I see a lot of Rust Crates being stuck in the 0.X release cycle.

What is missing before it can be declared a version 1.0.0?

Do you need community help to complete it?

Just a question.

Hello,
The repository contains one LICENSE file but the Cargo.toml file lists both MIT/X11 and Apache-2.0
Could you please clarify the licensing?

pub fn load_path_or_else<T, F>(
    path: impl AsRef<Path>,
    op: F
) -> Result<T, ConfyError>
where
    T: DeserializeOwned + Serialize,
    F: FnOnce() -> T,
{ }

Add the function so that user can pass a closure, which could return a custom value if the file doesn't exist or is empty.

The directories crate is maintained again while the crate that was supposed to be the continuation of directories is now unmaintained.

Since 0.4.0 it seems an old error has come back whereby if a configuration file is missing a field it gives an Error during load_config.

We should be on the latest stable Rust edition, currently we are on 2018 when 2021 is available.

First of all, I love the convenience that this package provides but I ran into a problem that might have a simple solution.
I want to have multiple config files per project. For example, I would like to store/load the following files on Linux:

~/.config/app-name/config1.toml
~/.config/app-name/config2.toml
~/.config/app-name/config3.toml

With confy's current functionality I can only generate a ~/.config/app-name/app-name.toml file but I'd like to have multiple configs per project.

I've got a quick solution working for me on a fork but it would be nice if this kind of functionality could be in the official package.

confy requires serde for serialization and deserialization and the documentaion is not clear about this. Also confy will fail in file is missing my_app so Default trait was also added.

Add confy and serde to your cargo.toml

confy = "0.4.0"
serde = { version = "1.0", features = ["derive"] }

Add Default trait to generate a file with default values if missing.

use serde::{Serialize, Deserialize};

#[derive(Default, Debug, Serialize, Deserialize)]
struct MyConfig {
    name: String,
    comfy: bool,
    foo: i64,
}

fn main() -> Result<(), io::Error> {
    let cfg: MyConfig = confy::load("my_app")?;
    println!("{:#?}", cfg);
    Ok(())
}

Hi! Let's release 0.5.0!

@nyanpasu64 mentioned #48 that we maybe should run CI also on other images than ubuntu as well.

What'd you think?

Are there any opinions on adding JSON as a serialization format? My project uses JSON because it requires interop between multiple components, and JSON's tooling is very strong and ubiquitous. There's also value in how restrictive the data model is. While I want to use confy, I can't justify adding another serialization format (and all the tooling requirements and conversion ambiguity that come with it) to my project.

I see that YAML is already a config option. Is there desire for adding one for JSON? If so, I would be happy to take it on and make a PR following the same general structure as the YAML feature. I understand there are arguments against JSON as a configuration format however (lack of comments for example) and don't want to jump in adding it if the maintainers have no desire for the feature.

Is there any plan to make a release? Things like #8 are fixed but not on crates.io yet, which IMHO makes this great library unusable.

The unwrap call unwrapping toml::from_str's output in line 105 of /src/lib.rs (currently in master 7115b65 and 0.3.1 95b1aac) does that:

Ok(mut cfg) => Ok(toml::from_str(&cfg.get_string().unwrap()).unwrap()),

Will it be reasonable to create an enum ConfyError { BadTomlData } so I could return it in the load function at the point of toml::from_str's failure?

It would change load's type signature into a more general one or a totally different one if we return both IoErrors and ConfyError as Error trait values, I am not sure, so it might not be a good idea.

User should know which should file should they edit to change the program behaviour

Hello,

I've recently come across this crate and am quite a fan! With that being said, during my use of the crate, I discovered that I couldn't use the ? operator in the context of a function returning Result<(), Box<dyn std::error::Error>>. This seems to have become a somewhat idiomatic way of dealing with errors in Rust.

The failure crate, while certainly having its upsides, has had its use in libraries discouraged (due to a lack of stabilization) and the standard library has been gradually incorporating fixes that the crate was intended to provide.

Finally, failure pulls in backtrace as a dependency, which breaks cross compilation from Linux -> macOS, which is what has brought me here. I'd suggest replacing the failure crate with the std::error implementation and would be more than happy to make a pull request.

Let me know what you think!

Hello

Unfortunately confy::load(...) fails with SerializeTomlError(ValueAfterTable) error if there is something defined after a HashMap in the configuration struct. For example, my config struct is like this does not work:

#[derive(Debug, Serialize, Deserialize)]
struct Config {
    title:   String,
    api_key: String,
    updated: String,
    centers: std::collections::HashMap<String,String>,
    tasks: Vec<String>,
}

I got it working by swapping the order of centers HashMap and tasks Vector, but I'd rather keep then in the order... Is there any way to make this happen, or is this problem due to how TOML works?

add a new pub fn and remove config

Hello, I'm using this crate in one project. The latest version of the crate depends on directories 4.0, but version 5.0.1 is the current version. Any thoughts on updating this dependency and other dependencies and publishing a new version of confy? I can send a PR with the dependency update and handle any API changes that might happen.

The directories crate has a breaking change when upgrading to version 3 (specifically macOS change). https://github.com/dirs-dev/directories-rs#3
I did not check more, but I will before making a PR.

The general idea behind this feature is the following. Take a configuration file:

my_option = "foo"

and a CLI which also wants to provide the option my_option which, if provided overrides the config state and if isn't, just uses the default provided by the configuration.

Open Questions

• A similar problem was already solved by config (and others), can we just use their work/ be compatible with them?
• It should also be to use this feature with clap.rs directly, structopt or thunder so a more generic approach is generally good

Not sure if I'm just being stupid, but it looks like when you add config values after the original toml file has been created it errors with Bad TOML data. I would expect it to just use the specified default trait and use that instead.

I want to start using confy in my project, but I can't figure out how to include a manually editable configuration file?
From my understanding of the practical example in the readme, it is loading a .toml file named "app-name" out as the values. I try to include a app-name.toml in the root of my project, but it does not work.
Is it possible to do this?

Here's the output I'm getting:
thread '<unnamed>' panicked at 'Config read error: BadTomlData(Error { inner: ErrorInner { kind: Custom, line: Some(0), col: 0, at: Some(0), message: "missing field secret_key", key: [] } })', src/utils/config.rs:25:36

From this code:

#[derive(serde::Serialize, serde::Deserialize)]
pub struct Config {
  pub secret_key: String,
  pub test_attr: u8,
}

impl ::std::default::Default for Config {
  fn default() -> Self {
    Self {
      test_attr: 0,
      secret_key: "secret".into(),
    }
  }
}

pub fn load_config() -> Config {
  confy::load("app-name").expect("Config read error")
}

In my project root folder app-name.toml alongside cargo.toml:

secret_key = "something"
test_attr = 123

Is this a feature request? I dont know.

(The Discord chat room doesn't seem to load, so I'm asking here)

In the design philosophy of confy, is the developer allowed (or encouraged) to edit the config file manually?

I'm asking because based on the documentation, the creation of a new config file is done by:

1. Implement the Default trait on the config struct
2. Run confy::load()

Editing the config is done by confy::store(). So the creation and update of the config are done through Rust code.

So if I understand correctly, this crate is used to persist configurations between different executions. But not as a config file solution similar to dotenv or rc files (e.g. .bashrc, .vimrc), where people write a configuration file manually so they can override the program's behavior without re-compiling the code?

If the user is encouraged to directly edit the config file instead of using the confy::store() function, then I believe we need to put more thoughts into how a first-time user can create a new config file. Maybe the developer needs to implement a separate "first-time configuration wizard" CLI that help the user choose the configuration? It would be great to clarify the common use cases and patterns in the documentation besides the two core APIs.

Thank you for this little, but very useful crate.

#12 fixes a blocker: confy should never panic confronted to erroneous configuration files.

Unfortunately version 0.3.1, the last published on crates.io, comes without bug-fix and panics!
As a result my crate getreu/tp-note: fast note-taking with templates and filename synchronization also panics. This is a blocker for Windows user!

Please release immediately when a bug-fix is available and tested. I tested your last commit 5a58388 and it works well.

This is more a question/suggestion than an issue/bug report, since I cannot seem to find an explanation on this behaviour.

Assuming there is an AppConfig struct in place that contains of a handful values and there is a default implementation for those values in place as well. Then this config is being loaded on the very first launch and presumably stored on exit.

When the AppConfig struct changes, however, and for example a new value was introduced, it looks like confy simply refuses to load the configuration altogether, even though a default for that new value was specified:

Error: BadTomlData(Error { inner: ErrorInner { kind: Custom, line: Some(0), col: 0, at: Some(0), message: "missing field `api_endpoint`", key: [] } })

Would it be possible/make sense to "merge" the existing config and the defaults for value that does not exist in the .toml yet, in order to allow loading the configuration?

I'm using confy 0.3.1, and it seems that I am unable to remove items from a vec in my config (adding items works fine). But when I was trying to build a simple example to demonstrate the issue, I found a very confusing one.

Here's my code:

use serde::{Deserialize, Serialize};

#[derive(Debug, Serialize, Deserialize)]
struct MyItem {
    name: String,
}

#[derive(Debug, Serialize, Deserialize)]
struct MyConfig {
    items: Vec<MyItem>,
}

/// `MyConfig` implements `Default`
impl ::std::default::Default for MyConfig {
    fn default() -> Self {
        Self {
            items: vec![MyItem { name: "foo".into() }, MyItem { name: "bar".into() }],
        }
    }
}

fn main() -> Result<(), ::std::io::Error> {
    let mut cfg: MyConfig = confy::load("foobar")?;
    println!("{:?}", cfg);
    cfg.items.remove(0);
    println!("{:?}", cfg);
    confy::store("foobar", cfg)?;
    Ok(())
}

This is what I get in the config file after running it (any number of times):

[[items]]
name = 'bar'

[[items]]
name = 'bar'

I expected one item after the first run and zero after subsequent runs. And the default vec has two different items (foo and bar) whereas the toml file have the same item twice.

Hi,

https://docs.rs/confy/0.3.1/src/confy/lib.rs.html#27 points to the non-existing https://docs.rs/crates/serde, but should link to the correct page (https://docs.rs/serde) since commit 40d9dfdf266fda310a52bf6c57798f0d71ae742e. I'm not sure what's wrong here, deployment wrong or something. I would have opened a PR but I am not sure there's anything to fix in the docs themselves.

Best,
Lexi

In my properties i have the following enum for a datetime. Which uses camelcase

pub enum DateTimeFormat {
    Rfc3339,
    Rfc2822,
    Epochms
}

My application toml file sets the property as such

datetime_format = 'Rfc3339'

If I use non cap

datetime_format = 'rfc3339'

it fails at

confy::load_path(inpath)

Is there an acceptable way of rectifying this in a deserializer

Please release more often. Even a few smaller improvements can make a big difference, according to what the user needs.

For example: I am desperately waiting for
pub fn get_configuration_file_path(name: &str) -> Result<PathBuf, ConfyError>.

The commit 83c3c22 could be easily shipped as intermediate version.

I've noticed that when confy tries to write to a file path, it overwrites the file in-place:

In my experience (though not with apps using confy), overwriting files in-place causes problems. One possible issue (for files shared between many apps) is that either other applications will be able to read half-written contents, or the file will be locked and unreadable for the duration of the write. A more serious integrity issue is that if the application terminates during the file write (either from panicking, or from being terminated in taskmgr or Ctrl-C'd), it will result in a corrupted file being written to disk, with both the old and new file data being lost.

As an example of what can go wrong, a Python app I wrote (corrscope) saves config files directly to disk (since I didn't know better at the time), overwriting the previous file. Every few months I get a report from a user who can't open the app because the config file is corrupt and fails to load.

What causes saving to fail?

Confy's store_path() first opens the file, then calls either toml::to_string_pretty or serde_yaml::to_string (both of which are fallible and return Result<...>), and use the ? operator on the Result. If the function called fails, then store_path() exits, but the file on the disk has already been truncated and the damage is done. (I'm not sure if TOML or YAML serialization is likely to fail, either reliably or on edge cases, in unfinished programs on developer machines or in released applications on user machines.)

In corrscope, I suspect a similar event can occur if the serializer is left in a corrupted state. (I use a persistent global YAML serializer object for both documents and settings, since I hadn't learned Rust at the time, which would've taught me to avoid shared mutability.)

On top of this, there's the risk of getting terminated or Ctrl-C'd during the serialization operation (or if the serialization hangs and the app needs to be killed). I'm not sure if a system crash after the app finishes writing can lead to corruption, in various filesystems and journaling modes.

Solutions

A common strategy is to write to a temporary file and rename it on top of the original. While imperfect (may not always preserve permissions or owners), it's the least bad strategy I'm aware of for saving generic textual config files. (An alternative, advocated by danluu, is to not rename on top of the original file, but use an external log file for crash recovery. However, this introduces complexity in the app, and is unworkable if other apps expect plaintext config files and don't understand how to recover from log files.)

Additionally if you want to recover from system crashes as well, you need to fsync the temporary file before renaming it to the original filename. If you don't do so, it's possible that the temporary file gets renamed and overwrites the original file (and the rename gets flushed to disk), but the system crashes before the temporary file's contents are written to disk.

Yet another level of protection is fsyncing the directory to ensure that if the system crashes after renaming the temporary file over the old file, the rename can't be reverted. However this is slower than not fsyncing the directory, is not necessary to prevent corruption on Linux, and is not possible on Windows.

https://github.com/untitaker/rust-atomicwrites is a crate that performs atomic file writing. However it performs a directory fsync on Linux, which I think is unnecessary, slow, and inconsistent between platforms. I would rewrite or vendor the crate without performing a directory fsync.

Is there any chance to create a load_or_die feature such that application don't start if not able to find configuration file? I'd be happy to implement this given the opportunity.

This would simplify the main API and allow more configuration options without blowing up the main code body.

Suggested by #2 (review)