In the first meeting, we discussed how to define what a CLI, so we can say what kind of projects we want to focus on.

@TeXitoi proposed this definition:

A program that is launch in a terminal with parameter, no user interaction during the execution, and that do work on files, network and/or output to terminal

Which also means: not an ncurses TUI, and not a daemon

(To settle this issue, write a PR that adds a definition to the Readme, and have it merged)

Next meeting is on Friday, march 9, 2018, at 5pm UTC via https://gitter.im/rust-lang/WG-CLI.

You can find (and add to) a preliminary agenda on https://public.etherpad-mozilla.org/p/rust-cli-wg.

# Rust CLI working group

## Meta

- Chat: https://gitter.im/rust-lang/WG-CLI
- Coordination repo: https://github.com/rust-lang-nursery/cli-wg
- Next meeting: Friday 2018-03-09 17:00 UTC
- Nickname: Rust CLIQuE (Rust CLI Quality Enhancement)
- Survey URL: https://goo.gl/forms/lqUTazC78j26ISu53

Legend: `->` = action item

## First newsletter issue: Looking for content!

- Survey open until March 25: https://goo.gl/forms/lqUTazC78j26ISu53

## Agenda next Meeting

- What are we doing?
	- https://github.com/rust-lang-nursery/cli-wg/issues?q=is%3Aissue+is%3Aopen+label%3A%22tracking+issue%22
- Other stuff if we have time (add yours here!)

## Agenda Meeting March 9

- What are we doing?
	- https://github.com/rust-lang-nursery/cli-wg/issues?q=is%3Aissue+is%3Aopen+label%3A%22tracking+issue%22
	- Find next actionable thing for each issue
		- Next meeting in two weeks -- what can we achieve until then?
		- Maybe assign people for each ticket? or open new tickets for short term goals
		-> killercup: plan assert_cli story with edpage
		-> killercup'll create an issue for config-file configure adaptor
- Survey check-in
	- Anything we have already learned from the survey?
- Nominate some people who've been active so far to "CLI WG core team"?
	- create github team and assign people to issues
- Other stuff if we have time (add yours here!)
	- rust-clique github orga to create a friendly place for cli libraries to go to

### Team members

- eijebong (diesel, servo and randomly bumping dependencies all around the place...)
- epage (Ed Page): cobalt, crate-ci
- DPC (Dylan DPC): co-maintainer of uuid-rs & github-rs
- kbknapp (Kevin Knapp): clap, clog, cargo-{outdated, count, graph}8
- killercup (Pascal Hertleif): WG lead, quicli, assert_cli, diesel, cargo-edit
- TeXitoi (Guillaume P.):  structopt, benchmarksgame in rust, osmpbfreader, mdo, mimirsbrunn (a geocoder in rust based on elasticsearch), a bit of rust-geo and rs-es
- vitiral (Garrett Berg): artifact, ergo, path_abs, taken
- yoshuawuyts (Yoshua Wuyts)
- sgrif (Sean Griffin): diesel

## Slogan ideas

Writing CLI apps in Rust is a frictionless experience with known best practices that work across platforms, good documentation, and effortless distribution

## Definition of CLI

A program that is launch in a terminal with parameter, no user interaction during the execution, and that do work on files, network and/or output to terminal

ie not an ncurses TUI
ie not a daemon (@kbknapp: I would argue that I daemon could very well fall into the CLI realm of what's applicable here)

## Goals

Areas
- Fun / low friction
- Cross-platform
- Native to each platform
- End-user consumable (distribution)
- Releasing with confidence (testing, etc)
- Ease of use

- Rust is known to be a very good language for writing CLI tools
	- Writing and distributing a CLI in rust should be more fun than doing so in python.
- "Writing a CLI tool" is a good way to get started with Rust as a beginner
- Make it easy to ship well-tested binaries
- "Writing seamless binaries in Rust should be effortless"
- Rust CLI apps work effectively the same on all platforms
- Rust CLI apps should be easy to build, test and package

### How to get there

- Identify important use-cases and get crates ready
	-> Small survey?
- Blog posts, tutorials, guides, etc. on how to write CLI apps in Rust
	- Cookbook? https://rust-lang-nursery.github.io/rust-cookbook/
	- quicli docs are written in a cookbook/"let's get shit done" way https://killercup.github.io/quicli/
	- "guides as tests" philosophy
	- CI book started (will include binary distribution): https://crate-ci.github.io/index.html
	- Organizing bin crates (when to mix bin/lib, when to keep separate, managing distinct dependencies)
- Testing / Stability
	- Stability this year is a non goal. This year is about experimentation.
	- Working on stability for next year or the year after is a goal.
	- Do something like "crater" for the CLI ecosystem -- make it easy to have your CLI auto tested with the latest "CLI libraries"
- Create "conglomeration" crates that exposes a "unified" API.
	- This is what ergo is attempting: https://github.com/rust-crates/ergo
- Presence on the new Rust site
	- Begin describing "ideal workflows"

Crate-wise, I think the thing that could be valuable is to expand on an idea I've seen floated from time to time, creating higher level abstractions over the existing libs. For example, std::path is pretty low level when you compare it to pathlib in Python. If we provide some higher level libs (at the cost of performance or lack of platform neutrality) that'd be a big help for people. They could then progress to the lower level stuff in inner loops 

## Aspects of writing a CLI tool

- cargo templates?
  -> https://internals.rust-lang.org/t/pre-rfc-cargo-templates/5056
- CLI args handling
	- Really solid.
	- Next step: Ship clap 3 with structopt-style derive built-in.
		-> help out! https://github.com/kbknapp/clap-rs/issues/1037 and https://github.com/kbknapp/clap-rs/milestone/73
- Colored output
	- cf. https://internals.rust-lang.org/t/terminal-platform-abstraction/6746/5
	- some crates: termstyle, colored, crossterm, term color, ansi_term
- stabilize `?` in main (https://github.com/rust-lang/rust/issues/43301)
	-> help out implementing and testing this issue
	- have a good story/guidelines around which exit codes to return
- Streamline logging story
	- figure out log vs slog debate
- Error handling. Not all errors are created equal. For example, I/O pipe errors.
	- Yes! Providing some official guides on common footguns would go a long way (closed pipes, etc.)
	- Also ^C handling: some apps will want to die right away, others will want to delay and maybe disconnect remote links nicely, others will want to do cleanup only locally
- CI for providing binaries; general guidance on installation options
	- cf. https://github.com/japaric/trust/
	- https://github.com/crate-ci/crate-ci.github.io by @epage
	- cargo-install doesn't respect Cargo.lock
		-> help fix https://github.com/rust-lang/cargo/issues/2263 (already merged )
- CLI integration tests
	-> assert_cli 1.0 (@killercup and @epage)
	-> dir-diff 1.0
	- Best practices, i.e. testing application code, and not duplicating tests from deps
- Generate documentation
	- rustbook only covers Rust API, not CLI API
	- documentation  of env vars a CLI app repsects/uses
		- auto-generate from something like ?
		- how to incorporate external crates, like env_logger?
	- there seems to be no cohesive answer for man pages yet
		- ripgrep is using aciidoc: https://github.com/BurntSushi/ripgrep/pull/776
		-> issue in clap https://github.com/kbknapp/clap-rs/issues/552 (@kbknapp, awaiting clap v3-alpha)
	- flatdoc or mdbook templates/best practices?
		http://ricostacruz.com/flatdoc/
		https://rust-lang-nursery.github.io/mdBook/
- Bottom-line status output, using \r. Some CLI tools will want to show progress, e.g. whilst waiting for slow remote resources (apt-get, etc)
- Readline-like library for friendly input handling (not just basic cooked stdin tty handling)

### Gotchas

- test harness
  - how to set up clean filesystem/etc environments for testing (containers? chroot jails?
    - tempfile helps
    - dir-diff helps with validation
    - Mocking
    
- error handling (`failure` is looking good but does not yet have clear messaging/documentation about how to architect/structure error handling in your app)

- file path handling
  - `..`, .`.`, `//` not being auto- handled
  - windows gotchas, like long file paths (which require extended-length paths aka "verbatim paths" as they're called in std)
  - absolute paths on Windows (UNC prefices)
    - https://github.com/notion-cli/notion/blob/master/crates/verbatim/src/lib.rs
  - UTF-8 filenames on MacOS (the filesystem normalizes UTF-8 strings)
  
- how to package/deploy/distribute

- serialization/deserialization idioms
  - learning how different formats (JSON, TOML, etc) interact with the serde data model
  - best practices and idioms, e.g. "intermediate representation" types that map more closely to the serialization format
- Where to read config files from
  - [XDG](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html)?)
  - app_dirs crate
    - Is it being maintained?
    - What can be done to help with this?
- Config best practices
  - config crate (handle merging env with file)
  
- streaming
  - how to thread progress computation through chained readers
  - handling different archive formats (this may be specific to me but I bet others will hit it)
    - how to stream zip files (no crates.io crate supports this AFAIK)
    - how to compute progress percentage of decompressing files

### Packaging (Not cargo-install)

- Guides/How-Tos
	- Deb, RPM, AUR, Homebrew, Snap, Flatpak, AppImage, nixpkgs
- Automation 
	- cargo subcommands
	- Using CI
	
If cargo-install is an important use case for dev tools, how much should we look at improving it?

From the survey, string ergonomics are huge. Many, many CLI applications deal heavily with strings. In Rust, strings can be...difficult.

Granted, (IMO) Rust handles them correctly, sometimes the correctness doesn't actually matter for a given problem domain and just adds unnecessary gyration.

For example, we have:

• (&)('static)str
• Cow
• (&)String
• (&)OsStr
• (&)OsString

It's understanding it can be overwhelming. My personal opinion is we should first tackle/discuss the ergonomics of using OsStr(ing) as it's heavily used on Linux (where paths may not contain valid UTF-8).

IMO OsStr should have the same user experience as &str/String.

We could also probably start by either listing known issues/inconsistencies or any current issue links/RFCs on the matter.

FileDesc::drop silently drops the error if close() fails. How could we get this error out to the user code? Is there any other code that silently drops errors?

One model might be for std to keep a (short, fixed) queue of unreported errors globally. FileDesc::drop could push any error onto this queue. There could be a function for user code to call to check whether there are any pending errors to deal with, and return the error. (This maybe has some overlap with dealing with SIGINT and SIGTERM, when those are being caught.)

Any other ideas?

(moderated summary by @epage)

Context:

• "Rust makes writing crossplatform, tested, modern command line applications frictionless while incorporating industry best practices and providing great documentation." (goal)
• Focus is on CLI apps (see README for definition)

Assumptions and Implications

• Majority of CLIs will at more than just Rust developers (for targeting rust-developers, see #20)
• Users of CLIs will be somewhat technical considering they are using a terminal
• Use case priorities
  1. Using the app on the command line via the path
  2. Glued together with other apps in a script
  3. Drag and drop files onto an icon for the CLI
• Examples of content to distribute
  • one or more rust binaries
  • completion files
  • man pages (See #23)

Other solutions

• Go has equinox

Plan of attack

1. "How to package" documentation

• Spec out and document how to package

2. Automation

• Implement stager
• Reach out to binary packaging tools to consolidate effort with stager

In-ecosystem resources

• trust: example project that tarballs pre-built binaries releases via github releases
• crate-ci docs: A home for process automation documentation
• crate-ci meta: A home for discussing tooling and documentation, including discussing stager
• stager
• self-update
• cargo-deb
• cargo-wix
• cargo-arch
• cargo-rpm
• cargo-ebuild
• cargo-bundle

Open Issues

• (Windows) Adding manifests, see rust-lang/rfcs#721

Open Questions

• Prioritized list of package formats per platform?
  • Linux: deb, rpm, flatpak, snap
    • Are flatpak / snap's apps added to PATH?
  • Windows: MSI via wix, chocolatey
  • Mac: homebrew
    • How easily are pkg's available from command line?
• PPAs?
• Use case, challenges, and priority of source packages?
  • See mmstick/cargo-deb#40
• Importance of remote administration and provisioning?

@killercup's original post

In the first meeting, we identified packaging and distributing CLI apps a something we should improve.

We need to have a best-practice solution for getting from "I have a crate with a binary" to "I can ship this app and don't have to worry about cross-platform installation stuff".

Edit 2018-03-19: This issue is about distribution of rust binaries without using cargo-install.

We're searching for testimonials from companies that have been using Rust in CLI's in production and can tell their stories.

Primarily these are

1. Short, snappy quotes that we can put on the website directly, explaining how Rust and the ecosystem of crates made development of their CLI nicer/easier/better/etc
2. Blog posts on the company websites, videos or even talks at conferences that then go into detail that we link to from the website.
3. (Maybe) "Rust CLI Friends", being companies that use Rust in CLI's in production that don't have a blog or anything to link to, that could have their logo's featured without explicitly quoting them (the embedded-wg seems to have done this as well)

Summary / Motivation

The current cargo-install user experience is very poor. For both the user's
(people using cargo-install) and the developer's (people publishing binaries
through cargo-publish). For example having no upgrade story, having no way
to separate dependencies between libraries and binaries, and having no awareness
of binaries on the crates.io website.

CLI applications are currently the third most popular category
on crates.io. From general purpose tools like ripgrep, tokei, and hyperfine,
to rust development tools like the [diesel CLI], to the myriad of cargo plugins
which one of the biggest categories on crates.io. Despite crates.io initially only
being designed for libraries it's clear that people want to use it to distribute and
share Rust applications.

Cargo and the crates.io is a lot of developer's first distribution of their CLI
application. Improving this experience helps smooth out building a CLI
application to rust by having a complete distribution experience out of the box
with Cargo.

Upgrading

The is no current way in cargo to upgrade a binary. The user's options are to
run cargo install --force <package> or
cargo uninstall <package> && cargo install <package>. The problem with both of
these options is that neither respects the currently installed package's version
so to upgrade a cargo package requires a complete reinstall of the application
regardless of whether there is a newer version.

This is a very poor experience as Rust developer's we know how long Rust release
compile times are. Even with with all the great work the compiler does to reduce
compile times there is no better compile time than having to not compile at all.
Cargo should provide upgrade functionality either through a new subcommand a la
brew upgrade <package> or through a flag like pip pip install -u <package>.

Separation of dependencies

Currently cargo allows you to specify the developer's intent of a dependency
with [dev-dependencies] allowing a developer to use different dependencies for
development that the end user of their library or application doesn't need to
download or worry about compiling for their system. cargo-install does not
respect this and will download and build these packages.

Cargo currently does not have the ability to separate dependencies for binaries
and libraries. While at first this doesn't seem like a big issue since a binary
that depends on the library has to get all of their dependencies. This becomes
a problem for the reverse libraries downloading their binary's dependencies.
Which becomes an issue when installing a CLI through cargo-install.

Cargo should provide a way to specify library or binary specific dependencies.
This could be done in a similar way to development dependencies and having
[lib-dependencies], [bin.dependencies], and [bin.bin_name.dependencies]
which would allow developer's specify which packages are used only for libraries
or binaries while retaining the current behaviour and having [dependencies] be
for crates that used in all.

It might not be a big deal to get these packages but they do add a non trivial
amount of compile time. We also should not be wasting a user's bandwidth there
is still a lot of situations where bandwidth is tight whether the user lives in
an area with poor service or is using cellular which are no where near the
level's of bandwidth or stability, so any reduction of data is a
big improvement.

Showcasing applications

Currently crates.io only shows instructions on how to add a crate as library
dependency. Cargo should show cargo install <package> for binary only, and
show other binaries that are part of that crate.

Unresolved questions

• How cargo would remember features of an installed app?

cf. #45 for general idea

Let's get a website up and running, with some drafts for content, but no need to have anything ready and polished.

Decide

• How to implement this? mdbook? Jekyll?
• First content

(moderated summary by WG)

Context

Plan of Action

In-ecosystem resources

• Common directories
  • app-dirs-rs
  • app-dirs2
  • directories
• Read/write config
  • configure
  • config
  • twelf
• keyring for storing passwords

External inspiration

• cosmiconfig

Challenges

• Cross platform paths in config files. See also #10
• Integrating information from environment, command line arguments, defaults, and cascading config files
• Every existing applications is probably littered with special cases, migration schemes and special rules how settings are handled. I'm not sure it is possible to encode all this into an API without making it extremely hard to use.
• Good error messages, see serde-rs/serde#1184

@spacekookie's original poist

In the first meeting, we discussed the task of file management on different platforms (particularly configurations) and how it can be made better.

@Eijebong summarised it like this

If I need a config file, I don't want to know that it should be ${XDG_CONFIG:${XDG_HOME}/.config:/home/${user}/.config on linux, %AppDir%/App/ on windows and something else on osx [...]

There is a crate for "determing system configuration" (app-dirs-rs) but it seems unmaintained and not up to date

When thinking about networked services, one of the basic command line requirements is to accept a port to listen to.

This is usually expressed through the --port command. But that's not the only version; while talking in person, we've encountered the following variations:

• --port to accept a port (number)
• -p as a shorthand for --port
• $PORT, environment variable to accept a port (number)
• $LISTEN_FD to forward a socket, usually from Systemd
• launchd (MacOS) also has a thing

For an average program, this is a lot to think about. If we were to write this for every application we build, it'd both end up being repetitive boilerplate, and easy to get wrong.

So the question we're asking here is: how can we turn this information into a reusable component?

There's a few more constraints we need to think about here:

• This should be a component of an argument parsing solution, but stay flexible enough that other arguments can be specified.
• Handling ports is an example. Other solutions that could be used in conjunction would be: handling log levels, handling database ports, tokens, and more.
• It should be scoped to just handling arguments from the command line. Ideally there would be no friction between the parsing of arguments, and using the result in different frameworks / crates.

We're not sure what the solution to these problems are, but we do know that it would be beneficial to solve them. We'll continue from here by experimenting, and we encourage people interested in this to do the same.

Thanks!

Let's collect content for the fourth issue of this WG's newsletter here!

(will post the notes taken during the meeting here)

Show an example like
this
or this.

Subset of #38

• define a small but somewhat stable API
• example crate that shows the full workflow ("with a build script and everything")

cf. https://github.com/rust-clique/man

• assert_cmd based on predicates
• assert_fs
• first draft of nice guide

Creating a project by using cargo-generate binary, the command is wrongly spelt.

In tutorial/setup.md it is written as

cargo generate --git https://github.com/rust-clique/cargo-template-cli

but should be

cargo-generate --git https://github.com/rust-clique/cargo-template-cli

In the first meeting, we've discussed that we should have a best practice for error handling in CLI apps.

Open issues in other crates that we can target to improve the CLI ecosystem as discussed in the first meeting

CLI Args handling:

• clap 3.0 release

? in Main:

• RFC 1937

CI for binaries:

• trust
• crate-ci documentation

CLI Integration tests:

• assert_cli
• dir_diff

Man page / help docs:

• clap-rs

Next meeting is on Friday, March 23, 2018, at 6pm UTC via https://gitter.im/rust-lang/WG-CLI.

The main ones are SIGINT (^C) and SIGTERM (kill). For CLI apps that require no special cleanup, the default handling is fine (i.e. exit immediately and let the OS cleanup).

For CLI apps that need to do special cleanup (e.g. flush to files or database, nicely terminate network connections, deconfigure devices, etc), these signal handlers need to be caught. Some possible models:

• Catch signal, set a global flag, and have a call to check that global flag regularly in inner loops, and convert to an error to pass up the stack to do cleanup before exiting.

• Try and do cleanup in the signal handler. But this is pretty hard as the calls that are permitted in a signal handler are quite restricted (see man 7 signal-safety on Linux)

In the case of catching ^C, if the global flag is already set, then on the second ^C consider exiting immediately, to give the user a way out if the CLI app is being unresponsive.

I don't know Windows. What's the model there for these kinds of signals?

If there's a crate already out there that handles this well, then someone link that here and perhaps we can consider this dealt with.

Cargo and many sys creates depend on environmental variables for configuration, but there's no standard way to configure default env vars for a project.

Most of this configuration (such as target-cpu, crt-static and whether libraries should be linked statically) is absolutely critical for making high quality redistributable executables.

For example, forgetting about set RUSTFLAGS="-C target-feature=+crt-static" on Windows makes executables that don't work without the latest Visual Studio runtime, which is not available on Windows by default. Users get a scary warning about missing DLL, and googling DLL's name finds lots of shady sites trying serve adware/malware.

Similarly, forgetting to set all dependencies static on macOS with Homebrew means sys creates will use pkg-config, which will hardcode Homebrew-specific paths in the executable, and the executable won't run on other people's computers. "Works only on developers' machines" problem is not easy to spot before getting complaints from users for shipping a broken executable.

Env vars are also shared global mutable state, so even if you remember to set them, they make switching between projects harder and more error-prone.

• Cargo should use Cargo.toml for all configuration, so that git clone; cargo build can be made to do the right thing, instead of having to wrap cargo in another build process and remember to use it every time instead of the usual, simplest — but invalid — cargo build by mistake.

• The features mechanism should be extended, or another mechanism added, to correctly support selection of static vs dynamic linking of libraries. (e.g. you use foo-rs crate, but want to configure foo-sys to use static linking).

• If "Cargo-native" changes to Cargo.toml are undesirable and env vars are here to stay, then Cargo.toml should at least have a section for specifying env vars for the project, when it's build as the top-level project (i.e. not as a dependency. Dependencies setting global variables would be bad).

The book currently refers to a template you can get with cargo-generate to start writing your CLI crate.

This template repo doesn't exist! We should crate one.

Feel free to be inspired by quicli.

In the first meeting, we discussed how to decide what to work on. One approach is writing an "ideal guide".

The basic idea is to write guides about how to write a CLI app in Rust, but not by describing how it works right now, but how we imagine it should work some time in the future.

Let's collect content for the second issue of this WG's newsletter here!

The first issue was posted here.

(moderated summary by @epage)

Context

Common inputs to a CLI

• Files
• Command line flags
• Environment variables
• stdin
• signals

Common outputs to a CLI

• Files (sometimes unique, sometimes mutating the input)
• stdout

Plan of Attack

Testing crate(s)

• Make it easy to initialize a tempdir
  • Copy files over from tests dir
  • touch files
  • dump string to a file in tempdir
• cmd assertions
  • stdout
  • exit code
• file system assertions
  • file exists
  • file content assertions
  • fixture dir is a subset of target dir, reporting the differences
  • fixture dir exactly matches target dir, reporting the differences

In-ecosystem resources

• tempfile
• dir-diff
• assert_cli
• cli_test_dir
  • Can serve as inspiration but changing it based on input from this WG seems limited
  • From a deleted issue: "At this point, the API of cli_test_dir is unlikely to break backwards compatibility in any major ways. I'm happy to add new features to it that people need. And if anybody would like to re-export parts of it, I'm happy to do that, too."

Challenges

• Mocking
  • Should this be a priority?
    • For unit tests, we should probably encourage a higher level abstraction over the file system that can instead be mocked
    • For integration and end-to-end tests, we should probably run against real resources
  • https://github.com/iredelmeier/filesystem-rs
  • https://github.com/twmb/rsfs
  • https://github.com/manuel-woelker/rust-vfs
• Symbolic links on windows
• Testing colored output
• Testing signal handling

@killercup's original post

In the first meeting, we identified that testing CLI apps cross-platform is not trivial and that we want to improve the situation.

It seems like we're coming across a few small "footguns", some common some less so, so it may be nice to collect them in one place so when writing something like #6 we don't forget anything.

This could happen in #6, but I'd prefer if it were separate so this can dive a little deeper and actually make a list.

Off the top of my head from stuff that's come up today in the gitter and elsewhere:

• Reading stdin line by line is slow, because it does a heap alloc at each line. Easiest solution is BufReader::read_lines
• Using String as an input type for arbitrary data that doesn't necessarily need to be UTF-8
• Using println! when writing lots of small output, which locks stdout at each call. Prefer locking stdout manually and using something like write!
• I think #28 probably qualifies too, although it's probably far less common. The workout is simple, but not really intuitive IMO (calling libc::close manually, and checking for errors)
• println! can panic on a broken pipe
• std::env::args panics at invalid UTF-8 (the fix is to use std::env::args_os but this is not very intuitive, and with issues like #26 it's far more friction than it should be)

If you've got more I'll add them. Of course I think we should also be open to discussion about the ones I listed as to why they should, or should not be listed and/or other fixes for them. 😄

        let mut line = String::with_capacity(1024);
        while buf.read_line(&mut line).unwrap() > 0 {
            self.proc_str(&(line));
            line.clear();
        }

Kevin K. @kbknapp: 
> The reason i was asking is that since we are looking at strengthening existing crates, we should know what & who we are targetting
Totally agree!
My opinion (@killercup's could be different, I'm not sure :stuck_out_tongue_winking_eye:) is that if working group can develop a spec proposal for crate improvements, or a spec for non-existing crates and then release these specs to the either the community/crate owners for things that could be worked on it'd be a huge win
There are tons of people in the community that want to help and work on something, but just need to know what to work on, or how it should be built
And you're totally correct, that unless we know what already exists, we can't really determine where the gaps are

Pascal Hertleif @killercup: @kbknapp totally agree! we need to work on defining solutions, ideally with experts in the problem space (like, people who already have crates that do stuff), before implementing (by creating or improving)

Kevin K. @kbknapp: yep, exaclty :thumbsup:

Pascal Hertleif @killercup: i should've probably put a label like "Research Phase" on the current work

Kevin K. @kbknapp: I think it'd be great to get some candid feedback from people like @BurntSushi and @sharkdp (as well as other, like the Exa dev, Habitat devs, and so many more), etc who maintain large production CLIs and see what their pain points were as well as what crates they're currently using.
I know we've already a few of those messages from people in this group, which is invaluable
But maybe another way to get some info would be a quick survey sent out to a targeted few (i.e. known CLI authors), and put out a community call for all CLI authors to participate as well
Cause it's totally understandable that some may not have the time, or even approval, to participate in something like a working group. But a survey should be simple

Pascal Hertleif @killercup: @kbknapp i agree and will try to reach out to some people again (standalone and not as part of "wanna join this working group", just asking for feedback)

Kevin K. @kbknapp: Also, I didn't mean any offense to anyone who I didn't list above...theres so many! I know @Aaronepower has some I use all the time, etc :wink:

Pascal Hertleif @killercup: setting up and evaluating a survey is a lot of work though

Kevin K. @kbknapp: agreed, but I don't think it needs to be super formal or involved. It could be, "what crates are using", "any pain points?", "any strengths", just something to give a more wide reaching audience so we don't accidentally over optimize for a select few
just thoughts :grinning:

Pascal Hertleif @killercup: and they're good thoughts! if someone (you? :grinning:) wants to help with running the survey we can totally do it :grinning:

Dylan DPC @Dylan-DPC: can help if needed :grinning:

Kevin K. @kbknapp: Yeah I can probably help as well, I don't mind giving some feedback on the design of any survey or going over results
Maybe we can start with an initial survey design in the etherpad, cause that's easier for everyone to edit than an issue?

Pascal Hertleif @killercup: @kbknapp @Dylan-DPC great!

Kevin K. @kbknapp: I'll create a tracking issue as well

Pascal Hertleif @killercup: :thumbsup:

We'd like to put together a survey. Ideally we'd like to send this survey out to a number of individuals and organizations specificially, but also open it up to the Rust community to fill out.

I think it'd be a good idea to split this into several "phases"

• Design
• Execution
• 🏃‍♂️ Analysis
• Reveal of Results

The design phase will be open and available for all to edit on the ether pad. Once we're happy with the design of the survey we'll move into the execution phase which I for see lasting a minimum of two weeks, potentially a month. In this phase we should post the survey to the subreddit, {u,i}rlo forums, twitter, etc.

Finally the analysis piece will be conducted immediately following execution completion. The analysis piece will last as long as necessary due to participants time, but if at all possible no longer than two weeks.

The results phase will be a "report" of such to the CLI-WG with a proposal of some areas of focus or potential low hanging fruit such as existing gaps in the ecosystem, areas of focus for improvement, etc.

As of now, it's not planned to release these results to the wider community, unless they're interested in the results and of course at the discretion of the Rust team.

Having said all that, let's begin the design 😄

Being able to document exit codes would be a useful for many programs, especially when reimplementing existing programs where exit codes can have special meanings.

This thread serves to gather resources about discussions happening around this topic, with the goal that eventually we'll be in a position where it's possible to document all exit codes from our programs.

Related Issues

• signal handling https://github.com/rust-lang-nursery/cli-wg/issues/27
• man pages https://github.com/rust-lang-nursery/cli-wg/issues/38
• error handling in CLI apps https://github.com/rust-lang-nursery/cli-wg/issues/12
• support development of well-behaved CLI apps https://github.com/rust-lang-nursery/cli-wg/issues/21

Links

• https://internals.rust-lang.org/t/mini-pre-rfc-redesigning-process-exitstatus/5426/15
• rust-lang/rust#48711

CLI applications are expected to behave in a certain way. This is especially important when those applications are used in scripts.

• Exit status should reflect success or failure (see also: rust-lang/rust#48453)
• Correct use of standard out and standard error
• Accept input from standard in (optional)
• Command-line option consistency
• Exit gracefully when a "broken pipe" error occurs
• Enable/disable ANSI color escapes (see also: https://github.com/rust-lang-nursery/cli-wg/issues/15)
• Structured output (log files, json, etc)
• Correctly handles signals (SIGHUP, SIGINT, SIGCHLD, etc)
• Respect hard/soft resource limits (e.g. max open files)

I can always add more from the comments.

In the first meeting, we discussed setting a slogan for this working group, to make it easy to describe to others what we're trying to achieve and to remind ourselves what we want to focus on.

I'll post a few proposals below, feel free to add yours.

@yoshuawuyts from gitter

On

• panic
• request

Report

• Method to report information
  • email from Cargo.toml?
  • repo url from Cargo.toml?
• Information to help CLI author
  • OS
  • Rust version
  • Crate version
  • git commit (if possible) in case they are using master rather than a published version

@Centril and I talked a bit about writing property tests for CLI apps. The general idea is to write input generation and assertions on CLI app invocations without much overhead (e.g. without having to deal with manually formatting types as CLI arguments).

The most trivial implementation that already works is probably this:

proptest! {
    #[test]
    fn same_input_back(string: String) {
        assert_cli::Assert::command(&["echo", &string])
            .stdout().is(&string)
            .unwrap();
    }
}

It generates random strings and asserts that echos gives us back the same string it was fed with.

We found that aside from the prop-testing specific challenges, there are also CLI-specific helpers needed, e.g. to easily generate temp dirs with (also generated) files, and that OS-specific or environment specific settings need to be able to be passed as arguments (or to be overwritten to reduce/eliminate dependence on global state).

In general, we think that we can use assert_cli to write these helpers. What we need to find is a way to express relations between arguments (inputs) and the asserts (like "stdout contains").

For the record but not meant as a concrete suggestions as something to implement today, I proposed a macro/DLS like:

cli_check!(cargo_binary("brighten"), ("--color": Color, "--amount": 0f32..=1) -> (stdout: Color));

that passes two arguments to a binary called brighten and asserts that, given valid arguments, the tool writes a string to stdout that can be parsed as Color. Obviously missing here is a way to setup the environment. Side-effect of this syntax is that generating invalid (or omitting) arguments and asserting that the program fails is easy.

(moderated summary by the WG)

• .., .., // not being auto- handled
• windows gotchas, like long file paths (which require extended-length paths aka "verbatim paths" as they're called in std)
  • https://github.com/dherman/verbatim
  • rust-lang/rust#32689
  • BurntSushi/ripgrep#364
  • How do we deal with these and relative paths?
• absolute paths on Windows (UNC prefices)
  • https://github.com/notion-cli/notion/blob/master/crates/verbatim/src/lib.rs
• UTF-8 filenames on MacOS (the filesystem normalizes UTF-8 strings)
  • BurntSushi/ripgrep#845
• atomic file writes
• Path.join: A method that both concatenates a path or completely replaces the existing one based on the input ... when did one ever have that use-case?
• "Simple" API that comes at the cost of performance for rapid prototyping / throwaway "scripts", see also https://github.com/rust-lang-nursery/cli-wg/issues/26
  • Priority should be on working out how the normal rust stuff should behave before wrapping it up in the high level, python-pathlib-like API.
• format! a path for a command line argument or mutating a path can cause problems with the non-UTF8 nature of OsStr

@killercup's original post

In the first meeting, we talked a bit about the pain points of dealing with files and path in a cross-platform manner.

One idea is to create or improve crates that provide higher-level abstractions than std in this area.

(Might no make it in time for Edition 2018 Preview 2)

First public preview release of Clap 3!

Issues tracked at https://github.com/kbknapp/clap-rs/milestone/73

• using assert_cmd’s features to quickly run cargo binaries with different inputs and assert their outputs
• generating temp dirs with demo files

Since we are not just creating libraries but also collecting and curating knowledge around writing CLI crates, we need a good place to document this. I propose we create a web site that is hosted as part of this repo.

For this, we added a mdBook to this repository whose rendered version you can find here: https://rust-lang-nursery.github.io/cli-wg/

Content

• A "learn Rust by writing a CLI app" tutorial
  • @killercup has an initial draft ready to go but it'll be a lot of work to really polish this
• Best practices
  • What we learned from #29 and #21
• Hidden treasures of the ecosystem (which will then no longer be hidden)
  • cf. #11

Prior art:

• Rust WASM book: https://rustwasm.github.io/book/
• @epage has already created https://github.com/crate-ci a https://crate-ci.github.io which track, develop, and document tooling and best practices around maintaining crates.

Alternatives/additions

• Add content to other resources! Cross-linking should be encouraged.

Content team

Who is responsible for (maintaining) the content?

Move "testable"/pure functions into "library" part,
write unit tests/docs tests.
Only makes sense if there are enough.
Otherwise keep in main.rs.
Discuss advantages of thinking of library API.

The survey mentions the ergonomics of cross compiling quite frequently. While not directly tied to writing CLI applications, it's somewhat implied if you're releasing for multiple platforms.

I've opened this tracking issue to discuss the current best practices/guides, and if there are specific ways in which we can improve the experience.

Add a first draft for the signal handling chapter of the CLI book (#45).

• What are signals
• Easy crate to deal with Ctrl+C
• Handling signals with threads and channels?

Here's something to get you started:

Processes like command line applications need to react to signals sent by the operating system.
The most common example is probably Ctrl+C.

To handle these signals in Rust you need to consider how you can receive these signals as well as how you can react to them.

See #27 for a lot of discussion around signals and crates for handling them!

(forked from #8)

This is for discussing documentation formats and authoring.

This is a tracking issue for man page creation. The initial goal is to provide a flow to generate man pages from Structopt instances. This is a specialization of https://github.com/rust-lang-nursery/cli-wg/issues/23.

Example

The goal is to be able to use code similar to this.

// src/cli.rs
extern crate structopt;

#[derive(StructOpt, Debug)]
pub struct Opts {
  /// TCP port to listen to.
  #[structopt(
    short = "p", long = "port", env = "PORT", default_value = "8080"
  )]
  port: usize,
}

// src/lib.rs
#[macro_use]
extern crate structopt;

pub mod cli;

// build.rs
extern crate structop_to_man;

use structopt::{clap::Shell, StructOpt};

include!("src/lib.rs");

fn main() {
  let outdir = ::std::env::var_os("OUT_DIR").expect("OUT_DIR not found.");
  let mut app = cli::Opts::clap();
  app.gen_completions("my_app", Shell::Fish, &outdir); // generate shell completions
  structop_to_man::gen_man(&app, &outdir); // generate man pages
}

Checklist

I think the following steps are required to make this happen.

• Base man page generator (roff) - https://github.com/killercup/roff-rs
• Higher level man page generator (wrap roff, write to Writer).
• Parse Clap struct & convert to individual fields (see clap-rs/clap#1285 for current status).
• Combine Clap struct parser + man page generator into single crate that can generate a man page and write it to Writer.

While I was writing rust-cli/man#8, I started to wonder: how could we create git-style subcommands?

So in Git, each subcommand is also available as a standalone binary. So git add also exists as git-add. A cool pattern this is commonly used for is to load extra commands at runtime. If we wanted to add the git foo command, all we'd need to do is make git-foo available in $PATH. I believe cargo takes a similar approach too.

I feel this problem is reasonably well understood, but I haven't seen much talk about it. Is there any good prior art available? Can this be done using clap? And is there perhaps a way we could document on how to best approach this?

Thanks!

I was trying to get people to embrace this crate before as a unified way to control colors but it did not get far: https://crates.io/crates/clicolors-control (that crate also turns on ansi colors for the windows console as an added bonus)

The idea was that you have one common crate that turns on and off colors for the entire cli app.

I'm fine finding a different mechanism for this but right now controlling colored output is really not a particularly fun experience.

Anything got some thoughts on this matter?

This is a tracking/discussion issue for stdout and stderr output escaping.

This is pretty generic problem, however normally not that many programs face it.

My example

In Mainframer we pipe output from ssh to the Rust program's stdout and stderr respectively.

If ssh is forced to run in interactive mode (-t) where pseudo-terminal is allocated and the program we're running via ssh does some complicated progress output (Gradle build system) it can lead to Terminal state being corrupted.

Generic example

If you cat some binary file, there is a good chance it'll break Terminal state.

To my understanding, some unexpected control symbols leak through stdout/stderr and break Terminal state.

Hacky Solutions

In some cases (not my though) running:

$ reset

After the Rust program will fix Terminal.

In my case, running:

$ stty sane

Fixes the Terminal.

Related to #8

I package my apps with a bash script sort of like this:

cargo build --release
tar cjf app-$VERSION.tar.gz target/release/app
rsync etc.

The problematic part is $VERSION. I would like to read the version from Cargo.toml, but can't without string manipulation in bash (yuck!)

I'd be great if there was something like cargo metadata --query package.version that outputs nothing by the given Cargo.toml key.

In the first meeting, we discussed ecosystem discoverability.

The goal is to make it easier for people to get the right tools they need to smoothly write CLI apps in Rust.

also cc #57

Let's collect content for the third issue of this WG's newsletter here!