Code Monkey home page Code Monkey logo

tin-summer's Introduction

The Tin Summer

Windows build status Build Status

If you do a significant amount of programming, you'll probably end up with build artifacts scattered about. sn is a tool to help you find those artifacts.

sn is also a replacement for du. It has nicer output, saner commands and defaults, and it even runs faster on big directories thanks to multithreading.

Installation

Script

Enter the following in a command prompt:

curl -LSfs https://japaric.github.io/trust/install.sh | sh -s -- --git vmchale/tin-summer

Binary install

If the script doesn't work for you, you can download a binary from the releases page.

Cargo

If your platform doesn't have binaries, or you just want to build from source, get cargo. Then:

 $ cargo install tin-summer

Make sure you are on nightly; otherwise

 $ rustup run nightly cargo install tin-summer

Use

To list directory and file sizes for the current directory:

$ sn f

To get a list of directory sizes concurrently, excluding version control:

 $ sn p --exclude '\.git|\.pijul|_darcs|\.hg'

To get a sorted list of the 12 biggest directories in $DIR:

 $ sn sort $DIR -n12

To search current directory for directories with build artifacts:

 $ sn ar

To look for artifacts or directories containing artifacts that occupy more than 200MB of disk space:

 $ sn ar -t200M

Accessibility

To turn off colorized output:

export CLICOLOR=0

Comparison (or, 10 Things I Hate About du)

Reasons to use du

  • Reads disk usage, not just file sizes
  • Optionally dereferences symlinks
  • Slightly faster on small directories
  • Stable and well-supported

Reasons to use sn

  • Faster on large directories
  • Uses regex for exclusions, making it dramatically faster than du when used with the --exclude flag.
  • Defaults to human-readable output
  • Colorized output
  • Nicer help via clap
  • Provides sorted output
  • Finds build artifacts
  • Reads file sizes, not disk usage
  • Extensible in Rust

Benchmark results

Directory Size Tool Command Time
600MB sn sn p 60.74 ms
600MB sn sn d 99.92 ms
600MB du du -hacd2 88.28 ms
4GB sn sn p 185.2 ms
4GB sn sn d 271.9 ms
4GB du du -hacd2 195.5 ms
700MB sn sn p 91.05 ms
700MB sn sn d 176.3 ms
700MB du du -hacd2 153.8 ms
7MB sn sn p 19.48 ms
7MB sn sn d 12.72 ms
7MB du du -hacd2 10.13 ms

These commands are all essentially equivalent in function, except that sn p may use more threads than sn a or du. Results were obtained using Gabriel Gonzalez's bench tool. You can see pretty criterion graphs here or here.

In summary: yes, sn actually is faster on larger directories, but it is also slower on small ones. I'm hoping to make it faster in the future; the current naïve concurrency model has obvious directions for improvement.

Screenshots (alacritty + solarized dark)

The Tin Summer

Displaying a user's timeline in a terminal.

du

Displaying a user's timeline in a terminal.

Heuristic for build artifacts

Currently, sn looks for files that either have an extension associated with build artifacts, or executable files that are ignored by version control. It also looks for "build directories", like .stack-work, elm-stuff, etc. and if it finds a configuration file like tweet-hs.cabal, it considers all their contents to be build artifacts.

Languages Supported

The following is a list of languages sn artifacts has been tested with. The intent is to support basically anything, so feel free to open a PR or start an issue.

  • Haskell (incl. GHCJS)
  • Rust
  • Julia
  • Python
  • Elm
  • Nim
  • Vimscript
  • TeX
  • Idris
  • FORTRAN
  • Ruby
  • C
Autoclean

sn can clean up your artifacts for you, but only for the above-indicated languages. It is still experimental, but it has been tested and should not delete unwanted files (though it may not clean everything it should).

tin-summer's People

Contributors

killercup avatar matklad avatar vmchale avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

tin-summer's Issues

Prettify things: Change title and ordering in README

If this tool is intended to be both a replacement for du and build artefacts, change the title accordingly.
You could make two headlines linking to 1.cleanup functionality and 2.du-replacement option.

Please link to the supported languages in the first 15 lines of the README.

Question about crate/cmd name

Hi and thanks for a very nice tool!

I am confused about the cargo crate name tin-summer and it's relation to the installed command line tool sn.

From what I can guess from the README, especially the oskar1.png, sn might derive from "file-sniffer"

Might a better crate name, or command line name, or possibly just a logical connection between the two be provided?

Personally, I wouldn't mind fu, as in "file usage", as it would align nicely with du (disk usage) naming, but I can see that would raise some eyebrows.

Also, there is a command line tool sn from the mono project, which would be in path if user installed mono:

$ sn
Mono StrongName - version 5.0.1.1
StrongName utility for signing assemblies
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Usage: sn [-q | -quiet] options [parameters]

 -q | -quiet    	Quiet mode (minimal display)

Help options
 -? | -h        	Show this help screen about the tool
 -? | -h config 	Configuration options
 -? | -h csp    	Crypto Service Provider (CSP) related options
 -? | -h convert	Format convertion options
 -? | -h sn     	StrongName signing options

EDIT: I realized sn is the chemical symbol for tin. But the issue regarding name collide with mono tool might still be worth discussing.

Make `directories` subcommand sortable

It seems like only artifacts is sortable right now, it would be great if other commands like directories were as well.

$ sn --sort d -d 1 .
error: Found argument '--sort' which wasn't expected, or isn't valid in this context
	Did you mean to put '--sort' after the subcommand 'artifacts'?

USAGE:
    sn <SUBCOMMAND>
$ sn d -d 1 --sort .
error: Found argument '--sort' which wasn't expected, or isn't valid in this context

USAGE:
    sn directories --depth <NUMBER>

For more information try --help

Thanks, great tool!

tin-summer does not count the size of directories

How to reproduce:

Run this in an empty directory:

for i in {1..1000}; do mkdir $i; done
echo -n "test" > 123/file

If we now use du to report the "disk usage" and the "apparent size" of that directory, we get:

▶ du -s -B1
4096	.

▶ du -s -B1 --apparent-size
60064	.

However, sn only reports the 4 bytes from the 123/file:

▶ sn files -d0
4    b	 .

▶ sn parallel -d0
4    b	 .

Line 192, Missing `mode` method for `Permissions` in tin-summer v1.21.14

Hello,

When compiling on Win 11:

   Compiling tin-summer v1.21.14
error[E0599]: no method named `mode` found for struct `Permissions` in the current scope
   --> C:\Users\RFID\.cargo\registry\src\github.com-1ecc6299db9ec823\tin-summer-1.21.14\src\walk_parallel\single_threaded.rs:192:39
    |
192 |             if metadata.permissions().mode() == 0o755 || REGEX_GITIGNORE.is_match(path_str) {
    |                                       ^^^^ method not found in `Permissions`

For more information about this error, try `rustc --explain E0599`.
error: could not compile `tin-summer` due to previous error
warning: build failed, waiting for other jobs to finish...
error: failed to compile `tin-summer v1.21.14`, intermediate artifacts can be found at `C:\Users\RFID\AppData\Local\Temp\cargo-installLTttvs`

Windows

It seems Windows is not currently supported

Release version has lockfile that is out of sync

The current 1.21.14 release tag has a Cargo.lock file that is not in sync with Cargo.toml. It cannot be build with cargo build --locked and hence cannot be used to produce reproducible package builds.

panic report(s) :)

thread 'main' panicked at 'OS String invalid.: "./rust/rust/build/x86_64-unknown-linux-gnu/test/run-make/linker-output-non-utf8.stage2-x86_64-unknown-linux-gnu/zzz�"', /checkout/src/libcore/result.rs:860
stack backtrace:
   0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
             at /checkout/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::panicking::default_hook::{{closure}}
             at /checkout/src/libstd/sys_common/backtrace.rs:71
             at /checkout/src/libstd/sys_common/backtrace.rs:60
             at /checkout/src/libstd/panicking.rs:355
   2: std::panicking::rust_panic_with_hook
             at /checkout/src/libstd/panicking.rs:371
             at /checkout/src/libstd/panicking.rs:549
   3: std::panicking::begin_panic
             at /checkout/src/libstd/panicking.rs:511
   4: std::panicking::begin_panic_fmt
             at /checkout/src/libstd/panicking.rs:495
   5: core::panicking::panic_fmt
             at /checkout/src/libstd/panicking.rs:471
   6: core::result::unwrap_failed
   7: libsniff::prelude::read_all
   8: libsniff::prelude::read_all
   9: libsniff::prelude::read_all
  10: libsniff::prelude::read_all
  11: libsniff::prelude::read_all
  12: libsniff::prelude::read_all
  13: libsniff::prelude::read_all
  14: libsniff::prelude::read_all
  15: sniff::main
  16: main
  17: __libc_start_main
  18: _start

Need pretty printing cross-platform

In Windows the file size printing is ugly.
Would be nice to have pretty printing cross-platform.

PS C:\Users\RFID\Desktop\Development> sn ar
←[32m887  b←[0m  .\debbie\migrations
←[32m1.3  kB←[0m         .\debbie\src
←[32m30   kB←[0m         .\debbie
←[32m357  b←[0m  .\eddie\.github
←[32m963  b←[0m  .\eddie\.vscode
←[32m408  b←[0m  .\eddie\docs
←[32m8.0  kB←[0m         .\eddie\examples
←[32m29   kB←[0m         .\eddie\src
←[32m39   kB←[0m         .\eddie
←[32m2.9  kB←[0m         .\egui\.cargo
←[32m9.0  kB←[0m         .\egui\.github
←[32m121  b←[0m  .\egui\.vscode
←[32m4.4  MB←[0m         .\egui\docs
←[32m148  kB←[0m         .\egui\eframe
←[32m965  kB←[0m         .\egui\egui
←[32m46   kB←[0m         .\egui\egui-wgpu
←[32m43   kB←[0m         .\egui\egui-winit
←[32m62   kB←[0m         .\egui\egui_demo_app
←[32m265  kB←[0m         .\egui\egui_demo_lib
←[32m76   kB←[0m         .\egui\egui_extras
←[32m36   kB←[0m         .\egui\egui_glium
←[32m66   kB←[0m         .\egui\egui_glow
←[32m4.0  kB←[0m         .\egui\egui_web
←[32m66   kB←[0m         .\egui\emath
←[32m1.7  MB←[0m         .\egui\epaint
←[32m82   kB←[0m         .\egui\examples
←[32m8.3  MB←[0m         .\egui\media
←[32m8.0  kB←[0m         .\egui\sh
←[32m16   MB←[0m         .\egui
←[32m358  b←[0m  .\estienne\.github
←[32m640  b←[0m  .\estienne\benches
←[32m18   kB←[0m         .\estienne\src
←[32m38   kB←[0m         .\estienne
←[32m504  b←[0m  .\LISA\.github
←[32m12   kB←[0m         .\LISA\.jj
←[32m1.4  kB←[0m         .\LISA\.vscode
←[32m492  kB←[0m         .\LISA\examples
←[32m27   kB←[0m         .\LISA\src
←[32m624  kB←[0m         .\LISA
←[32m545  kB←[0m         .\slr\examples
←[32m3.7  kB←[0m         .\slr\src
←[32m558  kB←[0m         .\slr
←[32m100  b←[0m  .\tin-summer\.cargo
←[32m77   b←[0m  .\tin-summer\bash
←[32m3.2  kB←[0m         .\tin-summer\benches
←[32m3.3  kB←[0m         .\tin-summer\ci
←[32m1.2  kB←[0m         .\tin-summer\ion
←[32m3.4  kB←[0m         .\tin-summer\man
←[32m164  kB←[0m         .\tin-summer\screenshots
←[32m94   kB←[0m         .\tin-summer\src
←[32m306  kB←[0m         .\tin-summer
←[32m17   MB←[0m         .

Remove `update` command

I don't think a small command line utility should have a built in update command. That task is best left for an out of band mechanism such as cargo install or a package manager.

It is fine to have precompiled binaries ready for download but please reconsider making an install script a primary way of installing and updating this tool.

Name collision

$ sn --version
Mono StrongName - version 5.0.0.0
StrongName utility for signing assemblies
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Unknown option --version
$ pacman -Qo sn
/usr/bin/sn is owned by mono 5.0.0.100-2

De-humanize sizes

In general the feature set seems to be very usable, but to make it actually usable for me, I need to have exact bytesizes to get processed by another script.

So is there an option that actually always lists bytes?

Failed to parse .gitignore

$ ~/.cargo/bin/sn r
Warning: failed to parse .gitignore/.ignore at "path/to/coreutils/.gitignore", ignoring

path/to/coreutils contains an up-to-date clone of git://git.sv.gnu.org/coreutils.

Unhandled Exception.

Bug Report

Output

Unhandled Exception:
System.TypeInitializationException: The type initializer for 'System.Console' threw an exception. ---> System.TypeInitializationException: The type initializer for 'System.ConsoleDriver' threw an exception. ---> System.Exception: Magic number is wrong: 542
  at System.TermInfoReader.ReadHeader (System.Byte[] buffer, System.Int32& position) [0x00028] in <a84b655e5e6a49ee96b338ec792f5580>:0 
  at System.TermInfoReader..ctor (System.String term, System.String filename) [0x0005f] in <a84b655e5e6a49ee96b338ec792f5580>:0 
  at System.TermInfoDriver..ctor (System.String term) [0x00055] in <a84b655e5e6a49ee96b338ec792f5580>:0 
  at System.ConsoleDriver.CreateTermInfoDriver (System.String term) [0x00000] in <a84b655e5e6a49ee96b338ec792f5580>:0 
  at System.ConsoleDriver..cctor () [0x0004d] in <a84b655e5e6a49ee96b338ec792f5580>:0 
   --- End of inner exception stack trace ---
  at System.Console.SetupStreams (System.Text.Encoding inputEncoding, System.Text.Encoding outputEncoding) [0x00007] in <a84b655e5e6a49ee96b338ec792f5580>:0 
  at System.Console..cctor () [0x0008e] in <a84b655e5e6a49ee96b338ec792f5580>:0 
   --- End of inner exception stack trace ---
  at Mono.Tools.SN.Main (System.String[] args) [0x0004a] in <f57b897282924139a6887b1b45cb2427>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: System.TypeInitializationException: The type initializer for 'System.Console' threw an exception. ---> System.TypeInitializationException: The type initializer for 'System.ConsoleDriver' threw an exception. ---> System.Exception: Magic number is wrong: 542
  at System.TermInfoReader.ReadHeader (System.Byte[] buffer, System.Int32& position) [0x00028] in <a84b655e5e6a49ee96b338ec792f5580>:0 
  at System.TermInfoReader..ctor (System.String term, System.String filename) [0x0005f] in <a84b655e5e6a49ee96b338ec792f5580>:0 
  at System.TermInfoDriver..ctor (System.String term) [0x00055] in <a84b655e5e6a49ee96b338ec792f5580>:0 
  at System.ConsoleDriver.CreateTermInfoDriver (System.String term) [0x00000] in <a84b655e5e6a49ee96b338ec792f5580>:0 
  at System.ConsoleDriver..cctor () [0x0004d] in <a84b655e5e6a49ee96b338ec792f5580>:0 
   --- End of inner exception stack trace ---
  at System.Console.SetupStreams (System.Text.Encoding inputEncoding, System.Text.Encoding outputEncoding) [0x00007] in <a84b655e5e6a49ee96b338ec792f5580>:0 
  at System.Console..cctor () [0x0008e] in <a84b655e5e6a49ee96b338ec792f5580>:0 
   --- End of inner exception stack trace ---
  at Mono.Tools.SN.Main (System.String[] args) [0x0004a] in <f57b897282924139a6887b1b45cb2427>:0 

Details

This was obtained trying to use the binary release tin-summer-1.21.1-x86_64-unknown-linux-gnu.tar.gz

Additionally, trying to use the cargo installation I received:

Unhandled Exception:                                                                                            
System.TypeInitializationException: The type initializer for 'System.Console' threw an exception. ---> System.Ty
peInitializationException: The type initializer for 'System.ConsoleDriver' threw an exception. ---> System.Excep
tion: Magic number is wrong: 542
  at System.TermInfoReader.ReadHeader (System.Byte[] buffer, System.Int32& position) [0x00028] in <a84b655e5e6a4
9ee96b338ec792f5580>:0                  
  at System.TermInfoReader..ctor (System.String term, System.String filename) [0x0005f] in <a84b655e5e6a49ee96b3
38ec792f5580>:0                                                                                                
  at System.TermInfoDriver..ctor (System.String term) [0x00055] in <a84b655e5e6a49ee96b338ec792f5580>:0
  at System.ConsoleDriver.CreateTermInfoDriver (System.String term) [0x00000] in <a84b655e5e6a49ee96b338ec792f55
80>:0
  at System.ConsoleDriver..cctor () [0x0004d] in <a84b655e5e6a49ee96b338ec792f5580>:0
   --- End of inner exception stack trace ---
  at System.Console.SetupStreams (System.Text.Encoding inputEncoding, System.Text.Encoding outputEncoding) [0x00
007] in <a84b655e5e6a49ee96b338ec792f5580>:0
  at System.Console..cctor () [0x0008e] in <a84b655e5e6a49ee96b338ec792f5580>:0
   --- End of inner exception stack trace ---
  at Mono.Tools.SN.Main (System.String[] args) [0x0004a] in <f57b897282924139a6887b1b45cb2427>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.TypeInitializationException: The type initializer for 'System.Console' threw an exception. ---> System.TypeInitializationException: The type initializer for 'System.ConsoleDriver' threw an exception. ---> System.Exception: Magic number is wrong: 542
  at System.TermInfoReader.ReadHeader (System.Byte[] buffer, System.Int32& position) [0x00028] in <a84b655e5e6a49ee96b338ec792f5580>:0
  at System.TermInfoReader..ctor (System.String term, System.String filename) [0x0005f] in <a84b655e5e6a49ee96b338ec792f5580>:0
  at System.TermInfoDriver..ctor (System.String term) [0x00055] in <a84b655e5e6a49ee96b338ec792f5580>:0
  at System.ConsoleDriver.CreateTermInfoDriver (System.String term) [0x00000] in <a84b655e5e6a49ee96b338ec792f5580>:0
  at System.ConsoleDriver..cctor () [0x0004d] in <a84b655e5e6a49ee96b338ec792f5580>:0
   --- End of inner exception stack trace ---
  at System.Console.SetupStreams (System.Text.Encoding inputEncoding, System.Text.Encoding outputEncoding) [0x00007] in <a84b655e5e6a49ee96b338ec792f5580>:0
  at System.Console..cctor () [0x0008e] in <a84b655e5e6a49ee96b338ec792f5580>:0
   --- End of inner exception stack trace ---
  at Mono.Tools.SN.Main (System.String[] args) [0x0004a] in <f57b897282924139a6887b1b45cb2427>:0

System details

screenneo

`clean` command removed gpg private key

Running sn clean ~ on 1.21.8 deleted my gpg private key. If I didn't have a backup, I'd be pretty upset right now.

Couldn't believe it at first so I reran a couple times, and it indeed removes the entire "~/.gnupg/private-keys-v1.d" folder. Obviously, this should be fixed ASAP. Could it be possible somebody sneeked in some malicious code? Dunno if I can trust this key anymore.

Incorrect disk usage/directory size

I got this on an NFS volume, and there are many many many small (1-2KB) files in the directory I'm listing (not directly tho).

$ du -hd0 logs ; sn p -d0 logs
1.3T	logs # du output
3.4  TB	 logs # sn output

tin-summer counts hardlinks twice

How to reproduce:

In an empty directory, create a 1MB file and a hard link to it:

dd if=/dev/zero bs=1M count=1 > file
ln file hardlink

sn now reports a total size of 2 MB:

▶ sn files -a
1.0  MB	 ./hardlink
1.0  MB	 ./file
2.0  MB	 .

Whereas it should rather be 1 MB:

▶ du -sh                
1,0M	.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.