Code Monkey home page Code Monkey logo

fcp's Introduction

fcp

CI status fcp crate Packaging status

fcp is a significantly faster alternative to the classic Unix cp(1) command.

fcp aims to handle the most common use-cases of cp with much higher performance.

fcp does not aim to completely replace cp with its myriad options.

Note: fcp is optimized for systems with an SSD. On systems with a HDD, fcp may exhibit poor performance.

Installation

Please note that fcp supports only Unix-like operating systems (e.g. Linux, macOS, etc.).

Pre-built binaries

Pre-built binaries for some systems can be found under this repository's releases.

Via cargo

fcp requires Rust version 1.53.0 or newer. fcp can be installed using cargo by running the following:

cargo install fcp

Arch Linux

fcp can be installed on Arch Linux via the fcp-bin AUR.

NixOS

As of NixOS 21.11 fcp is included in the stable channel. For earlier versions, fcp is available through nixpkgs-unstable. Assuming you've already added the nixpkgs-unstable channel, fcp can be installed by running the following:

nix-env -iA unstable.fcp

macOS

fcp can be installed on macOS via Homebrew by running the following:

brew install fcp

Usage

Usage information can be found by running fcp --help, and has been reproduced below:

fcp 0.2.1

USAGE:
    fcp [OPTIONS] SOURCE DESTINATION_FILE
    Copy SOURCE to DESTINATION_FILE, overwriting DESTINATION_FILE if it exists

    fcp [OPTIONS] SOURCE ... DESTINATION_DIRECTORY
    Copy each SOURCE into DESTINATION_DIRECTORY

OPTIONS:
    -h, --help
            Output this usage information and exit.

    -V, --version
            Output version information and exit.

Benchmarks

fcp doesn't just claim to be faster than cp, it is faster than cp. As different operating systems display different performance characteristics, the same benchmarks were run on both macOS and Linux.

macOS

The following benchmarks were run on a 2018 MacBook Pro1 (2.9 GHz 6-Core Intel Core i9, 16 GiB RAM, SSD) with APFS as the filesystem.

Large Files

The following shows the result of a benchmark which copies a directory containing 13 different 512 MB files using cp and fcp, with fcp being approximately 822x faster on average (note the units of the axes for each plot)2:

fcp is approximately 822x faster than cp, with fcp's average time to copy being approximately 4.5 milliseconds, while cp's average time to copy is approximately 3.7 seconds

Linux Kernel Source

The following shows the result of a benchmark which copies the source tree of the Linux kernel using cp and fcp, with fcp being approximately 6x faster on average:

fcp is approximately 6x faster than cp, with fcp's average time to copy being approximately 5.1 seconds, while cp's average time to copy is approximately 30 seconds

Linux

The following benchmarks were run on a bare-metal AWS EC2 instance (a1.metal, 16 CPUs, 32 GiB RAM, SSD) with XFS as the filesystem.

Linux Kernel Source

The following shows the result of a benchmark which copies the source tree of the Linux kernel using cp and fcp, with fcp being approximately 10x faster on average:

fcp is nearly 10x faster than cp, with fcp's average time to copy being approximately 675 milliseconds, while cp's average time to copy is approximately 6.02 seconds

Large Files

The following shows the result of a benchmark which copies a directory containing 13 different 512 MB files using cp and fcp, with fcp being approximately 1.4x faster on average:

fcp is approximately 1.4x faster than cp, with fcp's average time to copy being approximately 8 seconds, while cp`'s average time to copy is approximately 11.3 seconds

Methodology

fcp's high-performance can be attributed to several factors, but is primarily the result of leveraging parallelism, distributing the work of walking directories and copying their contents across all of your machine's cores. This leads to a significant performance increase on systems with an SSD, as more I/O requests are issued over the same period of time (as compared to a single-threaded approach), resulting in a higher-average queue depth, thus allowing higher utilization of the SSD (as a function of its maximum IOPS) and correspondingly higher throughput.

Additionally, on macOS (and perhaps in the future on other operating systems) fcp utilizes the system's underlying copy-on-write capability, dramatically reducing the time needed to copy large files.

These two factors – in addition to an overall performance-conscious approach to this problem – serve to explain fcp's significantly improved performance relative to cp.


[1] While in general you should avoid benchmarking on laptops, fcp is a developer tool and many developers work primarily on laptops. Also unlike with Linux where you can rent by the second, the minimum tenancy for AWS EC2 macOS instances is 24 hours, and these benchmarks took less than an hour.

[2] The massive difference in performance in this case is due to fcp using fclonefileat and fcopyfile under the hood.

fcp's People

Contributors

svetlitski avatar vladimyr avatar xychu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

fcp's Issues

Request for musl build?

I recently discovered this cool program and thought I'd try it out on the one non-laptop SSD system I have. Unfortunately:

$ fcp
fcp: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by fcp)

In the past when I've seen things like this with Rust projects, I've asked if the developers could make a musl build instead of (or in addition to) a glibc build. Do you think fcp could have a musl build? Or perhaps it requires glibc?

(Of course, it might be "I should update my OS", but it's a government system so OS updates are...let's call it deliberate in pace.)

Disappointing results on workstation with btrfs raid1

Hi

I decided to try out this utility on my workstation which seems to bottleneck on single core during file copying and so far results that I'm getting are extremely disappointing at best.

image

Using traditional cp utility resulted in copying file in around 8:10 (~996 MB/s), while fcp accomplished the same task in 9:39 (~843 MB/s).

image

Of course both results are extremely disappointing considering it's copying from NVME Gen4 RAID1 to different NVME Gen4 RAID1 which has theoretical throughput of around 14 GB/s read and 6 GB/s write (used drives are FireCuda 530 and Samsung 990 PRO and they don't have any problem with reaching advertised speeds during fio seq i/o benchmarks)

It's worth to point out that in both cases neither cp nor fcp exceeded 100% of single core usage which corresponds to negligible CPU usage on 24c/48t Threadripper workstation. Disks are far from being 100% utilized and in matter of fact it's possible to hack bash script that achieves higher performance for copying single file.

size=`du -m "$1" | grep -o "^[0-9]\+"`
threads=8
block=1000
count=$((size/block/threads+1))
for t in `seq 0 $((threads-1))` ; do
        o=$((t*count))
        echo dd if="$1" of="$2" iseek=$o oseek=$o bs=${block}M count=$count
        (dd if="$1" of="$2" iseek=$o oseek=$o bs=${block}M count=$count) &
done
wait

this thing copies the same file in 5:24 (~1507 MB/s) and hammers disks utility to 100% :/

I'm not sure where to start troubleshooting. All those results are very pathetic considering raw SSDs performance.

Support for common flags from GNU ls

fcp is pretty great, and I make a lot of use out of it!

One feature that would be nice is a few flags...

[] -n/--no-clobber
[] -a/--archive
[] -d/--no-dereference
[] --preserve={all, mode,ownership,timestamps,context,links,xattr}

The -a and -n are very to prevent overwriting files, as well as preserving extra metadata (especially on macOS!)

[Feature] Support clonefile(2) on macOS

fcp is an awesome project, and amazingly fast.

However, it could be even faster if it used the clonefile(2) syscall on macOS. This is instantaneous and creates a copy-on-write file backed by the same filesystem blocks as the original. For copies that do not cross onto different filesystems / devices, this is effectively identical to a manual copy, while being O(1) instead of O(n).

Rendered man page here: https://www.manpagez.com/man/2/clonefile/
Raw man page here: https://opensource.apple.com/source/xnu/xnu-3789.21.4/bsd/man/man2/clonefile.2.auto.html

Compile error: "use of unstable library feature 'array_from_ref' "

Hi, when I typed cargo install fcp the following error was given:

error[E0658]: use of unstable library feature 'array_from_ref'
   --> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/fcp-0.2.0/src/lib.rs:190:61
    |
190 |         (Ok(metadata), _) if metadata.is_dir() => copy_into(array::from_ref(source), dest),
    |                                                             ^^^^^^^^^^^^^^^
    |
    = note: see issue #77101 <https://github.com/rust-lang/rust/issues/77101> for more information

error: aborting due to previous error

For more information about this error, try `rustc --explain E0658`.
error: could not compile `fcp`

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: failed to compile `fcp v0.2.0`, intermediate artifacts can be found at `/tmp/cargo-installoQO1VQ`

Caused by:
  build failed

So what should I do to get it to compile? Also, maybe the answer should be put on the readme page to help others :)

[Feature] A progress meter would be nice

Currently fcp is a "do one thing, do it well" tool, but it lacks a lot of introspection.

It would be nice if there were a --progress flag that printed out the number of files / bytes copied per second, so that one could gauge how fast a copy is performing.

Feature request: Speed limit and IOPS limit.

Sometimes (especially when the 12309 or something gets in the way) unfortunate filesystem copy operation may affect the whole system.

One of workarounds may be limiting speed.

Ideally it should support setting or changing the limit after copy operation has already started (like with pv).

Deadlock when copying to a FAT32 volume

My wife asked me to transfer her photos from the backup location on my server to her usb thumb drive. In order than she can mount this drive on Windows I have it formatted as FAT32. I thought I'd give fcp a go to speed up the transfer since it usually takes a very long time with rsync or cp. While the transfer did begin at an amazing speed, it eventually deadlocks waiting for a FUTEX that apparently never gets released. I was able to determine this by stracing the process.

I attempted this several times to see if it was a fluke and it deadlocked every time after transfering about 8-10G of data

Benchmarks question: what is copying time?

Benchmark plots show some time period in seconds.

What does that time mean? The time between CLI tool start and when it exits? Or the time when copying actually finishes under the hood (so e.g. removable storage can be ejected)?

Also were benchmarks performed hot (when involved data was already in RAM) or cold (e.g. like after Mac analogue of /proc/sys/vm/drop_caches, if it exists)?

Also how benchmarks look with NFS or removable storage?

support copy-on-write on Linux

For filesystems that support it, Linux offers the FICLONE operation of the ioctl syscall which results in what is commonly called a reflink and defers creating another copy of the data until necessary. This could speed up copying on Linux dramatically in some cases.

windows build fails with `cargo install fcp`

error
❯ cargo install fcp
    Updating crates.io index
  Installing fcp v0.2.1
   Compiling autocfg v1.1.0
   Compiling crossbeam-utils v0.8.8
   Compiling cfg-if v1.0.0
   Compiling lazy_static v1.4.0
   Compiling rayon-core v1.9.2
   Compiling scopeguard v1.1.0
   Compiling libc v0.2.123
   Compiling num_cpus v1.13.1
   Compiling bitflags v1.3.2
   Compiling either v1.6.1
   Compiling memoffset v0.6.5
   Compiling crossbeam-epoch v0.9.8
   Compiling rayon v1.5.2
   Compiling crossbeam-channel v0.5.4
   Compiling nix v0.22.3
   Compiling crossbeam-deque v0.8.1
   Compiling fcp v0.2.1
error[E0433]: failed to resolve: could not find `unix` in `os`
 --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\lib.rs:9:14
  |
9 | use std::os::unix::fs::{MetadataExt, PermissionsExt};
  |              ^^^^ could not find `unix` in `os`

error[E0433]: failed to resolve: could not find `sys` in `nix`
 --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:6:10
  |
6 | use nix::sys::stat::Mode;
  |          ^^^ could not find `sys` in `nix`

error[E0432]: unresolved import `nix::unistd`
 --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:7:5
  |
7 | use nix::unistd;
  |     ^^^^^^^^^^^ no `unistd` in the root

error[E0433]: failed to resolve: could not find `unix` in `os`
  --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:10:14
   |
10 | use std::os::unix::fs::{self as unix, DirBuilderExt, FileTypeExt, OpenOptionsExt, PermissionsExt};
   |              ^^^^ could not find `unix` in `os`

error[E0432]: unresolved import `std::os::unix`
  --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:10:14
   |
10 | use std::os::unix::fs::{self as unix, DirBuilderExt, FileTypeExt, OpenOptionsExt, PermissionsExt};
   |              ^^^^ could not find `unix` in `os`

error[E0433]: failed to resolve: use of undeclared type `Mode`
  --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:79:16
   |
79 |     let mode = Mode::from_bits_truncate(permissions.mode().try_into()?);
   |                ^^^^ use of undeclared type `Mode`

error[E0599]: no method named `mode` found for struct `DirBuilder` in the current scope
  --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:61:10
   |
61 |         .mode(mode)
   |          ^^^^ method not found in `DirBuilder`

error[E0599]: no method named `mode` found for struct `OpenOptions` in the current scope
  --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:69:10
   |
69 |         .mode(mode)
   |          ^^^^ method not found in `OpenOptions`

error[E0599]: no method named `mode` found for struct `Permissions` in the current scope
  --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:79:53
   |
79 |     let mode = Mode::from_bits_truncate(permissions.mode().try_into()?);
   |                                                     ^^^^ method not found in `Permissions`

error[E0599]: no method named `is_fifo` found for struct `std::fs::FileType` in the current scope
   --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:102:29
    |
102 |         } else if file_type.is_fifo() {
    |                             ^^^^^^^ help: there is an associated function with a similar name: `is_file`

error[E0599]: no method named `is_socket` found for struct `std::fs::FileType` in the current scope
   --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:104:29
    |
104 |         } else if file_type.is_socket() {
    |                             ^^^^^^^^^ method not found in `std::fs::FileType`

error[E0599]: no method named `is_char_device` found for struct `std::fs::FileType` in the current scope
   --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:106:29
    |
106 |         } else if file_type.is_char_device() {
    |                             ^^^^^^^^^^^^^^ method not found in `std::fs::FileType`

error[E0599]: no method named `is_block_device` found for struct `std::fs::FileType` in the current scope
   --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:108:29
    |
108 |         } else if file_type.is_block_device() {
    |                             ^^^^^^^^^^^^^^^ method not found in `std::fs::FileType`

error[E0599]: no method named `mode` found for struct `Permissions` in the current scope
  --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\lib.rs:48:72
   |
48 |                 let mut dest = fs::create(dest, metadata.permissions().mode())?;
   |                                                                        ^^^^ method not found in `Permissions`

error[E0599]: no method named `mode` found for struct `Permissions` in the current scope
  --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\lib.rs:62:70
   |
62 |     fs::create_dir(dest, fs::symlink_metadata(source)?.permissions().mode())?;
   |                                                                      ^^^^ method not found in `Permissions`

error[E0599]: no method named `ino` found for struct `Metadata` in the current scope
   --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\lib.rs:102:64
    |
102 |         .map(|ancestor| fs::metadata(ancestor).map(|meta| meta.ino()));
    |                                                                ^^^ method not found in `Metadata`

error[E0599]: no method named `ino` found for struct `Metadata` in the current scope
   --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\lib.rs:108:68
    |
108 |         .map(|source| fs::symlink_metadata(source).map(|meta| meta.ino()))
    |                                                                    ^^^ method not found in `Metadata`

error[E0599]: no method named `ino` found for struct `Metadata` in the current scope
   --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\lib.rs:201:46
    |
201 |         (_, Ok(metadata)) if source_metadata.ino() == metadata.ino() => fatal(format!(
    |                                              ^^^ method not found in `Metadata`

error[E0599]: no method named `ino` found for struct `Metadata` in the current scope
   --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\lib.rs:201:64
    |
201 |         (_, Ok(metadata)) if source_metadata.ino() == metadata.ino() => fatal(format!(
    |                                                                ^^^ method not found in `Metadata`

Some errors have detailed explanations: E0432, E0433, E0599.
For more information about an error, try `rustc --explain E0432`.
error: could not compile `fcp` due to 19 previous errors
warning: build failed, waiting for other jobs to finish...
error: failed to compile `fcp v0.2.1`, intermediate artifacts can be found at `C:\Users\hitus\AppData\Local\Temp\cargo-installK8TlaQ`

Caused by:
  build failed

Explanation

Can you explain why is it so much faster?

Why this method not in kernel/base linux or other systems?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.