Currently, trying to call self encryption on a small file leads to undesirable results.
We should just have 1 chunk + data map for a small file, instead we get 3 chunks. + data map.
It is from this function get_num_chunks inside self encryption.

Also, earlier we had a method to encrypt small files separately, but now it is removed, maybe that is the cause? Anyways, move that functionality into self-encryption crate.

Also add logging to self encryption.

When the self encryptor stores data on the network it assumes that the address of the data is equal to it's hash. In the new Unpublished ImmutableData type the address of the data is the hash of the (name + owner).
Due to this there is a mismatch between the addresses in the DataMap and the actual data address on the network, which causes a NoSuchData error.

serde_cbor is unmaintained

The serde_cbor crate is unmaintained. The author has archived the github repository.

Alternatives proposed by the author:

*ciborium
*minicbor

See advisory page for additional details.

This will require a change to the memmap crate which would have to be agreed and published by the author of that crate.

I'm hitting problems with a couple of the libraries, but see that the CI results on github say they're ok. The other one that fails is message_filter, but not for build, only for cargo test.

Here's the result for self_encryption:

$ rustc --version
rustc 1.6.0-nightly (5b4986fa5 2015-11-08)
$ cargo update
    Updating registry `https://github.com/rust-lang/crates.io-index`
      Adding libc v0.1.12
$ cargo build
   Compiling self_encryption v0.2.2 (file:///home/mrh/src/safe-nightly/self_encryption)
src/lib.rs:143:26: 143:43 error: unresolved name `sha512::HASHBYTES` [E0425]
src/lib.rs:143 const HASH_SIZE: usize = sha512::HASHBYTES;
                                        ^~~~~~~~~~~~~~~~~
src/lib.rs:143:26: 143:43 help: run `rustc --explain E0425` to see a detailed explanation
error: aborting due to previous error
Could not compile `self_encryption`.

merged into the aes crate

The aesni crate has been merged into the aes crate. The new repository
location is at:

<https://github.com/RustCrypto/block-ciphers/tree/master/aes>

AES-NI is now autodetected at runtime on i686/x86-64 platforms.
If AES-NI is not present, the aes crate will fallback to a constant-time
portable software implementation.

To prevent this fallback (and have absence of AES-NI result in an illegal
instruction crash instead), continue to pass the same RUSTFLAGS which were
previously required for the aesni crate to compile:

RUSTFLAGS=-Ctarget-feature=+aes,+ssse3

See advisory page for additional details.

Note: There may be an update to any test that used a hard coded output to check encryption,

merged into the aes crate

The aes-soft crate has been merged into the aes crate. The new repository
location is at:

<https://github.com/RustCrypto/block-ciphers/tree/master/aes>

AES-NI is now autodetected at runtime on i686/x86-64 platforms.
If AES-NI is not present, the aes crate will fallback to a constant-time
portable software implementation.

To force the use of a constant-time portable implementation on these platforms,
even if AES-NI is available, use the new force-soft feature of the aes
crate to disable autodetection.

See advisory page for additional details.

Readme shows raw table marker. Needs fixed.
cc @nbaksalyar

$ cargo bench
     Running target/release/lib-a1c7f0b70117a838

running 5 tests
test bench_write_then_read_a_200B ... bench:     11708 ns/iter (+/- 532) = 34 MB/s
test bench_write_then_read_b_1KB  ... bench:     42412 ns/iter (+/- 3723) = 48 MB/s
test bench_write_then_read_c_1MB  ... thread '<main>' panicked at 'index out of bounds: the len is 1048576 but the index is 1048576', /home/rustbuild/src/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libcollections/vec.rs:1356

This is on Debian.

Crate provides you with http://maidsafe.net/self_encryption/self_encryption/latest
But it is actually http://maidsafe.net/self_encryption/0.2.0/self_encryption/index.html ?

If asked to encrypt a folder rather than a file, then error.
(Also wildcards that match more than one file appear not supported - eg normal/*)

$ ./basic_encryptor  -e /media/sf_VMSharedFolder/normal
! chunk_store_test AlreadyExists
thread '<main>' panicked at 'called `Result::unwrap()` on an `Err` value: Error { repr: Os(21) }', /home/rustbuild/src/rust-buildbot/slave/beta-dist-rustc-linux/build/src/libcore/result.rs:729

Rust 1.1 now targets musl, this will means 100% statically linked binaries with zero dependencies

Initial run of RustFmt on libraries that are not being worked on at the moment.

I am tackling this issue myself as part of the code bounty program as suggested by Fraser and approved by David.

Add the following lint checks to lib.rs (crust is a good example to check)

#![deny(deprecated, improper_ctypes, missing_docs, non_shorthand_field_patterns,
overflowing_literals, plugin_as_library, private_no_mangle_fns, private_no_mangle_statics,
raw_pointer_derive, stable_features, unconditional_recursion, unknown_lints, unsafe_code,
unsigned_negation, unused, unused_allocation, unused_attributes, unused_comparisons,
unused_features, unused_parens, while_true)] 

#![warn(trivial_casts, trivial_numeric_casts, unused_extern_crates, unused_import_braces,
unused_qualifications, variant_size_differences)]

This test is exposing at least one bug in the library. When the test fails, usually two or four threads panic and the test hangs indefinitely.

Thank you for contributing to the project!
We recommend you check out our "Contributing to the SAFE Network" guide if you haven't already.

Describe the bug

Originally found through integrating sn_client libraries into sn_httpd: maidsafe/safe_network#1621 (comment)

In short, when attempting to retrieve small parts of medium sized files (up to 1.5 MB), the relative_pos is incorrectly calculated for index 2+. This results in relative_pos being too low, causing an overlap with prior calls to seek_info when stepping through a file.

To Reproduce
Steps to reproduce the behavior:

1. Call seek_info with a file smaller than 1.5 MB, using a small length such as 64 KB and increment the pos by 64 KB each time.
2. Once index 2 is referenced, the relative_pos will return duplicate bytes from the prior call.

Expected behavior
There should be no duplicate bytes returned from prior calls to seek_info().

Screenshots
This is a comparison between 2 files, one using a large length (512 KB) which is successful, vs one which used 128 KB and had duplicate bytes:

Desktop (please complete the followin
g information):

• OS: Linux

Smartphone (please complete the following information):

• Device: Laptop

Additional context

We need to revert rust-crypto to version "0.2.30" since it has a critical issue in Release mode.

These lines in fn decrypt_chunk do absolutely nothing:

            if content.len() == 0 {
                ()
            }

I can apprently encrypt OK, Decrypting is less fun.
Latest git pull from all libs and last nights rust reported ready to roll

First I check to ensure I have no "stale" files lying around from previous attempts, then I succesfully encrypt a file on my Desktop, then I fail to restore that file to a different name.

willie@gagarin:/projects/MaidSafe/self_encryption$ locate chunk_store_test
willie@gagarin:/projects/MaidSafe/self_encryption$ cargo run --verbose --example basic_encryptor -- -e /home/willie/Desktop/WindsorDavies.jpg
Fresh byteorder v0.3.10
Fresh regex-syntax v0.1.2
Fresh rustc-serialize v0.3.15
Fresh strsim v0.3.0
Fresh libc v0.1.8
Fresh regex v0.1.33
Fresh cbor v0.3.7
Fresh num_cpus v0.2.6
Fresh rand v0.3.8
Fresh libsodium-sys v0.0.5
Fresh docopt v0.6.66
Fresh asynchronous v0.4.5
Fresh tempdir v0.3.4
Fresh sodiumoxide v0.0.5
Fresh self_encryption v0.1.4 (file:///home/willie/projects/MaidSafe/self_encryption)
Running target/debug/examples/basic_encryptor -e /home/willie/Desktop/WindsorDavies.jpg
chunk written
chunk written
chunk written
chunk written
willie@gagarin:/projects/MaidSafe/self_encryption$ cargo run --verbose --example basic_encryptor -- -d /home/willie/Desktop/WindsorDavies.jpg /home/willie/Desktop/OhDearHowSad.jpg
Fresh strsim v0.3.0
Fresh rustc-serialize v0.3.15
Fresh regex-syntax v0.1.2
Fresh libc v0.1.8
Fresh byteorder v0.3.10
Fresh regex v0.1.33
Fresh num_cpus v0.2.6
Fresh rand v0.3.8
Fresh libsodium-sys v0.0.5
Fresh cbor v0.3.7
Fresh docopt v0.6.66
Fresh asynchronous v0.4.5
Fresh tempdir v0.3.4
Fresh sodiumoxide v0.0.5
Fresh self_encryption v0.1.4 (file:///home/willie/projects/MaidSafe/self_encryption)
Running target/debug/examples/basic_encryptor -d /home/willie/Desktop/WindsorDavies.jpg /home/willie/Desktop/OhDearHowSad.jpg
! chunk_store_test AlreadyExists
thread '' panicked at 'called Result::unwrap() on an Err value: Decode(Other("Expected Unicode string or variant map, but got Break"))', ../src/libcore/result.rs:731
Process didn't exit successfully: target/debug/examples/basic_encryptor -d /home/willie/Desktop/WindsorDavies.jpg /home/willie/Desktop/OhDearHowSad.jpg (exit code: 101)
willie@gagarin:/projects/MaidSafe/self_encryption$

crate has been renamed to block-cipher

This crate has been renamed from block-cipher-trait to block-cipher.

The new repository location is at:

<https://github.com/RustCrypto/traits/tree/master/block-cipher>

See advisory page for additional details.

Hi,

I've just started to play with this library. I am using the sample code from example/basic_encryptor.rs. When trying to encrypt a very small file (< 20 characters), no chunk is written to chunk_store_test. But the data_map seems fine, because I can decrypt and retrieve back my original file. If I try with a bigger file, I've got chunks in chunk_store_test directory. Is that normal?

If that helps, I'm using rustc 1.0.0-nightly (2baf34825 2015-04-21) (built 2015-04-22)

Thanks!

In cases such as usage with unpublished immutable data, we should be able to delete unpublished data at an address.
If the data at the address is a data-map, we should be able to delete the individual chunks referenced in the map. This should be done recursively until all the chunks are deleted.

Trait functions which the user is supposed to handle, don't cater for errors. Presently they look like this:

pub trait Storage {
    /// Fetch the data bearing the name
    fn get(&self, name: &[u8]) -> Vec<u8>;
    /// Insert the data bearing the name.
    fn put(&mut self, name: Vec<u8>, data: Vec<u8>);
}

So anyone implementing those and running into errors is stuck as there is no proper value to return to SE. These traits should probably return S.E defined errors and S.E crate can then decide what to do in case of errors - either return immediately with and error or rollback if possible etc. That would be much cleaner.

A check on chunk content hash is required before decryption in case of corruption. The example should fail gracefully without further corruption of chunks if the check fails.

A way to have the decryption key built into the main application to decrypt on runtime when running the encrypted executable file

Replace rust_sodium dependency in self_encryption with crate aes.

• AES128 is preferred to be used to replace the encryption mechanisms in the modules wherever rust_sodium is currently used.
• Removal of rust_sodium makes self_encryption a Rust-only crate by moving away from C dependencies.

In the current implementation, when a file is self encrypted the chunks are held locally until self_encryptor.close() is called. When the SAFE Client Libraries consume these functions for Immutable Data the data is encrypted and stored locally and in the end all the chunks are sent to the network at once which increases the load on the network.

It would be nice to be able to start writing / streaming chunks to the network as they are being encrypted.

obtuse@kali:~/rust/self_encryption$ cargo run --verbose --example basic_encryptor -- -e LICENSE
Fresh regex v0.1.30
Fresh byteorder v0.3.9
Fresh libc v0.1.7
Fresh gcc v0.3.5
Fresh strsim v0.3.0
Fresh rustc-serialize v0.3.14
Fresh rand v0.3.8
Fresh docopt v0.6.64
Fresh cbor v0.3.6
Fresh tempdir v0.3.4
Fresh time v0.1.25
Compiling self_encryption v0.1.1 (file:///home/obtuse/rust/self_encryption)
Running rustc src/lib.rs --crate-name self_encryption --crate-type lib -g --out-dir /home/obtuse/rust/self_encryption/target/debug --emit=dep-info,link -L dependency=/home/obtuse/rust/self_encryption/target/debug -L dependency=/home/obtuse/rust/self_encryption/target/debug/deps --extern crypto=/home/obtuse/rust/self_encryption/target/debug/deps/libcrypto-67f1ad94f3ffe5f4.rlib --extern tempdir=/home/obtuse/rust/self_encryption/target/debug/deps/libtempdir-46f6e863906dc7d1.rlib --extern rustc_serialize=/home/obtuse/rust/self_encryption/target/debug/deps/librustc_serialize-9ef26f158d5284e0.rlib --extern rand=/home/obtuse/rust/self_encryption/target/debug/deps/librand-b924d9fc5b3eb5b8.rlib --extern docopt=/home/obtuse/rust/self_encryption/target/debug/deps/libdocopt-40f44425b0f8ac23.rlib --extern cbor=/home/obtuse/rust/self_encryption/target/debug/deps/libcbor-a029b1d21f636c96.rlib -L native=/home/obtuse/rust/self_encryption/target/debug/build/rust-crypto-67f1ad94f3ffe5f4/out -L native=/home/obtuse/rust/self_encryption/target/debug/build/time-e758cbe877e9589d/out
Fresh rust-crypto v0.2.31
src/lib.rs:382:10: 382:13 error: binary operation + cannot be applied to type collections::vec::Vec<u8> [E0369]
src/lib.rs:382 (vec + &n_1_vec[48..64] + &n_2_vec[..] , n_1_vec[0..32].to_vec() , n_1_vec[32..48].to_vec())
^~~
error: aborting due to previous error
Could not compile self_encryption.
Caused by:
Process didn't exit successfully: rustc src/lib.rs --crate-name self_encryption --crate-type lib -g --out-dir /home/obtuse/rust/self_encryption/target/debug --emit=dep-info,link -L dependency=/home/obtuse/rust/self_encryption/target/debug -L dependency=/home/obtuse/rust/self_encryption/target/debug/deps --extern crypto=/home/obtuse/rust/self_encryption/target/debug/deps/libcrypto-67f1ad94f3ffe5f4.rlib --extern tempdir=/home/obtuse/rust/self_encryption/target/debug/deps/libtempdir-46f6e863906dc7d1.rlib --extern rustc_serialize=/home/obtuse/rust/self_encryption/target/debug/deps/librustc_serialize-9ef26f158d5284e0.rlib --extern rand=/home/obtuse/rust/self_encryption/target/debug/deps/librand-b924d9fc5b3eb5b8.rlib --extern docopt=/home/obtuse/rust/self_encryption/target/debug/deps/libdocopt-40f44425b0f8ac23.rlib --extern cbor=/home/obtuse/rust/self_encryption/target/debug/deps/libcbor-a029b1d21f636c96.rlib -L native=/home/obtuse/rust/self_encryption/target/debug/build/rust-crypto-67f1ad94f3ffe5f4/out -L native=/home/obtuse/rust/self_encryption/target/debug/build/time-e758cbe877e9589d/out (exit code: 101)

arr! macro erases lifetimes

Affected versions of this crate allowed unsoundly extending
lifetimes using arr! macro. This may result in a variety of
memory corruption scenarios, most likely use-after-free.

See advisory page for additional details.

Remove size limits on files. This will mean streaming and flushing to disk first. That allows changes in the file to be handled off-line (i.e. where an app uses this directly and is rendering scenes or similar). On file close all chunks should go to the network.
The temp disk store must be encrypted and cleared when the file is written to the network.

I'm having problems downloading 5MB file when I request chunks anything less than that.

For instance, file is about 5MB, I request chunk 0-65536.

In the prepare_window function self.sequencer is expanded to accomodate 0-65536 elements.
(I'm talking about sources which are on crates.io at the moment, seems like things might have changed a little as I view source from this repository, if it was already fixed tell me)

However, downloaded chunk size seems exactly 1 MB. And it goes onto writing those chunks in sequencer which is currently only 0-65536 and "out of bounds" error happens. I've added a hack to extend sequencer one by one (probably won't be efficient with memory mapped files), it works now. Not sure if this is the best solution though. Left comments in all caps.

    /// Prepare a sliding window to ensure there are enough chunk slots for writing, and to read in
    /// any absent chunks from external storage.
    fn prepare_window(&mut self, length: u64, position: u64) {
        // EXPANSION - EXPANDS ONLY ACCORDING TO REQUESTED RANGE
        if (length + position) as usize > self.sequencer.len() {
            let tmp_size = self.sequencer.len();
            self.sequencer.extend(repeat(0).take((length as usize + position as usize) - tmp_size));
        }
        if self.file_size < (3 * MIN_CHUNK_SIZE) as u64 {
            return
        }
        let mut first_chunk = self.get_chunk_number(position);
        let mut last_chunk = self.get_chunk_number(position + length);
        if self.file_size < (3 * MAX_CHUNK_SIZE) as u64 {
            first_chunk = 0;
            last_chunk = 3;
        } else {
            for _ in 0..2 {
                if last_chunk < self.get_num_chunks() {
                    last_chunk += 1;
                }
            }
        }
        // [TODO]: Thread next - 2015-02-28 06:09pm
        let mut vec_deferred = Vec::new();
        for i in first_chunk..last_chunk {
            let mut found = false;
            for itr in self.chunks.iter() {
                if itr.number == i {
                    let pos = self.get_start_end_positions(i).0;
                    if itr.location == ChunkLocation::Remote {
                        vec_deferred.push(self.decrypt_chunk(i)
                            .chain::<_,String,_>(move |res|{
                                Ok((pos, res.unwrap()))
                            })
                        );
                    }
                    found = true;
                    break;
                }
            }
            if !found {
                self.chunks.push(Chunks{number: i, status: ChunkStatus::ToBeHashed,
                                        location: ChunkLocation::InSequencer});
            }
        }
        for (pos, vec) in
            Deferred::vec_to_promise(vec_deferred, ControlFlow::ParallelCPUS).sync().unwrap() {
            let mut pos_aux = pos;
            for itr2 in vec.iter() {
                // PREVIOUS VERSION - CRASHING
                //self.sequencer[pos_aux as usize] = *itr2;
                // HACK
                if self.sequencer.len() != (pos_aux as usize) {
                    self.sequencer[pos_aux as usize] = *itr2;
                } else {
                    self.sequencer.extend(repeat(*itr2).take(1));
                }
                pos_aux += 1;
            }
        }
    }

Unclear if function get_pad_key_and_iv is used beyond just the basic_encryptor.

Following through the simple example

cargo build --example basic_encryptor --release

Which works well normally.

This panics, in the case that a block is missing:

src/lib.rs:247:21
from fn get_pad_key_and_iv

let src_hash = &chunk_hashes[chunk_index];

So, done just by moving a block out of the way.

$ ./basic_encryptor -d /tmp/chunk_store_test/secret_key "/tmp/chunk_store_test/hello"
thread '' panicked at 'index out of bounds: the len is 2 but the index is 2', src/lib.rs:247:21

Compilation error with latest nightly rustc 1.3.0-nightly (6d71ce536 2015-07-06)

Compiling self_encryption v0.2.0 (file:///home/user/MaidSafe-Testing/maidsafe_rust/self_encryption)
src/lib.rs:521:13: 521:22 error: variable does not need to be mutable
src/lib.rs:521         let mut start;
                           ^~~~~~~~~
src/lib.rs:102:28: 102:34 note: lint level defined here
src/lib.rs:102         unsigned_negation, unused, unused_allocation, unused_attributes, unused_comparisons,
                                          ^~~~~~
error: aborting due to previous error
Could not compile `self_encryption`.

To learn more, run the command again with --verbose.
   Compiling self_encryption v0.2.0 (file:///home/user/MaidSafe-Testing/maidsafe_rust/self_encryption)
src/lib.rs:521:13: 521:22 error: variable does not need to be mutable
src/lib.rs:521         let mut start;
                           ^~~~~~~~~
src/lib.rs:102:28: 102:34 note: lint level defined here
src/lib.rs:102         unsigned_negation, unused, unused_allocation, unused_attributes, unused_comparisons,
                                          ^~~~~~
error: aborting due to previous error
Build failed, waiting for other jobs to finish...
src/lib.rs:521:13: 521:22 error: variable does not need to be mutable
src/lib.rs:521         let mut start;
                           ^~~~~~~~~
src/lib.rs:102:28: 102:34 note: lint level defined here
src/lib.rs:102         unsigned_negation, unused, unused_allocation, unused_attributes, unused_comparisons,
                                          ^~~~~~
src/lib.rs:634:13: 634:25 error: variable does not need to be mutable
src/lib.rs:634         let mut data_map: datamap::DataMap;
                           ^~~~~~~~~~~~
src/lib.rs:102:28: 102:34 note: lint level defined here
src/lib.rs:102         unsigned_negation, unused, unused_allocation, unused_attributes, unused_comparisons,
                                          ^~~~~~
src/lib.rs:665:13: 665:25 error: variable does not need to be mutable
src/lib.rs:665         let mut data_map: datamap::DataMap;
                           ^~~~~~~~~~~~
src/lib.rs:102:28: 102:34 note: lint level defined here
src/lib.rs:102         unsigned_negation, unused, unused_allocation, unused_attributes, unused_comparisons,
                                          ^~~~~~
src/lib.rs:707:13: 707:25 error: variable does not need to be mutable
src/lib.rs:707         let mut data_map: datamap::DataMap;
                           ^~~~~~~~~~~~
src/lib.rs:102:28: 102:34 note: lint level defined here
src/lib.rs:102         unsigned_negation, unused, unused_allocation, unused_attributes, unused_comparisons,
                                          ^~~~~~
src/lib.rs:749:13: 749:25 error: variable does not need to be mutable
src/lib.rs:749         let mut data_map: datamap::DataMap;
                           ^~~~~~~~~~~~
src/lib.rs:102:28: 102:34 note: lint level defined here
src/lib.rs:102         unsigned_negation, unused, unused_allocation, unused_attributes, unused_comparisons,
                                          ^~~~~~
src/lib.rs:791:13: 791:25 error: variable does not need to be mutable
src/lib.rs:791         let mut data_map: datamap::DataMap;
                           ^~~~~~~~~~~~
src/lib.rs:102:28: 102:34 note: lint level defined here
src/lib.rs:102         unsigned_negation, unused, unused_allocation, unused_attributes, unused_comparisons,
                                          ^~~~~~
src/lib.rs:836:13: 836:25 error: variable does not need to be mutable
src/lib.rs:836         let mut data_map: datamap::DataMap;
                           ^~~~~~~~~~~~
src/lib.rs:102:28: 102:34 note: lint level defined here
src/lib.rs:102         unsigned_negation, unused, unused_allocation, unused_attributes, unused_comparisons,
                                          ^~~~~~
src/lib.rs:882:13: 882:25 error: variable does not need to be mutable
src/lib.rs:882         let mut data_map: datamap::DataMap;
                           ^~~~~~~~~~~~
src/lib.rs:102:28: 102:34 note: lint level defined here
src/lib.rs:102         unsigned_negation, unused, unused_allocation, unused_attributes, unused_comparisons,
                                          ^~~~~~
src/lib.rs:912:13: 912:25 error: variable does not need to be mutable
src/lib.rs:912         let mut data_map: datamap::DataMap;
                           ^~~~~~~~~~~~
src/lib.rs:102:28: 102:34 note: lint level defined here
src/lib.rs:102         unsigned_negation, unused, unused_allocation, unused_attributes, unused_comparisons,
                                          ^~~~~~
src/lib.rs:943:13: 943:25 error: variable does not need to be mutable
src/lib.rs:943         let mut data_map: datamap::DataMap;
                           ^~~~~~~~~~~~
src/lib.rs:102:28: 102:34 note: lint level defined here
src/lib.rs:102         unsigned_negation, unused, unused_allocation, unused_attributes, unused_comparisons,
                                          ^~~~~~
src/lib.rs:989:13: 989:25 error: variable does not need to be mutable
src/lib.rs:989         let mut data_map: datamap::DataMap;
                           ^~~~~~~~~~~~
src/lib.rs:102:28: 102:34 note: lint level defined here
src/lib.rs:102         unsigned_negation, unused, unused_allocation, unused_attributes, unused_comparisons,
                                          ^~~~~~
error: aborting due to 11 previous errors
Could not compile `self_encryption`.