Debug self-encryption of small files about self_encryption HOT 7 CLOSED

@RolandSherwin how exactly do you envision this to work, when you say small files what's the range of sizes you actually mean here? is it between 3 * MIN_CHUNK_SIZE and 3 * MAX_CHUNK_SIZE ? if so, for a file of size let's say 2 * MAX_CHUNK_SIZE we'll need to generate datamap + 2 chunks, right? and that'd be a different mechanism as SE needs at least three chunks to work...

from self_encryption.

I don't know if this is a good place to raise this, but it would be nice to have an API for self encrypting a block of memory. Currently it only allows self encryption of a file on disk and its not good to force data to be written to disk just for this purpose.

from self_encryption.

Hey @bochaco. Right, I see! Also, we were having troubles when actually trying to upload those extra chunks. We fetched store cost + paid + sent over chunk+payment, but they were never accepted.
I can try to reproduce it locally.

from self_encryption.

I can try to reproduce it locally.

These may be two separate bugs.

Firstly uploading individual small files does work (ie safe files upload -p <FILE>).

But uploading a directory (ie safe files upload -p <DIRECTORY>) fails with payments never being enough, and wallet gets drained (logs posted on forum for Qi here).

from self_encryption.

After digging in, the issue with small files ending up in 1 chunk (plus the data-map chunk) is due to the fact that files that are smaller than 3 * MAX_CHUNK_SIZE are split into 3 chunks, thus if the content of the file is by chance something like "hello\nhello\nhello\n" it ends up being split into three equal chunks: "hello\n".
So that seems all ok to me, at least, as long as we are ok with dealing that way with small files as mentioned above.

from self_encryption.

@bochaco That sounds like the issue riddim isolated as he was just adding the same text to a file to see what happened to the upload cost.

It doesn't explain the issue I was experiencing and reported to Qi Ma. Is anyone looking into that?

from self_encryption.