Add APIs to make NativeMacLayeredInputStream more secure about conceal HOT 4 CLOSED

I'm a strong believer in that the API should do the right thing and I think you have some good points here. I'm aware of this and this is definitely a limitation of any streaming API. This is the case because we've made a decision to use the least amount of memory possible in the API, so holding all the data in memory is not an option for us. The second option suggested which is to invent our own cryptographic construction (the notion of a block), which we'd rather not do just because there are a lot of tricky problems to deal with here, for example we need to make sure that the blocks form a hash mac chain, otherwise you can rearrange parts of the stream and still have the stream's mac be valid. We'd rather use publicly vetted crypto algorithms here, and might replace HMAC in the future. However, I agree we should definitely change the documentation to make that more clear.

We've decided to keep the InputStream abstraction because Java programmers love dealing and passing around streams rather than passing around byte arrays. This makes the API more compatible with other libraries because streams are so general, and it is the norm rather than the exception that reading would happen at several layers of abstraction higher than the crypto library just because Java. I think this is a very useful API if you consider that one of the most common ways streams are read in java is:

ByteArrayOutputStream bout = new ByteArrayOutputStream();
while ((read = in.read(buffer) == -1)) {
   bout.write(buffer, 0, read);
}

We're planning to add some features to the Crypto api in the near future which will alleviate these concerns for people who don't mind using it, and don't have these deep abstraction layers:

Crypto {
   get(x)InputStream(byte[] buffer);
   get(x)InputStream(File file);
}

Which makes the Crypto lib own the writing to a buffer and a file and only passes back the buffer or File back only if it's valid which would I think address your concern.

I'm going to mark this as a enhancement request. You're welcome to submit a pull request if you're passionate about working on this.

from conceal.

What restrictions do we have on the format of the stream? I would like to propose modifying it to look like this:

block[0] || MAC(0 || block[0]) || block[1] || MAC(1 || block[1]) || ... || block[n] || MAC(n || block[n])

where all blocks but the last have a preselected, constant size. This way we can read a block at a time and verify it before returning any data from it. If that's acceptable, I will look for a well established way of doing something like this (I think this is secure by a simple application of MAC's security guarantees, but I may very well be missing something) and submit a pull request.

EDIT: Sorry, should have replied when I've really woken up. I will try to find a sufficiently vetted protocol that does something similar and get back if i do find it. If that wouldn't be convincing enough for you anyway, please say so.

from conceal.

I think what would have a huge impact on this project would be the implementation of the abstraction that I mentioned before:

Crypto {
   get(x)InputStream(byte[] buffer);
   get(x)InputStream(File file);
}

I'd rather not re-invent crypto constructions precisely because it's really difficult to get them right. For example in the suggested construction, let's say the attacker sees 2 messages:

m1 = b01 || b11
m2 = b02 || b12
the tags for these two messages would be (effectively)
t1 = Mac(0 || b01) || Mac(1 || b11)
t2 = Mac(0 || b02) || Mac(1 || b12)

an attacker could then guess a mac on a message m3
m3 = b01 || b12
t3 = Mac(0 || b01) || Mac (1 || b12)
breaking the mac security guarantee

I love that you're thinking of new schemes to make this library better. The point of this proof is not to shoot down your idea, but to stress the importance of peer review in security constructions.

If you would like to work on the other idea that I mentioned let me know 👍

from conceal.

I've changed the title of the issue to be less alarmist and to reflect the current state of the discussion.

from conceal.