RFC8823: acme4j response does not match CA expectation about acme4j HOT 8 CLOSED

First of all, thank you for the detailed explanation! It was really helpful for reproducing the problem.

Actually, I have anticipated this issue...

According to RFC-8823, the CA needs to generate two tokens (Token Part 1 and Token Part 2). Unfortunately the RFC is not very clear about how the two parts are then supposed to be concatenated by the client. Section 3 only says:

6. The ACME client concatenates "token-part1" (received over email) and "token-part2" (received over HTTPS [RFC2818]) to create the ACME "token" and calculates keyAuthorization (as per Section 8.1 of RFC8555).

There are two possible ways of concatenation.

• Both token-parts are base64-decoded and the byte arrays are concatenated. The result is then base64url-encoded again, in order to compute the key-authorization.

• Both token-parts are just string-concatenated, and the result is then used for key-authorization.

Both ways give different results, because of the padding bits that may be appended by base64 encoding. Sadly the RFC gives no concrete example of a concatenation so one could compare the own implementation with an expected result.

I had already raised this question in the ACME mailing list in June: https://mailarchive.ietf.org/arch/msg/acme/mOtPJpJHmmJeuzGBIaHuDcX8LBI/

Brian Sipos then gave good arguments for the string concatenation, and deemed the RFC-8823 text sufficient to explain this: https://mailarchive.ietf.org/arch/msg/acme/suw60485gKpcP9Cx5tl9gE1SQQI/

I got no other answers that were relevant. For this reason, acme4j uses string concatenation for computing the key-authorization value. Assuming that, like in your example:

token 1    = "5T6P7VjzlZO_LkBgHYBwpQmYsnXkPpZTikh_DkJAMM0"
token 2    = "l0Euldt78kNFEmb5K3k-rv4fGsSgvBBlDxUQO_IbmFk"
thumbprint = "gAJVFtivkss40JqEsHy2v6r6oxd8xrkVaF3Z1VzkpIo"

The string concatenation gives:

string-concat = "5T6P7VjzlZO_LkBgHYBwpQmYsnXkPpZTikh_DkJAMM0l0Euldt78kNFEmb5K3k-rv4fGsSgvBBlDxUQO_IbmFk"
key-auth      = "5T6P7VjzlZO_LkBgHYBwpQmYsnXkPpZTikh_DkJAMM0l0Euldt78kNFEmb5K3k-rv4fGsSgvBBlDxUQO_IbmFk.gAJVFtivkss40JqEsHy2v6r6oxd8xrkVaF3Z1VzkpIo"

response      = "OLzGVBMykMOx0F8OSZaf94jXpFctHswLUzbTTqVF1cM"
              = d4b4a416dd99c00d4948ec6a50a59bc28e44d96ba8a589c7d0e4a348720f99cc

This is what acme4j gives as response to your CA. Your CA however seems to expect a byte array concatenation:

array-concat = "5T6P7VjzlZO_LkBgHYBwpQmYsnXkPpZTikh_DkJAMM2XQS6V23vyQ0USZvkreT6u_h8axKC8EGUPFRA78huYWQ"
key-auth     = "5T6P7VjzlZO_LkBgHYBwpQmYsnXkPpZTikh_DkJAMM2XQS6V23vyQ0USZvkreT6u_h8axKC8EGUPFRA78huYWQ.gAJVFtivkss40JqEsHy2v6r6oxd8xrkVaF3Z1VzkpIo"

response     = "DZsxvV81Xp6D9DoBIV7QXiIhgve03G1l9bR3DAQILOM"
             = 0d9b31bd5f355e9e83f43a01215ed05e222182f7b4dc6d65f5b4770c04082ce3

Note how string-concat and array-concat are different. This is the only reason for the different response. If acme4j were to use byte array concatenation, it would give the result that is expected by your CA.

It is a difficult situation now, because in my oppinion it is unclear whether your CA or acme4j implements that RFC incorrectly. There are good arguments on both sides, and personally I would even favor byte array concatenation. But the RFC itself (sadly) gives no further clues.

I have just contacted the author of RFC-8823 and asked him for clarification. I will report back as soon as I got an answer.

from acme4j.

S/MIME is supported since acme4j v2.12. You need to add the acme4j-smime module as dependency though. It is a separate module because it uses javax.mail and thus requires a JavaMail implementation in the classpath.

The acme4j-smime module provides EmailIdentifier as email identifier, and also supplies helper methods for parsing challenge emails and generating response emails. Please note that S/MIME support is experimental at the moment.

More about S/MIME support is documented here: https://shredzone.org/maven/acme4j/challenge/email-reply-00.html

Thank you for the test server link. It supports both S/MIME and STAR, so I can use it to end the experimental state of both extensions, finally.

I'm closing the issue because it seems that everything you need is already present. If something is missing, feel free to reopen it.

from acme4j.

@shred:

acme4j does not seem to be compatible with the latest development version of
the CA software 🤷 .

There seems to be a disagreement about the SHA256 hash value of the ACME reply
being generated. Padding the keyAuthorization with == didn't solve the
mismatch, either. What is the input of the hash sent by acme4j?

Public key generated by acme4j as seen by CA:

-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEaV3pFhaa1RbDkbcDjWkoWytKvuo74sup
TEvGHA8yursjIxZovBHX+sK79Whv/n+fbSSfcsPc3zZHkzkzIYndW2RK+pIPyau8
vAHi1k9BaUj7RC4x94BHIuWule1/fcVU
-----END PUBLIC KEY-----

Creation of an email challenge email-reply-00 by CA:

hex(token-part2): 97412e95db7bf243451266f92b793eaefe1f1ac4a0bc10650f15103bf21b9859
hex(token-part1): e53e8fed58f39593bf2e40601d8070a50998b275e43e96538a487f0e424030cd
thumbprint(pubkey): gAJVFtivkss40JqEsHy2v6r6oxd8xrkVaF3Z1VzkpIo
keyAuthorization: 5T6P7VjzlZO_LkBgHYBwpQmYsnXkPpZTikh_DkJAMM2XQS6V23vyQ0USZvkreT6u_h8axKC8EGUPFRA78huYWQ.gAJVFtivkss40JqEsHy2v6r6oxd8xrkVaF3Z1VzkpIo

CA outgoing email subject:

Subject: ACME: 5T6P7VjzlZO_LkBgHYBwpQmYsnXkPpZTikh_DkJAMM0

Reply by acme4j:

-----BEGIN ACME RESPONSE-----
OLzGVBMykMOx0F8OSZaf94jXpFctHswLUzbTTqVF1cM
-----END ACME RESPONSE-----

Computed hex from response:

> base64 -d | xxd
OLzGVBMykMOx0F8OSZaf94jXpFctHswLUzbTTqVF1cM==
00000000: 38bc c654 1332 90c3 b1d0 5f0e 4996 9ff7  8..T.2...._.I...
00000010: 88d7 a457 2d1e cc0b 5336 d34e a545 d5c3  ...W-...S6.N.E..

Expected reply by CA:

hex(expected reply): 0d9b31bd5f355e9e83f43a01215ed05e222182f7b4dc6d65f5b4770c04082ce3

The testcode used can be found here:
https://gitlab.com/platynum/certification-authority/flows/-/blob/development/test/acme4j/src/test/java/SmimeTest.java

Full trace of the failed certificate generation can be found here:
https://gitlab.com/platynum/certification-authority/flows/-/jobs/1505052777#L3075

from acme4j.

Thanks for your in depth explanation and the link to the threads regarding this topic. To me this is confusing as well, and I tend towards the array-concat method, because of this in RFC8823 3.1:

   The Subject header field has the following syntax: "ACME: <token-
   part1>", where the prefix "ACME:" is followed by folding white
   space (FWS; see [RFC5322]) and then by <token-part1>, which is
   the base64url-encoded first part of the ACME token that MUST be
   at least 128 bits long after decoding.

This says, that the decoded <token-part1> has to be used as the first part of the token used for the keyAuthorization. Although, this does not explain, how <token-part2> has to be concatenated. I assume, that mixing a binary <token-part1> with a stringish <token-part2> is not intended.

from acme4j.

As Brian Sipos said in his response:

[...] RFC 8823 does not make any requirements about the content of the "token-part2" text value. The fact that the example looks like base64url encoding implies that, but I don't see any requirement on the token-part2 generation other than its minimum entropy.

So actually, <token-part2> could be anything, and mixing a base64url encoded <token-part1> with a stringish <token-part2> might be the intention.

Well, I guess we can agree that RFC8823 leaves a lot of room for interpretation, as we are both (independently) not certain about how to do proper concatenation. 😃 I hope that Alexey Melnikov (as the author of the RFC) can shed some light on it.

from acme4j.

Alexey Melnikov has answered in the mailing list: https://mailarchive.ietf.org/arch/msg/acme/KusfZm3qC50IfcAAuTXtmbFK0KM/

I can confirm that my intent was to suggest straight string concatenation without any base64url-decode and re-encoding.

This means that acme4j handles the challenge according to the intention of the RFC author, and (sorry to say that) the bug is on the CA side. 😉

However, I am glad that we can close this issue with certainty.

from acme4j.

This is not a bug on the CA side, but an interoperability issue caused by an ambigous / incomplete specification. Whatever ... The server side implementation has been tweaked for maximum interoperability. It was able to generate a certificate using acme4j as a client: https://gitlab.com/platynum/certification-authority/flows/-/jobs/1510875646#L5488

This issue can be closed now.

PS Using Order.autoRenewal() and Order.getAutoRenewalCertificate() seem to be necessary.

from acme4j.

I agree. But it seems that in the ACME mailing list no one sees a need to further clarify that problem, e.g. in an RFC8823 errata. I am quite sure that I will be having this discussion again some day.

Anyway, thank you for your report!

from acme4j.