[BUG] v3007.0 broken multi-master authentication when connecting with older minions about salt HOT 7 OPEN

Aditionally to the situation of the master being on 3007 and the minion on 3005, the opposite is also a problem:

I have a master being on 3005.5 with the minion being upgraded to 3007.0. The same problem is occuring that the key is not recognized anymore, and the recommended solution of deleting it and restarting the minion does not help either.
Error is consistently Unable to sign_in to master: Invalid master key.

Having a closer look at the data passing through the function, it seems that the cleaned key for the master has an extra \n linebreak added to the end which gets removed by the clean function. At this point, the comparision fails.
The minion key however does not have an added newline at the end of the file and does not suffer from this error.

Again, making the clean_key() function a noop by returning the passsed key variable unchanged resolves this problem but is only a workaround.

from salt.

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

from salt.

this looks related to #66126

from salt.

Thanks @whytewolf for the reply. This fix to 3007.x seems promising https://github.com/saltstack/salt/pull/66161/files - I'll test it shortly.

from salt.

Unfortunately, the above fix (#66161) won't address our case where the clean_key() on the master "messes up" the master's signature verification on older minions which don't have this function in their code base. As far as I can see, the only way to move forward is either to upgrade all the minions to the version that introduced this new code at the same time as upgrading the master(s) (which is not easily feasible for saltenv's with 100k+ nodes), or apply the patch suggested in the description of this bug case.

from salt.

Aditionally to the situation of the master being on 3007 and the minion on 3005, the opposite is also a problem:

I have a master being on 3005.5 with the minion being upgraded to 3007.0. The same problem is occuring that the key is not recognized anymore, and the recommended solution of deleting it and restarting the minion does not help either. Error is consistently Unable to sign_in to master: Invalid master key.

Having a closer look at the data passing through the function, it seems that the cleaned key for the master has an extra \n linebreak added to the end which gets removed by the clean function. At this point, the comparision fails. The minion key however does not have an added newline at the end of the file and does not suffer from this error.

Again, making the clean_key() function a noop by returning the passsed key variable unchanged resolves this problem but is only a workaround.

Yes, that's exactly what I have found as well. We have no need for use of the clean_key() function in our environment, it only "breaks" the signature validation.

from salt.

Aditionally to the situation of the master being on 3007 and the minion on 3005, the opposite is also a problem:

I have a master being on 3005.5 with the minion being upgraded to 3007.0. The same problem is occuring that the key is not recognized anymore, and the recommended solution of deleting it and restarting the minion does not help either. Error is consistently Unable to sign_in to master: Invalid master key.

Having a closer look at the data passing through the function, it seems that the cleaned key for the master has an extra \n linebreak added to the end which gets removed by the clean function. At this point, the comparision fails. The minion key however does not have an added newline at the end of the file and does not suffer from this error.

Again, making the clean_key() function a noop by returning the passsed key variable unchanged resolves this problem but is only a workaround.

the master being a lower version then the minion has never and will never be a supported setup.

see the first paragraph of https://docs.saltproject.io/salt/install-guide/en/latest/topics/upgrade.html

so while we are working on the fix for this in cases where the master is higher version. we will not do so for in cases where the minion is higher version. if the fix ends up working in such cases. great, but it is not a goal.

from salt.