Integrate new API call with plugin configuration reloading. about spire HOT 9 CLOSED

Thank you @edwbuck for opening this issue.
It seems that the mechanism using the plugin_data_file setting with a SIGUSR1 signal is not a good fit for your use case. Could you please provide a short description of your use case so we can better understand the need for this RPC? Thanks!

from spire.

The use case is that in addition to reloading plugins on disk, there is a desire to expose the plugin configuration to an API.

This permits one to perform an API call, from a remote system, to change the plugin configuration. The end result for success would look like:

1. The plugin received the new configuration data and accepted it.
2. The plugin new configuration data is on disk so the system, should it restart, restarts with the new configuration data.

And in a failure scenario:

1. For one reason or another the API call fails (bad config, unchangeable setting, etc.)
2. The plugin data on disk is not updated, so the system won't restart with a bad configuration.

Ideally, only plugins that have been reconfigured to use plugin_data_file would be impacted.

the SIGUSR1 proposal that was put forward by sorin was woefully out-of-date, but the push to make reconfiguration possible was welcomed. It is now a matter of exposing the reconfiguration approach to an API so external management tools can alter plugin configurations without launching a shell and sending a SIGUSR1.

This is not a proposal to remove SIGUSR1, as users that lack company-specific management tooling should still enjoy the benefits of reconfiguration.

from spire.

Thank you Edwin for the details. Maybe @sorindumitru can provide some more context for this request? Particularly, what's the main obstacle to use the SIGUSR1 approach in your use case?

from spire.

My need for this would be to get some feedback from the reload process to see if the configuration is ok or not, to allow for some kind of automated recovery (e.g. roll back configuration to previous config).

At the moment I can alarm based on the log file, but reacting to the error in an automated way is a bit error prone.

from spire.

@amartinezfayo

Can you provide some more context for this request?

Yes, the context is that after asking SPIRE to reconfigure, the config files ideally should be in the state that permits SPIRE server to relaunch in the same way it was running when it shutdown.

Under the current SIGUSR1 approach, monitoring and applying these configuration changes involves many different steps, all which must be coordinated:

1. Backup the file
2. Write the file
3. Send SIGUSR1
4. In the Catalog component, sent the configuration to the plugin.
5. In the Catalog component, see if an error response was returned by the plugin. (which will be logged as a side-effect).
6. Monitor the log (to detect the side effect of failed plugin reconfiguration).
7. If the configuration failed, restore the backup.

Steps 1, 2, and 7 require a sysadmin (or tooling) to run the file commands "cp", "cat", and "mv".
Steps 3 require a sysadmin to send the "kill" command
Steps 4 and 5 are internal to SPIRE
Step 6 requires a scanner to scan the log files, and that the SPIRE system is configured with log levels high enough to log the reconfiguration in the first place.

This means there are 4 or 5 command line calls, a need to build a log file parser, and a need to ensure that SPIRE is configured to emit the necessary logged messages. A failure in any one of these items fails to set the system in a state where it is likely to startup with the same running configuration it had when it shutdown.

With an API as the entry point of the new configuration file, we can reorder this as:

1. Call the API.
2. In the Catalog component, send the configuration to the plugin.
3. In the Catalog component, see if an error response was returned by the plugin. (better than scanning log files)
4. If the plugin didn't raise an error, rewrite the plugin's plugin_data_file with the same string used in step #2.
5. The return error is sent back to the API caller.

Step 1 is done by an external system, or perhaps even a dedicated tool. It waits till it gets a GRPC response.
Step 2 and 3 are done by SPIRE internally, same as before. Additionally Step 4 is done by SPIRE, and Step 5 is done by SPIRE.

This means that there are fewer independent processes involved, and less need / worry to coordinate them. The config file is only changed AFTER the plugin is successfully reconfigured, meaning that there is a smaller opportunity for the config file and the running server to be different when the plugin fails to accept the new configuration.

Particularly, what's the main obstacle to use the SIGUSR1 approach in your use case?

The need to coordinate the steps above to restore the configuration file to the one that is running in memory.

from spire.

Thank you for the details.
We have discussed this request and we think that providing a way to validate a configuration of a plugin would be good to have. Instead of exposing a new API in the server/agent, this could be achieved through a command in the SPIRE Server CLI and SPIRE Agent CLI, that would output the result of the validation of the configuration of a specified plugin.
At the same time, we don't think that SPIRE should take care of writing/persisting configuration files, as managing configuration files is outside of the scope of SPIRE. We think that this handling should happen outside of SPIRE.

from spire.

Instead of exposing a new API in the server/agent, this could be achieved through a command in the SPIRE Server CLI and SPIRE Agent CLI, that would output the result of the validation of the configuration of a specified plugin.

This sounds reasonable, and will re-establish some aspects of the two-phase (verify and then apply) change pattern that was initially proposed. One remaining item is where to place the new configuration that is being verified. Ideally, it would not be overwriting the running configuration, because that would simply re-establish the issue that we are trying to avoid (overwriting the configuration file until it is applied).

As with any file placement strategy, there are many different patterns. Please consider how the new configuration files can rest on disk without overwriting the old. There are a few obvious choices, and probably a few that I didn't think to mention:

• Putting all the files of the new configuration into a different directory, and using a cli option to relocate the "verify" command to the new directory.
• Putting a prefix or suffix on the new files, and using a cli option to specify the matching "new configuration" files.
• Some other pattern

At the same time, we don't think that SPIRE should take care of writing/persisting configuration files, as managing configuration files is outside of the scope of SPIRE. We think that this handling should happen outside of SPIRE.

Thank you for this design decision. I'll keep it in mind for future efforts.

from spire.

One remaining item is where to place the new configuration that is being verified. Ideally, it would not be overwriting the running configuration, because that would simply re-establish the issue that we are trying to avoid (overwriting the configuration file until it is applied).

The SPIRE CLI command would just take the location of the configuration of a specific plugin that is being verified, and return the result of the verification. I think that is up to the user to choose the best strategy to store those configuration files that are being used for verification, and SPIRE shouldn't really depend on that strategy.

from spire.

Closing this issue in favor of #5253 and #5268.

from spire.