Port validation mechanics from private repo about ngen-cal HOT 3 OPEN

Having a quick glance at validation_run.py, this is something that the plugin system was developed for.

from ngen-cal.

@SnowHydrology resurfaced the desire for this feature. Without thinking about it deeply, i'm not sure if it makes more sense to implement this as a core ngen.cal feature or implement this as a plugin that ngen.cal maintains.

Below are the key features @SnowHydrology is asking for:

• specify a validation simulation time period. This is different from the time period used for calibration.
• configure how many of the top n performing parameter sets from calibration to run as validation runs.
• Compute and report performance metrics for the validation parameter sets.

from ngen-cal.

If we are to go the plugin route, I think we should consider exposing "helper" functions to:

• modify realization configs with updated parameter values
• potentially compute performance metrics (on the fence about this)
• consider how we want to report results (ideally we could standardize that, but likely far too early to get it right)

from ngen-cal.