[Research] Improve data source linking for Inventory/Metric threshold/Log threshold rules about kibana HOT 8 CLOSED

Option 3 - Is it possible to set this programatically (via the API)? If the value was known it can be reconciled and enforced by some external mechanism.

from kibana.

@neiljbrookes yes and no, these are saved objects and there are mechanisms in Kibana for retrieving and overwriting saved objects, so this is possible in a way. If we allow for reading/writing this value via API in any way, though, would it prevent the type of outage/incident that triggered this latest investigation? My sense is that the UI would still be open to a user unknowingly changing the source setting without realizing they had just changed the data source for all running rules, which is what we want to prevent.

Can you elaborate on how you see API access preventing this type of incident (we want to expose all of our settings via API, so we'll aim to make these kinds of things possible either way, eventually)?

from kibana.

UPDATE: I spoke with @vinaychandrasekhar and after talking through the options, he would like us to start by implementing Option 1 from this research above. If possible, he also suggested looking into displaying the currently selected index pattern on the RULE DETAILS page, along with a link to the settings page where it can be updated. @benakansara can you create an issue that outlines that work? Let me know if you need more clarity and we can have a call tomorrow. Sorry for the back-and-forth confusion!

After a few rounds of conversation I think we're settling on "Option 2" here, providing some kind of checkbox to "lock this rule to a specific index pattern". I don't think we need to support data views here, since we don't support them in the metrics/infrastructure UI settings currently, and that retains one of the advantages of migrating to "Custom Threshold", too.

@benakansara can you create a new issue referencing this one, that lays out the work for locking in this setting for the metric threshold rule? TBD: if we should do the same for other rule types, folks can discuss that part in the new ticket, I think.

from kibana.

@jasonrhodes , thanks for the details.

You are right, my Option 3 would still be open to the situation where a user can change the setting in the UI, and we then (eventually) reset it to a known value. This isn't the best solution to prevent it, unless we can also disable the ability to change it in the UI.

Ideally all settings will be defined in code and applied via API calls....without any possibility of user intervention in the UI ! We manage 100's of kibanas, and keeping them consistent is challenging.

from kibana.

I think choosing option 1 is a good idea especially, since in option 2, removing an index from the setting will make the View in app URL for the log threshold confusing, as we only pass the time here with the assumption that the indices for the rule and setting are the same.

from kibana.

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

from kibana.

I discussed this with @jasonrhodes . Option 1 is good but is a "band-aid" temporary solution.

Option 2 where users start with a default rule definition setting that ties to the Discover or whatever data source setting but allows the user to change would be more useful. In addition to / instead of a warning in option 1 above, here could we offer users a choice when they change the data source setting in Discover / App, to ask them whether they want to reflect that change in the associated alerting rules, or would they like to sever the tie-in (we only show this if we see "connected" rules and don't show it otherwise). We should chat about this longer term solution live and come up with a plan / approach.

from kibana.

Decision has been made for the next release and the issue has been created, so I'll close this Research ticket. Thanks for your feedback, everyone.

from kibana.