Test case:

  @Test
  public void shouldMergeHashesIfNeeded() {
    def changes = '''

---
  source: global.yaml
  set:
    some::hash:
      global_value: true

---
  source: myteam.yaml
  merge:
    - some::hash
  set:
    some::hash:
      myteam_value: false
'''

    def result = generateFromYaml(hierarchy, changes, 'myteam.yaml', [
        'global.yaml': '',
        'myteam.yaml': '',
        'myteam/stage.yaml': ''
    ])

    def expected = [
        'global.yaml': [:],
        'myteam.yaml': [
            'some::hash': [
                'global_value': true,
                'myteam_value': false
            ]
        ],
        'myteam/stage.yaml': ''
    ]

    assert result == expected
  }

This will help interaction with YAML files that are partially human edited, and partially managed by monarch.

For example, the YAML might look like:

# human comments
human::managed::key: value

# --- Begin generated by monarch
monarch::managed::key: value
# --- End generated by monarch

# more human comments
another::human::managed::key: value

Should just treat these as empty and not care

Right now we blow up if a source we are looking for is found in multiple places in the hierarchy.

Essentially, like hiera proper:

---
:hierarchy:
  - "nodes/%{trusted.certname}"
  - "environment/%{server_facts.environment}"
  - "virtual/%{::is_virtual}"
  - common

So in a change:

# when no source: ... try facts instead
facts:
  trusted.certname: fake.com
  server_facts.environment: production
  ::is_virtual: true
set:
  etc: values here

It should merge the hashes in the lower source, then "unmerge" when targeting the higher source (or above)

Use case and some prior discussion can be found in #36

Basically, it would be useful to be able to programmatically modify a changeset.

That being said, without a YAML parser and dumper that maintain comments and formatting, doing so will canonicalize the changeset document, sorting it, reformatting it, and stripping it of comments.

I looked internally and we have no meaningful comments in our changesets, so maybe this is not such a big deal. More discussion on this kind of thing here: #15

The check is intended to remove redundant values, but it does not check if it is there from multiple places or just one.

It's helpful to have examples of how to do things to get started from.

Already have a ton of stuff in hiera for a project, would be great to generate the changeset from that as a starting point. And maybe an example for how to cleanup duplicate keys by generate a changeset then generate the hiera data from the changeset.

Would maybe be convenient to do something like

---
  source: foo
  variables:
    version: 1.2.3
  set:
    myapp1::version: ${version}
    myapp2::version: ${version}
    myapp3::version: ${version}

But maybe not

This would be useful when you are creating a new data source and you maybe experimenting with moving keys around. You'd have to add the moved keys to remove blocks to clean up from the old data source. If this feature was added, you can say flag the change as "comprehensive" or something like that, and it was ignore existing data (maybe "ignore_existing" or "overwrite" are better flag names).

In case you want to apply to something low level directly and You Know What You Are Doing

Use case:

• Dev environment has ongoing keys being changed
• These changes are not reflected in a release changeset, and release is updating to an older version
• Want to able able to target parent in hierarchy put not change dev version

Had a random tab creep into a changeset and got the following error:
Exception in thread "main" while scanning for the next token
found character '\t(TAB)' that cannot start any token. (Do not use \t(TAB) for indentation)
in 'reader', line 19, column 1:

^

at org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:420)
at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:226)
at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:558)
at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
at org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:143)
at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:224)
at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:155)
at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:229)
at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:155)
at org.yaml.snakeyaml.composer.Composer.composeDocument(Composer.java:122)
at org.yaml.snakeyaml.composer.Composer.getNode(Composer.java:84)
at org.yaml.snakeyaml.constructor.BaseConstructor.getData(BaseConstructor.java:104)
at org.yaml.snakeyaml.Yaml$1.next(Yaml.java:471)
at io.github.alechenninger.monarch.YamlMonarchParser.parseChanges(YamlMonarchParser.java:53)
at io.github.alechenninger.monarch.MonarchOptionsFromInputs.lambda$changes$5(MonarchOptionsFromInputs.java:73)
at java.util.Optional.map(Optional.java:215)
at io.github.alechenninger.monarch.MonarchOptionsFromInputs.changes(MonarchOptionsFromInputs.java:69)
at io.github.alechenninger.monarch.OverridableOptions$1.<init>(OverridableOptions.java:54)
at io.github.alechenninger.monarch.OverridableOptions.lambda$changes$11(OverridableOptions.java:53)
at io.github.alechenninger.monarch.Main.run(Main.java:89)
at io.github.alechenninger.monarch.Main.main(Main.java:179)

The tab was at the beginning of a blank line. I could see it being expected behavior but was definitely annoying. Could at least throw a better exception message (even just including the offending filename would be a big help)

Somehow this didn't actually turn out to be a huge problem yet :D

I'm guessing that yaml.loadAll (from org.yaml.snakeyaml.Yaml) is parsing it as 'null' for some reason, meaning this could affect any yaml file that is parsed throughout the app.

Exception in thread "main" java.lang.NullPointerException
at io.github.alechenninger.monarch.Change.fromMap(Change.java:41)
at io.github.alechenninger.monarch.YamlMonarchParser.parseChanges(YamlMonarchParser.java:55)
at io.github.alechenninger.monarch.MonarchOptionsFromInputs.lambda$changes$5(MonarchOptionsFromInputs.java:73)
at java.util.Optional.map(Optional.java:215)
at io.github.alechenninger.monarch.MonarchOptionsFromInputs.changes(MonarchOptionsFromInputs.java:69)
at io.github.alechenninger.monarch.OverridableOptions$1.(OverridableOptions.java:54)
at io.github.alechenninger.monarch.OverridableOptions.lambda$changes$11(OverridableOptions.java:53)
at io.github.alechenninger.monarch.Main.run(Main.java:89)
at io.github.alechenninger.monarch.Main.main(Main.java:179)

It would also be nice to have multiple root hierarchies in a single config, e.g.:
config.yaml
hierarchy:

• group/A.yaml
  • group/A/dev.yaml
  • group/A/prod.yaml
• group/B.yaml
  • group/B/dev.yaml
  • group/B/prod.yaml

Then you could have multiple change sets that could work off of group/A.yaml or group/B.yaml without having to specify multiple config files. Trying to have multiple roots now gives me a NullPointerException:
Exception in thread "main" java.lang.NullPointerException
at io.github.alechenninger.monarch.MonarchParsers.getExtensionForFileName(MonarchParsers.java:44)
at io.github.alechenninger.monarch.Main.readDataForHierarchy(Main.java:131)
at io.github.alechenninger.monarch.Main.run(Main.java:100)
at io.github.alechenninger.monarch.Main.main(Main.java:179)

We could include when the data source was last generated, and what changeset was used, or even a history of changesets used

You can specify a source in your changeset that is not in the hierarchy. I dont think it is trying to apply the changeset, but at the very least it is still rewriting your hieradata output in alphabetical order.

Should probably verify that the source is declared in your hierarchy and fail loudly.

Not everybody using the hiera data will be looking at monarch files. While it's good to have comments in the changeset we loose the ability to comment the hiera yaml. This can help folks understand why something was set etc.

...since cli parser natively supports multi arg values with spaces (and config arg uses this)

For example, imagine a dynamic hierarchy like this:

hierarchy:
  sources:
    - common.yaml
    - environment/%{environment}.yaml
    - team/%{team}.yaml
    - team/%{team}/%{environment}.yaml
    - node/%{hostname}.yaml
  potentials:
    # snip...
    hostname:
      - myhost.prod.mycompany.com:
        environment: prod
        team: myteam

Then, if you defined a target by hostname=myhost.prod.mycompany.com, you would have a complete hierarchy, without having to manually define environment and team for that node too, which you already know statically in advance.

Additionally, the reverse is true. If you targeted environment=prod or team=myteam, you would know that host was a descendant.

This should work:

hierarchy:
  foo:
    - bar
    - baz
mergeKeys:
  - key1
  - key2

Right now this only works if you indicate the hierarchy and mergeKey values are strings (using >)

...in such a way that ignores the rest of the hash and only changes or adds the key in that hash.

Target is a more intuitive name

I think this happens when:

hierarchy:
  sources:
    - %{foo}.yaml
  potentials:
    foo:
      -

So you can put team specific config in one file, and user specific config in another

Otherwise it's a big mess

Something like .monarch-config.yaml

Look at other projects for conventions

...in such a way that doesn't remove anything else about the hash; only that key in the hash.