adobe / himl Goto Github PK

https://github.com/adobe/himl/blob/master/himl/config_generator.py#L204
os.chdir(self.cwd) should be used before the for loop where os.chdir(path) is used.

I made this change in local and it works.

Consider a config folder structure as below

configs
├─ prod
│   |─ zone1.yaml
│   └─ zone2.yaml
└─ default.yaml

default.yaml contains

path:
    parent: data
    result_path: "{{path.parent}}/{{zone.name}}/{{path.exp_name}}"

prod/zone1.yaml

zone:
   name: zone_A
path:
    exp_name : exp1

prod/zone2.yaml

zone:
   name: zone_B
path:
    exp_name : test
run:
    description: "testing himl"

code snippet:

from himl import ConfigProcessor
config = ConfigProcessor().process(path='configs/prod', print_data=True)

Expected Behaviour

default.yaml has default configurations that will be overridden by any of the selected zone configurations. An error to ask to chose between zone1.yaml and zone2.yaml files.

Actual Behaviour

the code resulted in merging the files by itself and provides the following as output.

path:
  exp_name: test
  parent: data
  result_path: data/zone_B/test
zone:
  name: zone_B
run:
  description: testing himl

Platform and Version

Platform: Ubuntu 20.04 LTS
himl version: v0.7.1

Expected Behaviour

Optionally install himl with all extras dependencies: vault and aws interpolation

Actual Behaviour

Need to additionally install either with himl[vault] or install himl followed by boto3 and hvac.

Reproduce Scenario (including but not limited to)

Steps to Reproduce

Test 1

mkdir himl-test
pipenv install himl
echo "from himl import ConfigProcessor" > "test.py"
pipenv run python3 test.py

Traceback (most recent call last): File "/Users/uparnva/gitlab/pearsontechnology/gpt/baybridge/pdp/temp/himl-test/test.py", line 1, in <module> from himl import ConfigProcessor File "/Users/uparnva/.local/share/virtualenvs/temp-8eaOm6yW/lib/python3.10/site-packages/himl/__init__.py", line 11, in <module> from .config_generator import ConfigGenerator, ConfigProcessor File "/Users/uparnva/.local/share/virtualenvs/temp-8eaOm6yW/lib/python3.10/site-packages/himl/config_generator.py", line 20, in <module> from .interpolation import InterpolationResolver, EscapingResolver, InterpolationValidator, SecretResolver, \ File "/Users/uparnva/.local/share/virtualenvs/temp-8eaOm6yW/lib/python3.10/site-packages/himl/interpolation.py", line 14, in <module> from .inject_secrets import SecretInjector File "/Users/uparnva/.local/share/virtualenvs/temp-8eaOm6yW/lib/python3.10/site-packages/himl/inject_secrets.py", line 12, in <module> from .secret_resolvers import AggregatedSecretResolver File "/Users/uparnva/.local/share/virtualenvs/temp-8eaOm6yW/lib/python3.10/site-packages/himl/secret_resolvers.py", line 13, in <module> from .simplessm import SimpleSSM File "/Users/uparnva/.local/share/virtualenvs/temp-8eaOm6yW/lib/python3.10/site-packages/himl/simplessm.py", line 11, in <module> import boto3 ModuleNotFoundError: No module named 'boto3'

pipenv install boto3
pipenv run python3 test.py

Traceback (most recent call last): File "/Users/uparnva/gitlab/pearsontechnology/gpt/baybridge/pdp/temp/himl-test/test.py", line 1, in <module> from himl import ConfigProcessor File "/Users/uparnva/.local/share/virtualenvs/temp-8eaOm6yW/lib/python3.10/site-packages/himl/__init__.py", line 11, in <module> from .config_generator import ConfigGenerator, ConfigProcessor File "/Users/uparnva/.local/share/virtualenvs/temp-8eaOm6yW/lib/python3.10/site-packages/himl/config_generator.py", line 20, in <module> from .interpolation import InterpolationResolver, EscapingResolver, InterpolationValidator, SecretResolver, \ File "/Users/uparnva/.local/share/virtualenvs/temp-8eaOm6yW/lib/python3.10/site-packages/himl/interpolation.py", line 14, in <module> from .inject_secrets import SecretInjector File "/Users/uparnva/.local/share/virtualenvs/temp-8eaOm6yW/lib/python3.10/site-packages/himl/inject_secrets.py", line 12, in <module> from .secret_resolvers import AggregatedSecretResolver File "/Users/uparnva/.local/share/virtualenvs/temp-8eaOm6yW/lib/python3.10/site-packages/himl/secret_resolvers.py", line 15, in <module> from .simplevault import SimpleVault File "/Users/uparnva/.local/share/virtualenvs/temp-8eaOm6yW/lib/python3.10/site-packages/himl/simplevault.py", line 15, in <module> import hvac ModuleNotFoundError: No module named 'hvac'

pipenv install hvac
pipenv run python3 test.py

Success.

Test 2

pipenv --rm
pipenv install "himl[vault]"  
pipenv run python3 test.py

Success.

Platform and Version

Sample Code that illustrates the problem

Logs taken while reproducing problem

Expected Behaviour

test: 1
home_path: "{{env(HOME)}}"

=>

test: 1
home_path: /home/zadkiel

Actual Behaviour

test: 1
home_path: "{{env(HOME)}}"

=>

test: 1

Details

Retrieving values from the environment would be a great addition.
Currently it can be done with YQ after the values compilation but it does not support the hierarchical overrides that himl allows.

Consider the following .yaml file.

examples/simple/default.yaml

deep:
  key1: v1
  key2: v2
deep_list:
- item1
- item2

Request

A feature to filter deep keys like, deep.key1, deep.key2. Currently, the cli supports only top-level keys to filter. (ie, deep and deep_list in the example)

Expected Behaviour

When using dynamic data sources default merging strategies should be used.

Actual Behaviour

Method missing args:

TypeError: ConfigGenerator.merge_value() missing 3 required positional arguments: 'type_strategies', 'fallback_strategies', and 'type_conflict_strategies'

Expected Behaviour

secret: "{{vault.key/path/to/secret/key}}"

should output:

secret: "secret_content"

Actual Behaviour

secret: null

Details

It looks like this is a documentation error.
Tested with vault.key( instead of vault.key/

First of all, great work! We have been using himl for a while, and it's been great in helping us to manage our configuration needs.

One recent requirement is to allow lower level configuration overriding upper level for a list, instead of the default append behavior.

i.e.
examples/simple/default.yaml

list:
 - key: key1
 - value: value1 

examples/simple/region/cluster1.yaml

list:
 - key: key1
 - value: value2

so instead of merging the two, I'd like to allow cluster1.yaml to override default.yaml value, if the keys are the same.

the deep merge pkg allows passing custom function for different merging strategies, himl ConfigGenerator is building Merger object but doesn't allow custom function to pass in to override default behavior:

merger = Merger([(list, ["append"]), (dict, ["merge"])], ["override"], ["override"])

Would it be possible to refactor the ConfigProcessor and ConfigGenerator so that a custom func can be passed to the merger, thus allowing greater flexibility on merging behavior? Maybe ConfigProcessor can be passed an optional param to allow this?

Thanks, and keep up the great work!

Currently looking at how to integrate himl with piera (which I'm using extensively).

Hopefully I can sort it and can come back with the results.

https://github.com/b1naryth1ef/piera

It would be helpful if an efficient way to read the config (or process it) from S3 storage. boto3 client get the file ('s3://path/to/.yaml') as BytesIO which will not be merged with any default.yaml in the config folder.

For a mature hierarchical config it can become cumbersome to manage the all the nuances and keep track of where a config should be placed or what limitations some values are.
A fail early and validation mechanisms are highly beneficial and could prevent malformed or unintended configs to be used further in the pipeline, potentially avoiding misconfigurations.

Expected Behaviour

VAULT_SKIP_VERIFY=1 himl myvaultyaml.yaml

=> Just works

Actual Behaviour

requests.exceptions.SSLError: HTTPSConnectionPool(host='vault.xx.com', port=443): Max retries exceeded with url: /v1/auth/token/lookup-self (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1125)')))

Himl goes through each value and checks for "{{ interpolation.key }}".
If we want to pass the following value "{{ template custom.value }}" that can be used by an end application, interpolation will fail as custom.value is not found.

It would be nice to have a flag or way to explicitly say a value should not be interpolated.

The only workaround for this is be setting "skip_interpolation_validation=True".

In cases when stdout is collected (logging system, elk, splunk) an option to mask secret data on stdout would be useful.

Add support for secret interpolation from a vault path and namespace.

Himl cli should support an argument to output current installed version

On the stated example/simple/default.yaml trying to access key1 in deep.key1 results in empty dict.

examples/simple/default.yaml

deep:
  key1: v1
  key2: v2
deep_list:
- item1
- item2

Commands tried from himl directory

himl --filter deep.key1 examples/simple
himl --filter deep[key1] examples/simple
himl --filter deep{key1} examples/simple

Expected Behaviour

key1: v1

Actual Behaviour

{}

Platform and Version

Platform: Ubuntu 20.04 LTS

himl v0.7.1

Expected Behaviour

If I am setting a key/object in my top level globals.yaml at a specific lower level, I want to be able to remove that key (e.g. not just set the value: null but actually del). For example:

global.yaml

modules_to_deploy:
  module1: <config-mod1>
  module2: <config-mod2>
  module3: <config-mod3>

then in my special_dr_site.yaml:

modules_to_deploy:
  module2: #remove

After merging, for my special_dr_site, I want the resulting yaml to be:

modules_to_deploy:
  module1: <config-mod1>
  module3: <config-mod3>

Actual Behaviour

Not sure if/how this can be achieved?

Reproduce Scenario (including but not limited to)

simply try the simple example above.

Steps to Reproduce

see above example

Platform and Version

Any

These integrations https://github.com/adobe/himl#feature-secrets-retrieval depends on external packages which are not required for the basic operation of the package.

What do you think of moving these dependencies (boto, hvac...) to extra_requires ?
https://setuptools.readthedocs.io/en/latest/userguide/dependency_management.html#id7

Then users will be able to install the simple package with pip install himl and add functionnality with pip install himl[vault, aws]

Loader=yaml.SafeLoad --> Loader=yaml.SafeLoader

Expected Behaviour

Execute himl cli command in current directory as:

cd working-dir
himl . --output-file output.yaml

Output of the file should not be empty. No data is appended to the output file.

Actual Behaviour

cd working-dir
himl . --output-file output.yaml
cat output.yaml
{}

Output of the file is empty.
If I go up one folder and use the working-dir in path:

pwd
working-dir
cd ..
himl working-dir/ --output-file output.yaml

Data is being generated in the output.yaml

Platform and Version

MacOS Ventura: 13.4.1 (c)
himl v0.12.0 (installed via brew)

Python: 3.7.5
Himl: 0.5.0

Current yaml Library configuration used by HIML is treating multiline string dumps with a \n.
This can be improved by adding a scalar_representer to the BaseRepresenter.

eg.
For the following field:

testkey: |-
        # Set to true to log user information returned from LDAP
          verbose_logging = true

          [[servers]]
          # Ldap server host
          host = "someaddress"

          # Default port is 389 or 636 if use_ssl = true
          port = 389

          start_tls = true

will be dumped as:

testkey: "# Set to true to log user information returned from LDAP\n  verbose_logging\
        \ = true\n\n  [[servers]]\n  # Ldap server host\n  host = \"someaddress\"\n\
        \n  # Default port is 389 or 636 if use_ssl = true\n  port = 389\n\n  start_tls\
        \ = true"

Expected result:

testkey: |-
        # Set to true to log user information returned from LDAP
          verbose_logging = true

          [[servers]]
          # Ldap server host
          host = "someaddress"

          # Default port is 389 or 636 if use_ssl = true
          port = 389

          start_tls = true

I tested locally a version that fixes this, let me know if a PR is ok that would support this fix.

Expected Behaviour

vault:

/path/to/key:
  data: |
    -----BEGIN CERTIFICATE-----
    MIIC1TCCAb2gAwIBAgIJAIAI+nCNTUAuMA0GCSqGSIb3DQEBBQUAMBoxGDAWBgNV
    BAMTD3d3dy5leGFtcGxlLmNvbTAeFw0yMTA0MDkxNTIyMjlaFw0zMTA0MDcxNTIy
    MjlaMBoxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTCCASIwDQYJKoZIhvcNAQEB
    BQADggEPADCCAQoCggEBAK1klavp8ccE9es+8q2YQc9jRFmUIjwWXEzHQWXjoLQj
    ofqWOGP0XiUikx6Sj6USrgnIKbnt4mynn7awrh+yDm4q6rilqWurx7av/UVSTax9
    uMDB5Gf9hF87zHuS24I0EO7TKheRIdO8PLpLPe7a+SYylwmailLfnQZVPw/+Bovj
    2sa0fWRVy3qpvRoovPD77D9XrkS1LG6AaYFOmGboWL62rFB9xTy8vtfSIg+dHs2V
    NWVAVAIwqWMWY5E/WjBdDtZuCwpYl/eztoR6gjJqbI4g9lbpv/NCeAalMM8xKcQj
    /ZdI0j4H0IOVa+HNWwG+J46LDLV0AJcqKrd4PCEEJXUCAwEAAaMeMBwwGgYDVR0R
    BBMwEYIPd3d3LmV4YW1wbGUuY29tMA0GCSqGSIb3DQEBBQUAA4IBAQB5bdOZqNae
    7nH69lSbunttLHYRHpkDhS7gNagns3lskq8+sUfanbq6Jf0iSIbwY2UO8M9/vzHb
    OvnhwiXRHMMK1W5E9a3r/9lVApOzdnVlyM0OPiGuubaVYMD7v0dwhEq0eETdrr29
    veI1R/8XSNcJD6XuW5OgQ5gooOL6Ao7aogPIDW3FMwSxm/5nS5xgp+924xFlk8zg
    moyuFiM99eH9OzALcIj21B/iBCrw1eM+BRgbr/azdWortnhPQpQB2LH6AMB2oQn0
    E05HDs1ZJ7Y1ovSrwN5L57YAPvP+87z8OeUkFwgQnu10y3CBuSysmXaFI1N2iv/2
    S5ukJUcRP1RC
    -----END CERTIFICATE-----

input.yaml:

certificate: "{{vault.key(/path/to/key/data)}}"

$ himl input.yaml
certificate: | 
  -----BEGIN CERTIFICATE-----
  MIIC1TCCAb2gAwIBAgIJAIAI+nCNTUAuMA0GCSqGSIb3DQEBBQUAMBoxGDAWBgNV
  BAMTD3d3dy5leGFtcGxlLmNvbTAeFw0yMTA0MDkxNTIyMjlaFw0zMTA0MDcxNTIy
  MjlaMBoxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTCCASIwDQYJKoZIhvcNAQEB
  BQADggEPADCCAQoCggEBAK1klavp8ccE9es+8q2YQc9jRFmUIjwWXEzHQWXjoLQj
  ofqWOGP0XiUikx6Sj6USrgnIKbnt4mynn7awrh+yDm4q6rilqWurx7av/UVSTax9
  uMDB5Gf9hF87zHuS24I0EO7TKheRIdO8PLpLPe7a+SYylwmailLfnQZVPw/+Bovj
  2sa0fWRVy3qpvRoovPD77D9XrkS1LG6AaYFOmGboWL62rFB9xTy8vtfSIg+dHs2V
  NWVAVAIwqWMWY5E/WjBdDtZuCwpYl/eztoR6gjJqbI4g9lbpv/NCeAalMM8xKcQj
  /ZdI0j4H0IOVa+HNWwG+J46LDLV0AJcqKrd4PCEEJXUCAwEAAaMeMBwwGgYDVR0R
  BBMwEYIPd3d3LmV4YW1wbGUuY29tMA0GCSqGSIb3DQEBBQUAA4IBAQB5bdOZqNae
  7nH69lSbunttLHYRHpkDhS7gNagns3lskq8+sUfanbq6Jf0iSIbwY2UO8M9/vzHb
  OvnhwiXRHMMK1W5E9a3r/9lVApOzdnVlyM0OPiGuubaVYMD7v0dwhEq0eETdrr29
  veI1R/8XSNcJD6XuW5OgQ5gooOL6Ao7aogPIDW3FMwSxm/5nS5xgp+924xFlk8zg
  moyuFiM99eH9OzALcIj21B/iBCrw1eM+BRgbr/azdWortnhPQpQB2LH6AMB2oQn0
  E05HDs1ZJ7Y1ovSrwN5L57YAPvP+87z8OeUkFwgQnu10y3CBuSysmXaFI1N2iv/2
  S5ukJUcRP1RC
  -----END CERTIFICATE----- 

output:

$ cat output.yaml | yq -r .certificate
-----BEGIN CERTIFICATE-----
MIIC1TCCAb2gAwIBAgIJAIAI+nCNTUAuMA0GCSqGSIb3DQEBBQUAMBoxGDAWBgNV
BAMTD3d3dy5leGFtcGxlLmNvbTAeFw0yMTA0MDkxNTIyMjlaFw0zMTA0MDcxNTIy
MjlaMBoxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTCCASIwDQYJKoZIhvcNAQEB
BQADggEPADCCAQoCggEBAK1klavp8ccE9es+8q2YQc9jRFmUIjwWXEzHQWXjoLQj
ofqWOGP0XiUikx6Sj6USrgnIKbnt4mynn7awrh+yDm4q6rilqWurx7av/UVSTax9
uMDB5Gf9hF87zHuS24I0EO7TKheRIdO8PLpLPe7a+SYylwmailLfnQZVPw/+Bovj
2sa0fWRVy3qpvRoovPD77D9XrkS1LG6AaYFOmGboWL62rFB9xTy8vtfSIg+dHs2V
NWVAVAIwqWMWY5E/WjBdDtZuCwpYl/eztoR6gjJqbI4g9lbpv/NCeAalMM8xKcQj
/ZdI0j4H0IOVa+HNWwG+J46LDLV0AJcqKrd4PCEEJXUCAwEAAaMeMBwwGgYDVR0R
BBMwEYIPd3d3LmV4YW1wbGUuY29tMA0GCSqGSIb3DQEBBQUAA4IBAQB5bdOZqNae
7nH69lSbunttLHYRHpkDhS7gNagns3lskq8+sUfanbq6Jf0iSIbwY2UO8M9/vzHb
OvnhwiXRHMMK1W5E9a3r/9lVApOzdnVlyM0OPiGuubaVYMD7v0dwhEq0eETdrr29
veI1R/8XSNcJD6XuW5OgQ5gooOL6Ao7aogPIDW3FMwSxm/5nS5xgp+924xFlk8zg
moyuFiM99eH9OzALcIj21B/iBCrw1eM+BRgbr/azdWortnhPQpQB2LH6AMB2oQn0
E05HDs1ZJ7Y1ovSrwN5L57YAPvP+87z8OeUkFwgQnu10y3CBuSysmXaFI1N2iv/2
S5ukJUcRP1RC
-----END CERTIFICATE-----

Actual Behaviour

vault:

/path/to/key:
  data: |
    -----BEGIN CERTIFICATE-----
    MIIC1TCCAb2gAwIBAgIJAIAI+nCNTUAuMA0GCSqGSIb3DQEBBQUAMBoxGDAWBgNV
    BAMTD3d3dy5leGFtcGxlLmNvbTAeFw0yMTA0MDkxNTIyMjlaFw0zMTA0MDcxNTIy
    MjlaMBoxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTCCASIwDQYJKoZIhvcNAQEB
    BQADggEPADCCAQoCggEBAK1klavp8ccE9es+8q2YQc9jRFmUIjwWXEzHQWXjoLQj
    ofqWOGP0XiUikx6Sj6USrgnIKbnt4mynn7awrh+yDm4q6rilqWurx7av/UVSTax9
    uMDB5Gf9hF87zHuS24I0EO7TKheRIdO8PLpLPe7a+SYylwmailLfnQZVPw/+Bovj
    2sa0fWRVy3qpvRoovPD77D9XrkS1LG6AaYFOmGboWL62rFB9xTy8vtfSIg+dHs2V
    NWVAVAIwqWMWY5E/WjBdDtZuCwpYl/eztoR6gjJqbI4g9lbpv/NCeAalMM8xKcQj
    /ZdI0j4H0IOVa+HNWwG+J46LDLV0AJcqKrd4PCEEJXUCAwEAAaMeMBwwGgYDVR0R
    BBMwEYIPd3d3LmV4YW1wbGUuY29tMA0GCSqGSIb3DQEBBQUAA4IBAQB5bdOZqNae
    7nH69lSbunttLHYRHpkDhS7gNagns3lskq8+sUfanbq6Jf0iSIbwY2UO8M9/vzHb
    OvnhwiXRHMMK1W5E9a3r/9lVApOzdnVlyM0OPiGuubaVYMD7v0dwhEq0eETdrr29
    veI1R/8XSNcJD6XuW5OgQ5gooOL6Ao7aogPIDW3FMwSxm/5nS5xgp+924xFlk8zg
    moyuFiM99eH9OzALcIj21B/iBCrw1eM+BRgbr/azdWortnhPQpQB2LH6AMB2oQn0
    E05HDs1ZJ7Y1ovSrwN5L57YAPvP+87z8OeUkFwgQnu10y3CBuSysmXaFI1N2iv/2
    S5ukJUcRP1RC
    -----END CERTIFICATE-----

input.yaml:

certificate: "{{vault.key(/path/to/key/data)}}"

$ himl input.yaml
certificate: '-----BEGIN CERTIFICATE-----

  MIIC1TCCAb2gAwIBAgIJAIAI+nCNTUAuMA0GCSqGSIb3DQEBBQUAMBoxGDAWBgNV

  BAMTD3d3dy5leGFtcGxlLmNvbTAeFw0yMTA0MDkxNTIyMjlaFw0zMTA0MDcxNTIy

  MjlaMBoxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTCCASIwDQYJKoZIhvcNAQEB

  BQADggEPADCCAQoCggEBAK1klavp8ccE9es+8q2YQc9jRFmUIjwWXEzHQWXjoLQj

  ofqWOGP0XiUikx6Sj6USrgnIKbnt4mynn7awrh+yDm4q6rilqWurx7av/UVSTax9

  uMDB5Gf9hF87zHuS24I0EO7TKheRIdO8PLpLPe7a+SYylwmailLfnQZVPw/+Bovj

  2sa0fWRVy3qpvRoovPD77D9XrkS1LG6AaYFOmGboWL62rFB9xTy8vtfSIg+dHs2V

  NWVAVAIwqWMWY5E/WjBdDtZuCwpYl/eztoR6gjJqbI4g9lbpv/NCeAalMM8xKcQj

  /ZdI0j4H0IOVa+HNWwG+J46LDLV0AJcqKrd4PCEEJXUCAwEAAaMeMBwwGgYDVR0R

  BBMwEYIPd3d3LmV4YW1wbGUuY29tMA0GCSqGSIb3DQEBBQUAA4IBAQB5bdOZqNae

  7nH69lSbunttLHYRHpkDhS7gNagns3lskq8+sUfanbq6Jf0iSIbwY2UO8M9/vzHb

  OvnhwiXRHMMK1W5E9a3r/9lVApOzdnVlyM0OPiGuubaVYMD7v0dwhEq0eETdrr29

  veI1R/8XSNcJD6XuW5OgQ5gooOL6Ao7aogPIDW3FMwSxm/5nS5xgp+924xFlk8zg

  moyuFiM99eH9OzALcIj21B/iBCrw1eM+BRgbr/azdWortnhPQpQB2LH6AMB2oQn0

  E05HDs1ZJ7Y1ovSrwN5L57YAPvP+87z8OeUkFwgQnu10y3CBuSysmXaFI1N2iv/2

  S5ukJUcRP1RC

output:

```shell
$ cat output.yaml | yq -r .certificate
-----BEGIN CERTIFICATE-----
MIIC1TCCAb2gAwIBAgIJAIAI+nCNTUAuMA0GCSqGSIb3DQEBBQUAMBoxGDAWBgNV
BAMTD3d3dy5leGFtcGxlLmNvbTAeFw0yMTA0MDkxNTIyMjlaFw0zMTA0MDcxNTIy
MjlaMBoxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTCCASIwDQYJKoZIhvcNAQEB
BQADggEPADCCAQoCggEBAK1klavp8ccE9es+8q2YQc9jRFmUIjwWXEzHQWXjoLQj
ofqWOGP0XiUikx6Sj6USrgnIKbnt4mynn7awrh+yDm4q6rilqWurx7av/UVSTax9
uMDB5Gf9hF87zHuS24I0EO7TKheRIdO8PLpLPe7a+SYylwmailLfnQZVPw/+Bovj
2sa0fWRVy3qpvRoovPD77D9XrkS1LG6AaYFOmGboWL62rFB9xTy8vtfSIg+dHs2V
NWVAVAIwqWMWY5E/WjBdDtZuCwpYl/eztoR6gjJqbI4g9lbpv/NCeAalMM8xKcQj
/ZdI0j4H0IOVa+HNWwG+J46LDLV0AJcqKrd4PCEEJXUCAwEAAaMeMBwwGgYDVR0R
BBMwEYIPd3d3LmV4YW1wbGUuY29tMA0GCSqGSIb3DQEBBQUAA4IBAQB5bdOZqNae
7nH69lSbunttLHYRHpkDhS7gNagns3lskq8+sUfanbq6Jf0iSIbwY2UO8M9/vzHb
OvnhwiXRHMMK1W5E9a3r/9lVApOzdnVlyM0OPiGuubaVYMD7v0dwhEq0eETdrr29
veI1R/8XSNcJD6XuW5OgQ5gooOL6Ao7aogPIDW3FMwSxm/5nS5xgp+924xFlk8zg
moyuFiM99eH9OzALcIj21B/iBCrw1eM+BRgbr/azdWortnhPQpQB2LH6AMB2oQn0
E05HDs1ZJ7Y1ovSrwN5L57YAPvP+87z8OeUkFwgQnu10y3CBuSysmXaFI1N2iv/2
S5ukJUcRP1RC
-----END CERTIFICATE-----

-----END CERTIFICATE----- '

output:

```shell
$ cat output.yaml | yq -r .certificate
-----BEGIN CERTIFICATE-----
MIIC1TCCAb2gAwIBAgIJAIAI+nCNTUAuMA0GCSqGSIb3DQEBBQUAMBoxGDAWBgNV
BAMTD3d3dy5leGFtcGxlLmNvbTAeFw0yMTA0MDkxNTIyMjlaFw0zMTA0MDcxNTIy
MjlaMBoxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTCCASIwDQYJKoZIhvcNAQEB
BQADggEPADCCAQoCggEBAK1klavp8ccE9es+8q2YQc9jRFmUIjwWXEzHQWXjoLQj
ofqWOGP0XiUikx6Sj6USrgnIKbnt4mynn7awrh+yDm4q6rilqWurx7av/UVSTax9
uMDB5Gf9hF87zHuS24I0EO7TKheRIdO8PLpLPe7a+SYylwmailLfnQZVPw/+Bovj
2sa0fWRVy3qpvRoovPD77D9XrkS1LG6AaYFOmGboWL62rFB9xTy8vtfSIg+dHs2V
NWVAVAIwqWMWY5E/WjBdDtZuCwpYl/eztoR6gjJqbI4g9lbpv/NCeAalMM8xKcQj
/ZdI0j4H0IOVa+HNWwG+J46LDLV0AJcqKrd4PCEEJXUCAwEAAaMeMBwwGgYDVR0R
BBMwEYIPd3d3LmV4YW1wbGUuY29tMA0GCSqGSIb3DQEBBQUAA4IBAQB5bdOZqNae
7nH69lSbunttLHYRHpkDhS7gNagns3lskq8+sUfanbq6Jf0iSIbwY2UO8M9/vzHb
OvnhwiXRHMMK1W5E9a3r/9lVApOzdnVlyM0OPiGuubaVYMD7v0dwhEq0eETdrr29
veI1R/8XSNcJD6XuW5OgQ5gooOL6Ao7aogPIDW3FMwSxm/5nS5xgp+924xFlk8zg
moyuFiM99eH9OzALcIj21B/iBCrw1eM+BRgbr/azdWortnhPQpQB2LH6AMB2oQn0
E05HDs1ZJ7Y1ovSrwN5L57YAPvP+87z8OeUkFwgQnu10y3CBuSysmXaFI1N2iv/2
S5ukJUcRP1RC
-----END CERTIFICATE-----

Details

At the end, the output is correct but it does not look well in the output.yml file and is difficult to read.

My directory structure for my project looks like this

$ tree live/environments
live/environments
├── _default.yaml
├── dev.yaml
└── prod.yaml

I want to be able to selectively choose to merge dev.yaml with _defaults.yaml or prod.yaml with _defaults.yaml. Hell if I wanted to I should be able to merge dev.yaml with prod.yaml.

Expected Behaviour

# Non-existent API
config_processor.process_custom(default="live/environments/_default.yaml", override="live/environments/dev.yaml")

Merges dev.yaml on top of _default.yaml

Actual Behaviour

config_processor.process(path="live/environments")

Takes the last file, prod.yaml and merges with _default.yaml.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update python:3.12-slim docker digest to f11725a
• chore(deps): update dependency boto3 to v1.34.145
• chore(deps): update dependency backports.functools_lru_cache to v2
• chore(deps): update dependency hvac to v2
• chore(deps): update docker/build-push-action action to v6
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Expected Behaviour

When enabling the flag skip_interpolations=True for ConfigProcessor we would expect strings that would otherwise be interpolated to be left in tact.

Actual Behaviour

Value strings with "{{ some.param.value }}" are replaced with the referenced values.

Reproduce Scenario (including but not limited to)

Parse a hierarchical config with "{{ some.param.value }}" and the flag skip_interpolations=True set. Something like;

config = config_processor.process(
    path=path,
    filters=filters,
    exclude_keys=exclude_keys,
    output_format=output_format,
    print_data=True,
    skip_interpolations=True
)

Steps to Reproduce

See above. On inspection of the class method ConfigProcessor.process() we notice that a call to generator.resolve_interpolations() is outside of the if not skip_interpolations: test;

https://github.com/adobe/himl/blob/master/himl/config_generator.py#L73

However, commenting out this line still doesn't seem to prevent these params being interpolated. Adding a debug print() statement here and returning immediately afterwards (in an effort to disable interpolation completely) shows that this interpolation func is still being called but will prevent the the strings from being processed.

Platform and Version

Darwin 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18 00:46:32 PDT 2022; root:xnu-8020.101.4~15/RELEASE_ARM64_T6000 arm64

Python 3.8.9

attrs==21.4.0
backports.functools-lru-cache==1.6.4
boto3==1.21.1
botocore==1.24.46
certifi==2022.5.18.1
charset-normalizer==2.0.12
deepdiff==5.8.1
deepmerge==1.0.1
himl==0.9.0
hvac==0.11.2
idna==3.3
importlib-resources==5.7.1
jmespath==0.10.0
jsonschema==4.5.1
lru-cache==0.2.3
ordered-set==4.1.0
pathlib2==2.3.7.post1
pyrsistent==0.18.1
python-dateutil==2.8.2
PyYAML==6.0
requests==2.27.1
s3transfer==0.5.2
six==1.16.0
urllib3==1.26.9
zipp==3.8.0

Sample Code that illustrates the problem

networks:
    o3:
      name: "some_name"
      ipv4:
        subnet_name: "some_other_name"

network:
    name: "{{ networks.o3.name }}"
    shared: false
    subnet:
      name: "{{ networks.o3.ipv4.subnet_name }}"

Logs taken while reproducing problem

With the current code base:

  network:
    name: some_name
    shared: false
    subnet:
      name: some_other_name

When interpolation is completely disabled in code:

  network:
    name: '{{networks.o3.name}}'
    shared: false
    subnet:
      name: '{{networks.o3.ipv4.subnet_name}}'

Expected Behaviour

The complex example should compile using the example given in the documentation.

Actual Behaviour

The compilation failed:

 $ himl-config-merger examples/complex --output-dir merged_output --levels env region cluster --leaf-directories cluster
Traceback (most recent call last):
  File "/home/kim/.local/bin/himl-config-merger", line 8, in <module>
    sys.exit(run())
  File "/home/kim/.local/lib/python3.10/site-packages/himl/config_merger.py", line 184, in run
    merge_configs(dirs, opts.hierarchy_levels,
  File "/home/kim/.local/lib/python3.10/site-packages/himl/config_merger.py", line 93, in merge_configs
    merge_logic(config)
  File "/home/kim/.local/lib/python3.10/site-packages/himl/config_merger.py", line 115, in merge_logic
    config_processor.process(path=path, output_format="yaml", print_data=False, multi_line_string=True))
  File "/home/kim/.local/lib/python3.10/site-packages/himl/config_generator.py", line 58, in process
    generator.process_hierarchy()
  File "/home/kim/.local/lib/python3.10/site-packages/himl/config_generator.py", line 251, in process_hierarchy
    yaml_content = self.yaml_get_content(yaml_file)
  File "/home/kim/.local/lib/python3.10/site-packages/himl/config_generator.py", line 199, in yaml_get_content
    content = yaml.load(f, Loader=yaml.SafeLoader)
  File "/home/kim/.local/lib/python3.10/site-packages/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
  File "/home/kim/.local/lib/python3.10/site-packages/yaml/constructor.py", line 51, in get_single_data
    return self.construct_document(node)
  File "/home/kim/.local/lib/python3.10/site-packages/yaml/constructor.py", line 60, in construct_document
    for dummy in generator:
  File "/home/kim/.local/lib/python3.10/site-packages/yaml/constructor.py", line 413, in construct_yaml_map
    value = self.construct_mapping(node)
  File "/home/kim/.local/lib/python3.10/site-packages/yaml/constructor.py", line 218, in construct_mapping
    return super().construct_mapping(node, deep=deep)
  File "/home/kim/.local/lib/python3.10/site-packages/yaml/constructor.py", line 141, in construct_mapping
    raise ConstructorError("while constructing a mapping", node.start_mark,
yaml.constructor.ConstructorError: while constructing a mapping
  in "examples/complex/env=dev/region=us-west-2/cluster=cluster1/cluster.yaml", line 2, column 7
found unhashable key
  in "examples/complex/env=dev/region=us-west-2/cluster=cluster1/cluster.yaml", line 2, column 8

Reproduce Scenario (including but not limited to)

See above, HEAD was at 298793c, version installed was 0.12.0

Steps to Reproduce

git checkout 298793c94fb9f6e8d68af00f4a09f8b5c6ef4d24
himl-config-merger examples/complex --output-dir merged_output --levels env region cluster --leaf-directories cluster

Platform and Version

Kubuntu 22.04.2 LTS, version 0.12.0

Sample Code that illustrates the problem

The examples/complex in this repo

Logs taken while reproducing problem

See above

Integrate with vals to support it's whole set of kv integrations

https://github.com/variantdev/vals

Secret injection should be a separate option from key interpolation.

This is a feature request

It would be really nice to be able to use a jinja template to define variables programmatically.
I think jinjyaml may integrate really well with himl.

Example of jinja defined variables.

---
env: prod
deep:
  key3: v3
deep_list:
  - item3

jinja_array: !j2 |
  {% for i in range(10) %}
  - sub{{i}}: {{loop.index}}
  {% endfor %}

jinja_deep_list: !j2 |
  {% for i in deep_list %}
  - sub-jinja-{{i}}
  {% endfor %}

Produces

deep:
  key1: v1
  key2: v2
  key3: v3
deep_list:
- item1
- item2
- item3
env: prod
jinja_array:
- sub0: 1
- sub1: 2
- sub2: 3
- sub3: 4
- sub4: 5
- sub5: 6
- sub6: 7
- sub7: 8
- sub8: 9
- sub9: 10
jinja_deep_list:
- sub-jinja-item1
- sub-jinja-item2
- sub-jinja-item3

I currently have a very alpha poc here master...chekolyn:jinjyaml_poc to produce the above example. It does introduce a new package dependency but it would open a lot of possibilities with the use of jinja syntax.

looks like this was introduced with #162 @azun @danielcoman

Expected Behaviour

CLI example works as advertised.

himl examples/complex/env=dev/region=us-east-1/cluster=cluster2

Actual Behaviour

running the examples results in NameError: name 'filter_config' is not defined

Traceback (most recent call last):
  File "/opt/homebrew/bin/himl", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/himl/main.py", line 84, in run
    ConfigRunner().run(args)
  File "/opt/homebrew/lib/python3.11/site-packages/himl/main.py", line 30, in run
    self.do_run(opts)
  File "/opt/homebrew/lib/python3.11/site-packages/himl/main.py", line 41, in do_run
    config_processor.process(cwd, opts.path, filters, filter_config, excluded_keys, opts.enclosing_key, opts.remove_enclosing_key,
                                                      ^^^^^^^^^^^^^
NameError: name 'filter_config' is not defined

Reproduce Scenario (including but not limited to)

README.md example should work

Steps to Reproduce

git clone https://github.com/adobe/himl.git
cd himl
himl examples/complex/env=dev/region=us-east-1/cluster=cluster2

Platform and Version

Sample Code that illustrates the problem

bash-5.2$ git clone https://github.com/adobe/himl.git
cd himl
himl examples/complex/env=dev/region=us-east-1/cluster=cluster2
Cloning into 'himl'...
remote: Enumerating objects: 1069, done.
remote: Counting objects: 100% (323/323), done.
remote: Compressing objects: 100% (42/42), done.
remote: Total 1069 (delta 302), reused 286 (delta 281), pack-reused 746
Receiving objects: 100% (1069/1069), 195.73 KiB | 2.08 MiB/s, done.
Resolving deltas: 100% (638/638), done.
Traceback (most recent call last):
  File "/opt/homebrew/bin/himl", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/himl/main.py", line 84, in run
    ConfigRunner().run(args)
  File "/opt/homebrew/lib/python3.11/site-packages/himl/main.py", line 30, in run
    self.do_run(opts)
  File "/opt/homebrew/lib/python3.11/site-packages/himl/main.py", line 41, in do_run
    config_processor.process(cwd, opts.path, filters, filter_config, excluded_keys, opts.enclosing_key, opts.remove_enclosing_key,
                                                      ^^^^^^^^^^^^^
NameError: name 'filter_config' is not defined

Logs taken while reproducing problem

see above