vmware-tanzu-labs / yaml-overlay-tool Goto Github PK

Use JSONPath to query & patch YAML files with ease.

License: MIT License

Go 99.47% Dockerfile 0.40% Shell 0.13%

yaml kubernetes golang

yaml-overlay-tool's Introduction

YAML Overlay Tool (Yot)

YAML Overlay Tool, or Yot for short, often pronounced /yaucht/, is a tool to assist with patching YAML files. Yot uses JSONPath to query YAML documents within YAML files, and to perform a change. YAML Overlay Tool operates on YAML nodes. It is able to preserve and inject head, foot, and line comments into the new output versions of the files that you manipulate.

Getting started

Check out the list of YAML Overlay Tool's features.
Install YAML Overlay Tool.
Read the full documentation and user guide at docs.yaml-overlay-tool.io.
Try out an interactive YAML Overlay Tool tutorial to get you started quickly

Why create another YAML tool?

Yot is designed to be flexible, simple, and familiar; with a focus on end-user and developer experience. Whether you want to use a templating language to transform YAML data, or just change a couple values in a YAML document, Yot can make it possible.

Our philosophy is to treat YAML manifests as source code. We don't want to manage templated YAML files. We want to manage patches (overlays) and keep all potential templating outside of the source YAML files.

Templated files are hard to manage over time, making them difficult to read. Yot allows us to take YAML documents from multiple sources and transform them to fit our environment requirements through overlays. This practice allows us to manipulate a generic YAML file, without contaminating the original file. This practice also stops the cycle of having to update and manage complex YAML templates.

At the same time, Yot's instructions file specification provides you with documentation-as-code. This is because you have documented all the required changes to source YAML files in one place, including what is required to get an application running in your environment.

The use of JSONPath queries and templating give the tool familiar interfaces, making adoption easier, and providing for a more pleasant end-user experience. The specification, also known as the instructions file, is assembled in a declarative way, where we only operate on what is clearly defined. We take actions based on JSONPath query results. We provide flexibility by allowing your instructions to be templated if needed. See the full documentation, which will help get you moving along with Yot!

Author

Andrew J. Huffman
Jeff Davis

License

MIT
NOTICE

Contributing

Please see our Contribution Guide

Code of conduct

Please see our project's Code of Conduct

Communication

E-Mail

Please join our mailing list on Google Groups: yaml-overlay-tool-users

Kubernetes Slack

Please join us on the Kubernetes Community Slack #yaml-overlay-tool channel

yaml-overlay-tool's People

Contributors

Stargazers

Watchers

Forkers

dghant1024 jefedavis lizz7000 scottd018 andriykalashnykov ssansh ekmixon insideyew ahuffman

yaml-overlay-tool's Issues

Feature: Perform Overlays via CLI command

As a user, I would like to occasionally perform one-off overlays via the command line with an "overlay" sub-command. I would expect it to perform like a commonOverlay and have options for all the same features the instructions file has on a common overlay.

switch to camelCase to align with k8s yaml spec standards

while yaml is flexible and will support almost any casing standard I believe we should switch the yaml schema to leverage camelCase, allowing a more consistent experience for end users that are leveraging this tool for managing kubernetes manifests. it may seem like a small thing but it removes any confusion of "is this section part of the k8s schema and I need to use camelCase or is this section yot and need to use snake_case"

Feature: Multi-site/Multi-cluster/Multi-config via templating and site-specific values (instructions reusability)

As a user, I have several environments and/or configurations to manage. I often have a set of values that are common to all the differing configurations, but I also have some values that need to be overridden with alternate values for specific environments.

I would like to be able to pass in a named directory, where the directory is the name of my site, which contains a collection of yaml files that apply to that site, and many others as well. I would also like to pass in a set of common values, that the site specific values will get merged over to perform overrides when templating.

Order of Operations:

Common Values are read in
Site Values are read in
Instructions are templated based on configured templating language
Repeat for each add'l multi config values

These multi-config values should be passed in with a -m /path/to/my/site/dev or --multi-config. This option can be passed 1 or many times.

Additionally, I would like to have a parent directory containing several sub-directories that are named by their site/purpose, and all contain yaml files which are values for templating. I would like to also be able to pass them all in at once with -m or --multi-config.

Templating is inoperable and/or missing documentation

This could be a combination of me doing this wrong, but the docs for templating is completely missing so it's not clear how to accomplish templating in an overlay. Here is what I'm doing:

/tmp/values.yaml

---
updateCa: "true"

/tmp/instructions.yaml

---
commonOverlays:
  - name: "inject oidc settings to api server"
    query: spec.containers[0][?(@.name == 'kube-apiserver')].command
    action: merge
    value:
      - --oidc-client-id=tanzu
      - --oidc-issuer-url=https://test
      - --oidc-username-claim=email
      - --oidc-groups-claim=group
      {{if .updateCa eq "true"}}
      - --oidc-ca-file=/etc/kubernetes/pki/tanzu-ca.crt
      {{end}}

yamlFiles:
  - path: kube-apiserver.yaml

kube-apiserver.yaml (partial)

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  labels:
    component: kube-apiserver
    tier: control-plane
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=10.10.10.10
....

Command run (w/ error):

yot -f values.yaml -i instructions.yaml -o out/
15:53:23 [ERROR] template: instructions.yaml:11:11: executing "instructions.yaml" at <.updateCa>: updateCa is not a method but has arguments

Also tried with $updateCa in the template instead with this result:

yot -f values.yaml -i instructions.yaml -o out/
15:53:59 [ERROR] unable to process template for instructions file instructions.yaml, template: instructions.yaml:11: undefined variable "$updateCa"

Version:

yot version 0.4.0

Change document_query key to list and add some sort of grouping with an operator of 'and' and 'or'

To allow for more powerful document_query parameters we need to change this to a list.
To further qualify results we need to add some sort of operator key to allow for and/or groupings of queries.

Docs: Update setup guide for installation of packaged releases

Update guide for homebrew, deb, rpm, apk, snap installs.

Feature: URIs as path

As a user, I would like the ability to point to an upstream git repository URL or Kubernetes URI to use as a source file in my instructions file. I would like the tool to pull these files to a temporary location, and then perform overlays from the instructions file.

In case of a git repository, it would be nice to have the ability to additionally make an automatic commit/push once overlayed, but needs to be thought about since an end-user should also manually review the changes to ensure contents are as expected.

Kubernetes URIs would be nice to be able combine with the -s stdout option and direct the output into kubectl or other alternative manifest application tools.

%k formatter does not work on scalar nodes

Query (attempted in both yamlFiles and commonOverlays):

  - name: "adjust tanzu namespace spec values"
    query:
      - spec[*]
      # - spec.*     # also tried this 
    value: "%v" #+workload:%k:default="%v":type=string
    documentQuery:
      - conditions:
          - query: kind
            value: TanzuNamespace

Output (notice the missing key format input to the output manifest):

spec:
  tanzuNamespaceName: tanzu-identity #+workload::default="tanzu-identity":type=string
  tanzuLimitRangeDefaultCpuLimit: 50m #+workload::default="50m":type=string
  tanzuLimitRangeDefaultMemoryLimit: 64Mi #+workload::default="64Mi":type=string
  tanzuLimitRangeDefaultCpuRequest: 50m #+workload::default="50m":type=string
  tanzuLimitRangeDefaultMemoryRequest: 64Mi #+workload::default="64Mi":type=string
  tanzuLimitRangeMaxCpuLimit: 1000m #+workload::default="1000m":type=string
  tanzuLimitRangeMaxMemoryLimit: 1024Mi #+workload::default="1024Mi":type=string
  tanzuResourceQuotaCpuRequests: "6" #+workload::default="6":type=string
  tanzuResourceQuotaMemoryRequests: 8Gi #+workload::default="8Gi":type=string
  tanzuResourceQuotaCpuLimits: "6" #+workload::default="6":type=string
  tanzuResourceQuotaMemoryLimits: 8Gi #+workload::default="8Gi":type=string

Allow for environment vars to set command line options

To allow for other tools and workflows to more readily integrate with yot, we should allow command line options to be set with environment variables.

Unable to merge in a new map value within an array

Working with a kubernetes service, I want to inject a targetPort key to my output manifest.

The relevant data in question looks like this:

spec:
  ports:
    - port: 80
  selector:
    app.kubernetes.io/app: guestbook
    app.kubernetes.io/tier: frontend
  type: LoadBalancer

The overlay I'm attempting to apply looks like this:

- name: targetPort
  query: spec.ports[0].targetPort
  value: 80  # myport
  action: merge
  onMissing:
    action: inject

I expect an output result of:

spec:
  ports:
    - port: 80
      targetPort: 80 # myport
  selector:
    app.kubernetes.io/app: guestbook
    app.kubernetes.io/tier: frontend
  type: LoadBalancer

Instead I get the original value in the first yaml snippet.

Docs: Add information on using templating in instructions files

Need to build out how to use templating in Yot via instructions files.

Get stackoverflow when adding removeComments: true to yot.config file

My ./yot.config looks like:

outputDirectory: .out
removeComments: true

I get the following output:

runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0xc020260390 stack=[0xc020260000, 0xc040260000]
fatal error: stack overflow

runtime stack:
runtime.throw(0x1488bb1, 0xe)
        /opt/hostedtoolcache/go/1.16.5/x64/src/runtime/panic.go:1117 +0x72
runtime.newstack()
        /opt/hostedtoolcache/go/1.16.5/x64/src/runtime/stack.go:1069 +0x7ed
runtime.morestack()
        /opt/hostedtoolcache/go/1.16.5/x64/src/runtime/asm_amd64.s:458 +0x8f

goroutine 1 [running]:
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:178 +0x110 fp=0xc0202603a0 sp=0xc020260398 pc=0x13d2e30
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202603d0 sp=0xc0202603a0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260400 sp=0xc0202603d0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260430 sp=0xc020260400 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260460 sp=0xc020260430 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260490 sp=0xc020260460 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202604c0 sp=0xc020260490 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202604f0 sp=0xc0202604c0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260520 sp=0xc0202604f0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260550 sp=0xc020260520 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260580 sp=0xc020260550 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202605b0 sp=0xc020260580 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202605e0 sp=0xc0202605b0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260610 sp=0xc0202605e0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260640 sp=0xc020260610 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260670 sp=0xc020260640 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202606a0 sp=0xc020260670 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202606d0 sp=0xc0202606a0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260700 sp=0xc0202606d0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260730 sp=0xc020260700 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260760 sp=0xc020260730 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260790 sp=0xc020260760 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202607c0 sp=0xc020260790 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202607f0 sp=0xc0202607c0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260820 sp=0xc0202607f0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260850 sp=0xc020260820 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260880 sp=0xc020260850 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202608b0 sp=0xc020260880 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202608e0 sp=0xc0202608b0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260910 sp=0xc0202608e0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260940 sp=0xc020260910 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260970 sp=0xc020260940 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202609a0 sp=0xc020260970 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202609d0 sp=0xc0202609a0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260a00 sp=0xc0202609d0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260a30 sp=0xc020260a00 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260a60 sp=0xc020260a30 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260a90 sp=0xc020260a60 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260ac0 sp=0xc020260a90 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260af0 sp=0xc020260ac0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260b20 sp=0xc020260af0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260b50 sp=0xc020260b20 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260b80 sp=0xc020260b50 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260bb0 sp=0xc020260b80 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260be0 sp=0xc020260bb0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260c10 sp=0xc020260be0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260c40 sp=0xc020260c10 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260c70 sp=0xc020260c40 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260ca0 sp=0xc020260c70 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260cd0 sp=0xc020260ca0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260d00 sp=0xc020260cd0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260d30 sp=0xc020260d00 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260d60 sp=0xc020260d30 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260d90 sp=0xc020260d60 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260dc0 sp=0xc020260d90 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260df0 sp=0xc020260dc0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260e20 sp=0xc020260df0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260e50 sp=0xc020260e20 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260e80 sp=0xc020260e50 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260eb0 sp=0xc020260e80 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260ee0 sp=0xc020260eb0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260f10 sp=0xc020260ee0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260f40 sp=0xc020260f10 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260f70 sp=0xc020260f40 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260fa0 sp=0xc020260f70 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020260fd0 sp=0xc020260fa0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261000 sp=0xc020260fd0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261030 sp=0xc020261000 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261060 sp=0xc020261030 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261090 sp=0xc020261060 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202610c0 sp=0xc020261090 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202610f0 sp=0xc0202610c0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261120 sp=0xc0202610f0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261150 sp=0xc020261120 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261180 sp=0xc020261150 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202611b0 sp=0xc020261180 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202611e0 sp=0xc0202611b0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261210 sp=0xc0202611e0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261240 sp=0xc020261210 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261270 sp=0xc020261240 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202612a0 sp=0xc020261270 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202612d0 sp=0xc0202612a0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261300 sp=0xc0202612d0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261330 sp=0xc020261300 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261360 sp=0xc020261330 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261390 sp=0xc020261360 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202613c0 sp=0xc020261390 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202613f0 sp=0xc0202613c0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261420 sp=0xc0202613f0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261450 sp=0xc020261420 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261480 sp=0xc020261450 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202614b0 sp=0xc020261480 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202614e0 sp=0xc0202614b0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261510 sp=0xc0202614e0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261540 sp=0xc020261510 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261570 sp=0xc020261540 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202615a0 sp=0xc020261570 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc0202615d0 sp=0xc0202615a0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261600 sp=0xc0202615d0 pc=0x13d2d6e
github.com/vmware-tanzu-labs/yaml-overlay-tool/internal/instructions.removeCommentsFromNode(0xc000227540)
        /home/runner/work/yaml-overlay-tool/yaml-overlay-tool/internal/instructions/yamlfile.go:188 +0x4e fp=0xc020261630 sp=0xc020261600 pc=0x13d2d6e
...additional frames elided...

Feature: Store YAML Files as Objects

For use as a library, YAML files should be stored as objects on a structure, as some programs will want to use their own YAML files that may or may not be able to point to a file on local file systems.

See https://github.com/vmware-tanzu-labs/yaml-overlay-tool/blob/port-to-golang/internal/lib/instructions.go#L41

Relative paths do not work when going up a directory

Taking this file into consideration:

---
yamlFiles:
  #
  # common
  #
  - path:       "../vendor/tanzu-namespace.yaml.j2"
    outputPath: "../manifests/namespace.yaml"
    overlays:
      - name: "insert workload markers for parent namespace"
        query:
          - spec.tanzuNamespaceName
          - metadata.name
        value: cnpo-identity-system #+workload:namespace:default="cnpo-identity-system":type=string
        action: replace

      - name: "define tanzu namespace spec values as markers"
        query: spec
        value:
          "%k": "%v" #+workload:%k:default="%v":type=string

The following error is produced:

.source/cnpo.tanzulabs.vmware.com/identity/common/manifests/namespace.yaml
Error: unable to read instructions file .source/cnpo.tanzulabs.vmware.com/identity/dex/config/overlay.yaml: EOF
14:14:50 [ERROR] unable to read instructions file .source/cnpo.tanzulabs.vmware.com/identity/dex/config/overlay.yaml: EOF

Moving the overlay so that it does not have to go up a directory fixes the issue.

JSONpath Filter Query Inop

What i would like to do (works in straight JSON path):

  # - name: "replace dex values with appropriate resource name markers"
  #   query:
  #     - "..*[?(@.name == 'dex')]"
  #     - "..*[?(@.serviceAccountName == 'dex')]"
  #   value: dex #+workload:dexResourceName:default="dex":type=string

What i have to do in order to work around:

  - name: "ensure dex resources have the appropriate markers"
    path: "../vendor/dex/"
    overlays:
      - name: "inject dex resource name marker"
        value: dex #+workload:dexResourceName:default="dex":type=string
        query:
          - metadata.name
          - metadata.labels['app.kubernetes.io/name']
          - spec.template.metadata.labels['app.kubernetes.io/name']
          - spec.selector.matchLabels['app.kubernetes.io/name']
          - spec.selector['app.kubernetes.io/name']
          - roleRef.name
          - subjects[*].name
          - ..*.serviceAccountName
          - ..*.containers[*].name
          - ..*.configMap.name
          - ..*.ports[*].name

Fix yot's python lint issues

See results from linting run. Lots of little nits to fix up.

Feature: Ability to template instructions files for multi-site/multi-cluster configurations

In order to handle complex multi-site and multi-cluster configurations, templating of instructions files is required.

This will allow a user of Yot to parameterize and template overlays in instructions files VS. treat YAML configurations/manifests as templates, thus removing the need to manage the technical debt of templated YAML configurations.

This should allow the user multiple templating options they may already be accustomed to, such as:

Go templating
Jinja2
Others (please comment on templating languages you would like to see added into the tool)

check license text on each file

Add a --remove-comments feature

Sometimes I would like manifests that get overlayed to be pristine and allow my overlay to document the changes that I'm making. By default, I feel that leaving the manifests untouched (with comments) is desired, however there may be a use case to strip out the comments from the source manifests prior to overlaying.

Proposal: Port `yot` to Go

To be more aligned with and attractive to the cloud native community, we would like to port yot from Python to Go.

Decisions:

Templating language considerations
- Pongo2 (close to jinja2)
- Go Templating Language
- Something else?
- Could this be pluggable?
- Requirements:
  - for loops
  - if statements
  - var substitution
  - plugins/custom macros
Pros?
* Potentially more community interest in contribution
* Performance gains? <- already fast since we're dealing with text processing
* Easier to package and deliver?
Cons?
- What are the disadvantages to moving to Go?
- ???

This is all open for conversation and highly encouraged. If you have opinions on this, please share!

refactor yamlFile SubObject for more streamlined processing

currently we use a sub object called File to process paths contained in directories, we should refactor this to perform this logic during unmarshal and get rid of the sub object

Feature: Allow `outputPath` to be an absolute path

Currently the outputPath key only allows for a relative path from the location of the instructions file. It would be ideal to allow a user to specify an absolute path to where they would like output to land, even if outside of the -o /output/path for specific use-cases.

Feature: YTT Templating Support

As a user, I would like to have support for YTT Templating within instructions files. I understand I will probably lose my comments and have no ability to inject comments within YAML files.

Feature: Add JSON Support for output and value files (templating)

As a user, I would sometimes like to be able to reuse JSON based values files when templating. I would also like to output JSON files as well as an option.

Allow more control over where YAML files get output

As a consumer of YOT, I would like the ability to control the output of the files which are produced by YOT. It would be nice to be able to create the file structure of my choosing when outputting these files. See also #22 for a similar issue.

E.g.

instructions.yaml:

yamlFiles:
  - name: "example of manipulating an output file path"
    path: 01_something.yaml
    outputFilename: i_hate_files_with_number_prefixes.yaml

The following command would be run:

yot -i instructions.yaml -o manifests/

Which would produce:

manifests/
└── i_hate_files_with_number_prefixes.yaml

0 directories, 1 file

Also, notice the missing yamlFiles directory. I do not believe users should be tied to an opinionated directory structure when outputting with the -o option. Instead, they should end up in the same structure with the -o as the root, and any filePath overrides taking precedence for the specified files.

Lint rules changed recently and have been giving CI run failures

Need to fix the linting issues that recently sprung up.

Create a logging mechanism for debugging

In order to debug what has happened during a yot run, a logging mechanism should exist.

Add Python Linter Github Action

We should add pylint so the python code is linted on push to ensure consistency and best practices

Feature: Ability to include overlays off of the main instructions file to better organize my content

As a user, I have many overlays. So many that my instructions files have grown quite large. I would like the ability to use an include keyword under commonOverlays, or yamlFiles[*].overlays, or yamlFiles[*].documents.overlays.

Example Instructions file:

commonOverlays:
  - include: /path/to/broken/out/overlays.yaml
yamlFiles:
  - name: my upstream manifests
    path: /tmp/downloads
    overlays:
      - include: /path/to/broken/out/overlays2.yaml
    documents:
      - name: specific document
        path: 0
        overlays:
          - include: /path/to/broken/out/overlays3.yaml

The files would contain arrays of overlays, without any indentation, and would be loaded up underneath the appropriate keywords commonOverlays and yamlFiles.

Feature: GO Templating Sprig Support

As a user I would like to use all of the templating features that are commonly available in other familiar tools in my instructions files when templating with Go templating. It would be nice to have Sprig support.

Injecting comments, while attempting to retain an original integer value forces me to quote the format marker

I'm trying to inject a comment within a k8s deployment after the replicas field like:

- query: spec.replicas
  value: "%v" # my comment here

Due to the % character not allowed by the YAML standard to start a value, I'm forced to quote the %v format marker. When the new YAML file is generated, this causes my integer to become a string, which is not good in cases where APIs are strongly typed.

Feature: Ability to take a single YAML document from a YAML file, and loop (via templating) to create additional YAML documents from it

As a user, I have a collection of iterable values, a single document YAML file, and I would like to iterate over the initial YAML document (treated as a starting point/template) to generate 1 or many new YAML documents to be output to the YAML file from the iterable values.

This could either be a new action or simply a method to use a source document within a YAML file.

Feature: Jinja2 Templating Support

As a user, I would like to have jinja2 templating support within instructions files.

Proposal: Changes to format marker syntax

Due to issue #105 , we should consider changing the syntax of format markers.

I propose to encapsulate the %v, %k, %h, %l, and %f format markers with curly braces as such: {%v}.

In cases where a user would like to manipulate the original values with the go-sed implementation, the format marker should change to either: {%v, s/cool/neat/g} or {%v: s/coll/neat/g}.

Packaging of snaps

As a user, it would be nice to be able to install Yot via snap for Ubuntu and Arch Linux, and any other supported platform.

Combine action runs twice if preceded with a replace

Real-world users probably won't hit this, but if you perform a replace on a Yaml Node, followed by a 'combine' action, due to the nature of the deep copy performed, the combine action will run more than once.

Header/Footer comments land in wrong places sometimes when injected into a YAML document

During injection of header/footer comments, they do not always land where they "should" within the YAML document.

This is due to an issue in upstream go library yaml.v3. The problem is essentially an issue with how/where to handle comments in maps.

The headers will land below the target injection point, and the footers can end up sometimes above the target injection point.

We should make an effort to contribute to the upstream library to solve the problem within the library.

Signed Checksums

Checksums are not currently being signed via the release process. We need to setup keys to perform signing during the release process with go releaser.

Consistent Versioning for Release Artifacts

Given the release path:

https://github.com/vmware-tanzu-labs/yaml-overlay-tool/releases/download/v0.1.0/yot_0.1.0_Linux_x86_64.tar.gz

Version is listed as v0.1.0 after the download/ path
Version is listed as ** 0.1.0** in the tarball.

Keeping these consistent will allow users to more easily download their given versions. E.g.:

curl -L https://github.com/vmware-tanzu-labs/yaml-overlay-tool/releases/download/${YOT_VERSION}/yot_${YOT_VERSION}_Linux_x86_64.tar.gz

Packaging for homebrew

As a user, it would be convenient to have a homebrew (brew) tap for my MacOS system to install Yot with.

Add a configuration file

Add a configuration to allow config for the jinja2 templating engine, which would allow for specifying plugins and filter paths to enhance jinja2's current default experience.
As a user, I may want to make use of add'l jinja2 functionality, or customize how jinja2 behaves. To achieve this we need a configuration file that can be parsed and the values can be passed to the instantiation of jinja2 in the tool.

Packaging of RPM/Deb/APKs on release

As a user, it would be convenient to have the ability to install Yot from a package manager such as RPM/Deb/APK.

Delete action panics if given a keyname query

example manifest

---
commonOverlays:
  - name: "Test":
    query: metadata.labels.name~
    action: delete
yamlFiles:
  - path: examples/kubernetes/manifests/

results in panic: slice index out of bounds

Feature: Allow for regex based formatting of keys, values, head comments, foot comments, and line comments

As a user, I would like to use the original values returned from my JSONPath query to be able to manipulate the original values, not just re-insert the original values.

Feature: Allow an absolute path in `path` yamlFile key

Currently the path key only supports paths relative to the instructions file. The path should also accept an absolute path to a YAML file or directory of YAML files on the filesystem.

Create a step-by-step tutorial with real-world examples

Create a walk-through guide that takes k8s manifests from somewhere upstream and teaches an end-user how-to fish.

Allow delete action to delete an entire document

given an overlay of

- query: $
   action: delete

we should be able to delete the entire document and not output anything

this could be useful in templating allowing if/then statements to control if a file gets included in the final output. an example that comes to mind would be if you didn't want to include a ingress or a secret

Using outputPath may produce 2 identical files

Consider this instruction set:

yamlFiles:
  #
  # dex
  #
  - name: "adjust common file path location"
    path: "../vendor/dex/01-common.yaml"
    outputPath: "dex/unused/dex-common.yaml"

  - name: "ensure dex resources have the appropriate markers"
    path: "../vendor/dex/"
    overlays:
      - name: "inject dex resource name marker"
        value: dex #+workload:dexResourceName:default="dex":type=string
        query:
          - metadata.name
          - metadata.labels['app.kubernetes.io/name']
          - spec.template.metadata.labels['app.kubernetes.io/name']
          - spec.selector.matchLabels['app.kubernetes.io/name']
          - spec.selector['app.kubernetes.io/name']
          - roleRef.name
          - subjects[*].name
          - ..*.serviceAccountName
          - ..*.containers[*].name
          - ..*.configMap.name
          - ..*.ports[*].name

When the files are written, there is overlap between the directory path and the file path between the 2 separate yamlFile declarations. The above will produce the following with the same content:

the original file (01-common.yaml)
the new output file (dex/unused/dex-common.yaml)

Here is a sample (more files included) to show what was produced:

static/components/core/identity/manifests/
├── dex
│   ├── 01-common.yaml
│   ├── 02-certs-selfsigned.yaml
│   ├── 03-cm.yaml
│   ├── 04-rbac.yaml
│   ├── 05-0-secret.yaml
│   ├── 05-deployment.yaml
│   ├── 06-service.yaml
│   ├── dex-deployment.yaml
│   ├── dex-rbac.yaml
│   ├── dex-service.yaml
│   └── unused
│       ├── dex-certs.yaml
│       └── dex-common.yaml
├── gangway
│   ├── 01-common.yaml
│   ├── 02-certs-selfsigned.yaml
│   ├── 02-gangway-service.yaml
│   ├── 03-config.yaml
│   ├── 03-dex-ca-config.yaml
│   ├── 03-secret.yaml
│   ├── 05-deployment.yaml
│   ├── dex-ca-config-map.yaml
│   ├── gangway-config-map.yaml
│   ├── gangway-deployment.yaml
│   ├── gangway-secret.yaml
│   ├── gangway-service.yaml
│   └── unused
│       ├── gangway-certs.yaml
│       └── gangway-common.yaml
└── ldap
    ├── ldap-deployment.yaml
    └── ldap-service.yaml

5 directories, 28 files

Consider Complex Strings when Templating

I know templating isn't implemented yet, but consider the following use case (Jinja2), so that we may template complex strings (e.g. ConfigMaps) in conjunction with overlaying YAML:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ldif-config
  namespace: "{{ tanzu_identity.namespace }}"
data:
  config-ldap.ldif: |-

    dn: ou=People,{{ tanzu_identity.ldap.base_dn }}
    objectClass: organizationalUnit
    ou: People

    dn: cn={{ user.first_name }},ou=People,{{ tanzu_identity.ldap.base_dn }}
    objectClass: person
    objectClass: inetOrgPerson
    sn: {{ user.last_name }}
    cn: {{ user.first_name }}
    mail: {{ user.email }}
    userpassword: {{ user.password }}

    dn: cn={{ user.first_name }},ou=People,{{ tanzu_identity.ldap.base_dn }}
    objectClass: person
    objectClass: inetOrgPerson
    sn: {{ user.last_name }}
    cn: {{ user.first_name }}
    mail: {{ user.email }}
    userpassword: {{ user.password }}

    # Group definitions.

    dn: ou=Groups,{{ tanzu_identity.ldap.base_dn }}
    objectClass: organizationalUnit
    ou: Groups

    dn: cn=admins,ou=Groups,{{ tanzu_identity.ldap.base_dn }}
    objectClass: groupOfNames
    cn: admins

    dn: cn=developers,ou=Groups,{{ tanzu_identity.ldap.base_dn }}
    objectClass: groupOfNames
    cn: developers

Allow multiple queries to use the same value

As a consumer, I'd like the ability to use multiple queries to inject the exact same value in multiple places in order to cut down on copy/paste.

Here is what we have now:

yamlFiles:
  - name: "tanzu namespace markers"
    path: "static/components/core/identity/base/common/namespace.yaml"
    overlays:
      - query: metadata.name
        value: tanzu-identity #+workload:namespace:default="tanzu-identity":type=string
        action: merge
      - query: spec.tanzuNamespaceName
        value: tanzu-identity #+workload:namespace:default="tanzu-identity":type=string
        action: merge

Here is what it could look like:

yamlFiles:
  - name: "tanzu namespace markers"
    path: "static/components/core/identity/base/common/namespace.yaml"
    overlays:
      - queries: 
        - metadata.name
        - spec.tanzuNamespaceName
        value: tanzu-identity #+workload:namespace:default="tanzu-identity":type=string
        action: merge

This is similar to how the commonOverlays handles it.