It could be good to have a flag to check if a chart of the version that would be set is already available in the associated helm chart repository I think. That way, we could have the means to resolve this issue for example: jupyterhub/zero-to-jupyterhub-k8s#1451.

In mybinder.org, we use chartpress to build images and set image tags in a deployment config file (such as config/staging.yaml or config/prod.yaml). Currently, values.yaml is hardcoded - would be nice to allow this to change.

Since 55fe935, the --commit-range argument is no longer needed for chartpress to do a good job.

We should update the documentation to reflect this.

Recent published versions of ruamel.yaml have introduced breaking changes for chartpress. See:

• #17
• #19
• jupyterhub/zero-to-jupyterhub-k8s#831

To avoid this in the future, it might be a good idea to pin the version of ruamel.yaml to be used by chartpress. We can also use a narrow range of versions known to work.

I've started using chartpress --reset with the pre-commit package and it works very nicely to prevent accidental commits of modified chart values. It would be great to have the .pre-commit.hooks.yaml definition as a part of chartpress so one could simply include it like any other pre-commit hook. The user of chartpress would then have a .pre-commit.config.yaml in their repo's base directory with something like:

repos:
-   repo: https://github.com/jupyterhub/chartpress
    rev: master
    hooks:
    - id: chartpress

If then you try to commit a change to the chartpress-managed values you get something like:

git commit -am 'commiting something'
black................................................(no files to check)Skipped
Flake8...............................................(no files to check)Skipped
chartpress reset.........................................................Failed
hookid: chartpress

If you want to include something like this I'll happily contribute a PR.

Proposed change

To publish a latest image tag by default and make it configurable by a flag.

Who would use this feature?

It would allow for example a GitHub action setup to do a security scan of the latest image be able to do it without having a lot of code to extract the latest tagged image using chartpress or similar. This is a discussion arising from jupyterhub/zero-to-jupyterhub-k8s#1712 (comment).

(Optional): Suggest a solution

We add a boolean flag which defaults to something that makes us also push latest image tags when we --push the images. I suggest --push-latest-tag=false as the override of this default or perhaps --no-push-latest-tag or something like that.

It would be nice to get PyPI CD working.

We have configured chartpress to use "round trip mode" and to "preserve quotes". This means that if we load something, then change something in the python dictionary, and then write it back - it will preserve usage of no/single/double quotes for our strings.

There are four different situations where this can happen or not.

1. Chart.yaml

• Version set
• Version reset

1. values.yaml

• Tag set
• Tag reset

The bug is that chartpress doesn't preserve the quote formatting when updating the image tags.

This is where we define we want to use a roundtrip strategy and that we want to preserve the quotes.

This is where it fails for values.yaml

This is where it succeeds for Chart.yaml

The issue is resolved by not coercing the image tag to be a string here at this point.

This coercion was introduced in #16 to solve #15. The question in my mind now is if this is still relevant.

It seems as long as we use ruamel.yaml 0.15.44+ we should be fine!

I've also verified this by...

pip install "ruamel.yaml==0.15.43" && \
python -c "import sys; import ruamel.yaml; ruamel.yaml.YAML(typ='rt').dump({'x': '008'}, sys.stdout)"
        
pip install "ruamel.yaml==0.15.44" && \
python -c "import sys; import ruamel.yaml; ruamel.yaml.YAML(typ='rt').dump({'x': '008'}, sys.stdout)"

I'm submitting a PR where we stop coercing this and declare a dependency on ruamel >=0.15.44.

I'd like to see chartpress output details about the versions it will apply early on before starting to build the chart if possible.

Perhaps it makes the most sense to do this by something like a pip freeze command, but it should also make sure to write it when it updates Chart.yaml at the latest.

@rodriguez-facundo wrote:

Hi guys, the tag 0.9-b51ffeb for k8s-network-tools and k8s-image-awaiter docker images was overwrite on Saturday. I had a deployment to k8s working with that tag on Friday and today the awaiter is hanging up, unable to deploy a small change I did to my notebook. Any idea how can I get the image that was in that tag before Saturday update?

Dear rubber duck

I write to you to document and think more clearly...

Hmmmm why did this happen? I verified there was an override of the b51ffeb tag as Facu described, and that this tag was 4 months old relating to jupyterhub 1.0.0.

Hmmm... Neither do I understand how the rebuild caused image-awaiter to get stuck...

Investigation

In the logs here, i found that there was a push made by chartpress, our CLI helping out: https://travis-ci.org/jupyterhub/zero-to-jupyterhub-k8s/jobs/581728271

$> docker build -t jupyterhub/k8s-hub:0.9-ec3b97d images/hub --build-arg JUPYTERHUB_VERSION=1.0.0
$> docker push jupyterhub/k8s-hub:0.9-ec3b97d $> git log -n 1 --pretty=format:%h -- images/network-tools chartpress.yaml
$> docker build -t jupyterhub/k8s-network-tools:0.9-b51ffeb images/network-tools
$> docker push jupyterhub/k8s-network-tools:0.9-b51ffeb
$> git log -n 1 --pretty=format:%h -- images/image-awaiter chartpress.yaml
$> docker build -t jupyterhub/k8s-image-awaiter:0.9-b51ffeb images/image-awaiter
$> docker push jupyterhub/k8s-image-awaiter:0.9-b51ffeb
$> git log -n 1 --pretty=format:%h -- images/singleuser-sample chartpress.yaml
$> docker build -t jupyterhub/k8s-singleuser-sample:0.9-b51ffeb images/singleuser-sample --build-arg JUPYTERHUB_VERSION=1.0.0
$> docker push jupyterhub/k8s-singleuser-sample:0.9-b51ffeb

Why was this done though? The chartpress command was...

chartpress --commit-range 14dd29ab85f1...ec3b97df3f75 --push --publish-chart

I note that there was no --tag sent, so therefor in the chartpress code i can conclude that the following call to docker push was triggered because image_needs_pushing(image) was considered truthy.

    if push:
        if tag or image_needs_pushing(image_spec):
            check_call([ 'docker', 'push', image_spec ])
        else:
            print(f"Skipping push for {image_spec}, already on registry")

So why was the image_needs_pushing(image_spec) function truthy!?

@lru_cache()
def image_needs_pushing(image):
    """Return whether an image needs pushing
    Args:
    image (str): the `repository:tag` image to be build.
    Returns:
    True: if image needs to be pushed (not on registry)
    False: if not (already present on registry)
    """
    d = docker_client()
    try:
        d.images.get_registry_data(image)
    except docker.errors.APIError:
        # image not found on registry, needs pushing
        return True
    else:
        return False

• Was it because memoization through the functools @lru_cache() somehow returned a cached truthy value while it shouldn't have?
• Was it because an error was thrown, and the error was docker.errors.APIError, even though it did not end up meaning that the image was not found, but it was thrown for another reason?
• Was it because the image on dockerhub had actually been deleted or similar?
• Does this relate to what @clkao found in #40?
• Was it because the chartpress version we use in z2jh differed from the one in chartpress master?

I'm not sure if this is what chartpress is intended to do, but I'd like to try using chartpress to build an image defined in our jupyterhub.hub.extraContainers. extraContainers takes a list of dictionaries where each has an image key with value of the form name:tag. However valuesPath will insert name and tag keys.

To avoid #76 again.

Tags on commits not in current branch

Chartpress picks the version it sets, like 0.9.0-n123.gabc1234, based on the latest commit on the current branch. This means that if someone makes a patch release 0.9.1 with a cherry picked commit, it will be out of branch and chartpress will keep counting from 0.9.0. This is probably the preferred behavior, but it is a bit sensitive and requires the user to be aware about it.

I wonder if we perhaps should provide a warning and documentation about this in the readme?

Invalid helm versions

Helm3 requires the Chart.yaml version to not have a leading v, so if someone makes a tag like v0.9.0 chartpress would still set the Chart.yaml version field to v0.9.0 but that would be invalid. So, what should we do about this? I think the right call is to allow the v in the tag but make the version set in chart.yaml be with the v stripped while letting the docker images have the v part of the tag still.

I'd love some input on this change.

This part is being closed by #106.

In ccbbba7 I documented the current procedure for --publish-chart.

We currently act as if we have a --force-publish-chart flag set by default when we use --publish-chart. I think we should require --force-publish-chart to be passed to get the current behavior where a new chart of the same version will override an existing chart.

This relates to #60 that is about making chartpress provide a easy way to check if a chart is already published before running tests etc again if nothing has changed.

To learn about buildArgs, one requires to inspect the code currently. Fixxable by documenting an example in the README.md.

In #52 I added a + sign to the helm chart version, but the Helm chart version is used to construct a filename when we use --publish-chart. If this would be done on a windows computer, we would crash I think.

We should probably when working with the filename instead build to a filename where we replace + with _. What's important though, is that everything still works when updating and adding the files to index.yaml, so that the chart version we get from there still relies on +.

Wait, it may work... I don't know.

If chartpress is run with unstaged changes that makes us require a image rebuild, what should we do?

Currently, the process chartpress works like...

1. Get the last commit where dependent files for an image where changed.
2. See if that image is available locally or at DockerHub and build it if it isn't.

So, if we have unstaged changes, chartpress will not rebuild or update the helm chart and images. I think it could be good to do this by default, but that the user may want to understand if this happens by being warned they need to commit those changes for chartpress to care.

I propose we add a check for unstaged changes of relevance and print a warning message about this. Oh no wait... chartpress could mess up badly unless we fail sometimes... Consider if it realize it requires a rebuild, but there are more unstaged changes of relevance for the image build. Then it could end up building a image for a commit tag that isn't representing the state of that commit.

We have to decide how we want chartpress to behave in at least two situations.

There are unstaged changes influencing an image, and...

1. the tag is not available and needs to be rebuilt
2. the tag is already available and doesn't need to be rebuilt

I think we should fail hard for 1), and for 2) the user at least needs to be informed.

Technically, to conclude if there are unstaged changes, we could use the currently unused (since 55fe935) function path_touched where we would pass the commit range of HEAD for it to catch staged and unstaged changes that has not yet been committed.

More thorough testing

Partially closed by #67.

Managing testing content

There is a challenge with testing chartpress relating to its interaction with the git repository. For proper testing, we would need to make commits and tags to some git repository, within some branch.

The best idea that allows for these modifications etc is to work with a pytest fixture to create a new temp directory, copying testing content there, run git init etc, and add tags etc.

Discarded ideas:

• Having a separate repository with testing content etc that we would clone.
• Checking out a new branch, adding tags etc, and then discarding the branch and returning to previous branch.

Testing content

README.md
chartpress.yaml
images/Dockerfile
charttest/values.yaml
charttest/Chart.yaml

Tests

• Test --force-build
• Test --push
• Test --force-push
• Test static and rendered dynamic values (TAG, LAST_COMMIT) etc are passed to the Dockerfile as --build-arg.
• Test that the latest of a) latest tagged commit and b) latest modifying commit is used.
• Test --version
• Test --reset
• Test --skip-build
• Test --publish-chart
• Test --extra-message
• ...

(from jupyterhub/zero-to-jupyterhub-k8s#1260 (comment))

in a local dev env, i see Skipping build for jupyterhub/k8s-hub:0.9-c9f80ce, it already exists when running chartpress. but this is somehow not the case on travis, for example: https://travis-ci.org/jupyterhub/zero-to-jupyterhub-k8s/jobs/526434320#L536

setup.py says this should work with Python 3.4+: https://github.com/jupyterhub/chartpress/blob/0.2.2/setup.py#L47 but chartpress.py includes formatted string literals such as print(f"Skipping {name}, not touched in {commit_range}") which were added in Python 3.6.

The changes introduced in #52 and subsequent iterations created a backward-incompatible image and chart tagging scheme (which consequently broke most of our CI/CD pipelines). I don't dislike the new scheme since it is also more aligned with how e.g. setuptools_scm handles versions but I wonder if it would be desirable to provide an option to use the "old-style" tags e.g. <version>-<commit-sha>?

Add support for images defined as <registry>/<repo>:<tag>.

Possible solutions to keep compatibility with images defined as mapping:

1. check the image value in values.yaml
2. if it is mapping use current way of setting the dictionary;
3. else use formatted string "{registry}/{repo}:{tag}".format(...)

Please let me know what you think and I will send a PR.

@minrk @manics @yuvipanda I suggest we release 1.0.0, this project has plenty of tests and is well documented and such I think.

It can be practically helpful to have more flexibility to communicate more information with the changes to versions when you can adjust major.minor.patch and not only adjust minor.patch, so I think we should go for a 1.0.0 release!

I hope to address #89 first before release, but otherwise think this is ready for the release.

#62 is causing troubles in 0.4.0-0.4.2, but is fixed in master.

@minrk, are you okay with me releasing 0.4.3 tonight for example? I'd then add a CHANGELOG etc.

Background

I would like the developer experience for Z2JH to be as great as possible. Currently I think chartpress clutters things too much in the local git clone of the repo in Chart.yaml and values.yaml. It is generally quite cumbersome to make a change in the Z2JH repo and use the new chart, and I'd like to improve it. With the following suggested changes to chartpress, we could make a very smooth experience for new developers I think.

Suggestions

• Allow chartpress to be run from a different folder
  I think we should let chartpress mirror something like docker build . allowing a path to be specified, but still default to the CWD. Think of for example docker build ., they allow the same thing but specifying . is the most common thing to do.
• Avoid modifying values.yaml in place
  I think passing the chartpress argument --images-config images-config.yaml could be a good way to ask chartpress to output the image name and tag to an external .yaml file instead of overwriting the values in place in values.yaml.
• Avoid modifying Chart.yaml in place
  Is it important that we update the version all the time? Perhaps we can do it only when we are pushing the chart? We could perhaps also restore it after the chart has been pushed? I recall the chart is copied to a folder, perhaps we could only make the Chart.yaml in that copied folder be updated, and let the Chart.yaml within the original repo remain as it is?
• Allow general chartpress.yaml override
  I was about to write that we should allow override of the repo.git field, but perhaps it is even better to simply allow for an additional chartpress.yaml file that can override fields in the chartpress.yaml within the repo? We have an array of charts in chartpress.yaml, but we also have a name field within the elements of the array. I suggest that we would merge in a manner that would update elements with identical names, fail if the a single chartpress.yaml file contains multiple elements of the same name, and add an element if the chartpress-a.yaml and chartpress-b.yaml contains elements with different name. I've seen such merging behavior while working with kubernetes resources.
  To clarify the merge behavior in other words, the following...

  charts:
    - name: jupyterhub
      imagePrefix: jupyterhub/k8s-

  ... could instead be thought like whats below before a merge...

  charts:
    jupyterhub:
      imagePrefix: jupyterhub/k8s-

First I want to say awesome project!

I'm trying to build a chart repo which currently doesn't have any images. The README says images are optional however I am unable to get it to run without them.

If I omit imagePrefix and images I get:

Traceback (most recent call last):
  File "/opt/boxen/homebrew/bin/chartpress", line 11, in <module>
    load_entry_point('chartpress==0.2.0.dev0', 'console_scripts', 'chartpress')()
  File "/opt/boxen/homebrew/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/chartpress.py", line 204, in main
    prefix=chart['imagePrefix'],
  File "/opt/boxen/homebrew/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ruamel/yaml/comments.py", line 702, in __getitem__
    return ordereddict.__getitem__(self, key)
KeyError: 'imagePrefix'

If I set imagePrefix then I get:

Traceback (most recent call last):
  File "/opt/boxen/homebrew/bin/chartpress", line 11, in <module>
    load_entry_point('chartpress==0.2.0.dev0', 'console_scripts', 'chartpress')()
  File "/opt/boxen/homebrew/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/chartpress.py", line 205, in main
    images=chart['images'],
  File "/opt/boxen/homebrew/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ruamel/yaml/comments.py", line 702, in __getitem__
    return ordereddict.__getitem__(self, key)
KeyError: 'images'

If I set images to None I get:

Traceback (most recent call last):
  File "/opt/boxen/homebrew/bin/chartpress", line 11, in <module>
    load_entry_point('chartpress==0.2.0.dev0', 'console_scripts', 'chartpress')()
  File "/opt/boxen/homebrew/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/chartpress.py", line 208, in main
    push=args.push,
  File "/opt/boxen/homebrew/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/chartpress.py", line 65, in build_images
    for name, options in images.items():
AttributeError: 'NoneType' object has no attribute 'items'

If I set images to an empty list (images: []) I get:

Traceback (most recent call last):
  File "/opt/boxen/homebrew/bin/chartpress", line 11, in <module>
    load_entry_point('chartpress==0.2.0.dev0', 'console_scripts', 'chartpress')()
  File "/opt/boxen/homebrew/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/chartpress.py", line 208, in main
    push=args.push,
  File "/opt/boxen/homebrew/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/chartpress.py", line 65, in build_images
    for name, options in images.items():
AttributeError: 'CommentedSeq' object has no attribute 'items'

I pushed a 0.3.2 tag after updating the CHANGELOG.md and updated the README.md to reflect changes to the provided chartpress --help output. These are the changes between 0.3.1...0.3.2. I also added 9ded86b in order to return to a developer version 0.3.3.dev.

Is it okay if we publish this tag to PyPI? The current PyPI maintainers of the PyPI package chartpress are @minrk, @willingc and @yuvipanda.

If it seems OK, then this is my understanding of what would need to be done to publish the new package on PyPI.

# checkout the tag
git checkout 0.3.2

# package
python3 setup.py bdist_wheel

# upload to PyPI
twine upload dist/chartpress-0.3.2-py3-none-any.whl

Now that we check for local/remote existence of a image, we may end up failing to rebuild images if we want. I imagine a situation where we may want to get a fresher version of Ubuntu as base image for the hub image for example (FROM ubuntu:18.04), due to a security issue or similar, and that we want to re-push an update manually sometime.

I'm not sure.

Proposed change

To somehow enable chartpress to do various things just before packaging a Helm chart

• Copy a README.md file and images to the Helm chart folder to be packaged.
• Transform schema.yaml in z2jh to become a Helm 3 compliant values.schema.json for automatic validation.

Alternative options

To run such script separately before packaging a chart to be pushed.

Who would use this feature?

JupyterHub/BinderHub/Pebble in order to provide a README and values.schema.json.

(Optional): Suggest a solution

Perhaps to provide a hook to run?

While writing this down I started thinking it can make more sense to not embed this logic in chartpress, but instead run such logic separately before doing chartpress --publish as adding it to chartpress may not really help... Hmm...

Test against helm2 and helm3

I want a run of chartpress --reset to make someone that has developed in z2jh/binderhub locally be able to run git add . and then git commit instead of ending up to first clean up the changes of using chartpress to develop locally just because chartpress has modified Chart.yaml and values.yaml.

When we run chartpress --reset, we modify Chart.yaml and values.yaml. The configuration option resetTag that defaults to set-by-chartpress is only used within values.yaml while in Chart.yaml another logic is used. The logic is simple:

        if reset:
            version = chart['version'].split('-')[0]

In other words, it will take something like 0.9-asdf123 and make it 0.9. We currently have 0.9-dev. I think we should simply write 0.9 or specifically 0.9.0. Or, make chartpress append -dev like we do in z2jh (but not in binderhub's Helm chart's Chart.yaml).

It could be nice to reset to x.y.z-dev to make it clear that x.y.z isn't released yet.

In the event that the git short hash of a repository is something like 01234567, then the image tag is serialized unquoted, resulting in a broken values.yaml.

This is a bit hard to reproduce, as it would happen on only ~0.4% of the commits.

You can reproduce the issue by running:

git clone https://github.com/SwissDataScienceCenter/renku-ui
cd renku-ui
git checkout -b test 0192117
cd helm-chart
chartpress

Then, you can check the values.yaml file contents:

cat renku-ui/values.yaml

[...]
image:
  name: renku/renku-ui
  repository: renku/renku-ui
  tag: 0192117
  pullPolicy: IfNotPresent
[...]

As you can see, this will result in image.tag equals to 192117 and not 0192117.

A potential fix would be to always quote sha1 tags.

The image build context is very wide for binderhub, and as I don't realize a better option, it makes sense to me that we are allowed to specify that we want to build context to be excluded from the paths that impacts the decision on what version to set for the image and chart based on if they have changed or not.

Example:

Binderhub has the structure:

/helm-chart/chartpress.yaml
/helm-chart/binderhub (helm chart)
/helm-chart/images/binderhub/Dockerfile
/binderhub (python package)
/README.md

And the configuration declare a build context path of .. relative to the chartpress.yaml file's location which means it specifies the root structure of the repo. That means that a change to ANYTHING should cause a bump to the version of both the image and chart, and that is quite strict.

We could perhaps allow this path dependency on build context to be opt-out and allow manual specification of the paths that matters using the existing paths field of an image in chartpress.yaml.

This would allow binderhub helm chart to not rebuild on documentation changes etc.

@jhamman @scottyhq, I recall there was some issue with a dependency chart that didn't update. I imagine that it may be very relevant to have the --publish-chart flag make a call to helm package including the --dependency-update flag. Currently, we don't.

It probably makes sense to have this run as default within chartpress.

Usage:
  helm package [flags] [CHART_PATH] [...]

Flags:
      --app-version string   set the appVersion on the chart to this version
  -u, --dependency-update    update dependencies from "requirements.yaml" to dir "charts/" before packaging
  -d, --destination string   location to write the chart. (default ".")
  -h, --help                 help for package
      --key string           name of the key to use when signing. Used if --sign is true
      --keyring string       location of a public keyring (default "/home/erik/.gnupg/pubring.gpg")
      --save                 save packaged chart to local chart repository (default true)
      --sign                 use a PGP private key to sign this package
      --version string       set the version on the chart to this semver version

Bug description

If I make any edit to chartpress.yaml, for eg. add a new image definition or edit the the valuesPath for an existing image, I expect it to not cause a bump in version numbers for all the images or fresh re-builds, but this seems to be what is happening.

Steps to reproduce:

• Create chartpress.yaml referencing images to build
• Run chartpress - images build
• Make changes to one particular image or unrelated to the images folder
• Run chartpress - does not rebuild any images that it does not need to.
• Make an edit to chartpress.yaml - eg. change the valuesPath for an existing image

Expected behaviour

I expect chartpress to not rebuild any images since none of the contents of the images themselves have changed.

Actual behaviour

Chartpress now updates the version and rebuilds all images, even though nothing has actually changed in the images folder, just a change in the chartpress.yaml file.

How to reproduce

Described above - let me know if it helps to create a repository with a simplified test case or so.

Your personal set up

Encountered this on this repo https://github.com/developmentseed/osm-seed with chartpress running as part of a Github actions workflow, but I was able to reproduce this locally as well - ubuntu 20.04, running chartpress version 0.6.0.

(Thank you for an extremely useful project - let me know if I can help narrow this issue down).

It seems that updates from ruamel.yaml continue to affect chartpress.

Related: #17

Both issues come from the changes in ruamel.yaml changing the hierarchy of yaml objects.

Going back to one of the proposals in #24

Proposed change

Allow chartpress to be run from a different folder than that where the Helm chart are stored.

Alternative options

In some version before 1.0 it was possible to specify the name of a chart by including the folder where it is placed w.r.t. the chartpress config file (e.g. helm-chart/name-of-my-chart), however with the introduction of the version checking when publishing this is no longer possible as the "name" of the chart in the config (containing the folder) does not mach the name of the published chart.

Who would use this feature?

Anyone that want to be able to run chartpress from the root of a repository where the chart(s) are stored in a subfolder, which is a reasonable structure for any repository that holds both the code of an application and its Helm chart.

(Optional): Suggest a solution

This could be achieved by allowing an optional parameter in the configuration file to specify a path where to find the Helm charts and then simply have chartpress use that as cwd.

If there is consensus about this request and the proposed implementation I might be able to open a PR myself to address it.

I've found a reproducible issue since #52 relating to the use of chartpress. Apparently, if we use chartpress to publish a tagged commit, like 0.9.0-alpha.1, and later use chartpress to publish a later commit, it appears that Helm chart repository's index.yaml will get its entry about 0.9.0-alpha.1 replaced by 0.9.0-alpha.1+001.g152b8c9a, instead of having it added alongside...

jupyterhub/helm-chart@29f7cbe#diff-43a27642790006c93fea79f384f23b4fL2858-R2859

Reproduction

You need chartpress and helm, where helm is initialized with helm init --client-only.

pip install chartpress=0.4.2

rm -rf /tmp/{z2jh,helm-chart,index}
mkdir /tmp/{z2jh,helm-chart,index}

pushd /tmp/helm-chart
git clone https://github.com/jupyterhub/helm-chart . --no-checkout 
git checkout 6c4de08
popd

pushd /tmp/index
git init
helm repo index . --merge /tmp/helm-chart/index.yaml --url https://jupyterhub.github.io/helm-chart
git add index.yaml
git commit -m "index.yaml initial state: with 0.9.0-alpha.1"
rm --interactive=never /tmp/index/*
popd



pushd /tmp/z2jh
git clone https://github.com/jupyterhub/zero-to-jupyterhub-k8s . --no-checkout
git checkout 83af061d
chartpress --skip-build
helm package ./jupyterhub --destination /tmp/index
mv /tmp/index/jupyterhub-0.9.0-alpha.1+001.g152b8c9.tgz /tmp/index/jupyterhub-0.9.0-alpha.1_001.g152b8c9.tgz
popd

pushd /tmp/index
helm repo index . --merge /tmp/helm-chart/index.yaml --url https://jupyterhub.github.io/helm-chart
git add index.yaml
git commit -m "merge with 0.9.0-alpha.1+n.sha overrode 0.9.0-alpha.1 entry"
cp /tmp/index/* /tmp/helm-chart/
rm --interactive=never /tmp/index/*
#rm /tmp/index/index.yaml
popd



pushd /tmp/z2jh
git checkout 0.9.0-alpha.1
chartpress --skip-build
helm package ./jupyterhub --destination /tmp/index
popd

pushd /tmp/index
helm repo index . --merge /tmp/helm-chart/index.yaml --url https://jupyterhub.github.io/helm-chart
rm --interactive=never /tmp/index/jupyterhub-*
git add index.yaml
git commit -m "Should not do anything."

Reproduction results

   - apiVersion: v1
     appVersion: 1.0.1dev
-    created: "2019-10-17T22:17:04.353205591Z"
+    created: "2019-10-19T20:01:39.379225412+02:00"
     description: Multi-user Jupyter installation
-    digest: 4835bf4b9d3130ad5747052a0eec50f1e5b2ef5133b9084d3a4e5a9f3602cc3e
+    digest: 9789053b0136b06fe3be4e429c1b57d69cc097faac80cdaf21239afa36f650a7
     home: https://z2jh.jupyter.org
     icon: https://jupyter.org/assets/hublogo.svg
     kubeVersion: '>=1.11.0-0'
@@ -2847,8 +2847,8 @@ entries:
     - https://github.com/jupyterhub/zero-to-jupyterhub-k8s
     tillerVersion: '>=2.11.0-0'
     urls:
-    - https://jupyterhub.github.io/helm-chart/jupyterhub-0.9.0-alpha.1.tgz
-    version: 0.9.0-alpha.1
+    - https://jupyterhub.github.io/helm-chart/jupyterhub-0.9.0-alpha.1+001.g152b8c9.tgz
+    version: 0.9.0-alpha.1+001.g152b8c9

I started investigating jupyterhub/mybinder.org-deploy#1706 by checking the changes to index.yaml in https://jupyterhub.github.io/helm-chart/

I noticed the changes were surprisingly large (I don't know if this is related to the above issue).

E.g. compare
jupyterhub/helm-chart@f37566f
which seems to rewrite a large part of index.yaml, 1788 lines changed
with the previous change
jupyterhub/helm-chart@c945feb
which just adds the chart, 17 lines changed

I think this is roughly where the change in behaviour started to occur.

I haven't had time to dig any further yet.

pangeo-data/helm-chart's latest git tag was v0.2.2, this gave us a v0.2.2-n###.h###### version that is an invalid SemVer 2 version because it leads with v, and this makes it an invalid Helm 3 version.

Here is an reported issue about the build context of the images isn't used correctly to determine the chart version: jupyterhub/binderhub#919, and the workaround that had to be implemented was jupyterhub/binderhub#1014.

But, it was fixed in #68, which has not been released yet, so simply making a new release will solve this issue.

I raised an issue against the zero-to-jupyterhub-k8s project jupyterhub/zero-to-jupyterhub-k8s#1038 but if I've understood the build process correctly the change is probably across both this project (to support version tags) and the zero-to-jupyterhub-k8s project (to modify the CI scripts).

Looks like #3 has popped up again.

We use chartpress to publish the Dask helm chart but not the Dask docker images. Therefore we have no images section in our config. This seems to have broken with the latest release.

Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.6.7/bin/chartpress", line 10, in <module>
    sys.exit(main())
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/chartpress.py", line 693, in main
    for image_name, image_config in chart['images'].items():
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/ruamel/yaml/comments.py", line 753, in __getitem__
    return ordereddict.__getitem__(self, key)
KeyError: 'images'

Testing with the latest binderhub helm chart dev releases:

$ helm version`
version.BuildInfo{Version:"v3.0.0", GitCommit:"e29ce2a54e96cd02ccfce88bee4f58bb6e2a28b6", GitTreeState:"clean", GoVersion:"go1.13.4"}

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "jupyterhub" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈ 

This works:

helm upgrade binder-test jupyterhub/binderhub --install --version=0.2.0-c04966f --namespace=binder-test -f secret.yaml -f config.yaml

This fails:

helm upgrade binder-test jupyterhub/binderhub --install --version=0.2.0-028.9ba1fc3 --namespace=binder-test -f secret.yaml -f config.yaml
Error: failed to download "jupyterhub/binderhub" (hint: running `helm repo update` may help)

This also fails:

$ cd helm-chart/binderhub/
$ cat requirements.yaml 
dependencies:
- name: jupyterhub
  version: "0.9.0-alpha.1.071.b0d70c2"
  repository: "https://jupyterhub.github.io/helm-chart"
$ helm dependency update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "jupyterhub" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈
Error: dependency "jupyterhub" has an invalid version/constraint format: constraint Parser Error

It seems to me that we have some invalid logic regarding use of name or repository as a key when updating values.yaml files with image tags.

chartpress logic for whether to build/push an image currently doesn't check whether that image:tag exists. Instead, it assumes it does if --commit-range is specified and the last change is outside the range. This can be a faulty assumption in a number of cases, including:

• previous build or publish failed
• image prefix changed
• CI didn't build the relevant commit (e.g. due to aborting superseded builds)

Instead, we should modify the build-triggering condition to include a check for existence of the image:

• if image changed in commit range (current)
• or image doesn't exist

Alternately, we could skip the "didn't change" check, and only check for image existence (avoids triggering build if called with the same --commit-range twice).

I thought I'd let you know what I've been doing with Chartpress. Originally I planned to use it to publish a repo with multiple independently versioned Helm Charts with Travis following a similar process to Z2JH, but for releases only, then moved on to the idea of automating tagging based on when the Chart release version was changed.

I wasn't entirely happy with giving Travis full push access to a GitHub repo, then remembered I've got beta access to GitHub Actions so thought I'd try that out.

The result is master...manics:devel
The target repo is https://github.com/manics/kubernetes-omero

Since it's heavily modified for my own workflow I thought I'd mention it instead of opening PRs, but if you see something useful I'm happy to modify it if necessary to get it into this repo.

Running chartpress --list-images will also update Chart.yaml with a version which I consider unwanted.