openshift-eng / doozer Goto Github PK

View Code? Open in Web Editor NEW

14.0 14.0 26.0 5.01 MB

License: Apache License 2.0

Dockerfile 1.06% Shell 1.66% Python 97.26% Makefile 0.02%

doozer's People

Contributors

Stargazers

Watchers

doozer's Issues

Error in state.yaml after failed rebase

OS: rhel7
Arch: s390x
Doozer version: v0.4.12

I'm seeing an error when attempting to build 3.11. After cloning all repos and then starting a rebase, golang-github-openshift-oauth-proxy fails to rebase, giving the following error (full output here: http://pastebin.test.redhat.com/759623):

The following non-critical images failed to rebase:
golang-github-openshift-oauth-proxy

doozer logs an error (see line 140 in above pastebin) but the error output itself doesn't appear in the output. This also leads to a malformed state file:

...
images:rebase:
  images:
    aos3-installation: true
    atomic-openshift-cluster-autoscaler: true
    atomic-openshift-descheduler: true
    atomic-openshift-metrics-server: true
    atomic-openshift-node-problem-detector: true
    automation-broker-apb: true
    cluster-monitoring-operator: true
    configmap-reload: true
    csi-attacher: true
    csi-driver-registrar: true
    csi-livenessprobe: true
    csi-provisioner: true
    efs-provisioner: true
    golang-github-openshift-oauth-proxy: ''
    golang-github-prometheus-alertmanager: true
...

Where the expected false is replaced by single quotes. This causes any build commands to fail with:

[dorzel@devel42: oit-BB_eWT.tmp]$ doozer --latest-parent-version --local -i openshift-enterprise-registry images:build
/home/nfs/dorzel/.config/doozer/settings.yaml
2019-04-29 14:51:14,505 INFO Using branch from group.yml: rhaos-3.11-rhel-7
2019-04-29 14:51:14,876 INFO [containers/openshift-enterprise-registry] Distgit directory already exists; skipping clone: /tmp/oit-BB_eWT.tmp/distgits/containers/openshift-enterprise-registry
Traceback (most recent call last):
  File "/usr/bin/doozer", line 1831, in <module>
    main()
  File "/usr/bin/doozer", line 1812, in main
    cli(obj={})
  File "/usr/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/bin/doozer", line 801, in images_build_image
    img_obj = runtime.image_map[img]
KeyError: 'golang-github-openshift-oauth-proxy'

manually editing the state.yaml value to be false resolves this issue.

random issues hit while demoing this on a new install

git clone assumes you have accepted github's host key
git clone assumes that you have an ssh key that will allow cloning from github via ssh
rhpkg switch-branch attempted when building something that was already cloned and rebased
Ctrl-C on images:build cloning did not work for a long time
there are more -devel libs needed than we list
the error when your config.yaml fails to parse is not particularly friendly

OS_GIT_MAJOR: unbound variable error on local OCP builds

I'm seeing this error on any container built out of global_ose:

2019-03-26 14:32:47,546 ERROR [containers/openshift-enterprise-cli] Error running imagebuilder -mount : out=--> FROM openshift/golang-builder:1.10 as builder
--> WORKDIR /go/src/github.com/openshift/origin
--> COPY . .
--> RUN make build WHAT=cmd/oc
hack/build-go.sh cmd/oc 
/go/src/github.com/openshift/origin/hack/lib/constants.sh: line 110: OS_GIT_MAJOR: unbound variable
[ERROR] hack/lib/constants.sh:110: `ldflags+=($(os::build::ldflag "${OS_GO_PACKAGE}/pkg/version.majorFromGit" "${OS_GIT_MAJOR}"))` exited with status 1.
[ERROR] hack/lib/build/binaries.sh:172: `version_ldflags=$(os::build::ldflags)` exited with status 1.
[ERROR] hack/lib/build/binaries.sh:150: `local -a binaries=("$@")` exited with status 1.
[ERROR] hack/build-go.sh exited with code 1 after 00h 00m 02s
make: *** [build] Error 1
running 'make build WHAT=cmd/oc' failed with exit code 2

OS: fedora 29 and rhel7
Arch: x86_64
Doozer: v0.4.4

Steps to reproduce:

clean working directory
settings =

#Global option overrides that can only be set from settings.yaml
global_opts:
  rhpkg_push_timeout:
  distgit_threads:
  rhpkg_clone_timeout:

#Persistent working directory to use
working_dir: /home/jeyoung/doozerdata

#Git URL or File Path to build data
data_path:  https://github.com/openshift/ocp-build-data

#Sub-group directory or branch to pull build data
group: openshift-4.0 #(or openshift4.1)

#Username for running rhpkg / brew / tito
user: jeyoung

doozer --local --latest-parent-version -i openshift-enterprise-cli images:rebase
doozer --local --latest-parent-version -i openshift-enterprise-cli images:build

can't interrupt image push

$ doozer --group openshift-4.0 -i ose-multus-admission-controller images:push --to defaults
[...]
2019-01-14 21:10:54,375 INFO [containers/ose-multus-admission-controller] Pushing image to mirror [retry=0]: defaults/openshift/ose-multus-admission-controller:v4.0.0-0.139.0
2019-01-14 21:10:55,270 INFO [containers/ose-multus-admission-controller] Error pushing image -- retrying in 60 seconds
^C2019-01-14 21:11:33,786 WARNING SIGINT received, signaling threads to terminate...
^C2019-01-14 21:11:55,315 INFO [containers/ose-multus-admission-controller] Pushing image to mirror [retry=1]: defaults/openshift/ose-multus-admission-controller:v4.0.0-0.139.0
2019-01-14 21:11:56,215 INFO [containers/ose-multus-admission-controller] Error pushing image -- retrying in 60 seconds
^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C2019-01-14 21:12:56,274 INFO [containers/ose-multus-admission-controller] Pushing image to mirror [retry=2]: defaults/openshift/ose-multus-admission-controller:v4.0.0-0.139.0

UnicodeDecodeError when building 3.11 openshift-ansible RPM

openshift-ansible.spec contains non-ascii character which breaks doozer with Python 2.7.

During our daily release routine, openshift-ansible is built by build/ocp3 job rather than Doozer. This error will happen if you use the build/custom job to build openshift-ansible which uses Doozer:

E           Traceback (most recent call last):
E             File "/usr/lib64/python2.7/runpy.py", line 174, in _run_module_as_main
E               "__main__", fname, loader, pkg_name)
E             File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
E               exec code in run_globals
E             File "/workspaces/doozer/doozerlib/cli/__main__.py", line 2289, in <module>
E               main()
E             File "/workspaces/doozer/doozerlib/cli/__main__.py", line 2270, in main
E               cli(obj={})
E             File "/home/dev/.local/lib/python2.7/site-packages/click/core.py", line 764, in __call__
E               return self.main(*args, **kwargs)
E             File "/home/dev/.local/lib/python2.7/site-packages/click/core.py", line 717, in main
E               rv = self.invoke(ctx)
E             File "/home/dev/.local/lib/python2.7/site-packages/click/core.py", line 1137, in invoke
E               return _process_result(sub_ctx.command.invoke(sub_ctx))
E             File "/home/dev/.local/lib/python2.7/site-packages/click/core.py", line 956, in invoke
E               return ctx.invoke(self.callback, **ctx.params)
E             File "/home/dev/.local/lib/python2.7/site-packages/click/core.py", line 555, in invoke
E               return callback(*args, **kwargs)
E             File "/home/dev/.local/lib/python2.7/site-packages/click/decorators.py", line 64, in new_func
E               return ctx.invoke(f, obj, *args, **kwargs)
E             File "/home/dev/.local/lib/python2.7/site-packages/click/core.py", line 555, in invoke
E               return callback(*args, **kwargs)
E             File "/workspaces/doozer/doozerlib/cli/__main__.py", line 253, in rpms_build
E               results = results.get()
E             File "/usr/lib64/python2.7/multiprocessing/pool.py", line 572, in get
E               raise self._value
E           doozerlib.runtime.WrapException: 
E           Original traceback:
E           Traceback (most recent call last):
E             File "doozerlib/runtime.py", line 99, in wrapper
E               return func(*args, **kwargs)
E             File "doozerlib/runtime.py", line 111, in wrapper
E               return func(*args)
E             File "/workspaces/doozer/doozerlib/cli/__main__.py", line 251, in <lambda>
E               version, release, terminate_event, scratch, local=runtime.local, dry_run=dry_run),
E             File "doozerlib/rpmcfg.py", line 351, in build_rpm
E               self.update_spec()
E             File "doozerlib/rpmcfg.py", line 244, in update_spec
E               if "%global os_git_vars " in lines[i]:
E           UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 70: ordinal not in range(128)

License on pypy is INTERNAL

On https://github.com/openshift/doozer/blob/master/setup.py#L24 we say that license is internal, but then we publish the code... shouldn't it be adapted and add a proper LICENSE file to this repo?

Thanks

doozer won't overwrite dotfiles on rebase

When doing a rebase, dotfiles like .dockerignore in the source will be copied over to the dist-git repo, but only if they haven't before.

My test case is:

doozer  --latest-parent-version -i  ose-multus-admission-controller  images:rebase  --version v4.0.0 --release 0.139.1 -m "testing"

In the history of this repo, the .dockerignore was added, and then later modified. The modifications were never automatically synced, leading to quite a lot of difficulty in tracking down why stuff was missing from the build.

I think we need to tread lightly here as we don't necessarily want to copy everything from the source; certainly not .., and certainly not .git. Not sure about .gitignore and presumably not .oit.

I ran into this and have been looking through the code and still don't quite understand why it happens. rsync copies everything. I don't understand how it's copying something but only if it's not overwriting the destination (there are rsync options for that but I don't see them being used?)

Error getting data repo status: fatal: Not a git repository

In my doozer config I set:

data_path: https://github.com/openshift/ocp-build-data/ <-- with trailing slash

This causes a confusing error:

Error getting data repo status: fatal: Not a git repository (or any of the parent directories): .git

It works if I remove the trailing /

missing doozerlib after update

I updated doozer on my ppc64le build machine (rhel7), and it seems i'm missing a library:

[jeyoung@b ~]$ doozer version
Traceback (most recent call last):
  File "/bin/doozer", line 6, in <module>
    from doozerlib import state
ImportError: cannot import name state

openshift-eng / doozer Goto Github PK

doozer's People

Contributors

Stargazers

Watchers

Forkers

doozer's Issues

Error in state.yaml after failed rebase

random issues hit while demoing this on a new install

OS_GIT_MAJOR: unbound variable error on local OCP builds

can't interrupt image push

UnicodeDecodeError when building 3.11 openshift-ansible RPM

License on pypy is INTERNAL

doozer won't overwrite dotfiles on rebase

Error getting data repo status: fatal: Not a git repository

missing doozerlib after update

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent