cloudposse / build-harness Goto Github PK
View Code? Open in Web Editor NEWCollection of Makefiles to facilitate building Golang projects, Dockerfiles, Helm charts, and more
Home Page: https://cloudposse.com/accelerate
License: Apache License 2.0
Collection of Makefiles to facilitate building Golang projects, Dockerfiles, Helm charts, and more
Home Page: https://cloudposse.com/accelerate
License: Apache License 2.0
Could the packages/install
target work with custom git repos?
I was playing with idea of having a fork of https://github.com/cloudposse/packages
.
In general I like the idea of submitting PR's back to this repo, but wondering if these will get a bit too specific to my own use-cases. For now I've got my own fork that I would like to use with build-harness to install packages for my project needs.
Would love to set an variable in the top-level build-harnessMakefile
that could control this:
export PACKAGES_VERSION = master
export PACKAGES_VENDOR ?= cloudposse
export PACKAGES_REPO ?= packages.git
Then in the modules/packages/install
## Install packages
packages/install:
@if [ ! -d $(PACKAGES_PATH) ]; then \
echo "Installing packages ref=$(PACKAGES_VERSION)..."; \
rm -rf $(PACKAGES_PATH); \
$(GIT) clone -c advice.detachedHead=false --depth=1 -b $(PACKAGES_VERSION) https://github.com/$(PACKAGES_VENDOR)/$(PACKAGES_REPO) $(PACKAGES_PATH); \
rm -rf $(PACKAGES_PATH)/.git; \
fi
Currently I do this as a module override, but it generates a lot of noise:
... /build-harness-extensions/modules/packages/Makefile:9: warning: overriding recipe for target 'packages/install'
... /build-harness/modules/packages/Makefile:15: warning: ignoring old recipe for target 'packages/install'
1 - Is there any way to avoid printing out those warnings when overriding recipes?
2 - Would you accept a PR for making package-vendor/repo configurable?
The script to generate terraform docs uses an intermediate to pre process files with awk
as work around of a limitation in the terraform-docs
tool for terraform version 0.12. The temporary file created is not a unique file, it always generate the following file. /tmp/terraform-docs-XXXXXXXXXX.tf
In case the generate file is not cleared or in recursive usages mode the script fails frequent.
By removing .tf
from the template it always will use a unique file
Have a question? Please checkout our Slack Community or visit our Slack Archive.
If this repo can be used with other repositories for other organizations, both private and public, the watermarks and references to cloudposse, and all areas in general should be overwritable without having to use a custom README.md.gotmpl
All parts of the README.md
should be overwritable using only the README.yaml
Usage in a private module for a private organization
Configuration options for the yaml file that will overwrite all areas and documentation with an example yaml configuration for a private module.
Using a custom README.md.gotmpl
Related #223
When running the make terraform:lint
command on Mac I get an error that relates to the xargs options.
$ make terraform:lint
xargs: illegal option -- -
usage: xargs [-0opt] [-E eofstr] [-I replstr [-R replacements]] [-J replstr]
[-L number] [-n number [-x]] [-P maxprocs] [-s size]
[utility [argument ...]]
According to these resources the option --no-run-if-empty
is a GNU option only.
Can you remove that or look at weather it is really needed?
make readme
installs gomplate to vendor/gomplate
. However, when executing, https://github.com/cloudposse/build-harness/blob/master/modules/readme/Makefile#L32, the script tries to call the system version of gomplate instead of the installed version of gomplate.
For systems that do not have gomplate installed, the make readme fails. For example:
jonathanstimbp:bastion jonathan$ make readme
* Installing packages master...
Cloning into '/Users/jonathan/code/oss/bastion/build-harness/vendor/packages'...
remote: Enumerating objects: 921, done.
remote: Counting objects: 100% (921/921), done.
remote: Compressing objects: 100% (454/454), done.
remote: Total 921 (delta 316), reused 592 (delta 232), pack-reused 0
Receiving objects: 100% (921/921), 11.22 MiB | 26.42 MiB/s, done.
Resolving deltas: 100% (316/316), done.
* Installing gomplate to /Users/jonathan/code/oss/bastion/build-harness/vendor
mkdir -p -m a+rX /Users/jonathan/code/oss/bastion/build-harness/vendor
curl --retry 3 --retry-delay 5 --fail -sSL -o /Users/jonathan/code/oss/bastion/build-harness/vendor/gomplate https://github.com/hairyhenderson/gomplate/releases/download/v3.11.4/gomplate_darwin-amd64 && chmod +x /Users/jonathan/code/oss/bastion/build-harness/vendor/gomplate
make: gomplate: No such file or directory
make: *** [readme/build] Error 1
make readme
should succeed, utilizing the vendor/gomplate version instead of the system version. Perhaps the PATH variables are not being properly set or the script needs to be calling the local version instead.
Do not have gomplate installed on the machine. Run make readme
on any project.
No response
No response
No response
When docker
is installed in a path with spaces, it brakes the Makefile
make[2]: *** No rule to make target `/Users/hansd/Library/Application'. Stop.
make[1]: *** [/Users/hansd/Library/Application] Error 2
just runs
$ which docker
/Users/hansd/Library/Application Support/multipass/bin/docker
Eg this occurs when using multipass on a Mac (https://multipass.run/docs/docker-tutorial)
No response
issue:
build-harness/modules/docker/Makefile
Line 1 in b1e1c0f
ERROR: Unable to connect to helm server
make: *** [//build-harness/modules/helm/Makefile:24: helm/toolbox/upsert] Error 1
Have a question? Please checkout our Slack Community or visit our Slack Archive.
It would be nice to hide or disable unnecessary targets so it doesn't overwhelm the user.
The targets shown should be applicable to the module.
Not to overwhelm the user with targets that are not applicable.
Hide or disable modules.
N/A
N/A
which: no glide in (/usr/local/bin:/usr/bin:/bin:/sbin:/usr/local/sbin:/usr/sbin)
which: no go in (/usr/local/bin:/usr/bin:/bin:/sbin:/usr/local/sbin:/usr/sbin)
build-harness
make terraform/install
Installing Terraform 1.0.11 (linux) from https://releases.hashicorp.com/terraform/1.0.11/terraform_1.0.11_linux_aarch64.zip
curl: (22) The requested URL returned error: 404
Installing Terraform 1.0.11 (linux) from https://releases.hashicorp.com/terraform/1.0.11/terraform_1.0.11_linux_arm64.zip
See above
No response
Running in a linux docker container
No response
Have a question? Please checkout our Slack Community or visit our Slack Archive.
It would be nice to do something like this.
make readme ./path/to/our/README.md.gotmpl
A way to overwrite the template with documentation on how to do so
It would be nice to overwrite the README.md.gotmpl
for private modules as we may not need some of the cloudposse watermarks but would still like to use the buildharness.
See Describe the Feature
Only way I can think of doing this now would be extend the harness with our own README module. If that doesn't work, then we'd have to try forking the repo.
N/A
//build-harness/modules/git/bootstrap.Makefile:2: disabling git bootstrapping
//build-harness/modules/helm/Makefile.repo:15: unable to extrapolate current BRANCH
//build-harness/modules/semver/Makefile:58: disabled semver processing
//build-harness/modules/travis/Makefile:12: unable to extrapolate current BRANCH
//build-harness/modules/git/bootstrap.Makefile:2: disabling git bootstrapping
//build-harness/modules/helm/Makefile.repo:15: unable to extrapolate current BRANCH
//build-harness/modules/semver/Makefile:58: disabled semver processing
//build-harness/modules/travis/Makefile:12: unable to extrapolate current BRANCH
make codefresh/sync/pipelines
fails when repository does not exist and the codefresh account being used does not have a git integration named "github". See here
This should either be configurable via environment or select a context of type git.github
using codefresh get context --type git.github
Example interface:
make github/push-artifacts ARTIFACTS=files/ GIT_TAG=0.1.0
Use one of these projects. Pick the best tool for our use-case.
Add the binary package to cloudposse/packages
Have a question? Please checkout our Slack Community or visit our Slack Archive.
make list
to show all targets
Shows all targets
I forget the targets so I have to go back to the README.md
make list
Explain what alternative solutions or features you've considered.
Ref: https://stackoverflow.com/questions/4219255/how-do-you-get-the-list-of-targets-in-a-makefile
I tried adding that to my Makefile
after the included script and when I ran make list
, it only returned:
travis/docker-login
travis/docker-tag-and-push
helm reset
), then install fails✓ (cpco-testing-admin) conf ⨠ make helm/toolbox/upsert
INFO: Perform attempt - 1
Error: could not find tiller
INFO: Helm version: matches tiller version: .
INFO: Initializing helm client...
Remove all references to git.io/build-harness
and refer tocloudposse.tools/build-harness
instead.
See our Git.io deprecation documentation for more details, including instructions on how to update.
GitHub announced that the git.io
redirector service is shutting down on 2022-04-29. GitHub only provided 4 days notice of this shutdown.
This means all references to git.io/build-harness
will stop working.
GitHub amended their decision to completely shut down git.io
(and amended the above linked blog post)), but you should still make the change when convenient in order to preserve functionality going forward.
The helm/chart/build/package
target is dependent on yq. Our builds run on amd64, ppc64le and s390x Linux, but on the latter two, running yq results in cannot execute binary file: Exec format error
presumably because it is the amd64 version.
The helm/install target should install the platform-specific version of yq.
Write my own target to package helm charts with no dependency on yq. Other alternatives are to use something like echo
to append the text that yq writes to values.yaml
. A more sophisticated approach would be to grep for the text and replace it if it exists (maybe using sed
), else append it.
To recreate:
(main) directory $ make readme/init
cp: .../directory/build-harness/templates/README.yaml: No such file or directory
README.yaml created!
Proposed fix:
templates/
in path from README_TEMPLATE_YAML
variable.build-harness/modules/readme/Makefile
Line 5 in 0957353
I found two issues within the changes added with PR #304.
The first is that the make terraform/get-plugins
target fails to execute properly with 127 exit code
❯ make init
exit 0
Cloning https://github.com/cloudposse/build-harness.git#master...
Cloning into 'build-harness'...
remote: Enumerating objects: 150, done.
remote: Counting objects: 100% (150/150), done.
remote: Compressing objects: 100% (122/122), done.
remote: Total 150 (delta 9), reused 81 (delta 5), pack-reused 0
Receiving objects: 100% (150/150), 94.82 KiB | 2.56 MiB/s, done.
Resolving deltas: 100% (9/9), done.
❯ make terraform/install
Installing Terraform 1.1.2 (linux) from https://releases.hashicorp.com/terraform/1.1.2/terraform_1.1.2_linux_amd64.zip
################################################################################################################# 100.0%
/home/jbouse/test/build-harness/vendor/terraform version
Terraform v1.1.2
on linux_amd64
+ provider registry.terraform.io/hashicorp/random v3.1.0
❯ make terraform/get-plugins
/bin/bash: cho: command not found
make: *** [/home/jbouse/test/build-harness/modules/terraform/Makefile:19: terraform/get-plugins] Error 127
The second is that the make clean
target fails execute properly
❯ make clean
if [ -d "/home/jbouse/test/build-harness" ]; then \
if [ -d build-harness ] && [ $(stat -f %i "/home/jbouse/test/build-harness") == $(stat -f %i build-harness) ]; then \
rm -rf build-harness; \
else \
echo Not removing build harness from "/home/jbouse/test/build-harness" because it appears to be shared.; \
echo If you are sure you want to delete it, run: ; \
echo ' rm -rf "/home/jbouse/test/build-harness"'; \
fi; \
fi
stat: cannot read file system information for '%i': No such file or directory
stat: cannot read file system information for '%i': No such file or directory
/bin/bash: line 1: [: too many arguments
Not removing build harness from /home/jbouse/test/build-harness because it appears to be shared.
If you are sure you want to delete it, run:
rm -rf "/home/jbouse/test/build-harness"
In the first instance simply expect to have the message output as a result
❯ make terraform/get-plugins
terraform -get-plugins no longer supported
In the second instance expect the build-harness
to either be deleted with no output or the expected error message output.
❯ make clean
❯
or
❯ make clean
Not removing build harness from /home/jbouse/path/to/shared/build-harness because it appears to be shared.
If you are sure you want to delete it, run:
rm -rf "/home/jbouse/path/to/shared/build-harness"
Steps to reproduce the behavior:
make init
make lint
or make terraform/get-plugins
make clean
regexp.Replace ".*" "" in
README.md` templatedefineDatasource
wouldn't be pollute output, it just temporary solutionact
should work with our gh actions
in this repo.
It would simplify workflow and allow to test changes before pushing.
Especially this flag is interesting for continuous testing --watch - watch the contents of the local repo and run when files change
act
[cloudposse-terraform-workflow/pre-commit] 🚀 Start image=node:12.6-buster-slim
[lint/readme ] 🚀 Start image=node:12.6-buster-slim
[cloudposse-terraform-workflow/pre-commit] 🐳 docker run image=node:12.6-buster-slim entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
[lint/readme ] 🐳 docker run image=node:12.6-buster-slim entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
[cloudposse-terraform-workflow/pre-commit] 🐳 docker cp src=/workspace/cloudposse.terraform-opsgenie/. dst=/github/workspace
[lint/readme ] 🐳 docker cp src=/workspace/cloudposse.terraform-opsgenie/. dst=/github/workspace
[lint/readme ] ⭐ Run actions/checkout@master
[lint/readme ] ✅ Success - actions/checkout@master
[lint/readme ] ⭐ Run cloudposse/[email protected]
[lint/readme ] ☁ git clone 'https://github.com/cloudposse/build-harness' # ref=0.39.0
[cloudposse-terraform-workflow/pre-commit] ⭐ Run actions/checkout@v1
[cloudposse-terraform-workflow/pre-commit] ✅ Success - actions/checkout@v1
[cloudposse-terraform-workflow/pre-commit] ⭐ Run actions/setup-python@v1
[cloudposse-terraform-workflow/pre-commit] ☁ git clone 'https://github.com/actions/setup-python' # ref=v1
[lint/readme ] ❌ Failure - cloudposse/[email protected]
Error: open /.cache/act/[email protected]/action.yaml: no such file or directory
Sample action.yaml
from actions-setup-python@v1
repo:
---
name: 'Setup Python'
description: 'Set up a specific version of Python and add the command-line tools to the PATH.'
author: 'GitHub'
inputs:
python-version:
description: "Version range or exact version of a Python version to use, using SemVer's version range syntax."
default: '3.x'
architecture:
description: 'The target architecture (x86, x64) of the Python interpreter.'
default: 'x64'
outputs:
python-version:
description: "The installed python version. Useful when given a version range as input."
runs:
using: 'node12'
main: 'dist/index.js'
branding:
icon: 'code'
color: 'yellow'
master
b/c that's what our examples showFound a bug? Maybe our Slack Community can help.
make 4.4 (oct 31 2022) and 4.4.1 causes issues with the git submodule resulting in an infinite loop
cc: @Nuru
It should not be in an infinite loop
/usr/bin/make init
brew install make
to get the latest 4.4 version/usr/local/bin/make init
✗ make SHELL='sh -x' init
+ pwd
+ pwd
+ '[' -d build-harness ']'
+ pwd
+ config --get remote.origin.url
+ sed 's/\.git$//g'
+ sed 's/git@\(.*\):/https:\/\/\1\//g'
+ show -s --format=%aN
+ show -s --format=%aN
+ config --get remote.origin.url
+ sed 's/\.git$//g'
+ sed 's/git@\(.*\):/https:\/\/\1\//g'
+ rev-parse --verify --short HEAD
+ config --get remote.origin.url
+ sed 's/\.git$//g'
+ sed 's/git@\(.*\):/https:\/\/\1\//g'
+ rev-parse --verify --short HEAD
+ rev-parse --verify --short HEAD
+ config --get remote.origin.url
N/A
Worked on debugging this with @dudymas on it! Thank you!
Downgrading back to make 4.3 is successful
Commenting out this line prevents the issue
Line 42 in 72ee8e2
And the git
module is the specific one that seems to be failing which causes repeating the include.
build-harness/modules/git/bootstrap.Makefile
Lines 11 to 17 in 72ee8e2
/usr/bin/make --version
/usr/bin/make all
if you do not want to use the older make but also want to pin to 4.3 using homebrew
# uninstall 4.4 make
brew uninstall make
# find your make.rb file
find / -name make.rb
# override the make.rb file with the 4.3 version
wget https://raw.githubusercontent.com/Homebrew/homebrew-core/9345e061435f18a91437cc5a3db34a90acbb9f1b/Formula/make.rb make.rb
# install
brew install make.rb
# finally pin
brew pin make
required
function to enforce required fields existpackages/reinstall/%
target to reinstall a packageHello,
I tried to use /bin/terraform-docs.awk to generate documentation with terraform-docs.
An error occured with this variable from provider aws, resource "aws_ecs_service", property "ordered_placement_strategy" :
variable "ordered_placement_strategy" {
type = list(object({
type = string
field = string
}))
description = "Ordered task placement strategy on EC2 instances"
default = [
{
type = "spread"
field = "attribute:ecs.availability-zone"
}
]
}
Due to presence of "type" in the object definition, and default value, awk transformation failed to retrieve a valid hcl0.11 content.
Transformation result :
variable "ordered_placement_strategy" {
type = "object"
type = "string"
description = "Ordered task placement strategy on EC2 instances"
default = [
{
type = "spread"
}
Thank you for the awk hack anyway 👍
In order to ensure a known version of build-harness
is run by various automation workflows, and to support development and testing of build-harness
, if you have the build-harness
project checked out from GitHub and it is in the same directory tree at the same or higher level than your project, the checked out version will be used instead of the latest version.
This leads to problems and confusion by end users when they have a checked out version and it is old.
When using the local checked out version of build-harness
, display:
More users are confused and "bitten" by this behavior than benefit from it. Therefore:
BUILD_HARNESS_AUTO_INIT
is "true", with corresponding informational outputLove your project. But i found a really small issue. The smallest.
What?
You're missing a file in template\README.yaml file for readme:init target to work.
If you run your target readme/init or something on a project that has no readme.yaml yet, an attempt is made to copy over this file from build.harness\templates but there is none.
Found a bug? Maybe our Slack Community can help.
Running a make clean
doesn't actually remove $(BUILD_HARNESS_PATH)
. It just echos the command to do so. Not sure if this is intentional or not...
I would think make clean
should actually remove the build-harness directory + possibly the .build-harness
file.
git clone https://github.com/cloudposse/terraform-aws-ssm-parameter-store
make
make clean
build-harness/
is not removed.Anything that will help us triage the bug will help. Here are some ideas:
master
github/download-public-release
to use github/download-private-release
GITHUB_TOKEN
exists.GITHUB_TOKEN
exists we use an authenticated call that has limits 5000 per day per IPFrom terrafrom v0.12+, terraform validate does not support -check-variables
option anymore. See the discussion in the thread. From v0.12+ it won't check variable by default. Is it possible to remove -check-variables
from module/terraform/Makefile
?
Expected to see when using terraform version 0.13+, there's no error when running make terraform/validate
Steps to reproduce the behavior:
export TERRAFORM_VERSION = 0.15.4
build-harness
(curl -sSL -o .build-harness "https://git.io/build-harness"; echo .build-harness
)make terraform/install
terraform/get-modules
terraform/validte
╷
│ Error: Failed to parse command-line flags
│
│ flag provided but not defined: -check-variables
╵
For more help on using this command, run:
terraform validate -help
make: *** [terraform/validate] Error 1
Anything that will help us triage the bug will help. Here are some ideas:
0.58.0
0.13+
help/%
targetShow all help for docker targets.
make help/docker
MIT
and GPL3
licensesREADME.md
Running make terraform/validate
on Terraform v0.12.X prints the following message:
Terraform 0.12 does not support validate without skipping variables
I would expect the terraform/validate
target to not care about this skipping variables
flag as it doesn't seem needed, but maybe I am missing something.
git clone https://github.com/cloudposse/terraform-aws-ssm-parameter-store
make
make terraform/install TERRAFORM_VERSION=0.12.24
make terraform/validate
tf validate
locally without the Makefile / Build-harness -- I get the expected, useful output:Error: Missing required argument
The argument "region" is required, but was not set.
When running make help
or make help/all
some of the targets are not displayed, specifically docs/terraform.md
and docs/targets.md
. Also awk
complains about the regex being used.
Available targets:
awk: cmd. line:1: warning: regexp escape sequence `\_' is not a known regexp operator
The make help
or make help/all
should display all available targets and awk
should not error.
Steps to reproduce the behavior:
make init
make help
or make help/all
Anything that will help us triage the bug will help. Here are some ideas:
I tested in both BASH and ZSH just to rule out my ZSH was doing something funky.
Changed the regex used by awk
here, to the below regex:
/^[a-zA-Z\\_0-9%:\\\/-\\.]+:/
0.0.0.0
is invalid, but 0.0.0-rc1.0
is valid)when I run commands that would use docker containers (i.e. make pr/readme
) I get this error:
docker: Error response from daemon: manifest for cloudposse/build-harness:sha-a1e5103 not found: manifest unknown: manifest unknown.
This sha is the latest commit to main but it appears a build was not triggered.
container tag exists
run make init; make pr/readme
in any terraform module
No response
No response
No response
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.