cmmorrow / build-magic Goto Github PK

Is your feature request related to a problem? Please describe.
Currently, if build-magic fails to shutdown a Docker container (named build-magic) properly when using the docker runner, the container will continue to run. If trying to run build-magic again with the docker runner, the setup will fail with a generic "Setup failed" error message.

Describe the solution you'd like
Modify the container_up() function to first check to see if a container named build-magic is already running, and give a detailed error message that says "Setup failed because the build-magic Docker container is already running. Please stop and remove the container to fix this error."

Describe alternatives you've considered
Alternatively, investigate having build-magic stop and remove the container automatically before trying to start the Docker container. If doing this, print a message to let the user know the build-magic container is already running and is being shutdown before creating a new container.

Is your feature request related to a problem? Please describe.
All CI/CD tools support variables that can be dynamically substituted at runtime.

Describe the solution you'd like
Add support for user-defined variables that can be specified through the CLI with --variable as a key/value pair that correspond to a key in a config file. Variables should have a special syntax that will be parsed by build-magic and substituted for the user-defined variable. The config file should support variables with scope local to a Stage, as well as global scope applied to all stages.

Is your feature request related to a problem? Please describe.
It would be useful to check the system OS and arch before executing a stage.

Describe the solution you'd like
Provide an operating system and architecture parameters that can be applied to each runner. If the executing environment doesn't match the provided parameters, the stage is skipped.

Describe alternatives you've considered
This can be done with Local runner "environment" field, but it would be useful to allow this feature for all runners as a parameter.

Additional context
In the event a stage with a Local runner provides an environment and an "operating system" parameter, the parameter takes precedence.

Describe the bug
When running a job with a Docker runner with the cleanup action, the job fails during startup. The issue is when the cleanup action provision function runs, a permission error is raised when trying to read a .git directory. The fix should be to handle the case where a permission error is raised.

Is your feature request related to a problem? Please describe.
Corresponds to the TODO here: https://github.com/cmmorrow/build-magic/blob/main/build_magic/core.py#L160

Describe the solution you'd like
Need to bubble up the exit code from the last failing command and make sure it's propagated to the end of the Engine.run() method.

The problem occurs here: https://github.com/cmmorrow/build-magic/blob/main/build_magic/core.py#L436.

# Set the exit code.
fails = map(lambda r: True if r.exit_code > 0 else False, self._results)
if any(fails):
    self._result = 1

If any command fails during stage execution, self._result is set to 1 and is returned instead of the exit code from the failing command. The condition when if any(fails) is True should be changed to instead set self._result to the last non-zero exit code in fails.

Is your feature request related to a problem? Please describe.
It's nice that --environment can be used to skip stages on non-matching operating systems, but it would be better to allow for more granular control by being able to specify an OS version as well.

Describe the solution you'd like
Parse the Local command runner environment for OS and version number and handle accordingly.

Describe alternatives you've considered
Issue #84 also enables granular filtering by using parameters, but this method can be used as an alternative to the more verbose usage proposed in #84.

Describe the bug
When build-magic executes a long command, the fancy output is sometimes displayed incorrectly.

Screenshots

(  5/19 ) BUILD   : pyinstaller -n "build-magic_0.3.0rc1" -D --clean --add-data build_magic/static:build_magic/static build_magic/__main__.py
 ............................... RUNNING

Is your feature request related to a problem? Please describe.
Code reuse isn't great in the action functions.

Describe the solution you'd like
Refactor some of the action functions to make better use of code reuse.

Describe the bug
If the yaspin spinner is being displayed and build-magic is terminated with Ctrl+c, the spinner continues to run, and a second Ctrl+c is needed to kill build-magic, resulting in a python error.

Is your feature request related to a problem? Please describe.
Currently, provision and teardown functions only return a boolean.

Describe the solution you'd like
In case of an error, it would be good to capture it and bubble it up to the caller.

Is your feature request related to a problem? Please describe.
Currently, config files lack any meta data that can be used to identify and differentiate them.

Describe the solution you'd like
Add keys for version, maintainer, and date to the config file schema to optionally allow for adding meta data to config files.

Describe alternatives you've considered
VCS somewhat covers this, but doesn't allow for config files to be individually versioned which a user might want to do.

Is your feature request related to a problem? Please describe.
The sequence number of a command out of the total number of commands per stage isn't displayed.

Describe the solution you'd like
Add the command sequence per line with something that looks like: ( 1/8 ).

Describe the bug
Running the cleanup action on a working directory that performs git commands on a branch and has a .git directory appears to corrupts the branch state by deleting new files.

In particular, the gh-pages branch is corrupted after running with build-magic and using the cleanup action, resulting in the following message when running git pull origin main:

error: refs/heads/gh-pages does not point to a valid object!
error: refs/remotes/origin/gh-pages does not point to a valid object!

Describe the bug
When an alias is used in a config file which doesn't have a matching anchor (perhaps a typo) a pyyaml ComposerError is raised and not handled by build-magic.

Traceback (most recent call last):
  File "/Users/chrismorrow/.pyenv/versions/3.9.2/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/chrismorrow/.pyenv/versions/3.9.2/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/chrismorrow/repos/build-magic/build_magic/__main__.py", line 7, in <module>
    build_magic()
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/chrismorrow/repos/build-magic/build_magic/cli.py", line 226, in build_magic
    stages = get_stages_from_config(cfg, dict(variable))
  File "/Users/chrismorrow/repos/build-magic/build_magic/cli.py", line 387, in get_stages_from_config
    obj = yaml.safe_load(cfg)
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/yaml/__init__.py", line 162, in safe_load
    return load(stream, SafeLoader)
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/yaml/composer.py", line 82, in compose_node
    node = self.compose_sequence_node(anchor)
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/yaml/composer.py", line 111, in compose_sequence_node
    node.value.append(self.compose_node(node, index))
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/yaml/composer.py", line 82, in compose_node
    node = self.compose_sequence_node(anchor)
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/yaml/composer.py", line 111, in compose_sequence_node
    node.value.append(self.compose_node(node, index))
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/Users/chrismorrow/build-magic/lib/python3.9/site-packages/yaml/composer.py", line 68, in compose_node
    raise ComposerError(None, None, "found undefined alias %r"
yaml.composer.ComposerError: found undefined alias 'clean_pyinstaller'
  in "package.yaml", line 69, column 20

To Reproduce
Steps to reproduce the behavior:

1. Create a valid config file with an alias without corresponding anchor.
2. Run the config file with build-magic -C config.yaml.

Expected behavior
Build-magic should handle these errors and provide a user-friendly error message.

Desktop (please complete the following information):

• OS: MacOS
• Python 3.9
• Version 0.3.0rc1

Is your feature request related to a problem? Please describe.
In a CI/CD environment, command line tools are often downloaded as part of the stage/job being executed. It would be useful to have a built-in way to automatically download files to the current directory.

Describe the solution you'd like
Add a --download option to the command line and downloads section to each stage in the config file. Multiple URLs should be supported and build-magic should be able to download the files in parallel to save time. Multiple protocols should be supported such as HTTP, HTTPS, SFTP, SCP (maybe), and file. The protocol, URI, and port should be parsed from the value and handled accordingly.

Is your feature request related to a problem? Please describe.
A stage intended to run on Windows cannot run on Linux or MacOS. For this reason, it would be useful to be able to tag a stage with the intended OS.

Describe the solution you'd like
Since the environment: field isn't used for the Local command runner, this can optionally be used to identify which OS the runner is intended for. Options should include "windows" and "macos" or "darwin". Individual Linux distros should also be included such as "debian", "centos", and "arch". The use would be optional and if provided, a check for the operating system is run at the beginning of the stage to determine the OS. If an exit code of 0 is returned, continue with executing the stage. Otherwise, skip the stage and notify the user.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe.
Certain values such as user name, tag names, etc can change from user to user, job to job. It's necessary to support variable substitution in config files to allow for variable values at runtime.

Describe the solution you'd like
Add a --variable option to the command line that takes two arguments: the variable name, and the value to substitute for the corresponding variable name in the config file. The config file should support a standard way of identifying text as a variable.

Describe alternatives you've considered
Environment variables can do the job, but the format and easy of use is different between Windows and Unix.

Is your feature request related to a problem? Please describe.
Running commands from a build-magic config file is useful, but doesn't integrate so well with CI/CD workflows/pipelines.

Describe the solution you'd like
Add an --export option that can convert a build-magic config file to aGitHub workflow file or GitLab CI file. Both formats support running arbitrary shell commands, so build-magic commands can map to commands in both formats easily. Build-magic stages can map to GitHub workflow steps and to GitLab CI stages or jobs. Command runners and environments cannot map deterministically so they should be ignored and the commands in the stages should be assumed to match the CI runner.

Is your feature request related to a problem? Please describe.
Seeing a long shell command in the build-magic export doesn't always make it clear what is happening. It would be good to be able to provide an optional description of the command to display instead of the command itself.

Describe the solution you'd like
A command label can be added by using a --label option. Each --label should sequentially correspond to a provided command. If the number of commands do not match the number of labels, an error should be raised to let the user know.

In the case of config files, a label: object can optionally be provided for a command like the following:

commands:
  - build: make all
    label: Compile myapp and run tests.

The implementation should be done by adding --label to the CLI and the label: object to config_schema.json. The labels will need to be matched to commands and passed to the build_stage() function. The MacroFactory constructor will also need to be modified to accept the label as an argument and set it as a property. When Stage.run() executes the macro, it should display the description in the Macro's label property if it exists, otherwise the command is displayed.

Is your feature request related to a problem? Please describe.
I don't always want to use the build-magic CLI and would like to be able to wrap the build-magic functionality around custom logic.

Describe the solution you'd like
Expose a high-level Python API for creating Macros, creating Stages, executing Stages, specifying an Action, adding custom Actions, and parsing config files. Add the documentation for these methods.

Is your feature request related to a problem? Please describe.
It would be nice to be able to pipe arguments to build-magic from other commands. Examples of arguments could be file names used by the --copy option, explicit commands specified without using the --command option, as well as stage names in a config file.

Describe the solution you'd like
Add stdin support to the cli to support this request. Piped arguments need to be read from stdin unless click has a native solution.

Usage would work like the following examples:

# Explicit command
> cat command.sh | build-magic --verbose

# Copy file names
> ls | build-magic --copy . --wd myproject --verbose --command build "docker build . -t myproject"

Is your feature request related to a problem? Please describe.
If you specify options other than output formats in the CLI and provide a config file, the CLI options are ignored.

Describe the solution you'd like
It would be useful if the CLI options would instead override the corresponding settings in the config file. Since the CLI only has a single Stage focus, the options passed from the CLI must act like globals, and override the corresponding config file settings for EVERY Stage.

Is your feature request related to a problem? Please describe.
Getting the config file syntax just right can be tricky at best and frustrating at worst.

Describe the solution you'd like
It would be nice to have a command that generates a template config file to use as a starting point. It could work be adding a --template or similarly named option to the CLI that would generate a template config file named build_magic_template.yaml. An error will be raised if trying to generate a file in the directory that's already named build_magic_template.yaml.

Is your feature request related to a problem? Please describe.
When adding support for dotenv files, I noticed adding support for individual environment variables wouldn't be that difficult and could be useful if only one or two are needed, and the user doesn't want to both with making a dotenv file.

Describe the solution you'd like
Added a --env option to the CLI and environment variables: property to the config file for adding ad hoc environment variables. Allow for multiple --env options and environment variables: should be a list. Environment variables should tie into the existing code for passing envs but the code for reading and parsing dotenv file might need to be refactored a little to make this easier.

Describe the bug
Build-magic doesn't seem to handle the case where the environment doesn't match a local image name.

Is your feature request related to a problem? Please describe.
Homebrew is a popular package manager on MacOS and would make for a great way to install build-magic as opposed to installing via python and virtual environments.

Describe the solution you'd like
Create a homebrew formula for build-magic so that a user only needs to add the tap and install build-magic.

Is your feature request related to a problem? Please describe.
There are a lot of options and ways to use build-magic. It would be useful to have a detailed help file in the form of a man page on systems that support or add more info to the --help command.

Describe the solution you'd like
Add a man page for systems that support it and improve the --help output to duplicate the output of the man page for systems that do not support man pages.

Describe alternatives you've considered
Online documentation is good and should be highly detailed, but it's useful to have local help for quick reference built into the terminal.

Additional context
Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe.
Users shouldn't have to worry about installing a valid version of python, managing dependancies, and creating virtual environments to use build-magic. It would be much simpler to be able to install a rpm or deb package and have build-magic just work.

Describe the solution you'd like
Create rpm and deb packages for new build-magic releases using a build-magic config file. Each package should work for equivalent debian/ubuntu/mint and fedora/centos/RHEL versions. Pyinstaller will be used to create a single folder that can then be packaged and installed via a package manager. The SHA256 hashes and gpg signatures should also be provided for data integrity and trust. The packages should then be uploaded to Github along with signatures and hashes. Eventually, build-magic can be submitted to the official repos once stable and out of beta.

Describe alternatives you've considered
Eventually, Arch Linux packages should be supported as well.

Is your feature request related to a problem? Please describe.
The "cleanup" action will cleanup any new files created, but will not correct files that are renamed or modified from their original state.

Describe the solution you'd like
The "restore" action will create a backup of the working directory for the provision step, and delete and restore the working directory for the teardown step.

Is your feature request related to a problem? Please describe.
When a config file doesn't pass validation, the error message is extremely verbose and hard to interpret.

Describe the solution you'd like
Add a --lint option to the command line to provide more user friendly error messages about the config file schema.

Is your feature request related to a problem? Please describe.
Address TODO here: https://github.com/cmmorrow/build-magic/blob/main/build_magic/core.py#L123

Describe the solution you'd like
Just need to add a simple check to make sure stages is an instance of list.

Is your feature request related to a problem? Please describe.
Currently, there's no indication build-magic is doing anything when the start up (provision) is running, particularly with the Vagrant runner.

Describe the solution you'd like
Add a start up and teardown status to indicate when they are running and complete.

Describe alternatives you've considered
Alternatively a spinner or some other visual cue could be used.

build-magic release 0.4.0 checklist:

• Refactor CLI to move --info into a callback function.
• Refactor Stage.run() to clean up the code and improve reuse.
• Update README.
• Update documentation.
• Make sure unit tests pass on Windows.
• Create the release.
• Push to Pypi.
• Build and test the Linux packages.

Describe the bug
When running a job with the Docker runner and setting the host working directory, the setup fails with no error message.

To Reproduce
Run package.yaml

The problem is if the specified host working directory doesn't exist, an APIError is raised which doesn't give an error message on setup. The solution is to check for the existence of the host working directory in the runner before creating the Bind object.

Describe the bug
A large gap in the build-magic output to a terminal window is present on a new terminal window or after running clear.

To Reproduce
Steps to reproduce the behavior:

1. Open a new terminal window.
2. Run build-magic "echo 'hello world'".

Expected behavior
The output to a terminal should be consistent.

Screenshots

Is your feature request related to a problem? Please describe.
Stages with long execution times could benefit from running in parallel as long as they don't depend on one another.

Describe the solution you'd like
Allow for a parallel stage alternative to stage in a config file. By default, parallel stages that are defined sequentially are executed in parallel. Alternatively, a parallel stage can define a group key in which case parallel stages with the same group will be run together.

Additional context
Maybe the vagrant runner shouldn't implement parallel stages since the behavior might not be stable.

Also, a new display type should be considered for use with parallel stages.

Is your feature request related to a problem? Please describe.
Sometimes, commands can be repeated in a config file across multiple stages. Anchors and aliases come in handy in this situation, but it would be useful to have a stage-independent section for defining anchors that can be called from all stages.

Describe the solution you'd like
Create an optional prep: section in the config file that can be used for defining anchors. This can be a freeform, unstructured section with loose validation to allow the user to structure the content how they see fit.

Is your feature request related to a problem? Please describe.
Corresponds to the TODO here: https://github.com/cmmorrow/build-magic/blob/main/build_magic/runner.py#L153

Describe the solution you'd like
Validate the type of values in the artifacts and parameters arguments are strings and raise an error if they cannot be cast. This is preferable to raising an ambiguous error further along that might not reflect the root cause of the error. This should only be an issue when using the low-level API directly.

If a value in artifacts or parameters isn't a string, a Value error should be raised. This is to prepare for the high-level Python API.

Is your feature request related to a problem? Please describe.
Currently, all Stages in a config file must be executed to pass and are executed in the order they are defined. However, there might be instances where a user wants to run a particular stage or stages in a different order.

Describe the solution you'd like
The "target" feature can be used similar to the make command, where targets for make to execute are provided as arguments. In this case, the --target option can be added to the CLI, where the specified value matches a corresponding Stage name. Multiple --target options can be provided to run multiple Stages, and the Stages should be executed in the order they're provided.

To summarize:

• Add the --target argument to execute an individual stage in a config file.
• Allow for multiple --target args and execute them in order.
• Allow for a "default" config file name of build-magic.yaml so that the --config option can be omitted.
• Hardcode an "all" target which runs all the stages in order in the config file.
• Contextually parse a single arg as a target stage name as syntactic sugar so that the --target option can be eliminated.

Is your feature request related to a problem? Please describe.
When running a config file you didn't write, it would be useful to have a command that gives at-a-glance information such as the stage names and variable placeholders in the config file.

Describe the solution you'd like
Create a new info command that accepts an argument for the type of information to display and an argument for the config file name.

Is your feature request related to a problem? Please describe.
The docker runner allows for cleanup of files in the host working directory. Need to add similar behavior for the Vagrant runner.

Describe the solution you'd like
This will involve creating a new action that calls the vm_up() and vm_destroy() actions. It should work similar to the local cleanup by removing any newly created actions during the action.

Is your feature request related to a problem? Please describe.
Build-magic has packages for debian/ubuntu/mint and fedora/centos/rhel. Need to add a package for arch/manjaro.

Describe the solution you'd like
The PKGBUILD file and makepkg process should be automated in the package.yaml file in the build-magic repo, and should upload the resulting .zst file to the GitHub release.

See https://wiki.archlinux.org/title/creating_packages for more info on creating Arch Linux packages.

See https://github.com/cmmorrow/build-magic/blob/main/package.yaml for how the build-magic packages are built for debian and centos.

Note: Do not commit GitHub secrets to the repo!

Is your feature request related to a problem? Please describe.
Setting environment variables from a build-magic command fails because the environment variable only lives for the single command. For this reason, it would be great to be able to load a dotenv per stage to support user/project settings without having to manually enter them at runtime.

Describe the solution you'd like
Add a --dotenv option to the command line and dotenv file key to the config file schema to support loading a dotenv file per stage.

Is your feature request related to a problem? Please describe.
The --prompt command is used to interactively provide a variable value that shouldn't be printed to the screen, however, the text is still printed as part of the command output.

Describe the solution you'd like
Blank out with ****** in the command display for variable values that were entered via the --prompt option.

Describe the bug
When using the Docker runner and a command fails with an error, the error message isn't displayed and None is displayed instead.

Is your feature request related to a problem? Please describe.
Build-magic settings can be set at execution time for command runners, except for the vagrant runner. Setting the working directory is handled by adding a prefix to the macro for vagrant, but some settings, such as setting the bind directory can only be set in the Vagrantfile.

Describe the solution you'd like
Before running vagrant_up, read the Vagrantfile and modify the in-memory copy by adding a custom build_magic provisioner. It should be something like:

config.vm.provision "build_magic" do |build|
    build.vm.provision "shell", inline: "export HELLO=world"
end

Describe alternatives you've considered
It might also be possible to set these through ssh-config.

Acceptance Criteria:

• Be able to reliably provision a VM by setting the sync folder (bind directory), the working directory, and setting environment variables.
• Be able to reliably bootstrap the provision on top of other provisioning already in the Vagrantfile (shell, chef, ansible, etc).
• Be able to reliably add the provisioning at the end of all other provisioning already in the Vagrantfile.

Is your feature request related to a problem? Please describe.
Config files convey some meta data, but it would be useful to allow the user to provide a description for a build-magic job as well as stages.

Describe the solution you'd like
Added a description: field as job meta-data as well as a description: field for each stage.

Describe the bug
The Vagrant package logs a warning when the package is loaded and Vagrant isn't in PATH. This is currently suppressed by restricting all warning log level messages.

Expected behavior
It would be better to instead have build-magic check to see if Docker and Vagrant is installed, and only load the modules if the commands are found in PATH.

This is an open-ended issue for continuously improving unit test and integration test coverage. At the time of opening this issue, the unit test coverage is 95%.

You can find the coverage reports here: https://app.codecov.io/gh/cmmorrow/build-magic

Is your feature request related to a problem? Please describe.
Creating a config file from scratch can be frustrating. It would be nice to have build-magic provide a template as a starting point.

Describe the solution you'd like
Provide a --template command line option which will generate a generic config file.