magefile / mage Goto Github PK

If a command run by mage requires user input, mage should allow the user to enter that input.

I'd like to see a helper function in mg that could run deps sequentially without parallelism. Some of the deps in Hugo are memory intensive, and I'd like to just run them sequentially.

We can already do this now with multiple if err := A(); err != nil {} blocks (or create my own helper function), but a helper function in mg would be convenient.

I was going through the code and noticed there was copies of files found in the build package https://golang.org/pkg/go/build/

Any reason why not just to import it? I am thinking about long term maintenance implications !

if you have

func Foo() {
    mg.Deps(Bar, Baz)
}

and you run mage -v foo currently we print out

Running Foo

But it would be nice if it said

Running Bar
Running Baz
Running Foo

The Go project I'm working on has a sub-project in JavaScript that has its own build tools. To run them, I need to change directory. The current build script I'm using is:

func BuildAssets() {
	cmd := exec.Command("yarn", "install")
	cmd.Dir = "front"
	cmd.Stdout = os.Stdout
	cmd.Run()
}

Is there any way to use the much less cumbersome sh.RunV() and related methods in a different path?

The obvious solution of adding cd front && yarn to the command doesn't work as it searches for the executable in the PATH:

Error: failed to run "cd front && yarm install: exec: "cd front && yarm": executable file not found in $PATH"

Using os.Chdir should also work, but is quite awkward when everything else expects to be run from the base path. Perhaps this is the only way?

mg.Deps() doesn't seem to handle anonymous functions properly. Given a loop like below:

func Demo() error {
	cmds := []interface{}{}
	for i:=0; i < 10; i++ {
		k := i
		f := func() error {
			fmt.Println(k)
			return nil
		}
		cmds = append(cmds, f)
	}
	mg.Deps(cmds...)
	return nil
}

I get the following output:

Running target: Demo
0

I ran into this while generating dynamic build targets for a multi-platform build (to give a use case).

If you have common targets you want to use for multiple magefiles, right now there's no way to share the targets.

A . import statement could be a good implementation. It's pretty obvious and fairly straightforward to implement.

import . "example.com/my-mage/std-magefile"

Let users define a list of aliases for targets:

var Aliases = map[string]interface{}{
    "example-foo" : ExampleFoo,
}

Alias names would be compared vs. the lowercased target name specified on the command line. The code would check for alias matches first, and if found, use reflection to get the function name specified, and replace the CLI target name with the reflection generated name.

might be nice to do mage foo,bar to run both foo and bar. However, this may have adverse impacts on other features, so it requires some thought. Input herre is welcome

Currently the output of mage -version is:

Mage Build Tool 
Build Date: <not set>
Commit: <not set>

Just as we auto-expand env vars, it would be nice to expand glob (*.foo) into the correct filenames.

Also possibly add a config variable var AutoGlob=false

hugo website deployed to netlify

I've had several issues, so I'm not sure what's what anymore, but I thought I should raise this issue to get a complete picture.

This is with the latest go get mage, but I saw this also with the 1.0 version. It does not happen all the time, so may be a memory thing ... But I have never seen this with make doing the exact same thing.

#!/bin/bash -eo pipefail
git clone [email protected]:gohugoio/hugoDocs.git
cd hugo
mage vendor
mage check
Cloning into 'hugoDocs'...
remote: Counting objects: 42080, done.        
remote: Compressing objects: 100% (143/143), done.        
remote: Total 42080 (delta 115), reused 113 (delta 62), pack-reused 41873        
Receiving objects: 100% (42080/42080), 82.94 MiB | 57.03 MiB/s, done.
Resolving deltas: 100% (29294/29294), done.
go build testmain: /usr/local/go/pkg/tool/linux_amd64/link: signal: killed
go build testmain: /usr/local/go/pkg/tool/linux_amd64/link: signal: killed
go build testmain: /usr/local/go/pkg/tool/linux_amd64/link: signal: killed
go build testmain: /usr/local/go/pkg/tool/linux_amd64/link: signal: killed
go build testmain: /usr/local/go/pkg/tool/linux_amd64/link: signal: killed
go build testmain: /usr/local/go/pkg/tool/linux_amd64/link: signal: killed
go build testmain: /usr/local/go/pkg/tool/linux_amd64/link: signal: killed
go build testmain: /usr/local/go/pkg/tool/linux_amd64/link: signal: killed
go build testmain: /usr/local/go/pkg/tool/linux_amd64/link: signal: killed
Error: exit status 1
Error: exit status 1
Error: running "govendor test +local" failed with exit code 2
running "govendor test -race +local" failed with exit code 2
Exited with code 2

Auto-expand environment variables in sh should be configurable.. on or off for everything, or include/exclude command names. Include/exclude would be a list of command names to either turn expanding environment variables on or off for. For example, "don't expand environment variables when I run the 'foo' command.". Thus if you have a command that expects an argument with a $ in it, you can pass that to it without it getting auto-expanded.

All of the above also for the upcoming auto-glob in #55

API:

func CfgExpandEnv(m CfgMode, cmds ...string) {}
func CfgGlobFIles(m CfgMode, cmds ....string) {}

var CfgMode int
const (
    OnForAll CfgMode = iota // enable the config for all commands (default)
    OffForAll               // disable the config for all commands
    IncludeCmds             // disable the config except for the given commands
    ExcludeCmds             // enable the config except for the given commands
)

If you pass a list of commands for OnForAll or OffForAll, we panic. If you don't pass a list of commands for IncludeCmds or ExcludeCmds, we panics.

This would generally be used in an init function so that it takes effect for all targets... but nothing forces you to do it that way.

Examples of dependencies, usage of sh and mg, how to structure files for readability, etc.

"a Makefile replacement for go" gives the impression that this is a tool for building Go code. "A tool for writing Makefiles using Go" might be better.

I get that this is very subjective, but this is from the Hugo build:

▶ mage
  check            Run tests and linters
  checkvendor      Verify that vendored packages match git HEAD
  docker           Build hugo Docker container
  fmt              Run gofmt linter
  hugo             Build hugo binary
  hugonogitinfo    Build hugo without git info
  hugorace         Build hugo binary with race detector enabled
  install          Install hugo binary
  lint             Run golint linter
  test             Run tests
  test386          Run tests in 32-bit mode
  testcoverhtml    Generate test coverage report
  testrace         Run tests with race detector
  vendor           Install govendor and sync Hugo's vendored dependencies
  vet              Run go vet linter

Which is mostly beautiful. But the all lower-case command names looks "wrong" in my eyes. I would much prefer that it followed the original case of the method, but with a lowercase first letter, so hugoRace, testCoverHTMLetc. I would still expect the all lowercase variant to work.

this will help debug problems during generation

In the video demo your example of return errors.New("boo!") just prints out "boo!" and whilst the echo $? shows a non-zero exit code, there is nothing on the screen to visually show the command failed.

I know some people have prompts which display something when the previous command failed, however I think it would be nice to show it visually too. Nothing too big, but just big enough.

For example:

$ mage install
Error: boo!

As a comparison let's just look at what make does. We don't have to copy this nor am I holding it up as the ideal, but just a curiosity. For example, if my target just tried to cat a non-existant file:

$ make target
cat hello.go
cat: hello.go: No such file or directory
Makefile:2: recipe for target 'target' failed
make: *** [build] Error 1

Which (I think) is made up of the command, stderr from the command and then output from make itself. I'm happy to keep it real simple as above and just prefix the output with Error:.

(Note: I think we could or should also do the same for functions that don't return errors, so I'm not sure how we'd do that - perhaps if they wrote something on stderr? This comment is a non-goal of this ticket and perhaps another ticket should be opened, however it's pertinent to raise this here in case it impacts what we do here.)

Description

It's really small improvement but I think it's a good one.

Many projects have links to their web sites in the project's description (e.g. https://github.com/golang/go or https://github.com/golang/dep) so people don't need to scroll down and search for the link (usually the web site contains more info as readme file).

Proposal:

Add link to https://magefile.org/ in the project description.

That's all 😉.

I may investigate this later, but thought I should post this in case someone else has any tips:

Resolving deltas: 100% (28797/28797), done.
Error: magefile.go:6:2: could not import bytes (cannot find package "bytes" in any of:
	/go/src/github.com/gohugoio/hugo/vendor/bytes (vendor tree)
	/Users/finchnat/src/github.com/golang/go/src/bytes (from $GOROOT)
	/go/src/bytes (from $GOPATH))

I use the Linux binary from the v1 release.

Problem: When running mage with the included example code below the following error is reported.

make.go:8:2: could not import github.com/magefile/mage/mg (can't find import: github.com/magefile/mage/mg)

Expected: mage -l to generate and build with the example code, allowing the use of mg for mg.Dep(...).

// +build mage

package main

import (
	"fmt"

	"github.com/magefile/mage/mg"
)

func Build() {
	mg.Deps(Clean)
	fmt.Println("Building")
}

func Clean() {
	fmt.Println("Cleaning")
}

Results of go env

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/interlock/src/go"
GORACE=""
GOROOT="/home/interlock/.asdf/installs/golang/1.8.3/go"
GOTOOLDIR="/home/interlock/.asdf/installs/golang/1.8.3/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build518538770=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"```

Running the recommended commands from the README.md:

go get -u -d github.com/magefile/mage
cd $GOPATH/src/github.com/magefile/mage
go run bootstrap.go

Getting:

Running target: Build
exec: git rev-parse --short HEAD
exec: git describe --tags
exec: go install -a -ldflags=-X "github.com/magefile/mage/mage.timestamp=2017-10-27T09:39:17-05:00" -X "github.com/magefile/mage/mage.commitHash=accd8ef" -X "github.com/magefile/mage/mage.gitTag=v2.0.1-3-gaccd8ef
" github.com/magefile/mage
go install runtime/internal/sys: open /usr/local/go/pkg/darwin_amd64/runtime/internal/sys.a: permission denied
Error: running "go install -a -ldflags=-X "github.com/magefile/mage/mage.timestamp=2017-10-27T09:39:17-05:00" -X "github.com/magefile/mage/mage.commitHash=accd8ef" -X "github.com/magefile/mage/mage.gitTag=v2.0.1-3-gaccd8ef
" github.com/magefile/mage" failed with exit code 1
exit status 1

macOS 10.12.6

$ go version
go version go1.9.2 darwin/amd64
$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/jake/Projects/Labs/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/h2/whwtkkx168n8333397n69m2r0000gn/T/go-build684654678=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

Having a magefile like this:

// Creates "something"
func Create() error {
  return nil
}

Will result in an error. It looks like the quotes aren't being escaped when I check out the intermediate file. Probably a trivial fix but posting here before doing anything to verify it should be fixed.

Some of the tests are forced to use go run to run full-stack tests for go. It would be a lot easier and better if we could just run the mage main function on a specific directory. This should be pretty trivial.

This will also open up the ability to have one magefile run a target in another magefile (outside the current directory).

Is there no way to pass in variables to a target? I see the docs state a function with no args.

Something like:

mage buildDocker -imageName myimage -tag 1.0

I'm really new to golang so maybe there's some other way to do this without mage needing to care, but seems like it would be hard to know what parameters a target needs without digging into the code implementation vs just looking a function definition.

The code is trivial, but it might be nice to add a helper to provide parallelisation limits.

func ensureGoTool(tool, pkg string) func() error {
	return func() error {
		_, err := exec.LookPath(tool)
		if err != nil {
			fmt.Printf("couldn't find tool %s, attempting to download\n", tool)
			if _, err := sh.Exec(nil, os.Stdout, os.Stderr, "go", "get", "-u", pkg); err != nil {
				return err
			}
			_, err := exec.LookPath(tool)
			if err != nil {
				return fmt.Errorf("couldn't download tool %s from %s", tool, pkg)
			}
		}
		return nil
	}
}

func Vendor() error {
	mg.Deps(ensureGoTool("vndr", "github.com/LK4D4/vndr"))
	return sh.Run("vndr")
}
func Lint() error {
	mg.Deps(ensureGoTool("gometalinter", "github.com/alecthomas/gometalinter"))
	return sh.Run("gometalinter", "--config", ".gometalinter.json", "./pkg/...")
}
func Do() error {
	mg.Deps(Vendor, Lint)
	return nil
}

The problem is that with mage do, ensureGoTool only get's run once. There is probably some optimization to prevent similar tasks from being called multiple times. How do I disable it in this case?

You should be able to declare that a target produces one or more files from one or more files. If the source files have been modified more recently than the target files (or if the target files don't exist), then run the target, otherwise skip it, assuming it's done.

Hi,

latest release 2.0.1 doesn't support alias. If I use deps as dependency management tool, only 2.0.1 version will be fetched.

My question: can we release a new version of mage?

Migrating my personal project to mage currently and I tried to run eval in sh.Run(). The program outputted
Error: failed to run "eval (docker-machine env cryptotraderbot): exec: "eval": executable file not found in $PATH"

Is there any way to do this with mage or is it currently not supported ?

Relevant code:

if err := sh.Run("eval", "$(docker-machine", "env", "cryptotraderbot)"); err != nil {
     return err
}

Currently there's very little information about what went wrong if you mess up your mage file or mage otherwise doesn't do what you want. It should be pretty easy to add a -debug flag to mage itself to enable verbose logging of code parsing and generation.

create a new flag -init that creates a new default magefile.

we should have a magefile to build all release targets

For now, we don't need a general build target, but it might be worth making one just so people can see how it works.

This would allow for users to write functions like:

func Build(ctx context.Context) error {
  cmd := exec.CommandContext(ctx, "/usr/local/go/bin/go", "build", "./cmd/...")
  cmd.Stdout = os.Stdout
  cmd.Stderr = os.Stderr
  err := cmd.Run()
}

And then if the user hits control-c to terminate the mage process, the context will instantly cancel and cause the process started by the Build function to be killed.

Mage itself would need to start and manage these contexts.

It would be nice to also have a command line flag -t that would set a timeout on these contexts.

What's the normally intended use of stdout/stderr in mage files for user functions? I notice mage has no logging wrappers for adding build specific information, so I'm not super clear on the design intent here?

See gohugoio/hugo#3939

Good thing I tested.

So I just replaced make with mage and got:

mage hugoRace check

Wich does not fail. If you do not support multiple targets you really should exit with error + some error message.

Could be as easy as running tests, where a clickable link is inserted above each task.

This commit 0c1c4dc

Seem to have broken Hugo (cannot find any of the targets)

func init() {
	if exe := os.Getenv("GOEXE"); exe != "" {
		goexe = exe
	}

	// Update the GOARCH.
	if output, err := sh.Output(goexe, "env", "GOARCH"); err == nil {
		goarch = output
	} else {
		panic(fmt.Sprintf("couldn't get GOARCH: %v", err))
	}
}

> mage
2018/01/26 21:55:25 exec: go env GOARCH

I expected that output only when invoking mage -v.

When the actual targets get run, the sh commands are properly muted.

Your release binary distro does not resemble "anything else" ... and I would suggest doing something more standardize so people can use the bash scripts they already set up for similar stuff. Package the binary mage (i.e. no need for renaming on extract) in a tar file with the execution permissions set. I suspect most Windows AV scanners will jump at the exe downloads etc. Also, I suspect sha 256 hashes are more common.

I started to update my scripts, and what I thought should be a 5 minute task is now turning into real work.

Goreleaser does this right.

It wouldn't be too hard to auto-generate a CLI for each target so the user could pass in positional args or flags per arg.

e.g.

func Build(OS string, ARCH string) error {
     // build with GOOS and GOARCH
}

$ mage build -os darwin -arch amd64
building with GOOS=darwin GOARCH=amd64

Resolving deltas: 100% (29367/29367), done.
Error: failed to check types in directory: magefile.go:14:2: could not import github.com/magefile/mage/mg (cannot find package "github.com/magefile/mage/mg" in any of:
	/go/src/github.com/gohugoio/hugo/vendor/github.com/magefile/mage/mg (vendor tree)
	/usr/local/go/src/github.com/magefile/mage/mg (from $GOROOT)
	/go/src/github.com/magefile/mage/mg (from $GOPATH))
Exited with code 1

And while I do understand the error above and know the workaround for it, I did expect this to just work without any "go getting". I guess the released binary is just for bootstrapping and not really something one can use for added stability?

Having it in my vendor file would not work either (chicken and egg).

I basically always want to run with verbose mode on, so it would be nice if I could just set an environment variable and not need to remember to type -v.

Asking mage to display supported options yields:

$ mage -h
mage [options] [target]
Options:
  -f    force recreation of compiled magefile
  -h    show this help
  -init
        create a starting template if no mage files exist
  -keep
        keep intermediate mage files around after running
  -l    list mage targets in this directory
  -v    show verbose output when running mage targets
  -version
        show version info for the mage binary

but typing the following yields:

$ mage --help
mage [options] [target]
Options:
  -f    force recreation of compiled magefile
  -h    show this help
  -init
        create a starting template if no mage files exist
  -keep
        keep intermediate mage files around after running
  -l    list mage targets in this directory
  -v    show verbose output when running mage targets
  -version
        show version info for the mage binary
mage [options] [target]
Options:
  -f    force recreation of compiled magefile
  -h    show this help
  -init
        create a starting template if no mage files exist
  -keep
        keep intermediate mage files around after running
  -l    list mage targets in this directory
  -v    show verbose output when running mage targets
  -version
        show version info for the mage binary

instead of:

$ mage --help
flag provided but not defined: -help
mage [options] [target]
Options:
  -f    force recreation of compiled magefile
  -h    show this help
  -init
        create a starting template if no mage files exist
  -keep
        keep intermediate mage files around after running
  -l    list mage targets in this directory
  -v    show verbose output when running mage targets
  -version
        show version info for the mage binary
Error: flag provided but not defined: -help

You may want to fix this 😉

When writing things using function commentary such as:

// Build builds and re-generates all needed dependencies and compiles all binaries for $APP_NAME.
func Build() {
// ... etc

This gets recorded in the mage task list as:

$ mage -l 
Targets:
  build      Build builds and re-generates all needed dependencies and compiles all binaries for $APP_NAME.

What I would expect would be something akin to:

$ mage -l 
Targets:
  build      builds and re-generates all needed dependencies and compiles all binaries for $APP_NAME.

I have a proof of concept patch here that implements this in a rather hacky way that seems to work pretty well.

bail if either flag is set if fs.NArgs() > 0 || fs.NFlags() > 1 - i.e. if any other args were specified.

Also move them into a separate section of the usage output called "commands".

The --clean flag should be treated the same way from #19

As seen in Hugo's Makefile, I like the default target to show a list of available targets. Is this possible with mage?