jimporter / bfg9000 Goto Github PK

Before releasing 0.1, I'd like to at least make a best effort to fix all the OS X tests.

It might be useful to add the ability to copy/link a directory, e.g. to make it easier to package up resource files. Obviously, you can already do this with command(), but a platform-agnostic way could be useful. One question is what to call the link function. Symlinks and hardlinks are both nice, but "link" is too confusing: it sounds more like linking an executable.

I'm not even 100% sure I'd want these steps in the first place, so I'm leaving this open to let folks discuss it. If there's a consensus in favor of this, I'll add support.

To make portable executables (and DSOs) on Linux, we should use rpath. (LD_LIBRARY_PATH is crap.) This presents some issues with how to create a rpath that works for both pre- and post-installation of the binary. Here are some possible solutions:

1. Always build binaries so their directory structure in the build dir matches the installation structure. However, this would require a fair bit of extra bookkeeping, since bfg's install directive is separate from the build directives. This also wouldn't work if we support setting $(bindir) and friends from the environment.
2. Use LD_LIBRARY_PATH for pre-installation execution. This could end up being extra management for the user if they want to run the binary by hand before installing it.
3. Use chrpath or patchelf to modify the runpath when installing. This is probably the most-complete solution, but it imposes an additional dependency on bfg.

Once the API stabilizes some more, I should add docstrings to everything. I figure by the time 0.2 is out, things will be stable enough...

Normally, the way you'd use a tool like ccache or distcc is to prepend it to your CC env var like so: CC=ccache gcc. This causes problems with bfg though. Command variables (like CC) are converted into absolute paths so that builds are less dependent on the user's current environment. That is, CC=gcc becomes CC=/usr/bin/gcc.

Problems

1. How can we tell which parts of that variable would need to be converted into absolute paths? Would we just have to know how ccache and distcc work? What if the user had a similar tool that we didn't recognize? What if there was a tool whose format looked like this, but the gcc wasn't meant to refer to an actual file in the path? (For a simpler example of this problem, see #27.)
2. Currently, bfg assumes that a command variable's value is a single command that can be passed to which() to get the absolute path. We can split the value into a list, but that doesn't solve everything.
3. bfg also uses the absolute-path-resolution to make sure that all the required build tools exist during configuration. If, say, gcc isn't installed, bfg will error out. This prevents users from starting a build, only to find out an hour in that they forgot to install something. However, we can obviously check for gcc's existence some other way.

Possible Solutions

1. Make bfg understand ccache and distcc so that we know to treat the argument after it as a file that needs to be turned into an absolute path, e.g. ccache gcc becomes /usr/bin/ccache /usr/bin/gcc. This doesn't help us for commands we don't know about, though.
2. Don't convert to absolute paths, but come up with a way of using the environment variables from configuration (read: when bfg9000 is first run for the project) during the actual build process. This would prevent users from making any after-the-fact tweaks to the environment, which might be a bad thing. It would prevent users from tweaking env vars to, say, run unit tests in a slightly different configuration. Arguably, this should be handled internally in the build.bfg file though.
3. Don't convert to absolute paths, and just rely on the user protecting themselves from danger. This is the simplest, but also the most error-prone for users.
4. Provide a totally-different way of enabling ccache or distcc. This has all the problems of (1), in that users won't be able to use tools bfg doesn't know about, and doesn't seem to have much benefit.

Thoughts

So far, (1) and (3) seem like the best solutions. (2) is pretty complex and breaks probably-useful workflows, while (4) is strictly worse than (1). At the moment I'm leaning towards (1), but (3) is definitely simpler. It really comes down to a question of what's more important: simplicity or reproducibility.

One related question (see #27) is if there are commands that should not have their absolute paths resolved. Obviously, shell built-ins fall under this, but arguably, so does mkdir and other commands that GNU Make assumes exist.

The CC toolchain can generate Windows import libraries (e.g. with MinGW or cross-compiling). As an initial step towards supporting import libraries everywhere, let's add support for it in CC.

Currently, we autodetect the best build system and get version info by running it and grabbing the version. However, this fails if the name is something we don't expect (or it's not in the PATH). We should support checking standard env vars to get the build backend's name, e.g. $MAKE. This may end up obviating #12.

exit() in a build.bfg file should just exit the script, not all of bfg9000. I'm not sure if this already works or not, but I'd guess not. At minimum, this should have tests.

In order to support Java, we'll need to add a concept of "runtime" to bfg. Currently, all the code assumes native runtimes. We could add the JVM as either a new Platform (that has the native platform as a member) or create a Runtime type with both Native and JVM instances/subtypes.

It would be nice if people could add custom rules, languages, etc to bfg9000 so that they don't have to rely on me to support their use cases (especially important for really unusual use cases that I wouldn't want to add into bfg9000 proper).

The code for the builtin functions is already largely structured like this, with one file for each group of builtins, so it would just be a matter of letting users load their own modules at the same time. One way of handling this would be to look in my_project/bfg9000 and load any .py files in there before executing the build.bfg file.

I'm not 100% on this stuff, but the general idea is (on Linux) to append .1.2.3 or what-have-you to the filename, set the soname to libfoo.so.1, and set up some symlinks as needed. No idea how this works on OS X, and I'm sure it doesn't work on Windows at all.

Current plan:

• Create VersionedSharedLibrary to track the real .so and its aliases
• Add a post-build separate build step to symlink libfoo.so -> libfoo.so.1.2.3 (likewise for libfoo.so.1) and make it depend upon the link step for libfoo.so.1.2.3.
• Add a post-install step to symlink as appropriate too (use ldconfig here?)

VS2015 support already works, with both Ninja and MSBuild. However, I think the MSBuild files are broken for VS2013, etc. We should at least support VS2013 for 0.1.

We really need some simple docs and a purdy homepage for bfg9000 before we release.

To make builds more reproducible, we shouldn't rely on commands being in the PATH; instead, resolve them to absolute paths before writing them out in the build file. This will also help ameliorate #13.

To make life easier for users, we should provide installable packages for bfg. While setuptools is helpful, users should be able to use their OS's packager. We want the following:

• Linux
  • Red Hat (.rpm): distutils
  • Debian (.deb): stdeb
• Windows (.exe): py2exe
  • Chocolatey
• OS X (.app): py2app
  • Homebrew

MSBuild support is kind of crappy right now; it can only build software, not all the other fun stuff bfg supports. We should fix this. Below is a list of the features we'd need to support:

• Commands
• Tests See #62
• Installation See #61
• Aliases
• Default

Depends on #18.

Pretty self-explanatory. Something like global_options('-pthread', phase='link')? Eventually, we'll probably want to add the ability to apply this only to a certain runtime (native, CLR, JVM, etc).

To help make more reproducible builds, we could keep track of the exact results of package/executable lookups (e.g. system_package()). This way, even if something about the environment changes, we'll still have consistent results (although sometimes this might just mean a build failure).

One open question is: where should this information be stored? Currently, I see .bfg_environ as independent of the build.bfg file, and so the results of lookups wouldn't go there. This implies that .bfg_environ is immutable even when build.bfg is edited, which is a potentially-nice property (keeps things simple). However, these results do technically come from the environment, so maybe they really do belong there.

We already support this for tests (under POSIX). However, we need a general way to set environment variables that get persisted correctly: local to each sub-command for test(), but for all sub-commands for command(). As usual, Windows is a pain in the ass here.

This would be especially helpful for Appveyor, since free accounts only get one concurrent build. Travis might benefit a small amount too.

single-quoting the $@ causes the patsubst to not match, thus we just mkdir -p $targetdir/.dir and then touch the new directory $targetdir/.dir. I think the intent was to create a 0-sized file .dir in $targetdir.

Removing the quotes breaks spaced-things, we need to use double-quotes here, but I see no easy way to do so without significant changes. Not sure how important it is.

There should be a way to specify that there's a build.bfg file in a subdirectory that needs to be parsed. Remember that paths in the sub-build.bfg file should be relative to the current directory, not the root directory.

Tasks:

• Paths should be resolved relative to the build.bfg file
  • Investigate how to propagate this info to builtins in a safe way
• Execute submodules
• Allow exporting relevant bits from submodules(?)
• Generate appropriate build files
  • Single Makefile? Multiple (recursive vs include)?

If an executable (or shared lib) links to a shared lib, said lib obviously needs to be present at runtime. To prevent errors, we should automatically mark runtime deps for installation if something requires it. This probably only applies to locally-built shared libs, but maybe Windows would want external shared libs to be copied over too? Windows is weird.

I think this technically works, but it's kind of a pain, partly because zlib takes some effort to use under Windows (and has a funky library naming scheme). We could change the library used in the test (e.g. libogg?). We might also want to consult more of the Windows-specific environment variables to make things easier on people.

This way, if the tool (or backend) you wish to use isn't in the path, you can still use fancy new features.

Not all platforms have the install utility. We should just issue a warning if it's not present. (This requires having a system to emit warnings.)

This already kind of works, but we need to be sure that everything is actually correct on Windows (at least, correct enough that simple projects can be built).

There should be a function for build.bfg files to ensure that bfg9000's version meets the requirements (it should probably also have an optional check for the python version).

OS X has rpaths, although they seem pretty different from Linux. Still, we should try to support them. This may take a fair amount of preliminary work...

GCC (or clang with the CC interface) can be used on Windows. We should support this. We can do so by examining the CC and CXX environment variables and then picking the appropriate compiler type.

Currently, if I add a new rule for bfg9000 that needs special handling, I have to go and modify BuildInputs too. This will make life hard for people extending bfg9000 (see #48), so we should fix it.

Pretty self-explanatory. We should fix all the bits required to let people use Python 3. One of these days, I'm sure people will stop using Python 2... :) The most obvious thing to fix is all the uses of basestring.

Currently, bfg assumes for some file types that a file is a dependency iff it has a creator associated with it. This isn't quite right. A file is a dep iff it's internal to the project. Probably the best way to handle this is to treat includes and libs as sources of deps, but packages as non-deps.

It should be impossible to install something found via system_package, system_executable, boost_package, etc. They're already installed! One way to do this would be to give all Files an external attribute to keep track of whether it's from the system (i.e. external to the project).

Static libraries don't allow you to pass link options or libs to them. However, we could allow this and just keep track of the options that were passed, and then forward them on to whatever links to our static lib. This probably requires #5 so that we can record these link options when installing our static lib.

Currently, the install command uses a variety of GNU tools (install, cp -r, and mkdir -p). Write a new program that does this in a simpler way so that we can just use that. Then Windows users won't need GnuWin32 CoreUtils to install stuff.

Currently, we use mkdir -p for installing directories and for creating subdirs under Make. However, not all platforms support mkdir -p (e.g. Windows, some POSIX systems). We should create a set of tools to pick the right program to use on all platforms.

Problems

1. We might want the user to be able to set the MKDIR_P environment variable as, e.g. mkdir -p, and that doesn't really work with how we treat those variables right now: as a single word to be sent on to the which() function for finding the absolute path (see #28). On the other hand, maybe this isn't something the user should be able to set.
2. Do we want to convert mkdir to an absolute path like we do for other commands? Do we just want to do it sometimes (e.g. for gmkdir on Windows)? "Always" or "never" are the simplest answers, but 1) it seems weird to reference mkdir by abs path on Linux, and 2) it seems weird not to for gmkdir on Windows.

Thoughts

First, it should be noted that long-term, mkdir -p will probably only apply to the Make backend. This means that we can probably assume the existence of other GNU tools. In that case, it's probably ok not to have a MKDIR_P environment variable. Additionally, I'm leaning towards always using the abs path because that's more consistent with everything else.

Running automated tests is a pretty important part of a build system! We should have some built-in way of managing this, even if it's just for relatively simple test frameworks for now. Eventually, I'll need to do some research on how various test frameworks expect to be run so that I can come up with a one-size-fits-most solution.

To my knowledge, there's no existing meta-build system that supports globbing source files and regenerating the build files if the result of the glob changes. For instance, while CMake supports globbing and automatic build file regeneration, the two can't be used together [1].

Obviously, the hard part here is doing this fast: incremental builds should only regenerate the build files if a relevant file was added or deleted. Here's a potential solution:

• Provide built-in globbing/find functions that remember their search patterns
• When initially generating a build file, create a .bfg_watch file that lists these patterns and their results
• When performing an incremental build, open the .bfg_watch file, and go through each of the glob patterns to see if their results changed
  • Lots of optimization potential here: for instance, check the mtime of directories to see if their file list has changed at all; otherwise, it can be skipped
• If so, regenerate the build file

If something you do (e.g. edit via emacs) creates a temp file in a folder that's been searched for files, that folder will be marked as changed, triggering regeneration of the backend build script. It would be nice if we could be smarter about this, but that'll probably require ninja-build/ninja#987 to be fixed first.

Evaluate using a continuous integration service:

• CircleCI
• TravisCI

To link to external libraries, we should read from pkg-config files and use the appropriate options.

Fairly self-explanatory, I think.

Currently, if there's an error in the build script, we just let the exception bubble up and kill the program. We should make the error output nicer.

Currently, variable expansion in paths can cause the string that gets sent to the shell to have spaces, but no quotes. Obviously, this is a problem. It doesn't matter in other contexts though, since 1) spaces in filenames are just a nightmare for Make, and 2) Ninja handles escaping for variables on its own.

One open question is how to handle shell globs (currently used by the install command). We could just pull the * out of the path itself and call it a day. Note that this shouldn't matter in the long-term since the install command is due for replacement.

This is mainly necessary for propagating the source language from a static library to a whole-archive to a shared library. The hard part will be deciding on when functions like executable() refer to a build and when they refer to a pre-existing file.