fastbuild / fastbuild Goto Github PK
View Code? Open in Web Editor NEWHigh performance build system for Windows, OSX and Linux. Supporting caching, network distribution and more.
Home Page: https://fastbuild.org
High performance build system for Windows, OSX and Linux. Supporting caching, network distribution and more.
Home Page: https://fastbuild.org
We have a custom compiler/linker that links in clang which we use to generate reflection info for our engine and games.
The compiler emits a reflection "obj" file for each source file as well as some ancillary generated source.
The "linker" concats all the "obj" files into a big reflection db.
The kicker here is that FBuild doesn't appear to have a path for generating response files for clang and even if it did, we'd still have a bit of trouble with FBuild auto-detecting what to do based on tool file names.
Would it be reasonable to add an optional var to Library/Linker nodes which allows you to force response files?
Library('my-lib')
{
.UseResponseFiles = true/false/default
}
The currently distributed binary only supports Linux platforms with GCC 4.9 (CXXABI_1.3.8 required).
This excludes Centos7 which has GCC 4.8.
Is there a way the distributed binary can target a lower CXXABI ?
If this is bothersome for you, I could otherwise write CMakeLists for the purpose of bootstrapping to a lower CXXABI.
Thanks
Place source code of fastbuild in a folder that contains spaces in its name.
In my case it was c:\Users\User Name\Projects\fastbuild
Run fbuild on the code. the Windows res compilation will fail because command line for res compiler was malformed.
Hi,
I have a main fbuild.bff that includes other ones:
...
All is good but there are linker errors, so I modify linker options in "exe.bff" but it tells me "BFF file 'exe.bff' has changed (reparsing will occur)" and rebuilds everything including all libs.
I use '-cache' on the commandline but it doesn't seem to have any impact.
Thx
Is there a plan to support modules?
It seems to be working in visual studio (I don't know about other platforms).
https://blogs.msdn.microsoft.com/vcblog/2015/12/03/c-modules-in-vs-2015-update-1/
Caching fails on objects which follow a name.cpp.obj format. This makes caching incompatible with things such as Unreal Engine 4 which uses this format.
The responsible bit of code is ObjectNode.cpp:389:
// use preprocesed output path as input
Node * sourceFile = GetSourceFile();
fullArgs.Replace( sourceFile->GetName().Get(), tmpFileName.Get() );
This ends up replacing the output file name as well, which causes the object to just get created in my temp directory, leading to the build eventually failing when it can't find the object.
My dirty not-so-nice solution was to look for a character (' ' or '"') at the end of the source file name.
//Replace Instances followed by space
{
AString safeFileName;
safeFileName = GetSourceFile()->GetName();
safeFileName += " ";
AString safeTempName;
safeTempName = tmpFileName;
safeTempName += " ";
fullArgs.Replace(safeFileName.Get(), safeTempName.Get());
}
//Replace Instances followed by "
{
AString safeFileName;
safeFileName = GetSourceFile()->GetName();
safeFileName += "\"";
AString safeTempName;
safeTempName = tmpFileName;
safeTempName += "\"";
fullArgs.Replace(safeFileName.Get(), safeTempName.Get());
}
There's definitely better ways, this was just the fastest fix.
Showinclude write a human readable output, you should not parse it.
You can use tracker.exe instead.
It is used by MSBUILD to identify the input and output files of a process.
It will allow you to identify every files used in a build step and to know what is generated by this build step.
tracker.exe works with every kind of process: cl, lib, link, mt, midl... any.exe.
The DirectoryListNode.DoBuild() does get called, and it actually searches the filesystem, however the ExeNode does not realize that the there's one fewer ObjectNode to link, and thinks everything is just dandy, instead of relinking without the objectfile belonging to the removed cpp file.
Hi Franta,
first of all, I'm so glad you're working to further improve FASTBuild, which is already so brilliant, and that you found the time to host the sources on github. Seems it will pay off, you've already got forkers to contribute :-).
Just wanted to point at the links to fastbuild.org on your releases page here on github, those contained in "See fastbuild.org for:", seems they're broken, because they contain a backslash.
Other than that, keep up the good work, and, btw, I already succeeded in seducing few colleagues at work into using FASTBuild at least for inofficial, development builds of a large project 👍 .
Best regards,
Lucian
When a Visual Studio solution is generated, the generated GUIDs in the solution and project files do not match. When opening the generated .sln file, Visual Studio is forced to immediately update the GUID mismatch, resulting in a change in the opened file. This causes problems with version control.
There is a fix for this issue in pull request #52.
The Test()
function has the required parameter .TestOutput
which, as far as I can tell, can only be set to a filepath on disk. I would like to see other options here, such as redirecting it directly to stdout.
I use FASTBuild from within Sublime Text which captures stdout and presents that live to the user. Seeing the output of my unit tests from within the editor, especially when they fail, would be very useful.
Or is this possible already and I missed something in the docs? Maybe with a magic string that couldn't be a path, such as .TestOutput = ':stdout'
?
One useful feature I would like to see in FASTBuild is retrieving values from registry, that would be useful for example for getting paths and then copy executable after build, it's possible now if we set environment variable and use #import or with Exec and custom program. It's windows-only though. What do you think about it ?
Attempting to add a Dll to the .Libraries on and Executable fails. It seems as though the Node is an Exe rather than a Dll. I assume this means the dynlib just isn't supported yet?
FASTBuild Error #1005 - Executable() - Unsupported node type in 'Libraries'. (Node: '/Users/kylerocha/dev/Bump/Project/master_rengine/bin/OSX-Debug/dll/rEngineDLL.dll', Type: 'Exe')
Executable( '$ProjectName$-Exe-$Platform$-$Config$' )
^
--here
BFFParser::ParseIncludeDirective
always append the path of the currrent file, assuming that given string is a relative path.
Ex :
#include "c:\test.bff"
c:\test.bff(1,11): FASTBuild Error #1032 - Unable to open include 'c:\c:\test.bff'.
Trying to compile a hello world with 3000 threads takes forever to finish. It works "fine" for 1000, so i guess that's an issue spawning more threads than the ones needed to build the solution.
Sadly it doesn't provide much information as to why other than "This website contains elements classified as malware"
http://trafficlight.bitdefender.com/info?url=http%3A//fastbuild.org/&language=en_US
Hi !
I'm helping a bit on the CMake generator for Fsatbuild, and I've run into a problem, where very long command lines fail with Fastbuild.
When linking a large library, the command line can easily exceed 32K (I have 57K right here, and it's not the largest lib in the project) as CMake likes absolute paths better. And even if it used relative paths, this particular command line would be around 36K long.
I googled a bit and it seems there is a way around this:
http://blogs.msdn.com/b/oldnewthing/archive/2003/12/11/56043.aspx
I guess this would be a better way of passing information to the created process.
Best
The entire Data directory appears to be missing, resulting in tests failing. Example missing files:
Data/TestIncludeParser/fbuildcore.gcc.ii
Data/TestBuildAndLinkLibrary/fbuild.bff
Etc.
Hi,
The following code is not accepted by BFF parser :
.Count = 0
.Count + 1
Print( .Count )
Output :
fbuild.bff(2,8): FASTBuild Error #1027 - Cannot concatenate <Int> to <Any> for Variable '.Count'.
.Count + 1
^
\--here
The code of BFFParser::ParseVariableDeclaration()
checks that :
else if ( ( openToken >= '0' ) && ( openToken <= '9' ) )
{
if ( concatenation )
{
Error::Error_1027_CannotConcatenate( operatorIter, varName, BFFVariable::VAR_ANY, BFFVariable::VAR_INT );
return false;
}
Is there a particular reason to forbid integer concatenation ?
I can't seem to figure out how to get fast build to not eat compiler warnings. You get to see them if there is an error but not after a successful compile.
Am I missing something?
With this code (missing ' at the end of something)
.Test =
[
.Foo = 'something
.Bar = 'other'
]
One gets this error
fbuild.bff(30,1): FASTBuild Error #1002 - Matching closing token ] not found.
Which is a bit misleading as it's a ' that is missing and not a ]
Use .o on unix
In the example below FileA.hpp is not added as a dependency and modifications to FileA.hpp don't trigger a rebuild of main.cpp.
main.cpp
#include "FileA.hpp" // Missed since line starts with whitespace
# include "FileB.hpp" // Correctly added as a dependency
#include "FileC.hpp" // Correctly added as a dependency
This is especially problematic for us as we include Bullet in our project and it uses tabs before #include all over the place.
P.S. Thank you for making fastbuild, its awesome!!
I was expecting -clean to do this.
This flag should remove all object, executable, libraries and fdb files
Found an issue with -config
option :
FBuild.exe -config test.bff
This will use 'test.bff' and write the dependency graph in 'fbuild.fdb' (which is static).
Then if the user change the configuration file path :
FBuild.exe -config another_test.bff
The code will check first 'fbuild.fdb', which is valid since the last execution, and skip parsing even if -config
changed.
I thing that ideally the path to the fdb file should be the same as the configuration file, except for extension.
For example like this one: https://github.com/fastbuild/fastbuild/blob/master/Code/Core/Core.bff
I'm not sure if I'm doing this wrong or if this is a bug but following yields an error.
.ParentScopeArray = {}
{
.LocalVariableString = 'hello'
^ParentScopeArray + { .LocalVariableString }
}
Error:
FASTBuild Error #1026 - Variable '.LocalVariableString' not found for concatention.
^ParentScopeArray + { .LocalVariableString }
^
\--here
Hello,
I have found a weird behavior of Alias() with the following code :
Alias( 'all' )
{
.Targets = { 'to_be_defined' }
}
Exec( 'to_be_defined' )
{
.ExecOutput = 'output.txt'
.ExecInput = 'input.bff'
.ExecExecutable = 'cmd.exe'
}
Which generates this confusing error :
fbuild.bff(5,1): FASTBuild Error #1100 - Exec() - Target 'to_be_defined' already defined.
Exec( 'to_be_defined' )
^
\--here
I would have thought Alias() was only referencing targets, not defining them.
The documentation seems to imply that targets must be already defined though :
.Targets = { 'Library-Alias' // A previously defined Alias()
'tmp/Libraries/X64/Release/Core.dll' // A previously defined DLL()
'External/SDK/VS2012/libs/libcmt.lib' } // An external DLL import library
Is this behavior normal ?
We have some use cases where it would be handy to extract different parts of a file path to compose a new path and or make things relative. Something to the effect of...
.ProjectSourceRoot = "C:\my_project\source"
.InputFilePath = "C:\my_project\source\game\main.cpp
.InputFileDir = "$InputFilePath:directory$"
-- ouput: "C:\my_project\source\game\"
.InputFileName = "$InputFilePath:file$"
-- ouput: "main"
.InputFileExtension = "$InputFilePath:extension$"
-- ouput: "cpp"
.InputRelativePath = "$InputFilePath:relative_to($ProjectSourceRoot$)$"
-- output: "game\main.cpp"
I came across an error with this snippet :
.UntypedArray = {}
.Struct = []
.ArrayOfStructs = { .Struct }
.UntypedArray + .ArrayOfStructs
Which generates :
fbuild.bff(4,10): FASTBuild Error #1034 - Operation not supported: 'ArrayOfStrings' + 'ArrayOfStructs'.
Empty arrays seem to be typed by default as arrays of string when declared with {}, which later causes a typing error when I try to concatenate it with an array of structs.
Hello Franta!
Automatic isolation (from unity) of files that are worked on looks like a great feature. Unfortunately, as it is now, it is tied to how Perforce marks those files.
I would like to add support for other VCS (git in particular) for determining if file should be isolated, and I see 2 main ways to do this:
Pros:
Cons:
Add plugin support, similarily to .CachePlugin (maybe call it .VCSPlugin ?)
Pros:
Cons:
Please let me know which approach (if any) do you like more.
It will be really nice to have somekind of fbuild --list-build-targets
command switch that will enumerate all build targets one can build
Currently the only useful #defines are __WINDOWS__
, __LINUX__
, __OSX__
and anything that derives directly from those. It is not possible to add additional custom conditional compilation.
some use cases:
The only way to do these things today is to add more and more configurations in an ever expanding matrix of options
I propose adding a command line option "-define NAME" that would add an implicit #define to the .bff parsing.
Attached is a patch that does this.
Hi, I'm currently evaluating fastbuild to replace our game engine build system. I've got everything setup and building ok but for one area.
We support runtime compiling of a game dll module which can be hot loaded while the game is being debugged in visual studio. To do this I need to be able to change the name of the game.pdb file on each dll compilation as visual studio will lock the current one.
I've got this working using and environment import to randomise the name, but doing this causes a reparsing of the .bff to occur and all dependencies to also be rebuilt. I thought this would be ok with a special dll declaration with dependencies removed for the rtc, but it also triggers a full game rebuild when you next come to build and run through visual studio (which is not going to fly)
I can't see a solution to this at the moment other than the extra .bff dependency work mentioned in #55. Any ideas?
I guess this is by design, but I was wondering what the reasoning is behind the design decision to store the "timestamp of last built" and the dependencies found for each node into the serialized depgraph, instead of in a different file. If kept in a different file, I think you would no longer be in the situation that any .bff file change always causes a rebuild of everything
Using #import, if the given environment variable doesn't exist, the parsing of the .bff fails. this makes #import less useful than it could have been since if I use it anywhere, I now have to make sure this variable is always defined and I can't even do a simple ./FBuild in the .bff folder.
A reasonable solution is to add a second space delimited, optional argument to #import that would be the default value, in case the given name doesn't exist. this is similar to python's dict.get()
pro: gives maximum flexibility
con: some might find it a weird syntax
Another alternative is that #import just sets the variable to an empty string,
pro: no syntax change, simpler concept
con: less powerful
con: break backward compatibility as far as error handling is concerned.
let me know what you prefer and I'll code it.
Hi,
I tried that with no success:
.UnityInputExcludePath = '
.UnityInputExcludePath + '
One line is Ok, but once I add the second line then none of them are taken into account.
thx,
Hi again,
So, after my pull request about unity time stamp "fix" (#18) I continued to investigate my false-nothing-to-build-state while using unities.
I am French and my compiler does not output :
Note: including file: X:\A\B\C
But :
Remarque : inclusion du fichier : X:\A\B\C
Which is not recognized by the parser, resulting in ObjectNodes without dynamic dependencies.
The problem is coming from CIncludeParser::ParseMSCL_Preprocessed(), which does not handle compiler message localization :
In the end It was not just the unities, but all ObjectNodes that were missing their includes.
Your code is handling all the dependencies nicely, It just needs a better include parser ;)
the website of fastbuild Quick Start Guide just for windows?There isnot for iMac?Thanks !
For security reasons we have FBuildWorker running under a special domain user, and removed this users read permissions for the D: drive. This results in HasEnoughDiskSpace Worker.cpp failing, as GetDiskFreeSpaceExA(...) returns false when access is denied.
My solution to this was to simply remove the second part of that if, but a more detailed way to check if it's something actually wrong with the drive might be needed.
Hi, i wrote a small bash completion script for fbuild (including flags and targets completion)
its not exactly belongs to the code repository, whats the preferred way to distribute it?
I am building UE4 using the article from the wiki here. Sometimes when using remote workers the build fails with the following error:
cl: Command line error D8049 : cannot execute 'C:\tmp[...]\c1xx.dll': command line is too long to fit in debug record
I have observed the following:
I'm attaching the command I used to reproduce it in my test application. It's actually saved from Process.cpp (fullArgs
parameter) in Fastbuild.
I'm on Win7, using VS2015 professional.
Hi,
I need to call the preprocessor in my build chain. For that I use an ObjectList() with my compiler and just add '/P /Fi "%2"' to .CompilerOptions, which causes cl.exe to write preprocessor output to %2.
This fits in nicely in theory, but I got an access violation when running it. With a debugger I found the crash in ObjectNode::ProcessIncludesMSCL(), dereferencing output which was null. This is only asserted and not handled :
// FBuild\FBuildCore\Graph\ObjectNode.cpp(437)
// ProcessIncludesMSCL
//------------------------------------------------------------------------------
bool ObjectNode::ProcessIncludesMSCL( const char * output, uint32_t outputSize )
{
Timer t;
{
ASSERT( output && outputSize );
CIncludeParser parser;
bool result = parser.ParseMSCL_Output( output, outputSize );
if ( result == false )
{
FLOG_ERROR( "Failed to process includes for '%s'", GetName().Get() );
return false;
}
Called from here :
// FBuild\FBuildCore\Graph\ObjectNode.cpp(247)
// compiled ok, try to extract includes
if ( ProcessIncludesMSCL( ch.GetOut().Get(), ch.GetOutSize() ) == false )
{
return NODE_RESULT_FAILED; // ProcessIncludesMSCL will have emitted an error
}
The problem seems to come from the /ShowIncludes flag of cl.exe. On the MSDN we can read :
The /showIncludes option emits to stderr, not stdout.
For some obscure reason this is not the case when calling cl.exe with "/c %1 /Fo %2" or even by adding "/E" or "/EP", in those cases -and in fact most of the time- the output is written on stdout.
But when I add "/P /Fi" then /ShowIncludes will write on stderr, stdout will be empty and ch.GetOutput() will return nullptr.
Hi,
I wonder if it's possible to override settings an a per file basis?
Say that you have something like this
.CompilerInputFiles = { "file0.cpp", "file1.cpp " }
.Section
{
// These files needs to have opts disabled.
.CompilerFlags = "-O0"
.CompilerInputFiles = { "file2.cpp", "file3.cpp " }
}
Is that possible?
The dev branch version of Fastbuild outputs this in my french VS2015 :
1> LINK : .netmodule ou module MSIL compilÚ avec /GL trouvÚá; redÚmarrage de l'Údition de liens avec /LTCGá; ajoutez /LTCG Ó la ligne de commande de l'Údition de liens pour amÚliorer les performances de l'╔diteur de liens
I did not check the code, but could this be related to recent changes ?
Environment used: Windows 7, Visual Studio 2013
The problem appears with long argument lists, that are somewhat near the maximum command size (32767) boundary. The problem is that the total command length exceeds the boundary in certain cases, when the need for a response file is deducted wrongly.
Need for response file is deduced by the length of arguments provided for the linker. This deduction should also take into account the length of linker-command itself. In addition there are two slashes and a white space added in Process::Spawn, which adds further three characters to total length.
Following snippet demonstrates and fixes the immediate problem with linking process, but it seems that there are other call sites that use the same logic (LibraryNode, CSNode, ObjectNode). Additionally this fix is only for Windows, so probably a more general fix needs to be implemented (maybe let Process::Spawn() handle the response file logic?).
Problematic deduction:
Code/Tools/FBuild/FBuildCore/Graph/LinkerNode.cpp
Snippet:
const uint32_t MAX_COMMANDLINE_CHARS = 32767;
// Magic corresponds to number of characters added in Process::Spawn(), when __WINDOWS__ is defined.
const uint32_t MAGIC = 3;
const uint32_t totalCommandLength = m_Linker.GetLength() + fullArgs.GetLength() + MAGIC;
const bool useResponseFile = (totalCommandLength > MAX_COMMANDLINE_CHARS) && (GetFlag(LINK_FLAG_MSVC) || GetFlag(LINK_FLAG_GCC) || GetFlag(LINK_FLAG_SNC) || GetFlag(LINK_FLAG_ORBIS_LD) || GetFlag(LINK_FLAG_GREENHILLS_ELXR) || GetFlag(LINK_FLAG_CODEWARRIOR_LD));
In case a source code generator produces more than one output from the given inputs, the Exec
function cannot be used safely, as it does not allow registering all the outputs of the task.
The CMake tests try to pass lots of strange characters by define, and the following fails:
-DTARGETSTRING=""hello`~!@^$*)(+-=}{][:^'.?/ ,&|^^<>world;""
Notice that there are two spaces after the forward slash.
Only the first space in a sequence is kept in this situation.
This could be simplified to:
-DFOO="" ""
In the resulting source file, FOO would be " " instead of " ".
It seems the fastbuild tokenizer does not recognize the escape double quote and simplifies the whitespace when it should not.
Thanks
FBuildWorker creates a copy of itself in exe's directory and names itself "FBuildWorker.exe.copy".
This is so that updates via source control etc are not blocked by the application being open.
The current solution means that the FBuildWorker.exe cannot be in a network drive, and many workers referencing the same location. Because when a new worker spawns, it finds it cannot create the .copy file.
My suggestion would be to turn this into:
FBuildWorker.exe.hostname
Or to always copy the FBuildWorker into the local TMP files location on the local machine.
This would mean that the brokerage location can house the FBuildWorker, and when a new FBuildWorker is deployed to that directory, all workers will update.
Perhaps there are reasons to avoid this though?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.