"D8049: command line is too long to fit in debug record" when building UE4 about fastbuild HOT 23 CLOSED

For future answer seekers getting this message on UE4 projects with or without FASTBuild:
add bLegacyPublicIncludePaths = false; to your [ProjectName].Build.cs file (or whichever module is failing).

This requires you to fix all your header includes though, but that's just healthy for your project.
The reason your build is failing is because legacy behavior is to append all public include paths to the build tool's command line. This doesn't scale and fails eventually as a project grows for tools with smaller commandline buffer. Turning off legacy include paths stops appending them to the command line at the cost of you having to put in the actual correct path for each header. But you'd want to have correct include paths anyways, right? :)

UE4 has a commandlet to help with fixing your include paths:
RunUAT.bat RebasePublicIncludePaths -Project="path/to/project.uproject" -UpdateDir="path/to/code" -Write

If you have many modules you might want to turn it on for all of them, but you can also put that same boolean into your *.Target.cs files to force it for all modules in your targets.

from fastbuild.

@Flassari The bLegacyPublicIncludePaths = false fixed the issue for me (UE4 4.25). Thank you!

from fastbuild.

Some background: On Windows, the normal command line args limit is just short of 32KiB. When this limit is exceeded, FASTBuild writes the command line to a "response file" and passes a specially formatted arg to cl.exe that references that. One thing to note is that the command line limit includes the exe path, which FASTBuild takes into account when deciding whether to use the response file. It's possible that this might explain why a particular length arg works in FASTBuild, but doesn't in your test.

Now, this particular error seems to be something a little different. As you've probably noted, there's basically zero useful info on MSDN about it, but if I had to guess, I'd guess that under some circumstances, there is some additional constraint which might mean that we need to use response files for something even shorter than 32KiB (the ~20KiB you are seeing).

Interestingly, I've never seen this particular error before (while I have seen the 32KiB limit being hit), which makes me wonder if there is something about the args that is relevant? I looked at the args you sent and nothing out of the ordinary stood out. I'll try to do some testing to see if I can narrow something down.

It'll also be interesting to see if response files fix this problem. You could test that by calling cl.exe directly as you've been doing, but with the response file args (detailed here) or by changing the limit in FASTBuild to force response files for shorter limits.

I feel like we're probably not that far away from working out exactly what the problem case is, which means we'll likely be able to add a work around.

from fastbuild.

I re-read your message where you already noted that response files didn't solve the problem. Sorry I missed that the first time.

I did some digging as well, and it just seems that the only solution here is to shorten the list of args by removing some of the include paths or defines (of which there really are a lot).

Includes: I'm not sure how much control you have over this given the needs of the unreal engine. I wonder if this is something you can reach out to Unreal about?

Defines: Again, are they all needed? If they are, maybe it's possible to move some of these defines into the precompiled header?

I'm not really sure if there are any other work-arounds available to us if response files don't work. I imagine this must also be a problem with the Unreal build tool, unless there is some work around it uses? Again, perhaps you can reach out to unreal support for some ideas?

For the "failing on remote but not locally" issue, it's entirely possible you are the very edge of a threshold and an additional few characters in the exe path (on the remote machine) is enough to push things over the limit. I am a little puzzled by why your local test fails and FASTBuild locally does not though, unless you really are very close to the edge, and perhaps just an additional space or quotes around the exe is enough to push it over?

from fastbuild.

Thanks for quick reply. Yes, unfortunately response files didn't help. I also tried to submit the command separated into two (shorter) response files, but still the same problem appeared. Experimented with tabs vs newlines in those files, but still didn't help.

When it comes to shortening includes or defines (which I agree there are tons of) it's not really viable. It would mean lots of changes to the UE4 build system, which is probably too much for me in this case.

And I don't think I'm very close to the edge: to have my local test not fail with this error I had to remove a couple hundred of arguments from the command. The one attached to this issue (command.txt) is already shortened btw - it is a couple lines above the threshold, but it wasn't like that originally.

Currently I made a workaround by not sending to remote workers those files that are using cl.exe and which have more than 20k characters. It's a very crude solution, but since there are not really that many of such files, it looks like it doesn't slow down too much. At least I hope so...

Finally yes, reaching out to Unreal support sounds good, but just today I found that I didn't have the latest VS2015 (update 2), so, since a cl.exe bug is not ruled out, I still need to check with that one. I'll come back with my findings.

My final, a bit wild idea is that not all proper .dlls are sent out (wrong .bff) and since other machines also have visual studio installed they load an incompatible library (for example by looking first at other architectures) and fail. I also need to check it somehow.

from fastbuild.

Currently I made a workaround by not sending to remote workers those files that are using cl.exe and which have more than 20k characters.

I'm a little puzzled by why this happens. Running remotely could certainly have a slightly longer path, but for the most part it would be much the same. I feel like there might be something subtle going on here that we're not fully appreciating.

My final, a bit wild idea is that not all proper .dlls are sent out (wrong .bff)

Generally, if the Compiler() section of your bff was not set correctly all your remote compilation would fail. Missing/incorrect DLLs generally fail in a very specific way, whereas this is something entirely new.

Let me know how you go with your update to update 2. I will also try to find some time to write a unit test that reproduces the behavior you're seeing. If I manage to do that, maybe it will provide some hint as to what's going on.

from fastbuild.

I've finally been able to test it with Update 2, and unfortunately my problem persists. I still strongly suspect a bug with VS, so will raise a bug with it. I'll update here how it goes.

from fastbuild.

Do you see the problem if you call cl.exe directly form the command line? (i.e. eliminating both my code and your code from the equation)

from fastbuild.

When I try to do it through cmd.exe it doesn't allow me to paste in the full command line (I think because of the circa-8k characters limit in shell).

Is there another way to do it?

from fastbuild.

We got hit by this problem. I can suggest a solution that worked for us: ensure all the workers have a short temporary directory name. The default %TEMP% value (something like %LocalAppData%\Temp) indeed leads to too long command lines.

This is basically our worker launch script:

if not exist c:\tmp  mkdir c:\tmp
set TMP=c:\tmp
set TEMP=c:\tmp
set FASTBUILD_BROKERAGE_PATH=\\fastbuild\fastbuild\tokens
xcopy /f /y /r \\fastbuild\fastbuild\bin\win64\FBuildWorker.exe %TEMP%
start %TEMP%\FBuildWorker.exe

from fastbuild.

This has just started happening to us (also UE4) with our upgrade to UE4.14. I have saved the two command lines for one that works and one that doesn't. In the working file there are 503 include paths. In the one that is failing there are 526. The include paths of roughly the same length.

The number of characters for the command lines are 23543 and 24814 for UE4.13 and UE4.14 respectively. Like the OP, this is only happening on remote workers.

from fastbuild.

I have submitted a change that fixes the issue for us, but may not be suitable for all. See gkoreman@428ca02

This has automatically been added to my pull-request #163 because I am a github noob and can't figure out how to make it stand alone.

from fastbuild.

Here are a couple of options:

Put gkoreman's fix on a toggle
If we put a toggle in the Compiler options, we wouldn't have to break recode.

Auto-detect Unreal engine and use gkoreman's fix
If we detect Unreal is being used (via a define present on the cmd line perhaps) we could apply gkoreman's fix. A bit hacky though.

Detect problem cmd line length and use gkoreman's fix
If we knew the character limit that triggers the issue (even conservatively), we could activate the fix only in those cases. We would break recode only in cases where it wouldn't be working anyway (since it doesn't compile :))

Going forward, I'm still a little puzzled as to why this doesn't fail locally. I guess if we're near some threshold of length, compiling remotely might bump it over depending on the working dir. If that's the case though, I'm a little worried that this is actually a ticking time-bomb for local builds (though that would be a problem for all users, not just FASTBuild users I guess). Has anyone with Unreal support available reached out to Epic about reigning in their include paths?

from fastbuild.

One more option:

Change the remote TEMP dir setup
As per samhocevar's comment, if the TMP dir could be shortened automatically on the remote end, that might help. I suspect this might only be helping a little bit when you are only just over whatever the limit is.

from fastbuild.

I did some experiments combined with hex viewing the .obj files and I now have a much better understand of what's going on.

It seems that when the command line args are embedded into the object file during compilation, relative include paths ( like C++ /I Editor/UnrealEd/Private are prepended with the current working directory. Full include paths are not altered.

So compiling locally in C:\Test versus remotely in C:\Users\Franta\AppData\Local\Temp.fbuild.tmp\0x00000000\core_1001 results in an extra 63 bytes per relative include. There are 453 relative includes in the Command.txt example (which means 28KiB more space used remotely compared to locally with the two example dirs above). I was able to get it to fail locally by creating a longer local working directory and adding extra includes. I didn't measure it precisely, but it seems the limit is 64KiB.

This correlate's with samhocevar's TMP dir modification (which would reduce the remote working dir space)

For completeness, here are a couple of potential strategies to get to a fix for this:

a) Have FASTBuild make the /I options into full paths On the remote machine, instead of relying on the compiler, we could make the includes full paths, using the working dir of the host machine instead of the remote machine.
Pro: This would make the command buffer usage consistent with local compilation.
Con: A little bit of work (possibly a change to the worker protocol as we might not have the necessary info)

b) Shorten the .fbuild.tmp dir Currently, the typical tmp dir on a remote host is something like C++ C:\Users\Franta\AppData\Local\Temp\.fbuild.tmp\0x00000000\core_1001 . We don't have control over all of this path, but at least some of it could be shortened (the .fbuild.tmp and the core_1001. We could cut at least 9 characters (which gives us 4KiB more breathing space in the Unreal case)
Pro: Trivial change to make
Con: Just buys some time - not really a proper fix

c) Change the TMP dir location We could make FASTBuild do what samhocevar's script does and have FASTBuild operate in C:.fastbuild.tmp\ or some similar shorter path.
Con: Later versions of Windows don't like you writing into the root of C:. (Might have permission problems)
Con: Conceivably this folder could still be longer than the local working dir

I think option b) is interesting only as a short term fix. Option c) is likely to cause new problems, so I list it mostly for completeness.
That leaves option a) is the best bet since I think it would fix it "properly". If I'm right, it would never fail remotely unless it also fails locally, at which point you really just have to reign in your command line options.

Thoughts?

from fastbuild.

from fastbuild.

It's an interesting idea, but sadly even if we strip out the /I options after pre-processing, there will still be full paths embedded in the obj file for a) debug info and b) __FILE__ directives. Additionally, to improve performance, the pre-processed output is not always used with the MSVC compiler (for example when using pre-compiled headers) so it wouldn't really help there (since naturally we can't remove the include paths in that case :) ).

In short, any change to how the /I paths are handled when building remotely won't really have any bearing on cacheability.

from fastbuild.

I've implemented the manual path expansion when compiling on the remote machine. This makes the length of the command line embedded in the obj match what happens locally. This fix (00f6b4c) will be in the next version (v0.93). I'll leave this issue open until that new version is released.

from fastbuild.

v0.93 has now been release which includes this fix.

from fastbuild.

You Can use LONG PATH TOOL for solve this problem.It is great.

from fastbuild.

I had the same problem integrating C++ Hot Reload (hotreload.tech) into UE4, it's only solved in VS2019, they will not fix this for VS2017 brrr... !!!

from fastbuild.

For future answer seekers getting this message on UE4 projects with or without FASTBuild:
add bLegacyPublicIncludePaths = false; to your [ProjectName].Build.cs file (or whichever module is failing).

This requires you to fix all your header includes though, but that's just healthy for your project.
The reason your build is failing is because legacy behavior is to append all public include paths to the build tool's command line. This doesn't scale and fails eventually as a project grows for tools with smaller commandline buffer. Turning off legacy include paths stops appending them to the command line at the cost of you having to put in the actual correct path for each header. But you'd want to have correct include paths anyways, right? :)

UE4 has a commandlet to help with fixing your include paths:
RunUAT.bat RebasePublicIncludePaths -Project="path/to/project.uproject" -UpdateDir="path/to/code" -Write

If you have many modules you might want to turn it on for all of them, but you can also put that same boolean into your *.Target.cs files to force it for all modules in your targets.

I tried this but it did not seem to work. it is not clear where to put bLegacyPublicIncludePaths = false
so I tried lots of places. I also ran the script but on my project Source directory
using windows 10 UE 4.25 clion 2019.3.5 and msvc

from fastbuild.

Specifically, I have resolved this problem in UE4 version 4.26.2, while building "Development Editor|Win64" from the command line with Visual Studio 2017, by adding
bLegacyPublicIncludePaths = false;
as the first line of method
public ShaderCompileWorker(ReadOnlyTargetRules Target) : base(Target)
in file
\Engine\Source\Programs\ShaderCompileWorker\ShaderCompileWorker.Build.cs

from fastbuild.