[Compile Commands] Quotes are stripped from preprocessor definitions in cache json files about cquery HOT 21 OPEN

This may be a bug upstream in libclang. I don't believe cquery does any special processing w.r.t. quotes .

from cquery.

I suppose that is possible - however it's worth noting that the code itself does compile with MSVC, GCC, and most importantly, Clang regardless of the presence of these errors provided by cquery.

from cquery.

This appears to be a combination of a number of issues - none of which are the fault of cquery.
First, the official documentation for the JSon Compilation Database format (found here), provides the following example:

[
  { "directory": "/home/user/llvm/build",
    "command": "/usr/bin/clang++ -Irelative -DSOMEDEF=\"With spaces, quotes and \\-es.\" -c -o file.o file.cc",
    "file": "file.cc" },
  ...
]

This example does not work as expected. When libclang loads a compile_commands.json it appears to go through multiple stages: loading the JSon data, and applying transformations for escape characters. When loading the example above, the \" pieces of the string are transformed into " and used by the first transformation stage to guarantee that all content between the two quotes are part of the same define (for SOMEDEF). Similary the \\ is transformed into a single \ and then ignored. To get the expected behavior one would need the following:

[
	{
		"directory": "/home/user/llvm/build",
		"command": "/usr/bin/clang++ -Irelative -DSOMEDEF=\"\\\"With spaces, quotes and \\\\-es.\\\"\" -c -o file.o file.cc",
		"file": "file.cc"
	}
]

Additionally the script I was using to generate compile_commands.json was not correctly generating these escape patterns, which in turn was creating the issue I was experiencing.

from cquery.

In this case "arguments" is a better fit than "command".

from cquery.

I just want to mention that I'm running into the same problem with compile_commands.json generated by Ninja (which seems to be "supported" according to your documentation). Ninja writes "command", not "arguments", and if the "command" includes these kinds of escaped quotes (in -D defines), the command doesn't seem to get parsed succesfully at all, which means that cquery (in VS Code) doesn't work at all ... but I wasn't able to find any error message indicating what went wrong.
Manually editing my compile_commands.json to switch to "arguments" made cquery work (it took a while for me to figure this out!), but the quotes still seem to be removed from the -D argument, because I'm then getting compile errors where the define is used (similar to @SE2Dev's example).

This might not be cquery's fault, but it seems to make cquery unusable in some cases. I'm probably going to need to write a script that transforms "command" to "arguments" and "fixes" the escape sequences in a way that works (if possible) ...

from cquery.

@Boddlnagg What's the tool you use to generate build.ninja? If it is cmake, cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=YES. Don't use ninja -t compdb

from cquery.

I do use ninja -t compdb, because the project doesn't use cmake, nor any other off-the-shelf meta-build tool. I'm generating the ninja buildfile myself, and hoped that this would (as a byproduct) allow me to get compile_commands.json for free. But it seems like I need to do that manually as well.

from cquery.

A proper "arguments" output from Ninja requires adding vector<string> functions https://github.com/ninja-build/ninja/blob/master/src/eval_env.cc#L98
This does not seem easy to fix in the upstream.

You can also chime in in ninja-build/ninja#1381

from cquery.

This is not related to command vs arguments after all.
Ninja generates a command that contains -DSOMEDEF=\\\"foobar\\\", this works with rtags, but not with cquery. I can postprocess the compile_commands.json with in the way that @SE2Dev suggested and rewrite the problematic parameters to -DSOMEDEF=\"\\\"foobar\\\"\" (using a sed script sed --expression='s/\(\\\\\\\"[^"]\+\\\\\\\"\)/\\\"\1\\\"/g')

Therefore I think that this is a cquery bug.

from cquery.

Given this works with rtags this seems like a valid issue. @Boddlnagg the code to handle this is in project.cc, would you like to take a look at fixing?

from cquery.

from cquery.

from cquery.

https://github.com/Andersbakken/rtags/blob/2abf5213815a7389df28742464d4a9ef0335f91f/src/Server.cpp#L525-L538

It seems like this is how rtags handles it. It doesn't seem like such a burden to get it working automatically.

from cquery.

I think it is fundamentally impossible to fix this in Ninja, since for Ninja a command is also just simply a string, not an array of arguments. Ninja is cross-platform, and on Windows Ninja just passes that string to CreateProcess, and then the process itself is responsible for parsing that string.

Furthermore, as was already mentioned in this thread, the command and arguments JSON fields should be exchangable, and I would expect cquery to work with any of them just fine. It is not only Ninja that produces a single command string (Bear does the same thing, if I remember correctly).

@jacobdufault Maybe I'm able to find some time and come up with a PR, but it might take a while

from cquery.

Sure, take your time.

from cquery.

% cat b.cc
#include <stdio.h>
int main() { puts(D); }

% clang++ b.cc -o b -DD=\\\"ab\\\" && ./b
In file included from <built-in>:341:
<command line>:1:12: warning: missing terminating '"' character [-Winvalid-pp-token]
#define D \"ab\"

% clang++ b.cc -o b '-DD="ab"' && ./b  
ab

If you put extra double quotes around arguments, it will conflict with other sane compile_commands.json producers.

ninja is not suitable to produce compile_commands.json and such use is a minority. You could improve the current https://github.com/cquery-project/cquery/wiki/compile_commands.json#stdout-of-an-external-command facility instead of spending more lines of code to work around the ninja quirk.

from cquery.

@MaskRay let's see what @Boddlnagg can come up with.

If working around the quirk is relatively simple then that is a significantly better solution than adding to the wiki. If we solve it here, then no user has to deal with it.

I'm also hesitant to recommend cquery's ability to run stdout of an external command, since it doesn't support windows. Also, it seems to duplicate something that can be done pretty easily by the build system, so I'm more inclined to remove it entirely.

from cquery.

Reading over the issue again, I think we should either fix libclang so it properly extracts the quotes or write a custom compile_commands.json loader.

from cquery.

Reading over the issue again, I think we should either fix libclang so it properly extracts the quotes or write a custom compile_commands.json loader.

That sounds like the right solution to me as well. But I don't think that I can help much there ...

from cquery.

this breaks clang-check in exact same way as it breaks cquery.

Currently I work around it by checking the json for escapes, and moving offending -Ds into into cquery.index.extraClangArguments

from cquery.

@svorobiev It'd be awesome if you filed a bug against upstream, esp since clang-check is broken.

from cquery.