Issue Description:

Unable to build LLBuild after following commits:

Git (git [email protected]:apple/swift-llbuild.git)

1. Allow clients to provide fine-grained diagnostics (#276) (detail)

Git (git [email protected]:apple/swift.git)

1. [stringref-upgrade] Use StringRef::consume_front instead of our own (detail)

2. [stringref-upgrade] Return a StringRef from (detail)

3. [stringref-upgrade] Change (detail)

https://ci.swift.org/job/oss-swift-incremental-RA-linux-ubuntu-16_10/3474/

11:07:42 FAILED: lib/BuildSystem/CMakeFiles/llbuildBuildSystem.dir/BuildSystem.cpp.o 
11:07:42 /usr/bin/clang++    -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-16_10/llbuild/include -std=c++14 -fno-rtti -fno-exceptions -Wmost -Wdocumentation  -Woverloaded-virtual -Wparentheses -Wswitch -Wunused-function -Wunused-variable -Wunused-value -Wempty-body -Wuninitialized -Wconstant-conversion -Wint-conversion -Wbool-conversion -Wenum-conversion -Wsign-compare -Wnewline-eof -Wdeprecated-declarations -Winvalid-offsetof -Wnon-virtual-dtor -fPIC -include "/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-16_10/llbuild/include/libstdc++14-workaround.h" -Wdocumentation -Wimplicit-fallthrough -Wunreachable-code -Woverloaded-virtual -O3 -MMD -MT lib/BuildSystem/CMakeFiles/llbuildBuildSystem.dir/BuildSystem.cpp.o -MF lib/BuildSystem/CMakeFiles/llbuildBuildSystem.dir/BuildSystem.cpp.o.d -o lib/BuildSystem/CMakeFiles/llbuildBuildSystem.dir/BuildSystem.cpp.o -c /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-16_10/llbuild/lib/BuildSystem/BuildSystem.cpp
11:07:42 In file included from /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-16_10/llbuild/lib/BuildSystem/BuildSystem.cpp:13:
11:07:42 /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-16_10/llbuild/include/llbuild/BuildSystem/BuildSystem.h:207:21: error: no type named 'vector' in namespace 'std'
11:07:42                std::vector<Command*>) = 0;
11:07:42                ~~~~~^
11:07:42 /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-16_10/llbuild/include/llbuild/BuildSystem/BuildSystem.h:207:27: error: expected ')'
11:07:42                std::vector<Command*>) = 0;
11:07:42                           ^
11:07:42 /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-16_10/llbuild/include/llbuild/BuildSystem/BuildSystem.h:206:53: note: to match this '('
11:07:42   virtual void cannotBuildNodeDueToMultipleProducers(Node* output,
11:07:42                                                     ^
11:07:42 In file included from /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-16_10/llbuild/lib/BuildSystem/BuildSystem.cpp:15:
11:07:42 /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-16_10/llbuild/include/llbuild/BuildSystem/BuildSystemFrontend.h:238:16: error: 'cannotBuildNodeDueToMultipleProducers' marked 'override' but does not override any member functions
11:07:42   virtual void cannotBuildNodeDueToMultipleProducers(Node* output,
11:07:42                ^
11:07:42 /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-16_10/llbuild/include/llbuild/BuildSystem/BuildSystemFrontend.h:238:16: warning: 'llbuild::buildsystem::BuildSystemFrontendDelegate::cannotBuildNodeDueToMultipleProducers' hides overloaded virtual function [-Woverloaded-virtual]
11:07:42 /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-16_10/llbuild/include/llbuild/BuildSystem/BuildSystem.h:206:16: note: hidden overloaded virtual function 'llbuild::buildsystem::BuildSystemDelegate::cannotBuildNodeDueToMultipleProducers' declared here: type mismatch at 2nd parameter ('int' vs 'std::vector<Command *>')
11:07:42   virtual void cannotBuildNodeDueToMultipleProducers(Node* output,
11:07:42                ^
11:07:42 1 warning and 3 errors generated.

Attachment: Download

Issue Description:

Appears that the build.db schema has changed and no longer contains a rule_results.id column.

Reproduction steps:

• Build a project with Xcode10b5

• Start the llbuild-ui tool in debug mode

• Attempt to load the build.db file

• See attached screenshot.

cmake version: 3.25.3
llbuild version: 5.2

It seems like llbuild cannot build cmake ninja manifests that utilize the | operator 👀

Error:

/home/theo/dev/dyadikos/build/_deps/vulkan-headers-subbuild/build.ninja:151:102: error: expected ':' token
  build vulkan-headers-populate-prefix/src/vulkan-headers-populate-stamp/vulkan-headers-populate-update | ${cmake_ninja_workdir}vulkan-headers-populate-prefix/src/vulkan-headers-populate-stamp/vulkan-headers-populate-update: CUSTOM_COMMAND vulkan-headers-populate-prefix/src/vulkan-headers-populate-stamp/vulkan-headers-populate-download

According to https://ninja-build.org/manual.html, it seems like it is implicit dependency syntax.

Implicit dependencies, either as picked up from a depfile attribute on a rule or from the syntax | dep1 dep2 on the end of a build line. The semantics are identical to explicit dependencies, the only difference is that implicit dependencies don’t show up in the $in variable.

relates to:

• SR-1141 [llbuild] Add support for understanding Swift parseable output

Issue Description:

We should support scheduling tasks which are multithreaded (for example, the Swift compiler) and which should recognized as consuming additional resources. Ideally this would be done in a way that we could automatically throttle appropriately once we have a mechanism for the Swift driver to tell us exactly how many threads it is using.

Issue Description:

Currently, llbuild stores the full key for all dependencies in the database. On a large project like LLVM, this leads to a lot of redundancy because individual dependencies (e.g., a header file) are repeated a large number of times.

This is particularly redundant because almost all keys have a corresponding rule which embeds the name as well.

We should move to storing the dependency indirectly via an ID to the rule. I forget the exact numbers, but this reduces the dependency database for LLVM by a very large mount (IIRC, around 80%).

Description

When trying to build swift 5.9.2 in Homebrew, the build fails with:

2023-12-24T16:55:37.9717810Z /tmp/swift-20231224-11721-bkt1we/llbuild/lib/Core/SQLiteBuildDB.cpp:28:10: fatal error: 'sqlite3.h' file not found
2023-12-24T16:55:37.9717913Z #include <sqlite3.h>
2023-12-24T16:55:37.9717995Z          ^~~~~~~~~~~
2023-12-24T16:55:37.9718084Z 1 error generated.

Despite sqlite3 being included as a dependency.

Reproduction

See Homebrew/homebrew-core#157113 for the full build

Expected behavior

Completed build

Environment

Swift 5.9.2

Additional information

While this very well might be a Homebrew issue, we could use some help trying to figure out if it is.

Issue Description:

We need expose functionality to turn assertions on or off in llbuild for the Swift toolchain. We will also need to pass the right value while building llbuild on swift-ci. This is controlled by the Swift build script.

Issue Description:

llbuild-analyze is a local package in llbuild which offers a command line interface for calculating the critical path given a build database. The critical path of a build defines the chain of dependent tasks that has the longest duration when adding up the individual durations of those tasks. The tool offers to export the critical path as a JSON file which can then be used for further investigation.
Adding more serialization options would result in more actionable workflows. This issue tracks adding Chromium Tracing as an export schema option which would allow users to open the created critical path file and view it in their Chrome browser or other tool which is able to read and visualize the format.
The format is well defined and all the necessary information should be available in the CriticalBuildPath object that gets returned by the algorithm - it contains all tasks that were part of the build (including their timing information) and the dependency chain that forms the critical path.
The tool lives in products/llbuild-analyze/Sources/CriticalPathTool.swift and offers the basic infrastructure of supporting multiple export formats. Those consist of json and GraphViz at the moment.

Attachment: Download

Issue Description:

....this is an odd bug.

swift-build-tool returns error code 127 when it happens to be invoked via NSTask.

An identical invocation that isn't via NSTask works fine. ...I am as puzzled as you.

To assist with reproducing this issue, I've created a portable Docker image that exactly matches my environment. Reproduction steps:

1. Extract the files to some directory
2. docker build .

Expected results: Container builds
Actual results: Docker container fails to build

Essentially:

1. The docker container tries to invoke launch.swift
2. launch.swift creates an NSTask that calls swift-build-tool
3. swift-build-tool returns error code 127 to launch.swift
4. launch.swift hits fatalError codepath

If you instead invoke swift-build-tool in exactly the same way without NSTask, everything's fine:

RUN swift launch.swift #doesn't work
# RUN swift-build-tool -f .atllbuild/llbuild.yaml #works fine

Theories:

1. Maybe corelibs-Foundation is doing something strange?
2. could swift-build-tool be expecting a tty / stdin / stdout?

Issue Description:

Discovered in #20
the order of deps should probably be maintained.

Issue Description:

This is about the llbuild repo. The file 'unittests/Basic/LaneBasedExecutionQueueTest.cpp' is not included in the 'unittests/Basic/CMakeLists.txt' file, so it is not compiled and tested for linux.

When I tried including it in a PR, the test './BasicTests/LaneBasedExecutionQueueTest.workingDirectory' failed.

The Android NDK turned on this flag a couple years ago, so I had to disable it when I started cross-compiling llbuild on my Android CI, because of build errors like this:

lib/llvm/Support/Unix/Process.inc:218:21: error: no matching function for call to 'RetryAfterSignal'
      if ((NullFD = RetryAfterSignal(-1, ::open, "/dev/null", O_RDWR)) < 0)
                    ^~~~~~~~~~~~~~~~
include/llvm/Support/Errno.h:34:13: note: candidate template ignored: couldn't infer template argument 'Fun'
inline auto RetryAfterSignal(const FailT &Fail, const Fun &F,
            ^
1 error generated.

I'm not sure if these fortify checks are specific to the Android NDK or would be applied on other platforms too.

Issue Description:

In the following example, it isn't clear that llbuild should rebuild the link on the second iteration – the semantics of the build file are such that the ln is up to date, but because it doesn't exist, and we `stat()`, then we rebuild

$ cat t.llbuild 
client:
  name: basic
  version: 0

targets:
  "": ["foo-stat"]

commands:
  C1:
    tool: shell
    outputs: ["foo"]
    args: ln -sf does/not/exist foo

  C2:
    tool: shell
    inputs: ["foo"]
    outputs: ["foo-stat"]
    args: stat -s foo | tee foo-stat

$ llbuild buildsystem build -f t.llbuild 
/bin/sh -c "ln -sf does/not/exist foo"
/bin/sh -c "stat -s foo | tee foo-stat"
st_dev=16777225 st_ino=14250438 st_mode=0120755 st_nlink=1 st_uid=501 st_gid=0 st_rdev=0 st_size=14 st_atime=1453531949 st_mtime=1453531949 st_ctime=1453531949 st_birthtime=1453531949 st_blksize=4096 st_blocks=8 st_flags=0

$ llbuild buildsystem build -f t.llbuild 
/bin/sh -c "ln -sf does/not/exist foo"

We should probably have a way for the client to specify the appropriate semantics to use for file based nodes. It isn't clear to me yet whether that should be an attribute tied to the file or the command, though.

Issue Description:

Trying to build a ninja manifest that declares an output file to a nonexistent subdirectory, the ninja implementation of llbuild fails to create the subdirectories, which results in build failure.

This does not match the upstream ninja behavior.

A minimal ninja file to reproduce this:

rule CAT
     command = cat ${in} > ${out}build

subdir/output: CAT input

This build file requires a file named `input` to exist in the directory where this is run.

relates to:

• SR-1143 [llbuild] Support for scheduling multithreaded tasks

Issue Description:

We should have support for understanding swiftc's "parseable output".

This will allow us to report improved status about compilation of large projects, because we can report on the status of individual files as they are compiled by swiftc.

This is also important for building additional features on (like better multithreaded scheduling) that rely on being able to get richer status information from the compiler.

Attachment: Download

Issue Description:

I’m trying to swift build this package: https://github.com/harlanhaskins/Postgres.swift
on Ubuntu 14.04.3

System information:
harlan@8ball ~/Postgres.swift> lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.3 LTS Release: 14.04 Codename: trusty

Swift-build-tool error message:

Cloning Packages/CLibPQ swift-build-tool: /home/buildslave/jenkins/workspace/oss-swift-linux-packages-ubuntu_14_04-one-off-build/llbuild/lib/Basic/FileInfo.cpp:54: static llbuild::basic::FileInfo llbuild::basic::FileInfo::getInfoForPath(const std::string &): Assertion `result.device == (unsigned)buf.st_dev && result.inode == (unsigned)buf.st_ino && result.size == (unsigned)buf.st_size && result.modTime.seconds == (unsigned)seconds && result.modTime.nanoseconds == (unsigned)nanoseconds' failed. 0 swift-build-tool 0x00000000004f138e llvm::sys::PrintStackTrace(llvm::raw_ostream&) + 46 1 swift-build-tool 0x00000000004f22f9 2 swift-build-tool 0x00000000004f2603 3 libpthread.so.0 0x00007f96383e7340 4 libc.so.6 0x00007f9637624cc9 gsignal + 57 5 libc.so.6 0x00007f96376280d8 abort + 328 6 libc.so.6 0x00007f963761db86 7 libc.so.6 0x00007f963761dc32 8 swift-build-tool 0x00000000004df22e llbuild::basic::FileInfo::getInfoForPath(std::string const&) + 430 9 swift-build-tool 0x00000000004bdc21 llbuild::buildsystem::BuildNode::getFileInfo() const + 129 10 swift-build-tool 0x00000000004ae74a 11 swift-build-tool 0x00000000004c5600 12 swift-build-tool 0x00000000004be39d 13 swift-build-tool 0x00000000004be29b llbuild::core::BuildEngine::build(std::string const&) + 43 14 swift-build-tool 0x00000000004aaa32 15 swift-build-tool 0x00000000004aa963 llbuild::buildsystem::BuildSystem::build(llvm::StringRef) + 67 16 swift-build-tool 0x000000000048901f llbuild::buildsystem::BuildSystemFrontend::build(llvm::StringRef) + 1263 17 swift-build-tool 0x00000000004862db 18 swift-build-tool 0x0000000000485f5a main + 170 19 libc.so.6 0x00007f963760fec5 __libc_start_main + 245 20 swift-build-tool 0x0000000000485de4 swift-build: Child process exited with signal

FB11020478 - Xcode 14 incremental builds are all serial(expecting parallel wherever applicable)

We have enclosed the Xcode project("SerialReproducer.zip") and a video("Xcode14-repro.mov") demonstrating the issue in this ticket. (Repro1 as well as Repro 2).

Attachments:

1. Xcode14-repro.mov
2. SerialReproducer.zip

Setup:
The Xcode project was created from the latest Xcode Version 14.0.
In the reproducer, there are 5 libraries - LibA, LibB, LibC, LibD, LibE.

Scenario:

1. LibA, LibB, LibC, LibD → all depend on LibE.
2. This means if there are changes in LibE, it should trigger a recompilation of files in LibA, LibB, LibC, LibD targets.
3. Since LibA, LibB, LibC, LibD targets are disjoint from each other, they should compile in parallel.

Repro1: Repro to get all serial incremental builds:

1. Select the scheme: "All".
2. Do a clean build. Notice LibA-LibD targets are compiled in parallel.
3. Now, make any changes in "LibE.swift". It can be adding a new class or struct, any meaningful change inside Xcode.
4. Build the Scheme "All".
5. Notice that targets: LibA, LibB, LibC, LibD follow a serial compilation sequence. Ideally, LibA-LibD should compile in parallel in the same way as they were compiled in the Clean Build case.
6. Make any change in LibE.swift and you will always see serial incremental builds for all successive changes/invocations.

But this is not always the case. The new build system in Xcode 14 does give parallel builds under a very special edge-case.

Repro2: Repro to get parallel builds:

1. Follow the steps in "Repro1" to setup the initial state.
2. Now make a change in LibA.swift, LibB.swift, LibC.swift, LibD.swift, LibE.swift.(it can be commenting out code, adding a new class etc) Notice: we have made changes to all the files that are a part of the compilation sequence.
3. Now build the scheme: "All".
4. Notice that the parallel incremental builds are back. We can see from the Xcode's assistant build timeline that LibA, LibB, LibC, LibD are being compiled in parallel.
5. Now, make any change just in "LibE.swift". It can be adding a new class, anything.
6. Build the scheme and notice that LibA-LibD are compiled in parallel.
7. We now get parallel builds for all the successive changes in LibE.swift.
8. This initially in Repro 1 was only giving serial builds for the exact same change. Maybe there is a state that is missing in the initial incremental build and that's why it's all serial.

For any medium to large projects, not getting parallel incremental builds brings a significant regression that will be affecting the developer experience. Can the fix be prioritized and are there any workarounds that we can explore till the fix lands in an Xcode 14 release?

Issue Description:

This test failure is seen in apple/swift#5041

FAIL: llbuild :: BuildSystem/Build/missing-inputs.llbuild (20 of 118)
******************** TEST 'llbuild :: BuildSystem/Build/missing-inputs.llbuild' FAILED ********************
Script:
--
rm -rf /home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/tests/BuildSystem/Build/Output/missing-inputs.llbuild.tmp.build
mkdir -p /home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/tests/BuildSystem/Build/Output/missing-inputs.llbuild.tmp.build
cp /home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/llbuild/tests/BuildSystem/Build/missing-inputs.llbuild /home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/tests/BuildSystem/Build/Output/missing-inputs.llbuild.tmp.build/build.llbuild
'/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/bin/llbuild' buildsystem build --serial --chdir /home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/tests/BuildSystem/Build/Output/missing-inputs.llbuild.tmp.build &> /home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/tests/BuildSystem/Build/Output/missing-inputs.llbuild.tmp.out || true
/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llvm-linux-x86_64/bin/FileCheck /home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/llbuild/tests/BuildSystem/Build/missing-inputs.llbuild --input-file /home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/tests/BuildSystem/Build/Output/missing-inputs.llbuild.tmp.out --check-prefix=CHECK-FAILURE
touch /home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/tests/BuildSystem/Build/Output/missing-inputs.llbuild.tmp.build/input
'/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/bin/llbuild' buildsystem build --serial --chdir /home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/tests/BuildSystem/Build/Output/missing-inputs.llbuild.tmp.build &> /home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/tests/BuildSystem/Build/Output/missing-inputs.llbuild.tmp2.out || true
/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llvm-linux-x86_64/bin/FileCheck /home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/llbuild/tests/BuildSystem/Build/missing-inputs.llbuild --input-file /home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/tests/BuildSystem/Build/Output/missing-inputs.llbuild.tmp2.out --check-prefix=CHECK-FAILURE-2
--
Exit Code: 1

Command Output (stdout):
--
$ "rm" "-rf" "/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/tests/BuildSystem/Build/Output/missing-inputs.llbuild.tmp.build"
$ "mkdir" "-p" "/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/tests/BuildSystem/Build/Output/missing-inputs.llbuild.tmp.build"
$ "cp" "/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/llbuild/tests/BuildSystem/Build/missing-inputs.llbuild" "/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/tests/BuildSystem/Build/Output/missing-inputs.llbuild.tmp.build/build.llbuild"
$ "/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/bin/llbuild" "buildsystem" "build" "--serial" "--chdir" "/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/tests/BuildSystem/Build/Output/missing-inputs.llbuild.tmp.build"
# redirected output from '/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/tests/BuildSystem/Build/Output/missing-inputs.llbuild.tmp.out':
/bin/sh -c true
<unknown>:0: error: build had 1 command failures

<unknown>:0: error: cannot build 'output-1' due to missing input

note: command had no output on stdout or stderr
error: command failed with exit status: 1
$ "true"
$ "/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llvm-linux-x86_64/bin/FileCheck" "/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/llbuild/tests/BuildSystem/Build/missing-inputs.llbuild" "--input-file" "/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/tests/BuildSystem/Build/Output/missing-inputs.llbuild.tmp.out" "--check-prefix=CHECK-FAILURE"
# command stderr:
/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/llbuild/tests/BuildSystem/Build/missing-inputs.llbuild:12:18: error: expected string not found in input
# CHECK-FAILURE: missing input 'input' and no rule to build it
                 ^
/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/tests/BuildSystem/Build/Output/missing-inputs.llbuild.tmp.out:1:1: note: scanning from here
/bin/sh -c true
^
/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/llbuild-linux-x86_64/tests/BuildSystem/Build/Output/missing-inputs.llbuild.tmp.out:4:26: note: possible intended match here
<unknown>:0: error: cannot build 'output-1' due to missing input
                         ^

error: command failed with exit status: 1

Issue Description:

There is a subtle bug in the current dependency scanning where we will immediately start executing a task that we have scanned and realized needs to run, but that may occur before the point at which the downstream task of which it is a dependent has actually requested it.

This generally only shows up when making us of dynamic dependencies, which is why it hasn't been noticed until now.

As a concrete example, suppose a build executes in which
C executes, computes A, realizes based on A that it needs B
so we end up with
C depends { A, B }

Now suppose that the a new build is triggered, in which the result of A differs:
C executes, computes A, realizes based on A that it needs B'

In this situation, the current dependency scanning may also traverse to B, and trigger it to rebuild even though it is not actually used.

Issue Description:

We should have support for defining the default target to build in BuildSystem manifest files.

Currently we have a hard coded value of "" (the empty string) as the default target, but this is not very readable or obvious.

This would be analogous to the same feature in Ninja:
https://ninja-build.org/manual.html#_default_target_statements

Issue Description:

The SIGUNUSED define (31) has been removed as part of glibc 2.26, which is used in Fedora 27 and Ubuntu 17.10.

The workaround I have used for compiling under Fedora 27 is to use sed to replace SIGUNUSED with SIGSYS, which is defined with the same value. This may be different on other architectures; I have checked Debian on a Raspberry Pi and SIGSYS and SIGUNUSED are defined the same.

A custom Swift driver toolchain can be used in Xcode by overriding the SWIFT_EXEC as documented in the swift-driver README (https://github.com/apple/swift-driver).

As we are experimenting with llbuild changes locally, how can we point Xcode to use the custom llbuild - mostly for checking the impact of the changes on the end-to-end developer workflows in a production Xcode environment?

Issue Description:

In Ninja 1.6, rule declarations began to support per-file scoping, see https://ninja-build.org/manual.html#ref_scope

llbuild's Ninja implementation should support this (and match whatever related behavioral changes happened).

Issue Description:

This one took us a few days to narrow down unfortunately.

With the new build system enabled, the following in the Info.plist ...

{{ <key>NSExtension</key>}}
{{ <dict>}}
{{ <key>NSExtensionAttributes</key>}}
{{ <dict>}}
{{ <key>NSExtensionActivationRule</key>}}
{{ <string>SUBQUERY ( extensionItems, $extensionItem, SUBQUERY ( $extensionItem.attachments, $attachment, ANY $attachment.registeredTypeIdentifiers UTI-CONFORMS-TO "public.url").@count == 1 ).@count == 1</string>}}

... will be transformed to ...

<key>NSExtension</key>
<dict>
<key>NSExtensionAttributes</key>
<dict>
<key>NSExtensionActivationRule</key>
<string>SUBQUERY ( extensionItems, $extensionItem, SUBQUERY ( , , ANY UTI-CONFORMS-TO "public.url").@count == 1 ).@count == 1</string>

The problem being that the New Build System substitutes $IDENTIFIER whereas the Old Build System would only do that for $(IDENTIFIER).

Because of the moment in the build where this happens, there is no workaround. We tried entity encoding the $ and putting things in a CDATA section, but that did not work.

We switched back to the Old Build System and this problem went away.

If we visit Cupertino, do we get some stickers as a reward? 🙂

is duplicated by:

• SR-1467 SwiftPM not building on iCloud Drive after file changes

Issue Description:

➜  ~ mkdir "my swift projects"
➜  ~ cd my\ swift\ projects 
➜  my swift projects mkdir myproject
➜  my swift projects cd myproject 
➜  myproject swift build --init
Creating executable package: myproject
Creating Package.swift
Creating .gitignore
Creating Sources/
Creating Sources/main.swift
Creating Tests/
➜  myproject swift build
Compile Swift Module 'myproject' (1 sources)
Linking .build/debug/myproject
➜  myproject touch Sources/main.swift 
➜  myproject swift build
➜  myproject 

(note: no build)

Expected:

➜  ~ cd Developer 
➜  Developer mkdir myproject
➜  Developer cd myproject 
➜  myproject swift build --init
Creating executable package: myproject
Creating Package.swift
Creating .gitignore
Creating Sources/
Creating Sources/main.swift
Creating Tests/
➜  myproject swift build
Compile Swift Module 'myproject' (1 sources)
Linking .build/debug/myproject
➜  myproject touch Sources/main.swift 
➜  myproject swift build
Compile Swift Module 'myproject' (1 sources)
Linking .build/debug/myproject
➜  myproject

Issue Description:

swift build fails due to path name. Discovered when following the Swift Getting Started Using the Package Manager.

Example parent folder name which causes swift build fail: 4`2/`6=Test Case

TEST CASE

Uses /bin/zsh shell

# create parent folder which will cause a build fail
export UNICODE1=`echo '\x60'`
cd ~/Desktop
mkdir 4${UNICODE1}2:${UNICODE1}6=Test\ Case
cd 4\`2:\`6=Test\ Case/
# follow Getting Started steps 
mkdir Hello
cd Hello
swift package init
swift build

RESULTING ERROR

Compile Swift Module 'Hello' (1 sources)
<unknown>:0: error: error reading dependency file: '/Users/me/Desktop/4`2:`6=Test Case/Hello/.build/debug/Hello.build/Hello.d' (missing ':' following rule)
<unknown>:0: error: error reading dependency file: '/Users/me/Desktop/4`2:`6=Test Case/Hello/.build/debug/Hello.build/Hello.d' (missing ':' following rule)
<unknown>:0: error: error reading dependency file: '/Users/me/Desktop/4`2:`6=Test Case/Hello/.build/debug/Hello.build/Hello.d' (missing ':' following rule)
<unknown>:0: error: build had 1 command failures
error: exit(1): /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift-build-tool -f /Users/me/Desktop/4`2:`6=Test\ Case/Hello/.build/debug.yaml

Attachment: Download

Issue Description:

Hi all,
It was recommended I create a ticket, so here it is.

As a test, I just wanted to import the CCairo system module. That and everything else mentioned here will be in a zip archive below. So I created a directory named Sources, and two folders named example and CCairo. I had put a cairo.so.2xxxx file in the "Link: " section of the module.modulemap file, and when run, I had a main.swift file that only had "import CCairo" inside of it.

When ran, the thing spat this:

Linking Executable: .build/debug/example
/usr/bin/ld: cannot find -l/usr/lib/x86_64-linux-gnu/libcairo.so
clang: error: linker command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: link command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: build had 1 command failures
swift-build: exit(1): ["/home/mgage/Downloads/swift-2.2-SNAPSHOT-2015-12-01-b-ubuntu15.10/usr/bin/swift-build-tool", "-f", "/home/mgage/Sources/example/.build/debug/example.o/llbuild.yaml"]

I went on the mailing list with the above machine-jabber, and was told it looks like the linker itself had a bug. I'm using the latest (2.2) and had added the thing to my path so I could use it from anywhere. Also, I'm on Ubuntu GNOME 15.04, so to take care of that possible question. Also, I'm new to Swift, but I thought it must be a big deal given that Apple hardly ever does anything like this.

It will produce "Function has token parameter but isn't an intrinsic" error on nearly every file, which may be related to llvm/llvm-project#40056

Function has token parameter but isn't an intrinsic
  call void @"?readFile@CAPIManifestActions@?A0x8C4B31D0@@UEAA?AV?$unique_ptr@VMemoryBuffer@llvm@@U?$default_delete@VMemoryBuffer@llvm@@@std@@@std@@VStringRef@llvm@@0PEBUToken@ninja@llbuild@@@Z.cold.1"(%"class.llvm::ErrorOr"* %bufferOrError, token %44) #29, !dbg !8274
in function ?readFile@CAPIManifestActions@?A0x8C4B31D0@@UEAA?AV?$unique_ptr@VMemoryBuffer@llvm@@U?$default_delete@VMemoryBuffer@llvm@@@std@@@std@@VStringRef@llvm@@0PEBUToken@ninja@llbuild@@@Z
fatal error: error in backend: Broken function found, compilation aborted!
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: T:\\Library\\Developer\\Toolchains\\unknown-Asserts-development.xctoolchain\\usr\\bin\\clang.exe -target x86_64-unknown-windows-msvc -g -gcodeview -O2 -DSWIFT_PACKAGE=1 -fblocks -I C:\\Users\\stevapple\\workspace\\swift-llbuild\\products\\libllbuild\\include -I C:\\Users\\stevapple\\workspace\\swift-llbuild\\lib\\Ninja\\include -fmodule-map-file=C:\\Users\\stevapple\\workspace\\swift-llbuild\\.build\\x86_64-unknown-windows-msvc\\release\\llbuildNinja.build\\module.modulemap -I C:\\Users\\stevapple\\workspace\\swift-llbuild\\lib\\BuildSystem\\include -fmodule-map-file=C:\\Users\\stevapple\\workspace\\swift-llbuild\\.build\\x86_64-unknown-windows-msvc\\release\\llbuildBuildSystem.build\\module.modulemap -I C:\\Users\\stevapple\\workspace\\swift-llbuild\\lib\\Core\\include -fmodule-map-file=C:\\Users\\stevapple\\workspace\\swift-llbuild\\.build\\x86_64-unknown-windows-msvc\\release\\llbuildCore.build\\module.modulemap -I C:\\Users\\stevapple\\workspace\\swift-llbuild\\lib\\Basic\\include -fmodule-map-file=C:\\Users\\stevapple\\workspace\\swift-llbuild\\.build\\x86_64-unknown-windows-msvc\\release\\llbuildBasic.build\\module.modulemap -I C:\\Users\\stevapple\\workspace\\swift-llbuild\\lib\\llvm\\Support\\include -fmodule-map-file=C:\\Users\\stevapple\\workspace\\swift-llbuild\\.build\\x86_64-unknown-windows-msvc\\release\\llvmSupport.build\\module.modulemap -I C:\\Users\\stevapple\\workspace\\swift-llbuild\\lib\\llvm\\Demangle\\include -fmodule-map-file=C:\\Users\\stevapple\\workspace\\swift-llbuild\\.build\\x86_64-unknown-windows-msvc\\release\\llvmDemangle.build\\module.modulemap -DLLVM_ON_WIN32 -Dlibllbuild_EXPORTS -D_MT -D_DLL -Xclang --dependent-lib=msvcrt -MD -MT dependencies -MF C:\\Users\\stevapple\\workspace\\swift-llbuild\\.build\\x86_64-unknown-windows-msvc\\release\\libllbuild.build\\Ninja-C-API.cpp.d -std=c++14 -c C:\\Users\\stevapple\\workspace\\swift-llbuild\\products\\libllbuild\\Ninja-C-API.cpp -o C:\\Users\\stevapple\\workspace\\swift-llbuild\\.build\\x86_64-unknown-windows-msvc\\release\\libllbuild.build\\Ninja-C-API.cpp.o
1.      <eof> parser at end of file
2.      Code generation
3.      Running pass 'Function Pass Manager' on module 'C:\Users\stevapple\workspace\swift-llbuild\products\libllbuild\Ninja-C-API.cpp'.
4.      Running pass 'Module Verifier' on function '@"?readFile@CAPIManifestActions@?A0x8C4B31D0@@UEAA?AV?$unique_ptr@VMemoryBuffer@llvm@@U?$default_delete@VMemoryBuffer@llvm@@@std@@@std@@VStringRef@llvm@@0PEBUToken@ninja@llbuild@@@Z"'
 #0 0x00007ff838734fd9 (C:\windows\System32\KERNELBASE.dll+0x34fd9)
 #1 0x00007ff6e93b335a (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x170335a)
 #2 0x00007ff6e93b6f53 (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x1706f53)
 #3 0x00007ff6e7d3ce60 (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x8ce60)
 #4 0x00007ff6e93be66e (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x170e66e)
 #5 0x00007ff6e93be751 (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x170e751)
 #6 0x00007ff6e8dd062d (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x112062d)
 #7 0x00007ff6e8cc80ac (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x10180ac)
 #8 0x00007ff6e8cc8310 (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x1018310)
 #9 0x00007ff6e8cc85ca (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x10185ca)
#10 0x00007ff6e8cc7d67 (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x1017d67)
#11 0x00007ff6e9767bc8 (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x1ab7bc8)
#12 0x00007ff6e9766d3f (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x1ab6d3f)
#13 0x00007ff6e97675e5 (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x1ab75e5)
#14 0x00007ff6ebe4552f (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x419552f)
#15 0x00007ff6eabdc053 (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x2f2c053)
#16 0x00007ff6e9da3d3e (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x20f3d3e)
#17 0x00007ff6ebe43fdb (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x4193fdb)
#18 0x00007ff6e9da3b3e (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x20f3b3e)
#19 0x00007ff6e9d678df (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x20b78df)
#20 0x00007ff6e9e1c389 (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x216c389)
#21 0x00007ff6e7d3e292 (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x8e292)
#22 0x00007ff6e7d314c6 (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x814c6)
#23 0x00007ff6e9c78567 (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x1fc8567)
#24 0x00007ff6e93b347f (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x170347f)
#25 0x00007ff6e9c78b58 (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x1fc8b58)
#26 0x00007ff6e9bef449 (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x1f3f449)
#27 0x00007ff6e9bef5ec (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x1f3f5ec)
#28 0x00007ff6e9bd691c (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x1f2691c)
#29 0x00007ff6e7d33acc (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x83acc)
#30 0x00007ff6ebb35d7c (T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\clang.exe+0x3e85d7c)
#31 0x00007ff83a367034 (C:\windows\System32\KERNEL32.DLL+0x17034)
#32 0x00007ff83a9c26a1 (C:\windows\SYSTEM32\ntdll.dll+0x526a1)
clang: error: clang frontend command failed with exit code 70 (use -v to see invocation)
swift.org clang version 15.0.0
Target: x86_64-unknown-windows-msvc
Thread model: posix
InstalledDir: T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin

Although it's most probably a clang bug, we can see if it's workaround-able, so it won't block us for building "release" Swift tools with SwiftPM.

Attachment: Download

Issue Description:

When WMO is used, swift-llbuild outputs errors of the form:

<unknown>:0: error: unable to open dependencies file '.atllbuild//llbuildtmp/Fisher-Yates_Shuffle.d'

I guess this is because -emit-dependencies doesn't do anything in this case, but it is not clear how to resolve this. My current workaround is to compile twice, both with and without the flag, which I think is illogical.

To reproduce, open the attached project, and run

$ swift-build-tool -f .atllbuild/llbuild.yaml
Compiling Swift Module 'example' (2 sources)
<unknown>:0: error: unable to open dependencies file '.atllbuild//llbuildtmp/Fisher-Yates_Shuffle.d'
<unknown>:0: error: build had 1 command failures

Issue Description:

When compiling with "buildbot_incremental,tools=RA,stdlib=RA,xcode", swiftpm, llbuild both build in Xcode debug mode even though release is specified.

I worked around this by disabling the building of llbuild/swiftpm with that preset.

Issue Description:

This is enhancement request for adding a CI job for running llbuild tests under the sanitizers.

When I tried running them under ASan I found an issue (fixed here: #691)

Issue Description:

We have disabled the tests for llbuild's signal handling on Linux, because they behaved flaky on CI. We need to make them reliable and reenable.

Issue Description:

llbuild's CMakeLists.txt explicitly adds -std=c++14 as a cxx flag:

    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++14 -fno-rtti -fno-exceptions")

This means it can't be compiled with clang 3.4, which supports c\+\+14, but the flag to -std is called c\+\+1y

Can we just change llbuild's CMakeLists.txt to use c\+\+1y?

Issue Description:

With latest master (a5d08ff), when running Utilities/bootstrap from a path with spaces, it shows the following error message. The exit code is 0, which is strange as it appears to be an error.

~/L/M/c/D/swift-package-manager ❯❯❯ Utilities/bootstrap -v
(...)
--- bootstrap: note: building self-hosted 'swift-build': env SWIFT_EXEC=/Users/bouke/Library/Mobile Documents/com~apple~CloudDocs/Developer/swift-package-manager/.build/debug/swiftc SWIFT_BUILD_PATH=/Users/bouke/Library/Mobile Documents/com~apple~CloudDocs/Developer/swift-package-manager/.build SYSROOT=/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk /Users/bouke/Library/Mobile Documents/com~apple~CloudDocs/Developer/swift-package-manager/.build/debug/swift-build-stage1 --disable-sandbox -Xlinker -rpath -Xlinker @executable_path/../lib/swift/macosx -v --build-tests
xcrun --sdk macosx --show-sdk-platform-path
xcrun --find clang
'/Users/bouke/Library/Mobile Documents/com~apple~CloudDocs/Developer/swift-package-manager/.build/debug/swiftc' --driver-mode=swift -L '/Users/bouke/Library/Mobile Documents/com~apple~CloudDocs/Developer/swift-package-manager/.build/lib/swift/pm/4' -lPackageDescription -swift-version 4 -I '/Users/bouke/Library/Mobile Documents/com~apple~CloudDocs/Developer/swift-package-manager/.build/lib/swift/pm/4' -target x86_64-apple-macosx10.10 -sdk /Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk '/Users/bouke/Library/Mobile Documents/com~apple~CloudDocs/Developer/swift-package-manager/Package.swift' -fileno 5
'/Users/bouke/Library/Mobile Documents/com~apple~CloudDocs/Developer/swift-package-manager/.build/debug/swift-build-tool' -f '/Users/bouke/Library/Mobile Documents/com~apple~CloudDocs/Developer/swift-package-manager/.build/debug.yaml' test -v
sh: /Users/bouke/Library/Mobile: No such file or directory

Issue Description:

The Clang compilation database format:
http://clang.llvm.org/docs/JSONCompilationDatabase.html
is a useful way for external tools to be able to introspect on part of a build process. llbuild should have built-in support for it.

Issue Description:

As part of optimising storage in db, keys (refer: https://bugs.swift.org/browse/SR-1140) are now stored in `key_names` table. The delete query (deleteFromKeysStmt) is created which should delete the keys from table when they are no longer required.

Reference:
https://bugs.swift.org/browse/SR-1140
#15

Issue Description:

Rev lock has occurred with the swift-bindings in llbuild and swift, since swift builds after llbuild, llbuild cannot build the bindings unless a sufficiently new swift toolchain is already installed.

Not sure what a good solution is.

Issue Description:

llbuild has a number of places where it prints out a command line to the console. When doing so, we should quote the command such that it will execute properly if rerun from the shell, which requires performing shell escaping.

For example:

$ cat t.llbuild 
client:
  name: basic
  version: 0

targets:
  "": ["<dummy>"]

commands:
  C.dummy:
    tool: shell
    args: ["echo", "hello \"world\"!"]
    outputs: ["<dummy>"]

$ llbuild buildsystem build --no-db -f t.llbuild 
echo "hello "world"!"
hello "world"!

The first command printed there echo "hello "world"!" is not escaped properly.

********************
Failed Tests (10):
  llbuild :: BuildSystem/Build/mkdir.llbuild
  llbuild :: Ninja/Build/console-pool-no-control-fd.ninja
  llbuild :: Ninja/Build/file-limit.ninja
  llbuild :: Ninja/Build/phony-inputs.ninja
  llbuild :: SwiftBuildTool/changed-commands.swift-build
  llbuild :: SwiftBuildTool/dependencies.swift-build
  llbuild :: SwiftBuildTool/swift-compiler-whole-module-optimization.swift-build
  llbuild :: SwiftBuildTool/swift-compiler.swift-build
  llbuild :: SwiftBuildTool/swift-shared-library.swift-build
  llbuild :: SwiftBuildTool/swift-version.swift-build

consoleText.txt

Issue Description:

The package file for sourcekit-lsp pulls down swift-package-manager, which itself pulls down llbuild. A new commit to llbuild has added a dependency on swift-tools-support-core, which is duplicated from SwiftPM, resulting in a total of 14 resolution errors from duplicate target names. For instance, Xcode 11b7 says:

: multiple products named 'TSCTestSupportTests' in: SwiftPM, swift-tools-support-core
: multiple products named 'TSCBasicTests' in: SwiftPM, swift-tools-support-core
: multiple products named 'TSCBasicPerformanceTests' in: SwiftPM, swift-tools-support-core
: multiple products named 'TSCUtilityTests' in: SwiftPM, swift-tools-support-core
: multiple targets named 'TSCBasic' in: SwiftPM, swift-tools-support-core
: multiple targets named 'TSCBasicPerformanceTests' in: SwiftPM, swift-tools-support-core
: multiple targets named 'TSCBasicTests' in: SwiftPM, swift-tools-support-core
: multiple targets named 'TSCLibc' in: SwiftPM, swift-tools-support-core
: multiple targets named 'TSCTestSupport' in: SwiftPM, swift-tools-support-core
: multiple targets named 'TSCTestSupportExecutable' in: SwiftPM, swift-tools-support-core
: multiple targets named 'TSCTestSupportTests' in: SwiftPM, swift-tools-support-core
: multiple targets named 'TSCUtility' in: SwiftPM, swift-tools-support-core
: multiple targets named 'TSCUtilityTests' in: SwiftPM, swift-tools-support-core
: multiple targets named 'TSCclibc' in: SwiftPM, swift-tools-support-core

Issue Description:

While building apple/swift-package-manager#456 on Linux, I regularly hit this crash in llbuild:

--- bootstrap: note: building stage1: /home/ddunbar/public/swift-project/build/Ninja-ReleaseAssert/llbuild-linux-x86_64/bin/swift-build-tool -f /home/ddunbar/public/swift-project/build/Ninja-ReleaseAss\
ert//swiftpm-linux-x86_64/.bootstrap/build.swift-build                                                                                                                                                    
swift-build-tool: /home/ddunbar/public/swift-project/llbuild/lib/Core/BuildEngine.cpp:948: void (anonymous namespace)::BuildEngineImpl::reportCycle(): Assertion `!cycleList.empty()' failed.             
0  swift-build-tool 0x00000000005205ee llvm::sys::PrintStackTrace(llvm::raw_ostream&) + 46                                                                                                                
1  swift-build-tool 0x0000000000521569                                                                                                                                                                    
2  swift-build-tool 0x0000000000521873                                                                                                                                                                    
3  libpthread.so.0  0x00007fe250691d10                                                                                                                                                                    
4  libc.so.6        0x00007fe24f847267 gsignal + 55                                                                                                                                                       
5  libc.so.6        0x00007fe24f848eca abort + 362                                                                                                                                                        
6  libc.so.6        0x00007fe24f84003d                                                                                                                                                                    
7  libc.so.6        0x00007fe24f8400f2                                                                                                                                                                    
8  swift-build-tool 0x00000000004f6960                                                                                                                                                                    
9  swift-build-tool 0x00000000004f4bec                                                                                                                                                                    
10 swift-build-tool 0x00000000004ec87d                                                                                                                                                                    
11 swift-build-tool 0x00000000004ec77b llbuild::core::BuildEngine::build(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 43                                     
12 swift-build-tool 0x00000000004ccaa2                                                                                                                                                                    
13 swift-build-tool 0x00000000004cc913 llbuild::buildsystem::BuildSystem::build(llvm::StringRef) + 67                                                                                                     
14 swift-build-tool 0x00000000004a35cd llbuild::buildsystem::BuildSystemFrontend::build(llvm::StringRef) + 1405                                                                                           
15 swift-build-tool 0x00000000004a00f6                                                                                                                                                                    
16 swift-build-tool 0x000000000049fced main + 173                                                                                                                                                         
17 libc.so.6        0x00007fe24f832a40 __libc_start_main + 240                                                                                                                                            
18 swift-build-tool 0x000000000049fb69 _start + 41                                                                                                                                                        
--- bootstrap: error: build failed with exit status -6                                                                                                                                                    

I don't see any way this is actually triggered by the code change, I suspect either a race condition that is happening to be triggered.

Issue Description:

llbuild-analyze is a local package in llbuild which offers a command line interface for calculating the critical path given a build database. The critical path of a build defines the chain of dependent tasks that has the longest duration when adding up the individual durations of those tasks. The tool offers to export the critical path as a JSON file which can then be used for further investigation.
Adding more serialization options would result in more actionable workflows. This issue tracks adding GraphViz as an export schema option which would allow users to open the created critical path file and view it in tools supporting GraphViz files.
The format is well defined and all the necessary information should be available in the CriticalBuildPath object that gets returned by the algorithm - it contains all tasks that were part of the build (including their timing information) and the dependency chain that forms the critical path.
The tool lives in products/llbuild-analyze/Sources/CriticalPathTool.swift and offers the basic infrastructure of supporting multiple export formats. Those consist of json and GraphViz at the moment while GraphViz is very rudimentary.
The current implementation exports all tasks which is a huge output even for small projects. A more actionable approach would export just the elements in the critical path with an option to add direct dependencies for a better overview.
Optionally there could be a command line option to specify how many transitive dependencies should be part of the output.

Attachment: Download

relates to:

• SR-81 Cant Build Hello World on Ubuntu 14.04

Issue Description:

I looked through my system and it turns out the header doesn't resolve. swift-build-tool doesn't seem to have a sophisticated error handling routine for unresolved dependencies.

swift-build: exit(127): ["swift-build-tool", "-v", "-f", "/users/u21/harlan/swiftpq-example/.build/debug/CLibPQ.o/llbuild.yaml"]

Issue Description:

Currently, llbuild is intended to be used in a way where all build tasks are directly exposed to the build engine running within a single process.

This is problematic when working with existing build system designs which depend on running out-of-process build tools. The traditional way we have thought about solving this was by turning each tool individually into library-based tool (potentially with its own IPC boundary) which we could then interact with more directly. However, this is difficult and disruptive.

As an alternative, we should extend llbuild to expose an IPC protocol to subprocesses (e.g., an environment variable connected to a named pipe or Unix domain socket), and a new library which clients can link against.

This library would expose the following functionality:

• Clients could register new sets of rule domains they could "match".

• Clients could request new work be performed on their behalf (potentially matching those keys).

• Clients could communicate scheduling information (such as when they are blocked waiting on another task, or when they are launching a new thread).

Under the covers, this would work by shuttling this traffic over the IPC pipe to the active build engine, which would then translate that into native APIs. Similary, when rule requests appear they would be forwarded to any actively running clients which had registered support for that rule domain (we could probably use a simple prefix matching scheme for routing).

The benefit of this approach is that it is quite easy to integrate with existing tools, and in such a way that the tool transparently uses llbuild if available, but if not simply does the work itself.

For example, consider a tool which does some computation, where one part is easily cached and may be shared with many things in the build:

func foo(name: String, input: Data) {
   let z = computeSomeProperty(input)
   ... do more work ...
}

then, assume the tool can unambiguous identify this computation (give it a name), we could write something like:

import llbuild

func init() {
  // Register rule callbacks.
  llbuild.registerRule(prefix: "ComputeSomeProperty-") { (rule, engine) in
    let name = ... extract name from rule ...
    engine.requiresInput(name) { input in
      let output = computeSomeProperty(data)
      engine.provideOutput(output)
    }
  }
}

func foo(name: String, input: Data) {
  // This will block (and notify the engine we are blocked) until available.
  let output = llbuild.computeRule("ComputeSomeProperty-\(name)")
  ... do more work ...
}

When run in a build, the engine will now only schedule this task to run once, after that it will be cached (in memory) and stored in the database. Any subsequent requests for the same value will be vended directly. We can make the library convenient to use so that if llbuild isn't connected, we just do the work as usual with minimal overhead.

duplicates:

• SR-1468 SwiftPM not building path with spaces after file changes

Issue Description:

For easier moving between my iMac and MacBook, I use iCloud Drive to sync the projects. However `swift build` doesn't recognize the changes and doesn't build the package.

Reproduce:

➜  ~ cd ~/Library/Mobile\ Documents/com\~apple\~CloudDocs          
➜  com~apple~CloudDocs mkdir myproject
➜  com~apple~CloudDocs cd myproject 
➜  myproject swift build --init
Creating executable package: myproject
Creating Package.swift
Creating .gitignore
Creating Sources/
Creating Sources/main.swift
Creating Tests/
➜  myproject swift build
Compile Swift Module 'myproject' (1 sources)
Linking .build/debug/myproject
➜  myproject touch Sources/main.swift 
➜  myproject swift build
➜  myproject

(note: no build)

Expected:

➜  ~ cd Developer 
➜  Developer mkdir myproject
➜  Developer cd myproject 
➜  myproject swift build --init
Creating executable package: myproject
Creating Package.swift
Creating .gitignore
Creating Sources/
Creating Sources/main.swift
Creating Tests/
➜  myproject swift build
Compile Swift Module 'myproject' (1 sources)
Linking .build/debug/myproject
➜  myproject touch Sources/main.swift 
➜  myproject swift build
Compile Swift Module 'myproject' (1 sources)
Linking .build/debug/myproject
➜  myproject

Issue Description:

llbuild has support for generating a GraphViz file to visualize build graphs once a build is done, but this is not exposed from the llbuild ‘build system’ tool yet. It would be really useful if this were added in.

Issue Description:

My Xcode is located at `/Applications/Xcodes/Xcode 9.3 GM.app`, and `xcode-select` is pointing to it.

When I try to `swift build` things, SPM dies because it can't find `/Applications/Xcodes/Xcode`.

Issue Description:

While working on a reproducer for https://bugs.swift.org/browse/SR-13895 I was able to get it to reproduce while using swift-llbuild2, but I couldn't dig into the issue myself with a development Swift toolchain since I can't get swift-llbuild2 to build with either a locally build Swift toolchain or the latest development toolchain snapshot from https://swift.org/download/.

Logs: https://gist.github.com/DavidGoldman/933260ebc751ffe486c1537f0af4d8b6

❯ swift build                                                                                                                                                                In file included from /Users/davg/dev/github/swift-llbuild2/.build/checkouts/swift-nio-ssl/Sources/CNIOBoringSSL/ssl/tls_record.cc:109:In file included from /Users/davg/dev/github/swift-llbuild2/.build/checkouts/swift-nio-ssl/Sources/CNIOBoringSSL/include/CNIOBoringSSL_ssl.h:149:In file included from /Users/davg/dev/github/swift-llbuild2/.build/checkouts/swift-nio-ssl/Sources/CNIOBoringSSL/include/CNIOBoringSSL_pem.h:67:In file included from /Users/davg/dev/github/swift-llbuild2/.build/checkouts/swift-nio-ssl/Sources/CNIOBoringSSL/include/CNIOBoringSSL_x509.h:76:In file included from /Users/davg/dev/github/swift-llbuild2/.build/checkouts/swift-nio-ssl/Sources/CNIOBoringSSL/include/CNIOBoringSSL_obj.h:62:20:/Users/davg/dev/github/swift-llbuild2/.build/checkouts/swift-nio-ssl/Sources/CNIOBoringSSL/include/CNIOBoringSSL_span.h:139:7: error: missing '#include <stdlib.h>'; 'abort' must be declared before it is used      abort();      ^/Applications/Xcode_12.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/stdlib.h:131:7: note: declaration here is not visiblevoid     abort(void) __cold __dead2;         ^

Issue Description:

When building llbuild with SwiftPM, we currently get the following error:

In file included from /Users/david/Projects/swift/llbuild/lib/llvm/Support/Path.cpp:14:
/Users/david/Projects/swift/llbuild/lib/llvm/Support/include/llvm/Support/COFF.h:541:5: error: expected identifier
    DEBUG,
    ^
<command line>:2:15: note: expanded from here
#define DEBUG 1
              ^
1 error generated.

This seems to be because clang is invoked with a -DDEBUG=1 define that clashes with llvmSupport's DataDirectoryIndex enum:

enum DataDirectoryIndex {
  EXPORT_TABLE = 0,
  IMPORT_TABLE,
  RESOURCE_TABLE,
  EXCEPTION_TABLE,
  CERTIFICATE_TABLE,
  BASE_RELOCATION_TABLE,
  DEBUG, // HERE
  ARCHITECTURE,
  GLOBAL_PTR,
  TLS_TABLE,
  LOAD_CONFIG_TABLE,
  BOUND_IMPORT,
  IAT,
  DELAY_IMPORT_DESCRIPTOR,
  CLR_RUNTIME_HEADER,

  NUM_DATA_DIRECTORIES
};

The only solution I've found for now is to tell SwiftPM to undefine DEBUG:

swift build -Xcxx -UDEBUG