I couldn't see a tracking issue or roadmap for a stable 1.0 release. Does one exist? If not, is that a desirable goal, and what might be needed to achieve it?

As a general rule of thumb I like to avoid unstable (0.*) crates, though jobserver has been stable enough for me that I've added to my list of exceptions.

(side note: thank you Alex! (and other contributors))

Not sure if this is the right place, but uchardet-sys fails on windows-gnu at the moment on my fork with this error:

-- Build files have been written to: C:/projects/rust-uchardet/target/debug/build/uchardet-sys-f767bae7b802ab5d/out/build
running: "cmake" "--build" "." "--target" "install" "--config" "Debug" "--"
--- stderr
make: *** internal error: invalid --jobserver-auth string '__rust_jobserver_semaphore_2679511779'.  Stop.
thread 'main' panicked at '
command did not execute successfully, got: exit code: 2
build script failed, must exit now', C:\Users\appveyor\.cargo\registry\src\github.com-1ecc6299db9ec823\cmake-0.1.27\src\lib.rs:627:4
note: Run with `RUST_BACKTRACE=1` for a backtrace.

Presumably it eventually runs into this check: https://github.com/jaakristioja/make/blob/6371bb5dcef4ab5de16a1fddb2ae21c44588c6b8/posixos.c#L85

related to rust-lang/cc-rs#721

I was wondering if it would be safe to alias c_int along the lines of what is done here

https://github.com/tcharding/rust-bitcoinconsensus/blob/9b3496a5c827e99cf10debe2c36f7b20fd622cc4/src/types.rs

This would allow this lib to not depend on libc on unix?

For example, building in an alpine container:

6.767 error[E0425]: cannot find function `preadv2` in crate `libc`
6.767     --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/jobserver-0.1.29/src/unix.rs:598:15
6.767      |
6.767 598  |           libc::preadv2(
6.767      |                 ^^^^^^^ help: a function with a similar name exists: `preadv`
6.767      |
6.767     ::: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libc-0.2.153/src/unix/linux_like/linux/mod.rs:4868:13
6.767      |
6.767 4868 | /             pub fn preadv(
6.767 4869 | |                 fd: ::c_int,
6.767 4870 | |                 iov: *const ::iovec,
6.767 4871 | |                 iovcnt: ::c_int,
6.767 4872 | |                 offset: ::off_t,
6.767 4873 | |             ) -> ::ssize_t;
6.767      | |__________________________- similarly named function `preadv` defined here
6.767
6.768 error[E0425]: cannot find value `RWF_NOWAIT` in crate `libc`
6.768     --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/jobserver-0.1.29/src/unix.rs:606:19
6.768      |
6.768 606  |             libc::RWF_NOWAIT,
6.768      |                   ^^^^^^^^^^ help: a constant with a similar name exists: `IPC_NOWAIT`
6.768      |
6.768     ::: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libc-0.2.153/src/unix/linux_like/linux/mod.rs:2102:1
6.768      |
6.768 2102 | pub const IPC_NOWAIT: ::c_int = 0o4000;
6.768      | ----------------------------- similarly named constant `IPC_NOWAIT` defined here
6.768
6.821 For more information about this error, try `rustc --explain E0425`.

https://docs.rs/libc/latest/x86_64-unknown-linux-musl/libc/index.html?search=preadv2

(copied from rust-lang/rust#113730 (comment))

These lines seem questionable:

They lead the fds not being available to child processes by default (unless reverted by Client::configure) but from_env_ext() does not remove the environment variables. Which means child processes are instructed to access file descriptor numbers for jobserver communication that aren't open anymore and may have been reopened to point to other files. This seems like a violation of IO-safety (rust-lang/rust#116059 (comment)).

Either cloexec shouldn't be set or the environment variables should also be removed and only be added back via configure.

The master jobserver-rs depends on rand 0.7 but the last released version depends on rand 0.6.5 which fails to build in rust-lang/rust. Could we publish the master branch ?

Ref: rust-lang/cc-rs#1039

On older glibc version, the preadv2 symbol isn't available.

I think it might makes sense to use syscall directly to avoid this link-time issue.

This is a writeup of an issue where dropping the helper thread can take up to 1 second on unix (discovered while investigating rust-lang/cargo#7844). Here is a repro:

use std::sync::*;
use std::sync::atomic::*;

fn main() {
    let client = jobserver::Client::new(4).unwrap();
    static COUNT: AtomicU32 = AtomicU32::new(0);
    let tokens = Arc::new(Mutex::new(Vec::new()));
    let helper = client.into_helper_thread(move |token| {
        tokens.lock().unwrap().push(token);
        COUNT.fetch_add(1, Ordering::SeqCst);
    }).unwrap();

    // Request more tokens than what are available.
    for _ in 0..5 {
        helper.request_token();
    }
    // Wait for at least some of the requests to finish.
    while COUNT.load(Ordering::SeqCst) < 3 {
        std::thread::yield_now();
    }
    // Drop helper
    let t = std::time::Instant::now();
    drop(helper);
    let d = t.elapsed();
    if d.as_secs_f64() > 0.5 {
        println!("took {:?} seconds!!!!!!!!!!!!!", d);
    }
}

Running this usually (not always) takes about 1 second to drop.

The gist is that if you request too many tokens, and then try to drop the helper thread, it is unable to stop it. The reason is:

• for_each_request is stuck on this line, waiting for poll.
• Helper::join attempts to wake it up. But the acquire loop does not check why it is being interrupted, and simply tries to go back to polling.

It looks like #22 attempts to address this (and this test seems to pass with that change), but I have not reviewed that PR in detail.

cargo and rustc (and other tools using jobserver) will fail with a "failed to acquire jobserver token: early EOF on jobserver pipe" error in the next scenario reported in rust-lang/rust#46981 (comment) as ICE in rustc:

• make executes a subprocess (e.g. shell) with the jobserver pipe closed (marked with FD_CLOEXEC), but the MAKEFLAGS env var will still contain --jobserver-auth=3,4 (or some other pair of integers) in the subprocess.
  This is where make is wrong, if it closes the descriptors for a subprocess, then it should probably clean up MAKEFLAGS for it as well, but it doesn't, so we have to live with make possibly providing garbage descriptors.
• the subprocess (e.g. shell) opens some files so descriptors 3 and 4 are taken again now, but refer to entirely unrelated files.
• the shell runs cargo, which sees that 3 and 4 are open, concludes that they refer to a jobserver, and fails when trying to read from them.

jobserver could be more resilient in the face of garbage descriptors if it checked not only that the descriptors are valid, but also that they actually refer to a pipe rather than random unrelated files.

https://github.com/getong/rust-esplora-client/actions/runs/7746969254/job/21126393201?pr=1

I run a ci error on it, the tag is 0.1.27, but the code base has been fixed.
I don't know why this error occurs, I think it could be solved by making a new tag.

See our platform support page.

Especially for Linux, we should be testing against the appropriate kernel and libc versions, not later ones. We may be able to get away with fudging it occasionally on "not the most common OS for builds to run on", but otherwise...

On unix, when the named fifo is used and on Windows, it is possible to acquire without blocking.

For named fifo on unix, we can simply set pipe to be non-blocking before reading.

https://github.com/rust-lang/cc-rs/blob/2f587f5561fb9c3c02e5c9e186b2bea98691608a/src/parallel/job_token/unix.rs#L113

On windows, we can use WaitForSingleObject(self.sem, 0):

https://github.com/rust-lang/cc-rs/blob/2f587f5561fb9c3c02e5c9e186b2bea98691608a/src/parallel/job_token/windows.rs#L44

On Linux, we can also applies an optimization, to turn an annoymous pipe into a named fifo:

https://github.com/rust-lang/cc-rs/blob/2f587f5561fb9c3c02e5c9e186b2bea98691608a/src/parallel/job_token/unix.rs#L77

Though that optimization is verified to be incorrect on macOS, opening /dev/fd/$fd seems to return the same file description.

In the manual of GNU make 13.1.1 POSIX Jobserver Interaction:

Your tool should also examine the first word of the MAKEFLAGS variable and look for the character n. If this character is present then make was invoked with the ‘-n’ option and your tool should stop without performing any operations.

Do we want jobserver to follow the behaviour? It might break someone's misconfigured make invocation, though.

Client::configure only sets CARGO_MAKEFLAGS, and there is no way to retrieve the value it sets in the Command because there is no API for that. So it's not possible to set MAKEFLAGS yourself to pass cargo's jobserver to GNU make in a build script.

I've recently started getting the following errors in my build commands from this library on my local dev environment.

Apple M1 Max
Ventura 13.4.1

Works fine in my gitlab actions though. Any suggestions ?

Now started failing in my gitlab runners also

error[E0412]: cannot find type `RawFd` in `std::os::fd`
 --> /Users/dstorey/.cargo/registry/src/github.com-1ecc6299db9ec823/jobserver-0.1.27/src/error.rs:2:27
  |
2 | type RawFd = std::os::fd::RawFd;
  |                           ^^^^^ not found in `std::os::fd`
  |
help: consider importing this type alias
  |
1 | use std::os::unix::prelude::RawFd;
  |
help: if you import `RawFd`, refer to it directly
  |
2 - type RawFd = std::os::fd::RawFd;
2 + type RawFd = RawFd;
  |

error[E0603]: module `fd` is private
 --> /Users/dstorey/.cargo/registry/src/github.com-1ecc6299db9ec823/jobserver-0.1.27/src/error.rs:2:23
  |
2 | type RawFd = std::os::fd::RawFd;
  |                       ^^ private module
  |
note: the module `fd` is defined here

The first --jobserver-auth it finds is used, instead of the last like the Make manual asks:

Be aware that the MAKEFLAGS variable may contain multiple instances of the --jobserver-auth= option. Only the last instance is relevant.

$ echo 'fn main() {}' | MAKEFLAGS='--jobserver-auth= --jobserver-auth=3,4' rustc -
warning: failed to connect to jobserver from environment variable `MAKEFLAGS="--jobserver-auth= --jobserver-auth=3,4"`: cannot parse jobserver environment variable value: expected `fifo:PATH` or `R,W`, found ``
  |
  = note: the build environment is likely misconfigured

From rust-lang/rust#120515.

The approach outlined in this comment

https://github.com/alexcrichton/jobserver-rs/blob/9d5e6da2157a3db4e60e276185292d4f65cdaf0d/src/unix.rs#L118-L133

probably is inefficient on linux. The kernel recently gained an optimization where a write to a pipe only wakes up one reader as long as that one reader empties the pipe: torvalds/linux@0ddad21 Note the wake_next_reader flag.

Waking up a polling process will never empty the pipe since it first has to return to userspace before that might issue a read syscall, So it'll end up waking more readers than necessary which leads to lots of unnecessary context switches and wasted CPU cycles.

It would be better to simply attempt reading from the fd and only start polling when it returns EWOULDBLOCK.

It might even be beneficial to dup() the file descriptor and explicitly set the copy to blocking mode if the current process has no need for non-blocking operation.

Edit: I included a lot of hedge words, but here's the word of god saying that poll-based wakeups indeed do not benefit from that optimization: https://lkml.org/lkml/2020/2/12/1019

Ref: rust-lang/cargo#13483 rust-lang/cc-rs#962

If you do something like:

if let Some(client) = unsafe { jobserver::Client::from_env() } {
    client.configure(&mut cmd);
}

Where cmd is an existing Command.

By the time you execute the command and the pre_cmd that Client::configure set up, the Client instance has already been dropped. And dropping the Client actually closes the file descriptors (which is documented in Client::from_env. So by the time the pre-command executes, the file descriptors are either closed, or worse, were opened for something else.

It seems like Client::configure should have an explicit requirement on the lifetime of the command not exceeding that of the client.

This issue is a continuation of #27 (comment).

Jobserver users may want to report warnings or errors if the jobserver cannot be correctly inherited from the environment.

For example, rustc previously ICEd on jobserver errors (rust-lang/rust#46981), but now it either swallows them (when jobserver acquire or release fails) or silently creates a new jobserver.
However, for diagnosing issues in the build environment we'd ideally want to

• Warn if the jobserver descriptors were passed but couldn't be parsed or turned out closed
  • both name of the used env var (e.g MAKEFLAGS) and possible parsing errors should be available for reporting
• Error or warn if the jobserver descriptors are open/accessible, but corrupted, i.e. reopened by someone else and no longer refer to pipes
  • name of the used env var should be available in this case as well

Cargo probably wants something like this as well.

This crate can support producing extended error types from Client::from_env to support such diagnostics.

Docs: https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
Release announcement: https://lwn.net/Articles/913253/

On systems that support it GNU Make 4.4 will now use --jobserver-auth=fifo:PATH where PATH points to a named pipe rather than --jobserver-auth=R,W. I haven't checked if this breaks anything, but based on looking at the code it seems like it will cause the jobserver to be ignored entirely.

It looks like the crate fails to compile with a wasm target because wasm::Client::try_acquire method improperly returns an Option<Acquired> rather than a Result<Option<Acquired>>. This is fixed by #84 .

For some additional context, the compiler error output looks like this on my machine:

   Compiling jobserver v0.1.30
error[E0308]: mismatched types
  --> /path/to/.cargo/registry/src/index.crates.io-6f17d22bba15001f/jobserver-0.1.30/src/wasm.rs:51:13
   |
48 |     pub fn try_acquire(&self) -> io::Result<Option<Acquired>> {
   |                                  ---------------------------- expected `Result<Option<imp::Acquired>, std::io::Error>` because of return type
...
51 |             None
   |             ^^^^ expected `Result<Option<...>, ...>`, found `Option<_>`
   |
   = note: expected enum `Result<Option<imp::Acquired>, std::io::Error>`
              found enum `Option<_>`
help: try wrapping the expression in `Ok`
   |
51 |             Ok(None)
   |             +++    +

error[E0308]: mismatched types
   --> /path/to/.cargo/registry/src/index.crates.io-6f17d22bba15001f/jobserver-0.1.30/src/wasm.rs:54:16
    |
54  |             Ok(Acquired(()))
    |             -- ^^^^^^^^^^^^ expected `Option<Acquired>`, found `Acquired`
    |             |
    |             arguments to this enum variant are incorrect
    |
    = note: expected enum `Option<imp::Acquired>`
             found struct `imp::Acquired`
help: the type constructed contains `imp::Acquired` due to the type of the argument passed
   --> /path/to/.cargo/registry/src/index.crates.io-6f17d22bba15001f/jobserver-0.1.30/src/wasm.rs:54:13
    |
54  |             Ok(Acquired(()))
    |             ^^^------------^
    |                |
    |                this argument influences the type of `Ok`
note: tuple variant defined here
   --> /path/to/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/result.rs:506:5
    |
506 |     Ok(#[stable(feature = "rust1", since = "1.0.0")] T),
    |     ^^
help: try wrapping the expression in `Some`
    |
54  |             Ok(Some(Acquired(())))
    |                +++++            +

Trying to bootstrap rustc for x86_64-unknown-linux-gnux32 fails due to mismatching sizes for self.thread.as_pthread_t():

   Compiling jobserver v0.1.8
   Compiling env_logger v0.4.3
error[E0308]: mismatched types
   --> /home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/jobserver-0.1.8/src/lib.rs:628:40
    |
628 |                     libc::pthread_kill(self.thread.as_pthread_t(), libc::SIGUSR1);
    |                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^ expected u64, found u32

error: aborting due to previous error

error: Could not compile `jobserver`.

Caused by:
  process didn't exit successfully: `/home/glaubitz/rust/rust/build/bootstrap/debug/rustc --crate-name jobserver /home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/jobserver-0.1.8/src/lib.rs --error-format json --crate-type lib --emit=dep-info,link -C opt-level=2 -C metadata=d0dd8f3422c113bb -C extra-filename=-d0dd8f3422c113bb --out-dir /home/glaubitz/rust/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnux32/release/deps --target x86_64-unknown-linux-gnux32 -L dependency=/home/glaubitz/rust/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnux32/release/deps -L dependency=/home/glaubitz/rust/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps --extern libc=/home/glaubitz/rust/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnux32/release/deps/liblibc-0d148f7402aae6cb.rlib --cap-lints allow` (exit code: 101)
warning: build failed, waiting for other jobs to finish...
error: build failed
thread 'main' panicked at 'command did not execute successfully: "/home/glaubitz/rust/rust/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnux32" "--release" "--features" " jemalloc llvm" "--manifest-path" "/home/glaubitz/rust/rust/src/rustc/Cargo.toml" "--message-format" "json"
expected success, got: exit code: 101', src/bootstrap/compile.rs:873:8
note: Run with `RUST_BACKTRACE=1` for a backtrace.
failed to run: /home/glaubitz/rust/rust/build/bootstrap/debug/bootstrap build
Build completed unsuccessfully in 0:29:29
Makefile:22: recipe for target 'all' failed
make: *** [all] Error 1