daandemeyer / reproc Goto Github PK

I replaced argv here https://github.com/DaanDeMeyer/reproc/blob/3e7caf8/reproc/examples/drain.c#L20 with this line but the output is always empty:

char *cmd[] = {
    "curl",
    "--proto",
    "=https",
    "--tlsv1.2",
    "-sSf",
    "https://sh.rustup.rs",
    "\0" /* tried without this also */
};

What am I doing wrong?

Hi and thank you for reproc!

I use reproc for a personal project and today I updated it but my project doesn't work anymore...

This my usage:

• Start a subprocess (an SMT solver instance that permanently wait for user input on stdin and prints result on stdout) with process.start();
• Write a string on subprocess stdinwith process.write(reinterpret_cast<const uint8_t*>(s.data()), static_cast<unsigned int>(s.length()));
• Read subprocess answer with character by character with:

uint8_t buf[1];
std::string s = "";
while(true)
{
    std::pair<size_t, std::error_code> bytes_read_and_error = process.read(reproc::stream::out, buf, 1);
    char c = (char)buf[0];
    if(c == '\n') break;
    s += c;
}

It works well before the update but now I always get a Resource temporarily unavailable on the read.

BUT, if I add a sleep(5)after the writeit works.

Do you know what's the issue?

Thank you!

Hi,

I am still playing with reproc and multithreading by I think that I need your help.
I will try to explain my issue with a minimal pseudo code.

Context: I use reproc to communicate with multiple SMT solver processes.

This is my Solver class:

class Solver
{
   int m_solver_id;
   reproc::process m_solver_process;
};

During my program initialization I start 2 solver processes like that:

std::vector<Solver *> solvers;
for(int i = 0 ; i < 2 ; i++)
{
    Solver* solver = new Solver();
    solver->m_solver_id = i;
    solver->m_solver_process = reproc::process();
    solver->m_solver_process.start({"cvc4"});
}

From here, my 2 solver processes are ready and are running.

Also, some times I need to restart one solver process with this function:

void restartSolver(Solver* solver)
{
    reproc::stop_actions process_stop_actions = {
        { reproc::stop::terminate, reproc::milliseconds(100)},
        { reproc::stop::kill, reproc::milliseconds(100)},
        {}
      };
    solver->m_process.stop(process_stop_actions);
    solver->m_solver_process = reproc::process();
    solver->m_solver_process.start({"cvc4"});
    return;
}

Now, during my program workflow, I have multiple thread that read/write on solvers stdin and stdout.

BUT, if from thread n°0 I am performing read/write operations on solver n°0, and if at the same time thread n°1 call restartSolver() on solver n°2, then I get a Operation not permitted error on the solver n°0 poll call.

Do you know why?

Thank you for your help.

Edit: Currently using 11.0.2 reprocxx version. I will try with 12.0.0 soon.

Edit: I am now on 12.1.0, so I remove the poll line of my read fonction because I want a blocking read. Now I get this error message Interrupted system call on the read. Also, I notice that I only get this error when I run my project from Xcode but not from my terminal...

I mean merge all headers into one like reproc.h, and all .c to a big reproc.c

This is an excellent lib and we use it easily to mimic the io device like QProcess in Qt.
I believe user would like to embed it in their own repos to simplify their interface/script/building env.

Or maybe the cmake object library can do the same thing.
Can we have an option to support this?

Thanks for it,
Anhong

Hello @DaanDeMeyer I'm writting a new program using your library:

#pragma once

#include <filesystem>
#include <string>
#include <future>
#include <entt/entity/registry.hpp>
#include <reproc++/reproc.hpp>
#include <antara/gaming/core/real.path.hpp>
#include <antara/gaming/ecs/system.hpp>

namespace fs = std::filesystem;

namespace antara::gaming::blockchain {

    /*struct nspv_output
    {
        nspv_output(reproc::process& background) noexcept;
        std::string output;
        std::mutex output_mutex;
        std::future<std::error_code> async_drain;
    };*/

    class nspv final : public ecs::logic_update_system<nspv> {
    public:
        nspv(entt::registry &registry, fs::path tools_path = core::assets_real_path() / "tools") noexcept;
        void update() noexcept final;
        bool spawn_nspv_instance(const std::string& coin) noexcept;
        ~nspv() noexcept final;
    private:
        std::filesystem::path tools_path_;
        using nspv_registry = std::unordered_map<std::string, reproc::process>;
        nspv_registry registry_;
    };
}

REFL_AUTO(type(antara::gaming::blockchain::nspv))

#include <loguru.hpp>
#include <future>
#include <mutex>
#include <antara/gaming/blockchain/nspv.system.hpp>
#include <reproc++/sink.hpp>

namespace antara::gaming::blockchain {
    nspv::nspv(entt::registry &registry, fs::path tools_path) noexcept :
            system(registry), tools_path_(std::move(tools_path)) {
        LOG_SCOPE_FUNCTION(INFO);
        DVLOG_F(loguru::Verbosity_INFO, "assets tool path: {}", tools_path_.string());
    }

    void nspv::update() noexcept {
        for (auto &&[coin, background] : registry_) {
            std::stringstream ss;
            background.drain(reproc::stream::out, reproc::sink::ostream(ss));
            DVLOG_F(loguru::Verbosity_INFO, "nspv output: {}", ss.str());
        }
    }

    nspv::~nspv() noexcept {
        LOG_SCOPE_FUNCTION(INFO);
        for (auto &&[coin, background] : registry_) {
            auto ec = background.stop(reproc::cleanup::terminate, reproc::milliseconds(2000), reproc::cleanup::kill,
                                      reproc::infinite);
            if (ec) {
                VLOG_SCOPE_F(loguru::Verbosity_ERROR, "error: %s", ec.message().c_str());
            }
        }
    }

    bool nspv::spawn_nspv_instance(const std::string &coin) noexcept {
        LOG_SCOPE_FUNCTION(INFO);
        registry_[coin] = reproc::process(reproc::cleanup::terminate, reproc::milliseconds(2000),
                                          reproc::cleanup::kill, reproc::infinite);
        std::array<std::string, 1> args = {tools_path_ / "nspv"};
        auto ec = registry_[coin].start(args, tools_path_.string().c_str());
        if (ec) {
            DVLOG_F(loguru::Verbosity_ERROR, "error: {}", ec.message());
            return false;
        }
        return true;
    }
}

I try to read in an infinite loop in the function update the output of each process that I'm running.

But unfortunately it's seem's blocking at the first try.

What I'm looking for retrieving output at each frame ?

I read at the example but it's only use stream when the execution is finished for example, no ?

I get process status with wait while running
I have -110 (seems to be -ETIMEDOUT)
If I kill the process from a terminal, I get 137 with kill -9, 133 with -5
in both case the error_code message is : success

How can I "translate" these status, I can't find a function to convert status to string, and a table to build a switch case?
When process die, cause kill in my case, is it ok if the error_code is (0:success)?

with my best regards

When options.working_directory is non-null, even set to a copy of the current working directory, the child process exits immediately (before execing the requested program), with a non-zero status.

Sample test code (slightly modified version of the cmake-help example):

#include <reproc++/reproc.hpp>
#include <reproc++/sink.hpp>

#include <array>
#include <iostream>
#include <unistd.h>

static int fail(std::error_code ec)
{
  std::cerr << ec.message();
  return 1;
}

// Uses reproc++ to print CMake's help page.
int main()
{
  reproc::process process;

  // The `process::start` method works with any container containing strings and
  // takes care of converting the vector into the array of null-terminated
  // strings expected by `reproc_start` (including adding the `NULL` value at
  // the end of the array).
  std::array<std::string, 2> argv = { "cmake", "--help" };
  reproc::options options;
  options.working_directory = "/tmp";

  // reproc++ uses error codes to report errors. If exceptions are preferred,
  // convert `std::error_code`'s to exceptions using `std::system_error`.
  std::error_code ec = process.start(argv, options);

  // reproc++ converts system errors to `std::error_code`'s of the system
  // category. These can be matched against using values from the `std::errc`
  // error condition. See https://en.cppreference.com/w/cpp/error/errc for more
  // information.
  if (ec == std::errc::no_such_file_or_directory) {
    std::cerr << "cmake not found. Make sure it's available from the PATH.";
    return 1;
  } else if (ec) {
    return fail(ec);
  }

  std::string output;

  // `process::drain` reads from the stdout and stderr streams of the child
  // process until both are closed or an error occurs. Providing it with a
  // string sink makes it store all output in the string(s) passed to the string
  // sink. Passing the same string to both the `out` and `err` arguments of
  // `sink::string` causes the stdout and stderr output to get stored in the
  // same string.
  ec = process.drain(reproc::sink::string(output, output));
  if (ec) {
    return fail(ec);
  }

  std::cout << output << std::flush;

  // It's easy to define your own sinks as well. Take a look at `sink.cpp` in
  // the repository to see how `sink::string` and other sinks are implemented.
  // The documentation of `process::drain` also provides more information on the
  // requirements a sink should fulfill.

  // By default, The `process` destructor waits indefinitely for the child
  // process to exit to ensure proper cleanup. See the forward example for
  // information on how this can be configured.

  return process.exit_status();
}

Hey, any reason why the variable is assigned twice time here ?

I run the testcase with separate capture of stdout and stderr:

[yuri@yv /usr/home/yuri/testcase]$ ./a.out ./exec.sh 
Hello! (stderr)
stdout: Hello! (stdout)
stderr: [yuri@yv /usr/home/testcase]$ 

stderr from the child process instead goes straight to the calling process' stderr.

testcase.zip

I tried to use the "drain" example to read the output. The idea is to have a program that could send commands to a python REPL and read/parse the output. The problem is that, even if I set the nonblocking option, it seems to get stuck. Note that I have seen the same problem on Linux and Windows.
Note that this does not seem to happen if I write my own child process reading input and sending output.
Any idea or hint on how to fix that?
Here's the code to reproduce it:

#include <reproc++/drain.hpp>
#include <reproc++/reproc.hpp>

#include <array>
#include <chrono> // std::chrono::seconds
#include <iostream>
#include <thread> // std::this_thread::sleep_for

static int fail(std::error_code ec)
{
  std::cerr << ec.message();
  return ec.value();
}

// Uses `reproc::drain` to show the output of the given command.
int main(int argc, const char **argv)
{
   reproc::options options;
  options.nonblocking = true;
  reproc::process process;

  const char *cmd[] = { "python", NULL };
  std::error_code ec = process.start(cmd);

  if (ec == std::errc::no_such_file_or_directory) {
    std::cerr << "Program not found. Make sure it's available from the PATH.";
    return ec.value();
  } else if (ec) {
    return fail(ec);
  }

  std::string output;
  reproc::sink::string sink(output);

  // sleep a bit to let the REPL fire up...
  std::this_thread::sleep_for(std::chrono::milliseconds(200));
  uint8_t repl_cmd[] = "exit()\n";
  process.write(repl_cmd, sizeof(repl_cmd));

  ec = reproc::drain(process, sink, sink); // <- It's stuck in the read!
  if (ec) {
    return fail(ec);
  }

  std::cout << output << std::flush;

  int status = 0;
  std::tie(status, ec) = process.wait(reproc::infinite);
  if (ec) {
    return fail(ec);
  }

  return status;
}

My use case is to run a process and get its output (both stdout and stderr). the process must not run more than n milliseconds (because some commands are expected to freeze). I basically want to do the following:

• run the given command for at most n milliseconds (when n milliseconds pass, first try cleaning it up gracefully and kill it if it's not responding after some very short amount of time)
• get the output (both stdout and stderr) in newly allocated strings (char *)

I feel that the API is a bit complex and I don't have much knowledge about how pipes work, so a wrapper API that does this or an example would be appreciated. here is my current code which somehow works but I think it's leaking processes:

/**
 * Runs the given command in the background, waits
 * for it to finish and returns its exit code.
 *
 * @note Only works for stdout for now.
 *
 * @param args NULL-terminated array of args.
 * @param get_stdout Whether to get the standard out
 *   (true) or stderr (false).
 * @param[out] output A pointer to save the newly
 *   allocated stdout or stderr output.
 * @param ms_timer A timer in ms to
 *   kill the process, or negative to not
 *   wait.
 */
int
system_run_cmd_w_args (
  const char ** args,
  int           ms_to_wait,
  bool          get_stdout,
  char **       output,
  bool          warn_if_fail)
{
  g_message ("ms to wait: %d", ms_to_wait);

  *output = NULL;
  size_t size = 0;
  int r = REPROC_ENOMEM;
  reproc_event_source children[1];
  bool have_events = false;
  bool read_once = false;

  reproc_options opts;
  memset (&opts, 0, sizeof (reproc_options));
  opts.stop.first.action = REPROC_STOP_WAIT;
  opts.stop.first.timeout = 100;
  opts.stop.second.action = REPROC_STOP_TERMINATE;
  opts.stop.second.timeout = 100;
  opts.stop.third.action = REPROC_STOP_KILL;
  opts.stop.third.timeout = 100;
  opts.deadline = ms_to_wait;
  opts.nonblocking = true;

  reproc_t * process = reproc_new ();

  char err_str[8000];

  if (!process)
    {
      sprintf (
        err_str,
        "create process failed for %s", args[0]);
      goto finish;
    }

  g_message ("starting...");
  r = reproc_start (process, args, opts);
  if (r < 0)
    {
      sprintf (
        err_str,
        "process failed to start for %s",
        args[0]);
      goto finish;
    }

  g_message ("closing...");
  r = reproc_close (process, REPROC_STREAM_IN);
  if (r < 0)
    {
      sprintf (
        err_str,
        "process failed to close for %s",
        args[0]);
      goto finish;
    }

  children[0].process = process;
  children[0].interests =
    get_stdout ? REPROC_EVENT_OUT : REPROC_EVENT_ERR;

  for (;;)
    {
      if (r < 0)
        {
          r = r == REPROC_EPIPE ? 0 : r;
          goto finish;
        }

      uint8_t buffer[4096];
      g_message ("polling for %d ms...", ms_to_wait);
      reproc_poll (children, 1, ms_to_wait);
      g_message ("polled");
      if (children[0].events)
        {
          have_events = true;
        }
      else
        {
          g_message ("no events");
          break;
        }

      g_message ("reading...");
      r =
        reproc_read (
          process,
          get_stdout ?
            REPROC_STREAM_OUT : REPROC_STREAM_ERR,
          buffer, sizeof (buffer));
      if (r < 0)
        {
          g_message ("failed during read");
          if (read_once)
            {
              r = 0;
            }
          break;
        }
      g_message ("read");
      read_once = true;

      size_t bytes_read = (size_t) r;

      char * result =
        realloc (*output, size + bytes_read + 1);
      if (result == NULL)
        {
          r = REPROC_ENOMEM;
          g_message ("ENOMEM");
          goto finish;
        }

      *output = result;

      // Copy new data into `output`.
      memcpy(*output + size, buffer, bytes_read);
      (*output)[size + bytes_read] = '\0';
      size += bytes_read;

      if (r == REPROC_EPIPE)
        {
          break;
        }
    }

  if (have_events && !read_once && r < 0)
    {
      sprintf (
        err_str,
        "failed to get output from process %s",
        args[0]);
      goto finish;
    }

  if (*output)
    {
      g_message ("output:\n%s", *output);
    }
  else
    {
      g_message ("no output");
    }

  g_message ("waiting...");
  /* uses deadline */
  r = reproc_wait (process, REPROC_DEADLINE);
  g_message ("waited");

  if (r < 0)
    {
      sprintf (
        err_str,
        "failed to wait for process %s",
        args[0]);
      goto finish;
    }

  finish:
  g_message ("finishing...");

  if (r < 0)
    {
      if (warn_if_fail)
        {
          g_warning ("%s", err_str);
          g_warning ("%s", reproc_strerror (r));
        }
      else
        {
          g_message ("%s", err_str);
          g_message ("%s", reproc_strerror (r));
        }
    }

  g_message ("destroying...");
  reproc_destroy (process);
  g_message ("destroyed");

  return (r < 0) ? r : 0;
}

I hope you can see that this very complex for a relatively simple use case (not sure I'm using the API properly though)

Have you considered either using vfork() instead of fork() or adding support for vfork() as an alternative to fork() for the POSIX implementation?

We are using a hand-rolled process library at my current employer that is based on Folly's subprocess implementation. We depend on vfork() since we occassionally have to fork huge processes.

There are some drawbacks over fork() especially in a multi-threaded context. Some helpful information: http://ewontfix.com/7/

Hi,
The presence of the following comment environment.hpp

Note that passing an empty container to this method will start the child
process with no environment. To have the child process inherit the environment
of the parent process, call the default constructor instead.

Highlights by its presence that it is not expected behavior.
It is furthermore dangerous to have an empty env as we will highlight it later.

If a user need add an env variable, seeing that there is a environment object and that the environment is immutable (no push),
he will try to use this constructor.

This is without consequence for users of Linux or small applications.
But for Windows this mean that your application won't have any of its system Env variables that are read by windows libraries.
This means that a user trying to just add an env variable might suddenly, and just for windows, have networking issues.
for instance: A non-recoverable error occurred during a database lookup. (os error 11003)

I just was in such a situation and it took a few hours to 3 devs to find it out.

Furthermore the library doesn't offer any way to extend the env which is a pretty basic need, a much more basic one than starting a process with no env.

Therefor creating a limited env should only be a conscious choice on the part of the user and the default behavior should be extension.

Current reference based design was a bad decision as it forces a process to be assigned to an event source whereas in reproc the process is optional. This creates a significant mismatch between the two APIs.

The big problem is that the idiomatic solution to this problem requires std::optional which is only available from C++17 onwards.

I tried moving the reproc++ polling interface into a separate library reproc++-poll that could be C++17 only without requiring C++17 for the rest of the reproc++ interface and while this works well, there's one massive blocker, reproc::drain and subsequently reproc::run as well are built on reproc::poll so those would have to move to reproc++-poll as well which would remove most of the useful stuff from reproc++.

We could use a pointer instead of a reference as an interim fix but this clashes with the value based design of reproc::process.

So the best solution seems to be to wait until we can move reproc++ to C++17. When we do this depends on reproc++'s users. I have no data on this, but I'm guessing the poll interface isn't actually used that much and for now, making sure reproc++ works with C++11 is probably more important for most users than having a better interface for reproc++::poll.

I'm opening this issue for visibility on the issue and to allow anyone to give feedback on whether reproc++ should move to C++17 to fix this issue or stay on C++11 to be a viable option for a larger range of projects. Eventually the move will happen, it's just a matter of deciding when exactly. My current intent is to not do the move unless there's a pressing need for a better reproc::poll API or some data shows up that indicates a majority of C++ developers are using at least C++17.

Resulting ld bug:

> wasm-ld: error: 'atomics' feature is used by d1_lib.o, so --shared-memory 
> must be used
>
> This one means that `dl_lib.o` was compiled with -pthread already 
> (-pthread enables the 'atomics' feature), so its atomic operations have 
> been lowered to real atomic WebAssembly instructions. However, because 
> you're not passing -pthread at link time, the linker tries to produce a 
> module with an unshared memory, which would make those atomic instructions 
> invalid.
>
> wasm-ld: error: Target feature 'atomics' used in d1_lib.o is disallowed by 
> ConcurrentScheduler.cpp.o. Use --no-check-features to suppress.
>
> This line tells us that ConcurrentScheduler.cpp.o was compiled without 
> -pthread, which means its atomic operations were lowered to non-atomic 
> WebAssembly instruction because the atomics feature was not enabled. That 
> means that ConcurrentScheduler.cpp.o is not safe to link with dl_lib.o 
> because the resulting program would not be thread-safe, even if the program 
> was thread-safe at the source level. When using the LLVM backend, the 
> wasm-ld linker helpfully recognizes this error and does not let you link an 
> unsafe program. Fastcomp would just silently link your program into a 
> thread-unsafe binary.

If I have a process that I suspect has already exited, but want to kill it if not, is this the right sequence?
i.e. Is the noop the 'exit' if already exited part? Followed by terminate and kill?

reproc::stop_actions stop = {
            { reproc::stop::noop, reproc::milliseconds(0) },
            { reproc::stop::terminate, reproc::milliseconds(TerminateProcessMilliseconds) },
            { reproc::stop::kill, reproc::milliseconds(KillProcessMilliseconds) }
        };
proc->stop(stop);

It wasn't clear to me from the examples the correct approach here.
The scenario is a process that has been running for a while, but has been asked to exit by another mechanism (it has received a network message, for example). So I 'hope' that it has already gone, but if not, I wish to close it.

Hello i have the following error when i try to launch a process:

command line: [C:\Users\Public\Documents\atomicDEX-Desktop\cmake-build-debug\bin\assets\tools/mm2/mm2], from directory: [C:\Use
rs\Public\Documents\atomicDEX-Desktop\cmake-build-debug\bin\assets\tools/mm2/]

error when spawning process No mapping for the Unicode character exists in the target multi-byte code page

        options.working_directory = strdup(tools_path.string().c_str());

        SPDLOG_DEBUG("command line: [{}], from directory: [{}]", args[0], options.working_directory);
        const auto ec = m_mm2_instance.start(args, options);
        std::free((void*)options.working_directory);
        if (ec)
        {
            SPDLOG_ERROR("error when spawning process {}", ec.message());
            std::exit(EXIT_FAILURE);
        }

I've been trying to start a process and read out the results asynchronously, but so far my app always waits until the process is done. I checked out the forward.cpp example, but that technique didn't seem to work. Any suggestions?

To be clear, I would like to start a process and continuously read the results in a background thread so that I can still interact with ui of my app.

hello the line auto ec = registry_.at(coin).background.start(args, tools_path_.string().c_str());

is not compiling for me anymore:

/Users/romanszterg/CLionProjects/antara-gaming-sdk/modules/blockchain/antara/gaming/blockchain/nspv.system.cpp:60:61: fatal error: reference to type 'const reproc::options' could not bind to an rvalue of type 'const std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::value_type *' (aka 'const char *')
        auto ec = registry_.at(coin).background.start(args, tools_path_.string().c_str());
                                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/romanszterg/CLionProjects/antara-gaming-sdk/cmake-build-debug-sfml/_deps/reproc-src/reproc++/include/reproc++/reproc.hpp:91:56: note: passing argument to parameter 'options' here
                                        const options &options = {}) noexcept;

reproc version: v2.0.0-beta.3
reproc language: C
IDE: MSVC 2017 x64
OS: Windows 10 x64

Issue
When executing a program with the command reproc_start, I still see a command prompt opening and closing.

Expected result
The executed program through reproc should be completely invisible.

Possible fix
The addition of the flag CREATE_NO_WINDOW in the variable creation_flags fixes the problem. From the Window Dev Center documentation:

CREATE_NO_WINDOW : The process is a console application that is being run without a console window. Therefore, the console handle for the application is not set.
This flag is ignored if the application is not a console application, or if it is used with either CREATE_NEW_CONSOLE or DETACHED_PROCESS.

The line reproc/src/reproc/windows/process.c:63 should be replaced by:

DWORD creation_flags = CREATE_NEW_PROCESS_GROUP | CREATE_NO_WINDOW;

I'm currently writing a daemon-like server where each connection can often crash (it's a server meant for running tests). I use forks to allow tests to die (segfaults for example) but keeping the server still accepting new connections.
There's not a lot of child management going on, besides trivial connection setup, so a reference to child connections (processes) at some point becomes irrelevant. One could simply leave the children to their own devices, but being unable to detach from them stops the process from being killed (it becomes a zombie), and thus a server can't be restarted as the current running one never "terminates".

This isn't at all important for my own project right now, but it would be something nice to have :)

Hi Daan,
Would it be possible to have a variation of reproc_read that doesn't block but just returns a status code indicating if data is waiting to be read or not?
Or alternatively a reproc_read that blocks for a specified period (e.g. in milliseconds) which returns 0 if no data is waiting to be read, > 0 if data is available and < 0 in case of error?
Groeten,
Brecht

With reproc v9 I could call test binary to run the tests, but with v10 the test binary file is not generated.

What is the reason to link Reproc to Threads::Threads?

Windows,
Visual Studio 2017
I tried to compile run.cpp as is and got lots of errors.
e.g. error C2039: 'parent': is not a member of 'reproc::options::<unnamed-type-redirect
error C2039: 'deadline': is not a member of 'reproc::options'

This is probably on me but I am not sure. My repo is here. Could you please help:

In file included from /home/reinaldo/Documents/cpp/neovim-related/gnvim/src/process_handler.cpp:21:
In file included from /home/reinaldo/Documents/cpp/neovim-related/gnvim/build/_deps/reproc-src/reproc++/include/reproc++/reproc.hpp:7:
In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../include/c++/8.2.1/memory:80:
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../include/c++/8.2.1/bits/unique_ptr.h:79:16: error: invalid application of 'sizeof' to an incomplete type 'reproc_type'
        static_assert(sizeof(_Tp)>0,
                      ^~~~~~~~~~~
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../include/c++/8.2.1/bits/unique_ptr.h:382:4: note: in instantiation of member function 'std::default_delete<reproc_type>::operator()' requested here
          get_deleter()(__p);
          ^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../include/c++/8.2.1/bits/unique_ptr.h:289:2: note: in instantiation of member function 'std::unique_ptr<reproc_type, std::default_delete<reproc_type>
      >::reset' requested here
        reset(__u.release());
        ^
/home/reinaldo/Documents/cpp/neovim-related/gnvim/build/_deps/reproc-src/reproc++/include/reproc++/reproc.hpp:95:67: note: in instantiation of member function 'std::unique_ptr<reproc_type,
      std::default_delete<reproc_type> >::operator=' requested here
  REPROCXX_EXPORT process &operator=(process &&) noexcept = default;
                                                                  ^
/home/reinaldo/Documents/cpp/neovim-related/gnvim/build/_deps/reproc-src/reproc++/include/reproc++/reproc.hpp:12:8: note: forward declaration of 'reproc_type'
struct reproc_type;
       ^
1 error generated.
make[2]: *** [tests/CMakeFiles/test_process_handler.dir/build.make:76: tests/CMakeFiles/test_process_handler.dir/__/src/process_handler.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:279: tests/CMakeFiles/test_process_handler.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 94%] Linking CXX executable ../bin/gnvim
[ 94%] Built target gnvim

Would you like to replace any double quotes by angle brackets around file names for include statements?

Reproc is a great project, Looking at the recent reproc source code, I found that there seems to be a problem in the source code. When converting argv to commandline, there is no escaping operation. The relevant code is as follows:

Looking through strings_join I didn't find any escape code:

If any of the strings in argv have spaces, quotation marks, and single quotation marks, the process that is started may not be the one you want (there are spaces in the program path), or the process argv that was started does not match the expected one.

This problem is the same as the WindowsTermainl: #1090 I reported to Windows Terminal.

The fix can refer to PR: WindowsTerminal: Fix The conhost command line is not properly escaped

Since 9b35466 it seems that polls do not reliably receive an event::exit anymore.

Example:

#include <array>
#include <cassert>
#include <iostream>
#include <memory>
#include <string>
#include <unistd.h>
#include <vector>

#include <reproc++/reproc.hpp>

int main()
{
  reproc::process process;

  reproc::options options;
  options.working_directory = "/tmp";
  options.redirect.in.type = reproc::redirect::discard;
  options.redirect.out.type = reproc::redirect::default_;
  options.redirect.err.type = reproc::redirect::discard;

  std::vector<std::string> argv = { "/bin/echo", "Hi!" };

  auto ec = process.start(argv, options);
  assert(!ec);

  reproc::event::source source{ .process = process,
                                .interests = reproc::event::out |
                                             reproc::event::exit,
                                .events = 0 };

  while (true) {
    auto ec = reproc::poll(&source, 1);
    if (ec) {
      std::cerr << "Failure to poll: " << ec.message() << '\n';
      continue;
    }

    if ((source.events & reproc::event::out) != 0) {
      std::array<uint8_t, 4096> buf{};
      auto [size, ec] = process.read(reproc::stream::out, buf.begin(),
                                     buf.size());
      if (!ec) {
        std::cerr << "Received on out: " << size << '\n';
      } else {
        std::cerr << "Failure to read from out: " << ec.message() << '\n';
      }
    }

    if ((source.events & reproc::event::exit) != 0) {
      auto [status, ec] = process.wait(reproc::milliseconds(0));

      if (ec) {
        std::cerr << "ERROR: " << ec.message() << '\n';
        exit(1);
      } else {
        std::cerr << "Process has exited\n";
        return 0;
      }
    }
  }
}

Before 9b35466 this produced the following output:

Received on out: 4
Failure to read from out: Broken pipe
Process has exited

With 9b35466 I now see

Received on out: 4
Failure to read from out: Broken pipe
Failure to poll: Broken pipe
Failure to poll: Broken pipe
Failure to poll: Broken pipe
Failure to poll: Broken pipe
Failure to poll: Broken pipe
Failure to poll: Broken pipe
Failure to poll: Broken pipe
Failure to poll: Broken pipe
<repeats>

I am not sure if 9b35466 consciously changed the API, but the current behavior is hard to use since as a user one now needs to derive an event::exit from a read error on a event::[out|err]. The API would be much easier to use if polling would still emit an event::exit here.

Hello i have the following code:

#pragma once

#include <future>
#include <string>
#include <thread>
#include <reproc++/reproc.hpp>
#include <config/config.hpp>

namespace antara::mmbot
{

    class thread_safe_string_sink
    {
    public:
        thread_safe_string_sink(std::string &out, std::mutex &mutex)
                : out_(out), mutex_(mutex)
        {}

        bool operator()(const uint8_t *buffer, unsigned int size)
        {
            std::lock_guard<std::mutex> lock(mutex_);
            out_.append(reinterpret_cast<const char *>(buffer), size);
            return true;
        }

    private:
        std::string &out_;
        std::mutex &mutex_;
    };

    class mm2_client
    {
    public:
        explicit mm2_client(const antara::mmbot::config &cfg) : mmbot_cfg_(cfg)
        {
            VLOG_SCOPE_F(loguru::Verbosity_INFO, pretty_function);
            using namespace std::literals;
            std::array<std::string, 1> args = {std::filesystem::current_path() / "assets/mm2"};
            auto path = (std::filesystem::current_path() / "assets/").string();
            auto ec = background_.start(args, std::addressof(path));
            if (ec) {
                VLOG_SCOPE_F(loguru::Verbosity_ERROR, "error: %s", ec.message().c_str());
            }
            std::string output;
            std::mutex mutex;

            auto drain_async = std::async(std::launch::async, [this, &output,
                    &mutex]() {
                thread_safe_string_sink sink(output, mutex);
                return this->background_.drain(reproc::stream::out, sink);
            });


            drain_async.wait_for(2s);
            {
                std::lock_guard<std::mutex> lock(mutex);
                VLOG_SCOPE_F(loguru::Verbosity_INFO, "%s", output.c_str());
            }
        }

        ~mm2_client()
        {
            VLOG_SCOPE_F(loguru::Verbosity_INFO, pretty_function);
            auto ec = background_.stop(reproc::cleanup::kill, reproc::infinite, reproc::cleanup::terminate, reproc::milliseconds(0));
            if (ec) {
                VLOG_SCOPE_F(loguru::Verbosity_ERROR, "error: %s", ec.message().c_str());
            }
        }

    private:
        [[maybe_unused]] const antara::mmbot::config &mmbot_cfg_;
        reproc::process background_{reproc::cleanup::kill, reproc::infinite, reproc::cleanup::terminate, reproc::milliseconds(0)};
    };
}

My goal is simply to launch the program in background and continue the execution of my program that will use this program in background (mm2)

But when i write the unit tests:

TEST_CASE ("test launch mm2 in background")
    {
        auto cfg = load_mmbot_config(std::filesystem::current_path() / "assets", "mmbot_config.json");
        antara::mmbot::mm2_client mm2(cfg);

        nlohmann::json json_data = {{"method", "version"}, {"userpass", cfg.mm2_rpc_password}};
        auto resp = RestClient::post(antara::mmbot::mm2_endpoint, "application/json", json_data.dump());
        CHECK_EQ(200, resp.code);
        DVLOG_F(loguru::Verbosity_INFO, "body: %s", resp.body.c_str());
    }

The program looks like it is starting correctly but does not continue its execution and does not call the destructor, should it be launched in a separate thread? I did not really understand.

The idea of ​​the constructor, it is to launch the program in background, wait one or two seconds, display the output to check that the program is well started, and continue the execution of my main program.

When i comment the async logging part, it's seem's to work (unit tests is passing) and i have any output of version printed.

Roman Sztergbaum
Komodo Software Developer and Blockchain Architect

Hi, this is my context:

My main thread launches two threads (each thread start a process and perform reads and writes).
On my main thread I wait until the faster thread finish (the winner).
Then, the main thread want to stop the loser thread in order to continue the execution of my program.

If I call close/drain/stop on the thread loser process from my main thread then I get Assertion failed: (process->in != 0), function reproc_write because the loser thread currently perform some read or write on the process.

Do you know how I can "kill" the loser thread process even if there is a read or write operation on it?

Thank you

I'm writing a reproc binding to lua, from what I tested REPROC_ENV_EXTEND does not allow me to replace existing envvars. Is this an explicit design decision or is it a bug?

An example of replacing existing envvars would be trying to append to PATH.

Hi @DaanDeMeyer,

First of all, thanks a lot for the great library!
I'm trying to migrate from my own half-baked implementation to your library and it feels much much better than anything I could've implemented myself!

There is one thing I cannot figure out yet. Is there an API to determine whether a child process signaled or not?
Currently, I can only see the exit status, but it is not very helpful since value 134 could mean either exit(134) or abort().

I would appreciate any hints on the subject.

Cheers,
Alex.

It looks like this branch of #if defined(ATTRIBUTE_LIST_FOUND) was not updated when the options parameter was added. Therefore, a compilation error occurs if the ATTRIBUTE_LIST_FOUND is not defined.

It seems that on Windows one can put an attribute to the StartInfo struct startInfo.Verb = "runas";

https://stackoverflow.com/questions/133379/elevating-process-privilege-programmatically

I am wondering if that could be added as a feature?

Hello Daan, Great library. Thank you very much for releasing.

Can you accept a pull request for child PID ? Ive exposed the process->handle on both c and c++ side
through reproc_childPid and childPid methods respectively. I am using the childPids for accounting purposes and
so I needed the API.

Thanks,

-mp

Thanks for this excellent project!
I have a software requirement that trying to do pimpl using reproc but no idea how to integrate.

Basically what I need is to have a base class comm_interface. then I have many derived classes, like asio_comm_interface, it connects to a running server and communicates via socket, I also want to provide a reproc_comm_interface, then the user can spawn a process locally and communicate with it by read/write stdio.

Because we design it in PIMPL way, the lib user shouldn't know anything about reproc, or asio, they just do

auto p = new reproc_comm_interface(/*args*/);

then use interfaces defined in common_interface to do read/write.
we plan to only provides a header libnvc.hpp and a libnvc.a
But currently no matter how, I have to build the libreproc.a and ship to the user, use have to link to both.

Is there any way we can do to collect all symbols just in libnvc.a, not an extra libreproc.a?
asio doesn't have this issue since it's header only.
we also depend on mpack, it provides a single file to include all and we can build with it, then it's ok.

Thanks,
etorth

When compiling with Visual Studio 2015, the following error was encountered with reproc.c:

reproc.c(42): error C2065: 'false': undeclared identifier

as the boolean type is not valid for the C compiler.

This has been corrected by adding:

#ifdef _MSC_VER
#ifndef false
#define false 0
#endif
#endif

Right after the include block in reproc.c

reproc version: v2.0.0-beta.4
reproc language: C
IDE: Apple LLVM version 9.0.0 (Xcode 9.2)
OS: macOS 10.12.6 x64

Catch2

This is not an issue but more a comment about some troubles related to the usage of Catch2 to create unit tests and lldb to manage debug sessions.

Catch2 listen signals (like SIGTERM) and then abort the current test that call REPROC_TERMINATE. This gives Catch2 message like that:

FAILED:
{Unknown expression after the reported line}
due to a fatal error condition:
SIGTERM - Termination request signal

To remove this failure, it is needed to add a signal hander. This code is enough for that:

// Do nothing
void signal_handler(int) {}
// ...
// At the beginning of the unit test
TEST_CASE("...") {
  std::signal(SIGTERM, _test_signal_handler);
}

LLDB

In case you try to debug with LLDB (and maybe GDB) you may have surprise. At least under macOS, the error returned is not what you want. For example, for an invalid command (e.g /I/am/invalid), the returned error is not REPROC_FILE_NOT_FOUND but REPROC_PERMISSION_DENIED. I do not thing there is a workaround for that.

My unit tests on travis are failing with the following line on windows (msys):

Assertion failed: status <= 0x7fffffff, file ../subprojects/reproc/reproc/src/process.windows.c, line 470

https://travis-ci.org/github/zrythm/zrythm/jobs/727719758

this is what my test code is doing:

  char * output;
  int ret;

  const char * args[] = {
    "bash", "-c", "sleep 6 && echo hello", NULL, };
  const char * args_stderr[] = {
    "bash", "-c", "sleep 6 && >&2 echo hello", NULL, };
  ret =
    system_run_cmd_w_args (
      args, 1, true, &output, false);
  g_assert_cmpint (ret, !=, 0);
  ret =
    system_run_cmd_w_args (
      args, 1, false, &output, false);
  g_assert_cmpint (ret, !=, 0);
  ret =
    system_run_cmd_w_args (
      args_stderr, 1, true, &output, false);
  g_assert_cmpint (ret, !=, 0);
  ret =
    system_run_cmd_w_args (
      args_stderr, 1, false, &output, false);
  g_assert_cmpint (ret, !=, 0);

  ret =
    system_run_cmd_w_args (
      args, 2000, true, &output, false);
  g_assert_cmpint (ret, !=, 0);
  ret =
    system_run_cmd_w_args (
      args, 2000, false, &output, false);
  g_assert_cmpint (ret, !=, 0);
  ret =
    system_run_cmd_w_args (
      args_stderr, 2000, true, &output, false);
  g_assert_cmpint (ret, !=, 0);
  ret =
    system_run_cmd_w_args (
      args_stderr, 2000, false, &output, false);
  g_assert_cmpint (ret, !=, 0);

here is the function I'm testing:

/**
 * Runs the given command in the background, waits
 * for it to finish and returns its exit code.
 *
 * @note Only works for stdout for now.
 *
 * @param args NULL-terminated array of args.
 * @param get_stdout Whether to get the standard out
 *   (true) or stderr (false).
 * @param[out] output A pointer to save the newly
 *   allocated stdout or stderr output.
 * @param ms_timer A timer in ms to
 *   kill the process, or negative to not
 *   wait.
 */
int
system_run_cmd_w_args (
  const char ** args,
  int           ms_to_wait,
  bool          get_stdout,
  char **       output,
  bool          warn_if_fail)
{
  g_message ("ms to wait: %d", ms_to_wait);

  *output = NULL;

  reproc_options opts;
  memset (&opts, 0, sizeof (reproc_options));
  opts.stop.first.action = REPROC_STOP_WAIT;
  opts.stop.first.timeout = REPROC_DEADLINE;
  opts.stop.second.action = REPROC_STOP_TERMINATE;
  opts.stop.second.timeout = 1000;
  opts.stop.third.action = REPROC_STOP_KILL;
  opts.stop.third.timeout = REPROC_INFINITE;
  opts.deadline = ms_to_wait;

  reproc_sink sink = reproc_sink_string (output);
  int r =
    reproc_run_ex (
      args, opts,
      get_stdout ? sink : REPROC_SINK_NULL,
      get_stdout ? REPROC_SINK_NULL : sink);

  g_message ("output: %s", *output);

  if (r < 0)
    {
      if (warn_if_fail)
        {
          g_warning ("%s", reproc_strerror (r));
        }
      else
        {
          g_message ("%s", reproc_strerror (r));
        }
    }

  return (r < 0) ? r : 0;
}

any idea what this means?

I have compiled the reproc library on windows using MSYS2 using the command:

cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr -DREPROC++=OFF -DBUILD_SHARED_LIBS=OFF ..

so I have disabled the build of the shared library to have only the static library.

Now the pkg-config --libs command on reproc will return "-lreproc" and not the ws2_32 library required on Windows. This is because the ws2_32 library is declared as private in the ".pc" file:

prefix=@CMAKE_INSTALL_PREFIX@
exec_prefix=${prefix}
includedir=${prefix}/@CMAKE_INSTALL_INCLUDEDIR@
libdir=${exec_prefix}/@CMAKE_INSTALL_LIBDIR@

Name: @TARGET@
Description: @PROJECT_DESCRIPTION@
URL: @PROJECT_HOMEPAGE_URL@
Version: @PROJECT_VERSION@
Cflags: -I${includedir}
Libs: -L${libdir} -l@TARGET@
Libs.private: @REPROC_THREAD_LIBRARY@ @REPROC_WINSOCK_LIBRARY@ @REPROC_RT_LIBRARY@

The problem is that I have installed only the static library and the ws2_32 is needed.

The pkg-config documentation states:

Libs.private: The link flags for private libraries required by this package but not exposed to applications. The same rule as Cflags applies here.

In my understanding, when the dynamic library is disabled and the static library is going to be used all the libraries required should be listed in "Libs:" in the .pc file because "they are exposed to applications".

This is likely related to an old terminal issue, reported on GH here:

• microsoft/terminal#346
• microsoft/terminal#335

My setup is Windows 10, Visual Studio 2019 (16.1.5) and problem can be observed when running background.cpp reproc++ sample under VS. The observable results is that console window cannot be closed (Visual Studio Debugger Console cannot be closed if there are any child processes left, and the zombie process can't be stopped, can only be forcefully killed).

For me this doesn't repro in cmd.exe but does reproduce in Visual Studio Debugger Console as well as cmd replacements like @malxau's yori. I suspect that this is caused by a race between process->running set to false (i.e. child process terminating by itself) and platform specific process_terminate() being called (which in turn calls GenerateConsoleCtrlEvent()). In regular terminal I get lucky, in debugger console and yori I don't.

I tried a simple workaround by calling FreeConsole()/AttachConsole(child)/SetConsoleCtrlHandler(NULL, TRUE) before the GenerateConsoleCtrlEvent() call but that makes it impossible for the parent process to reattach to its console later on since the first FreeConsole() drops console's refcount to 0 an causes its release.

A dummy process would have to be spawned to hold to the parent console while it generates CTRL_BREAK_EVENT on the child but I haven't tried that yet and I'm not 100% convinced this is a correct solution to this problem.

Hi, is there a example of integrating reproc with event lib like libevent? It looks like the example included is by launching a new thread -- which is fine, but I have a program that already using libevent and want to see what's the best model

11.0.1 Release Version PACKAGE_VERSION is 11.0.0

project(
  reproc
  VERSION 11.0.0 
  DESCRIPTION "Cross-platform C99/C++11 process library"
  HOMEPAGE_URL "https://github.com/DaanDeMeyer/reproc"
  LANGUAGES C
)

project(
  reproc
  VERSION 11.0.1 
  DESCRIPTION "Cross-platform C99/C++11 process library"
  HOMEPAGE_URL "https://github.com/DaanDeMeyer/reproc"
  LANGUAGES C
)

I tried the example from README, but it turns out that there is an unclosed brace in the installed header:

$ cc -o proc proc.cpp `pkg-config --cflags --libs reproc++`
proc.cpp:8:2: error: expected '}'
}
 ^
/usr/local/include/reproc/drain.h:6:12: note: to match this '{'
extern "C" {
           ^
1 error generated.

reproc-14.1.0
FreeBSD 12.2
clang-10

/reproc.dir/src/windows/pipe.c.obj   -c _deps/reproc-src/reproc/src/windows/pipe.c
_deps/reproc-src/reproc/src/windows/pipe.c:178:14: error: conflicting types for 'pipe_wait'
REPROC_ERROR pipe_wait(HANDLE *ready, HANDLE out, HANDLE err)
             ^
_deps/reproc-src/reproc/src\pipe.h:51:1: note: previous declaration is here
pipe_wait(const reproc_handle *pipes,

Is it possible to have a dev branch or a second branch ?

Because i'm between a release that i'm waiting and there is only master branch

So basically it's break my CI when there is a breaking change :/

(ROOT)\reproc\src\cpp\error.cpp:60:28: error: 'too_many_symbolic_link_levels' is not a member of 'std::errc'
       return error_code == std::errc::too_many_symbolic_link_levels;

the mingw version is:

Using built-in specs.
COLLECT_GCC=C:\MinGW\bin\g++.exe
COLLECT_LTO_WRAPPER=C:/MinGW/bin/../libexec/gcc/mingw32/5.1.0/lto-wrapper.exe
Target: mingw32
Configured with: ../../../src/gcc-5.1.0/configure --build=mingw32 --enable-languages=ada,c,c++,fortran,lto,objc,obj-c++ --enable-libgomp --enable-lto --enable-graphite --enable-libstdcxx-debug --enable-threads=posix --enable-version-specific-runtime-libs --enable-fully-dynamic-string --enable-libstdcxx-threads --enable-libstdcxx-time --with-gnu-ld --disable-werror --disable-nls --disable-win32-registry --disable-symvers --enable-cxx-flags='-fno-function-sections -fno-data-sections -DWINPTHREAD_STATIC' --prefix=/mingw32tdm --with-local-prefix=/mingw32tdm --with-pkgversion=tdm-1 --enable-sjlj-exceptions --with-bugurl=http://tdm-gcc.tdragon.net/bugs
Thread model: posix
gcc version 5.1.0 (tdm-1)

Thank you for reproc project.

I wonder if in some time will be a release? With a release version will be easier for the Linux distributions to make a package.

If I set this option, I can make a child output to a file path instead of the console:
options.redirect.out.path = "c:/foo/bar.txt"

How would I make the child process output to the same stdout as the parent (so the output appears in the console of the parent process)? Is that possible?