Compilation in parallel on macOS does not work about pycutest HOT 5 OPEN

Thanks Tom,

Glad you managed to get it working with multiprocessing (my python parallel framework of choice). The joblib error you're seeing is similar to a recent upstream bug report:
joblib/joblib#1529 (comment)
The joblib developers are claiming it's nothing to do with them but that is unlikely, it looks to me like a relatively rare genuine upstream bug (the fact that you and someone else on a different platform both experience a very similar issue is too much of a coincidence).

from pycutest.

Hmm this looks suspiciously like an upstream issue to me:

Fatal Python error: init_sys_streams: can't initialize sys standard streams
Python runtime state: core initialized
OSError: [Errno 9] Bad file descriptor

Could you also try parallel compilation on macOS with another Python extension?

EDIT: also probably worth trying with a newer Python version (11/12).

from pycutest.

Hi Jari @jfowkes,

I made some tests, and the same problem occurred for Python3.11 and Python3.12. However, it seems that the following code works:

import logging
import multiprocessing

import pycutest


def get_logger(name):
    logger = logging.getLogger(__name__)
    if len(logger.handlers) == 0:
        logger.setLevel(logging.INFO)

        # Attach a console handler (thread-safe).
        handler = logging.StreamHandler()
        handler.setFormatter(logging.Formatter('[%(levelname)-8s] %(message)s'))
        logger.addHandler(handler)
    return logger


def load_problem(problem_name):
    logger = get_logger(__name__)
    logger.info(f'Loading {problem_name}')
    try:
        pycutest.import_problem(problem_name)
        logger.info(f'{problem_name} successfully loaded')
    except Exception as exc:
        logger.warning(f'{problem_name} failed to load: {exc}')


if __name__ == '__main__':
    problem_names = pycutest.find_problems(constraints='unconstrained', n=[1, 10])
    with multiprocessing.Pool() as p:
        p.map(load_problem, problem_names[:5])

I tried to change the name of all the Python files in PyCUTEst (I was particularly worried that system_paths might conflict with some other module), but this did not do the trick. I am not sure what is happening.

Cheers,
Tom.

from pycutest.

Hey @jfowkes @ragonneau,
Any news/updates the topic? We are having a similar issue on our library!
Thank you!

from pycutest.

Hi @LacombeLouis, this is an upstream issue as far as we are concerned. As a workaround I would recommend switching over to multiprocessing.

from pycutest.