pybind / pybind11_mkdoc Goto Github PK

View Code? Open in Web Editor NEW

37.0 37.0 20.0 55 KB

Pybind11 tool for making docstrings from C++ comments

License: MIT License

Python 92.15% C++ 6.77% C 1.08%

pybind11_mkdoc's People

Contributors

Stargazers

Watchers

Forkers

mgeplf noahamsel luxonis global-localhost global19 global19-atlassian-net tonywelte cluttrdev zcy543814 corna python-repository-hub tammojan tttapa jeguzzi njroussel aloisklink zeroxer dagnic westlicht

pybind11_mkdoc's Issues

Searching default clang library file fails if expected directory structure does not match environment

When not providing a LIBCLANG_PATH environment variable the libclang shared object library file is search for in
/usr/lib/llvm-*/lib/ directories containing libclang.so.1. On systems where this directory structure does not match the clang installation the implemented search fails with an IndexError since it expects to find at least one such path.

Use CMake

I have a library that depends on a number of different libraries. I use CMake to find them, and I would like pybind11_mkdoc to do the same. How do I do that?

Setting clang include directory fails on non-matching environments

Setting the clang include directory as it is implemented right now leads to AttributeError: 'NoneType' object has no attribute 'encode' erros from within clang/cindex.py on systems where the expected directories do not exist.

pybind11_mkdoc/pybind11_mkdoc/mkdoc_lib.py

Lines 309 to 313 in 956b3f9

    
           clang_include_dir = max( 
        
               glob(os.path.join(llvm_dir, 'lib', 'clang', '*') 
        
           ), default=None, key=folder_version) 
        
           parameters.extend(['-isystem', clang_include_dir])

If there are no directories matching the search pattern a default value of None is used which then leads to passing -isystem None as an argument to clang that causes the above error.

Documentation not explicit on dependency on "libclang-dev", not having it results in "'NoneType' object has no attribute 'encode'"

Documentation not explicit on dependency on "libcmake-dev", not having it results in "'NoneType' object has no attribute 'encode'".

Tested on master, on this file: https://github.com/EthicalML/vulkan-kompute/blob/master/single_include/kompute/Kompute.hpp

I also tried on a simpler file (https://github.com/EthicalML/vulkan-kompute/blob/master/src/include/kompute/Manager.hpp) but still get the same error:

Processing "Manager.hpp" ..
Waiting for jobs to finish ..
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/alejandro/miniconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/alejandro/miniconda3/lib/python3.7/site-packages/pybind11_mkdoc/mkdoc_lib.py", line 232, in run
    tu = index.parse(self.filename, self.parameters)
  File "/home/alejandro/miniconda3/lib/python3.7/site-packages/clang/cindex.py", line 2722, in parse
    self)
  File "/home/alejandro/miniconda3/lib/python3.7/site-packages/clang/cindex.py", line 2816, in from_source
    args_array = (c_char_p * len(args))(*[b(x) for x in args])
  File "/home/alejandro/miniconda3/lib/python3.7/site-packages/clang/cindex.py", line 2816, in <listcomp>
    args_array = (c_char_p * len(args))(*[b(x) for x in args])
  File "/home/alejandro/miniconda3/lib/python3.7/site-packages/clang/cindex.py", line 109, in b
    return x.encode('utf8')
AttributeError: 'NoneType' object has no attribute 'encode'

Is this due to my environment? I'm testing in Ubuntu 18, Python3. Happy to provide further details.

clang not correctly detected

In a new conda environment, I've installed

conda install -c conda-forge python
python -m pip install .

which outputs

Processing /Users/tdegeus/Downloads/pybind11_mkdoc
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting clang (from pybind11_mkdoc==2.6.2.dev1)
  Using cached clang-16.0.1.1-py3-none-any.whl (31 kB)
Building wheels for collected packages: pybind11_mkdoc
  Building wheel for pybind11_mkdoc (pyproject.toml) ... done
  Created wheel for pybind11_mkdoc: filename=pybind11_mkdoc-2.6.2.dev1-py3-none-any.whl size=9378 sha256=b8feb15bae6058b1dfe3085aef3d28bfb96a33aa4df57c21192383ac86085198
  Stored in directory: /private/var/folders/4z/74x3z8gs7pq77drzss82p5mc0000gq/T/pip-ephem-wheel-cache-89cdsw8p/wheels/57/8e/75/8ef6e06a545a06d229053aff7b0d90101be8b484c73921802b
Successfully built pybind11_mkdoc
Installing collected packages: clang, pybind11_mkdoc
Successfully installed clang-16.0.1.1 pybind11_mkdoc-2.6.2.dev1

Yet I'm getting

python -m pybind11_mkdoc -o python/docstrings.h include/GooseEYE/GooseEYE.h
Processing "include/GooseEYE/GooseEYE.h" ..
Waiting for jobs to finish ..
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/site-packages/clang/cindex.py", line 4136, in register_function
    func = getattr(lib, item[0])
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/ctypes/__init__.py", line 392, in __getattr__
    func = self.__getitem__(name)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/ctypes/__init__.py", line 397, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: dlsym(0x846d0020, clang_CXXMethod_isCopyAssignmentOperator): symbol not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/threading.py", line 1052, in _bootstrap_inner
    self.run()
  File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/site-packages/pybind11_mkdoc/mkdoc_lib.py", line 248, in run
    cindex.conf.lib.clang_createIndex(False, True))
    ^^^^^^^^^^^^^^^
  File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/site-packages/clang/cindex.py", line 212, in __get__
    value = self.wrapped(instance)
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/site-packages/clang/cindex.py", line 4217, in lib
    register_functions(lib, not Config.compatibility_check)
  File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/site-packages/clang/cindex.py", line 4164, in register_functions
    register(f)
  File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/site-packages/clang/cindex.py", line 4161, in register
    return register_function(lib, item, ignore_errors)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/site-packages/clang/cindex.py", line 4142, in register_function
    raise LibclangError(msg)
clang.cindex.LibclangError: dlsym(0x846d0020, clang_CXXMethod_isCopyAssignmentOperator): symbol not found. Please ensure that your python bindings are compatible with your libclang.so version.

Further information:

$ mamba list
# packages in environment at /Users/tdegeus/miniforge3/envs/doc:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h93a5062_5    conda-forge
ca-certificates           2023.11.17           hf0a4a13_0    conda-forge
clang                     16.0.1.1                 pypi_0    pypi
libexpat                  2.5.0                hb7217d7_1    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libsqlite                 3.44.2               h091b4b1_0    conda-forge
libzlib                   1.2.13               h53f4e23_5    conda-forge
ncurses                   6.4                  h463b476_2    conda-forge
openssl                   3.2.0                h0d3ecfb_1    conda-forge
pip                       23.3.1             pyhd8ed1ab_0    conda-forge
pybind11-mkdoc            2.6.2.dev1               pypi_0    pypi
python                    3.12.0          h47c9636_0_cpython    conda-forge
readline                  8.2                  h92ec313_1    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
tk                        8.6.13               h5083fa2_1    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h57fd34a_0    conda-forge

$ mamba info

          mamba version : 1.5.3
     active environment : doc
    active env location : /Users/tdegeus/miniforge3/envs/doc
            shell level : 2
       user config file : /Users/tdegeus/.condarc
 populated config files : /Users/tdegeus/miniforge3/.condarc
                          /Users/tdegeus/.condarc
          conda version : 23.10.0
    conda-build version : not installed
         python version : 3.11.6.final.0
       virtual packages : __archspec=1=m1
                          __osx=14.0=0
                          __unix=0=0
       base environment : /Users/tdegeus/miniforge3  (writable)
      conda av data dir : /Users/tdegeus/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/osx-arm64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-arm64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/tdegeus/miniforge3/pkgs
                          /Users/tdegeus/.conda/pkgs
       envs directories : /Users/tdegeus/miniforge3/envs
                          /Users/tdegeus/.conda/envs
               platform : osx-arm64
             user-agent : conda/23.10.0 requests/2.31.0 CPython/3.11.6 Darwin/23.0.0 OSX/14.0 solver/libmamba conda-libmamba-solver/23.11.1 libmambapy/1.5.3
                UID:GID : 503:20
             netrc file : None
           offline mode : False

The problem is persistent when I install clang from conda-forge

In addition, the problem is persistent in my CI: tdegeus/GooseEYE#105

I guess that that the issue could be that I'm not installing directly using

python -m pip install git+git://github.com/pybind/pybind11_mkdoc.git@master

However, that command times-out (also on the CI)

Add support for a docstring format that Sphinx understands

Currently a function's or method's arguments are output with the following format:

Parameter ``arg1``:
    This is the description.
    This is a continued line of the description.

Parameter ``arg2``:
    This is the description.

Throws:
    Exception1 This is when the exception is thrown.

Throws:
    Exception2 This is when the exception is thrown.

Sphinx then renders this as (depending on the theme)

Parameter arg1:
This is the description.
This is a continued line of the description.

Parameter arg2:
This is the description.

Throws:
Exception1 This is when the exception is thrown.

Throws:
Exception2 This is when the exception is thrown.

The docstring format that Sphinx understands that this closest to is the Google Style docstring (https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html#google-vs-numpy). This would look like the following:

Parameters:
    arg1: This is the description.
        This is a continued line of the description.
    arg2: This is the description.

Raises:
    Exception1: This is when the exception is thrown.
    Exception2: This is when the exception is thrown.

Which would render as (depending on the theme)

Parameters:

arg1 -- This is the description. This is a continued line of the description.
arg2 -- This is the description.

Raises:

Exception1 -- This is when the exception is thrown.
Exception2 -- This is when the exception is thrown.

Please could pybind11_mkdoc output docstrings in a format that Sphinx understands so that they render like any other Python docstring would.

The quickest way to implement this would to aim to have pybind11_mkdoc output Google style docstrings. Given how close the current output format is to the Google docstring style, this boils down to supporting grouped sections (eg Parameters and Raises).

However we could chose to output as rst and still get the desired output with Sphinx. Obviously this would be much more work. For the most flexibility, the user could choose the output format that they desire. Personally I don't need this extra flexibility because I use Google style anyway.

writing output to file fails if corresponding dir path does not exist

If the output file name contains a non-existing path, trying to open it for writing fails, i.e.

$ python -m pybind11_mkdoc -o <non/existing/path/out_file.hpp> input_headers...

results in a FileNotFoundError.

This can be fixed by adding a call to os.makedirs(os.path.dirname(output), exist_ok=True) (requires Python 3.2+).

fatal error: 'stdlib.h' file not found

Hi,

When trying to use the pybind11_mkdoc tool on the example in the README,it seems to work fine. However, if I add a standard include:

#include <string>

I get the error:

/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/cstdlib:75:15: fatal error: 'stdlib.h' file not found

Command: python -m pybind11_mkdoc example.hpp -o test.hpp appending -isystem /usr/include does not help.

I have this file in my /usr/include directory and the pybind11_mkdoc seems to be adding it to the include directories.

I have no other issues with clang / llvm. Any ideas?

System details:

Ubuntu 18.04.4 LTS

LLVM 9.0.0

clang version 9.0.0-2~ubuntu18.04.2 (tags/RELEASE_900/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

python-clang 11.0

Thanks, Matt.

Doxygen @ tags not supported

Doxygen defines tags like \return and \param that have special meanings in documentation. pybind11_mkdoc understands these tags and "translates" them appropriately by reformatting the comment. According to the Doxygen manual,

All commands in the documentation start with a backslash (\) or an at-sign (@). If you prefer you can replace all commands starting with a backslash below by their counterparts that start with an at-sign.

But pybind11_mkdoc does not understand @return and @param, and thus does not translate them at all.

clang.cindex.TranslationUnitLoadError: Error parsing translation unit

Hi, I'm trying to run the pybind11_mkdoc in a single file called score.h, but I got this error below:

C:\Users\nyck\Desktop\maialib>python -m pybind11_mkdoc maiacore\include\maiacore\score.h
Processing "'maiacore\include\maiacore\score.h'" ..
Waiting for jobs to finish ..
error: error reading ''maiacore\include\maiacore\score.h''
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "C:\Users\nyck\AppData\Roaming\Python\Python310\site-packages\pybind11_mkdoc\mkdoc_lib.py", line 246, in run
    tu = index.parse(self.filename, self.parameters)
  File "C:\Users\nyck\AppData\Roaming\Python\Python310\site-packages\clang\cindex.py", line 2722, in parse
    return TranslationUnit.from_source(path, args, unsaved_files, options,
  File "C:\Users\nyck\AppData\Roaming\Python\Python310\site-packages\clang\cindex.py", line 2837, in from_source
    raise TranslationUnitLoadError("Error parsing translation unit.")
clang.cindex.TranslationUnitLoadError: Error parsing translation unit.

The file score.h is a valid C++ header that contains a valid Doxygen documentation in it.

Please, could you help me?

My system:

Windows 10 x64
Clang LLVM 15.0.6 installed
Python 3.10
pybind11_mkdoc-2.6.2.dev1 (filename=pybind11_mkdoc-2.6.2.dev1-py3-none-any.whl)

error: ISO C++11 requires at least one argument for the "..." in a variadic macro

When compiling bindings output with g++ --std=c++17 -Wall -Werror -pedantic-errors, generated header is not conformant:

project/bindings.cpp:168:55: error: ISO C++11 requires at least one argument for the "..." in a variadic macro
  168 |     py::enum_<PreStatus>(m, "PreStatus", DOC(PreStatus))

Can #include'd files be missing?

My C++ code depends on an external library, so I add the flag -I${nlohmann_SOURCE_DIR}/include to ensure that #include can find the headers. Now #include doesn't really need to work for pybind11_mkdoc to do its job, and indeed, if I do not add the flag -I${EXTERNAL_SOURCE_DIR}/include, pybind11_mkdoc still runs successfully. However, I get the following error message

fatal error: 'nlohmann/json_fwd.hpp' file not found

at the line where this file is #included. Can we eliminate this error message and guarantee to the user that pybind11_mkdoc works even if included files can't be found, or should we require that all #includes succeed?

	clang_include_dir = max(
	glob(os.path.join(llvm_dir, 'lib', 'clang', '*')
	), default=None, key=folder_version)

	parameters.extend(['-isystem', clang_include_dir])