gbiggs / pykg-config Goto Github PK
View Code? Open in Web Editor NEWPython drop-in replacement for pkg-config
License: BSD 3-Clause "New" or "Revised" License
Python drop-in replacement for pkg-config
License: BSD 3-Clause "New" or "Revised" License
pykg-config ============================================================================== A pkg-config replacement. pykg-config is an input- and output-compatible implementation of pkg-config written in Python for greater ease of portability. It is designed to be a drop-in replacement: command lines that work for pkg-config should produce identical output from pykg-config. Requirements ------------ pykg-config uses the new string formatting operations that were introduced in Python 2.6. It will not function with an earlier version of Python. It has been been tested with Python 2.6, 2.7, 3.1, 3.2, 3.3, and 3.4. Installation ------------ There are several methods of installation available: 1. Download the source (either from the repository or a source archive), extract it somewhere, and run pykg-config from that directory. 2. Download the source (either from the repository or a source archive), extract it somewhere, and use distutils to install it into your Python distribution: a. Extract the source, e.g. to a directory C:\pykg-config\ b. Run setup.py to install pykg-config to your default Python installation: C:\pykg-config\> python setup.py install c. If necessary, set environment variables. These should be set by default, but if not you will need to set them yourself. On Windows, you will need to ensure that your Python site-packages directory is in the PYTHONPATH variable and the Python scripts directory is in the PATH variable. Typically, these will be something like C:\Python26\Lib\site-packages\ and C:\Python26\Scripts\, respectively (assuming Python 2.6 installed in C:\Python26\). 3. Use the Windows installer. This will perform the same job as running setup.py (see #2), but saves opening a command prompt. You may still need to add paths to your environment variables Under Unix-like operating systems, if you do not already have the original pkg-config available, you should create a symbolic link to pykg-config. This allows build system scripts that look for pkg-config to find pykg-config without modification. Place this link somewhere in your path. For example, assuming pykg-config was installed to /usr/local: $ ln -s /usr/local/bin/pykg-config.py /usr/local/bin/pkg-config On Windows, a batch file, pkg-config.bat, is installed along with pykg-config into the Scripts directory. This should function as a drop-in replacement for pkg-config in build system scripts, such as CMake's UsePkgConfig.cmake module, provided that this directory is in your PATH environment variable. Package paths ------------- Paths are searched in this order: 1. All paths listed in the PKG_CONFIG_PATH environment variable. 2. All paths listed in the PKG_CONFIG_LIBDIR environment variable, if set. 3. (Windows only) The registry keys HKEY_CURRENT_USER\Software\pkg-config\PKG_CONFIG_PATH and HKEY_LOCAL_MACHINE\Software\pkg-config\PKG_CONFIG_PATH. For both of these, paths should be set as values of the key. The value name has no meaning to pkg-config; use it for your own reference. The value type must be REG_SZ (a string), and the data should be a single path. 4. All paths listed in the --with-pc-path option when setup.py is executed, if set. Otherwise, all paths in ${prefix}/lib[64]/pkgconfig/ and ${prefix}/share/pkgconfig/, where ${prefix} is a system prefix (typically this will be /usr/). If you are using Windows, I recommend you add paths to PKG_CONFIG_PATH. This is the easiest place to add paths to and the easiest to check for errors. Google can tell you how to add an environment variable in Windows. Unfortunately, because Windows does not have a centralised directory structure, you will probably have to add every package you install to this variable. If you are lucky, some nice packages will do it when they are installed, but I haven't yet seen one that does this. Hard-coded package path ----------------------- It is possible, when installing using setup.py, to specify a hard-coded list of paths to be searched for .pc files. Use the following setup.py command to do so: python setup.py build_py --with-pc-path=<desired paths here> install The list of paths should be specified as a single string, with the paths separated by a semi-colon (';') on Windows or a colon (':') on other platforms. pkg-config (.pc) file things-to-watch-out-for --------------------------------------------- The pkg-config format does not deal with spaces in values very well. If you have an include or lib path with a space in it (common on Windows), pkg-config will cheerfully treat up to the first space as an include or lib path, and then ignore all remaining words. This is despite correctly parsing the file with escaped spaces (the final value processing step is where it drops the rest). By contrast, pykg-config uses Python's shlex module to split values, preserving things like escaped spaces. This is an advantage on Windows (provided your .pc files properly escape their spaces), but does mean output is incompatible with pkg-config. In the interest of user-friendliness, on Windows, full compatibility is _disabled_ by default (i.e. paths with escaped spaces are handled correctly). On other platforms, full compatibility is _enabled_ by default. You can manually turn it on or off using the --full-compatibility and --less-compatibility switches. The standard Windows path format (using \) does not play well with some build systems, such as CMake. Fortunately, CMake correctly handles paths specified using Unix-style separators (/), so if your .pc files specify their paths using that format you shouldn't have any problems. Miscellaneous notes ------------------- 1. Default target compiler The CMake pkg-config module does not handle Microsoft Visual C++ style libdir specifications (/libpath:). For this reason, even on Windows, the output defaults to the standard non-MSVC-compatible format. You can change which style is used with the --msvc-syntax and --no-msvc-syntax options. 2. Testing compatibility The test script, test_compatibility.py, performs a set of unit tests in an attempt to maintain compatibility with the version of pkg-config mentioned in CORRESPONDING_VERSION in pykg-config.py. Executing it will run through all the tests. If they all pass, pykg-config.py is producing the same output as the pkg-config installed on your system. 3. Search prefix and system directories pkg-config searchers in directories below a prefix that is defined at compile time. Typically, this prefix will be /usr/lib/ or /usr/lib64/, depending on the toolchain used to compile pkg-config. This path is then hard-coded into the binary and cannot be changed at run-time. The only way to change it is by recompiling pkg-config and forcing a different value for the system libpath onto it. This presents a problem for pykg-config. Checking the Python system prefix (via sys.prefix) is not a guaranteed solution, as it is often different from the libpath used by the compiler that builds pkg-config. While not much of a problem for pykg-config's main target (Windows), in order to ensure compatiblity it is best to try and meet this problem. A nasty hack is used to try and fix this in the majority of cases. At startup, pykg-config looks at the value of sys.path. It finds the first entry containing 'pythonxy.zip' (where x and y are the major and minor version numbers). If the end of the containing directory name is '64', it assumes that the system is running 64-bit libraries as the primary toolchain. This will cause paths such as /usr/lib/ (which are created from the sys.prefix value) to turn into /usr/lib64/. You can test if the result is accurate for your system by ensuring that the 'test_print_errors_with_error' test will have an error caused by a package in /usr/lib64 (see the comments for that test), then running test_compatibility.py. If the error messages are the same (i.e. the test passes), then the hack has worked. 5. Why? Because building pkg-config for Windows in an easy-to-distribute, easy-to-use way is a pain. The core functionality of pkg-config is simple and easy to provide using Python (which provides several modules useful for such things as parsing text files, which is what .pc files are). It was also fun.
Using pykg-config-1.3.0...
/usr/bin/python setup.py build_py --with-pc-path=/sw/lib/pkgconfig:/sw/share/pkgconfig:/opt/X11/lib/pkgconfig:/opt/X11/share/pkgconfig:/usr/X11/lib/pkgconfig:/usr/X11/share/pkgconfig:/usr/lib/pkgconfig
running build_py
error: error in 'with_pc_path' option: '/sw/lib/pkgconfig:/sw/share/pkgconfig:/opt/X11/lib/pkgconfig:/opt/X11/share/pkgconfig:/usr/X11/lib/pkgconfig:/usr/X11/share/pkgconfig:/usr/lib/pkgconfig' does not exist or is not a directory
setup.py appears to call ensure_dirname on the whole option, even though the option is documented and implemented to take a colon-delimited list of directories. This sort of test would need to parse the string and iterate through the list not take the string as it was given.
However, I think having a directory not exist should at most be a warning, not an error. I might want to configure for paths that none of my already-installed packages have populated, so their dirs would not exist (pykg-config is pretty low in the dep tree, so it could easily wind up being built on a very sparse system). Also, for portability I want a unified build-recipe I can use on a variety of target platforms (or cross-compile), so I can't have my local build machine reject the platform-specific paths needed by the individual targets.
See this bug for pkg-config:
https://bugs.freedesktop.org/show_bug.cgi?id=16101
pykg-config has the same problem although it removes the duplicates in a bit different way (pkg-config leaves the first occurrence of duplicated flag while pykg-config leaves the last).
When .pc file contains strings with escaped quotes, e.g.
Cflags: -DCG_PROGRAMS_PATH="/opt/shaders/"
behaviour of pkg-config and pykg-config.py is different.
pkg-config will preserve the escaping and output string unmodified:
-DCG_PROGRAMS_PATH="/opt/shaders/"
pykg-config.py will remove escaping:
-DCG_PROGRAMS_PATH="/opt/shaders/"
Note that when passing strings via -D option to gcc the quotes must be escaped.
The use of "all paths in ${prefix}/lib[64]/pkgconfig/ and ${prefix}/share/pkgconfig/, where ${prefix} is a system prefix (typically this will be /usr/)." as the default search paths is different from pkg-config. As the README notes, pkg-config hardcodes a search-path list at its compile-time, which means the vendor or sysadmin can define any arbitrary list and pkg-config will always use it by default. But pykg-config does not appear to have a way to permanently alter the default. I can use env vars to override it at runtime, but that means all my users have to be doing it (and risking breaking it by strict env-cleanup via sudo, or other site policies). It would be useful if setup.py took a --with-pc-path flag (like pkg-config's ./configure) to force hardcoding a specific default.
I get an error with importing '_winreg' with python 3 on windows. As it says here, this module has been renamed to 'winreg' in python 3.
With pkg-config it's possible to use in .pc file variables which are undefined in that .pc file as long as all the variables are defined using --define-variable option.
With pykg-config.py it doesn't work - one must define each variable in .pc file even if the value will be changed by command line options.
Can you upload a source distribution to PyPI? This is needed to make pykg-config easy_install
able.
It would be really handy if pykg-config was easy_install
able so that other packages could use it as a setup_requires
dependency. For Python packages that depend on external libraries which can be located with pkg-config
, it would be useful to be able to fall back to using pykg-config
if pkg-config
is missing. See, for example, healpy/healpy#88.
As an illustration, the following trivial setup.py
script:
from setuptools import setup
setup(
name="pykg_config_test",
setup_requires="pykg-config"
)
produces the following error message:
$ python setup.py build
No local packages or download links found for pykg-config
Traceback (most recent call last):
File "setup.py", line 5, in <module>
setup_requires="pykg-config"
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 112, in setup
_setup_distribution = dist = klass(attrs)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/dist.py", line 221, in __init__
self.fetch_build_eggs(attrs.pop('setup_requires'))
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/dist.py", line 245, in fetch_build_eggs
parse_requirements(requires), installer=self.fetch_build_egg
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pkg_resources.py", line 586, in resolve
dist = best[req.key] = env.best_match(req, self, installer)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pkg_resources.py", line 831, in best_match
return self.obtain(req, installer) # try and download/install
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pkg_resources.py", line 843, in obtain
return installer(requirement)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/dist.py", line 295, in fetch_build_egg
return cmd.easy_install(req)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 592, in easy_install
raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pykg-config')
I think that all one has to do is run python setup.py sdist upload
as described at http://wiki.python.org/moin/CheeseShopTutorial#Package_Distribution_Files.
All of pkgsearcher.init_search_dirs() uses isdir() at runtime to screen the entries in PKG_CONFIG* env-vars and for the automatically determined other places to search. But pc_path entries (alternative compile-time-defined search paths) are fed to listdir() without testing. Presumably it's trusting that the directories existed at compile-time, so they still do, and crashes if they don't (this is the converse of Issue #12). No need for that poor runtime experience (and that's not what pkg-config does either).
pkg-config allows the path to a package file to be specified on the command line, avoiding a search. pykg-config treats the file path as a package name and searches for it.
I have a system with the following files:
protobuf.pc
prefix=/usr
exec_prefix=/usr
libdir=/usr/lib
includedir=/usr/include
Name: Protocol Buffers
Description: Google's Data Interchange Format
Version: 3.15.2
Libs: -L${libdir} -lprotobuf -lpthread
Libs.private: -lz
Cflags: -I${includedir} -pthread
Conflicts: protobuf-lite
protobuf-lite.pc
prefix=/usr
exec_prefix=/usr
libdir=/usr/lib
includedir=/usr/include
Name: Protocol Buffers
Description: Google's Data Interchange Format
Version: 3.15.2
Libs: -L${libdir} -lprotobuf-lite -lpthread
Cflags: -I${includedir} -pthread
Conflicts: protobuf
Running
python pykg-config.py --cflags --libs protobuf
fails because PackageNotFoundError
is not defined at
pykg-config/pykg_config/result.py
Line 253 in 73d80a7
pkg.name
is not defined at pykg-config/pykg_config/result.py
Line 258 in 73d80a7
If I fix those issues, it then always raises a PackageConflictError
because protobuf and protobuf-lite are mutually exclusive. I'm only trying to use one of them though, so it seems like that shouldn't have a conflict.
I would like to contribute a fix for this, but I'm not sure what the expected behavior here is. Should this conflict checking code maybe just be removed? When I run pkg-config
on a PC running Ubuntu 20.04, it doesn't appear to care about conflicts.
$ pkg-config --cflags --libs protobuf
-pthread -lprotobuf -pthread
$ pkg-config --cflags --libs protobuf protobuf-lite
-pthread -lprotobuf -pthread -lprotobuf-lite -pthread
pykg-config doesn't obey PKG_CONFIG_SYSROOT_DIR. It looks like the value is being stored in global_variables['pc_sysrootdir']
in pykg-config.py, but read from Options().get_option('pc_sysrootdir')
in package.py. I did a quick hack to fix this, and it also has the problem that using os.path.join()
has a quirk that if it thinks a path component is absolute, it will ignore the previous components so that:
os.path.join('C:\Foo\Bar', '/usr/include') => '/usr/include'
I think this can be fixed by changing
include_dir = join(pc_sysrootdir, flag[2:].strip())
to
include_dir = join(pc_sysrootdir,flag[2:].strip().lstrip('/'))
(and the same for libpath) in package.py
This patch fixes recursion handling.
At the moment if package A requires B and C, and D requires A and E then pykg-config.py will not provide any flags defined for E when requesting flags for D. The reason is that when processing dependencies for D it notices that A was already loaded and stops processing the rest of dependencies defined for D. Instead it should just skip A and continue processing the rest of the dependencies.
--- gbiggs-pykg-config-1189f96/pykg_config/result.py.old 2012-03-27 17:31:31.229471333 +0200
+++ gbiggs-pykg-config-1189f96/pykg_config/result.py 2012-03-27 17:28:42.427769635 +0200
@@ -241,7 +241,7 @@
for dep in dependencies:
#0. Check if this package is already loaded
if self._get_loaded_package(dep.name):
return
continue
#1. Search for a package matching the spec
pkg = self.searcher.search_for_package(dep, self.globals)
#2. Check for conflicts
From man page of pkg-config:
Note that variable references are written "${foo}"; you can escape literal "${" as "$${".
When pkg-config processes .pc file containing literal "$${" it will strip the first "$" and will output it as "${".
pykg-config.py doesn't strip the first "$".
This means that output for .pc files containing literals "$${" is different.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.