Feature: Non-zero exit code when dedecting incompatibilities

Is your feature request related to a problem? Please describe

I would like to use licensecheck in a GitHub workflow/action and that action should fail if incompatible licenses are detected.

Describe the solution you'd like

a) Default behavior should be to fail if incompatibility is detected (exit code > 0).

b) Provide a command line flag like --allow-incompatibility to succeed (exit code = 0), even when incompatibilities are detected.

Additional context

Example GitHub workflow:

https://github.com/fraunhofer-isi/micat/blob/main/.github/workflows/back_end_license_check.yml

Bug: crash if setup.cfg exists with no metadata section

Bug

I have read the comment above and have completed each step
I have filled out the system info
I have described the bug, filled in the expected outcome and the actual
outcome including screenshots where appropriate

System info

OS: Darwin ***** 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:38:37 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T6000 arm64
Version: licensecheck==2023.0.1

Describe the bug

It crashes if you have a setup.cfg file that does not contain [metadata] section (for example, with only flake8 configuration).

Expected outcome

List licenses used.

Actual outcome

Traceback (most recent call last):
  File "/Users/nicolas.karolak/Library/Caches/pypoetry/virtualenvs/silvr-app-EZRmeXuX-py3.10/bin/licensecheck", line 8, in <module>
    sys.exit(cli())
  File "/Users/nicolas.karolak/Library/Caches/pypoetry/virtualenvs/silvr-app-EZRmeXuX-py3.10/lib/python3.10/site-packages/licensecheck/__init__.py", line 90, in cli
    depsWithLicenses = get_deps.getDepsWithLicenses(
  File "/Users/nicolas.karolak/Library/Caches/pypoetry/virtualenvs/silvr-app-EZRmeXuX-py3.10/lib/python3.10/site-packages/licensecheck/get_deps.py", line 112, in getDepsWithLicenses
    myLiceTxt = packageinfo.getMyPackageLicense()
  File "/Users/nicolas.karolak/Library/Caches/pypoetry/virtualenvs/silvr-app-EZRmeXuX-py3.10/lib/python3.10/site-packages/licensecheck/packageinfo.py", line 157, in getMyPackageLicense
    metaData = getMyPackageMetadata()
  File "/Users/nicolas.karolak/Library/Caches/pypoetry/virtualenvs/silvr-app-EZRmeXuX-py3.10/lib/python3.10/site-packages/licensecheck/packageinfo.py", line 136, in getMyPackageMetadata
    if "license" in config["metadata"]:
  File "/opt/homebrew/Cellar/[email protected]/3.10.10_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/configparser.py", line 965, in __getitem__
    raise KeyError(key)
KeyError: 'metadata'

Question: Supported python versions

What are the supported python versions for this package?

In the pyproject.toml file I found a reference to python 3.8 in the [tool.poetry.dependencies] section but I tried to use licensecheck in a pacakge of mine supporting python>=3.8 and it seems that there are issues at least with python 3.8 (while with python 3.9 and 3.10 everything seems to work fine)

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/licensecheck/__main__.py", line 6, in <module>
    cli()
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/licensecheck/__init__.py", line [90](https://github.com/NicolaDonelli/py4ai-core/actions/runs/3824810054/jobs/6507268900#step:4:91), in cli
    depsWithLicenses = get_deps.getDepsWithLicenses(
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/licensecheck/get_deps.py", line 116, in getDepsWithLicenses
    packages = packageinfo.getPackages(reqs)
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/licensecheck/packageinfo.py", line 109, in getPackages
    packageinfo.add(getPackageInfoLocal(requirement))
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/licensecheck/packageinfo.py", line 36, in getPackageInfoLocal
    packagePath = resources.files(requirement)
AttributeError: module 'importlib.resources' has no attribute 'files'

Bug: licensecheck command applied to requirements file return more dependencies than the ones tracked in file.

Bug

System info

-OS: Ubuntu 22.04.2 LTS
-Python: 3.10.6
-licensecheck version: 2023.3

Describe the bug

While checking a requirements file, the licensecheck command's output return more packages than the ones reported in the requirements.txt.

Here I attached a zipped version of an environment to reproduce the error:
example.zip

To reproduce the error:

unzip the file;
go to the example folder;
install licensecheck>=2023.2;
run licensecheck

Expected outcome

           Info

┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Item ┃ Value ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ program │ licensecheck │
│ version │ 2023.3.0 │
│ license │ mit │
│ project_license │ mit │
└─────────────────┴──────────────┘

                         List Of Packages

┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Compatible ┃ Package ┃ License(s) ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ✔ │ annotated-types │ MIT License │
│ ✔ │ certifi │ Mozilla Public License 2.0 (MPL 2.0) │
│ ✔ │ cfg-load │ MIT License │
│ ✔ │ charset-normalizer │ MIT License │
│ ✔ │ deprecated │ MIT License │
│ ✔ │ idna │ BSD License │
│ ✔ │ mpu │ MIT License │
│ ✔ │ numpy │ BSD License │
│ ✔ │ pandas │ BSD License │
│ ✔ │ pydantic │ MIT License │
│ ✔ │ pydantic-core │ MIT License │
│ ✔ │ python-dateutil │ Apache Software License;; BSD License │
│ ✔ │ pytz │ MIT License │
│ ✔ │ pyyaml │ MIT License │
│ ✔ │ requests │ Apache Software License │
│ ✔ │ scipy │ BSD License │
│ ✔ │ six │ MIT License │
│ ✔ │ tzdata │ Apache Software License │
│ ✔ │ tzlocal │ MIT License │
│ ✔ │ urllib3 │ MIT License │
│ ✔ │ wrapt │ BSD License │
└────────────┴────────────────────┴───────────────────────────────────────┘

This are the exactly the packages reported in the requirements.txt.

Actual outcome

                  Info

┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Item ┃ Value ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ program │ licensecheck │
│ version │ 2023.1.3 │
│ license │ MIT LICENSE │
│ project_license │ NO LICENSE/ UNKNOWN LICENSE │
└─────────────────┴─────────────────────────────┘

                                                                List Of Packages

┏━━━━━━━━━ ┃ Compatible ┃ Package ┡━━━━━━━━━ │ ✔ │ Bottleneck │ ✔ │ Brotli │ ✔ │ Deprecated │ ✔ │ Jinja2 │ ✔ │ Pillow │ ✖ │ PyQt5 │ ✔ │ PySocks │ ✔ │ PyYAML │ ✔ │ QtPy │ ✔ │ SQLAlchemy │ ✔ │ Sphinx │ ✔ │ XlsxWriter │ ✔ │ annotated-types │ ✔ │ asv │ ✔ │ backports.zoneinfo │ ✔ │ beautifulsoup4 │ ✔ │ black │ ✔ │ boto3 │ ✔ │ brotlicffi │ ✔ │ brotlipy │ ✔ │ bump2version │ ✖ │ certifi │ ✔ │ cfg-load │ ✔ │ chardet │ ✔ │ charset-normalizer │ ✔ │ check-manifest │ ✔ │ click │ ✔ │ cryptography │ ✔ │ cython-lint │ ✔ │ doit │ ✔ │ email-validator │ ✔ │ fastparquet │ ✔ │ flake8 │ ✔ │ fsspec │ ✔ │ gcsfs │ ✔ │ gmpy2 │ ✔ │ html5lib │ ✖ │ hypothesis │ ✔ │ idna │ ✔ │ jupytext │ ✔ │ lxml │ ✔ │ matplotlib │ ✔ │ mpmath │ ✔ │ mpu │ ✔ │ mypy │ ✔ │ myst-nb │ ✔ │ numba │ ✔ │ numexpr │ ✔ │ numpy │ ✔ │ numpydoc │ ✔ │ odfpy │ ✔ │ openpyxl │ ✔ │ pandas │ ✔ │ pandas-gbq │ ✔ │ pooch │ ✔ │ psycopg2 │ ✔ │ pyOpenSSL │ ✔ │ pyarrow │ ✔ │ pycodestyle │ ✔ │ pydantic │ ✔ │ pydantic-core │ ✔ │ pydata-sphinx-theme │ ✔ │ pydevtool │ ✔ │ pymysql │ ✔ │ pyreadstat │ ✔ │ pyroma │ ✔ │ pytest │ ✔ │ pytest-asyncio │ ✔ │ pytest-cov │ ✔ │ pytest-flake8 │ ✔ │ pytest-mccabe │ ✔ │ pytest-mock │ ✔ │ pytest-timeout │ ✔ │ pytest-xdist │ ✔ │ python-dateutil │ ✔ │ python-magic │ ✔ │ python-snappy │ ✔ │ pytz │ ✔ │ pyxlsb │ ✔ │ requests │ ✔ │ rich-click │ ✔ │ ruff │ ✔ │ s3fs │ ✔ │ scikit-umfpack │ ✔ │ scipy │ ✔ │ simplejson │ ✔ │ six │ ✔ │ sphinx_design │ ✔ │ tables │ ✔ │ tabulate │ ✔ │ threadpoolctl │ ✔ │ tox │ ✔ │ types-psutil │ ✔ │ typing_extensions │ ✔ │ tzdata │ ✔ │ tzlocal │ ✔ │ urllib3 │ ✔ │ │ ✔ │ wrapt │ ✔ │ xarray │ ✔ │ xlrd │ ✖ │ zest.releaser │ ✔ │ zstandard └───────── ━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ License(s) ┃
━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ BSD LICENSE │
│ MIT LICENSE │
│ MIT LICENSE │
│ BSD LICENSE │
│ HISTORICAL PERMISSION NOTICE AND DISCLAIMER (HPND) │
│ GPL V3 │
│ BSD │
│ MIT LICENSE │
│ MIT LICENSE │
│ MIT LICENSE │
│ BSD LICENSE │
│ BSD LICENSE │
│ MIT LICENSE │
│ BSD LICENSE │
│ APACHE SOFTWARE LICENSE │
│ MIT LICENSE │
│ MIT LICENSE │
│ APACHE SOFTWARE LICENSE │
│ MIT │
│ MIT │
│ MIT LICENSE │
│ MOZILLA PUBLIC LICENSE 2.0 (MPL 2.0) │
│ MIT LICENSE │
│ GNU LESSER GENERAL PUBLIC LICENSE V2 OR LATER (LGPLV2+) │
│ MIT LICENSE │
│ MIT LICENSE │
│ BSD LICENSE │
│ APACHE SOFTWARE LICENSE;; BSD LICENSE │
│ MIT LICENSE │
│ MIT LICENSE │
│ CC0 1.0 UNIVERSAL (CC0 1.0) PUBLIC DOMAIN DEDICATION │
│ APACHE SOFTWARE LICENSE │
│ MIT LICENSE │
│ BSD LICENSE │
│ BSD LICENSE │
│ GNU LESSER GENERAL PUBLIC LICENSE V3 OR LATER (LGPLV3+) │
│ MIT LICENSE │
│ MOZILLA PUBLIC LICENSE 2.0 (MPL 2.0) │
│ BSD LICENSE │
│ MIT LICENSE │
│ BSD LICENSE │
│ PYTHON SOFTWARE FOUNDATION LICENSE │
│ BSD LICENSE │
│ MIT LICENSE │
│ MIT LICENSE │
│ MIT LICENSE │
│ BSD LICENSE │
│ MIT LICENSE │
│ BSD LICENSE │
│ BSD LICENSE │
│ APACHE SOFTWARE LICENSE;; GNU GENERAL PUBLIC LICENSE (GPL);; GNU LIBRARY OR LESSER GENERAL PUBLIC LICENSE (LGPL) │
│ MIT LICENSE │
│ BSD LICENSE │
│ BSD LICENSE │
│ BSD LICENSE │
│ GNU LIBRARY OR LESSER GENERAL PUBLIC LICENSE (LGPL) │
│ APACHE SOFTWARE LICENSE │
│ APACHE SOFTWARE LICENSE │
│ MIT LICENSE │
│ MIT LICENSE │
│ MIT LICENSE │
│ BSD LICENSE │
│ MIT LICENSE │
│ MIT LICENSE │
│ APACHE SOFTWARE LICENSE │
│ MIT LICENSE │
│ MIT LICENSE │
│ APACHE SOFTWARE LICENSE │
│ MIT LICENSE │
│ BSD LICENSE │
│ MIT LICENSE │
│ MIT LICENSE │
│ DFSG APPROVED;; MIT LICENSE │
│ MIT LICENSE │
│ APACHE SOFTWARE LICENSE;; BSD LICENSE │
│ MIT LICENSE │
│ BSD LICENSE │
│ MIT LICENSE │
│ GNU LESSER GENERAL PUBLIC LICENSE V3 OR LATER (LGPLV3+) │
│ APACHE SOFTWARE LICENSE │
│ MIT LICENSE │
│ MIT LICENSE │
│ BSD LICENSE │
│ BSD LICENSE │
│ BSD LICENSE │
│ ACADEMIC FREE LICENSE (AFL);; MIT LICENSE │
│ MIT LICENSE │
│ MIT LICENSE │
│ BSD LICENSE │
│ MIT LICENSE │
│ BSD LICENSE │
│ MIT LICENSE │
│ APACHE SOFTWARE LICENSE │
│ PYTHON SOFTWARE FOUNDATION LICENSE │
│ APACHE SOFTWARE LICENSE │
│ MIT LICENSE │
│ MIT LICENSE │
urllib3-secure-extra │ MIT LICENSE │
│ BSD LICENSE │
│ APACHE SOFTWARE LICENSE │
│ BSD LICENSE │
│ GNU GENERAL PUBLIC LICENSE (GPL) │
│ BSD LICENSE │
───┴──────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

The main issue is that there are more packages than the ones reported in the requirements file but also the version of licensecheck in the header is wrong.

Question. Choose one license in license groups.

As I see according 2021.3 changes - we can choose groups of licenses. But what if I want to use one of license in groups, but not others? How to catch this case?

Bug: licensecheck relies on Pypi even if private registries are declared in Pyproject.toml

Bug

System info

OS: Ubuntu 22.04
Version: 2023.4.2

Describe the bug

Here is a sample (simplified) pyproject.toml

[build-system]
requires = [ "poetry>=0.12",]
build-backend = "poetry.masonry.api"

[tool.poetry]
name = "my-project"
version = "0.0.0"
description = "My Amazing Project"
authors = [ "Brice Santus",]
license = "MIT"

[[tool.poetry.source]]
name = "devpi"
url = "https://private-devpi.custom-domain.com/root/+simple/"
default = true

[tool.poetry.dependencies]
python = "~3.9"
flask = "^2"
deduplication = "==1.2.1"

In such case we are using 2 dep: flask and deduplication. In our dev context both are retrieved through Devpi (or any mirroring private registry). flask is the well known public dep while deduplication is here a custom internal lib stored in our devpi instance.

While licensecheck will run it will try to retrieve data for deduplication from https://pypi.org/ and will find https://pypi.org/project/deduplication/ which is completely unrelated with our own lib.
This behaviour will cause licensecheck to retrieve a wrong information. (And in this particular case, to crash because https://pypi.org/project/deduplication/ is not well register and its metadata can't be well parser by packaging builtin lib)

I'm trying to find the best way to solve it and I would like your advices. Two options:

Implement support of custom registry for remote check of dependencies: not so trivial depending on how each registry could retrieve packages inof
Make evolve the ignorePackages flag to completely skip package processing and not only licence checking on them

For both options I can work on a Pull Request.

What do you think?

Expected outcome

We expect options on licensecheck to handle private dependencies stored in private registries

Actual outcome

licensecheck always rely on Pypi even when parsing private libs

Bug: A compatible dual licensed libarary is shown as incompatible

Bug

System info

OS: Windows
Version: 2023.1.3

Describe the bug

simplejson is licensed under MIT and AFL

Expected outcome

simplejson should be compatible to AGPL

Actual outcome

simplejson is shown not to be compatible

            Info               
┌─────────────────┬──────────────┐
│ Item            │ Value        │
├─────────────────┼──────────────┤
│ program         │ licensecheck │
│ version         │ 2023.1.3     │
│ license         │ mit          │
│ project_license │ agpl         │
└─────────────────┴──────────────┘
                                                                           
                             List Of Packages                              
┌────────────┬────────────────┬───────────────────────────────────────────┐
│ Compatible │ Package        │ License(s)                                │
├────────────┼────────────────┼───────────────────────────────────────────┤
│ ✖          │ simplejson     │ MIT License;; Academic Free License (AFL) │
└────────────┴────────────────┴───────────────────────────────────────────┘

Feature: give an option to only list running dependencies

Feature

I have read the comment above and have completed each step
I have filled in each heading below

Is your feature request related to a problem? Please describe

Currently when running licensecheck with poetry, it shows licenses for all dependencies (including development tools).
Most of the time only running dependencies matter since those are integrated with package's code.

Describe the solution you'd like

Since there might be use cases where someone wants to check all dependencies, consider adding an option where we can exclude dev-dependencies. Not sure what the default should be, I'm fine with either way.

Describe alternatives you've considered

I found a workaround by simply invoking two commands, but it would be great if that was not necessary:

poetry export --without-hashes -o requirements.txt
poetry run licensecheck -u requirements

Additional context

Info key not available within Pypi json

Hi,

I just noticed this error whenever I run licensecheck, or if I use the packageInfoFromPypi function for a list of packages.

I'll attach a picture with the actual error from PyCharm.

Feature: support setuptools packaging standard in pyproject.toml

I think it could be useful to add support for setuptools standard configuration used in pyproject.toml files as described here

I think this could be easily implemented by modifying packageinfo.getClassifiersLicense this way:

def getClassifiersLicense() -> Dict[str, Any]:
	"""Get the package classifiers and license from "setup.cfg", "pyproject.toml" or user input

	Returns:
		dict[str, Any]: {"classifiers": list[str], "license": str}
	"""
	if Path("setup.cfg").exists():
		config = configparser.ConfigParser()
		_ = config.read("setup.cfg")
		if "license" in config["metadata"]:
			return config["metadata"].__dict__
	if Path("pyproject.toml").exists():
		pyproject = tomli.loads(Path("pyproject.toml").read_text(encoding="utf-8"))
		tool = pyproject["tool"]
		if "poetry" in tool:
			return tool["poetry"]
		if "flit" in tool:
			return tool["flit"]["metadata"]
                if pyproject.get("project") is not None:
                        try:
                            return {"classifiers":  pyproject["project"]["classifiers"], "license": pyproject["project"]["license"]["text"]}
                         except KeyError:
                            pass

	return {"classifiers": [], "license": ""}

Feature: search licences from requirements.txt file

As I see - your tool can find licences only if packages already installed. Is it possible to find all licences from requirements.txt file (or optionally path to it)?

Update Matrix for AGPL_3_PLUS

Hey,

I am going to need to use AGPLV3with my project, since I am using https://github.com/ultralytics/ultralytics which is under AGPLV3. This project looks perfect to me, and I would like to contribute.

First, I would want to add #43. It is GPL compatible which should make it easy to fill in the table.

Secondly, I think some lines in the matrix are wrong for AGPL:

AGPL Should be compatible with itself, currently it is not.
AGPL Should be compatible with Apache V2 (Apache work can be used in AGPL, not the other way round). https://www.apache.org/licenses/GPL-compatibility.html)
Since AGPL is compatible with GPLV3+, it should also be compatible with ZOTO.

I think separating out Apache V1 and Apache V2 would also be useful, since that is a compatibility problem. But use of Apache V1 is very limited so probably not needed

"The Free Software Foundation considers all versions of the Apache License to be incompatible with the previous GPL versions 1 and 2.[2] Furthermore, it considers Apache License versions before 2.0 incompatible with GPLv3"

From https://en.wikipedia.org/wiki/Apache_License

I am happy to add all these changes in a PR. Being able to choose the license in the CLI to use for the project would also be a nice touch I can add. Thanks!

Thanks in advance,
Adam

Bug: When using pandas in my dependencies, all the extra_dependencies are checked even they are not installed

Before You Begin

Before proceeding, please make sure to follow these steps:

I have checked for similar issues in the project's issue tracker.
I have searched closed issues to see if a similar problem was reported
before.

Issue Details

I get pipeline crashes on various projects recently because of LicenseCheck upgrade to 2023.5.1 from 2023.1.1.
whenever I check in pandas in my deps, all its extra dependencies are checked as well including PyQT5 that is under GPL3 which is incompatible with MIT.

My licensecheck parameters

[tool.licensecheck]
using = "PEP631"

the dependencies:

dependencies = [
    "deprecated>=1.2.14",
    "pandas<2",
    "earthengine-api",
    "pyarrow"
]

and the result from the pre-commit hook:

┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Item            ┃ Value        ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ program         │ licensecheck │
│ version         │ 2023.5.1     │
│ license         │ MIT LICENSE  │
│ project_license │ MIT LICENSE  │
└─────────────────┴──────────────┘
                                                                                
                                List Of Packages                                
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Compatible ┃ Package              ┃ License(s)                               ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ✔          │ Bottleneck           │ BSD LICENSE                              │
│ ✔          │ Deprecated           │ MIT LICENSE                              │
│ ✔          │ Jinja2               │ BSD LICENSE                              │
│ ✖          │ PyQt5                │ GPL V3                                   │
│ ✔          │ QtPy                 │ MIT LICENSE                              │
│ ✔          │ SQLAlchemy           │ MIT LICENSE                              │
│ ✔          │ XlsxWriter           │ BSD LICENSE                              │
│ ✔          │ beautifulsoup4       │ MIT LICENSE                              │
│ ✔          │ dataframe-api-compat │ MIT LICENSE                              │
│ ✔          │ earthengine-api      │ APACHE SOFTWARE LICENSE                  │
│ ✔          │ fastparquet          │ APACHE SOFTWARE LICENSE                  │
│ ✔          │ fsspec               │ BSD LICENSE                              │
│ ✔          │ gcsfs                │ BSD LICENSE                              │
│ ✔          │ html5lib             │ MIT LICENSE                              │
│ ✔          │ hypothesis           │ MOZILLA PUBLIC LICENSE 2.0 (MPL 2.0)     │
│ ✔          │ lxml                 │ BSD LICENSE                              │
│ ✔          │ matplotlib           │ PYTHON SOFTWARE FOUNDATION LICENSE       │
│ ✔          │ numba                │ BSD LICENSE                              │
│ ✔          │ numexpr              │ MIT LICENSE                              │
│ ✔          │ numpy                │ BSD LICENSE                              │
│ ✔          │ odfpy                │ APACHE SOFTWARE LICENSE;; GNU GENERAL    │
│            │                      │ PUBLIC LICENSE (GPL);; GNU LIBRARY OR    │
│            │                      │ LESSER GENERAL PUBLIC LICENSE (LGPL)     │
│ ✔          │ openpyxl             │ MIT LICENSE                              │
│ ✔          │ pandas               │ BSD LICENSE                              │
│ ✔          │ pandas-gbq           │ BSD LICENSE                              │
│ ✔          │ psycopg2             │ GNU LIBRARY OR LESSER GENERAL PUBLIC     │
│            │                      │ LICENSE (LGPL)                           │
│ ✔          │ pyarrow              │ APACHE SOFTWARE LICENSE                  │
│ ✔          │ pymysql              │ MIT LICENSE                              │
│ ✔          │ pyreadstat           │ APACHE SOFTWARE LICENSE                  │
│ ✔          │ pytest               │ MIT LICENSE                              │
│ ✔          │ pytest-asyncio       │ APACHE SOFTWARE LICENSE                  │
│ ✔          │ pytest-xdist         │ MIT LICENSE                              │
│ ✔          │ python-dateutil      │ APACHE SOFTWARE LICENSE;; BSD LICENSE    │
│ ✔          │ pytz                 │ MIT LICENSE                              │
│ ✔          │ pyxlsb               │ GNU LESSER GENERAL PUBLIC LICENSE V3 OR  │
│            │                      │ LATER (LGPLV3+)                          │
│ ✔          │ s3fs                 │ BSD LICENSE                              │
│ ✔          │ scipy                │ BSD LICENSE                              │
│ ✔          │ tables               │ BSD LICENSE                              │
│ ✔          │ tabulate             │ MIT LICENSE                              │
│ ✔          │ tzdata               │ APACHE SOFTWARE LICENSE                  │
│ ✔          │ wrapt                │ BSD LICENSE                              │
│ ✔          │ xarray               │ APACHE SOFTWARE LICENSE                  │
│ ✔          │ xlrd                 │ BSD LICENSE                              │
│ ✔          │ zstandard            │ BSD LICENSE                              │
└────────────┴──────────────────────┴──────────────────────────────────────────┘

Even though a simple pip install pandas && pip show PyQT5 shows that it's not installed by default.

Note that I'm using pandas > 2 that have changed to extra_requires recently.

Expected Behavior

I would expect to not see it in the list as it's not installed

Bug: licensecheck fails to identify all licenses of docutils package

I am publishing a python package, licensed with a MIT license, that depends on docutils == 0.17.1. According to its license page docutils is licensed as Public Domain, Python Software, BSD2 and GPL but it seems that licenscecheck fails to correctly parse the GPL and BSD2.

Reproducible example

OS: MacOS Monterey 12.6

Requirements file: requirements.txt

docutils==0.17.1

Project License: MIT

Command: python -m licensecheck --using 'requirements:requirements.txt'
Output:

WARN: 'GNU GENERAL PUBLIC LICENSE (GPL)' License not identified so falling back to NO_LICENSE
┌──────┬────────────────────┬──────────────────────────────┐
│Compat│      Package       │           License            │
├──────┼────────────────────┼──────────────────────────────┤
│  ❌  │docutils            │Public Domain, Python Software│
└──────┴────────────────────┴──────────────────────────────┘

Expected Output:

WARN: 'GNU GENERAL PUBLIC LICENSE (GPL)' License not identified so falling back to NO_LICENSE

┌──────┬────────────────────┬────────────────────────────────────────┐
│Compat│      Package       │           License                      │
├──────┼────────────────────┼────────────────────────────────────────┤
│  ❌  │docutils            │Public Domain, Python Software, GPL, BSD│
└──────┴────────────────────┴────────────────────────────────────────┘

Feature. Add path to requirements.txt

In my project I use next requirements hierarchy:
requirements/base.txt
requirements/dev.txt
requirements/docs.txt
requirements/test.txt
requirements/lint.txt

I want to check one or multiple files in this hierarchy, but according 2021.2 release - I can't.
Same hierarchy we can see, for example, here:
https://github.com/aio-libs/aiohttp/tree/master/requirements
can you add optional flag path to file?

Bug: crash where no classifiers field exists

Bug

I have read the comment above and have completed each step
I have filled out the system info
I have described the bug, filled in the expected outcome and the actual
outcome including screenshots where appropriate

System info

OS: N/A
Browser: N/A
Version: 2021.4.1

Describe the bug

When classifiers is not defined in pyproject.toml licensecheck crashes with an error that can confuse new users.

Expected outcome

Program working

Actual outcome

Exception like this:

Traceback (most recent call last):
  File "<snip>/.venv/bin/licensecheck", line 8, in <module>
    sys.exit(cli())
  File "<snip>/.venv/lib/python3.8/site-packages/licensecheck/__init__.py", line 84, in cli
    dependenciesWLicenses = get_deps.getDepsWLicenses(
  File "<snip>/.venv/lib/python3.8/site-packages/licensecheck/get_deps.py", line 107, in getDepsWLicenses
    myLiceTxt = packageinfo.getMyPackageLicense()
  File "<snip>/.venv/lib/python3.8/site-packages/licensecheck/packageinfo.py", line 201, in getMyPackageLicense
    licenseClassifier = licenseFromClassifierlist(metaData["classifiers"])  # type:ignore
  File "/<snip>/.venv/lib/python3.8/site-packages/tomlkit/items.py", line 1007, in __getitem__
    return self._value[key]
  File "<snip>/.venv/lib/python3.8/site-packages/tomlkit/container.py", line 553, in __getitem__
    raise NonExistentKey(key)
tomlkit.exceptions.NonExistentKey: 'Key "classifiers" does not exist.'

Question: poetry.lock file required for license checking?

Question

I have read the comment above and have completed each step

Hi,

thanks for the great project. I didn't know that I needed this, but this is really helpful. Yesterday I added this via the pre-commit hook.

My question is: Is the poetry.lock file required for license checking? I was unsure what I was doing wrong, I got "WARN: 'UNKNOWN' License not identified so falling back to NO_LICENSE" consistently in CI.

I am writing a library and added poetry.lock to .gitignore.

I tried to create a minimal example of what is going wrong in CI here: https://github.com/afuetterer/minimal-example

I think the issue might come from the missing lockfile. Could that be it?

Incorrect format toml file to json.dumps

Bug

I have read the comment above and have completed each step
I have filled out the system info
I have described the bug, filled in the expected outcome and the actual
outcome including screenshots where appropriate

System info

OS: Linux niccolum-HP-ProBook-650-G3 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Version: 2021.5.2

Describe the bug

Fail on formatter Version to json.dumps

    # Format the results
    if simpleConf.get("format", "simple") in formatter.formatMap:
        print(dependenciesWLicenses)  # add here print
        print(
            formatter.formatMap[simpleConf.get("format", "simple")](dependenciesWLicenses),
            file=filename,
        )
    else:
        exitCode = 2

$ poetry run licensecheck --zero

Expected outcome

OK

Actual outcome

[{'name': 'uvicorn', 'version': <Version('0.16.0')>, 'namever': 'uvicorn 0.16.0', 'home_page': 'https://www.uvicorn.org/', 'author': 'Tom Christie', 'size': 158778, 'license': 'BSD License', 'license_compat': True}, {'name': 'typing-extensions', 'version': <Version('4.0.1')>, 'namever': 'typing_extensions 4.0.1', 'home_page': 'UNKNOWN', 'author': 'UNKNOWN', 'size': 22706, 'license': 'Python Software Foundation License', 'license_compat': True}, {'name': 'starlette', 'version': <Version('0.17.1')>, 'namever': 'starlette 0.17.1', 'home_page': 'https://github.com/encode/starlette', 'author': 'Tom Christie', 'size': 186061, 'license': 'BSD License', 'license_compat': True}, {'name': 'starlette-csrf', 'version': <Version('1.4.0')>, 'namever': 'starlette-csrf 1.4.0', 'home_page': 'https://github.com/frankie567/starlette-csrf', 'author': 'François Voron', 'size': 3968, 'license': 'MIT License', 'license_compat': True}, {'name': 'sniffio', 'version': <Version('1.2.0')>, 'namever': 'sniffio 1.2.0', 'home_page': 'https://github.com/python-trio/sniffio', 'author': 'Nathaniel J. Smith', 'size': 4750, 'license': 'MIT License, Apache Software License', 'license_compat': True}, {'name': 'single-source', 'version': <Version('0.2.0')>, 'namever': 'single-source 0.2.0', 'home_page': 'https://github.com/rabbit72/single-source', 'author': 'Daniil Shadrin', 'size': 3898, 'license': 'MIT License', 'license_compat': True}, {'name': 'secweb', 'version': <Version('1.3.0')>, 'namever': 'Secweb 1.3.0', 'home_page': 'https://github.com/tmotagam/Secweb', 'author': 'Motagamwala Taha Arif Ali', 'size': 19418, 'license': 'Mozilla Public License 2.0 (MPL 2.0)', 'license_compat': True}, {'name': 'rfc3986', 'version': <Version('1.5.0')>, 'namever': 'rfc3986 1.5.0', 'home_page': 'http://rfc3986.readthedocs.io', 'author': 'Ian Stapleton Cordasco', 'size': 94910, 'license': 'Apache Software License', 'license_compat': True}, {'name': 'pydantic', 'version': <Version('1.9.0')>, 'namever': 'pydantic 1.9.0', 'home_page': 'https://github.com/samuelcolvin/pydantic', 'author': 'Samuel Colvin', 'size': 43343619, 'license': 'MIT License', 'license_compat': True}, {'name': 'itsdangerous', 'version': <Version('2.0.1')>, 'namever': 'itsdangerous 2.0.1', 'home_page': 'https://palletsprojects.com/p/itsdangerous/', 'author': 'Armin Ronacher', 'size': 45403, 'license': 'BSD License', 'license_compat': True}, {'name': 'idna', 'version': <Version('3.3')>, 'namever': 'idna 3.3', 'home_page': 'https://github.com/kjd/idna', 'author': 'Kim Davies', 'size': 267666, 'license': 'BSD License', 'license_compat': True}, {'name': 'httpx', 'version': <Version('0.21.3')>, 'namever': 'httpx 0.21.3', 'home_page': 'https://github.com/encode/httpx', 'author': 'Tom Christie', 'size': 273039, 'license': 'BSD License', 'license_compat': True}, {'name': 'httpcore', 'version': <Version('0.14.4')>, 'namever': 'httpcore 0.14.4', 'home_page': 'https://github.com/encode/httpcore', 'author': 'Tom Christie', 'size': 180787, 'license': 'BSD License', 'license_compat': True}, {'name': 'h11', 'version': <Version('0.12.0')>, 'namever': 'h11 0.12.0', 'home_page': 'https://github.com/python-hyper/h11', 'author': 'Nathaniel J. Smith', 'size': 167163, 'license': 'MIT License', 'license_compat': True}, {'name': 'gunicorn', 'version': <Version('20.1.0')>, 'namever': 'gunicorn 20.1.0', 'home_page': 'https://gunicorn.org', 'author': 'Benoit Chesneau', 'size': 241482, 'license': 'MIT License', 'license_compat': True}, {'name': 'fastapi', 'version': <Version('0.71.0')>, 'namever': 'fastapi 0.71.0', 'home_page': 'https://github.com/tiangolo/fastapi', 'author': 'Sebastián Ramírez', 'size': 202999, 'license': 'MIT License', 'license_compat': True}, {'name': 'click', 'version': <Version('8.0.3')>, 'namever': 'click 8.0.3', 'home_page': 'https://palletsprojects.com/p/click/', 'author': 'Armin Ronacher', 'size': 341344, 'license': 'BSD License', 'license_compat': True}, {'name': 'charset-normalizer', 'version': <Version('2.0.10')>, 'namever': 'charset-normalizer 2.0.10', 'home_page': 'https://github.com/ousret/charset_normalizer', 'author': 'Ahmed TAHRI @Ousret', 'size': 130739, 'license': 'MIT License', 'license_compat': True}, {'name': 'certifi', 'version': <Version('2021.10.8')>, 'namever': 'certifi 2021.10.8', 'home_page': 'https://certifiio.readthedocs.io/en/latest/', 'author': 'Kenneth Reitz', 'size': 268577, 'license': 'Mozilla Public License 2.0 (MPL 2.0)', 'license_compat': True}, {'name': 'asgiref', 'version': <Version('3.4.1')>, 'namever': 'asgiref 3.4.1', 'home_page': 'https://github.com/django/asgiref/', 'author': 'Django Software Foundation', 'size': 59054, 'license': 'BSD License', 'license_compat': True}, {'name': 'anyio', 'version': <Version('3.4.0')>, 'namever': 'anyio 3.4.0', 'home_page': 'UNKNOWN', 'author': 'Alex Grönholm', 'size': 276992, 'license': 'MIT License', 'license_compat': True}, {'name': 'colorama', 'version': '0.4.4', 'namever': 'colorama 0.4.4', 'home_page': 'https://github.com/tartley/colorama', 'author': 'Jonathan Hartley', 'size': 27813, 'license': 'BSD License', 'license_compat': True}]

Traceback (most recent call last):
  File "/home/niccolum/projects/github/fellowmate/auth/.venv/bin/licensecheck", line 8, in <module>
    sys.exit(cli())
  File "/home/niccolum/projects/github/fellowmate/auth/.venv/lib/python3.10/site-packages/licensecheck/__init__.py", line 96, in cli
    formatter.formatMap[simpleConf.get("format", "simple")](dependenciesWLicenses),
  File "/home/niccolum/projects/github/fellowmate/auth/.venv/lib/python3.10/site-packages/licensecheck/formatter.py", line 96, in json
    return dumps(out, indent="\t")
  File "/usr/lib/python3.10/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.10/json/encoder.py", line 201, in encode
    chunks = list(chunks)
  File "/usr/lib/python3.10/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.10/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.10/json/encoder.py", line 325, in _iterencode_list
    yield from chunks
  File "/usr/lib/python3.10/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.10/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/usr/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Version is not JSON serializable

My pyproject.toml (only poetry tool):

[tool.poetry]
name = "auth backend"
version = "0.1.0"
description = ""
authors = ["Niccolum <[email protected]>"]
license = "MIT"
readme = "README.md"
homepage = "https://github.com/fellowmate"
repository = "https://github.com/fellowmate/auth"
classifiers = [
    "Development Status :: 1 - Planning",
    "Environment :: Web Environment",
    "Framework :: FastAPI",
    "License :: OSI Approved :: MIT License",
    "Operating System :: POSIX :: Linux",
    "Programming Language :: Python :: 3.10",
    "Topic :: Home Automation",
]

[tool.poetry.dependencies]
python = "^3.10"
gunicorn = "^20.1.0"
uvicorn = {extras = ["standart"], version = "^0.16.0"}
single-source = "^0.2.0"
fastapi = "^0.71.0"
httpx = "^0.21.3"
starlette-csrf = "^1.4.0"
Secweb = "^1.3.0"

[tool.poetry.dev-dependencies]
black = "^21.12b0"
flake8 = "^4.0.1"
bandit = "^1.7.1"
isort = "^5.10.1"
pre-commit = "^2.16.0"

pytest = "^6.2.5"
pytest-cov = "^3.0.0"
pytest-custom-report = "^1.0.1"
pytest-reverse = "^1.3.0"
pytest-lazy-fixture = "^0.6.3"
pytest-nice-parametrize = "^1.0.1"
pytest-asyncio = "^0.16.0"
pytest-pythonpath = "^0.7.3"
licensecheck = "^2021.5.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Feature: Support for PEP631: Declaring dependencies in pyptoject.toml

Feature

I have read the comment above and have completed each step
I have filled in each heading below

Is your feature request related to a problem? Please describe

I'm using the build system Hatch, without a separate requirements.txt file, and I would like to check my dependencies' licences without first having to extract that information with a separate step.

Describe the solution you'd like

Support is added to directly read dependencies from pyproject.toml.

Describe alternatives you've considered

Continue to use a wrapper script to first create a requirements.txt file and then delete it again.

Additional context

Bug: fail with pip>= 21.3

Bug

I have read the comment above and have completed each step
I have filled out the system info
I have described the bug, filled in the expected outcome and the actual
outcome including screenshots where appropriate

System info

OS: Ubuntu 18 LTS
Version: 2021.5

Describe the bug

New pip broke internal API. Similar issue with jazzband/pip-tools#1503

Expected outcome

Zero code or lisencecheck output

Actual outcome

licensecheck.............................................................Failed
- hook id: licensecheck
- exit code: 1

Traceback (most recent call last):
  File "/home/niccolum/.cache/pre-commit/repohao7hzis/py_env-python3.10/bin/licensecheck", line 5, in <module>
    from licensecheck import cli
  File "/home/niccolum/.cache/pre-commit/repohao7hzis/py_env-python3.10/lib/python3.10/site-packages/licensecheck/__init__.py", line 12, in <module>
    from licensecheck import formatter, get_deps
  File "/home/niccolum/.cache/pre-commit/repohao7hzis/py_env-python3.10/lib/python3.10/site-packages/licensecheck/get_deps.py", line 10, in <module>
    from licensecheck import license_matrix, packageinfo
  File "/home/niccolum/.cache/pre-commit/repohao7hzis/py_env-python3.10/lib/python3.10/site-packages/licensecheck/packageinfo.py", line 15, in <module>
    from pip._internal.utils.misc import get_installed_distributions
ImportError: cannot import name 'get_installed_distributions' from 'pip._internal.utils.misc' (/home/niccolum/.cache/pre-commit/repohao7hzis/py_env-python3.10/lib/python3.10/site-packages/pip/_internal/utils/misc.py)

Feature: Support option ignore-licenses in pyproject.toml and support multiple entries

Feature

Is your feature request related to a problem? Please describe

a) If I run lichensecheck with

licensecheck --ignore-licenses='Apache'

I get the expected results. However, If I put that option in pyproject.toml it does not work.

[tool.licensecheck]
using = 'PEP631'
ignore-licenses = 'Apache'
zero = true

I also tried

ignore-licenses = [ 'Apache']

and

ignore-licenses = 'Apache Software License'

b) Furthermor, if I try

licensecheck --ignore-licenses='Zope Public License, Apache Software License'

only Apache is ignored instead of ignoring both licenses.

Describe the solution you'd like

Allow to specify multiple licenses in pyproject.toml that should be ignored.

Describe alternatives you've considered

Specify them at the command line; did not work.

Example pyproject.toml:

https://github.com/fraunhofer-isi/micat/blob/main/back_end/pyproject.toml

Bug: Package parsing results in an exception when "celery" is added to a poetry project

Bug

I have read the comment above and have completed each step
I have filled out the system info
I have described the bug, filled in the expected outcome and the actual
outcome including screenshots where appropriate

System info

OS: Ubuntu
Version: 20.04

Describe the bug

licensecheck command fails following the addition of the celery package to a poetry project.

$ poetry add Celery
Using version ^5.2.7 for celery

Updating dependencies
Resolving dependencies... (0.8s)

Writing lock file

Package operations: 10 installs, 0 updates, 0 removals

  • Installing vine (5.0.0)
  • Installing wcwidth (0.2.6)
  • Installing amqp (5.1.1)
  • Installing prompt-toolkit (3.0.38)
  • Installing billiard (3.6.4.0)
  • Installing click-didyoumean (0.3.0)
  • Installing click-plugins (1.1.1)
  • Installing click-repl (0.2.0)
  • Installing kombu (5.2.4)
  • Installing celery (5.2.7)

$ licensecheck
Enter the project license
>Proprietary
Traceback (most recent call last):
  File ".../.venv/bin/licensecheck", line 8, in <module>
    sys.exit(cli())
  File ".../.venv/lib/python3.10/site-packages/licensecheck/__init__.py", line 90, in cli
    depsWithLicenses = get_deps.getDepsWithLicenses(
  File ".../.venv/lib/python3.10/site-packages/licensecheck/get_deps.py", line 131, in getDepsWithLicenses
    packages = packageinfo.getPackages(reqs)
  File ".../.venv/lib/python3.10/site-packages/licensecheck/packageinfo.py", line 117, in getPackages
    packageinfo.add(getPackageInfoLocal(requirement))
  File ".../.venv/lib/python3.10/site-packages/licensecheck/packageinfo.py", line 42, in getPackageInfoLocal
    packagePath = ilr.files(requirement)
  File "/usr/lib/python3.10/importlib/_common.py", line 22, in files
    return from_package(get_package(package))
  File "/usr/lib/python3.10/importlib/_common.py", line 67, in get_package
    if wrap_spec(resolved).submodule_search_locations is None:
  File "/usr/lib/python3.10/importlib/_adapters.py", line 16, in __getattr__
    return getattr(self.spec, name)
AttributeError: 'NoneType' object has no attribute 'submodule_search_locations'

Expected outcome

The licensecheck command should complete successfully and output the license details of main packages in the poetry project.

Actual outcome

The licensecheck command fails with a runtime exception.

Bug: Unexpected warnings for ignored license

Bug

System info

OS: Windows

Describe the bug

I try to ignore the 'ZOPE PUBLIC LICENSE'

licensecheck --ignore-licenses='ZOPE PUBLIC LICENSE'

and would expect that no warnings are shown about it. However, I get many warnings:

Expected outcome

Do not show 'License not identified ' warnings for licenses that are ignored.

Actual outcome

WARN: 'ZOPE PUBLIC LICENSE' License not identified so falling back to NO_LICENSE
WARN: 'ZOPE PUBLIC LICENSE' License not identified so falling back to NO_LICENSE
WARN: 'ZOPE PUBLIC LICENSE' License not identified so falling back to NO_LICENSE
WARN: 'ZOPE PUBLIC LICENSE' License not identified so falling back to NO_LICENSE
WARN: 'ZOPE PUBLIC LICENSE' License not identified so falling back to NO_LICENSE
WARN: 'ZOPE PUBLIC LICENSE' License not identified so falling back to NO_LICENSE
WARN: 'ZOPE PUBLIC LICENSE' License not identified so falling back to NO_LICENSE
WARN: 'ZOPE PUBLIC LICENSE' License not identified so falling back to NO_LICENSE
WARN: 'ZOPE PUBLIC LICENSE' License not identified so falling back to NO_LICENSE
WARN: 'ZOPE PUBLIC LICENSE' License not identified so falling back to NO_LICENSE
WARN: 'ZOPE PUBLIC LICENSE' License not identified so falling back to NO_LICENSE
WARN: 'ZOPE PUBLIC LICENSE' License not identified so falling back to NO_LICENSE
WARN: 'ZOPE PUBLIC LICENSE' License not identified so falling back to NO_LICENSE
WARN: 'ZOPE PUBLIC LICENSE' License not identified so falling back to NO_LICENSE
WARN: 'ZOPE PUBLIC LICENSE' License not identified so falling back to NO_LICENSE
WARN: 'ZOPE PUBLIC LICENSE' License not identified so falling back to NO_LICENSE

Related: #46

Bug: Permissive libraries are not compatible with closed licenses

Bug

I have read the comment above and have completed each step
I have filled out the system info
I have described the bug, filled in the expected outcome and the actual outcome including screenshots where appropriate

System info

Version: 2023.2

Describe the bug

Licenses such as MIT are marked as incompatible with a proprietary license

Expected outcome

Permissive libraries are compatible with closed licenses

Actual outcome

Permissive libraries are not compatible with closed licenses

Bug: failure in correctly parsing installed namespace packages

System info

OS: MacOS
Version: 13.4

Describe the bug

I just realised that licensecheck does not parse correctly namespace packages locally installed (while it does not seem to face the same problem for namespace packages on PyPI).

Suppose you have a namespace package called 'hexagonal-repository-gcs' that has the following nested structure:

hexagonal
└─── repository
      └───  gcs
             │ __init__.py
             │ ...

In particular the issue arises in the function packageinfo.getPackageInfoLocal due to the fact that the call to resources.files(requirement) with requirement='hexagonal-repository-gcs' fails with a ModuleNotFoundError since importlib.import_module('hexagonal-repository-gcs') throws that error.

This is correct by importlib since the right call to importlib.import_module should have been importlib.import_module('hexagonal.repository.gcs') because my dependency is a module in a namespace but this does not seem correct from the point of view of the packageinfo.getPackageInfoLocal because this breaks the execution raising the exception but the package is actually installed an most of the package info had been correctly retrieved (only the size is missing but it is a ancillary information that is not actually used in the core of licensecheck business).

As of now the code is:

def getPackageInfoLocal(requirement: str) -> PackageInfo:
	"""Get package info from local files including version, author
	and	the license.

	:param str requirement: name of the package
	:raises ModuleNotFoundError: if the package does not exist
	:return PackageInfo: package information
	"""
	try:
		# Get pkg metadata: license, homepage + author
		pkgMetadata = metadata.metadata(requirement)
		lice = licenseFromClassifierlist(pkgMetadata.get_all("Classifier"))
		if lice == UNKNOWN:
			lice = pkgMetadata.get("License", UNKNOWN)
		homePage = pkgMetadata.get("Home-page", UNKNOWN)
		author = pkgMetadata.get("Author", UNKNOWN)
		name = pkgMetadata.get("Name", UNKNOWN)
		version = pkgMetadata.get("Version", UNKNOWN)
		size = 0
		try:
			packagePath = resources.files(requirement)
			size = getModuleSize(cast(Path, packagePath), name)
		except TypeError:
			pass
		# append to pkgInfo
		return PackageInfo(
			name=name,
			version=version,
			homePage=homePage,
			author=author,
			size=size,
			license=lice,
		)

	except (metadata.PackageNotFoundError, ModuleNotFoundError) as error:
		raise ModuleNotFoundError from error

I'd suggest to use instead directly the size computed by metadata.Distribution to avoid trying to import the package:

def getPackageInfoLocal(requirement: str) -> PackageInfo:
	"""Get package info from local files including version, author
	and	the license.

	:param str requirement: name of the package
	:raises ModuleNotFoundError: if the package does not exist
	:return PackageInfo: package information
	"""
	try:
		# Get pkg metadata: license, homepage + author
		pkgMetadata = metadata.metadata(requirement)
		lice = licenseFromClassifierlist(pkgMetadata.get_all("Classifier"))
		if lice == UNKNOWN:
			lice = pkgMetadata.get("License", UNKNOWN)
		homePage = pkgMetadata.get("Home-page", UNKNOWN)
		author = pkgMetadata.get("Author", UNKNOWN)
		name = pkgMetadata.get("Name", UNKNOWN)
		version = pkgMetadata.get("Version", UNKNOWN)		
                size = sum([pp.size for pp in metadata.Distribution.from_name(requirement).files if pp.size is not None])
		# append to pkgInfo
		return PackageInfo(
			name=name,
			version=version,
			homePage=homePage,
			author=author,
			size=size,
			license=lice,
		)

	except metadata.PackageNotFoundError as error:
		raise ModuleNotFoundError from error

Note: the size computed by metadata.Distribution is not the same as the on retrieved by PyPI (for published packages, ofc), they can differ significantly but, since this information is actually of no use for licensecheck business, I wouldn't care very much about understanding the motivations of this difference and try to solve it.

Reduce default pool size to 1

It looks like this is querying pypi very rapidly by default. Even after I set POOL_SIZE=1, it rapidly exhausts the rate limit and then instead of waiting, it fails.

It looks like license_check itself isn't at fault, but the way that yolk is configured & how it isn't respecting rate limits.

This is on a requirements.txt file of about 20 packages.

xmlrpc.client.Fault: <Fault -32500: 'HTTPTooManyRequests: The action could not be performed because there were too many requests by the client. Limit may reset in 53 seconds.'>

Bug: failed parsing requirements.txt

Bug

The newly released version introduced a bug in how requirements.txt files are read.

Describe the bug

With a requirements file of the form (generated by pip-tools):

#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --no-emit-index-url --output-file=requirements/tmp.txt subset.in
#
aiohttp==3.8.5
    # via
    #   -c requirements/requirements_dev.txt
    #   langchain
aiosignal==1.3.1
    # via
    #   -c requirements/requirements_dev.txt
    #   aiohttp
anyio==3.7.1
    # via
    #   -c requirements/requirements_dev.txt
    #   starlette
async-timeout==4.0.2
    # via
    #   -c requirements/requirements_dev.txt
    #   aiohttp
    #   langchain
attrs==21.4.0
    # via
    #   -c requirements/requirements_dev.txt
    #   aiohttp
certifi==2023.7.22
    # via
    #   -c requirements/requirements_dev.txt
    #   requests
cfg-load==0.9.0
    # via
    #   -c requirements/requirements_dev.txt
    #   py4ai-core
charset-normalizer==3.2.0
    # via
    #   -c requirements/requirements_dev.txt
    #   aiohttp
    #   requests
click==8.1.6
    # via
    #   -c requirements/requirements_dev.txt
    #   nltk
    #   uvicorn
cramjam==2.6.2
    # via
    #   -c requirements/requirements_dev.txt
    #   fastparquet
dataclasses-json==0.5.13
    # via
    #   -c requirements/requirements_dev.txt
    #   langchain
deprecated==1.2.14
    # via
    #   -c requirements/requirements_dev.txt
    #   py4ai-core
dnspython==2.4.1
    # via
    #   -c requirements/requirements_dev.txt
    #   pinecone-client
exceptiongroup==1.1.2
    # via
    #   -c requirements/requirements_dev.txt
    #   anyio
faiss-cpu==1.7.4
    # via
    #   -c requirements/requirements_dev.txt
    #   -r subset.in
fastapi==0.100.1
    # via
    #   -c requirements/requirements_dev.txt
    #   fastapi-utils
    #   microservices-core
    #   microservices-hexagonal
fastapi-utils==0.2.1
    # via
    #   -c requirements/requirements_dev.txt
    #   microservices-core
    #   microservices-hexagonal
fastparquet==2023.7.0
    # via
    #   -c requirements/requirements_dev.txt
    #   -r subset.in
filelock==3.12.2
    # via
    #   -c requirements/requirements_dev.txt
    #   huggingface-hub
    #   torch
    #   transformers
frozenlist==1.4.0
    # via
    #   -c requirements/requirements_dev.txt
    #   aiohttp
    #   aiosignal
fsspec==2023.6.0
    # via
    #   -c requirements/requirements_dev.txt
    #   fastparquet
    #   huggingface-hub
h11==0.14.0
    # via
    #   -c requirements/requirements_dev.txt
    #   uvicorn
hexagonal-core @ git+https://bitbucket.org/3rdplace/[email protected]
    # via
    #   -c requirements/requirements_dev.txt
    #   hexagonal-repository-core
    #   microservices-hexagonal
hexagonal-repository-core @ git+https://bitbucket.org/3rdplace/[email protected]
    # via
    #   -c requirements/requirements_dev.txt
    #   hexagonal-repository-langchain
hexagonal-repository-langchain @ git+https://bitbucket.org/3rdplace/[email protected]
    # via
    #   -c requirements/requirements_dev.txt
    #   -r subset.in
    #   hexagonal-repository-pinecone
hexagonal-repository-pinecone @ git+https://bitbucket.org/3rdplace/[email protected]
    # via
    #   -c requirements/requirements_dev.txt
    #   -r subset.in
huggingface-hub==0.16.4
    # via
    #   -c requirements/requirements_dev.txt
    #   sentence-transformers
    #   transformers
idna==3.4
    # via
    #   -c requirements/requirements_dev.txt
    #   anyio
    #   requests
    #   yarl
jinja2==3.1.2
    # via
    #   -c requirements/requirements_dev.txt
    #   torch
joblib==1.3.1
    # via
    #   -c requirements/requirements_dev.txt
    #   nltk
    #   scikit-learn
langchain==0.0.246
    # via
    #   -c requirements/requirements_dev.txt
    #   hexagonal-repository-langchain
    #   hexagonal-repository-pinecone
langsmith==0.0.15
    # via
    #   -c requirements/requirements_dev.txt
    #   langchain
loguru==0.7.0
    # via
    #   -c requirements/requirements_dev.txt
    #   pinecone-client
markupsafe==2.1.3
    # via
    #   -c requirements/requirements_dev.txt
    #   jinja2
marshmallow==3.20.1
    # via
    #   -c requirements/requirements_dev.txt
    #   dataclasses-json
microservices-core @ git+https://bitbucket.org/3rdplace/[email protected]
    # via
    #   -c requirements/requirements_dev.txt
    #   microservices-hexagonal
microservices-hexagonal @ git+https://bitbucket.org/3rdplace/[email protected]
    # via
    #   -c requirements/requirements_dev.txt
    #   -r subset.in
mpmath==1.3.0
    # via
    #   -c requirements/requirements_dev.txt
    #   sympy
mpu[io]==0.23.1
    # via
    #   -c requirements/requirements_dev.txt
    #   cfg-load
multidict==6.0.4
    # via
    #   -c requirements/requirements_dev.txt
    #   aiohttp
    #   yarl
mypy-extensions==1.0.0
    # via
    #   -c requirements/requirements_dev.txt
    #   typing-inspect
networkx==3.1
    # via
    #   -c requirements/requirements_dev.txt
    #   torch
nltk==3.8.1
    # via
    #   -c requirements/requirements_dev.txt
    #   sentence-transformers
numexpr==2.8.4
    # via
    #   -c requirements/requirements_dev.txt
    #   langchain
numpy==1.25.1
    # via
    #   -c requirements/requirements_dev.txt
    #   fastparquet
    #   langchain
    #   numexpr
    #   pandas
    #   pinecone-client
    #   scikit-learn
    #   scipy
    #   sentence-transformers
    #   torchvision
    #   transformers
openapi-schema-pydantic==1.2.4
    # via
    #   -c requirements/requirements_dev.txt
    #   langchain
packaging==23.1
    # via
    #   -c requirements/requirements_dev.txt
    #   fastparquet
    #   huggingface-hub
    #   marshmallow
    #   transformers
pandas==2.0.3
    # via
    #   -c requirements/requirements_dev.txt
    #   fastparquet
    #   py4ai-core
pillow==10.0.0
    # via
    #   -c requirements/requirements_dev.txt
    #   torchvision
pinecone-client==2.2.2
    # via
    #   -c requirements/requirements_dev.txt
    #   hexagonal-repository-pinecone
py4ai-core==1.0.0
    # via
    #   -c requirements/requirements_dev.txt
    #   hexagonal-core
    #   hexagonal-repository-core
    #   hexagonal-repository-langchain
    #   microservices-core
    #   microservices-hexagonal
pydantic==1.10.12
    # via
    #   -c requirements/requirements_dev.txt
    #   -r subset.in
    #   fastapi
    #   fastapi-utils
    #   hexagonal-repository-langchain
    #   hexagonal-repository-pinecone
    #   langchain
    #   langsmith
    #   microservices-core
    #   microservices-hexagonal
    #   openapi-schema-pydantic
    #   py4ai-core
python-dateutil==2.8.2
    # via
    #   -c requirements/requirements_dev.txt
    #   pandas
    #   pinecone-client
python-multipart==0.0.6
    # via
    #   -c requirements/requirements_dev.txt
    #   -r subset.in
pytz==2023.3
    # via
    #   -c requirements/requirements_dev.txt
    #   cfg-load
    #   mpu
    #   pandas
pyyaml==6.0.1
    # via
    #   -c requirements/requirements_dev.txt
    #   cfg-load
    #   huggingface-hub
    #   langchain
    #   pinecone-client
    #   transformers
regex==2023.6.3
    # via
    #   -c requirements/requirements_dev.txt
    #   nltk
    #   transformers
requests==2.31.0
    # via
    #   -c requirements/requirements_dev.txt
    #   cfg-load
    #   huggingface-hub
    #   langchain
    #   langsmith
    #   pinecone-client
    #   torchvision
    #   transformers
safetensors==0.3.1
    # via
    #   -c requirements/requirements_dev.txt
    #   transformers
scikit-learn==1.3.0
    # via
    #   -c requirements/requirements_dev.txt
    #   sentence-transformers
scipy==1.11.1
    # via
    #   -c requirements/requirements_dev.txt
    #   py4ai-core
    #   scikit-learn
    #   sentence-transformers
sentence-transformers==2.2.2
    # via
    #   -c requirements/requirements_dev.txt
    #   -r subset.in
sentencepiece==0.1.99
    # via
    #   -c requirements/requirements_dev.txt
    #   sentence-transformers
six==1.16.0
    # via
    #   -c requirements/requirements_dev.txt
    #   cfg-load
    #   python-dateutil
sniffio==1.3.0
    # via
    #   -c requirements/requirements_dev.txt
    #   anyio
sqlalchemy==1.4.49
    # via
    #   -c requirements/requirements_dev.txt
    #   fastapi-utils
    #   langchain
starlette==0.27.0
    # via
    #   -c requirements/requirements_dev.txt
    #   fastapi
sympy==1.12
    # via
    #   -c requirements/requirements_dev.txt
    #   torch
tenacity==8.2.2
    # via
    #   -c requirements/requirements_dev.txt
    #   langchain
threadpoolctl==3.2.0
    # via
    #   -c requirements/requirements_dev.txt
    #   scikit-learn
tokenizers==0.13.3
    # via
    #   -c requirements/requirements_dev.txt
    #   transformers
tomli==2.0.1
    # via
    #   -c requirements/requirements_dev.txt
    #   -r subset.in
    #   hexagonal-core
    #   hexagonal-repository-core
    #   hexagonal-repository-langchain
    #   hexagonal-repository-pinecone
    #   microservices-core
    #   microservices-hexagonal
    #   py4ai-core
torch==2.0.1
    # via
    #   -c requirements/requirements_dev.txt
    #   sentence-transformers
    #   torchvision
torchvision==0.15.2
    # via
    #   -c requirements/requirements_dev.txt
    #   sentence-transformers
tqdm==4.65.0
    # via
    #   -c requirements/requirements_dev.txt
    #   huggingface-hub
    #   nltk
    #   pinecone-client
    #   sentence-transformers
    #   transformers
transformers==4.31.0
    # via
    #   -c requirements/requirements_dev.txt
    #   sentence-transformers
typing-extensions==4.7.1
    # via
    #   -c requirements/requirements_dev.txt
    #   -r subset.in
    #   fastapi
    #   hexagonal-core
    #   hexagonal-repository-core
    #   hexagonal-repository-langchain
    #   hexagonal-repository-pinecone
    #   huggingface-hub
    #   microservices-core
    #   microservices-hexagonal
    #   pinecone-client
    #   py4ai-core
    #   pydantic
    #   torch
    #   typing-inspect
    #   uvicorn
typing-inspect==0.9.0
    # via
    #   -c requirements/requirements_dev.txt
    #   dataclasses-json
tzdata==2023.3
    # via
    #   -c requirements/requirements_dev.txt
    #   pandas
tzlocal==5.0.1
    # via
    #   -c requirements/requirements_dev.txt
    #   mpu
urllib3==2.0.4
    # via
    #   -c requirements/requirements_dev.txt
    #   pinecone-client
    #   requests
uvicorn==0.23.1
    # via
    #   -c requirements/requirements_dev.txt
    #   microservices-core
    #   microservices-hexagonal
wrapt==1.15.0
    # via
    #   -c requirements/requirements_dev.txt
    #   deprecated
yarl==1.9.2
    # via
    #   -c requirements/requirements_dev.txt
    #   aiohttp

# The following packages are considered to be unsafe in a requirements file:
# setuptools

the current code snippet at lines 89-90 of licensecheck/get_deps.py

	for req in reqPath.read_text("utf-8").strip().split("\n"):
		reqs.add(resolveReq(req))

fails to resolve any requirement due to the new definition of the function resolveReq at line 35 of the same file:

resolveReq = lambda req: pkg_resources.Requirement.parse(req).project_name.lower()

while the old implementation:

with open(reqPath, encoding="utf-8") as requirementsTxt:
		for req in requirements.parse(requirementsTxt):
			reqs.add(str(req.name).lower())

worked correctly.

Feature: Automated parsing of licence in setup.cfg file

As of now, only pyproject.toml files are parsed by packageinfo.getMyPackageLicense. Since setup.cfg files are almost as common and as easy to parse, I was wondering if it weren't possibile to automatically parse them too.

My idea would be to modify the packageinfo.getMyPackageLicense function along these lines:

import configparser

def getMyPackageLicense() -> str:
	"""Get the pyproject data.
	Returns:
		str: license name
	Raises:
		RuntimeError: Must specify a license using license spdx or classifier (tool.poetry or tool.flit)
	"""
        if os.path.exists("pyproject.toml"):
            pyproject = tomli.loads(Path("pyproject.toml").read_text(encoding="utf-8"))
	    tool = pyproject["tool"]
	    metaData = {"classifiers": [], "license": ""}
	    if "poetry" in tool:
                 metaData = tool["poetry"]
	    elif "flit" in tool:
	        metaData = tool["flit"]["metadata"]
	   else:
	        return input("Enter the project license")
	   licenseClassifier = licenseFromClassifierlist(metaData.get("classifiers", []))
	   if licenseClassifier != UNKNOWN:
	       return licenseClassifier
	   if "license" in metaData:
	       return str(metaData["license"])
	   raise RuntimeError(
		"Must specify a license using license spdx or classifier (tool.poetry or tool.flit)"
	   )
     elif os.path.exists("setup.cfg"):
         config = configparser.ConfigParser()
         _ = config.read('setup.cfg')
         if "license" in config["metadata"]:
	     return str(config["metadata"]["license"])
         else :
             return input("Enter the project license ") 
     else:
         return input("Enter the project license ")

I am not an expert but, up to my knowledge, there is no way to configure poetry or flit using setup.cfg files, thus I skipped that part in my proposal.

Feature: Support for proprietary license

Feature

I have read the comment above and have completed each step
I have filled in each heading below

Is your feature request related to a problem? Please describe

Currently when a proprietary license is defined, for example classifiers contains:
License :: Other/Proprietary License
The licensecheck results with unhelpful error

KeyError: <License.NO_LICENSE: 200

Describe the solution you'd like

Ideally licensecheck should just list licenses used by dependencies and mark licenses blacklisted in configuration.

Describe alternatives you've considered

As a workaround I can mark my project that it uses MIT license and get the output, but that forces me to provide false information in my project.

Additional context

Bug: docutils license is not parsed as BSD

Bug

I have read the comment above and have completed each step
I have filled out the system info
I have described the bug, filled in the expected outcome and the actual
outcome including screenshots where appropriate

System info

OS: Linux LAPTOP-VQJVCNH4 5.15.90.1-microsoft-standard-WSL2
Version: 2023.1.1

Describe the bug

The theme project depends directly on docutils (it's the base of every sphinx related stuff) and it raises an incompatibility issue as it's not identified correctly.

Expected outcome

list of packages                                
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Compatible ┃ Package               ┃ License(s)                              ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ✔          │ docutils              │ BSD License                             │
└────────────┴───────────────────────┴─────────────────────────────────────────┘

Actual outcome

list of packages                                
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Compatible ┃ Package               ┃ License(s)                              ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ✖          │ docutils              │ BSD License;; GNU General Public        │
│            │                       │ License (GPL);; Python Software         │
│            │                       │ Foundation License;; Public Domain      │
└────────────┴───────────────────────┴─────────────────────────────────────────┘

It'd be nice to move the compatibility matrix into another package.

Feature

I have read the comment above and have completed each step
I have filled in each heading below

Is your feature request related to a problem? Please describe

I don't need the most of your machinery, I gonna make an own one. But I don't want to maintain an own license compatibility database.

Describe the solution you'd like

The db to be detached from the machinery.

Describe alternatives you've considered

Cooperate and redesign this package according to my ideas. But it would likely take long.

Additional context

I am trying to make a library computing a license of a piece of software automatically combined of other pieces of software with own licenses.

Bug: Not able to ignore packages using pyproject.toml configuration

Hi, I have the following configuration in my project for licensecheck tool, but even if I try to ignore pylint, the licensecheck will still mark it as non compatible.

[tool.licensecheck]
format = "simple"
ignore_packages = ["pylint", "hypothesis"]
zero = true

I would expect the licensecheck to force compatibility for those two packages and return a 0 value.

Otherwise, I truly appreciate your tool 💪 .

Bug: Classifier and License Mismatch

TLDR: the license classifier overrides the license which can cause confusion if the values differ

Bug

I have read the comment above and have completed each step
I have filled out the system info {N/A}
I have described the bug, filled in the expected outcome and the actual outcome including screenshots where appropriate

Example

poetry new test_licensecheck
cd test_licensecheck
poetry add licensecheck

Manually edit the toml file with a license and classifier as necessary

[tool.poetry]
license = "MIT License"
classifiers = [
	"License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)",
]

Expected outcome

It might be beneficial to have a check that prints a warning if the selected classifier differs from the license

> poetry run licensecheck
Warning: license classifier "GPLv3+" differs from license "MIT License". Using "GPLv3+"

┌──────────┬────────────────────┬────────────────────┐
│Compatible│Package             │License             │
├──────────┼────────────────────┼────────────────────┤
│True      │add-trailing-comma  │MIT License         │
│True      │appdirs             │MIT License         │

Actual outcome

Right now, the classifier overrides the license, which is confusing if they diverged and licensecheck only reports true/false. In the above example, you might expect all GPL license to be non-compatible because the license is set to MIT

LicenseCheck/licensecheck/packageinfo.py

Lines 195 to 198 in 42513cf

    
           licenseClassifier = licenseFromClassifierlist(metaData["classifiers"]) # type: ignore yapf: disable 
        
           if licenseClassifier != LICENSE_UNKNOWN: 
        
           	return licenseClassifier 
        
           return str(metaData["license"])

Feature: Add tests

As I see - this tool hasn't any tests. What do you think about adding tests for this tool?

Bug: licensecheck gives Indexerror: list index out of range when trying to run

Bug

[x ] I have read the comment above and have completed each step
[x ] I have filled out the system info
[x ] I have described the bug, filled in the expected outcome and the actual
outcome including screenshots where appropriate

System info

OS: Windows
Browser: Chrome
Version:

Describe the bug

When trying to run licensecheck after installing and installing poetry the module crashes after call:

(venv311) PS C:\Users\steve\PycharmProjects\Parcival> licensecheck                
An error occurred with poetry, try running 'poetry show' to see what went wrong! - (fall back to requirements)
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "C:\Users\steve\PycharmProjects\Parcival\venv311\Scripts\licensecheck.exe\__main__.py", line 7, in <module>
  File "C:\Users\steve\PycharmProjects\Parcival\venv311\Lib\site-packages\licensecheck\__init__.py", line 90, in cli
    myLice, depsWithLicenses = get_deps.getDepsWithLicenses(
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\steve\PycharmProjects\Parcival\venv311\Lib\site-packages\licensecheck\get_deps.py", line 130, in getDepsWithLicenses
    myLice = license_matrix.licenseType(myLiceTxt)[0]
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

Tryin to run 'poetry show' as recommendet i get:

(venv311) PS C:\Users\steve\PycharmProjects\Parcival> poetry show

Poetry could not find a pyproject.toml file in C:\Users\steve\PycharmProjects\Parcival or its parents

falling back to requirements as hinted also prompts the same error

(venv311) PS C:\Users\steve\PycharmProjects\Parcival> licensecheck -u requirements
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "C:\Users\steve\PycharmProjects\Parcival\venv311\Scripts\licensecheck.exe\__main__.py", line 7, in <module>
  File "C:\Users\steve\PycharmProjects\Parcival\venv311\Lib\site-packages\licensecheck\__init__.py", line 90, in cli
    myLice, depsWithLicenses = get_deps.getDepsWithLicenses(
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\steve\PycharmProjects\Parcival\venv311\Lib\site-packages\licensecheck\get_deps.py", line 130, in getDepsWithLicenses
    myLice = license_matrix.licenseType(myLiceTxt)[0]
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

Expected outcome

Getting the list of licenses or a more helpful ErrorMessage

Actual outcome

An IndexError crashing the module

Bug: Flesh out warning messages

Bug

I have read the comment above and have completed each step
I have filled out the system info
I have described the bug, filled in the expected outcome and the actual
outcome including screenshots where appropriate

System info

OS: Windows 10
Browser: NA
Version: 2022.1

Describe the bug

Warning messages can be too ambiguous. e.g. for WARN: License not identified so falling back to NO_LICENSE what license caused the issue? Does it need adding in a feature update

Expected outcome

More detailed warning messages

Actual outcome

As above

Feature: Add config file functionality

I can't find this in docs. Can we write config file (and add path to it, like in flake, pylint, gunicorn or other cli tools instead of flags?

Question: Could you publish a new release with the latest fix(es)

Before You Begin

Before proceeding, please make sure to follow these steps:

I have checked for similar questions in the project's issue tracker to avoid duplicates.
I have searched existing issues to see if this question has been asked before.

Your Question

@FredHappyface I would appreciate if you could publish a new release with the latest fix(es) - this one in particular. TIA !

Bug: packages without PyPI Classifiers can't be properly parsed

Bug

I have read the comment above and have completed each step
I have filled out the system info
I have described the bug, filled in the expected outcome and the actual
outcome including screenshots where appropriate

System info

OS: Ubuntu
Version: 20.04

Describe the bug

Trying to analyse a package that doesn't have any Classifiers on a PyPI results in an error.
Example packages:
https://pypi.org/project/kaleido/
https://pypi.org/project/jsbeautifier/

Error:

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File ".../.venv/lib/python3.8/site-packages/licensecheck/__main__.py", line 6, in <module>
    cli()
  File ".../.venv/lib/python3.8/site-packages/licensecheck/__init__.py", line 90, in cli
    depsWithLicenses = get_deps.getDepsWithLicenses(
  File ".../.venv/lib/python3.8/site-packages/licensecheck/get_deps.py", line 116, in getDepsWithLicenses
    packages = packageinfo.getPackages(reqs)
  File ".../.venv/lib/python3.8/site-packages/licensecheck/packageinfo.py", line 122, in getPackages
    packageinfo.add(getPackageInfoLocal(requirement))
  File ".../.venv/lib/python3.8/site-packages/licensecheck/packageinfo.py", line 34, in getPackageInfoLocal
    lice = licenseFromClassifierlist(pkgMetadata.get_all("Classifier"))
  File ".../.venv/lib/python3.8/site-packages/licensecheck/packageinfo.py", line 102, in licenseFromClassifierlist
    for val in classifiers:
TypeError: 'NoneType' object is not iterable

Expected outcome

When the package has no classifiers, return UNKNOWN license.

Actual outcome

When the package has no classifiers, the error is thrown since the code expects classifierList to be a list.

Bug: PEP631 Mode failure: KeyError: 'tool'

Thank you for creating this tool! I look forward to using it. Hopefully the below bug report is helpful enough for diagnosing.

Before You Begin

Before proceeding, please make sure to follow these steps:

I have checked for similar issues in the project's issue tracker.
I have searched closed issues to see if a similar problem was reported
before.

Issue Details

Trying to use licensecheck with a pyproject.toml file generated by flit.

Description

It seems that licensecheck is expecting the existence of optional fields in the pyproject.toml file.

Here is the project that I cloned (and that I want to check): https://github.com/structuralpython/pfse_starterkit

Steps to reproduce:

Create (and activate) new conda environment (mine was with Python 3.10)
Install cloned package (see above)
pip install licensecheck into environment
Navigate to repository root directory (where pyproject.toml lives)
Run licensecheck --using PEP631

Expected Behavior

Expected licensecheck to run

Actual Behavior

(pfse) PS C:\Users\xxxx\xxxx\examples\pfse_starterkit> licensecheck --using PEP631
Traceback (most recent call last):
  File "C:\Users\xxxx\miniconda3\envs\pfse\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\xxxx\miniconda3\envs\pfse\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\xxxx\miniconda3\envs\pfse\Scripts\licensecheck.exe\__main__.py", line 7, in <module>
    sys.exit(cli())
  File "C:\Users\xxxx\miniconda3\envs\pfse\lib\site-packages\licensecheck\__init__.py", line 76, in cli
    configparser.parseConfigList(
  File "C:\Users\xxxx\miniconda3\envs\pfse\lib\site-packages\fhconfparser\fhconfparser.py", line 75, in parseConfigList
    dispatchers[conf[1]](
  File "C:\Users\xxxx\miniconda3\envs\pfse\lib\site-packages\fhconfparser\fhconfparser.py", line 154, in parseToml
    self.data = {**_resolveNamespace(doc, tomlNamespace), **self.data}
  File "C:\Users\xxxx\miniconda3\envs\pfse\lib\site-packages\fhconfparser\fhconfparser.py", line 341, in _resolveNamespace
    doc = doc[part]
KeyError: 'tool'

System Information

Please provide the following additional information about your system or
environment:

Operating System (OS): Windows 10
Project Version (if applicable): 2023.5.1

Feature: Please consider "Zope Public License"

Is your feature request related to a problem? Please describe

I use the library waitress, having a "Zope Public License":

https://github.com/Pylons/waitress
https://github.com/Pylons/waitress/blob/main/LICENSE.txt

licensecheck yields

WARN: 'ZOPE PUBLIC LICENSE' License not identified so falling back to NO_LICENSE

Describe the solution you'd like

Consider the license

Describe alternatives you've considered

Support of a whitelist, so that I can alter the behavior of licensecheck by configuration.

Bug: Packages parsed by importlib as MultiplexedPath cannot have module size read properly.

Bug

I have read the comment above and have completed each step
I have filled out the system info
I have described the bug, filled in the expected outcome and the actual
outcome including screenshots where appropriate

System info

OS: Ubuntu
Version: 20.04

Describe the bug

Package mkdocstrings is interpreted by importlib as an importlib.readers.MultiplexedPath and isn't cast to pathlib.Path type with glob attribute.

Error during evaluation:

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File ".../.venv/lib/python3.8/site-packages/licensecheck/__main__.py", line 6, in <module>
    cli()
  File ".../.venv/lib/python3.8/site-packages/licensecheck/__init__.py", line 90, in cli
    depsWithLicenses = get_deps.getDepsWithLicenses(
  File ".../.venv/lib/python3.8/site-packages/licensecheck/get_deps.py", line 116, in getDepsWithLicenses
    packages = packageinfo.getPackages(reqs)
  File ".../.venv/lib/python3.8/site-packages/licensecheck/packageinfo.py", line 122, in getPackages
    packageinfo.add(getPackageInfoLocal(requirement))
  File ".../.venv/lib/python3.8/site-packages/licensecheck/packageinfo.py", line 46, in getPackageInfoLocal
    size = getModuleSize(cast(Path, packagePath), name)
  File ".../.venv/lib/python3.8/site-packages/licensecheck/packageinfo.py", line 188, in getModuleSize
    for f in path.glob("**/*")
AttributeError: 'MultiplexedPath' object has no attribute 'glob'

Expected outcome

Skip and download info from PyPI site.

Actual outcome

AttributeError is thrown and the code stops execution.

Bug: Apache2 is shown as incompatible with LGPL3 (LGPLV3)

Question

I have read the comment above and have completed each step

My software is under LGPLV3 and I have a few dependencies with Apache License 2.0

When I run licensecheck, the output shows the software under Apache License 2.0 cannot be integrated to a project under LGPLV3.
As far I know, there is no problem : https://en.wikipedia.org/wiki/License_compatibility.

Is it a bug from licensecheck ? Could you provide a document that explains the compatibility between licenses ?

Thanks

Feature: add allow/deny list of licences as cmdopt

As I see - your tool can only find all licences of packages. Is it possible to check according allow/deny lists, to find packages, which I can use or not?

Question: Are the licenses of sub dependencies considered?

According to

https://itnext.io/how-to-detect-unwanted-licenses-in-your-python-project-c78ebdeb51df

"Let's say you want to avoid GPL, if you would just look at your requirements you might miss some. For example, Pyiotools is launched under an MIT license, but it has 4 dependencies with GPL. Meaning you will have to replace those 4 packages before you can use Pyiotools without GPL."

=> Does licensecheck consider the nested requirements or only the top level requirements?

Bug: TypeError due to Typing

Bug

I have read the comment above and have completed each step
I have filled out the system info
I have described the bug, filled in the expected outcome and the actual
outcome including screenshots where appropriate

System info

OS: Linux 5.15.0-52-generic #58~20.04.1-Ubuntu
Browser:
licensecheck Version: 2022.2
python Version: 3.8

Describe the bug

After freshly installing the current version of licensecheck and typing the command licensecheck, I get the following error:

Traceback (most recent call last):
  File "/home/me/myproject/venv/bin/licensecheck", line 5, in <module>
    from licensecheck import cli
  File "/home/me/myproject/venv/lib/python3.8/site-packages/licensecheck/__init__.py", line 13, in <module>
    from licensecheck import formatter, get_deps
  File "/home/me/myproject/venv/lib/python3.8/site-packages/licensecheck/formatter.py", line 50, in <module>
    def ansi(packages: list[PackageCompat]) -> str:
TypeError: 'type' object is not subscriptable

I know this error occurs, when in Python 3.8, you want to use list as type hint, although in 3.8 you still need to use List.

Expected outcome

No error message. :)

Feature: Improve error message No such file or directory: 'requirements.txt'

Feature

Is your feature request related to a problem? Please describe

I got the Error message
No such file or directory: 'requirements.txt'
and found it confusing, because I do not have a requiremnts.txt on purpose and use pyproject.toml.

My underlying issue was, that I was not in the right directory ( forgot to cd back_end).

You might want to adapt the message to something like

"Could not find specification of requirements (requirements.txt or pyproject.toml)."

Bug: Upper case change doesn't work with `None` license

Bug

I have read the comment above and have completed each step
I have filled out the system info
I have described the bug, filled in the expected outcome and the actual
outcome including screenshots where appropriate

System info

OS: Ubuntu
Browser:
Version: 22.04.2

Describe the bug

Pre-commit update wanted to change LicenceCheck in my project from 2023.1.4 to 2023.3.
I have seen that a new uppercase string has been implemented. Additionally, the ModuleNotFound error has been raised, but I'm not sure if it's related.

Here is the log from the execution in the pre-commit hook (Python 3.11).
Execution: https://github.com/srai-lab/srai/actions/runs/5721376383/job/15502877608?pr=269

 During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/.cache/pre-commit/repovx3e8lm_/py_env-python3.11/lib/python3.11/site-packages/licensecheck/packageinfo.py", line 25, in getPackageInfoLocal
    pkgMetadata = metadata.metadata(requirement)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/importlib/metadata/__init__.py", line 998, in metadata
    return Distribution.from_name(distribution_name).metadata
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/importlib/metadata/__init__.py", line 565, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: No package metadata was found for mkdocs-jupyter

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/.cache/pre-commit/repovx3e8lm_/py_env-python3.11/lib/python3.11/site-packages/licensecheck/packageinfo.py", line 109, in getPackages
    packageinfo.add(getPackageInfoLocal(requirement))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.cache/pre-commit/repovx3e8lm_/py_env-python3.11/lib/python3.11/site-packages/licensecheck/packageinfo.py", line 49, in getPackageInfoLocal
    raise ModuleNotFoundError from error
ModuleNotFoundError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/.cache/pre-commit/repovx3e8lm_/py_env-python3.11/bin/licensecheck", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/runner/.cache/pre-commit/repovx3e8lm_/py_env-python3.11/lib/python3.11/site-packages/licensecheck/__init__.py", line 91, in cli
    myLice, depsWithLicenses = get_deps.getDepsWithLicenses(
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.cache/pre-commit/repovx3e8lm_/py_env-python3.11/lib/python3.11/site-packages/licensecheck/get_deps.py", line 161, in getDepsWithLicenses
    packages = packageinfo.getPackages(reqs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.cache/pre-commit/repovx3e8lm_/py_env-python3.11/lib/python3.11/site-packages/licensecheck/packageinfo.py", line 112, in getPackages
    packageinfo.add(getPackageInfoPypi(requirement))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.cache/pre-commit/repovx3e8lm_/py_env-python3.11/lib/python3.11/site-packages/licensecheck/packageinfo.py", line 71, in getPackageInfoPypi
    license=ucstr(licenseClassifier if licenseClassifier != UNKNOWN else info["license"]),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.cache/pre-commit/repovx3e8lm_/py_env-python3.11/lib/python3.11/site-packages/licensecheck/types.py", line 20, in __new__
    return super().__new__(cls, v.upper())
                                ^^^^^^^
AttributeError: 'NoneType' object has no attribute 'upper'

Expected outcome

Have a failure path for None licenses, maybe change the constructor for ucstr.

Actual outcome

__init__ function for ucstr fails with the None object.

Bug: Handle both single and double quotes in extras specification

Before You Begin

Before proceeding, please make sure to follow these steps:

I have checked for similar issues in the project's issue tracker.
I have searched closed issues to see if a similar problem was reported
before.

Issue Details

Description

The current logic only accounts for single quotes instead of handling both ' (single quotes) and " (double quotes) when parsing the extras specification. For example, when parsing the extras for typer, all option extras are also being included by default as a dependency.

Expected Behavior

Contents of my requirements.txt:

typer

When I just specify the single typer package as a sole dependency in requirements.txt, I expect only the core dependencies to be included (i.e. {'TYPING-EXTENSIONS', 'TYPER', 'CLICK'}).

Actual Behavior

All the optional packages are included too (i.e. {'MKDOCS', 'TYPING-EXTENSIONS', 'PYTEST-COV', 'PYTEST-SUGAR', 'COLORAMA', 'MKDOCS-MATERIAL', 'FLAKE8', 'PILLOW', 'MDX-INCLUDE', 'SHELLINGHAM', 'COVERAGE', 'CLICK', 'AUTOFLAKE', 'TYPER', 'MYPY', 'ISORT', 'BLACK', 'RICH', 'PYTEST', 'PRE-COMMIT', 'PYTEST-XDIST', 'CAIROSVG'}).

System Information

Please provide the following additional information about your system or
environment:

Python Version: 3.8.10
Operating System (OS):

Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:        20.04
Codename:       focal

Project Version (if applicable): HEAD of master @ 15cc02a0c636c5e0caac505d6bfcfd70e17871e9

Feature: support pre-commit hook

According
#6
And
#7

if it possible to realize that - can you add your tool as pre-commit hook? For checking requirements licences after requirements changes. Just exit with zero, if OK and with another code and description, if not.
pre-commit hooks example:
https://github.com/pre-commit/pre-commit-hooks/blob/master/.pre-commit-hooks.yaml

	licenseClassifier = licenseFromClassifierlist(metaData["classifiers"]) # type: ignore yapf: disable
	if licenseClassifier != LICENSE_UNKNOWN:
	return licenseClassifier
	return str(metaData["license"])

fhpythonutils / licensecheck Goto Github PK

licensecheck's People

Contributors

Stargazers

Watchers

Forkers

licensecheck's Issues

Is your feature request related to a problem? Please describe

Describe the solution you'd like

Additional context

Bug

System info

Describe the bug

Expected outcome

Actual outcome

Bug

System info

Describe the bug

Expected outcome

Actual outcome

Bug

System info

Describe the bug

Expected outcome

Actual outcome

Bug

System info

Describe the bug

Expected outcome

Actual outcome

Feature

Is your feature request related to a problem? Please describe

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Before You Begin

Issue Details

Expected Behavior

Reproducible example

Bug

System info

Describe the bug

Expected outcome

Actual outcome

Question

Bug

System info

Describe the bug

Expected outcome

Actual outcome

Feature

Is your feature request related to a problem? Please describe

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Bug

System info

Describe the bug

Expected outcome

Actual outcome

Feature

Is your feature request related to a problem? Please describe

Describe the solution you'd like

Describe alternatives you've considered

Bug

System info

Describe the bug

Expected outcome

Actual outcome

Bug

System info

Describe the bug

Expected outcome

Actual outcome

Bug

System info

Describe the bug

Expected outcome

Actual outcome

System info