This is a set of analyzers that will retrieve information from build logs.
Consider a log produced by the pip package installer:
We can also use pipenv, see the How it works section
Log file content
Processing /home/macermak/RedHat/aicoe/lab Collecting networkx (from thoth-lab==0.0.3) Collecting requests (from thoth-lab==0.0.3) Using cached https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl Collecting pandas (from thoth-lab==0.0.3) Using cached https://files.pythonhosted.org/packages/e6/de/a0d3defd8f338eaf53ef716e40ef6d6c277c35d50e09b586e170169cdf0d/pandas-0.24.1-cp36-cp36m-manylinux1_x86_64.whl Collecting plotly (from thoth-lab==0.0.3) Using cached https://files.pythonhosted.org/packages/fd/db/003b5cfbc710f4d4982440451185b952269e4080a57ae7e760a2ceb8ce0c/plotly-3.6.1-py2.py3-none-any.whl Collecting decorator>=4.3.0 (from networkx->thoth-lab==0.0.3) Using cached https://files.pythonhosted.org/packages/f1/cd/7c8240007e9716b14679bc217a1baefa4432aa30394f7e2ec40a52b1a708/decorator-4.3.2-py2.py3-none-any.whl Collecting idna<2.9,>=2.5 (from requests->thoth-lab==0.0.3) Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl Collecting urllib3<1.25,>=1.21.1 (from requests->thoth-lab==0.0.3) Using cached https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl Collecting certifi>=2017.4.17 (from requests->thoth-lab==0.0.3) Using cached https://files.pythonhosted.org/packages/9f/e0/accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/certifi-2018.11.29-py2.py3-none-any.whl Collecting chardet<3.1.0,>=3.0.2 (from requests->thoth-lab==0.0.3) Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl Collecting python-dateutil>=2.5.0 (from pandas->thoth-lab==0.0.3) Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl Collecting pytz>=2011k (from pandas->thoth-lab==0.0.3) Using cached https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl Collecting numpy>=1.12.0 (from pandas->thoth-lab==0.0.3) Using cached https://files.pythonhosted.org/packages/f5/bf/4981bcbee43934f0adb8f764a1e70ab0ee5a448f6505bd04a87a2fda2a8b/numpy-1.16.1-cp36-cp36m-manylinux1_x86_64.whl Collecting six (from plotly->thoth-lab==0.0.3) Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl Collecting retrying>=1.3.3 (from plotly->thoth-lab==0.0.3) Collecting nbformat>=4.2 (from plotly->thoth-lab==0.0.3) Using cached https://files.pythonhosted.org/packages/da/27/9a654d2b6cc1eaa517d1c5a4405166c7f6d72f04f6e7eea41855fe808a46/nbformat-4.4.0-py2.py3-none-any.whl Collecting traitlets>=4.1 (from nbformat>=4.2->plotly->thoth-lab==0.0.3) Using cached https://files.pythonhosted.org/packages/93/d6/abcb22de61d78e2fc3959c964628a5771e47e7cc60d53e9342e21ed6cc9a/traitlets-4.3.2-py2.py3-none-any.whl Collecting jupyter-core (from nbformat>=4.2->plotly->thoth-lab==0.0.3) Using cached https://files.pythonhosted.org/packages/1d/44/065d2d7bae7bebc06f1dd70d23c36da8c50c0f08b4236716743d706762a8/jupyter_core-4.4.0-py2.py3-none-any.whl Collecting jsonschema!=2.5.0,>=2.4 (from nbformat>=4.2->plotly->thoth-lab==0.0.3) Using cached https://files.pythonhosted.org/packages/77/de/47e35a97b2b05c2fadbec67d44cfcdcd09b8086951b331d82de90d2912da/jsonschema-2.6.0-py2.py3-none-any.whl Collecting ipython-genutils (from nbformat>=4.2->plotly->thoth-lab==0.0.3) Using cached https://files.pythonhosted.org/packages/fa/bc/9bd3b5c2b4774d5f33b2d544f1460be9df7df2fe42f352135381c347c69a/ipython_genutils-0.2.0-py2.py3-none-any.whl Building wheels for collected packages: thoth-lab Running setup.py bdist_wheel for thoth-lab: started Running setup.py bdist_wheel for thoth-lab: finished with status 'done' Stored in directory: /tmp/pip-ephem-wheel-cache-_4menpyv/wheels/c6/6e/7d/ef751ae03b7d549816c6658c7f8065cb889a8de3f0c2b3ca6c Successfully built thoth-lab selinon 0.1.0rc9 requires celery>=4, which is not installed. thoth-analyzer 0.1.0 requires distro, which is not installed. thoth-storages 0.9.5 requires aiogremlin==3.2.6rc1, which is not installed. tensorflow-tensorboard 1.5.1 has requirement bleach==1.5.0, but you'll have bleach 2.1.4 which is incompatible. tensorflow-tensorboard 1.5.1 has requirement html5lib==0.9999999, but you'll have html5lib 1.0.1 which is incompatible. tensorboard 1.6.0 has requirement bleach==1.5.0, but you'll have bleach 2.1.4 which is incompatible. tensorboard 1.6.0 has requirement html5lib==0.9999999, but you'll have html5lib 1.0.1 which is incompatible. notebook 5.7.0 has requirement jupyter-client>=5.2.0, but you'll have jupyter-client 5.1.0 which is incompatible. notebook 5.7.0 has requirement pyzmq>=17, but you'll have pyzmq 16.0.3 which is incompatible. kaggle 1.4.7.1 has requirement urllib3<1.23.0,>=1.15, but you'll have urllib3 1.24.1 which is incompatible. botocore 1.11.3 has requirement urllib3<1.24,>=1.20, but you'll have urllib3 1.24.1 which is incompatible. awscli 1.14.25 has requirement botocore==1.8.29, but you'll have botocore 1.11.3 which is incompatible. awscli 1.14.25 has requirement colorama<=0.3.7,>=0.2.5, but you'll have colorama 0.3.9 which is incompatible. awscli 1.14.25 has requirement PyYAML<=3.12,>=3.10, but you'll have pyyaml 3.13 which is incompatible. thoth-storages 0.9.5 has requirement yarl<1.2, but you'll have yarl 1.3.0 which is incompatible. spyder-kernels 0.2.6 has requirement jupyter-client>=5.2.3, but you'll have jupyter-client 5.1.0 which is incompatible. spyder-kernels 0.2.6 has requirement pyzmq>=17, but you'll have pyzmq 16.0.3 which is incompatible. rstcheck 2.2 has requirement sphinx<1.5,>=1.3, but you'll have sphinx 1.8.1 which is incompatible. gremlinpython 3.2.6 has requirement six==1.10.0, but you'll have six 1.12.0 which is incompatible. gremlinpython 3.2.6 has requirement tornado==4.4.1, but you'll have tornado 4.5.2 which is incompatible. Installing collected packages: decorator, networkx, idna, urllib3, certifi, chardet, requests, six, python-dateutil, pytz, numpy, pandas, retrying, ipython-genutils, traitlets, jupyter-core, jsonschema, nbformat, plotly, thoth-lab Found existing installation: decorator 4.3.2 Uninstalling decorator-4.3.2: Successfully uninstalled decorator-4.3.2 Found existing installation: networkx 2.2 Uninstalling networkx-2.2: Successfully uninstalled networkx-2.2 Found existing installation: idna 2.8 Uninstalling idna-2.8: Successfully uninstalled idna-2.8 Found existing installation: urllib3 1.24.1 Uninstalling urllib3-1.24.1: Successfully uninstalled urllib3-1.24.1 Found existing installation: certifi 2018.11.29 Cannot uninstall 'certifi'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. You are using pip version 18.1, however version 19.0.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command.
In the log file, there is a line 76 which contains the information about the package which broke the installation: certifi
. We want to extract this information. In the most common case, we want to get a report containing the observations about this particular build log.
from thoth.build_analyzers.analysis import build_breaker_report
log: str = ...
report: dict = build_breaker_report(log)
The result is a dictionary which can further be formatted by the convenient build_breaker_format_report
function.
from thoth.build_analyzers.analysis import build_breaker_format_report
print(build_breaker_format_report(report))
And you should see something like this:
Build breaker:
{
"already_satisfied": null,
"source": "thoth-lab",
"target": "certifi",
"version_installed": null,
"version_specified": ">=2017.4.17"
}
Probable reason:
76: Cannot uninstall 'certifi'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
Candidates:
[
"76: Cannot uninstall 'certifi'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall."
]
The logic is quite simple (and also a bit naive). Really, no rocket science, just a lot of string matching.
The approach currently handles logs produced by pip and pipenv package installers. For both of them, there is a set of patterns which, if found in a log, potentially indicate an error. These patterns are strings formatted according to the PEP3101 string formatting syntax and are stored in the resources
folder as pandas DataFrames.
We evaluate the 'goodness' of a match for each line of the log file with respect to the relevant set of patterns and label them based on the given thresholds. Currently, the ERROR
threshold is set to the score of >=0.6 on the scale from 0 through 1.0. Finally, we try to search for package names in the log file to construct a dependency table. We filter the ERROR
s containing a package name and we've got ourselves a list of build breakers.
The package is available on PyPI:
pip install thoth-build-analyzers
And you're good to go ๐
The package can also be used via its Command Line Interface.
Check output of thoth-build-analysis --help
after installing the package for more information.
Author: Marek Cermak [email protected], @AICoE - Project Thoth