doccaz / scc-tools Goto Github PK

$ ./vercheck-cache.py
Traceback (most recent call last):
File "./vercheck-cache.py", line 6, in
from platformdirs import user_cache_dir
ModuleNotFoundError: No module named 'platformdirs'

I consider this more an enhancement than a bug.

Can you either

• exclude RPMs that belong to the SUSE Manger Bootstrap repo or
• always fetch SUSE Manager repo on top

Some packages that vercheck claims to be "unsupported" ;-)

sle-manager-tools-release | SUSE Linux Enterprise 12 | 12-10.3.1
sle-manager-tools-release-POOL | SUSE Linux Enterprise 12 | 12-1.20
python2-rhnlib | SUSE Linux Enterprise 12 | 4.3.4-21.43.3
spacewalk-usix | SUSE Linux Enterprise 12 | 4.0.9-3.6.7
supportutils-plugin-salt | SUSE Linux Enterprise 12 | 1.2.0-6.16.1

sle-manager-tools-release | SUSE Linux Enterprise 15 | 15-113.8.1

If vercheck is run against a supportconfig file for a SLE11 system, zero-length report files are written. It should not generate these files as SCC does not support SLE11 queries.

Add a flag to allow the tool to check for versions on the current system.

Since we already download and manage local cached copies of the PINT databases, we should implement parameters/commands to query it on the CLI. It should be a quicker alternative to checking the PINT website.

SUSE PTFs (Program Temporary Fix) packages have been standardized to have a specific "vendor" tag. They've been erroneously categorized as "unsupported" and went to the wrong list.

I'm implementing an extra report/CSV file where they can live, called "suseptf".

HTTP error 429 is "Too many requests". It needs to be added to the retry_states list so it can be properly managed.

Do not emit the package versions while analyzing a supportconfig by default. Instead, create flags to show specific messages.
e.g. --show-unknown, --show-differences, --show-uptodate.

Maybe show a spinner or just the package count while it works?

Analyzing scc_ccd03b001421_210922_1732, vercheck shows an older version of dbus1 than they have installed. Also looks like its displaying the shipping version instead of updates channel:

dbus-1,1.12.2-8.6.1,1.12.2-8.3.1,Basesystem Module 15 SP1 x86_64

I assume the versions reported by repositories that match the "base_regex" have more priority over the other repositories. However, it seems I forgot to check if the version is actually higher than the original version and discard it if otherwise.

Instead of printing out matching types of packages as they're found, output a more easily-readable report.

In some supportconfigs hardware.txt does not exist. Suggest adding a fallback file (basic-environment.txt) to check, or adding a flag to manually specify arch when running with -d against a supportconfig.

could not determine architecture for the supportconfig directory. Please supply one with -a.

maybe -a x86_64 ???

I noticed that single package searches are not updating the local JSON package cache, so for example:

# vercheck.py -p 2468 -n kernel-default
loaded 6532 items from cache (/home/erico/.cache/scc-tools/scc_data.json)
searching for package "kernel-default" in product id "2468" (SUSE Linux Enterprise Desktop 15 SP5 x86_64)
cached data for kernel-default is too old ( -134 days), discarding cache entry
searching for kernel-default for product ID 2468 in SCC
latest version for kernel-default on product ID 2468(SUSE Linux Enterprise Desktop 15 SP5 x86_64) is 5.14.21-150500.55.52.1, found on Basesystem Module (sle-module-basesystem/15.5/x86_64)

will always complain about the old cached data. The full directory analysis seems to update the cache correctly.

Getting "got error 504 from the server" errors and noticed that the script just keeps going with new requests rather than retrying failed ones.
Adding handling for this could be helpful to prevent partial results

It's been reported that lately vercheck.py is taking around 10-12 minutes to complete, and is stuck in the "gathering results..." message for quite a while.

I've noticed that something changed in the SCC infrastructure that's leaving at least "max_threads" connections in the CLOSE_WAIT state until the kernel's tcp_fin_timeout runs out for each one. The default for recent kernels is 60 seconds.

By default vercheck.py writes CSV files to the current directory. Adding a flag to specify the output directory for these files would be helpful for creating automatic checks.

There is a TID (https://www.suse.com/support/kb/doc/?id=000019341) that describes os-release inconsistencies that were found on SLES for SAP. Basically, different CPE string schemes were used on each release.

Coincidentally, my table of product IDs in the code matches all the flavors up until SLES for SAP 12 SP4, but according to the TID, it can still be misleading and we should not rely on just the os-release information, but also check the installed base products:

Even when the base product is SLES for SAP, the contents of the os-release file should never have been used to identify this fact.

SLES for SAP 12 (no service pack): when registered and patched during the build process, had the correct settings in os-release (all indicating SLES, not 'SLES for SAP'). Also incorrectly has two additional product files in /etc/products.d/
SLES for SAP 12 (no service pack): when not registered and patched during build, had the wrong settings in os-release for NAME, VERSION_ID, PRETTY_NAME, ID and CPR_NAME
SLES for SAP 12.1: the NAME, PRETTY_NAME, ID and CPE_NAME fields were all changed (incorrectly) from a 'SLES' identifier to a 'SLES for SAP' identifier. The contents of the products.d directory correctly reflect the only two entries that should be present.
SLES for SAP 12.2: os-release contents followed the same pattern as SLES for SAP 12.1
SLES for SAP 12.3: os-release contents changed the NAME, PRETTY_NAME and ID fields 'back' to SLES only identifiers. CPE_NAME was left stating 'SLES for SAP' (this was an over-sight).
SLES for SAP 12.4: os-release contents followed the same pattern as SLES for SAP 12.3
SLES for SAP 15: os-release contents match the original correct syntax from SLES 12 (no service pack) with CPE_NAME also correctly reflecting SLES. The contents of the products.d directory has changed with this release to reflect the new packaging/module schema.

And indeed, my CPE strings for SLES for SAP 15 and up are incorrect. I'll need to implement an extra check for the base products (present in updates.txt).

As a collateral effect of issue #15 , packages that exist on both the main "Pool" repository and on PackageHub (with different versions!) now come up in the search results.

This leads to the wrong (?) package being selected as the latest version, like in this example for the package python3-hwdata on SUSE Manager 4.2:

2 packages with name containing "python3-hwdata" in SUSE Manager Server 4.2 x86_64
Name	Version	License	Module
python3-hwdata	2.3.5-bp153.1.13	GPL-2.0	SUSE Package Hub 15 SP3
python3-hwdata	2.3.5-1.21	GPL-2.0	SUSE Manager Server Module 4.2

The version comparison class will select 2.3.5-bp153.1.13 over 2.3.5-1.21. Still considering what to do in this case.

I noticed that whenever processing supportconfig data from a SUSE Manager server, the susemanager-* are not found in the repositories. It turns out that I need to use the proper product ID when doing my queries, instead of the base system product ID.

I can determine the product version from the basic-environment.txt with a regex, and from there use the proper ID.

I found following difference in the resulted logs

Analyzing supportconfig directory: scc_ABC-SGS-SP808_xxxxxx
product name = SUSE Linux Enterprise Server 15 SP1 x86_64 (id 1763, x86_64)
found 1171 total packages to check
gathering results...

Done.
writing CSV reports to scc_ABC-SGS-SP808_xxxxxx

notfound: gnome-desktop,2.28.2-0.6.18
different: gnome-desktop-lang,2.28.2-0.6.18,3.26.2-2.13

as package "gnome-desktop-lang" will be build as sub package of "gnome-desktop" i am pretty sure there is somewhere a error (probably parsing or program logic) hidden in vercheck.py

the affected packages according to rpm.txt

gnome-desktop SUSE Linux Enterprise 11 2.28.2-0.6.18
gnome-desktop-lang SUSE Linux Enterprise 11 2.28.2-0.6.18

probably they are leftovers from a upgrade from SLES11 to SLES15

A colleague asked for adding a --version parameter. This is a bit tricky when dealing with GIT, unless I start using proper release tags and using git archive to do so. Even so, it seems things are a wee bit more complicated to have something like the old "$Id$" string in other version control systems.

loaded 16277 items from cache (/home/user/.cache/scc-tools/scc_data.json)
loaded 5 items from cache (/home/user/.cache/scc-tools/public_cloud_amazon.json)

• cached data for amazon is too old, downloading
  -- Downloading data for active images on amazon...
  -- Downloading data for inactive images on amazon...
  -- Downloading data for deprecated images on amazon...
  -- Downloading data for deleted images on amazon...
  got non-fatal reply (502) from server, trying again in 5 seconds
  got non-fatal reply (502) from server, trying again in 5 seconds
  got non-fatal reply (502) from server, trying again in 5 seconds
  got non-fatal reply (502) from server, trying again in 5 seconds
  got non-fatal reply (502) from server, trying again in 5 seconds
  Traceback (most recent call last):
  File "/home/user/Downloads/scc-tools/vercheck.py", line 2006, in
  main()
  File "/home/user/Downloads/scc-tools/vercheck.py", line 1435, in main
  pc = PublicCloudCheck()
  ^^^^^^^^^^^^^^^^^^
  File "/home/user/Downloads/scc-tools/vercheck.py", line 1794, in init
  self.aws_cm = PublicImageCacheManager(provider='amazon')
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/Downloads/scc-tools/vercheck.py", line 1676, in init
  self.cache_data = self.get_image_states(provider)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/Downloads/scc-tools/vercheck.py", line 1778, in get_image_states
  image_data['deleted'] = self.fetch_image_states(provider, 'deleted')
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/Downloads/scc-tools/vercheck.py", line 1769, in fetch_image_states
  return return_data['images']
  ~~~~~~~~~~~^^^^^^^^^^
  KeyError: 'images'

Identify the architecture used in a supportconfig and use it for the searches.

Add another way to specify product IDs, like "sles-15-sp1" or "sles_sap-12-sp4" (using CPE codes for standardization).

After the 100% CPU usage problems, I'll port the code to use concurrent.futures.ThreadPoolExecutor.

References:
https://docs.python.org/3/library/concurrent.futures.html
https://stackoverflow.com/questions/6509261/how-to-use-concurrent-futures-with-timeouts

e20ca18 May have broken some of the comparing, wasn't seeing these before hand.

> vercheck.py -d.
Analyzing supportconfig directory: .
product name = SUSE Linux Enterprise Server 15 SP2 x86_64 (id 1939, x86_64)
found 838 total packages to check
gathering results...    
error comparing: 2.35-7.11.1 and 2.35.1-7.18.1 ([2, 35, '-', 7, 11, 1] and [2, 35, 1, '-', 7, 18, 1])
error comparing: 2.35-7.11.1 and 2.35.1-7.18.1 ([2, 35, '-', 7, 11, 1] and [2, 35, 1, '-', 7, 18, 1])
error comparing: 2.35-7.11.1 and 2.35.1-7.18.1 ([2, 35, '-', 7, 11, 1] and [2, 35, 1, '-', 7, 18, 1])

> grep -E '2.35.1-7.18.1|2.35-7.11.1' rpm.txt 
binutils                            SUSE Linux Enterprise 15            2.35.1-7.18.1
libctf-nobfd0                       SUSE Linux Enterprise 15            2.35.1-7.18.1
libctf0                             SUSE Linux Enterprise 15            2.35.1-7.18.1

Add the product ID codes for ppc64le, s390x and aarch64 to the known product list.

After yet another severe rate-limiting event from the API server, we found a situation where dozens of threads were waiting on the results from the server for hours, and the vercheck process hit a solid 100% CPU until it was killed.

Apparently, Python threading doesn't yield idle time back to the CPU properly. Will have to research that, or maybe go for the multiprocessing library as an alternative.

Hi,

according to https://docs.python.org/3/library/json.html#json.loads needs explicit encoding on older Python 3:
Changed in version 3.6: s can now be of type bytes or bytearray. The input encoding should be UTF-8, UTF-16 or UTF-32.

We applied this to get it working internally:

--- vercheck.py.old     2021-08-30 13:55:05.124482403 +0200
+++ vercheck.py 2021-08-30 13:15:28.445376243 +0200
@@ -478,11 +478,11 @@
                        if r.status == 200:
                                if tries > 0:
                                        print('thread %d got a good reply after %d tries' % (self.instance_nr, tries))
-                               return_data = json.loads(r.data)
+                               return_data = json.loads(r.data.decode('utf-8'))
                                valid_response = True
                        elif r.status in self.error_states:
                                if r.data:
-                                       json_data = json.loads(r.data)
+                                       json_data = json.loads(r.data.decode('utf-8'))
                                        print('cannot be processed due to error: [' + json_data['error'] + ']')
                                print('thread %d got a fatal error (%d). Results will be incomplete!\nPlease contact the service administrators or try again later.' % (self.instance_nr, r.status))
                                break

As reported by Ralph Roth:

ralph at p5520 in ~/Documents/Eigene_TechDox/HealthCheck/scc-tools on master [?]
$ ll
total 128K
drwxr-xr-x 3 ralph users  193 25. Nov 16:41 ./
drwxr-xr-x 3 ralph users 4,0K 25. Nov 15:57 ../
drwxr-xr-x 8 ralph users  163 25. Nov 15:56 .git/
-rw-r--r-- 1 ralph users 1,1K 25. Nov 15:56 LICENSE
-rw-r--r-- 1 ralph users 6,6K 25. Nov 15:56 README.md
-rw-r--r-- 1 ralph users  18K 25. Nov 16:41 vercheck-different-.csv
-rw-r--r-- 1 ralph users 2,2K 25. Nov 16:41 vercheck-notfound-.csv
-rw-r--r-- 1 ralph users  448 25. Nov 16:41 vercheck-unsupported-.csv
-rw-r--r-- 1 ralph users  45K 25. Nov 16:41 vercheck-uptodate-.csv
-rwxr-xr-x 1 ralph users  34K 25. Nov 15:56 vercheck.py*

The script was called with:

$ ./vercheck.py -d /home/ralph/Desktop/.../unpacked/scc_czdcpv-uxnfs204_211101_1146/

Seeing the release date of the package would be important to see at least for me. For instance, I wanted to see if a new version of suse-migration-sle15-activation was released but while I see the version number, I then need to go into the SCC to find out the release date.

❯ vercheck.py -v -p 1755 -n suse-migration-sle15-activation
Using product ID 1755 (SUSE Linux Enterprise Server for SAP 12 SP4 x86_64)
looking for suse-migration-sle15-activation on product id 1755
version 1.2.0-6.8.1 is available on repository [Public Cloud Module 12 x86_64]
version 1.2.0-6.5.1 is available on repository [Public Cloud Module 12 x86_64]
version 1.2.0-6.11.3 is available on repository [Public Cloud Module 12 x86_64]
latest version for suse-migration-sle15-activation is 1.2.0-6.11.3

When trying to analyze the supportconfig report dir (vercheck-nocache.py -d scc...) appear bulk of messages:
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)

Due to the potential amount of small writes with multiple instances, the current cache implementation (plain JSON file) only reads the cache at the startup, modifies it in-memory, then writes it at the end. This is not optimal, and as a collateral effect "only the last one wins". With an SQLite backend, I hope to mitigate this.

Curiously, the current implementation has the thread locking working fine and there have been no data corruptions, but SQLite would also help with that too.