perfume-bench uses pyup.io in its description, which causes it to be fetched, slowing it down.

#18 blocked incorrect repo resolution, so many now return no repository because none can be found.

MicrosoftDocs/azure-docs#51067 tried to get the public metadata to be better, but that fizzled.

There are too many azure packages spread across lots of repo to 'work around' the problem.

Upstream needs to be fixed.

The only package in the top 360 which depends on github or rtd tokens is jupyter. Need to find some fallback if those tokens are missing so that the top 360 can be used without any tokens.

fedora-python/portingdb#674 (comment)

That probably also works for openSUSE, but the current openSUSE approach is lower bandwidth and easily allows for subsets, however it doesnt include libraries which are not in the python projects, such as Application:Geo

Package misspellings is causing requests to *.travis-ci.org - there is nothing useful which can be found there.

Blocks usually occur in _caching.py

Also there are basic handlers for travis-ci.org in _scm_url_cleaner.py - they could return False to block network requests.

Spawned from #9:

Probably using raw bypasses restrictions on API use without tokens, however it might only buy a bit more access, so should be a failover.

pypillowfight & paperwork-backend & pyocr fail during the tests (not in the library):

tests/utils.py:268: in _test_names
    r = web_session.get(url, timeout=get_timeout(url))
...
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='gitlab.gnome.org', port=443): Max retries exceeded with url: /World/OpenPaperwork/libpillowfight (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f97179c4640>, 'Connection to gitlab.gnome.org timed out. (connect timeout=15)'))

The timeouts for known SCM should be much higher.

A problem with higher timeouts is the test runner also has a limit, and that isnt flexible. pygtk encountered this.

openSUSE package names need to be exactly matched in RPM .spec. Given they are being matched to PyPI names, it is worth also building a dataset of which package names are not using the same name as PyPI, even if they are normalised as the same name. e.g. different capitalisation, or different _, - and ..

Very frequent use of the same RTD, and http://www.sanhehu.org/ returns a lot of junk urls, especially static assets mostly fonts.

#40 (comment)

https://freedesktop.org/wiki/Software/pyxdg/ says http://cgit.freedesktop.org/xdg/pyxdg/ is the main repository, but https://github.com/takluyver/pyxdg is the mirror.

e.g. mwlib.ext

https://www.reportlab.org & http://www.reportlab.org fail.

ERROR    https_everywhere.adapter:adapter.py:124 handle_error requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='www.reportlab.org', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f097d74bf40>, 'Connection to www.reportlab.org timed out. (connect timeout=15)'))
ERROR    https_everywhere.adapter:adapter.py:124 handle_error requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='www.reportlab.org', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f097d74bf40>, 'Connection to www.reportlab.org timed out. (connect timeout=15)'))
ERROR    https_everywhere.adapter:adapter.py:124 handle_error requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='www.reportlab.org', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f097d74bf40>, 'Connection to www.reportlab.org timed out. (connect timeout=15)'))
ERROR    https_everywhere.adapter:adapter.py:124 handle_error requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='www.reportlab.org', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f097d74bf40>, 'Connection to www.reportlab.org timed out. (connect timeout=15)'))
ERROR    https_everywhere.adapter:adapter.py:124 handle_error requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='www.reportlab.org', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f097d74bf40>, 'Connection to www.reportlab.org timed out. (connect timeout=15)'))
ERROR    https_everywhere.adapter:adapter.py:124 handle_error requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='www.reportlab.org', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f097d74bf40>, 'Connection to www.reportlab.org timed out. (connect timeout=15)'))
WARNING  pypidb._pypi:_pypi.py:459 http://www.reportlab.org: HTTPSConnectionPool(host='www.reportlab.org', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f097d74bf40>, 'Connection to www.reportlab.org timed out. (connect timeout=15)'))

There is a lot of links to https://www.mediawiki.org/wiki/Special:ExtensionDistributor/Collection , which has no chance of helping.

They also load http://blog.pediapress.com/

timeout occurs in rtd, but it is after the rtd has resolved already.

pypidb/_rtd.py:30: in __init__
    token = _get_token("readthedocs.io")

gertty currently causes log

ERROR    https_everywhere.adapter:adapter.py:124 handle_error requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='ttygroup.org', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb191ddc580>, 'Connection to ttygroup.org timed out. (connect timeout=15)'))
ERROR    https_everywhere.adapter:adapter.py:124 handle_error requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='ttygroup.org', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb191ddc580>, 'Connection to ttygroup.org timed out. (connect timeout=15)'))

While it is useful for the library to be able to skip past those problems, the tests should highlight them so they can be raised upstream.

Another more critical one would be TooManyRedirects.

https://www.thevirtualbrain.org causes

ERROR https_everywhere.adapter:adapter.py:126 handle_error requests.exceptions.SSLError: HTTPSConnectionPool(host='www.thevirtualbrain.org', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

But http redirects to https, so we cant downgrade it otherwise there is too many redirects, and causes a timeout on https://ci.appveyor.com/project/jayvdb/pypidb/builds/31795479.

the-virtual-brain/tvb-gdist#24 already solves this for that package.

The maximum number of redirects should also be reduced.

Python 2.7 has lots of failures. While Python 2.7 support will only be minimal, the tox py27 should succeed, skipping any aspects which are not supported.

GitLab repos often have a trailing slash, and others

TestTopFirstThousandTail has two expected failures for azure-*

There are also many more in tests/data.py

yoyo has no files.

yoyo-migrations has a 403 on http://data-vocabulary.org/Breadcrumb , and fetches http://bitbucket.org/ and https://bitbucket.org/linuxnotes/ and then other bitbucket site urls like #23.

django-hijack-admin is handled with a rule; however django-hijack doesnt return a result.

https://www.saddi.com/software/flup is

403 Forbidden
You don't have permission to access this resource.

https://www.saddi.com/software/flup/ is ok

In addition to #18 and #19

• "marshmallow-enum", # no urls
• "requestsexceptions", # c.f. test_openstack.py

Commercial:

• "google-cloud-dataflow", # no urls
• "newrelic", # no urls
• "spotinst-agent", # spotinst/spotinst-sdk-python#65

Proprietary:

• "intel-openmp",
• "nvidia-ml-py",

A lot of URLs are hidden rtd sites. Currently they are not processed as rtd unless they are using readthedocs.io/org/etc

Current project is pygal, i.e. Kozea/pygal#493
www.pygal.org is one of these hidden rtd sites.

Occurring in rv = self._check_github_repo(slug):

requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

This should be a skip, or retry.

https://pypi.org/project/uuid/ when http://zesty.ca/python/ fails, esp http://zesty.ca/python/uuid.html which is really slow

INFO     pypidb._pypi:_pypi.py:507 looking up uuid
INFO     pypidb._cache:_cache.py:82 domain http://zesty.ca/python/ lookup error: -11

It doesnt appear in @zestyping 's https://github.com/zestyping?tab=repositories

And https://zesty.ca/python/ (https) always fails.

https://github.com/search?q=%22Get+the+hardware+address+on+Windows+by+running+ipconfig.exe%22&type=Code shows lots of copies of it.

https://github.com/python/cpython/blob/master/Lib/uuid.py is very similar - I believe this package was the basis of the cpython stdlib.

This only occurred once
https://cirrus-ci.com/task/4782984914010112?command=main#L449

self = <tests.test_google_code.TestRedirectGitHub testMethod=test_protorpc>
    def test_protorpc(self):
        url = self.converter.get_vcs("protorpc")
>       self.assertInsensitiveEqual(url, "https://github.com/google/protorpc")
tests/test_google_code.py:157: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/utils.py:72: in assertInsensitiveEqual
    self.assertEqual(s.lower(), s2.lower())
E   AssertionError: 'https://github.com/xzer/run-jetty-run' != 'https://github.com/google/protorpc'
E   - https://github.com/xzer/run-jetty-run
E   + https://github.com/google/protorpc
------------------------------ Captured log call -------------------------------
INFO     pypidb._pypi:_pypi.py:507 looking up protorpc
INFO     pypidb._pypi:_pypi.py:351 protorpc: from None added urls ['https://github.com/xzer/run-jetty-run']
INFO     pypidb._rules:_rules.py:854 email split: rafek @ google.com
INFO     pypidb._pypi:_pypi.py:435 r https://pypi.org/project/protorpc/0.12.0/
WARNING  pypidb._pypi:_pypi.py:446 https://pypi.org/project/protorpc/0.12.0/: 500 Server Error: HTTPS Everywhere for url: https://pypi.org/project/protorpc/0.12.0/
INFO     pypidb._pypi:_pypi.py:435 r https://pypi.org/project/protorpc/
WARNING  pypidb._pypi:_pypi.py:446 https://pypi.org/project/protorpc/: 500 Server Error: HTTPS Everywhere for url: https://pypi.org/project/protorpc/
INFO     pypidb._db:_db.py:71 Adding mapping protorpc = https://github.com/xzer/run-jetty-run

In addition to #20

• "bindep" (opendev)
• "clearbit"
• "comet-ml"
• "dbus-python" (fedora)
• "dm.xmlsec-binding" (fedora iirc)
• "pycuda" (repo is auth protected)
• "scons"

Currently relative URLs are extracted when get_html_hrefs or _url_extract_both is used in a rule in https://github.com/jayvdb/pypidb/blob/master/pypidb/_rules.py , instead of using the default _url_extractor_wrapper.

Some of these explicit controlling rule might be removed by always doing urlextract first, and follow with get_html_hrefs if there is no useful values from urlextract.

check_repo could use raw instead, and tests could be marked as skipped if the error was due to no token.

There are 80 entries in tests/data.py "missing_repos". These need debugging. Creating issues about what is occurring is a helpful contribution; fixing them is even better.

While rummaging around the internet looking for the repo for package-a, very often a package-b repo is found, and can be reliably verified, and it should be stored as a mapping.

This would remove the need for many of the rules which preload packages that are known to cause incorrect mappings.

The most reliable way to verify a repo->PyPI is to look for a link to PyPI in the readme.

A readme analysis which might be useful is https://github.com/austinhle/readme-analysis - I havent looked into it yet.

https://pypi.org/project/plaidml/ - it is Intel, but says it uses Apache license.

data.py says:

"plaidml-keras": "https://github.com/plaidml/plaidml"
...
setuppy_mismatches = [
    ...
    "plaidml",
    ...
]

https://github.com/plaidml/plaidml is mono-repo, and thus needs to be added to the whitelist in db

https://logreduce.softwarefactory-project.io/ has disappeared.

https://softwarefactory-project.io/cgit/ also doesnt mention logreduce.

@TristanCacqueray / @sf-ci , I see you are the maintainer of https://pypi.org/project/logreduce/; do you know what has happened?

The caches should be kept, and re-used to speed up future runs.

There are three packages mysql-connector-* listed in test_top.py 's expected_failures.

• "mysql-connector-python-rf",
• "mysql-connector",
• "mysql-connector-python",

raw.githubusercontent.com is one of the GitHub handlers, however it doesnt cater for wiki URLs

https://raw.githubusercontent.com/wiki/seb-m/pyinotify/Recent-Developments.md is valid

• mdx_gh_links -> (detected ok) https://github.com/Python-Markdown/github-links/
• gilt (invalid package name) -> python-gilt
• archinfo -> https://pypi.org/project/archinfo/ - no urls. Maintainers match https://github.com/angr/archinfo/graphs/contributors

Currently we use URLExtract find_urls which collects all urls, whereas we only want to process some of them until we have a valid result. Using gen_urls would allow reducing the amount of work done in URLExtract, which would be especially helpful for msgpack-python.

However, currently the algorithm also processes all of the urlextract results in order to measure the distance of them all to find the 'best' one. That needs to be altered to know when a best url has been reached even when more urls are yet to be found. That can be easily done by using the github repo checker, and other scm checkers, rather than relying on the guessing approach.

It is resolving to https://github.com/maxmind/libmaxminddb , but it should be https://github.com/maxmind/MaxMind-DB-Reader-python

This was exposed by removing the "language = Python" filter on the setuppy check.

https://bitbucket.org/mrabarnett/mrab-regex/issues/123 are being fetched, because the bitbucket SCM picker doesnt block those URLs.

For GitHub, GitLab, and many others, the scm and repo link are the same.

However even for GitHub, the homepage is often different, being a RTD or Github Pages site.

For sourceforge (and BitBucket until June?), they support multiple VCS so the type of VCS and link are important to interact with the repo. This info can be gleaned from the info processed while finding the SCM, but validating that may be more trouble than using a specialised library. This library can at least indicate that the link is only useful for the SCM, and describe which SCM is used by the project.

The most important aspect however is the results which are a webpage, and no SCM can be found, such as pychart. Many tools like rpm spec builders (upt, py2pack, etc) are looking for a homepage only.

Another cohort is trac, which is an SCM, but the code repository isnt readily available.

Also launchpad is often used only as SCM/bug tracker, but isnt being used the code repo.

One way to avoid many of the "harder"/"slower" problems in CI is to require Python 3 compatibility for the test cases being run.

It is reasonable to assume that there are very few Python 3 packages on code.google.com, so that SCM host could also be disabled in order to avoid #6

c.f. timbertson/termstyle#10 for one example

config ends up at https://www.allaboutcookies.org which has a cloudflare captcha

https://pypi.org/search/?q=allaboutcookies
https://github.com/search?q=allaboutcookies+language%3Apython&type=Code

It doesnt look good.
walter-clayton/covidantivirus#20

As this is so common, and so useless, it shouldnt go through the caching process.

amazon-dax-client = https://github.com/awsdocs/amazon-dynamodb-developer-guide

https://cirrus-ci.com/task/5686714953891840?command=main#L144

GeoIP is resolving to Perl repo https://github.com/maxmind/MaxMind-DB

c.f. #12

A CLI list command could iterate through a dataset and print out the SCM as a prettytable with color, or a format requested by the user.