wolever / pip2pi Goto Github PK
View Code? Open in Web Editor NEWpip2pi builds a PyPI-compatible package repository from pip requirements
License: Other
pip2pi builds a PyPI-compatible package repository from pip requirements
License: Other
hello, i want to clone a mirror of pip dependency packages. because in i can not connect to internet, when i develop a software. i want to download all of the packages in http://pypi.pubyun.com/simple/ or other website. i hope you can help me! thank you.
I'm trying to use pip2tgz
but it says that the pip
module has no attribute main
.
I tried running python, import pip
and it doesn't have a main module indeed. I believe this is due to the following change in pip:
Traceback of the error:
$ pip2tgz ./build/pip/ -r requirements.tx
Traceback (most recent call last):
File "/home/jjmontes/git/scs-control-3.1/env/bin/pip2tgz", line 11, in <module>
sys.exit(pip2tgz())
File "/home/jjmontes/git/scs-control-3.1/env/local/lib/python2.7/site-packages/libpip2pi/commands.py", line 58, in maintain_cwd_helper
return f(*args, **kwargs)
File "/home/jjmontes/git/scs-control-3.1/env/local/lib/python2.7/site-packages/libpip2pi/commands.py", line 413, in pip2tgz
pip_run_command(['install', '-d', outdir] + argv[2:])
File "/home/jjmontes/git/scs-control-3.1/env/local/lib/python2.7/site-packages/libpip2pi/commands.py", line 158, in pip_run_command
res = pip.main(pip_args)
AttributeError: 'module' object has no attribute 'main'
It seems like pip[0] and bandersnatch[1] are now normalizing package names in the simple index, so for example simple/Django/Django-1.7.tar.gz
becomes simple/dango/Django-1.7.tar.gz
.
As of 0.6.5 pip2pi will include a --normalize-package-names
option (default: disabled) and will issue a deprecation warning when it encounters a non-normalized package name. If no issues are found by about June 2015 --normalize-package-names
will become the default.
Thanks to @orodbhen for initially reporting this: #35. See also: pypa/pip#2136
0: pypa/pip#2001
1: https://bitbucket.org/pypa/bandersnatch/pull-request/7/prefer-normalized-directory-names-but-keep/diff
Running dir2pi (or pip2pi) on Windows results in a OSError: symbolic link priviledge not held.
Steps to reproduce:
D:\>mkdir LocalPyPi
D:\>pip2pi .\LocalPyPi pytest
You are using pip version 6.1.1, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pytest
Downloading pytest-2.8.1-py2.py3-none-any.whl (148kB)
100% |################################| 151kB 1.7MB/s
Saved d:\localpypi\pytest-2.8.1-py2.py3-none-any.whl
Collecting py>=1.4.29 (from pytest)
Downloading py-1.4.30-py2.py3-none-any.whl (81kB)
100% |################################| 86kB 1.1MB/s
Saved d:\localpypi\py-1.4.30-py2.py3-none-any.whl
Collecting colorama (from pytest)
Downloading colorama-0.3.3.tar.gz
Saved d:\localpypi\colorama-0.3.3.tar.gz
Successfully downloaded pytest py colorama
Done. 3 new archives currently saved in 'D:\\LocalPyPi'.
Traceback (most recent call last):
File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\lib\runpy.py", line 170, in _run_module_as_main
"__main__", mod_spec)
File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\Scripts\pip2pi.exe\__main__.py", line 9, in <module>
File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\lib\site-packages\libpip2pi\commands.py", line 510, in pip2pi
res = _dir2pi(option, subcmd_argv)
File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\lib\site-packages\libpip2pi\commands.py", line 305, in _dir2pi
os.symlink(symlink_source, symlink_target)
OSError: symbolic link privilege not held
Running dir2pi:
D:\>dir2pi .\LocalPyPi
Traceback (most recent call last):
File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\lib\runpy.py", line 170, in _run_module_as_main
"__main__", mod_spec)
File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\Scripts\dir2pi.exe\__main__.py", line 9, in <module>
File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\lib\site-packages\libpip2pi\commands.py", line 268, in dir2pi
return _dir2pi(option, argv)
File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\lib\site-packages\libpip2pi\commands.py", line 305, in _dir2pi
os.symlink(symlink_source, symlink_target)
OSError: symbolic link privilege not held
If both packages A and B depends on C, then C will be downloaded twice.
pip2tgz pipdir/ requests
requests-1.2.3.tar.gz on pipdirpip2tgz pipdir/ cachecontrol
dependency requests-1.2.3.tar.gz downloaded twice..
Does this package do anything different from pip's "download" command? Perhaps it was written before "pip download" was a thing? If so, should this package be deprecated in favor of that command?
Some packages have a versioning system like -. and this may be due to the fact that their version starts from 0 and hence does not need to put a zero after the hyphen.
Example:
Traceback (most recent call last):
File "/usr/local/bin/pip2pi", line 11, in
sys.exit(pip2pi())
File "/usr/local/lib/python2.7/site-packages/libpip2pi/commands.py", line 510, in pip2pi
res = _dir2pi(option, subcmd_argv)
File "/usr/local/lib/python2.7/site-packages/libpip2pi/commands.py", line 290, in _dir2pi
pkg_name, pkg_rest = file_to_package(pkg_basename, pkgdir)
File "/usr/local/lib/python2.7/site-packages/libpip2pi/commands.py", line 123, in file_to_package
raise InvalidFilePackageName(file, basedir)
libpip2pi.commands.InvalidFilePackageName: unexpected file name: 'xfl-.06.zip' (not in 'pkg-name-version.xxx' format; found in directory: '/root/pypi_copy/var/www/packGeppers')
https://pip.pypa.io/en/stable/user_guide/#config-file
so:
[easy_install]
# Windows
# index_url = file:///C:/pip2pi/simple/
# Linux
# index_url = file:///home/myusername/.pip2pi/simple/
# Mac
index_url = file:///Users/myusername/.pip2pi/simple/
should be
[easy_install]
# Windows
# index-url = file:///C:/pip2pi/simple/
# Linux
# index-url = file:///home/myusername/.pip2pi/simple/
# Mac
index-url = file:///Users/myusername/.pip2pi/simple/
When running pip2pi with pip 1.4.1 installed, I see
###############################################
## ##
## Due to lack of interest and maintenance, ##
## 'pip bundle' and support for installing ##
## from *.pybundle files is now deprecated, ##
## and will be removed in pip v1.5. ##
## ##
###############################################
Failed to settle the format of bz2
Trying to do pip2tgz packages/ statsmodels gives the error:
ImportError: statsmodels requires patsy. http://patsy.readthedocs.org
It seems like you have to have patsy already installed to successfully use pip2pi on statsmodels. I'm not sure if this is a bug in the statsmodels packaging or in pip2pi
I was trying to create a repo for installing protobuf, which depends on google-apputils. I ran into two different issues: Pip2pi crashes upon trying to add packages that require the 'ez-setup' package, and pip2pi automatically downloads and installs the 'distribute' package when required, but doesn't include it in the repo it creates.
$ pip2pi ../pip_repo/ google-apputils
[...]
import ez_setup
ImportError: No module named ez_setup
$ pip2pi ../pip_repo/ google-apputils ez_setup
Downloading/unpacking google-apputils
File was already downloaded [...]/pip_repo/google-apputils-0.4.1.tar.gz
Running setup.py (path:[...]/google-apputils/setup.py) egg_info for package google-apputils
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.14.tar.gz
Extracting in /tmp/tmpq4H5PI
Now working in /tmp/tmpq4H5PI/distribute-0.6.14
Building a Distribute egg in /tmp/pip_build_USERNAME/google-apputils
/tmp/pip_build_USERNAME/google-apputils/distribute-0.6.14-py2.7.egg
But ../pip_repo
doesn't include the distribute package! This is a problem if I want to make a repo to install those packages on a machine with no internet connection.
pip2pi ./ diskimage-builder
pip2tgz ./ diskimage-builder
dir2pi --normalize-package-names ./
pip install diskimage-builder --index=http://127.0.0.1/pip-packages/
returns: "Collecting diskimage-builder
Could not find a version that satisfies the requirement diskimage-builder (from versions: )
No matching distribution found for diskimage-builder
"
curl -I http://127.0.0.1/pip-packages/simple/diskimage-builder/diskimage-builder-1.21.0-py2.py3-none-any.whl | head -n1
HTTP/1.1 200 OK
It seems like pip install --download
does broken things when dealing with a local file, as documented in pypa/pip#1111
Specifically, pip install --download output/ foo.tar.gz
extracts foo.tar.gz
into the output directory.
I'd like to stop using my private build, since you merged my symlink pull request several weeks ago.
When using this tool there is no easy way currently to block pip from downloading python wheel packages when using pip2pi
tool.
Pip have the following flag that can be used when called
--no-use-wheel Do not Find and prefer wheel archives when searching indexes and find-links locations.
It would be great if this tool had this option as either a flag to pip2pi
or perhaps as a environment variable that can be used NO_WHEEL=true pip2pi ...
or something like that.
Hi Recently pip has release its version 8.1.2 which strictly not allow the package names with . like oslo.config . like pypa/pip#3666 . So I have local repository deployed in network which is using pip2pi and I am getting issues installing packages and installing deps with it. That issue was with devpi too but they have now solved it will look at this issue please?
pip
will try to lookup wheels in lower case URLs, while pip2pi
will generate in whatever case the package name has. One example of this is PyYAML
, which will not be found in a repo generated by pip2pi
. The workaround is trivial, just slap on a script afterwards that lower-cases the directories, and updates index.html.
With this command
pip2pi C:\HTdocs\Fileserver\Programming\Python\pypi_pkgs -n pep8
pip2pi interprets the target as remote target and tries to use rsync which fails. The problem is how a remote target is identified, namely by if ':' in target:
. I solved this by changing the line to if re.match('.*\..*:', target):
. Since I do not know much about regular expressions, I am not sure if this is a good solution.
Another point: Since rsync is not available on Windows by default, a note about rsync in the requirements section in readme.rst would be helpful.
When installing something like dogpile.cache or backports.shutil-get-terminal-size pip looks in directories like simple/dogpile-cache/ or simple/backports-shutil-get-terminal-size/
I am far from an expert on python packaging. I'm just noting what I observed with pip and the -v flags while trying to setup my own little pypi repo.
The proxy variables isn't honoured by pip2pi
Traceback (most recent call last):
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pip/basecommand.py", line 134, in main
status = self.run(options, args)
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pip/commands/bundle.py", line 52, in run
requirement_set = super(BundleCommand, self).run(options, args)
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pip/commands/install.py", line 252, in run
requirement_set.create_bundle(self.bundle_filename)
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pip/req.py", line 1353, in create_bundle
zip.write(filename, basename + '/' + name)
File "/usr/lib/python2.7/zipfile.py", line 1033, in write
st = os.stat(filename)
OSError: [Errno 2] No such file or directory: '/home/ubuntu/project/packages/_pip2tgz_temp/build/factory-boy/docs/index.rst'
Storing complete log in /home/ubuntu/.pip/pip.log
Traceback (most recent call last):
File "/home/ubuntu/venv/bin/pip2tgz", line 4, in
sys.exit(pip2tgz())
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/libpip2pi/commands.py", line 21, in maintain_cwd_helper
return f(_args, *_kwargs)
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/libpip2pi/commands.py", line 145, in pip2tgz
check_call(["pip", "bundle", "-b", build_dir, bundle_zip] + argv[2:])
File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pip', 'bundle', '-b', '/home/ubuntu/project/packages/_pip2tgz_temp/build', '/home/ubuntu/project/packages/_pip2tgz_temp/bundle.zip', '-r', 'requirements.txt']' returned non-zero exit status 2
When I run pip2pi packages/ -r requirements.txt
, pip say:
DEPRECATION: pip install --download has been deprecated and will be removed in the future. Pip now has a download command that should be used instead.
Currently, there is no problem. But I think it's better to update the pip install --download
to pip download.
pip expects Django to be mapped into a directory called django
We are having issues with packages that contain "-" in their names when using wheels and a current version of pip.
eg. for python-dateutil dir2py generates following structure:
$ /var/pypi/packages-develop/simple/python-dateutil > ls -l
total 4
-rw-r--r-- 1 pypi pypi 636 Apr 16 13:09 index.html
lrwxrwxrwx 1 pypi pypi 48 Apr 16 13:09 python-dateutil-2.7.2-py2.py3-none-any.whl -> ../../python_dateutil-2.7.2-py2.py3-none-any.whl
and a index.html
<a href='python-dateutil-2.7.2-py2.py3-none-any.whl'>python-dateutil-2.7.2-py2.py3-none-any.whl</a><br />
With this structure in place pip will fail to install python-dateutil:
pip -vvv install --force-reinstall https://*****/packages-develop/simple/ --only-binary python-dateutil python-dateutil
Created temporary directory: /tmp/pip-ephem-wheel-cache-r7m7ebh7
Created temporary directory: /tmp/pip-install-cl8flexr
Looking in indexes: https://****/packages-develop/simple/
Collecting python-dateutil
1 location(s) to search for versions of python-dateutil:
* https://*****/packages-develop/simple/python-dateutil/
Getting page https://***/packages-develop/simple/python-dateutil/
Looking up "https://****/packages-develop/simple/python-dateutil/" in the cache
Current age based on date: 3372
Ignoring unknown cache-control directive:
Freshness lifetime from request max-age: 600
Starting new HTTPS connection (1): *************
https://*********:443 "GET /packages-develop/simple/python-dateutil/ HTTP/1.1" 200 636
Ignoring unknown cache-control directive:
Updating cache with response from "https://*******/packages-develop/simple/python-dateutil/"
Caching due to etag
Analyzing links from page https://****/packages-develop/simple/python-dateutil/
Skipping link https://********/packages-develop/simple/python-dateutil/python-dateutil-2.6.1-py2.py3-none-any.whl (from https://******/packages-develop/simple/python-dateutil/); wrong project name (not python-dateutil)
Could not find a version that satisfies the requirement python-dateutil (from versions: )
Cleaning up...
No matching distribution found for python-dateutil
Exception information:
Traceback (most recent call last):
File "/var/lib/go-agent/pipelines/accclient/accclient/debian/accclient/usr/share/nnis/accclient/lib/python3.4/site-packages/pip/_internal/basecommand.py", line 228, in main
status = self.run(options, args)
File "/var/lib/go-agent/pipelines/accclient/accclient/debian/accclient/usr/share/nnis/accclient/lib/python3.4/site-packages/pip/_internal/commands/install.py", line 291, in run
resolver.resolve(requirement_set)
File "/var/lib/go-agent/pipelines/accclient/accclient/debian/accclient/usr/share/nnis/accclient/lib/python3.4/site-packages/pip/_internal/resolve.py", line 103, in resolve
self._resolve_one(requirement_set, req)
File "/var/lib/go-agent/pipelines/accclient/accclient/debian/accclient/usr/share/nnis/accclient/lib/python3.4/site-packages/pip/_internal/resolve.py", line 257, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "/var/lib/go-agent/pipelines/accclient/accclient/debian/accclient/usr/share/nnis/accclient/lib/python3.4/site-packages/pip/_internal/resolve.py", line 210, in _get_abstract_dist_for
self.require_hashes
File "/var/lib/go-agent/pipelines/accclient/accclient/debian/accclient/usr/share/nnis/accclient/lib/python3.4/site-packages/pip/_internal/operations/prepare.py", line 243, in prepare_linked_requirement
req.populate_link(finder, upgrade_allowed, require_hashes)
File "/var/lib/go-agent/pipelines/accclient/accclient/debian/accclient/usr/share/nnis/accclient/lib/python3.4/site-packages/pip/_internal/req/req_install.py", line 307, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/var/lib/go-agent/pipelines/accclient/accclient/debian/accclient/usr/share/nnis/accclient/lib/python3.4/site-packages/pip/_internal/index.py", line 533, in find_requirement
'No matching distribution found for %s' % req
pip._internal.exceptions.DistributionNotFound: No matching distribution found for python-dateutil
It turns out that pip fails to handle the filename python-dateutil properly. However if adjust the structure manually to:
$ /var/pypi/packages-develop/simple/python-dateutil > ls -l
total 4
-rw-r--r-- 1 pypi pypi 636 Apr 16 14:04 index.html
lrwxrwxrwx 1 pypi pypi 48 Apr 16 13:09 python_dateutil-2.7.2-py2.py3-none-any.whl -> ../../python_dateutil-2.7.2-py2.py3-none-any.whl
$ /var/pypi/packages-develop/simple/python-dateutil > cat index.html
<a href='python_dateutil-2.7.2-py2.py3-none-any.whl'>python_dateutil-2.7.2-py2.py3-none-any.whl</a><br />
eveything is fine. (note the changes "-" -> "_")
Can you please adjust dir2py to generate symlinks and not translate "_" to "-" in filenames?
Regards
Roman
I often add a new package to my mirror then I run
dir2pi -n /path/to/my/mirror
The command takes ages to complete because it restarts from the beginning each time. Is there an --update
option to just create the symlinks for missing packages?
Using this for the first time and having problems with this in my requirements.txt:
backports.ssl-match-hostname==3.4.0.2
I'm doing the standard:
pip2pi /tmp/packages -r .requirements/requirements.txt
Then doing this:
pip install --index-url=file:///tmp/packages/simple -r requirements.txt
This is the error:
OSError: [Errno 2] No such file or directory: '/tmp/packages/simple/backports.ssl-match-hostname/'
In /tmp/packages/simple/ this is the directory that gets creates:
backports.ssl_match_hostname
So, the "-" gets converted to "_" and is this just a pip problem or something with pip2pi.
Thanks!
The problem is here https://github.com/wolever/pip2pi/blob/master/setup.py#L35
It would be great to have some kind of --wheel
option which generates wheels where only source packages are downloaded.
Running pip2tgz packages -r requirements.txt
requirements.txt:
Django==1.5.1
Fabric==1.6.1
South==0.8.1
argparse==1.2.1
django-allauth==0.12.0
django-urlauth==0.1.8
django-bootstrap-form==2.0.6
httplib2==0.8
oauth2==1.5.211
psycopg2==2.5.1
python-openid==2.2.5
wsgiref==0.1.2
xlrd==0.9.2
pytz
git+https://github.com/dragonfly-science/dragonfly-data-entry.git
django-reversion==1.7.1
git+git://github.com/zmathew/django-backbone.git
django-nose==1.1
nose-cov
pyquery==1.2.4
# Django Debug Toolbar is lagging it's update from 0.9.4
git+https://github.com/django-debug-toolbar/django-debug-toolbar.git@7e4ecfd04#egg=django-debug-toolbar
django-widget-tweaks==1.3
django-extensions==1.1.1
git+https://github.com/ferrouswheel/dragonfab.git
mock
requests==1.2.3
python-dateutil==2.1
python-magic==0.4.3
python-statsd==1.6.0
# Django CMS extras
pillow
cov-core==1.7
coverage==3.6
cssselect==0.8
django-account==0.1.14
django-classy-tags==0.4
django-cms==2.4.2
django-mptt==0.5.5
django-sekizai==0.7
html5lib==1.0b2
lxml==3.2.1
six==1.3.0
unicodecsv
Error was:
Downloading/unpacking django-debug-toolbar from git+https://github.com/django-debug-toolbar/django-debug-toolbar.git@7e4ecfd04#egg=django-debug-toolbar (from -r ../requirements.txt (line 24))
Cloning https://github.com/django-debug-toolbar/django-debug-toolbar.git (to 7e4ecfd04) to ./packages/_pip2tgz_temp/build/django-debug-toolbar
Could not find a tag or branch '7e4ecfd04', assuming commit.
Running setup.py egg_info for package django-debug-toolbar
Running setup.py egg_info for package django-debug-toolbar
Exception:
...
OSError: [Errno 17] File exists: '.../pipcache/packages/_pip2tgz_temp/build/django-debug-toolbar'
Storing complete log in ~/.pip/pip.log
Traceback (most recent call last):
File ".virtualenvs/xxx/bin/pip2tgz", line 4, in <module>
sys.exit(pip2tgz())
File ".virtualenvs/xxx//local/lib/python2.7/site-packages/libpip2pi/commands.py", line 21, in maintain_cwd_helper
return f(*args, **kwargs)
File ".virtualenvs/xxx//local/lib/python2.7/site-packages/libpip2pi/commands.py", line 145, in pip2tgz
check_call(["pip", "bundle", "-b", build_dir, bundle_zip] + argv[2:])
File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pip', 'bundle', '-b', '.../pipcache/packages/_pip2tgz_temp/build', '.../pipcache/packages/_pip2tgz_temp/bundle.zip', '-r', '../requirements.txt']' returned non-zero exit status 2
A direct pip install using this requirements.txt file works fine and does not have this error.
Hello!
I tried pip2pi to create a local mirror for some python packages using a requirements.freeze.txt. And it failed on this minimal case:
$ pip2tgz /var/pypi/ -r requirements.freeze.txt
Downloading/unpacking git+ssh://[email protected]/smathot/pyhunspell.git (from -r requirements.freeze.txt (line 2))
Cloning ssh://[email protected]/smathot/pyhunspell.git to /tmp/pip-VqMZNN-export
Running setup.py (path:/tmp/pip-HIwbcr-build/setup.py) egg_info for package from git+ssh://[email protected]/smathot/pyhunspell.git
Traceback (most recent call last):
File "<string>", line 17, in <module>
IOError: [Errno 2] No such file or directory: '/tmp/pip-HIwbcr-build/setup.py'
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 17, in <module>
IOError: [Errno 2] No such file or directory: '/tmp/pip-HIwbcr-build/setup.py'
----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /tmp/pip-HIwbcr-build
Storing debug log for failure in /root/.pip/pip.log
Done. 0 archives currently saved in '/var/pypi/'.
And pip succeed in installing that:
$ pip install -r requirements.freeze.txt
Downloading/unpacking git+ssh://[email protected]/smathot/pyhunspell.git (from -r requirements.freeze.txt (line 1))
Cloning ssh://[email protected]/smathot/pyhunspell.git to /tmp/pip-0WdcpV-build
Running setup.py (path:/tmp/pip-0WdcpV-build/setup.py) egg_info for package from git+ssh://[email protected]/smathot/pyhunspell.git
Cleaning up...
My requirements.freeze.txt looks like that:
git+ssh://[email protected]/smathot/pyhunspell.git
And this pyhunspell package has a setup.py in it.
I can't use the official pip package since it's not up to date considering current hunspell versions (even on Debian stable :p )
Any update is welcome.
Cheers!
The index directory for package python-daemon-3K
is normalized incorrectly to python-daemon
rather than python-daemon-3k
.
Steps to reproduce
$ pip2pi -nz foo/ python-daemon-3K==1.5.8
Actual behavior
$ ls -l foo/simple
[...]
drwxrwx--- 2 reidpr reidpr 80 Jul 21 11:28 python-daemon
[...]
pip install
then fails with:
DEPRECATION: Failed to find 'python-daemon-3K' at file:///[...]/packages/simple/python-daemon-3k/. It is suggested to upgrade your index to support normalized names as the name in /simple/{name}.
Could not find a version that satisfies the requirement python-daemon-3K==1.5.8 (from -r requirements.txt (line 19)) (from versions: )
No matching distribution found for python-daemon-3K==1.5.8 (from -r requirements.txt (line 19))
Expected behavior
$ ls -l foo/simple
[...]
drwxrwx--- 2 reidpr reidpr 80 Jul 21 11:28 python-daemon-3k
[...]
Comments
-3k
makes it work.3K
is being erroneously interpreted as a version number.Hi,
On PyPi the latest version is 0.5.0 https://pypi.python.org/pypi/pip2pi/0.5.0 but looking at master code, version is 0.4.0 https://github.com/wolever/pip2pi/blob/master/libpip2pi/__init__.py
What went wrong?
Many thanks!
Maybe this is outside the scope of this project, but I have lots of things installed directly from GitHub and a private gitolite.
It seems like those get installed from the source and not locally.
Is there some way to make those install locally too?
When running dir2pi with symlinks and package normalization, dir2pi will fail if the same package is present with both a normalized and non-normalized package name. For instance, if I have these two wheels in a directory:
requests_oauthlib-0.4.2-py2.py3-none-any.whl # version from pypi
requests-oauthlib-0.4.2-py2.py3-none-any.whl # version created by pip2pi
running dir2pi will fail with an unhelpful error:
$ dir2pi --normalize-package-names wheelhouse
Traceback (most recent call last):
File "/usr/local/bin/dir2pi", line 11, in <module>
sys.exit(dir2pi())
File "/usr/local/lib/python2.7/dist-packages/libpip2pi/commands.py", line 268, in dir2pi
return _dir2pi(option, argv)
File "/usr/local/lib/python2.7/dist-packages/libpip2pi/commands.py", line 305, in _dir2pi
os.symlink(symlink_source, symlink_target)
OSError: [Errno 17] File exists
This should probably just print a warning and skip.
We're using dir2pi
from 0.6.2 to generate an internal package index for serving wheels and .tar.gz
s.
We have a package with a '-' in the name, like 'hello-world'. setup.py sdist
creates hello-world-1.2.0.tar.gz
, but setup.py bdist_wheel
creates hello-world-1.2.0-py2.py3-none-any.whl
, so when we attempt to install hello-world==1.2.0
from our index, it always downloads the sdist. When I install hello_world==1.2.0
it uses the wheel.
This doesn't appear to be an issue on the cheeseshop (pip install django-model-utils
will download the wheel named django_model_utils
), so I'm thinking it can be fixed here.
I have a directory full of packages like below
dmanna@ubuntu:~/pyt/pkg$ ls
cdiff-1.0.tar.gz
certifi-2018.4.16-py2.py3-none-any.whl
chardet-3.0.4-py2.py3-none-any.whl
click-6.7-py2.py3-none-any.whl
idna-2.7-py2.py3-none-any.whl
kazoo-2.5.0-py2.py3-none-any.whl
patroni-1.4.4.tar.gz
prettytable-0.7.2.tar.bz2
psutil-5.4.6.tar.gz
psycopg2-2.7.4.tar.gz
python_dateutil-2.7.3-py2.py3-none-any.whl
pytz-2018.4-py2.py3-none-any.whl
PyYAML-3.12.tar.gz
requests-2.19.1-py2.py3-none-any.whl
six-1.11.0-py2.py3-none-any.whl
tzlocal-1.5.1.tar.gz
urllib3-1.23-py2.py3-none-any.whl
After running dir2pi pyt/pkg
the content of simple
dmanna@ubuntu:~/pyt/pkg/simple$ ls
cdiff click kazoo psutil pytz six
certifi idna patroni psycopg2 pyyaml tzlocal
chardet index.html prettytable python-dateutil requests urllib3
Now on trying to install patroni
it is throwing the below error
dmanna@ubuntu:~$ sudo pip install --index-url=file:///home/dmanna/pyt/pkg/simple patroni
Downloading/unpacking patroni
Running setup.py (path:/tmp/pip_build_root/patroni/setup.py) egg_info for package patroni
Downloading/unpacking urllib3>=1.19.1,!=1.21 (from patroni)
Downloading/unpacking psycopg2>=2.5.4 (from patroni)
Running setup.py (path:/tmp/pip_build_root/psycopg2/setup.py) egg_info for package psycopg2
Downloading/unpacking PyYAML (from patroni)
Cleaning up...
Exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 278, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1178, in prepare_files
url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
File "/usr/lib/python2.7/dist-packages/pip/index.py", line 194, in find_requirement
page = self._get_page(main_index_url, req)
File "/usr/lib/python2.7/dist-packages/pip/index.py", line 568, in _get_page
session=self.session,
File "/usr/lib/python2.7/dist-packages/pip/index.py", line 670, in get_page
resp = session.get(url, headers={"Accept": "text/html"})
File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/sessions.py", line 467, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.7/dist-packages/pip/download.py", line 237, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/sessions.py", line 455, in request
resp = self.send(prep, **send_kwargs)
File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/sessions.py", line 558, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/pip/download.py", line 194, in send
stats = os.stat(pathname)
OSError: [Errno 2] No such file or directory: '/home/dmanna/pyt/pkg/simple/PyYAML/'
Storing debug log for failure in /home/dmanna/.pip/pip.log
Versions
Each time I run the followed command it downloads the packages from the original source, even if they are already in packages/ folder:
pip2pi /var/www/packages/ -r /var/www/requirements.txt
The packages I have in the list:
Django==1.5.1
MySQL-python==1.2.4
South==0.7.6
amqp==1.0.11
anyjson==0.3.3
argparse==1.2.1
billiard==2.7.3.28
celery==3.0.19
clint==0.3.1
django-admin-bootstrapped==0.3.2
django-allauth==0.10.1
django-bootstrap-toolkit==2.11.4
django-celery==3.0.17
django-dajaxice==0.5.5
django-debug-toolbar==0.9.4
django-sitetree==0.9.4
gunicorn==0.17.2
httplib2==0.8
kombu==2.5.10
mock==1.0.1
oauthlib==0.4.0
python-dateutil==2.1
python-openid==2.2.5
pytz==2013b
requests==1.2.0
requests-oauthlib==0.3.1
simplejson==3.3.0
six==1.3.0
tweepy==2.0
wsgiref==0.1.2
Or I have miss-configured it?
Currently dir2pi
builds into the /simple
subfolder of the given directory.
Can we add a option to specify the output directory?
Use Case for this:
pip2tgz -r requirements
in a local folderdir2pi PACKAGE_DIR OUTPUT_DIR
the project readme does not indicate any way that underneath the implementation actually builds and installs the packages.
this means for the command to succeed, i need to install various -dev
packages
for example i was looking for this project, to speed up docker builds, using multistage builds.
FROM debian:stretch
RUN apt-get update
RUN apt-get install -y --no-install-recommends python3-pip python3-setuptools
RUN pip3 install pip2pi
RUN pip2tgz packages/ tesserocr==2.3.0
RUN dir2pi packages/
this fails with
root@dind docker/w# docker build -t pip2pi:1 .
Sending build context to Docker daemon 35.33kB
Step 1/9 : FROM debian:stretch
---> 3bbb526d2608
Step 2/9 : COPY requirements.txt /usr/src/weblate/
---> Using cache
---> fafdfec54e56
Step 3/9 : RUN apt-get update
---> Using cache
---> 94348872497d
Step 4/9 : RUN apt-get install -y --no-install-recommends python3-pip
---> Using cache
---> 649d90c85c82
Step 5/9 : RUN pip3 install pip2pi
---> Using cache
---> 82b6573157ea
Step 6/9 : ENV VERSION 3.0.1
---> Using cache
---> e4256fa8b6f0
Step 7/9 : RUN apt-get install -y --no-install-recommends python3-setuptools
---> Using cache
---> d05874d08e84
Step 8/9 : RUN pip2tgz packages/ tesserocr==2.3.0
---> Running in 022e4f9ef3cd
DEPRECATION: pip install --download has been deprecated and will be removed in the future. Pip now has a download command that should be used instead.
Collecting tesserocr==2.3.0
Downloading https://files.pythonhosted.org/packages/49/fc/f5d3ca8606566073919a5964118591734e91ee8e42adcc0cc8fcefc3953a/tesserocr-2.3.0.tar.gz (54kB)
Saved /packages/tesserocr-2.3.0.tar.gz
...
Collecting tesserocr==2.3.0
Downloading https://files.pythonhosted.org/packages/49/fc/f5d3ca8606566073919a5964118591734e91ee8e42adcc0cc8fcefc3953a/tesserocr-2.3.0.tar.gz (54kB)
Saved /packages/tesserocr-2.3.0.tar.gz
Complete output from command python setup.py egg_info:
Failed to extract tesseract version from executable: [Errno 2] No such file or directory: 'tesseract'
Supporting tesseract v3.04.00
...
distutils.errors.DistutilsExecError: command 'x86_64-linux-gnu-gcc' failed with exit status 1
...
It would be great to have some kind of --wheel
option which generates wheels where only source packages are downloaded.
Would I be correct in saying that pip2pi does not support the new wheel files?
For example, so that pip2pi example.com:/var/www/pypi/ foo==1.2
will only download foo==1.2
packages if it do not exist in example.com:/var/www/pyp/
.
After some poking around, it doesn't look like there's a straightforward way to do this. To continue calling pip's install command directly:
InstallCommand.run
method sets RequirementSet.ignore_dependencies=True
when the --download
option is set. However we do want requirements to be checked, just checked against our own set of packages.InstallRequirement.check_if_exists
method would need to be overridden so that it checks against a set we provide instead of whatever logic it's using right now.Alternatively we could re-implement significant parts of InstallCommand.run
. But that would involve copy+pasting quite a bit of code, and even then I don't think those APIs are public… so that could become a maintenance nightmare.
Line 117 in function pip2tgz:
check_call(["pip", "bundle", "-b", build_dir, bundle_zip] + argv[2:])
This assumes that there is a global pip installation. I am bootstrapping pip in to virtual environments with no global pip. Alternatively, you could use something like "os.path.join(os.path.dirname(sys.executable), 'pip')".
In pip 6, handling of mixed-case directory names under simple/ is deprecated, and will result in a warning message. See this commit and this discussion.
I'm looking specifically at the wheel for python-dateutil. I did the following:
$ pip2tgz pkgs/ python-dateutil
I then tried dir2pi
with all the different normalization options (-n
, -N
, -a
). pip install
ing from this index does not work in any case:
$ pip install --index-url file:///<blah>/pkgs/simple/ python-dateutil -v
...
Looking in indexes: file:///<blah>/pkgs/simple/
Collecting python-dateutil
1 location(s) to search for versions of python-dateutil:
* file:///<blah>/pkgs/simple/python-dateutil/
Getting page file:///<blah>/pkgs/simple/python-dateutil/
file: URL is directory, getting file:///<blah>/pkgs/simple/python-dateutil/index.html
Analyzing links from page file:///<blah>/pkgs/simple/python-dateutil/index.html
Skipping link file:///<blah>/pkgs/simple/python-dateutil/python-dateutil-2.7.2-py2.py3-none-any.whl (from file:///<blah>/pkgs/simple/python-dateutil/index.html); wrong project name (not python-dateutil)
However, if I manually rename the package name in simple/python-dateutil
to use an underscore: python_dateutil-2.7.2-py2.py3-none-any.whl
, and edit index.html
to point to it, everything works:
$ pip install --index-url file:///<blah>/pkgs/simple/ python-dateutil -v
...
Analyzing links from page file:///<blah>/pkgs/simple/python-dateutil/index.html
Found link file:///<blah>/pkgs/simple/python-dateutil/python_dateutil-2.7.2-py2.py3-none-any.whl (from file:///<blah>/pkgs/simple/python-dateutil/index.html), version: 2.7.2
The problem seems to be with the way Wheel parses the filenames.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.