Code Monkey home page Code Monkey logo

pip2pi's Introduction

pip2pi builds a PyPI-compatible package repository from pip requirements

image

PyPI can go down, package maintainers can remove old tarballs, and downloading tarballs can take a long time. pip2pi helps to alleviate these problems by making it blindingly simple to maintain a PyPI-compatible repository of packages your software depends on.

$ pip2pi --help
Usage: pip2pi TARGET [PIP_OPTIONS] PACKAGES ...

Adds packages PACKAGES to PyPI-compatible package index at TARGET.

If TARGET contains ':' it will be treated as a remote path. The
package index will be built locally and rsync will be used to copy
it to the remote host.

PIP_OPTIONS can be any options accepted by `pip install -d`, like
`--index-url` or `--no-use-wheel`.

For example, to create a remote index:

    $ pip2pi example.com:/var/www/packages/ -r requirements.txt

To create a local index:

    $ pip2pi ~/Sites/packages/ foo==1.2

To pass arguments to pip:

    $ pip2pi ~/Sites/packages/ \
        --index-url https://example.com/simple \
        --no-use-wheel \
        -r requirements-base.txt \
        -r requirements-dev.txt \
        bar==3.1

Requirements

  1. pip
  2. A requirements.txt file for your project (optional, but useful)
  3. An HTTP server (optional, but useful)

Setup

Install pip2pi:

$ pip install pip2pi

And create the directory which will contain the tarballs of required packages, preferably somewhere under your web server's document root:

$ mkdir /var/www/packages/

Mirroring Packages

To mirror a package and all of its requirements, use pip2tgz:

$ pip2tgz packages/ foo==1.2
...
$ ls packages/
foo-1.2.tar.gz
bar-0.8.tar.gz

Note that pip2tgz passes package arguments directly to pip, so packages can be specified in any format that pip recognizes:

$ cat requirements.txt
foo==1.2
http://example.com/baz-0.3.tar.gz
$ pip2tgz packages/ -r requirements.txt bam-2.3/
...
$ ls packages/
foo-1.2.tar.gz
bar-0.8.tar.gz
baz-0.3.tar.gz
bam-2.3.tar.gz

Building a Package Index

A directory full of .tar.gz files can be turned into PyPI-compatible "simple" package index using the dir2pi command:

$ ls packages/
bar-0.8.tar.gz
baz-0.3.tar.gz
foo-1.2.tar.gz
$ dir2pi packages/
$ find packages/
packages/
packages/bar-0.8.tar.gz
packages/baz-0.3.tar.gz
packages/foo-1.2.tar.gz
packages/simple
packages/simple/bar
packages/simple/bar/bar-0.8.tar.gz
packages/simple/baz
packages/simple/baz/baz-0.3.tar.gz
packages/simple/foo
packages/simple/foo/foo-1.2.tar.gz

But that's a lot of work...

If running two commands seems like too much work... Take heart! The pip2pi command will run both of them for you... And it will use rsync to copy the new packages and index to a remote host! :

$ pip2pi example.com:/var/www/packages/ foo==1.2
...
$ curl -I http://example.com/packages/simple/foo/foo-1.2.tar.gz | head -n1
HTTP/1.1 200 OK

But that's still too much work...

Take heart! Your shell's alias command can help. Add an alias like this to your shell's runtime configuration file (hint: ~/.bashrc or similar):

alias pip2acmeco="pip2pi dev.acmeco.com:/var/www/packages/"

Now updating your package index will be as simple as:

$ pip2acmeco foo==1.2 -r bar/requirements.txt

Using Your New Package Index

To use the new package index, pass the --index-url= argument to pip:

$ pip install --index-url=http://example.com/packages/simple/ foo

Or, once it has been mirrored, prefix you requirements.txt with --index-url=...:

$ cat requirements.txt
--index-url=http://example.com/packages/simple/
foo==1.2

Without a web server

You can use your package index offline, too:

$ pip install --index-url=file:///var/www/packages/simple foo==1.2

Some Tips

When installing packages from source via python setup.py install or python setup.py install, you may need to create a setup.cfg, which points to your package index. Here are some examples for an offline package index in your Windows, Linux, or Mac file system:

[easy_install]
# Windows
# index_url = file:///C:/pip2pi/simple/

# Linux
# index_url = file:///home/myusername/.pip2pi/simple/

# Mac
index_url = file:///Users/myusername/.pip2pi/simple/

Note the triple /// after file: -- two for the protocol, the third for the root of the local file system.

Keywords

  • Mirror PyPI
  • Offline PyPI
  • Create offline PyPI mirror

pip2pi's People

Contributors

antevens avatar brauliov avatar encukou avatar georgevreilly avatar greg-hellings avatar jsutlovic avatar reubano avatar safiyat avatar sethrosenblum avatar shazow avatar tgross avatar themartorana avatar tikank avatar venkatraju avatar wogg avatar wolever avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pip2pi's Issues

Option to not download wheel packages

When using this tool there is no easy way currently to block pip from downloading python wheel packages when using pip2pi tool.

Pip have the following flag that can be used when called

--no-use-wheel   Do not Find and prefer wheel archives when searching indexes and find-links locations.

It would be great if this tool had this option as either a flag to pip2pi or perhaps as a environment variable that can be used NO_WHEEL=true pip2pi ... or something like that.

pip bundle will be removed in pip v1.5

When running pip2pi with pip 1.4.1 installed, I see

###############################################
##                                           ##
##  Due to lack of interest and maintenance, ##
##  'pip bundle' and support for installing  ##
##  from *.pybundle files is now deprecated, ##
##  and will be removed in pip v1.5.         ##
##                                           ##
###############################################

dir2pi fails when normalized/non-normalized wheels are present

When running dir2pi with symlinks and package normalization, dir2pi will fail if the same package is present with both a normalized and non-normalized package name. For instance, if I have these two wheels in a directory:

requests_oauthlib-0.4.2-py2.py3-none-any.whl # version from pypi
requests-oauthlib-0.4.2-py2.py3-none-any.whl # version created by pip2pi

running dir2pi will fail with an unhelpful error:

$ dir2pi --normalize-package-names wheelhouse
Traceback (most recent call last):
  File "/usr/local/bin/dir2pi", line 11, in <module>
    sys.exit(dir2pi())
  File "/usr/local/lib/python2.7/dist-packages/libpip2pi/commands.py", line 268, in dir2pi
    return _dir2pi(option, argv)
  File "/usr/local/lib/python2.7/dist-packages/libpip2pi/commands.py", line 305, in _dir2pi
    os.symlink(symlink_source, symlink_target)
OSError: [Errno 17] File exists

This should probably just print a warning and skip.

pip2pi has issues with statsmodels

Trying to do pip2tgz packages/ statsmodels gives the error:
ImportError: statsmodels requires patsy. http://patsy.readthedocs.org

It seems like you have to have patsy already installed to successfully use pip2pi on statsmodels. I'm not sure if this is a bug in the statsmodels packaging or in pip2pi

Duplicate file in temp directory causes failure with pip2tgz and requirements file

Running pip2tgz packages -r requirements.txt

requirements.txt:

Django==1.5.1
Fabric==1.6.1
South==0.8.1
argparse==1.2.1
django-allauth==0.12.0
django-urlauth==0.1.8
django-bootstrap-form==2.0.6
httplib2==0.8
oauth2==1.5.211
psycopg2==2.5.1
python-openid==2.2.5
wsgiref==0.1.2
xlrd==0.9.2
pytz
git+https://github.com/dragonfly-science/dragonfly-data-entry.git
django-reversion==1.7.1
git+git://github.com/zmathew/django-backbone.git
django-nose==1.1
nose-cov
pyquery==1.2.4
# Django Debug Toolbar is lagging it's update from 0.9.4
git+https://github.com/django-debug-toolbar/django-debug-toolbar.git@7e4ecfd04#egg=django-debug-toolbar
django-widget-tweaks==1.3
django-extensions==1.1.1
git+https://github.com/ferrouswheel/dragonfab.git 
mock
requests==1.2.3
python-dateutil==2.1
python-magic==0.4.3
python-statsd==1.6.0

# Django CMS extras
pillow
cov-core==1.7
coverage==3.6
cssselect==0.8
django-account==0.1.14
django-classy-tags==0.4
django-cms==2.4.2
django-mptt==0.5.5
django-sekizai==0.7
html5lib==1.0b2
lxml==3.2.1
six==1.3.0

unicodecsv

Error was:

Downloading/unpacking django-debug-toolbar from git+https://github.com/django-debug-toolbar/django-debug-toolbar.git@7e4ecfd04#egg=django-debug-toolbar (from -r ../requirements.txt (line 24))
  Cloning https://github.com/django-debug-toolbar/django-debug-toolbar.git (to 7e4ecfd04) to ./packages/_pip2tgz_temp/build/django-debug-toolbar
  Could not find a tag or branch '7e4ecfd04', assuming commit.
  Running setup.py egg_info for package django-debug-toolbar

  Running setup.py egg_info for package django-debug-toolbar

Exception:
...
OSError: [Errno 17] File exists: '.../pipcache/packages/_pip2tgz_temp/build/django-debug-toolbar'

Storing complete log in ~/.pip/pip.log
Traceback (most recent call last):
  File ".virtualenvs/xxx/bin/pip2tgz", line 4, in <module>
    sys.exit(pip2tgz())
  File ".virtualenvs/xxx//local/lib/python2.7/site-packages/libpip2pi/commands.py", line 21, in maintain_cwd_helper
    return f(*args, **kwargs)
  File ".virtualenvs/xxx//local/lib/python2.7/site-packages/libpip2pi/commands.py", line 145, in pip2tgz
    check_call(["pip", "bundle", "-b", build_dir, bundle_zip] + argv[2:])
  File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pip', 'bundle', '-b', '.../pipcache/packages/_pip2tgz_temp/build', '.../pipcache/packages/_pip2tgz_temp/bundle.zip', '-r', '../requirements.txt']' returned non-zero exit status 2

A direct pip install using this requirements.txt file works fine and does not have this error.

Don't re-download packages if they already exist in the index

For example, so that pip2pi example.com:/var/www/pypi/ foo==1.2 will only download foo==1.2 packages if it do not exist in example.com:/var/www/pyp/.

After some poking around, it doesn't look like there's a straightforward way to do this. To continue calling pip's install command directly:

  1. The InstallCommand.run method sets RequirementSet.ignore_dependencies=True when the --download option is set. However we do want requirements to be checked, just checked against our own set of packages.
  2. The InstallRequirement.check_if_exists method would need to be overridden so that it checks against a set we provide instead of whatever logic it's using right now.

Alternatively we could re-implement significant parts of InstallCommand.run. But that would involve copy+pasting quite a bit of code, and even then I don't think those APIs are public… so that could become a maintenance nightmare.

Installed through github and gitolite?

Maybe this is outside the scope of this project, but I have lots of things installed directly from GitHub and a private gitolite.

It seems like those get installed from the source and not locally.

Is there some way to make those install locally too?

Pip's "download" command?

Does this package do anything different from pip's "download" command? Perhaps it was written before "pip download" was a thing? If so, should this package be deprecated in favor of that command?

dir2pi normalization broken for some wheel packages

I'm looking specifically at the wheel for python-dateutil. I did the following:

$ pip2tgz pkgs/ python-dateutil

I then tried dir2pi with all the different normalization options (-n, -N, -a). pip installing from this index does not work in any case:

$ pip install --index-url file:///<blah>/pkgs/simple/ python-dateutil -v
...
Looking in indexes: file:///<blah>/pkgs/simple/
Collecting python-dateutil
  1 location(s) to search for versions of python-dateutil:
  * file:///<blah>/pkgs/simple/python-dateutil/
  Getting page file:///<blah>/pkgs/simple/python-dateutil/
   file: URL is directory, getting file:///<blah>/pkgs/simple/python-dateutil/index.html
  Analyzing links from page file:///<blah>/pkgs/simple/python-dateutil/index.html
    Skipping link file:///<blah>/pkgs/simple/python-dateutil/python-dateutil-2.7.2-py2.py3-none-any.whl (from file:///<blah>/pkgs/simple/python-dateutil/index.html); wrong project name (not python-dateutil)

However, if I manually rename the package name in simple/python-dateutil to use an underscore: python_dateutil-2.7.2-py2.py3-none-any.whl, and edit index.html to point to it, everything works:

$ pip install --index-url file:///<blah>/pkgs/simple/ python-dateutil -v
...
  Analyzing links from page file:///<blah>/pkgs/simple/python-dateutil/index.html
    Found link file:///<blah>/pkgs/simple/python-dateutil/python_dateutil-2.7.2-py2.py3-none-any.whl (from file:///<blah>/pkgs/simple/python-dateutil/index.html), version: 2.7.2

The problem seems to be with the way Wheel parses the filenames.

Wheel support

Would I be correct in saying that pip2pi does not support the new wheel files?

Check if a package is already downloaded

If both packages A and B depends on C, then C will be downloaded twice.

pip2tgz pipdir/ requests
requests-1.2.3.tar.gz on pipdir

pip2tgz pipdir/ cachecontrol
dependency requests-1.2.3.tar.gz downloaded twice..

Directories should be lower case

pip will try to lookup wheels in lower case URLs, while pip2pi will generate in whatever case the package name has. One example of this is PyYAML, which will not be found in a repo generated by pip2pi. The workaround is trivial, just slap on a script afterwards that lower-cases the directories, and updates index.html.

Update existing mirror?

I often add a new package to my mirror then I run

dir2pi -n /path/to/my/mirror

The command takes ages to complete because it restarts from the beginning each time. Is there an --update option to just create the symlinks for missing packages?

Wheel generation in pip2pi

It would be great to have some kind of --wheel option which generates wheels where only source packages are downloaded.

Issue with dir2pi while a package name has uppercase letters in it

I have a directory full of packages like below

dmanna@ubuntu:~/pyt/pkg$ ls
cdiff-1.0.tar.gz
certifi-2018.4.16-py2.py3-none-any.whl
chardet-3.0.4-py2.py3-none-any.whl
click-6.7-py2.py3-none-any.whl
idna-2.7-py2.py3-none-any.whl
kazoo-2.5.0-py2.py3-none-any.whl
patroni-1.4.4.tar.gz
prettytable-0.7.2.tar.bz2
psutil-5.4.6.tar.gz
psycopg2-2.7.4.tar.gz
python_dateutil-2.7.3-py2.py3-none-any.whl
pytz-2018.4-py2.py3-none-any.whl
PyYAML-3.12.tar.gz
requests-2.19.1-py2.py3-none-any.whl
six-1.11.0-py2.py3-none-any.whl
tzlocal-1.5.1.tar.gz
urllib3-1.23-py2.py3-none-any.whl

After running dir2pi pyt/pkg the content of simple

dmanna@ubuntu:~/pyt/pkg/simple$ ls
cdiff    click       kazoo        psutil           pytz      six
certifi  idna        patroni      psycopg2         pyyaml    tzlocal
chardet  index.html  prettytable  python-dateutil  requests  urllib3

Now on trying to install patroni it is throwing the below error

dmanna@ubuntu:~$ sudo pip install --index-url=file:///home/dmanna/pyt/pkg/simple patroni
Downloading/unpacking patroni
  Running setup.py (path:/tmp/pip_build_root/patroni/setup.py) egg_info for package patroni
    
Downloading/unpacking urllib3>=1.19.1,!=1.21 (from patroni)
Downloading/unpacking psycopg2>=2.5.4 (from patroni)
  Running setup.py (path:/tmp/pip_build_root/psycopg2/setup.py) egg_info for package psycopg2
    
Downloading/unpacking PyYAML (from patroni)
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1178, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 194, in find_requirement
    page = self._get_page(main_index_url, req)
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 568, in _get_page
    session=self.session,
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 670, in get_page
    resp = session.get(url, headers={"Accept": "text/html"})
  File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/sessions.py", line 467, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 237, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/sessions.py", line 455, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/sessions.py", line 558, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 194, in send
    stats = os.stat(pathname)
OSError: [Errno 2] No such file or directory: '/home/dmanna/pyt/pkg/simple/PyYAML/'

Storing debug log for failure in /home/dmanna/.pip/pip.log

Versions

  • pip 1.5.4 from /usr/lib/python2.7/dist-packages
  • pip2pi - 0.7.0
  • Ubuntu 14.04

Absolute path to target directory on Windows interpreted as remote target

With this command

pip2pi C:\HTdocs\Fileserver\Programming\Python\pypi_pkgs -n pep8

pip2pi interprets the target as remote target and tries to use rsync which fails. The problem is how a remote target is identified, namely by if ':' in target:. I solved this by changing the line to if re.match('.*\..*:', target):. Since I do not know much about regular expressions, I am not sure if this is a good solution.

Another point: Since rsync is not available on Windows by default, a note about rsync in the requirements section in readme.rst would be helpful.

Release v0.6.4 to PyPI

I'd like to stop using my private build, since you merged my symlink pull request several weeks ago.

Target Directory for dir2pi

Currently dir2pi builds into the /simple subfolder of the given directory.

Can we add a option to specify the output directory?

Use Case for this:

  • Collect all Packages with pip2tgz -r requirements in a local folder
  • Put them on a (slow) network drive with dir2pi PACKAGE_DIR OUTPUT_DIR

run without *building* packages

the project readme does not indicate any way that underneath the implementation actually builds and installs the packages.

this means for the command to succeed, i need to install various -dev packages

for example i was looking for this project, to speed up docker builds, using multistage builds.

FROM debian:stretch
RUN apt-get update
RUN apt-get install -y --no-install-recommends python3-pip python3-setuptools
RUN pip3 install pip2pi
RUN pip2tgz packages/ tesserocr==2.3.0
RUN dir2pi packages/

this fails with

root@dind docker/w# docker build -t pip2pi:1 .
Sending build context to Docker daemon  35.33kB
Step 1/9 : FROM debian:stretch
 ---> 3bbb526d2608
Step 2/9 : COPY requirements.txt /usr/src/weblate/
 ---> Using cache
 ---> fafdfec54e56
Step 3/9 : RUN apt-get update
 ---> Using cache
 ---> 94348872497d
Step 4/9 : RUN apt-get install -y --no-install-recommends python3-pip
 ---> Using cache
 ---> 649d90c85c82
Step 5/9 : RUN pip3 install pip2pi
 ---> Using cache
 ---> 82b6573157ea
Step 6/9 : ENV VERSION 3.0.1
 ---> Using cache
 ---> e4256fa8b6f0
Step 7/9 : RUN apt-get install -y --no-install-recommends python3-setuptools
 ---> Using cache
 ---> d05874d08e84
Step 8/9 : RUN pip2tgz packages/ tesserocr==2.3.0
 ---> Running in 022e4f9ef3cd
DEPRECATION: pip install --download has been deprecated and will be removed in the future. Pip now has a download command that should be used instead.
Collecting tesserocr==2.3.0
  Downloading https://files.pythonhosted.org/packages/49/fc/f5d3ca8606566073919a5964118591734e91ee8e42adcc0cc8fcefc3953a/tesserocr-2.3.0.tar.gz (54kB)
  Saved /packages/tesserocr-2.3.0.tar.gz
...
Collecting tesserocr==2.3.0
  Downloading https://files.pythonhosted.org/packages/49/fc/f5d3ca8606566073919a5964118591734e91ee8e42adcc0cc8fcefc3953a/tesserocr-2.3.0.tar.gz (54kB)
  Saved /packages/tesserocr-2.3.0.tar.gz
    Complete output from command python setup.py egg_info:
    Failed to extract tesseract version from executable: [Errno 2] No such file or directory: 'tesseract'
    Supporting tesseract v3.04.00
...

    distutils.errors.DistutilsExecError: command 'x86_64-linux-gnu-gcc' failed with exit status 1
...

pip2tgz fail whehe pip doesn't

Hello!

I tried pip2pi to create a local mirror for some python packages using a requirements.freeze.txt. And it failed on this minimal case:

$ pip2tgz /var/pypi/ -r requirements.freeze.txt 
Downloading/unpacking git+ssh://[email protected]/smathot/pyhunspell.git (from -r requirements.freeze.txt (line 2))
  Cloning ssh://[email protected]/smathot/pyhunspell.git to /tmp/pip-VqMZNN-export
  Running setup.py (path:/tmp/pip-HIwbcr-build/setup.py) egg_info for package from git+ssh://[email protected]/smathot/pyhunspell.git
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
    IOError: [Errno 2] No such file or directory: '/tmp/pip-HIwbcr-build/setup.py'
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

IOError: [Errno 2] No such file or directory: '/tmp/pip-HIwbcr-build/setup.py'

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /tmp/pip-HIwbcr-build
Storing debug log for failure in /root/.pip/pip.log

Done. 0 archives currently saved in '/var/pypi/'.

And pip succeed in installing that:

$ pip install -r requirements.freeze.txt 
Downloading/unpacking git+ssh://[email protected]/smathot/pyhunspell.git (from -r requirements.freeze.txt (line 1))
  Cloning ssh://[email protected]/smathot/pyhunspell.git to /tmp/pip-0WdcpV-build
  Running setup.py (path:/tmp/pip-0WdcpV-build/setup.py) egg_info for package from git+ssh://[email protected]/smathot/pyhunspell.git

Cleaning up...

My requirements.freeze.txt looks like that:

git+ssh://[email protected]/smathot/pyhunspell.git

And this pyhunspell package has a setup.py in it.
I can't use the official pip package since it's not up to date considering current hunspell versions (even on Debian stable :p )

Any update is welcome.
Cheers!

Path normalization for wheels

We're using dir2pi from 0.6.2 to generate an internal package index for serving wheels and .tar.gzs.

We have a package with a '-' in the name, like 'hello-world'. setup.py sdist creates hello-world-1.2.0.tar.gz, but setup.py bdist_wheel creates hello-world-1.2.0-py2.py3-none-any.whl, so when we attempt to install hello-world==1.2.0 from our index, it always downloads the sdist. When I install hello_world==1.2.0 it uses the wheel.

This doesn't appear to be an issue on the cheeseshop (pip install django-model-utils will download the wheel named django_model_utils), so I'm thinking it can be fixed here.

Dots in package names do not work correctly in simple indexes

When installing something like dogpile.cache or backports.shutil-get-terminal-size pip looks in directories like simple/dogpile-cache/ or simple/backports-shutil-get-terminal-size/

I am far from an expert on python packaging. I'm just noting what I observed with pip and the -v flags while trying to setup my own little pypi repo.

pip2pi issue? Or purely a pip issue?

Using this for the first time and having problems with this in my requirements.txt:

backports.ssl-match-hostname==3.4.0.2

I'm doing the standard:

pip2pi /tmp/packages -r .requirements/requirements.txt

Then doing this:

pip install --index-url=file:///tmp/packages/simple -r requirements.txt

This is the error:

OSError: [Errno 2] No such file or directory: '/tmp/packages/simple/backports.ssl-match-hostname/'

In /tmp/packages/simple/ this is the directory that gets creates:

backports.ssl_match_hostname

So, the "-" gets converted to "_" and is this just a pip problem or something with pip2pi.

Thanks!

Multiple download of packages

Each time I run the followed command it downloads the packages from the original source, even if they are already in packages/ folder:
pip2pi /var/www/packages/ -r /var/www/requirements.txt

The packages I have in the list:

Django==1.5.1
MySQL-python==1.2.4
South==0.7.6
amqp==1.0.11
anyjson==0.3.3
argparse==1.2.1
billiard==2.7.3.28
celery==3.0.19
clint==0.3.1
django-admin-bootstrapped==0.3.2
django-allauth==0.10.1
django-bootstrap-toolkit==2.11.4
django-celery==3.0.17
django-dajaxice==0.5.5
django-debug-toolbar==0.9.4
django-sitetree==0.9.4
gunicorn==0.17.2
httplib2==0.8
kombu==2.5.10
mock==1.0.1
oauthlib==0.4.0
python-dateutil==2.1
python-openid==2.2.5
pytz==2013b
requests==1.2.0
requests-oauthlib==0.3.1
simplejson==3.3.0
six==1.3.0
tweepy==2.0
wsgiref==0.1.2

Or I have miss-configured it?

Requires system-wide pip installation.

Line 117 in function pip2tgz:
check_call(["pip", "bundle", "-b", build_dir, bundle_zip] + argv[2:])

This assumes that there is a global pip installation. I am bootstrapping pip in to virtual environments with no global pip. Alternatively, you could use something like "os.path.join(os.path.dirname(sys.executable), 'pip')".

config file suggestions in README.rst use underscores instead of dashes

https://pip.pypa.io/en/stable/user_guide/#config-file

so:

[easy_install]
# Windows
# index_url = file:///C:/pip2pi/simple/

# Linux
# index_url = file:///home/myusername/.pip2pi/simple/

# Mac
index_url = file:///Users/myusername/.pip2pi/simple/

should be


[easy_install]
# Windows
# index-url = file:///C:/pip2pi/simple/

# Linux
# index-url = file:///home/myusername/.pip2pi/simple/

# Mac
index-url = file:///Users/myusername/.pip2pi/simple/

Normalize package names in simple index

It seems like pip[0] and bandersnatch[1] are now normalizing package names in the simple index, so for example simple/Django/Django-1.7.tar.gz becomes simple/dango/Django-1.7.tar.gz.

As of 0.6.5 pip2pi will include a --normalize-package-names option (default: disabled) and will issue a deprecation warning when it encounters a non-normalized package name. If no issues are found by about June 2015 --normalize-package-names will become the default.

Thanks to @orodbhen for initially reporting this: #35. See also: pypa/pip#2136

0: pypa/pip#2001
1: https://bitbucket.org/pypa/bandersnatch/pull-request/7/prefer-normalized-directory-names-but-keep/diff

Problems with wheels an pip>=10

We are having issues with packages that contain "-" in their names when using wheels and a current version of pip.

eg. for python-dateutil dir2py generates following structure:

$ /var/pypi/packages-develop/simple/python-dateutil > ls -l
total 4
-rw-r--r-- 1 pypi pypi 636 Apr 16 13:09 index.html
lrwxrwxrwx 1 pypi pypi  48 Apr 16 13:09 python-dateutil-2.7.2-py2.py3-none-any.whl -> ../../python_dateutil-2.7.2-py2.py3-none-any.whl

and a index.html

<a href='python-dateutil-2.7.2-py2.py3-none-any.whl'>python-dateutil-2.7.2-py2.py3-none-any.whl</a><br />

With this structure in place pip will fail to install python-dateutil:

pip -vvv install --force-reinstall  https://*****/packages-develop/simple/ --only-binary python-dateutil python-dateutil
Created temporary directory: /tmp/pip-ephem-wheel-cache-r7m7ebh7
Created temporary directory: /tmp/pip-install-cl8flexr
Looking in indexes: https://****/packages-develop/simple/
Collecting python-dateutil
  1 location(s) to search for versions of python-dateutil:
  * https://*****/packages-develop/simple/python-dateutil/
  Getting page https://***/packages-develop/simple/python-dateutil/
  Looking up "https://****/packages-develop/simple/python-dateutil/" in the cache
  Current age based on date: 3372
  Ignoring unknown cache-control directive: 
  Freshness lifetime from request max-age: 600
  Starting new HTTPS connection (1): *************
  https://*********:443 "GET /packages-develop/simple/python-dateutil/ HTTP/1.1" 200 636
  Ignoring unknown cache-control directive: 
  Updating cache with response from "https://*******/packages-develop/simple/python-dateutil/"
  Caching due to etag
  Analyzing links from page https://****/packages-develop/simple/python-dateutil/
    Skipping link https://********/packages-develop/simple/python-dateutil/python-dateutil-2.6.1-py2.py3-none-any.whl (from https://******/packages-develop/simple/python-dateutil/); wrong project name (not python-dateutil)
  Could not find a version that satisfies the requirement python-dateutil (from versions: )
Cleaning up...
No matching distribution found for python-dateutil
Exception information:
Traceback (most recent call last):
  File "/var/lib/go-agent/pipelines/accclient/accclient/debian/accclient/usr/share/nnis/accclient/lib/python3.4/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/var/lib/go-agent/pipelines/accclient/accclient/debian/accclient/usr/share/nnis/accclient/lib/python3.4/site-packages/pip/_internal/commands/install.py", line 291, in run
    resolver.resolve(requirement_set)
  File "/var/lib/go-agent/pipelines/accclient/accclient/debian/accclient/usr/share/nnis/accclient/lib/python3.4/site-packages/pip/_internal/resolve.py", line 103, in resolve
    self._resolve_one(requirement_set, req)
  File "/var/lib/go-agent/pipelines/accclient/accclient/debian/accclient/usr/share/nnis/accclient/lib/python3.4/site-packages/pip/_internal/resolve.py", line 257, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/var/lib/go-agent/pipelines/accclient/accclient/debian/accclient/usr/share/nnis/accclient/lib/python3.4/site-packages/pip/_internal/resolve.py", line 210, in _get_abstract_dist_for
    self.require_hashes
  File "/var/lib/go-agent/pipelines/accclient/accclient/debian/accclient/usr/share/nnis/accclient/lib/python3.4/site-packages/pip/_internal/operations/prepare.py", line 243, in prepare_linked_requirement
    req.populate_link(finder, upgrade_allowed, require_hashes)
  File "/var/lib/go-agent/pipelines/accclient/accclient/debian/accclient/usr/share/nnis/accclient/lib/python3.4/site-packages/pip/_internal/req/req_install.py", line 307, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/var/lib/go-agent/pipelines/accclient/accclient/debian/accclient/usr/share/nnis/accclient/lib/python3.4/site-packages/pip/_internal/index.py", line 533, in find_requirement
    'No matching distribution found for %s' % req
pip._internal.exceptions.DistributionNotFound: No matching distribution found for python-dateutil

It turns out that pip fails to handle the filename python-dateutil properly. However if adjust the structure manually to:

$  /var/pypi/packages-develop/simple/python-dateutil > ls -l
total 4
-rw-r--r-- 1 pypi pypi 636 Apr 16 14:04 index.html
lrwxrwxrwx 1 pypi pypi  48 Apr 16 13:09 python_dateutil-2.7.2-py2.py3-none-any.whl -> ../../python_dateutil-2.7.2-py2.py3-none-any.whl
$ /var/pypi/packages-develop/simple/python-dateutil > cat index.html 
<a href='python_dateutil-2.7.2-py2.py3-none-any.whl'>python_dateutil-2.7.2-py2.py3-none-any.whl</a><br />

eveything is fine. (note the changes "-" -> "_")

Can you please adjust dir2py to generate symlinks and not translate "_" to "-" in filenames?

Regards
Roman

dir2pi on Windows: OSError symbolic link privilege not held

Running dir2pi (or pip2pi) on Windows results in a OSError: symbolic link priviledge not held.

Steps to reproduce:

D:\>mkdir LocalPyPi
D:\>pip2pi .\LocalPyPi pytest
You are using pip version 6.1.1, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pytest
  Downloading pytest-2.8.1-py2.py3-none-any.whl (148kB)
    100% |################################| 151kB 1.7MB/s
  Saved d:\localpypi\pytest-2.8.1-py2.py3-none-any.whl
Collecting py>=1.4.29 (from pytest)
  Downloading py-1.4.30-py2.py3-none-any.whl (81kB)
    100% |################################| 86kB 1.1MB/s
  Saved d:\localpypi\py-1.4.30-py2.py3-none-any.whl
Collecting colorama (from pytest)
  Downloading colorama-0.3.3.tar.gz
  Saved d:\localpypi\colorama-0.3.3.tar.gz
Successfully downloaded pytest py colorama

Done. 3 new archives currently saved in 'D:\\LocalPyPi'.
Traceback (most recent call last):
  File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\lib\runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\Scripts\pip2pi.exe\__main__.py", line 9, in <module>
  File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\lib\site-packages\libpip2pi\commands.py", line 510, in pip2pi
    res = _dir2pi(option, subcmd_argv)
  File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\lib\site-packages\libpip2pi\commands.py", line 305, in _dir2pi
    os.symlink(symlink_source, symlink_target)
OSError: symbolic link privilege not held

Running dir2pi:

D:\>dir2pi .\LocalPyPi
Traceback (most recent call last):
  File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\lib\runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\Scripts\dir2pi.exe\__main__.py", line 9, in <module>
  File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\lib\site-packages\libpip2pi\commands.py", line 268, in dir2pi
    return _dir2pi(option, argv)
  File "D:\WinPython\WinPython-32bit-3.4.3.3\python-3.4.3\lib\site-packages\libpip2pi\commands.py", line 305, in _dir2pi
    os.symlink(symlink_source, symlink_target)
OSError: symbolic link privilege not held

Pip install --download has been deprecated

When I run pip2pi packages/ -r requirements.txt, pip say:

DEPRECATION: pip install --download has been deprecated and will be removed in the future. Pip now has a download command that should be used instead.

Currently, there is no problem. But I think it's better to update the pip install --download to pip download.

Bad symlinks during package build

Traceback (most recent call last):
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pip/basecommand.py", line 134, in main
status = self.run(options, args)
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pip/commands/bundle.py", line 52, in run
requirement_set = super(BundleCommand, self).run(options, args)
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pip/commands/install.py", line 252, in run
requirement_set.create_bundle(self.bundle_filename)
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pip/req.py", line 1353, in create_bundle
zip.write(filename, basename + '/' + name)
File "/usr/lib/python2.7/zipfile.py", line 1033, in write
st = os.stat(filename)
OSError: [Errno 2] No such file or directory: '/home/ubuntu/project/packages/_pip2tgz_temp/build/factory-boy/docs/index.rst'

Storing complete log in /home/ubuntu/.pip/pip.log
Traceback (most recent call last):
File "/home/ubuntu/venv/bin/pip2tgz", line 4, in
sys.exit(pip2tgz())
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/libpip2pi/commands.py", line 21, in maintain_cwd_helper
return f(_args, *_kwargs)
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/libpip2pi/commands.py", line 145, in pip2tgz
check_call(["pip", "bundle", "-b", build_dir, bundle_zip] + argv[2:])
File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pip', 'bundle', '-b', '/home/ubuntu/project/packages/_pip2tgz_temp/build', '/home/ubuntu/project/packages/_pip2tgz_temp/bundle.zip', '-r', 'requirements.txt']' returned non-zero exit status 2

Relax package version format checking when downloading from git-based urls

Some packages have a versioning system like -. and this may be due to the fact that their version starts from 0 and hence does not need to put a zero after the hyphen.

Example:

Traceback (most recent call last):
File "/usr/local/bin/pip2pi", line 11, in
sys.exit(pip2pi())
File "/usr/local/lib/python2.7/site-packages/libpip2pi/commands.py", line 510, in pip2pi
res = _dir2pi(option, subcmd_argv)
File "/usr/local/lib/python2.7/site-packages/libpip2pi/commands.py", line 290, in _dir2pi
pkg_name, pkg_rest = file_to_package(pkg_basename, pkgdir)
File "/usr/local/lib/python2.7/site-packages/libpip2pi/commands.py", line 123, in file_to_package
raise InvalidFilePackageName(file, basedir)
libpip2pi.commands.InvalidFilePackageName: unexpected file name: 'xfl-.06.zip' (not in 'pkg-name-version.xxx' format; found in directory: '/root/pypi_copy/var/www/packGeppers')

pip2pi broken on RHEL7.2 ?

pip2pi ./ diskimage-builder
pip2tgz ./ diskimage-builder
dir2pi --normalize-package-names ./

pip install diskimage-builder --index=http://127.0.0.1/pip-packages/
returns: "Collecting diskimage-builder
Could not find a version that satisfies the requirement diskimage-builder (from versions: )
No matching distribution found for diskimage-builder
"
curl -I http://127.0.0.1/pip-packages/simple/diskimage-builder/diskimage-builder-1.21.0-py2.py3-none-any.whl | head -n1
HTTP/1.1 200 OK

PIP 'module' object has no attribute 'main' (pip 10.0)

I'm trying to use pip2tgz but it says that the pip module has no attribute main.

I tried running python, import pip and it doesn't have a main module indeed. I believe this is due to the following change in pip:

  • Move all of pip's APIs into the pip._internal package, properly reflecting the fact that pip does not currently have any public APIs. (#4696, #4700)
  • Removed the deprecated --download option to pip install. (#2643)

Traceback of the error:

$ pip2tgz ./build/pip/ -r requirements.tx
Traceback (most recent call last):
  File "/home/jjmontes/git/scs-control-3.1/env/bin/pip2tgz", line 11, in <module>
    sys.exit(pip2tgz())
  File "/home/jjmontes/git/scs-control-3.1/env/local/lib/python2.7/site-packages/libpip2pi/commands.py", line 58, in maintain_cwd_helper
    return f(*args, **kwargs)
  File "/home/jjmontes/git/scs-control-3.1/env/local/lib/python2.7/site-packages/libpip2pi/commands.py", line 413, in pip2tgz
    pip_run_command(['install', '-d', outdir] + argv[2:])
  File "/home/jjmontes/git/scs-control-3.1/env/local/lib/python2.7/site-packages/libpip2pi/commands.py", line 158, in pip_run_command
    res = pip.main(pip_args)
AttributeError: 'module' object has no attribute 'main'

python 8.1.2 creates problem while installing packages

Hi Recently pip has release its version 8.1.2 which strictly not allow the package names with . like oslo.config . like pypa/pip#3666 . So I have local repository deployed in network which is using pip2pi and I am getting issues installing packages and installing deps with it. That issue was with devpi too but they have now solved it will look at this issue please?

pip2pi downloads but fails to include build-time requirements

I was trying to create a repo for installing protobuf, which depends on google-apputils. I ran into two different issues: Pip2pi crashes upon trying to add packages that require the 'ez-setup' package, and pip2pi automatically downloads and installs the 'distribute' package when required, but doesn't include it in the repo it creates.

$ pip2pi ../pip_repo/ google-apputils
[...]
    import ez_setup
ImportError: No module named ez_setup

$ pip2pi ../pip_repo/ google-apputils ez_setup                                                  
Downloading/unpacking google-apputils
  File was already downloaded [...]/pip_repo/google-apputils-0.4.1.tar.gz
  Running setup.py (path:[...]/google-apputils/setup.py) egg_info for package google-apputils
    Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.14.tar.gz
    Extracting in /tmp/tmpq4H5PI
    Now working in /tmp/tmpq4H5PI/distribute-0.6.14
    Building a Distribute egg in /tmp/pip_build_USERNAME/google-apputils
    /tmp/pip_build_USERNAME/google-apputils/distribute-0.6.14-py2.7.egg

But ../pip_repo doesn't include the distribute package! This is a problem if I want to make a repo to install those packages on a machine with no internet connection.

Wheel generation in pip2pi

It would be great to have some kind of --wheel option which generates wheels where only source packages are downloaded.

`python-daemon-3K` normalized incorrectly

The index directory for package python-daemon-3K is normalized incorrectly to python-daemon rather than python-daemon-3k.

Steps to reproduce

$ pip2pi -nz foo/ python-daemon-3K==1.5.8

Actual behavior

$ ls -l foo/simple
[...]
drwxrwx--- 2 reidpr reidpr  80 Jul 21 11:28 python-daemon
[...]

pip install then fails with:

  DEPRECATION: Failed to find 'python-daemon-3K' at file:///[...]/packages/simple/python-daemon-3k/. It is suggested to upgrade your index to support normalized names as the name in /simple/{name}.
  Could not find a version that satisfies the requirement python-daemon-3K==1.5.8 (from -r requirements.txt (line 19)) (from versions: )
No matching distribution found for python-daemon-3K==1.5.8 (from -r requirements.txt (line 19))

Expected behavior

$ ls -l foo/simple
[...]
drwxrwx--- 2 reidpr reidpr  80 Jul 21 11:28 python-daemon-3k
[...]

Comments

  • Manually re-naming the directory to append -3k makes it work.
  • I suspect that 3K is being erroneously interpreted as a version number.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.