jnoortheen / pipm Goto Github PK

Python dependency management workflow using setup.cfg and requirements files without reinventing the wheels

License: MIT License

Python 100.00%

pip python-package-management pip-manager python-requirements metadata python management-workflow

pipm's Introduction

Deprecation Notice: Since PEP-621 is now accepted, it is advised to use pyproject.toml to store the Project's dependency list. Here are some projects supporting PEP-621

pipm

Python package management using pip, requirements file & setup.cfg.

Installation

Adviced to install only inside virtualenv

Install from PyPI

pip install pipm

Note:

This tool manipulates all your requirements file. So be sure to use a version control software to keep track of the changes.

Quickstart

Both pip and pipm command will work as the same.
Create a virtualenv for the project and install pipm with

pip install pipm

Create an alias as alias pip=pipm or use as it is - pipm

I. Install

install all your dependencies from the base requirements file (requirements.txt)

pipm install
pipm install --all  # *requirements*.txt - all environment -> test/prod/dev

2. Add new packages to project

pipm install pkg-name
pipm install pkg-name --dev # as development dependency
pipm install pkg-name --test # as testing dependency

3. Removal of packages

Remove one or more packages. Their dependencies will also get uninstalled. No orphaned packages.

pipm uninstall pkg-name

4. update all your dependencies in requirements.txt

pipm update
pipm update --dev

Features

No new set of files. *-requirements.txt works like the lockfile with pinned versions
Just a wrapper around the standard pip's install/uninstall command. So all the cli options will work
Handles multiple requirements files and setup.cfg stores abstract dependencies.

Commands

1. install

a wrapper around standard pip install command and accepts all the standard options

Below are the things that pipm brings to the table

Extra functionality
- when package names are given it will be saved to the requirements.txt file in the current directory. If you have requirements directory structure with base.txt inside then that file will be used. Otherwise it will create one in the current directory.
- when no package name is given then it is equivalent to -r requirements.txt and it will install all requirements from the current directory
Additional options: It will search for the matching one in the following pattern <name>-requirements.txt or requirements/<name>.txt or requirements-<name>.txt the below saves to file when package name given otherwise equivalent to passing requirements file name.
1. --dev - saves to development requirements
2. --prod - saves to production requirements
3. --test - saves to testing requirements
4. --doc - saves to documentation requirements
5. --env <name> - if you have any special set of requirements that belong to a separate file you could pass the name here.

2. uninstall

a wrapper around standard pip uninstall command
alias rm is available
when uninstalling a package, this command also checks packages that are no longer required by any of user installed packages and removes them
ofcourse it removes the packages from requirements files

3. update

new command
equivalent to calling pip install with --upgrade flag
update a single package or the whole environment when no argument given.
by default the packages are updated interactively
- set --auto-update to disable this

4. save/freeze

extends the standard freeze command to save the currently installed packages

Development

clone the repository and create new virtualenv

git clone [email protected]:jnoortheen/pipm.git
cd pipm
pew new pipm -a .
pip install -r dev-requirements.txt

to test from local sources

pip install -e .

Commit message should follow this style-guide.

Testing

run invoke test from the root directory.

TODOs:

rm will check whether a package is present in setup.cfg

pipm's People

Contributors

Stargazers

Watchers

Forkers

sevviyal-ltd

pipm's Issues

TypeError: argument of type 'instance' is not iterable

Python 2.7.10 (default, Feb 7 2017, 00:08:15)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin

pip install pipm
...
Successfully built pipm
Installing collected packages: pipm
Successfully installed pipm-18.1

>>> pipm install bunch

Traceback (most recent call last):
  File "/Users/matt/.envs/xxx/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 143, in main
    status = self.run(options, args)
  File "/Users/matt/.envs/xxx/lib/python2.7/site-packages/pipm/commands.py", line 145, in run
    file.save(env=options.req_environment, user_reqs=result)
  File "/Users/matt/.envs/xxx/lib/python2.7/site-packages/pipm/file.py", line 253, in save
    setup_cfg.add_requirements(user_reqs, env)
  File "/Users/matt/.envs/xxx/lib/python2.7/site-packages/pipm/setup_cfg.py", line 96, in add_requirements
    update_config(config, base_key, key, reqs)
  File "/Users/matt/.envs/xxx/lib/python2.7/site-packages/pipm/setup_cfg.py", line 46, in update_config
    if base_key not in config:
TypeError: argument of type 'instance' is not iterable

CI failing

https://travis-ci.com/jnoortheen/pipm/jobs/173650146

__________________________ test_get_orphaned_packages __________________________

patched_dists = <MagicMock name='get_distributions' id='140657923192528'>

    def test_get_orphaned_packages(patched_dists):

>       freqs = operations.get_orphaned_packages(['pytest'])

/home/travis/build/jnoortheen/pipm/tests/test_operations.py:26: 

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

/home/travis/build/jnoortheen/pipm/pipm/operations.py:70: in get_orphaned_packages

    for r in dists[dist].requires():

/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py:2635: in requires

    dm = self._dep_map

/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py:2899: in _dep_map

    self.__dep_map = self._compute_dependencies()

/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py:2908: in _compute_dependencies

    for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:

/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py:2890: in _parsed_pkg_info

    metadata = self.get_metadata(self.PKG_INFO)

/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py:1410: in get_metadata

    value = self._get(self._fn(self.egg_info, name))

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pip._vendor.pkg_resources.PathMetadata object at 0x7fed7992bad0>

path = '/home/noor/.local/share/virtualenvs/pipm/lib/python3.7/site-packages/lazy_object_proxy-1.3.1.dist-info/METADATA'

    def _get(self, path):

>       with open(path, 'rb') as stream:

E       IOError: [Errno 2] No such file or directory: u'/home/noor/.local/share/virtualenvs/pipm/lib/python3.7/site-packages/lazy_object_proxy-1.3.1.dist-info/METADATA'

/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py:1522: IOError

requirements-dev.txt instead of dev-requirements.txt

Using filename suffixes instead of prefixes,
requirements-dev.txt instead of dev-requirements.txt,
will make all requirements-*.txt close together in project root.

I understand that is more complicated when doing in compatible with older project.

Incompatibility with recent pip

It might be useful to add pip-shims or pip-api as a layer to avoid breakages when pip changes its internals, which they do every release ;-)

[   46s] + py.test-2.7 --ignore=_build.python2 --ignore=_build.python3 --ignore=_build.pypy3 -v
[   47s] ============================= test session starts ==============================
[   47s] platform linux2 -- Python 2.7.16, pytest-4.6.5, py-1.8.0, pluggy-0.13.0 -- /usr/bin/python2
[   47s] cachedir: .pytest_cache
[   47s] rootdir: /home/abuild/rpmbuild/BUILD/pipm-19.0.0, inifile: pytest.ini
[   48s] collecting ... collected 22 items / 4 errors / 18 selected
[   48s] 
[   48s] ==================================== ERRORS ====================================
[   48s] ______________________ ERROR collecting pipm/__main__.py _______________________
[   48s] pipm/__main__.py:11: in <module>
[   48s]     from pipm.commands import (
[   48s] pipm/commands.py:3: in <module>
[   48s]     from pip._internal.commands import UninstallCommand, FreezeCommand
[   48s] E   ImportError: cannot import name UninstallCommand
[   48s] ______________________ ERROR collecting pipm/commands.py _______________________
[   48s] pipm/commands.py:3: in <module>
[   48s]     from pip._internal.commands import UninstallCommand, FreezeCommand
[   48s] E   ImportError: cannot import name UninstallCommand
[   48s] ___________________ ERROR collecting tests/test_commands.py ____________________
[   48s] tests/test_commands.py:3: in <module>
[   48s]     from pipm import commands
[   48s] pipm/commands.py:3: in <module>
[   48s]     from pip._internal.commands import UninstallCommand, FreezeCommand
[   48s] E   ImportError: cannot import name UninstallCommand
[   48s] ___________________ ERROR collecting tests/test_commands.py ____________________
[   48s] ImportError while importing test module '/home/abuild/rpmbuild/BUILD/pipm-19.0.0/tests/test_commands.py'.
[   48s] Hint: make sure your test modules/packages have valid Python names.
[   48s] Traceback:
[   48s] tests/test_commands.py:3: in <module>
[   48s]     from pipm import commands
[   48s] pipm/commands.py:3: in <module>
[   48s]     from pip._internal.commands import UninstallCommand, FreezeCommand
[   48s] E   ImportError: cannot import name UninstallCommand
[   48s] !!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!
[   48s] =========================== 4 error in 0.61 seconds ============================

setup.cfg and install_requires

Package dependencies are often put in the install_requires field of setup.cfg like:
https://github.com/eight04/ptt-article-parser/blob/de6639fbd0797e7c2fe9df853ed82cbeeed49bb4/setup.cfg#L30-L32

Is it possible to sync the version number with install_requires after pipm install/uninstall? For example:

# requirements.txt
a==0.1.0
b==0.2.0

# setup.cfg
a~=0.1.0
b~=0.2.0

After running command pipm update a

# requirements.txt
a==0.2.1
b==0.2.0

# setup.cfg
a~=0.2.1
b~=0.2.0

Doesn't work inside pyenv virtualenv

Installed this with the hope of moving away from pipenv

Running within pyenv virtualenv for python 3.7 throws a ModuleNotFound error.

Traceback (most recent call last):
  File "/Users/developer-3/.pyenv/versions/leadflo/bin/pipm", line 5, in <module>
    from pipm.__main__ import main
  File "/Users/developer-3/.pyenv/versions/leadflo/lib/python3.7/site-packages/pipm/__main__.py", line 20, in <module>
    from pipm.commands import (
  File "/Users/developer-3/.pyenv/versions/leadflo/lib/python3.7/site-packages/pipm/commands.py", line 8, in <module>
    from . import file
  File "/Users/developer-3/.pyenv/versions/leadflo/lib/python3.7/site-packages/pipm/file.py", line 8, in <module>
    from pip._internal.download import PipSession, get_file_content
ModuleNotFoundError: No module named 'pip._internal.download'

$ pip --version     
pip 20.1 from /Users/developer-3/.pyenv/versions/leadflo/lib/python3.7/site-packages/pip (python 3.7)