pip install -e . equivalent? about poetry HOT 207 CLOSED

I don't know how you develop, but I never need pip install -e .. I either upload my packages to PyPI or a private index, or I install from a git repository

Ugh, that is not developing, that is installing a finished package.

These are roughly equivalent, in your src dir:

pip install -e .
python setup.py develop

It's called "installing a package in development mode". They are the starting point of beginning development. If you are not familiar with that, then I honestly do not understand how you do development.

I know two ways of doing development without it:

export PYTHONPATH=
or in your code:
import sys
sys.path.append(....)

Are you recommending this? (I hope not)

from poetry.

Another use case is when you’re trying to fix something in a library used by your project. From your project’s virtualenv, you run pip install -e path/to/library so that every change you make in the library repo is directly picked up in your project where you are testing the results.

from poetry.

I see.

The thing is poetry is a dependency/project management tool and is not intended for such use cases.

However, if you really want to to this, you can build your project poetry build and take the setup.py file from the generated tarball and then you should be able to do pip install -e .

from poetry.

No, this is not possible right now. And I don't think I will add an equivalent since it always seemed like a hack to me. But I might be wrong.

Why do you need pip install -e .? So I can see if there is a "poetry way" to do what you want.

from poetry.

I'll chime in and say I also use pip install -e . when doing local development. It is helpful for a couple reasons:

1. It puts my project on the Python path. I don't have to depend on being in a specific directory for things to work which feels hacky to me.
2. It sets up scripts and entry points I wouldn't have otherwise.

from poetry.

@kootenpv I just tested and copying the setup.py file works.

I don't know how you develop, but I never need pip install -e .. I either upload my packages to PyPI or a private index, or I install from a git repository.

from poetry.

No, I don't modify sys.path(). And please don't be condescending.

So, if I develop a library (or an application for that matter) I put myself in my project directory. At this point I have access to my module: I can import it in my tests to test it.

You can test this yourself: clone the poetry repository place yourself in it install the dependencies with poetry install and execute poetry run pytest tests/ and everything will work. You do not need pip install -e .

If you need it in another project, and that is why you install it in develop mode, the you have dependencies between project and in this case I recommend to use a private index and release preversions or installing from git repository and updating when needed.

from poetry.

Well, poetry can help you develop local packages by orchestrating your dependencies, sure. But it is not meant to support the use case of having only local packages that depend on each other since this is not the best practice that poetry tries to endorse.

However, if you are willing to make a Pull Request, I would gladly take a look at it.

from poetry.

I often depend on some local packages, for which I want to be able to change some code and not have to reinstall.

from poetry.

I might be off the mark here, but in my experience, I usually use pip install -e . in order to allow my acceptance tests to see the python project I am testing. Is that what we are talking about here?

from poetry.

@merwok I understand your point, but most of the time you don't need it, in my opinion.

Regarding you last example, this will be supported in the next feature release since it will add the ability to install from a directory.

I know that there is a lot of different workflows out there, but poetry can't support them all. It tries to enforce some of them to help make the dependency management and packaging in Python easier and for that it has to be somewhat opinionated.

I am not saying it will never make it into poetry but since I am the sole developer I have to focus on a workflow that will suit most cases to develop applications and libraries.

from poetry.

Release 0.10.0 adds a new develop command which does something similar to pip install -e ..

The install command now supports a --develop option which installs path dependencies in development/editable mode. 

from poetry.

@kootenpv Don't get me wrong, I can understand why you would use pip install -e, especially if you want to debug a dependency.

And, like I said, directory dependencies will be introduced in the next feature release so, with them, you will be able to reproduce this behavior.

from poetry.

I don't see why you would need pip install -e . for your tests to work. If you are inside a virtualenv your dependencies should be picked up by your tests.

In poetry that would be:

poetry install  # install your dependencies
poetry run pytest tests/

from poetry.

There are many communities of Python users, using different categories of development tools. Depending on practices (e.g. code in src dir vs code at repo root) or tools (e.g. Django runserver vs. Pyramid pserve), you can rely on «current dir is sys.path[0]» or you always install in develop/editable mode.

from poetry.

@sdispater Thanks for your way of developing (private index / from git), I wasn't aware of another way.

I can agree that under some conditions it is more proper and might lead to fewer bugs!

In my case, I do a lot of data science and "live" in Interactive python. It's very convenient when I can just do pip install -e . once and I'm just able to change the code so that in another module it will be picked up and see the results quickly (rather than having to run any install command again). That is the step before writing tests.

I understand you are hesitant from your perspective as it does not seem like a really clean use case, and it looks like the solution would have to be a bit hacky?

Still, I hope you could guide us and be accepting of a solution that might help a few (and not bother others)!

Currently the only way I see is something like:

1. poetry build
2. find the build package (?)
3. extract it
4. move the setup.py a level up
5. find the python version (?) used by poetry and use it like python setup.py develop

Does this sound like the best way?

from poetry.

Sorted this one out. Should've put an empty __init__.py file in tests.

from poetry.

It now also does not allow me to use it in the interactive interpreter :/

I tried using poetry build, then copy out the setup.py to the root level but that doesn't work. I'm very disappointed :(

Until this becomes important, I wish there'd be a nice hack. I'm wondering though how do you develop - how do you avoid having to constantly reinstall?

from poetry.

@kootenpv Exactly.

I think it would take the form:

[tool.poetry.dependencies]
dependency = { "path" = "relative/path/to/folder/" }

And with the equivalent of -e:

[tool.poetry.dependencies]
dependency = { "path" = "relative/path/to/folder/", develop = true }

Regarding the issue when the dependency is also a poetry package: at the moment, there won't be another solution but to generate a temporary setup.py to pip install it. I am not a fan of it but for now it will do, at least until the ability to specify a build backend in pyproject.toml is supported (see https://www.python.org/dev/peps/pep-0517/).

from poetry.

Sorry to bring this up again, but after reading everything here and also searching on the latest documentation I still cannot find a way for specifying develop mode for a dependency.

I see you mentioned here: #34 (comment) the develop = true and I tried but that does'n work.

dependency = { "path" = "relative/path/to/folder/", develop = true }

So I have a questions @sdispater

Lets say you are splitting Poetry in two projects poetry and poetry-core (which I think you are)

How are you currently doing to make core installable in development mode for the poetry projetct?

My usecase is similar, I am developing an application my-app and I am also creator/maintainer of dynaconf I want to specify on my-app/pyproject.toml the dynaconf dependency pointing to my local dynaconf so if I change something on dependency the application gets the changes immediately.

The same happens on my work project at Red Hat (which I may advocate the adoption of poetry). The project is split between multiple plugins [pulpcore, pulpfile, pulprpm, ..] etc... we may need to be able to install everything in editable mode just like we do using pip install -e

Is there an alternative?

Thank you.

from poetry.

@winni2k Not at the moment, no. For now, poetry assumes each dependency is located in a remote repository (PyPI or a private one) or is a VCS dependency, and does not support local packages.

That might change in the future, however but I can't give you an ETA.

I am trying first to support the most common use cases, like installing from PyPI before diving into more specific workflow, especially if they are not the recommended way of managing packages.

from poetry.

@sdispater First of all thanks for poetry, I think it's looking fantastic as I dive into it more.

The big use-cases for editable-mode installs the way I see it are:

1. Making entry points and console scripts available in the virtualenv. As you've said yourself, and I agree with, these are properly covered by poetry run and poetry script which should make these available inside that environment. If entry points are not discoverable inside poetry run python then I'd be discouraged but I haven't tested this yet.

2. Using your project out-of-place... as you've said yourself your approach requires you to either munge the PYTHONPATH or to develop within the source folder. Editable mode installs a .pth file in the site-packages, thus placing your project on the path as long as you have access to the python binary.

3. Installing and working on third-party dependencies in your project. This, again, you're handling via [tools.poetry.dependencies] with editable=true which is probably an acceptable way to go although it's unfortunate that you need to temporarily edit a file that's generally version controlled in such a way that you probably don't want to commit. This is probably where poetry install --develop comes into play.

I may be missing something but point 2 seems to be the big case that's unaddressed.

from poetry.

from poetry.

I'm not trying to be rude, but I'd like to point out that people are still trying to sort this issue out.

As I've said already the discussions about standardizing editable installations are here:

• Pip, pyproject.toml and debug builds
• Specification of editable installation
• Third try on editable installs
• Next steps for editable / develop proof of concept

Anyone is free to take over. There is not much that can be done here in poetry in particular, until these discussions conclude.

from poetry.

Where is the documentation for this could be found?

from poetry.

just do poetry install it will do pip install -e .

from poetry.

I think there's one more usecase that is not covered with current feature (based on reading the above comments): Installing a poetry-managed library which is a dependency for a project that is not poetry-managed. I think that the install --develop flow only works from the project, to install dependencies in develop mode, right? When the project is not managed by poetry, you would need a way to run some poetry command in the library directory, with the project's virtualenv activated, to install the library in develop mode. I suspect that this is what the (now deprecated?) develop command did?

from poetry.

Is it related to #47 ?

If i summarize it, the need is that a poetry managed project should be installable as editable. e.g. i want to debug that library used by my project, i git clone it and poetry install -e the path ?

from poetry.

So, this issue is marked "closed" and yet it seems the command mentioned above develop is now deprecated. has pip install -e . survived through some other mechanism? or is there another thread discussing this?

from poetry.

@kolypto Thanks! ..but, wow. This issue has really encouraged me to steer clear of this tool. The Developer In Charge seems to ignore what looks like a strongly supported request.

from poetry.

Why don't people just use a .pth file? It's a bit more work, yes. But as far as I can tell, it should give good results in most cases, poetry or not.

..but, wow. This issue has really encouraged me to steer clear of this tool. The Developer In Charge seems to ignore what looks like a strongly supported request.

@eode

Why blame poetry or its maintainers, here? Looks like you could as well blame pip, since you want pip to do the editable installation. But truth is: pip is not to blame either.

Well... from my point of view this is a non issue. Editable installations were always a bit of a hack. A very convenient hack, but still...

As for the actual, long-term, non-hacky fix: it can not come from poetry alone (nor pip actually, editable installations were always a setuptools thing at the end of the day). The best placed organization to move this forward is PyPA. They had discussions regarding this topic (stalled, because people have other priorities, just like anyone else), and I will link some of them here for those of you who want to read on and hopefully give a hand and make this happen:

• Pip, pyproject.toml and debug builds
• Specification of editable installation
• Third try on editable installs
• Next steps for editable / develop proof of concept

For some of the use cases, like maybe for @albertogomcas (as far as I understood), maybe try something like

[tool.poetry.dependencies]
# ...
my-lib = "^0.1.0"

[tool.poetry.dev-dependencies]
# ...
my-lib = {path = "../my-lib", develop = true}

This will install my-lib by its local path as editable. But when building the distributions, it doesn't use the path dependency.

The full example is here: https://github.com/sinoroc/foosball
Also related: #1168 (comment)

from poetry.

Or maybe the goal is just to have project console scripts installed in virtualenv ?

from poetry.

@sdispater I don't understand why i have to put the develop mode in the file. If i want to debug a poetry based dependency, i don't want to modify my pyproject.toml for that, just to poetry install --develop ../my-deps-fork/.

About console scripts, I saw poetry script .... I find odd to not use scripts the same way i will do in production, especiallty when scripts may be calling them each-other, or if i have Makefile or shell script calling the script.

For example, I use https://github.com/amoffat/sh to write functionnal tests of https://github.com/dalibo/ldap2pg . see https://github.com/dalibo/ldap2pg/blob/master/tests/func/test_sync.py#L34-L36 . When developing, it just use virtualenvs scripts. On CI, it tests scripts installed from RPM, without modification nor configuration. It's harder to do this using poetry.

I still think that some editable mode of poetry is required for the comfort of developing project with poetry.

from poetry.

@finswimmer

$ git clone https://github.com/rochacbruno/dynaconf 
Cloning into 'dynaconf'...

$ git clone https://github.com/rochacbruno/flask 
Cloning into 'flask'...

$ poetry new myapp
Created package myapp in myapp

$ cd myapp
$ poetry env use 3.8

$ poetry add ../flask
... ok!

$ poetry add ../dynaconf 
[ValueError]
'/home/rochacbruno/Projects/python/dynaconf' does not start with '/tmp/poetry_dev_mode/myapp'

from poetry.

btw after NixOS/nixpkgs#105593 I realized I could do without editable install for the current package, I just add PYTHONPATH=".:$PYTHONPATH"

from poetry.

@sdispater

You can test this yourself: clone the poetry repository place yourself in it install the dependencies with poetry install and execute poetry run pytest tests/ and everything will work. You do not need pip install -e .

Hi! Could you please clarify how you run tests with pytest without installing poetry in the virtualenv? I can't import my package from pytest tests because it's not installed and resides one level higher than the tests.

It seems like poetry has the same source layout, but for some reason the imports work (although the tests fail). Am I missing something?

UPD: Even the pytest docs say that you can only run tests against an installed version of your package under this layout: https://docs.pytest.org/en/latest/goodpractices.html#tests-outside-application-code

from poetry.

poetry build and take the setup.py

What do you think about some option to generate the setup.py file directly?

from poetry.

@sdispater I found that the problem I mentioned above regarding the develop=true is because it uses --no-deps to install dependencies so installation fails if the other project is requiring any library. Why is it using --no-deps ?

from poetry.

There's no setup.py, but you can get one by extracting it from the source .tag.gz file:

$ poetry build && tar -xvf dist/*.tar.gz --wildcards --no-anchored '*/setup.py' --strip=1

After this, the folder can be installe with pip install -e. Cheers! :)

from poetry.

from poetry.

@aucampia I don't think you need the moves if you add

[build-system]
requires = ["setuptools", "poetry_core>=1.0"]
build-backend = "poetry.core.masonry.api"

to your pyproject.toml.

from poetry.

The pyproject.toml is what orchestrates your dependencies when using poetry that's why I suggest putting there.

I see pyproject.toml as a way to share a reproduceable set of dependencies for both runtime and development. Actually i would create a third env in between : CI. having a reduced set of dependency on CI is nice. e.g. hupper is not needed on CI.

I may be wrong :-)

from poetry.

Apparently not - full example here: https://github.com/phiresky/poetry-install-self-issue

pipenv install && pipenv run python a/b.py works

poetry install && poetry run python a/b.py throws ModuleNotFoundError: No module named 'c'

from poetry.

Took me a few times to get it to work - make sure to clear your .venv dir and poetry.lock.

Add develop = true to your dependency and it will be installed as an "editable install".

To confirm it is an editable install, you should see a <library-name>.pth file in
.venv/lib/python3.8/site-packages/.

Aside: The entry in this file gets added to the sys.path module search path by Python, which is how editable installs work. There is no symlinking like in Node.js.

[tool.poetry.dependencies]
python = "^3.8"
foo = {path = "../../libs/foo", develop = true}

Also if you create a new package with poetry init -n, make sure you add a dir with the same name as your package and an __init__.py file (e.g. foo/foo/__init__.py) or it will error saying foo is not a package. If you don't want to use the same dir name as your package you can modify with https://python-poetry.org/docs/pyproject/#packages. Alternatively, just use poetry new foo to set this up.

from poetry.

.pth are neat - I'm glad I learned about them and they mostly solved my issue:

By adding pa.pth (containing /path/to/pa) in pb/venv/lib/python3.X/site-packages/, importing PA works in PB's venv.

My (non-blocking) remaining issues are:

• This is rather awkward to explain to a third party willing to use PA who is not expert in python packaging.
• Since PA is not formally installed by pip, pycharm (and possibly other IDEs) complains that a requirement is missing.
• It turns out that PA also has a console entry point. Since it's not pip installed, python -m pa must be used instead of a mere pa.

Edit: pycharm does recognise the requirement. It didn't initially, until it somehow reindexed the venv.

Now, building on @kolypto's answer, the following a script (in PA) would replace the usual pip install -e . (in principle -- I haven't tested it):

poetry build
tar -xvf dist/*.tar.gz --wildcards --no-anchored '*/setup.py' --strip=1
pip install -e .
rm setup.py

Is that something that poetry could automate, as a workaround until pip install -e is fixed to handle pyproject.toml?

from poetry.

Just found out about dephell project register.

The way I read it, it could be very useful for many of the use cases in this discussion. Still not as good as native support in poetry itself (obviously, but as I already stated it probably will not happen until editable installations are standardized). In particular it removes the need for manual extraction of setup.py files and such things.

In a simple test I just did, I was able to install an external poetry-based library as editable in my current virtual environment with just one command:

dephell project register --from=pyproject /path/to/poetry/project

I am sure there are ways to improve on this even more, dephell seems extremely flexible and configurable.

from poetry.

Here's a dephell command that worked for me:

# being in dependant project directory
dephell project register --level=DEBUG \
--venv=$(poetry env info -p) \
--from-format=poetry --from-path=pyproject.toml \
../dependency-project-i-want-to-install-editable

from poetry.

I opened a bug in dephell's ticket tracker: dephell/dephell#476.

from poetry.

@abey79 @kolypto

sadly your suggestion does not quite work, I also tried something similar:

poetry build --format sdist
tar -xvf dist/*.tar.gz -O '*/setup.py' > setup.py

$ pip3 install --user --editable .
Obtaining file:///home/iwana/syncthing/sw-wpw/d/gitlab.com/aucampia/templates/python-poetry
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Installing collected packages: aucampia.template.poetry
  Running setup.py develop for aucampia.template.poetry
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/iwana/syncthing/sw-wpw/d/gitlab.com/aucampia/templates/python-poetry/setup.py'"'"'; __file__='"'"'/home/iwana/syncthing/sw-wpw/d/gitlab.com/aucampia/templates/python-poetry/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps --user --prefix=
         cwd: /home/iwana/syncthing/sw-wpw/d/gitlab.com/aucampia/templates/python-poetry/
    Complete output (3 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'setuptools'
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/iwana/syncthing/sw-wpw/d/gitlab.com/aucampia/templates/python-poetry/setup.py'"'"'; __file__='"'"'/home/iwana/syncthing/sw-wpw/d/gitlab.com/aucampia/templates/python-poetry/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps --user --prefix= Check the logs for full command output.

It works fine without --editable, and I think it should actually work also with --no-use-pep517, but it does not:

$ pip3 install --no-use-pep517 --user --editable .
Obtaining file:///home/iwana/syncthing/sw-wpw/d/gitlab.com/aucampia/templates/python-poetry
ERROR: Disabling PEP 517 processing is invalid: project specifies a build backend of poetry.masonry.api in pyproject.toml

It does work however if you remove pyproject.toml:

$ rm pyproject.toml 
rm: remove regular file 'pyproject.toml'? y
removed 'pyproject.toml'

$ pip3 install --user --editable .
Obtaining file:///home/iwana/syncthing/sw-wpw/d/gitlab.com/aucampia/templates/python-poetry
Installing collected packages: aucampia.template.poetry
  Running setup.py develop for aucampia.template.poetry
Successfully installed aucampia.template.poetry

from poetry.

I had to add --wildcards to the previous tar command so that it becomes poetry build --format sdist && tar --wildcards -xvf dist/*.tar.gz -O '*/setup.py' > setup.py.
As I am on nix, I dont have pip in the nix-shell though but source "$( poetry env info --path )/bin/activate" works (run poetry install first).

from poetry.

I guess this is a bit of a style question, but I like to build a wheel for local packages and use pip to install them into my venv. Is that the workflow that is intended by poetry?

from poetry.

@bersace If you want to test you console scripts, it's possible via the poetry script my-script. For example for poetry that would be poetry script poetry.

Also, I understand the use case of pip install -e path/to/some/dependency/ to allow debugging a dependency since you won't have to reinstall each time you make a modification. That's why it will be possible in the next feature release.

However, I do not think pip install -e . (or python setup.py develop) is that useful and as such introducing an equivalent in poetry is not something I plan on adding at the moment.

Now, that being said, if someone can give me a compelling use case where it's necessary, I will gladly change my view on the subject.

from poetry.

Thanks a lot for the nice surprise!

A couple of observations:

• the --develop option is missing from the CLI command install (good: badusage is thrown, and poetry develop documentation exists)
• I thought I can add a folder containing a local package and install it using poetry install --develop, but that's not how it works?
• poetry develop allowed me to install the "current" package, and import it, but only if I'm in that folder (e.g. in ~/mypackage. You cannot import mypackage from the example location ~/otherpackage)

Maybe I'm misunderstanding the usage?

from poetry.

@sdispater Sorry, I still don't quite understand why I can't import my package after running poetry develop. When I run poetry develop, the package gets installed in the current venv, right? So, if I import it from a script in a project's subfolder and run the script with poetry run foo/bar.py, it should work, since poetry run uses the same venv.

It doesn't work however. Is it a bug or expected behavior?

P.S. Congrats on the big release and thanks a ton for your work!

from poetry.

@p5a0u9l The behavior on installing the package in an editable mode (poetry develop) is now part of poetry install.

from poetry.

Still don't understand how pip install -e . is supposed to be solved:

[tool.poetry]
name = "cooli"
...

I have these files:

a/b.py

import c.x
print(c.x.test)

c/x.py

test = "hi"

I tried multiple things:

poetry install --develop cooli
poetry install --develop .
poetry add cooli --path . (just shows AssertionError)
poetry add .

With pipenv this works completely fine. With poetry, how is this supposed to work?

I always just get

Traceback (most recent call last):
  File "a/b.py", line 1, in <module>
    import c.x
ModuleNotFoundError: No module named 'c'

from poetry.

@finswimmer looks like poetry allows only relative paths, when I try absolute paths (when my deps are in other paths and not the same tree as myapp). this problem happens when I try to set absolute path, is it not allowed?

from poetry.

Where is the documentation for this could be found?

@lig I opened a PR to mention this in the docs: #2780

from poetry.

@abey79 In your case, what is the current build backend of pb? You mention it is not poetry. Is it setuptools?

I would think that the suggested trick should work for a dependency with a setuptools build backend as well. I have not tested it explicitly, so I might be wrong.

What about the following?

poetry install
poetry run pip uninstall pb
poetry run pip install --editable /path/to/pb

I know it is annoying to have to type it all and one might not want to consider it as a long term solution, but... does it allow you to work?

Did you get the chance to try and experiment with .pth files? More or less:

echo '/path/to/pb' > '/path/to/pa/venv/lib/site-packages/pb.pth

from poetry.

I gave it a try but the command above didn't work for me. This one worked:

cd /some/project/which/depends/on/another/poetry/project
dephell project register --venv venv --from-path pyproject.toml --from-format poetry /path/to/poetry/dependency/project/

Here, venv is the name of the virtual environment of the dependant project (which happens to not be poetry managed).

from poetry.

I understand the frustration, but I've come to accept the poetry project's reasoning. This is really up to Python developers to sort out. I explained here how I do an editable install, and where you can follow along Python's progress patching this up.

from poetry.

@sinoroc no, I haven't – yet. I was checking first if @kbakk would know, being an actual user of this approach.

from poetry.

Since I do actually want the command provided by the poetry package available globally I have resorted to doing this:

poetry build --format sdist
tar -xvf dist/*.tar.gz -O '*/setup.py' > setup.py
mv {,not-}pyproject.toml
pip3 install --user --editable .
mv {not-,}pyproject.toml

or all at once:

poetry build --format sdist && tar -xvf dist/*.tar.gz -O '*/setup.py' > setup.py && mv {,not-}pyproject.toml && pip3 install --user --editable .; mv {not-,}pyproject.toml

It is nasty, but it works.

EDIT: see further down for a better option.

from poetry.

What is odd to me is that it appears this used to work fine. My example. I started building a project called uvicore 6 months ago using whatever poetry version that was back then. I had a second project called wiki that had a pyproject.toml dependency of uvicore = {path = "../uvicore"} as you would imagine. Everything worked perfectly. I could edit uvicore code in one VSCode and changes were instantly shown in the wiki project just like with pip install -e ../uvicore/. Even when running pip list from within the wiki environment it shows the symlink properly uvicore 0.1.0 /home/xyz/Code/uvicore/uvicore. And listing the contents if my ~/.cache/pypoetry/virtualenvs/wiki-whatever/lib/python3.9/site-packages/ it did NOT show the uvicore folder as if it built it one time and not used the symlink live. Everything worked perfectly for months. Then a few weeks ago I upgraded poetry. And today I wiped my envs and re made them all. Now it builds and copies the build of uvicore into ~/.cache/pypoetry... and a pip list does NOT show it as symlinked. Now nothing is live at all. Something must have changed somewhere as this has worked fine for me up until I re did all my envs.

from poetry.

Ah sorry, with local I mostly meant "developing local packages". I do believe poetry is also for helping with that? It sounds like barely any work to automate that part and create e.g. the poetry develop command.

from poetry.

I'm not sure, but when you share a module in git, with other dev members, it's nice when it is updated automatically on pull. Or is this automatically managed in pycharm or something? I use emacs without virtualenvs but just simply different python versions.

from poetry.

In Pycharm you set the environment by choosing a python interpreter in your directory tree. Packages are discovered from that AFAIK.

from poetry.

I think I get what you mean: you want to enable the option for a package (A) to list a development version of another package (B).

I perhaps see a problem though.

I believe you're saying you could list B as a requirement in A in the new version by listing pip install -e path/to/some/dependency/ in A's poetry file (or some similar syntax, maybe just relative path)?

Isn't the problem then that if both are poetry packages, A will not be able to pip install -e B since poetry does not generate a setup.py for B?

from poetry.

@bersace The pyproject.toml is what orchestrates your dependencies when using poetry that's why I suggest putting there.

Now, that being said, I can add a --develop option to install (and to add too) to tell poetry: I want this dependency from my dependency set to be installed in development mode. The thing is with that option is if you forget to set it it will install the dependency in non editable mode.

Regarding an editable mode for poetry, I could add it also but this is, at the moment, not high on my priority list. I am the sole developer at the moment so I have to prioritize the feature I want to implement in poetry. But if someone wants to tackle this , they are welcome to and I'd gladly review it.

from poetry.

Indeed, it is very useful to have different sets for runtime, test and dev dependencies. (Also CI bootstrap and server-only deps!)

from poetry.

@bersace that's what extras as for, IMHO. Add an extra for testing that doesn't have hupper, but does have all of the testing extras for example.

from poetry.

@mmerickel Thanks for the kind words and the detailed use cases.

Regarding entry points, like you said, poetry script takes care of it. However, there is a limitation: you have to use it from project's directory. So, you won't be able to install your command "globally". This is where a develop command could be useful.

And regarding installing dependencies in editable mode, I think that a --develop option is the way to go. However, there are some details to sort out, like if you do poetry install --develop my-dependency and later simply do poetry install should the editable mode be disabled?

from poetry.

@sdispater So I just tested it:

pip install -e .
pip instal my_package

In my case it looks like it was installing it again, but the binding shows it is still using the develop version.

It looks like pip does not care about version when you just do pip install my_package (we knew that, but also not with editable/dev already installed). Whenever you write pip install my_package==1.0.0, it will install it from pip if it doesn't have it locally.

In case you are talking about whether it remove the editable mode status in the file when you poetry install, I guess you wouldn't want that, but maybe a warning that an editable package is being linked?

As for being allowed to publish to pypi - I think that should not be allowed. Or maybe an input You have an editable package X, are you sure want to publish [y/N]?

from poetry.

I think #47 (path dependencies) should be able to solve this use case now?

from poetry.

@radix #47 is a first step towards it but path dependencies cannot be installed in editable mode at the moment. This is something planned for the next feature release.

from poetry.

@sdispater For the editable project mode from point 2 above, I suppose it's important to ask whether poetry has goals of providing a PEP 517 compliant build system for projects in the future? Presumably if that were the case then this is solved via pip install -e <path> on the path containing the pyproject.toml which would tell poetry to install itself in editable mode. Similarly a pip install <path> (not editable) is effectively equivalent to poetry build && pip install dist/*.whl in a future when you can upload a project created with poetry to PyPI without needing to generate a fake setup.py. I know this is futuristic but it helps me understand where poetry is going and whether it even makes sense for poetry to support the develop mode - although I suppose it could support poetry develop and in the future embrace the pip integration. This seems like a pretty logical step for poetry as it already has ambitions of making projects redistributable via poetry build and poetry publish.

from poetry.

@mmerickel Yes, I intend to support a build system. I am not sure it will be integrated directly into poetry since I don't want people to have to depend on the whole poetry's codebase to build a project. So it will likely be another package whose sole purpose will be to build a project from a poetrypyproject.toml file which is more or less the purpose of the masonry package inside poetry so it might be extracted from there.

from poetry.

With the introduction of directory dependencies, I thought poetry add --dev --path . mypackage would be possible and would completely solve this issue. This would install the package itself in the virtual environment from the current directory.

However, you can't do that right now (sample taken from my project Cliar):

$ poetry add --dev --path . cliar
[InvalidProjectFile]
[dev-dependencies.cliar] {'path': '.'} is not valid under any of the given schemas

add [-D|--dev] [--git GIT] [--path PATH] [-E|--extras EXTRAS] [--optional] [--allow-prereleases] [--dry-run] [--] <name> (<name>)…

(The line cliar = {path = "."} is added to pyproject.toml though.)

@sdispater Is it an issue to report or a design decision?

P.S. My two cents on the topic, and why I need poetry add --dev --path . cliar to work. Like your own Cleo, Cliar is a package to create CLIs. Because the purpose of the tool is to generate a commandline interface from Python code, I'm testing it this way:

• Create a .py file with the interface definition. This file obviously needs to be able to import cliar in order to create the CLI to test.
• In test files, I'm calling the .py files with subprocess.run like so: run('python clitest.py arg1 arg2', shell=True).
• I'm checking the output with Pytest's capfd fixture.

It would be really nice to put all the tests in tests folder, put the sample .py files in a subfolder, and discover them in the tests using pytest-datadir plugin.

However, I currently cannot do that. Whereas test_*.py files can import cliar, the sample files can't since the cliar package is not installed.

The workaround I use now is place the tests and the sample in the root dir. But I'd rather keep my tests in an isolated directory.

If there is a way to solve my problem without poetry add --dev --path . cliar I'm not seeing, please advise.

Thanks!

from poetry.

@kootenpv You must pass values to the --develop option. Let's say you have my-pkg1  and my-pkg2 that are path dependencies, you would do:

poetry install --develop my-pkg1 --develop --my-pkg2

poetry develop installs the package in development mode for the project's virtualenv, so that it's available when executing poetry run. If you manage your virtualenvs yourself then it will be installed in the current activated virtualenv.

from poetry.

@moigagoo Yes it should work. So if it's not it's likely a bug.

from poetry.

@sdispater made a separate issue #162

I think this one can be closed, since the equivalent of pip install -e . does exist now.

from poetry.

Do you have the necessary __init__.py files?

from poetry.

__init__.py files aren't required anymore (as of python 3.3). But even if i create empty __init__.py files in /, /a, /c, it still doesn't work with poetry and does with pipenv.

from poetry.

@phiresky Poetry won't find your packages automatically since the name of the project is different. You need to specify them explicitely as documented here: https://poetry.eustace.io/docs/pyproject/#packages

from poetry.

Thank you! That works :)

from poetry.

Hello @rochacbruno,

could you please add some more details about your whole setup? Path dependencies are installed in development mode by default. This is what I've done your test case:

.
├── dynaconf
│   ├── CHANGELOG.md
│   ├── CODE_OF_CONDUCT.md
│   ├── CONTRIBUTING.md
│   ├── HISTORY
│   ├── LICENSE
│   ├── MANIFEST.in
│   ├── Makefile
│   ├── README.md
│   ├── azure-pipelines.yml
│   ├── docs
│   ├── dynaconf
│   ├── dynaconf.egg-info
│   ├── example
│   ├── pytest.ini
│   ├── readthedocs.yml
│   ├── release.sh
│   ├── requirements_dev.txt
│   ├── setup.cfg
│   ├── setup.py
│   ├── tests
│   └── tox.ini
└── myapp
    └── pyproject.toml

In the pyprojoct.toml of myapp I have:

[tool.poetry]
name = "myapp"
version = "0.1.0"
description = ""
authors = ["My Name <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.8"
dynaconf = {path = "../dynaconf"}

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

from poetry.

Hey @rochacbruno,

which version of poetry are you using? I'm unable to reproduce the error in your very good step-by-step manual with the current release (version 1.0.3).

fin swimmer

from poetry.

I find myself with similar needs to the use cases explained above.

To be explicit, I have three packages:

• mylib (which is tightly dependent in mydepa and mydepb)
• mydepa
• mydepb

They are all poetry-ready packages, and things work ok for developing independently. However, it is frequent that I need to do changes in tandem (an interface change in mydep1 requires a change in mylib).

The way I solve this now:

• Create a python environment
• Check out the code of everything.
• Generate a setup.py from the pyproject.toml of all three projects
• Install mylib with python setup.py develop (which installs published wheels of mydepa and mydepb)
• Install mydepa and mydepb with python setup.py develop (which smashes the wheel instalation of previous step)

Why this?:
I have found that having poetry in the [build-system] in pyproject.toml breaks installation with pip install -e . The editable installs call setuptools behind the scenes, but this only works for me if the build system is setuptools/wheel based, not poetry (ModuleNotFound setuptools). That is why I need to explicitly call python setup.py develop.

I suspect this is related #2276

So my current "solution" is a sad state where I basically set up poetry to then proceed to completely remove it from the chain, I be back in setuptools land. Additionally, I lose dev dependencies because they are not taken into setup.py.

For the proposals on this thread, I see that I could list a dependency with a folder origin, but then: will not I have to constantly undo that to push the packages for release?

So, how can this workflow be supported using poetry, or what are alternatives that accomplish the same?

from poetry.

@albertogomcas #2270 might be of interest to you.

from poetry.

I just ran into the exact issue/use-case that @matthijskooijman described. I have a dependancy project PA I wish to migrate to poetry and a dependant project PB with a classical setup.py. PA being a sort of an "engine" library, features in PB often require changes in PA. As a result I always pip install -e ../pa in PB to ease the co-development in both PA & PB.

According to @sinoroc's answer above, my best bet would be to migrate PB to poetry as well and use the dev-dependency trick. Although technically doable, I'm uncomfortable with this because it feels like myself and third parties would be arm twisted into poetry for all other current and future projects depending on PA. Of course, after reading this thread and linked conversations, I understand that a "clean" resolution of this issue (i.e. make pip install -e compatible with project.toml) is beyond the responsibility of the poetry project alone.

Fundamentally, I don't need to use pip install -e, I just want the result of it. So I'm wondering if there could be some "easy" workaround. I naively tried the following:

git clone https://path/to/pb
cd pb
python -m venv venv
source venv/bin/activate

# try 1
poetry install ../pa  #FAIL: install doesn't accept argument

# try 2
cd ../pa
poetry install  #FAIL: poetry doesn't use the current venv

Any chance something like this could be made to work? Or am I missing something and there already is a way to do something like this?

from poetry.

@sinoroc The build back-end of PB is setuptools indeed.

I'm confused by your first suggestion. You may have my issue in the wrong direction. I want to install "poetry-PA" as editable inside "setuptools-PB"'s venv, not the other way around like you seem to be suggesting. Or am I missing something?

That's how I setup PB:

git clone https://path/to/pb
cd pb
python -m venv venv
source venv/bin/activate

pip install .   # this installs PA from pypi/git/whatnot based on setup.py
pip uninstall pa
pip install -e ../pa  # this fails when PA is migrated to poetry

As for .pth files, they are new to me. I will read on them, see if they can help and report back.

from poetry.

I'm confused by your first suggestion. You may have my issue in the wrong direction.

@abey79 Indeed, I had misunderstood. I got it the wrong way around. My bad. And indeed, as of today, sadly, one can not pythonX.Y -m pip install --editable /path/to/poetry/project.

I would recommend giving .pth files a chance. In the end this is the "hack" used to do editable installations (in both setuptools and poetry). .pth files are always a good trick to know, can be used for many other things.

To all: Maybe think about supporting PyPA with funding so that they can employ/pay a developer/project manager to work on/standardize the editable feature. Tell your bosses how this feature is important for your workflows, and talk to PyPA.

from poetry.

First of all, thanks to the maintainers, Sébastien Eustace especially!

It may be fruitless to post here in a closed issue, but I'm adding my voice that this is a common workflow. I want to be able to develop the dependency at the same time I'm working on the dependant.

And also suggesting a workaround (I see it's the same as #34 (comment) but I'll include my write-up on this should others need this):

In a Poetry project, I am able to do pip install -e ../project-dependency, but only if the other dependency is not a Poetry project as well (build-system.build-backend = poetry.core.masonry.api). I'm then getting an error (described in #3153 (comment)). Reading here, it seems to be related to pip install -e not working with Poetry.

What I can do however, is to generate the setup.py file (as suggest above with poetry build and extract the setup.py file might work, dephell deps convert was what I used). Then run poetry run sh -c "cd ../project-dependency; python setup.py develop --no-deps" in my dependant project.

That will successfully create the .pth file I need:

cat $(poetry env info -p)/lib/python3.7/site-packages/easy-install.pth
/Users/kristofferb/Code/nep/project-dependency

When importing the project-depdency package I can then go ahead and work on both projects at the same time.

from poetry.

from poetry.

@kbakk thanks for that, exactly what I was looking for!

Do you know what's a standard way to undo the result of dephell register?

My project itself is also using poetry. If trying to run poetry install after dephell register I got this error:

AssertionError: Egg-link /Users/juhoautio/Library/Caches/pypoetry/virtualenvs/abc-123-py3.7/src/xyz does not match installed location of xyz (at /Users/juhoautio/projects/Xyz)

I wasn't able to use pip uninstall xyz either, to clean up.

What worked for me was this:

rm $(poetry env info -p)/lib/python3.7/site-packages/dephell.pth
rm $(poetry env info -p)/lib/python3.7/site-packages/xyz.egg-link

Now poetry install works again. But maybe I'm missing some simpler way for cancelling the effects of dephell register?

from poetry.

@juhoautio-rovio Have you also asked this question about uninstalling in dephell's ticket tracker?

from poetry.

@aucampia I've myself reverted to adding a dependency.pth file in my dependant project's venv.

from poetry.

@NeilGirdhar thanks for the info, your suggestion works if I replace --user with --prefix="${HOME}/.local/", i.e:

poetry build --format sdist
tar --wildcards -xvf dist/*.tar.gz -O '*/setup.py' > setup.py
pip3 install --prefix="${HOME}/.local/" --editable .

with the following in pyproject.toml

[build-system]
requires = ["setuptools","poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

One liners for uninstall and install:

# uninstall globally
pip3 uninstall -y "$(poetry version | gawk '{ print $1 }')"

# install editable
\rm -rv dist/; poetry build --format sdist && tar --wildcards -xvf dist/*.tar.gz -O '*/setup.py' > setup.py && pip3 install --prefix="${HOME}/.local/" --editable .

Added --wirldcards after @teto mentioned it.

Example of running it:

$ poetry build --format sdist && tar -xvf dist/*.tar.gz -O '*/setup.py' > setup.py && pip3 install --prefix="${HOME}/.local/" --editable .
Building aucampia.template.poetry (0.1.0)
  - Building sdist
  - Built aucampia.template.poetry-0.1.0.tar.gz
aucampia.template.poetry-0.1.0/setup.py
Obtaining file:///home/iwana/syncthing/sw-wpw/d/gitlab.com/aucampia/templates/python-poetry
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Installing collected packages: aucampia.template.poetry
  Running setup.py develop for aucampia.template.poetry
Successfully installed aucampia.template.poetry

from poetry.

@mreschke

Before poetry 1.1 directory path dependencies were installed in editable mode by default. You should set the develop attribute explicitly, to make sure the behavior is the same for all poetry versions.

-- https://python-poetry.org/docs/dependency-specification/#path-dependencies

You might want:

uvicore = { path = "../uvicore", develop = true }

from poetry.

@teto I am not sure what is the point of adding the current directory... But yes, adding the path to some directory containing some python code to PYTHONPATH should somehow allow to do something similar to an editable installation. It is more or less the same trick as the .pth file trick.

from poetry.