Drop `#egg` fragment on `pdm export` if supplied a `--no-egg-fragments` about pdm HOT 3 CLOSED

I'm close to closing this issue as I've come to refactor my thoughts considerably, with my feature request being an edge case and the egg quirk being part of a much larger, and ongoing conversation from 2013, (with it mostly dropped in [email protected] hopefully to drop completely in 2025. And with it only being supported within a VCS repository url that's editable to name the package 🙄

I was originally dealing with theoretical scenarios and have recently moved to concrete implementation which as of now I'm comfortable with needing to use local editable when needed with a sed in place. Since this will only likely occur during a build / CI step it's not that bad.

I found this one other comment today that I stumbled as I'm actively investigating, where mitsuhiko recommends using pdm's method of handling local editables which is also very topical as I'm also currently leveraging pdm for a refactor of a medium sized codebase to a monorepo - and I've looked a lot at the page mentioned a lot 😆

astral-sh/uv#592 (comment)

from pdm.

I was able to look at this a little deeper today, and determined that this is being set as a result of the requirements from a pyproject.toml or lock file being read in as a pdm.models.requirements.FileRequirement. Specifically its a result of its as_line():

I was able to also determine that for some reason the logic for will result in the FileRequirements which expand with the egg# fragment

In my own testing with mocking this code directly, I originally thought that I was following the same logic, but you can see I'm only getting the default dependencies of my modified pdm-example-monorepo which includes this pyproject.toml:

[project]
name = "monorepo"
version = "0.1.0"
dependencies = [
        #none of these work ! they're not supposed to
	#"-e pkg-core @ file:///${PROJECT_ROOT}/packages/pkg-core",
	#"-e pkg-first @ file:///${PROJECT_ROOT}/packages/pkg-first",
	#"-e pkg-second @ file:///${PROJECT_ROOT}/packages/pkg-second",
	#"-e pkg-third @ file:///${PROJECT_ROOT}/packages/pkg-third",
	#"-e pkg-fourth @ file:///${PROJECT_ROOT}/packages/pkg-fourth",
	#"./packages/pkg-core",
	#"./packages/pkg-first",
	#"./packages/pkg-second",
	#"./packages/pkg-third",
	#"./packages/pkg-fourth",
        # these will!
	"pkg-core @ file:///${PROJECT_ROOT}/packages/pkg-core",
	"pkg-first @ file:///${PROJECT_ROOT}/packages/pkg-first",
	"pkg-second @ file:///${PROJECT_ROOT}/packages/pkg-second",
	"pkg-third @ file:///${PROJECT_ROOT}/packages/pkg-third",
	"pkg-fourth @ file:///${PROJECT_ROOT}/packages/pkg-fourth",
]
requires-python = ">=3.8"

[tool.pdm.dev-dependencies]
dev = [
    "-e pkg-core @ file:///${PROJECT_ROOT}/packages/pkg-core",
    "-e pkg-first @ file:///${PROJECT_ROOT}/packages/pkg-first",
    "-e pkg-second @ file:///${PROJECT_ROOT}/packages/pkg-second",
    "-e pkg-third @ file:///${PROJECT_ROOT}/packages/pkg-third",
    "-e pkg-fourth @ file:///${PROJECT_ROOT}/packages/pkg-fourth",
]


[tool.pdm]
distribution = false

[build-system]
requires = ["pdm-backend"]
build-backend = "pdm.backend"

>>> from pdm.core import Core
>>> from pdm.project.core import Project
>>> core = Core()
>>> Project(core, '.')
<Project '/home/python/pdm-example-monorepo'>
>>> proj = Project(core, '.')

>>> packages = requirements.values()
>>> packages
dict_values([FileRequirement(name='pkg-core', marker=None, extras=set(), specifier=<SpecifierSet('')>, editable=False, prerelease=None, groups=['default'], url='file:///${PROJECT_ROOT}/packages/pkg-core', path=PosixPath('packages/pkg-core'), subdirectory=None), FileRequirement(name='pkg-first', marker=None, extras=set(), specifier=<SpecifierSet('')>, editable=False, prerelease=None, groups=['default'], url='file:///${PROJECT_ROOT}/packages/pkg-first', path=PosixPath('packages/pkg-first'), subdirectory=None), FileRequirement(name='pkg-second', marker=None, extras=set(), specifier=<SpecifierSet('')>, editable=False, prerelease=None, groups=['default'], url='file:///${PROJECT_ROOT}/packages/pkg-second', path=PosixPath('packages/pkg-second'), subdirectory=None), FileRequirement(name='pkg-third', marker=None, extras=set(), specifier=<SpecifierSet('')>, editable=False, prerelease=None, groups=['default'], url='file:///${PROJECT_ROOT}/packages/pkg-third', path=PosixPath('packages/pkg-third'), subdirectory=None), FileRequirement(name='pkg-fourth', marker=None, extras=set(), specifier=<SpecifierSet('')>, editable=False, prerelease=None, groups=['default'], url='file:///${PROJECT_ROOT}/packages/pkg-fourth', path=PosixPath('packages/pkg-fourth'), subdirectory=None)])

>>> content = FORMATS["requirements"].export(proj, packages, options)
>>> content
'# This file is @generated by PDM.\n# Please do not edit it manually.\n\npkg-core @ file:///home/python/pdm-example-monorepo/packages/pkg-core\npkg-first @ file:///home/python/pdm-example-monorepo/packages/pkg-first\npkg-fourth @ file:///home/python/pdm-example-monorepo/packages/pkg-fourth\npkg-second @ file:///home/python/pdm-example-monorepo/packages/pkg-second\npkg-third @ file:///home/python/pdm-example-monorepo/packages/pkg-third\n'
>>> print(content)
# This file is @generated by PDM.
# Please do not edit it manually.

pkg-core @ file:///home/python/pdm-example-monorepo/packages/pkg-core
pkg-first @ file:///home/python/pdm-example-monorepo/packages/pkg-first
pkg-fourth @ file:///home/python/pdm-example-monorepo/packages/pkg-fourth
pkg-second @ file:///home/python/pdm-example-monorepo/packages/pkg-second
pkg-third @ file:///home/python/pdm-example-monorepo/packages/pkg-third

Then when running pdm export --pyproject, it picks up the dev dependencies with #egg= fragments as they're FileRequirements:

pdm export --pyroject

# This file is @generated by PDM.
# Please do not edit it manually.

-e file:///home/python/pdm-example-monorepo/packages/pkg-core#egg=pkg-core
-e file:///home/python/pdm-example-monorepo/packages/pkg-first#egg=pkg-first
-e file:///home/python/pdm-example-monorepo/packages/pkg-fourth#egg=pkg-fourth
-e file:///home/python/pdm-example-monorepo/packages/pkg-second#egg=pkg-second
-e file:///home/python/pdm-example-monorepo/packages/pkg-third#egg=pkg-third

As I dug deeper, FileRequirement is what will resolve to, and the -e is always prefixed and #egg is always attached if the package is --editable.

I will get the same results of my repl session above if I run pdm export--pyproject --prod, which was using only the default dependencies and was explicitly set to dev=False with my hack using a NamedTuple instead of a argparse.Namespace:

pdm export --pyproject --prod      
# This file is @generated by PDM.
# Please do not edit it manually.

pkg-core @ file:///home/python/pdm-example-monorepo/packages/pkg-core
pkg-first @ file:///home/python/pdm-example-monorepo/packages/pkg-first
pkg-fourth @ file:///home/python/pdm-example-monorepo/packages/pkg-fourth
pkg-second @ file:///home/python/pdm-example-monorepo/packages/pkg-second
pkg-third @ file:///home/python/pdm-example-monorepo/packages/pkg-third

I think in order for this to be an acceptable feature, there would need to be proof in what happens with a vanilla pip install -e for comparison. Which I've done, and will x-post the relevant bits to here from astral-sh/uv#313 (comment)

In a vanila pip install -e 'file://...', there's a quirk if the path is relative inside a git directory, and will produce something like:

# replace .venv
$ uv v --seed
# install the package as editable with relative path
$ .venv/bin/pip install -e 'file://'$PWD'packages/pkg-second'

# ...

# show pip installation
$ .venv/bin/pip freeze | grep pkg-core

-e git+https://github.com/pdm-project/pdm-example-monorepo@b85261555ea063e68eaf744e225804149c27e64f#egg=pkg_core&subdirectory=packages/pkg-core

However, when not in a git directory, the same way of installation will result in what I'm after:

# replace .venv
$ uv v --seed
# get rid of .git, so it's not detected as a git repository
$ mv .git .gitold
# install the package as editable with relative installation, again
$ .venv/bin/pip install -e 'file://'$PWD'/packages/pkg-core'

# ...

# show pip installation
$ .venv/bin/pip freeze | grep pkg-core

# Editable install with no version control (pkg-core==0.1.0+editable)
-e /home/python/pdm-example-monorepo/packages/pkg-core

Ultimately I can see how I could implement this feature. Through FileRequirement, since there's some open loops on uv with their handling of git+http://... urls which do include egg= in my testing, it may be revisited.

from pdm.

I am closing this issue feature request.

Thank you for letting me dump my thoughts here!

The primary reason is that this feature wouldn't be useful to many people, the output of pdm export ... can easily be stream edited to remove the problem parts when using the exported requirements.txt.

Another reason is there's a better option (in my case at least) to use non-editable package paths and rely on project.dependencies in a production deployment setting where other tools that understand PEP-621 have no problems installing those packages and dependencies.

Some more context:

I have gone through many scenarios with boilerplate project structures. I've since discovered the best approach for my use case is to not rely on pdm export but instead use PEP-621, with project.dependencies for production deployments. Whereas before I wasn't considering that aspect of it, I was trying to use uv in the context of editable development, where egg would be used.

Now, in this specific use case, I am using both project.dependencies and tool.pdm.dev-dependencies (no need for project.optional-dependencies at this time, which are the equivalent to the extras=[...] a la setup-tools).

During development, I will use pdm. For deployment, I can use pdm sync --no-editables --prod or I can use uv pip install -r pyproject.toml. Since uv is very new, this seems like an OK place to evaluate it for now.

I could also still convert the requirements over through pdm export ..., stripping off the egg=... with sed, but with the approach of having production ("non-editable") project.dependencies, uv can read it directly. With every sub-package within the monorepo, which is referenced at the root of this project also using PEP-621, and project.packages for describing its dependencies, I don't foresee issues at this point, and the option to fallback now that I've adopted pdm into this project, with pdm run ... helpers or other project scripts in place that can staple those commands I don't see this as a desirable feature.

I expect in the future, egg=... will be a distant memory, and including this as a feature will make another thing to remove and make it somewhat more difficult to migrate any other users that would happen to adopt this command, naively¹.

For posterity and anyone that might stumble across this, the monorepo is laid out almost identical to pdm-project/pdm-example-monorepo.

To then use it for uv pip install -r pyproject.toml, the root pyproject.toml would look like this:

[project]
dependencies = [
    "pkg-core @ file:///${PROJECT_ROOT}/packages/pkg-core",
    "pkg-first @ file:///${PROJECT_ROOT}/packages/pkg-first",
    "pkg-second @ file:///${PROJECT_ROOT}/packages/pkg-second",
]
requires-python = ">=3.8"

[tool.pdm.dev-dependencies]
dev = [
    "-e pkg-core @ file:///${PROJECT_ROOT}/packages/pkg-core",
    "-e pkg-first @ file:///${PROJECT_ROOT}/packages/pkg-first",
    "-e pkg-second @ file:///${PROJECT_ROOT}/packages/pkg-second",
]

or as a diff:

2c2,6
< dependencies = []
---
> dependencies = [
>     "pkg-core @ file:///${PROJECT_ROOT}/packages/pkg-core",
>     "pkg-first @ file:///${PROJECT_ROOT}/packages/pkg-first",
>     "pkg-second @ file:///${PROJECT_ROOT}/packages/pkg-second",
> ]

Thanks again for your excellent work and this excellent project!

from pdm.