Cannot pip-install using poetry about ansible-podman-collections HOT 12 CLOSED

We added a feature in ansible-base 2.10 for ACD (ansible >= 2.10) and anyone else to do this. You just need to pip install your collection as an implicit namespace package under ansible_collections/(ns)/(coll) anywhere on sys.path, and we'll pick it up. If you install ansible >= 2.10 and look where it goes, you'll see how the Ansible community distribution is already doing this.

Rather than having a special dedicated location, we scan all sys.path top-level dirs at startup and append any that contain an ansible_collections package to the end of the collection loader list, so collections found there will be found and loaded. While we don't "recommend" installing collections this way in general (as simple packaging errors can trash other collections since they're all under the same root packages), we do take some special care at runtime to prevent collection packaging errors from DoSing other collections or Ansible itself.

We're also planning to enhance the ansible --version output in 2.11 to include more detail about the dynamically-discovered collection paths under sys.path- currently, we only show the "configured" collections paths.

from ansible-podman-collections.

The collection is installed in /usr/share/ansible/collections:
https://github.com/containers/ansible-podman-collections/blob/master/setup.cfg#L25-L31

data_files =
    share/ansible/collections/ansible_collections/containers/podman/ = README.md
    share/ansible/collections/ansible_collections/containers/podman/roles/ = roles/*
    share/ansible/collections/ansible_collections/containers/podman/plugins/ = plugins/*
    share/ansible/collections/ansible_collections/containers/podman/playbooks/ = playbooks/*
    share/ansible/collections/ansible_collections/containers/podman/scripts/ = scripts/*
    share/ansible/collections/ansible_collections/containers/podman/docs/ = docs/*
    share/ansible/collections/ansible_collections/containers/podman/meta/ = meta/*

So you need to specify a module-path for ansible-doc:
poetry run ansible-doc --module-path .venv/share/ansible/collections containers.podman.podman_container
should work.
@yajo tell me if it's OK.

from ansible-podman-collections.

I guess that'd work, but TBH I'd expect that a pip-installed collection was reachable on standard collection paths... the venv location might be different depending on how it's installed, after all...

from ansible-podman-collections.

@yajo standard collection path is in your $HOME/.ansible, and no package can be installed there.

from ansible-podman-collections.

I see... I guess this is more a problem of ansible itself then. https://docs.ansible.com/ansible/latest/reference_appendices/config.html#collections-paths explains that /usr/share/ansible/collections is a valid default path, so I guess that globally installing this collection with pip should work, but it breaks in the venv use case. Let me open an issue there.

from ansible-podman-collections.

Here it is: ansible/ansible#72384

Let's see what they say...

from ansible-podman-collections.

If it was a different collection which is not in ACD, I would install it to lib/python3.7/site-packages/ansible_collections/ then, right @nitzmahone ?
But if this collection already exists there like containers.podman, it would overwrite one from Ansible installation, which is not a good idea.
So collections that are part of ACD I wouldn't install into this path.

from ansible-podman-collections.

Well, you'd probably just pip install it to wherever pip wants to install it, but we'll respect the sys.path order used by the interpreter for precedence, and IIRC we explicitly ignore package init's above the collection level, so there shouldn't be any worry about packaging failures DoSing other collections.

But yes, if you're installing to the same site-packages (or whatever) as ACD did, and you overwrite an ACD-packaged collection with a discretely packaged one, the next ACD upgrade or uninstall will wreak havoc. This is one of the reasons we don't really recommend or document this as a general procedure- it's there, it's unlikely to go away, but it's also easy to make a mess and break stuff.

from ansible-podman-collections.

ACD (as well as other OS distros that roughly track it) have considered moving to metapackages (rather than the current monolithic Python-package-of-collections)- that would arguably make that problem easier to solve, but at the cost of a lot more packaging overhead and potential difficulty with other metadata and deps inherent to a metapackage design. I'm not sure where that discussion landed, but maybe @abadger or @gundalow can shed some light.

from ansible-podman-collections.

Wouldn't it make sense that ACD reads collections in this order?

1. site-packages/ansible_collections/
2. site-packages/ansible_collections_core/

This way, collection owners have a clear path where to let users pip-install their content, and ACD packagers just have to install builtin collections in a separate path that would also be available as a fallback.

If a collection was found in the 1st dir, it wouldn't be searched in the 2nd dir, so there'd be no conflict.

from ansible-podman-collections.

Wouldn't it make sense that ACD reads collections in this order?

1. `site-packages/ansible_collections/`

2. `site-packages/ansible_collections_core/`

This way, collection owners have a clear path where to let users pip-install their content, and ACD packagers just have to install builtin collections in a separate path that would also be available as a fallback.

If a collection was found in the 1st dir, it wouldn't be searched in the 2nd dir, so there'd be no conflict.

I don't think so.... That seems like it can be done with either installing to another site-packages directory or installing into another collection path. It seems like your trying to combine both of those features instead of picking one.

The ansible-base team didn't like the idea of metapackackages of the Ansible package from pip so i said we'd revisit that idea for 2.11.(collection owners like the idea of Ansible being a metapackackage so it's a question of figuring out what the concerns are on both sides and figuring out a path that takes care of the worst problems pointed out by both). Individual distros are looking to package the monolithic Ansible package from pip or to package individual collections depending on their own perception of their users needs.

from ansible-podman-collections.

I think we can close this issue. Feel free to reopen if there are still questions.

from ansible-podman-collections.