pulp / pulp_ansible Goto Github PK

View Code? Open in Web Editor NEW

53.0 8.0 52.0 2.79 MB

A Pulp plugin that manages Ansible content, i.e. roles, collections

Home Page: https://docs.pulpproject.org/pulp_ansible/

License: GNU General Public License v2.0

Python 99.30% Shell 0.16% Jinja 0.54%

pulp ansible roles collections galaxy

pulp_ansible's Introduction

pulp_ansible

A Pulp plugin to support hosting Role and Collection Ansible content.

For more information, please see the documentation.

Collection Support

Warning

The 'Collection' content type is currently in tech-preview. Breaking changes could be introduced in the future.

pulp_ansible can manage the multi-role repository content <https://galaxy.ansible.com/docs/using/ installing.html#multi-role-repositories> referred to as a Collection. The following features are supported:

ansible-galaxy collection publish - Upload a Collection to pulp_ansible for association with one or more repositories.
ansible-galaxy collection install - Install a Collection from pulp_ansible.

Configuring Collection Support

You'll have to specify the protocol and hostname the pulp_ansible REST API is being served on. For pulp_ansible to interact with ansible-galaxy correctly it needs the entire hostname. This is done using the ANSIBLE_HOSTNAME setting in Pulp. For example if its serving with http on localhost it would be:

export PULP_ANSIBLE_API_HOSTNAME='http://localhost:24817'
export PULP_ANSIBLE_CONTENT_HOSTNAME='http://localhost:24816/pulp/content'

or in your systemd environment:

Environment="PULP_ANSIBLE_API_HOSTNAME=http://localhost:24817"
Environment="PULP_ANSIBLE_CONTENT_HOSTNAME=http://localhost:24816/pulp/content"

How to File an Issue

New pulp_ansible issue.

Warning

Is this security related? If so, please follow the Security Disclosures procedure.

pulp_ansible's People

Contributors

Stargazers

Watchers

Forkers

daviddavis bmbouter dralley dkliban werwty snmpboy goosemania t-woerner cutwater codeheeler ipanova atix-ag mdellweg nixocio mikedep333 sjha4 asmacdo ompragash awcrosby lubosmj newswangerd romanblanco ironfroggy evgeni bmclaughlin s-hertel toshywoshy chrismeyersfsu melcorr egmar mcssteam zitanemeckova fao89 ggainey gerrod3 luto jctanner yadnyawalkya stosem drodowic rochacbruno tas-sos azurecloudmonk shaiahwren oyvindkolbu tjmullicani mayacostantini jerabekjiri sivel decko viehoang68 pedro-psb

pulp_ansible's Issues

error with pulp-ansible: KeyError: 'path'

Author: @jlsherrill ([email protected])

Redmine Issue: 5586, https://pulp.plan.io/issues/5586

I hit this when deploying a new pulp3 box with master, full tracebacK:

pulp: django.request:ERROR: Internal Server Error: /pulp/api/v3/
Traceback (most recent call last):
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/views/generic/base.py", line 71, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/views.py", line 505, in dispatch
response = self.handle_exception(exc)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/schemas/views.py", line 48, in handle_except
return super().handle_exception(exc)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/views.py", line 465, in handle_exception
self.raise_uncaught_exception(exc)
return view_func(*args, **kwargs)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/views/generic/base.py", line 71, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/views.py", line 505, in dispatch
response = self.handle_exception(exc)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/schemas/views.py", line 48, in handle_exception
return super().handle_exception(exc)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/views.py", line 465, in handle_exception
self.raise_uncaught_exception(exc)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
raise exc
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/views.py", line 502, in dispatch
response = handler(request, *args, **kwargs)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/schemas/views.py", line 37, in get
schema = self.schema_generator.get_schema(request, self.public)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/schemas/openapi.py", line 64, in get_schema
paths = self.get_paths(None if public else request)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/schemas/openapi.py", line 47, in get_paths
operation = view.schema.get_operation(path, method)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/schemas/openapi.py", line 104, in get_operation
operation['responses'] = self._get_responses(path, method)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/schemas/openapi.py", line 489, in _get_responses
serializer = self._get_serializer(path, method)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/schemas/openapi.py", line 446, in _get_serializer
return view.get_serializer()
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/generics.py", line 109, in get_serializer
kwargs['context'] = self.get_serializer_context()
File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/galaxy/v3/views.py", line 55, in get_serializer_context
def get_serializer_context(self):
KeyError: 'path'

Enable v3 api for distributions

Author: daviddavis (daviddavis)

Redmine Issue: 6317, https://pulp.plan.io/issues/6317

Currently, the ansible-galaxy client uses v2[0] to download collections from pulp_ansible. We have v3 views that we should expose. Note that there are some slight differences in Automation Hub's and pulp_ansible's formats though. I ran into this problem when working on #6132.

[0]

pulp_ansible/pulp_ansible/app/galaxy/views.py

Line 36 in 51757b5

    
           api_info = {"available_versions": {"v1": "v1/", "v2": "v2/"}, "current_version": "v1"}

Ensure ContentVersionFilter can use DRF sorting on inherited fields

Author: @bmbouter (bmbouter)

Redmine Issue: 5399, https://pulp.plan.io/issues/5399

Currently the ContentVersionFilter does not manually specify the ordering_fields option DRF supports.

I believe ContentVersionFilter gets "ordering_fields" automagically from it's ancestors, but I'm not 100% sure. This task is to determine for sure if sort is working currently or not.

Split the import_collection task into two tasks to reduce the amount of time the repository lock is held

Author: @bmbouter (bmbouter)

Redmine Issue: 6677, https://pulp.plan.io/issues/6677

Background

The import_collection task both analyzes a collection here and then creates content and adds it to a repository version here.

Problem

The import_collection runs long enough that to import a lot of content into a repository serializes these long tasks making the whole workload take long. It takes longer than it should because the collection analysis portion of the task runtime takes the majority of the time and the lock on repository is not needed during that. The lock is only needed when creating the RepositoryVersion itself.

Solution

Split import_collection into two tasks. analyze_and_create_collection and then add_collection_to_repository.
Have the analyze_and_create_collection task dispatch the add_collection_to_repository. The analyze_and_create_collection task requires no lock. The add_collection_to_respository should have the lock on the repository.

Pulp Ansible Galaxy V3 Collection Artifact download_url field can give unresolveable URLs

Author: ironfroggy (ironfroggy)

Redmine Issue: 5647, https://pulp.plan.io/issues/5647

Some configurations, notably with `localhost` and likely other internal non-TLD hostnames, allow the `download_url` field to give invalid URLs that cannot be resolved by an HTTP client consistently, like:

localhost:24816/pulp/content/orionuser1.collection_dep_a_sufvwzod.1.0.0

https://travis-ci.org/pulp/pulp_ansible/jobs/604970911#L1269-L1325

The problem appears to be caused by the lack of a schema, which would make the URL valid and resolvable.

Validate the requirements.yml file

Author: daviddavis (daviddavis)

Redmine Issue: 7734, https://pulp.plan.io/issues/7734

Validate the requirements.yml when a remote is saved.

Some examples:

Validate the presence of required keys (name, source) if the entry is a hash
Validate that type matches up with whatever types we support (ie only galaxy at this point)

Dependency Downloading for Ansible Collection sync

Author: sajha (sajha)

Redmine Issue: 5251, https://pulp.plan.io/issues/5251

Dependency Downloading when syncing collections:

When syncing a collection, we would like to have the collections listed as dependencies to be synced into the repository as well.
Example: https://docs.ansible.com/ansible/devel/dev_guide/collections_galaxy_meta.html#examples

For a collection to be synced, having the below dependencies:

dependencies:
    "other_namespace.collection1": ">=1.0.0"
    "other_namespace.collection2": ">=2.0.0,<3.0.0"
    "anderson55.my_collection": "*"    # note: "*" selects the highest version available

The 3 listed collections should also get synced when syncing the collection.

Collection model that has no CollectionVersion related objects needs to be removed somehow when orphaned

Author: @bmbouter (bmbouter)

Redmine Issue: 5521, https://pulp.plan.io/issues/5521

Problem

1. Upload a collection with <namespace=foo, name=bar, and version=1.0> and associate it with a repository_version so it's not an orphan.
2. Upload a second collection with <namespace=foo, name=bar, and version=2.0> and associate it with a repository_version so it's not an orphan.

At this point I expect the /pulp/api/v3/ansible/collection_versions to show you two CollectionVersions. Once story 5520 is implemented you would see exactly 1 Collection (deduplicated since they share Collection.

3. Now unassociate the version=1.0 collection
4. Run orphan cleanup and verify the Collection verson=1.0 was deleted from Pulp entirely. Now you'll have 1 Collection and 1 CollectionVersion
5. Now unassociate the version=2.0 collection
6. Run orphan cleanup again and verify the Collection version=2.0 was deleted from Pulp entirely.
7. Now list a Collection using the endpoint from 5520 and observe it still shows the Collection even though there are 0 CollectionVersions, this is the bug.

Solutions

Add a post_delete hook on CollectionVersion that will identify if it's the last CollectionVersion referring to that Collection and delete the Collection also if so.

FILES.json not found when syncing all of galaxy.ansible.com

Author: @gerrod3 (gerrod)

Redmine Issue: 8292, https://pulp.plan.io/issues/8292

Feb 22 18:24:22 ip-172-32-0-61 rq[240455]: pulp [c1c01c8eb5b4411dafd848e54fe8148e]: pulp_ansible.app.tasks.utils:INFO: Reading meta/runtime.yml from artifact/c2/939ffe56fe75ffd1ae96e6f0159421aa120c286a9464ac851bb00c25568e42
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]: pulp [c1c01c8eb5b4411dafd848e54fe8148e]: pulp_ansible.app.tasks.utils:INFO: Reading MANIFEST.json from artifact/c2/939ffe56fe75ffd1ae96e6f0159421aa120c286a9464ac851bb00c25568e42
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]: pulp [c1c01c8eb5b4411dafd848e54fe8148e]: pulp_ansible.app.tasks.utils:INFO: Reading FILES.json from artifact/c2/939ffe56fe75ffd1ae96e6f0159421aa120c286a9464ac851bb00c25568e42
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]: pulp [c1c01c8eb5b4411dafd848e54fe8148e]: pulp_ansible.app.tasks.utils:INFO: Reading meta/runtime.yml from artifact/3c/906e302f64afc9d49e7c39e4749dba43dfc94c5907468da670dbb511b6292d
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]: pulp [c1c01c8eb5b4411dafd848e54fe8148e]: pulp_ansible.app.tasks.utils:INFO: Reading MANIFEST.json from artifact/3c/906e302f64afc9d49e7c39e4749dba43dfc94c5907468da670dbb511b6292d
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]: pulp [c1c01c8eb5b4411dafd848e54fe8148e]: pulp_ansible.app.tasks.utils:INFO: Reading FILES.json from artifact/3c/906e302f64afc9d49e7c39e4749dba43dfc94c5907468da670dbb511b6292d
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]: pulp [c1c01c8eb5b4411dafd848e54fe8148e]: rq.worker:ERROR: Traceback (most recent call last):
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:   File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 975, in perform_job
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:     rv = job.perform()
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:   File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 696, in perform
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:     self._result = self._execute()
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:   File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 719, in _execute
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:     return self.func(*self.args, **self.kwargs)
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:   File "/home/ec2-user/devel/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 129, in sync
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:     d_version.create()
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:   File "/home/ec2-user/devel/pulpcore/pulpcore/plugin/stages/declarative_version.py", line 148, in create
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:     loop.run_until_complete(pipeline)
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:   File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:     return future.result()
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:   File "/home/ec2-user/devel/pulpcore/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:     await asyncio.gather(*futures)
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:   File "/home/ec2-user/devel/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:     await self.run()
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:   File "/home/ec2-user/devel/pulpcore/pulpcore/plugin/stages/content_stages.py", line 114, in run
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:     await self._post_save(batch)
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:   File "/home/ec2-user/devel/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 843, in _post_save
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:     get_file_obj_from_tarball(tar, "FILES.json", artifact.file.name)
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:   File "/home/ec2-user/devel/pulp_ansible/pulp_ansible/app/tasks/utils.py", line 144, in get_file_obj_from_tarball
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]:     raise FileNotFoundError(f"{file_path} not found")
Feb 22 18:24:22 ip-172-32-0-61 rq[240455]: FileNotFoundError: FILES.json not found

Collaborate with `ansible-galaxy collection` developers to validate the signature Collection content upon installation

Author: @bmbouter (bmbouter)

Redmine Issue: 5525, https://pulp.plan.io/issues/5525

With a CollectionVersion's signatures available in the response serializer of 5523 the ansible-galaxy client is in a position to verify it trusts content before installing it.

Configuring the ansible-galaxy client for what keys to trust when performing this validation is beyond the scope of this document.

New artifact is downloaded even when we already have a content unit with the same identity

Author: @fao89 (fao89)

Redmine Issue: 8047, https://pulp.plan.io/issues/8047

I'll share a pulp_ansible edge case,

We have rh-certified and community collections, both with the same identity (namespace, name, version),
the content ideally would be the same, but in some cases the sha256s are different.

API: https://galaxy.ansible.com/api/v2/collections/vmware/vmware_rest/versions/1.0.1/

Download URL: https://cloud.redhat.com/api/automation-hub/v3/artifacts/collections/published/vmware-vmware_rest-1.0.1.tar.gz

"sha256": "297358d05551bd104eb94be40edaa14dd55ea9f9a2f3953f58f0197689536c7d",
"size": 153850

API: https://cloud.redhat.com/api/automation-hub/v3/collections/vmware/vmware_rest/versions/1.0.1/

Download URL: https://galaxy.ansible.com/download/vmware-vmware_rest-1.0.1.tar.gz

"size": 153853,
"sha256": "d3f47947d6a0ac81f5fc8d65ef38abe0ec953765504c9421640d81f0f39462c7"

So when you sync from one source and re-sync from another source, the existent content is used, but it downloads a new artifact since the sha256 is different.
So in the end, we have:

Content (Source A)
ContentArtifact (Source A)
Artifact (Source A)
Artifact (Source B)

Backport 8957 to 0.7

Author: daviddavis (daviddavis)

Redmine Issue: 8958, https://pulp.plan.io/issues/8958

Backport #8957 to 0.7

As an user I want to sync collections on demand

Author: @fao89 (fao89)

Redmine Issue: 7470, https://pulp.plan.io/issues/7470

Enable on demand sync for collections

As a developer, I have performance tests running nightly on CI

Author: @fao89 (fao89)

Redmine Issue: 8478, https://pulp.plan.io/issues/8478

We need performance tests with volume (1000 collections or more) to run nightly for v3 endpoints

Add sync_mode parameter to endpoint that deploys sync task

Author: [email protected] ([email protected])

Redmine Issue: 3495, https://pulp.plan.io/issues/3495

sync_mode will be removed in https://pulp.plan.io/issues/3492, which will break sync.

As a user I want to sync roles which have no versions

Author: daviddavis (daviddavis)

Redmine Issue: 3441, https://pulp.plan.io/issues/3441

None

Full text search trigger is not executed on insert

Author: osapryki (osapryki)

Redmine Issue: 5310, https://pulp.plan.io/issues/5310

A trigger that updates full text search vector introduced in migration 0004_add_fulltext_search_indexes.py is not executed on INSERT.

Failed imports leave orphaned artifacts blocking subsequent attempts

Author: ironfroggy (ironfroggy)

Redmine Issue: 5666, https://pulp.plan.io/issues/5666

When an imported is begun and fails, after uploading the artifact and attempting to spawn the import task, the artifact itself is still present. As such, any attempt to publish the same artifact again fails with "Artifact already exists", even though it was never completed successfully and is not available.

Remove the Galaxy V2 API

Author: @bmbouter (bmbouter)

Redmine Issue: 5424, https://pulp.plan.io/issues/5424

The Galaxy V3 superseded the V2 API and the current ansible-galaxy CLI already supports V3. We should drop the V2 API and only carry the V3 API support.

Add a role support to existing one-shot uploader for pulp_ansible

Author: @bmbouter (bmbouter)

Redmine Issue: 4066, https://pulp.plan.io/issues/4066

As a user, I should be able to upload Role and RoleVersion content to Pulp via 1 call where the tarball asset is delivered, and Pulp can automatically recognize the content and create the correct Artifacts Roles and RoleVersions.

This would replace the existing multi-part user-driven approach which is too complicated.

Collaborate with `ansible-galaxy collection` developers to sign signature Collection content upon build and publish

Author: @bmbouter (bmbouter)

Redmine Issue: 5524, https://pulp.plan.io/issues/5524

With a CollectionVersion's signatures able to be stored by pulp_ansible, the client could publish a signature to the server along with the artifact itself.

Completly remove shell scripts from docs testing.

Author: @gerrod3 (gerrod)

Redmine Issue: 8170, https://pulp.plan.io/issues/8170

With the docs moving to use the Pulp-CLI, the shell scripts have become barren and potentially useless. With this pluin https://www.sphinx-doc.org/en/master/usage/extensions/doctest.html the CLI commands could be implanted into the workflows directly and still tested through the CI.

As a user, I can pull in an Ansible content from any git repository

Author: daviddavis (daviddavis)

Redmine Issue: 3363, https://pulp.plan.io/issues/3363

None

As a developer I want sync pipeline conditionally add/remove stages

Author: @fao89 (fao89)

Redmine Issue: 7952, https://pulp.plan.io/issues/7952

We need to provide a way to facilitate adding stages for namespace sync

As a user I want to be able to identify the "installed" collection origin

Author: @fao89 (fao89)

Redmine Issue: 7964, https://pulp.plan.io/issues/7964

signing collection metadata

As a user I have efficient collections endpoint

Author: @fao89 (fao89)

Redmine Issue: 8262, https://pulp.plan.io/issues/8262

Currently, the collections endpoint provides data from multiple tables which can harm the performance, as the query is done for every request.

We need to pre-compute the query to improve the performance

Context: https://github.com/pulp/pulp_ansible/pull/469/files#r573247512

[EPIC] Automation Hub Release Blockers

Author: @bmbouter (bmbouter)

Redmine Issue: 5517, https://pulp.plan.io/issues/5517

This epic captures the pulp_ansible changes required for the initial Automation Hub release, which pulp_ansible is the storage backend for.

All import tasks are assigned to the same worker

Author: osapryki (osapryki)

Redmine Issue: 5674, https://pulp.plan.io/issues/5674

Task `import_collection` now uses resource reservation. It reserves two resources: an artifact and a related repository.
When scheduling a task, the resource manager assigns task that has a resource reservations on a worker that runs a task with any of requested reservations. This causes the following side effect:

Because repository is a global object, all tasks related to that repository are always run sequentially due to being assigned on the same worker.

As an ansible-galaxy CLI user, I can configure a token and auth_url and have pulp_ansible protect my content

Author: @bmbouter (bmbouter)

Redmine Issue: 7118, https://pulp.plan.io/issues/7118

Background

The authentication capabilities of the ansible-galaxy CLI are described here: https://docs.ansible.com/ansible/latest/user_guide/collections_using.html#configuring-the-ansible-galaxy-client

There are two credentials:

auth_url: The url to fetch the session token from
token: The long-lived credential that will give a user a session-token

Requirements

pulp_ansible needs to have some way to hand out a session-token.
An AnsibleContentGuard that will protect a Distribution, requiring the user to use a session-token when fetching content.

Collection highest_version includes pre-releases

Author: @fao89 (fao89)

Redmine Issue: 8714, https://pulp.plan.io/issues/8714

Under /v3/collections/// highest_version.version includes pre-releases, for example "version": "2.2.8-pre1". The expectation likely is that this should not include pre-releases.

The list of all versions, /v3/collections///versions/ includes pre-releases and this is expected.

As a user I can filter Collections by their tags

Author: @bmbouter (bmbouter)

Redmine Issue: 5398, https://pulp.plan.io/issues/5398

Background

The Collection model current has a many-to-many relationship with Tags here

Desire

As a user, I can provide tags as a get style parameter similar to existing DRF filtering. Multiple tags specified should be treated as an AND so each collection should only be returned if it contains ALL of them.

Details

This should extend the CollectionVersionFilter here

Also this request was originally implemented by Galaxy here as two queries where the first one returns pks of collections, and then further fitlers the queryset. Note this implementation is an OR while we need an AND.

Galaxy later opened a PR to update their implementation to an AND see here

Ansible content units are missing fields in the API from ModelSerializer like 'id' and 'created'

Author: daviddavis (daviddavis)

Redmine Issue: 3792, https://pulp.plan.io/issues/3792

None

As a user, I have RBAC permissions on Repository Sync and Modify

Author: @bmbouter (bmbouter)

Redmine Issue: 5789, https://pulp.plan.io/issues/5789

None

Ansible collection upload URL missing /pulp/api/v3/

Author: kersom (kersom)

Redmine Issue: 5281, https://pulp.plan.io/issues/5281

From the docs:

http --form POST :24817/ansible/collections/ [email protected]

See: https://pulp-ansible.readthedocs.io/en/latest/restapi.html#operation/upload_collection
Also: https://pulp-ansible.readthedocs.io/en/latest/workflows/upload-collection.html#upload-a-collection-via-the-rest-api

As a user, I can use django-admin to manage object-level permissions and read data from other models

Author: @fao89 (fao89)

Redmine Issue: 7633, https://pulp.plan.io/issues/7633

Background

Administrators and developers have expressed desire to read Pulp data using django-admin.
Additionally, RBAC features of Pulp need an easy way for users to view/manage object-level permissions for Pulp data and django-admin provides that.

Solution

3.6 Already enabled django-admin at /admin/ so now we need to expand the models that are available there. Specifically we need to:

Add all the pulpcore objects
Ensure each does not allow data to be added or modified, only read
Ensure each object that meaningfully uses object-level permissions inherits from or uses GuardedModelAdmin from django-guardian.

[Epic] Upload Process Improvements

Author: @bmbouter (bmbouter)

Redmine Issue: 6672, https://pulp.plan.io/issues/6672

None

As a user I can copy collections by name from one repo to another

Author: @bmbouter (bmbouter)

Redmine Issue: 6197, https://pulp.plan.io/issues/6197

The endpoint would use the copy action endpoint: /pulp/api/v3/repositories/ansible/ansible/copy/.

It would take a criteria object (see below) and also a source_repository and destination_repository. A new repository version will be created in the destination repository.

{"TYPE": [{[fields]}]} so like {"collection_version": [{"namespace": "whatever", "name": "ok"}, {"namespace": "another}]}

It would copy all versions that match in all cases.

As a user, I have RBAC permissions on Upload of content

Author: @bmbouter (bmbouter)

Redmine Issue: 5790, https://pulp.plan.io/issues/5790

None

As a user, the Galaxy V3 APIs has working Python bindings

Author: @bmbouter (bmbouter)

Redmine Issue: 5238, https://pulp.plan.io/issues/5238

The viewset's aren't correctly represented (or at all) in the Python bindings. Here's what needs to be done:

1. Add swagger tags to those viewsets
2. Verify they are all present and organized in /pulp/api/v3/docs/
3. Verify the endpoints are all usable in the bindings.

As a user, I can view certificate-base signatures of Collection content

Author: @bmbouter (bmbouter)

Redmine Issue: 5523, https://pulp.plan.io/issues/5523

Problem

Users can get their collection content from many places, and they want to understand where it came from. One great way to do this while increasing security is to have the mechanism of data provenance be a signature along with the claim of who it was made by.

This signature would be for a specific CollectionVersion asset (for a specific version). If the asset is modified even one bit (literally) it wouldn't cryptographically validate anymore so every version would need a signature.

How these signatures are generated or verified is beyond the scope of this ticket.

Solution

Add a new Model to pulp_ansible called CollectionVersionSignature and have it be a ForeignKey to CollectionVersion. Have the backref relationship be called 'signatures'. Assuming both data tables are populated you could:

my_collection_version = CollectionVersion.objects.get(pk=1)
my_collection_version.signatures   #  <---- this would be the Queryset of related objects.

To expose this to the user, we would extend the CollectionVersion serializer to display the signatures themselves.

As a user, I can store a signature on a CollectionVersion

Author: @bmbouter (bmbouter)

Redmine Issue: 5791, https://pulp.plan.io/issues/5791

We need to add a "signature" TextField to the CollectionVersion. This will be populated by a new action url added to the end of content. For example:

/pulp/api/v3/content/ansible/collection_versions/<UUID>/sign/

This must be RBAC protected.

Provide a way for subclasses of pulp viewset that use serializers with href methods to override urls that are reversed

Author: alikins (alikins)

Redmine Issue: 7032, https://pulp.plan.io/issues/7032

To paraphrase https://github.com/ansible/galaxy_ng/issues/247#issuecomment-644886557:

The serializers used by the pulp collection viewsets (and therefore galaxy_ng collection viewsets) base the values for 'href' on url reverse() lookups, and end up with the pulp paths.

May mean galaxy_ng viewsets need to provide their own serializer.

But may be able to parameterize the reverse()'s to handle both cases. Possibly via providing a ViewSet.get_serializer_context()

ansible/galaxy_ng#254 is a fix / workaround that is based on subclassing the
related pulp collection Serializers and overriding the href related SerializerMethodFields methods.

ie, the 'get_href()' and similar methods.

The changes in #254 are kind of cut&paste and duplicate a chunk of pulp_ansible code (and diverge
from upstream as a result). So if there is a reasonable way to avoid that it's probably worth while.

https://github.com/ansible/galaxy_ng/issues/247#issuecomment-644983783 mentions a few vague ideas on approaches to handle this issue. In particular, the main #2 point there.

Gist is providing a way for the serializers get_href methods to use a url namespace (or specify url name explicitly) when the viewset classes have been subclassed (as galaxy_ng.app.api.v3.viewsets.collections.CollectionViewSet does).

There may be existing django or drf support for doing this but it isn't clear to me yet. (The 'content_app' attribute mentioned in https://github.com/ansible/galaxy_ng/issues/247#issuecomment-644983783 seems like it is intended for exactly this case, but I couldn't get it to work. But that is likely developer error on my part)
Maybe this is something drf-nested-routers / rest_framework_nested is meant to support?

As a user, I can sync Roles from one Pulp server to another

Author: daviddavis (daviddavis)

Redmine Issue: 3789, https://pulp.plan.io/issues/3789

With two systems needed, this may be more difficult to run in an automated way, but here is what is needed:

1. setup two pulp systems, A and B
2. on A sync some Roles from Galaxy and expose them via a Distribution
3. on B create a remote to sync those roles from the Distribution on A
4. Ensure the remote can sync from another Pulp successfully.

500 when publishing a repository without any versions

Author: daviddavis (daviddavis)

Redmine Issue: 3610, https://pulp.plan.io/issues/3610

Copied from pulp_file. Was able to reproduce this with an ansible publisher.

~/3/pulp_ansible ❯ http POST http://dev.pulp3:8000/api/v3/publishers/ansible/007682b4-f522-469b-92c9-baeca276529a/publish/ repository=http://dev.pulp3:8000/api/v3/repositories/96a7048f-bda6-4ccc-8317-a8dca80defae/

HTTP/1.0 500 Internal Server Error


Traceback:  

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/core/handlers/exception.py" in inner
  41.             response = get_response(request)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  187.                 response = self.process_exception_by_middleware(e, request)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  185.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/views/decorators/csrf.py" in wrapped_view
  58.         return view_func(*args, **kwargs)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/rest_framework/viewsets.py" in view
  95.             return self.dispatch(request, *args, **kwargs)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/rest_framework/views.py" in dispatch
  494.             response = self.handle_exception(exc)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/rest_framework/views.py" in handle_exception
  454.             self.raise_uncaught_exception(exc)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/rest_framework/views.py" in dispatch
  491.             response = handler(request, *args, **kwargs)

File "/home/vagrant/devel/pulp_file/pulp_file/app/viewsets.py" in publish
  147.             [repository_version.repository, publisher],

Exception Type: AttributeError at /api/v3/publishers/file/007682b4-f522-469b-92c9-baeca276529a/publish/
Exception Value: 'NoneType' object has no attribute 'repository'

As a user, fulltext search includes "content match" in the search results

Author: @bmbouter (bmbouter)

Redmine Issue: 5788, https://pulp.plan.io/issues/5788

Return a content_match object similar to that found in v2 API search results.

As an example, if the user searches for the keyword "dynatrace", the v2 API currently returns the following object for each item contained in collections.results[]:

content_match": {
    "total_count": 4,
    "contents": {
        "module": [
            "dynatrace_deployment",
            "dynatrace_comment"
            ],
        "role": [
            "dynatrace_custom_deployment",
            "dynatrace_problem_comment"
            ],
        "plugin": []
        }
   }

From the above, the UI knows that 2 modules and 2 roles matched.

Another thing we talked about fixing in Community Galaxy, and that we do not want to proliferate in AH, is the lack of a total count being displayed in the UI for each content type. In other words, from the above, the UI knows that 2 modules matched. However, it does not know (or at least does not display) that the collection contains a total of 2 modules. What we want to see in the UI is something like, "Matching modules: 2/2".

If you look at the response object returned in the v2 API, you will find a content_summary object. As part of this story, let's verify with the UI team that returning this object is sufficient or if something else is preferred. If something else, then add that. Otherwise, make sure content_summary is returned.

Here's an example of the content_summary object:

"content_summary": {
    "total_count": 4,
    "contents": {
        "module": [
            "dynatrace_deployment",
            "dynatrace_comment"
            ],
        "role": [
            "dynatrace_custom_deployment",
            "dynatrace_problem_comment"
            ],
        "plugin": []
        }
},

You can view the full response object from the Galaxy v2 API that includes the above examples here:

https://galaxy.ansible.com/api/internal/ui/search/?deprecated=false&keywords=dynatrace&order_by=-relevance&page=1

The code behind content_match can be viewed here:
https://github.com/ansible/galaxy/blob/devel/galaxy/api/internal/search.py#L177

Wrong url resolution when path is the last component of prefix

Author: osapryki (osapryki)

Redmine Issue: 5346, https://pulp.plan.io/issues/5346

When <path:path> component is the last one in GALAXY_API_ROOT, Django redirects all requests having this prefix to the GalaxyVersionView, which is defined in pulp_ansible/app/urls.py as:

path(GALAXY_API_ROOT, GalaxyVersionView.as_view()),

As a user, I have my collections scored and linted at creation time

Author: @bmbouter (bmbouter)

Redmine Issue: 5070, https://pulp.plan.io/issues/5070

Current Workflow

The import_collection task contained in pulp_ansible currently does this:

1. open up an Artifact and search for collection metadata
2. Create a Collection object from that metadata and save it
3. Create ResourceCreated records for the created Collection so the user knows what was created.

Future Workflow with linting + scoring

1. open up an Artifact and search for collection metadata
2. Run the collection through the linter + scorer
3. Create a Collection object from that metadata and save it (along with the scoring + linting data)
4. Create ResourceCreated records for the created Collection so the user knows what was created.

Details

What are the field types on the Collection that need to store scoring or linting result data.

Add role sync exampe of N namepsace names, e.g. two of them

Author: @bmbouter (bmbouter)

Redmine Issue: 8886, https://pulp.plan.io/issues/8886

https://galaxy.ansible.com/api/v1/roles/?namespace__name__in=ovirt,willow

As a Galaxy V3 user, I can search for what collections provide specific files

Author: @bmbouter (bmbouter)

Redmine Issue: 5851, https://pulp.plan.io/issues/5851

Similar to the yum "whatprovides" but reports on which collections contain a given path.

GET /pulp_ansible/galaxy/<path:path>/api/v3/collections/q=netapp

This will fulltext search the file paths stored in the DIGEST data for collections. That DIGEST data is stored/served as part of https://pulp.plan.io/issues/5850.

As a user, I want to able to distinguish signed and unsigned Collections

Author: @fao89 (fao89)

Redmine Issue: 8049, https://pulp.plan.io/issues/8049

Currently, as CollectionVersions are distinguished by name, namespace, and version.
This makes pulp keeping the artifact from the first provenance and ignoring artifacts from other provenances.
Ref: #8047

pulp_ansible should allow and differentiate signed from unsigned Collections.
One possibility would be adding the provenance to the content's identity.

Note: At pulp_rpm, signed and unsigned content cannot be part of the same repo version
https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/models/package.py#L233

pulp / pulp_ansible Goto Github PK

pulp_ansible's Introduction

pulp_ansible

Collection Support

Configuring Collection Support

How to File an Issue

pulp_ansible's People

Contributors

Stargazers

Watchers

Forkers

pulp_ansible's Issues

Background

Problem

Solution

Dependency Downloading when syncing collections:

Problem

Solutions

Background

Requirements

Background

Desire

Details

Background

Solution

Problem

Solution

Current Workflow

Future Workflow with linting + scoring

Details

Recommend Projects

Recommend Topics

Recommend Org