Code Monkey home page Code Monkey logo

pulp_rpm's Introduction

pulp_rpm Plugin

Rpm Nightly CI/CD

This is the pulp_rpm Plugin for Pulp Project 3.0+. This plugin provides support for RPM family content types, similar to the pulp_rpm plugin for Pulp 2.

For more information, please see the documentation or the Pulp project page.

pulp_rpm's People

Contributors

asmacdo avatar barnabycourt avatar beav avatar bmbouter avatar codeheeler avatar daviddavis avatar dkliban avatar dralley avatar fao89 avatar fdobrovolny avatar gerrod3 avatar ggainey avatar goosemania avatar hstct avatar ichimonji10 avatar ipanova avatar jeremycline avatar jlconnor avatar jortel avatar jwmatthews avatar lubosmj avatar mdellweg avatar mhrivnak avatar mikedep333 avatar pavelpicka avatar pcreech avatar pedro-psb avatar pkilambi avatar pulpbot avatar werwty avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

ehelms jwmatthews jlsherrill jessegonzalez pkilambi lzap daviddavis arnisoph mykelalvis preethit bowlofeggs beav atix-ag asmacdo dkliban bmbouter ipanova jeremycline mhrivnak skarmark andreagiardini msutter pombreda release-engineering hjensas roysjosh lfisher47 pombredanne zjhuntin tehsmyers florianheigl pcreech goosemania aeria shubham90 dokuhebi ulif bkearney mahendra-shivaswamy-devops eslynunez fdobrovolny werwty mibanescu whythefyouflying s1artie simonzays dralley chtaylo2 nightstar alexxnica kryndex dparalen bmcivor dekimsey pavelpicka zxiong codeheeler bherrin3 jayz12138 rochacbruno evgeni rohanpm ragabala nixocio mikedep333 glbyers ggainey daberkow msehnout lubosmj fao89 poil ianballou midnightercz h0neybadger hao-yu ehsan-rezaei emilyzheng wbclark sjha4 pieterlexis cognifloyd pmoravec mdellweg quba42 kong op-ct gerrod3 melcorr dannysauer holgerhees ctwo0002 taftsanders zapp42 jvlcek rajulkumar git-hyagi javihernandez ziegenberg jkasnar2410

pulp_rpm's Issues

As a user, I can upload modular content

Author: @dralley (dalley)

Redmine Issue: 6271, https://pulp.plan.io/issues/6271

A user should be able to upload a module metadata YAML file to Pulp, which will create modulemd and modulemd-default content units.

When new modules are created, they should be automatically associated with any referenced RPMs already present or which are added in the future.

In case modulemd-deafults, there is restriction that only one modulemd-defaults can exist for a repository, so we will either need to have a total replacement strategy, a merge strategy like with Errata, or require the user to delete the existing ones first. Validation will be very important.

A test scenario for an upload of a single RPM package

Author: @lubosmj (lmjachky)

Redmine Issue: 5805, https://pulp.plan.io/issues/5805

The commit merged via #1517 resolved the issue https://pulp.plan.io/issues/5699. It seems like we are missing tests which validate a single upload and its publication.

Therefore, the following test scenario is proposed:
1. Create an artifact:

http --form POST :24817/pulp/api/v3/artifacts/ file@./giraffe-0.67-2.noarch.rpm

2. Create an RPM package:

http POST :24817/pulp/api/v3/content/rpm/packages/ artifact="/pulp/api/v3/artifacts/ce225ae1-9cbd-4c20-8f31-c16ab2f0a6c6/" relative_path=giraffe-0.67-2.noarch.rpm

3. Add the created content to an empty repository:

http POST :24817/pulp/api/v3/repositories/rpm/rpm/75c0c59f-d453-42d3-8d6d-a53561b5d516/modify/ add_content_units:="[\"/pulp/api/v3/content/rpm/packages/3c2ac34c-470e-41b7-85b3-b3830efd4722/\"]"

4. Create a publication and check if the corresponding task was finished with a success:

http :24817/pulp/api/v3/publications/rpm/rpm/ repository=/pulp/api/v3/repositories/rpm/rpm/75c0c59f-d453-42d3-8d6d-a53561b5d516/

Ensure required fields are always set on the models

Author: @goosemania (ttereshc)

Redmine Issue: 5744, https://pulp.plan.io/issues/5744

Currently there is no enforcement at the db level to require certain fields for the majority of content models.

Package needs NEVRA, pkgId, checksum_type at minimum.
UpdateRecord - id, issued_date, updated_date, digest
UpdateCollectionPackage - NEVRA
Modulemd - NSVCA
Module-defaults - module, digest
PackageGroup/Environment/Category - id, digest
PackageLangpacks - matches, digest
RepoMetadataFile - data_type, checksum

DistributionTree - header_version, release_name, release_version, arch, build_timestamp
Checksum - path, checksum
Image - name, path
Addon - addon_id, uid, name, type, packages
Variant - variant_id, uid, name, type, packages

SON object inside obj['conditional_package_names']

Line 123 in comps_utils.py. I seem to get a SON object from obj['conditional_package_names'], which you can not iterate through that way (you get Exception: Too many values to unpack).

Adding .items() fixes the issue , which then returns a proper list. But I guess I'm not suppose to get a SON-object inside the obj['conditional_package_names'] am I right?

This is what obj['conditional_package_names'] looks like: SON([(u'libreoffice-langpack-en', u'libreoffice-core')])

Repository that I'm trying to sync is: rhel-6-x86_64-optional

Check is_modular flag unit is present after RPM upload.

Author: @fao89 (fao89)

Redmine Issue: 6341, https://pulp.plan.io/issues/6341

  1. Upload a modular RPM
  2. Upload a non-modular RPM
  3. Filter the modular and non_modular units
  4. Check whether the modular and non_modular units returned by this filter is accurate.

Reference:
#4930

CheckIsModularFlagAfterSyncTestCase - https://github.com/pulp/Pulp-2-Tests/blob/master/pulp_2_tests/tests/rpm/api_v2/test_modularity.py#L146

Fixture: https://repos.fedorapeople.org/repos/pulp/pulp/fixtures/rpm-with-modular/

Travis PR test runner needs to be able to skip certain tests every time, leaving them for the nightly run that runs all tests.

Author: @bmbouter (bmbouter)

Redmine Issue: 5354, https://pulp.plan.io/issues/5354

Some tests run long for example a full EPEL sync test takes 50 minutes. Each plugin iteslf could handle this, but it would be much better to have the plugin_template provide facilities.

Specifically a way to have a subset of tests run on Travis at PR time (instead of all tests) would be ideal.

Re-enable test_sync_advisory_no_updated_date

Author: @pavelpicka (ppicka)

Redmine Issue: 6605, https://pulp.plan.io/issues/6605

If advisory has same ID, version but update_date missing sync will fail.

If updated_date missing in advisory issued_date is not take into consideration in advisory conflict resolution time.

How to reproduce:

git clone https://github.com/pulp/pulp-fixtures
cd pulp-fixtures
1. create fixtures
make fixtures/rpm-unsigned
make fixtures/rpm-advisory-no-update-date
  1. create repository
  2. sync remote rpm-unsigned
  3. re-sync same repo with rpm-advisory-no-update-date remote
"error" {
"description": "'<' not supported between instances of 'datetime.datetime' and 'NoneType'"
...}

Test RHEL repositories daily in CI

Author: @goosemania (ttereshc)

Redmine Issue: 6765, https://pulp.plan.io/issues/6765

Motivation

Every now and then users report issues which are specific to RHEL content but currently we are testing only CentOS and Fedora.
We also need to test certificates, see #6735.

Solution

Get a test cert for CDN and configure it in Travis, see https://pulp.plan.io/issues/6735#How-to-add-these-tests.

Add tests to sync and publish the latest RHEL6, RHEL7 and RHEL8 (BaseOS and AppStream) repos. Those tests should run daily.

Assert that API calls related to orphans function correctly

Author: @fao89 (fao89)

Redmine Issue: 6343, https://pulp.plan.io/issues/6343

  1. Create an RPM repository and populate it with content. Delete the repository, thus leaving behind orphans.
  2. Make several orphan-related API calls, and assert that each call has the desired effect.

Note: kickstart repos also have sub-repos

Reference:
#1268

OrphansTestCase - https://github.com/pulp/Pulp-2-Tests/blob/master/pulp_2_tests/tests/rpm/api_v2/test_orphan_remove.py#L33

Fixtures:
https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-kickstart/
https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-signed/

Test - As a user, I can sync modular content

Author: kersom (kersom)

Redmine Issue: 5408, https://pulp.plan.io/issues/5408

As a user, I can sync modulemd content

  • support compressed and uncompressed version of modules.yaml
  • track a relationship with artifacts
  • mark RPMs as modular

As a user, I can sync modulemd_defaults content

  • replace with the newer one if checksum changed (can be done in RemoveDuplicates stage)
  • do we need to track a relationship to a modulemd? maybe not, can be painful (this part needs additional discussion agreement)

As a user, I can sync modular erratum content

Reuse Content from other repos

Author: @fao89 (fao89)

Redmine Issue: 6344, https://pulp.plan.io/issues/6344

If two repositories have some identical content, then Pulp should be able to re-use that content. This is true even if the content is placed in differing locations in the two repositories. Do the following:

Scenario 1:

  1. Create a pair of repositories. Give the two repositories a download policy of either on demand or streamed. Give
    the two repositories differing urls, where the urls reference a pair of repositories with some identical content and differing layouts.
  2. Sync each of the repositories.
  3. Publish each of the repositories.
  4. Fetch an identical content unit from each of the two repositories.

Scenario 2:

  1. Create a pair of repositories. Give the two repositories a download policy of either on demand or streamed. Give
    the two repositories differing urls, where the urls reference a pair of repositories with some identical content and differing layouts.
  2. Sync each of the repositories.
  3. Re-sync each of the repositories.
  4. Publish each of the repositories.
  5. Fetch an identical content unit from each of the two repositories.

References:
#6229
#6303

Applicability misreporting as [] for exactly one EL7 consumer

Author: iballou (iballou)

Bugzilla: https://bugzilla.redhat.com/buglist.cgi?quicksearch=1854966
Redmine Issue: 7105, https://pulp.plan.io/issues/7105

Related issue: https://pulp.plan.io/issues/6851

I had a number of Katello users fix their applicability issues by applying the patch that resulted from the issue above. However, a couple users are now reporting that there is always one client that isn't reporting applicability properly. The users report that the applicability changes over time. One host might start showing applicability again, but then another fails. Full details are in the following thread, starting at the linked message: https://community.theforeman.org/t/katello-3-15-doesnt-show-available-package-updates-anymore/18948/89

I suspect the issue is in Pulp because the package profiles and bound repositories are correct on the consumers that are exhibiting the applicability issue.

Pulp versions in use:
pulp-server-2.21.2-1.el7.noarch
pulp-rpm-plugins-2.21.2-1.el7.noarch
(Plus pulp/pulp#3990 patched in)

Pulp does not resync yum metadata files on change

Author: bherring (bherring)

Redmine Issue: 4824, https://pulp.plan.io/issues/4824

Pulp uses revision numbers in repomd.xml to determine if contents need to be updated on sync. However "modifyrepo" does not generate new revision numbers for non rpm data.

Steps:
1) Setup the following repo

$ mkdir /tmp/my-data
$ cd /tmp/my-data
$ wget https://partha.fedorapeople.org/test-repos/rpm-with-productid/elephant-0.3-0.8.noarch.rpm
$ createrepo .
$ echo "100000" >> productid
$ modifyrepo  --mdtype=productid productid repodata
$ grep revision repodata/repomd.xml
  <revision>1554217257</revision>

2) Sync this repo
3) Now update the repo

$ cd /tmp/my-data
$ echo "100001" >> productid
$ modifyrepo  --mdtype=productid productid repodata
$ grep revision repodata/repomd.xml
  <revision>1554217257</revision>

Notice that the revision number did not change even though a metadata file got updated. Try resyncing this change and notice that the productid change will get ignored.

Store a digest of the fields for an UpdateRecord

Author: daviddavis (daviddavis)

Redmine Issue: 3942, https://pulp.plan.io/issues/3942

Currently, we have a digest field that captures a hash digest of the update record's xml. The problem is that this is very fragile and could change based on stuff like whitespace or character encoding.

Solution

Instead, store the hash digest of UpdateRecord's fields along with the its UpdateCollections and UpdateCollectionPackages. using a detereministic data structure like an ordered dictionary.

Test that display_order is not None in comps.xml

Author: @fao89 (fao89)

Redmine Issue: 6340, https://pulp.plan.io/issues/6340

  1. Create a rpm repo and a remote.
  2. Sync the repo with the remote.
  3. Publish and distribute the repo.
  4. Check whether display_order is not None in comps.xml.

Reference:
#1787

similar test: ValidateNoChecksumTagTestCase - https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/tests/functional/api/test_publish.py#L232

Fixture: https://repos.fedorapeople.org/pulp/pulp/fixtures/srpm/

As a user, I want to create content from chunked uploads

Author: lieter (lieter)

Redmine Issue: 6719, https://pulp.plan.io/issues/6719

The pulpcore API supports uploading content for an artifact in a chunked way (https://docs.pulpproject.org/en/3.3/restapi.html#operation/uploads_update). It would be great if the RPM plugin could do this for a Package as well. This will allow a user to upload an RPM in a chunked way, without having to go through the motions of making a Package from the Artifact.

Solution

Extend SingleArtifactContentUploadSerializer to accept a chunked upload id. This upload id can be used instead of a file to create a content unit.

Unit test for is_previous_revision

Author: @fao89 (fao89)

Redmine Issue: 6408, https://pulp.plan.io/issues/6408

https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/shared_utils.py#L35

is_previous_revision is utilized on sync optimization, we need to test if it could handle revisions in many formats (timestamp, string, None).

Compare revisions in many formats

is_previous_revision("3.0.0", "2.0.0") -> False
is_previous_revision("2.0.0", "3.0.0") -> True
is_previous_revision("2.0.0", "2.0.0") -> True
is_previous_revision(1234, 1235) -> True  #timestamp
is_previous_revision(1235, 1234) -> False  #timestamp
is_previous_revision(1235, "2.0.0") -> False
is_previous_revision("2.0.0", 1235) -> False
is_previous_revision("3.1.0", "2.15.0") -> False
is_previous_revision("2.15.0", "3.1.0") -> True
is_previous_revision(None, "3.1.0") -> False
is_previous_revision("3.1.0", None) -> False
is_previous_revision(None, 1234) -> False
is_previous_revision(1235, None) -> False

pytest parametrize: http://doc.pytest.org/en/latest/example/parametrize.html#parametrizing-conditional-raising

Test - As a user, I have simple content copy between repositories

Author: kersom (kersom)

Redmine Issue: 5276, https://pulp.plan.io/issues/5276

Simple content copy means that content units can be copied from one repository into another repository without considering content-type-specific invariants, such as making sure dependencies are satisfied.

The most straightforwards current design is to model this after one-shot upload. A new URL endpoint will be defined at /api/v3/rpm/copy/. POSTing to that endpoint with valid parameters will yield an asynchronous task, which will result in a new repository version in the target repository.

The user workflow should look like this:

http --form POST http://localhost:24817/pulp/api/v3/rpm/copy/ source_repository=${S_REPO_HREF} destination_repository=${D_REPO_HREF}

Both the repository parameters are required. When called, all content will be copied.

The additions to Django boilerplate will look something like this (see also: copy.py for an example of the function that does the work, and OneShotUploadSerializer in serializers.py)

urls.py

urlpatterns = [
    url(r'rpm/upload/$', OneShotUploadView.as_view()),
    url(r'rpm/copy/$', CopyView.as_view())
]

can't delete kickstart repository

Author: @dkliban ([email protected])

Redmine Issue: 6629, https://pulp.plan.io/issues/6629

When I tried to delete a kickstart repository I got the following error in my Task:

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 ~]$ http :/pulp/api/v3/tasks/c59c1a92-1ce5-4424-a948-7fbf1124b4f5/
HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: Keep-Alive
Content-Length: 1755
Content-Type: application/json
Date: Fri, 01 May 2020 20:13:22 GMT
Keep-Alive: timeout=5, max=10000
Server: gunicorn/20.0.4
Vary: Accept,Cookie
X-Frame-Options: SAMEORIGIN

{
    "child_tasks": [],
    "created_resources": [],
    "error": {
        "description": "(\"Cannot delete some instances of model 'Repository' because they are referenced through a protected foreign key: 'Variant.repository'\", <QuerySet [<Variant: BaseOS>]>)",
        "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 886, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 664, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 670, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/home/vagrant/devel/pulpcore/pulpcore/app/tasks/repository.py\", line 22, in delete\n    models.Repository.objects.filter(pk=repo_id).delete()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/query.py\", line 710, in delete\n    collector.collect(del_query)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/deletion.py\", line 224, in collect\n    field.remote_field.on_delete(self, field, sub_objs, self.using)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/deletion.py\", line 27, in PROTECT\n    sub_objs\n"
    },
    "finished_at": "2020-05-01T20:13:12.201492Z",
    "name": "pulpcore.app.tasks.repository.delete",
    "parent_task": null,
    "progress_reports": [],
    "pulp_created": "2020-05-01T20:13:11.926923Z",
    "pulp_href": "/pulp/api/v3/tasks/c59c1a92-1ce5-4424-a948-7fbf1124b4f5/",
    "reserved_resources_record": [
        "/pulp/api/v3/repositories/rpm/rpm/9581879e-4b4c-4f53-86e9-90319e835ffb/"
    ],
    "started_at": "2020-05-01T20:13:12.082540Z",
    "state": "failed",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/82124443-d497-4889-9075-bed24a6cb29c/"
}

Ignoring --checksum-type option at metadata

Hi, I'm syncing a CentOS 5 repository for network installations and found a problem, RHEL5 family doesn't support 'sha256' hashing in xml repodata. Then I force to generate metadata with simple 'sha' with --checksum-type=sha1 in rpm create repo command.

But the group metadata is ignoring this parameter and generating metadata with sha256 checksum and provocates installation fail. Or yum grouplist in a running machine also fails.

<data type="group">
<locationhref="repodata/734915cb5f5a01aaccbcd072383947abf6a5644d52e092f9497b03e9d5270f73-group.xml.gz"/>
<checksum type="sha256">
734915cb5f5a01aaccbcd072383947abf6a5644d52e092f9497b03e9d5270f73
</checksum>
<timestamp>
1362049622.33
</timestamp>
<open-checksum type="sha256">
ed7ba6f88964212c9003b6f9e8f7cebf57a53a3b3a50f86bf943c1b092f546da
</open-checksum>
</data>

Test that in a published repo repodata filenames are not prefixed with a checksum

Author: kersom (kersom)

Redmine Issue: 4758, https://pulp.plan.io/issues/4758

Checksum in a filename helps with caching, allows to have old repodata in the directory, is a default way of publishing in Fedora/Centos/SUSE repos.
Pulp2 publishes with a checksum in the filenames.


<?xml version="1.0" encoding="UTF-8"?>
<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">
  <revision>1537430242</revision>
  <data type="primary">
    <checksum type="sha256">7d053b4c5880ecec3b8d5c54e0f436860953fbf69d936f1c54e10346c775e40b</checksum>
    <open-checksum type="sha256">7ff98c3c80e3a82b6addc196de712f5744af8283bb51d39d053c9000fae03e57</open-checksum>
    <location href="repodata/primary.xml.gz"/>   <------------------------------------ no checksum prefix
    <timestamp>1537430242</timestamp>
    <size>4164</size>
    <open-size>34974</open-size>
  </data>
  <data type="filelists">
    <checksum type="sha256">761a77c037257b989cbb0c65413419e55c5ee79ef4b0b6a33d262f1eb011443c</checksum>
    <open-checksum type="sha256">cbfd64a483c9888d3aa184938f92dd2427c6d6dd298cf83ce9d5f08359ad608d</open-checksum>
    <location href="repodata/filelists.xml.gz"/>    <------------------------------------ no checksum prefix
    <timestamp>1537430242</timestamp>
    <size>2110</size>
    <open-size>6912</open-size>
  </data>
  <data type="other">
    <checksum type="sha256">484e0c3f42959669ae5e627db74b944c160e71097be714d0394b255b8f92c27f</checksum>
    <open-checksum type="sha256">1a66e06a6a6a45722b7e712c05ce780744006c582f89ecbec17271d42e7cd815</open-checksum>
    <location href="repodata/other.xml.gz"/>     <------------------------------------ no checksum prefix
    <timestamp>1537430242</timestamp>
    <size>1933</size>
    <open-size>5830</open-size>
  </data>
    <data type="updateinfo">
    <checksum type="sha256">2db62d73ee56ed9cb5e6c4b5d2012f7b1625cbf530b63e48eafbf8d9ec4f1949</checksum>
    <open-checksum type="sha256">c9a07b3c82c402ff60d5f47b29ef10f780794d074c2ec968324a8485ca55acbd</open-checksum>
    <location href="repodata/updateinfo.xml.gz"/>  <------------------------------------ no checksum prefix
    <timestamp>1537430242</timestamp>
    <size>906</size>
    <open-size>32964</open-size>
  </data>
  ...
</repomd>

code in progress report is not unique in case distribution tree has subrepos

Author: @goosemania (ttereshc)

Redmine Issue: 7288, https://pulp.plan.io/issues/7288

Sync a repo which has a distribution tree with sub repos (more than 1 variant or addon).
This one https://fixtures.pulpproject.org/rpm-distribution-tree/ has 1 main and 4 sub repos. Thus the same code will be seen up to 5 times.

E.g.:

{
    "child_tasks": [],
    "created_resources": [
        "/pulp/api/v3/repositories/rpm/rpm/be452987-051d-40ac-bcf3-6188678a3fce/versions/1/"
    ],
    "error": null,
    "finished_at": "2020-08-06T12:39:51.557658Z",
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
    "parent_task": null,
    "progress_reports": [

        ...      
        {
            "code": "parsing.packages",
            "done": 5,
            "message": "Parsed Packages",
            "state": "completed",
            "suffix": null,
            "total": 5
        },
        ...

        {
            "code": "parsing.packages",
            "done": 35,
            "message": "Parsed Packages",
            "state": "completed",
            "suffix": null,
            "total": 35
        },

        {
            "code": "parsing.packages",
            "done": 2,
            "message": "Parsed Packages",
            "state": "completed",
            "suffix": null,
            "total": 2
        },
        {
            "code": "parsing.packages",
            "done": 3,
            "message": "Parsed Packages",
            "state": "completed",
            "suffix": null,
            "total": 3
        },
        {
            "code": "parsing.packages",
            "done": 1,
            "message": "Parsed Packages",
            "state": "completed",
            "suffix": null,
            "total": 1
        },
        ...
    ],
    "pulp_created": "2020-08-06T12:36:27.046600Z",
    "pulp_href": "/pulp/api/v3/tasks/970fe05a-4599-4a34-aafc-6204c0bee823/",
    "reserved_resources_record": [
        "/pulp/api/v3/remotes/rpm/rpm/ce3e77e3-44eb-4039-a198-32da85ac6fee/",
        "/pulp/api/v3/repositories/rpm/rpm/be452987-051d-40ac-bcf3-6188678a3fce/"
    ],
    "started_at": "2020-08-06T12:36:27.172623Z",
    "state": "completed",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/c3b1489e-90b5-48e1-9e03-3d7336f1e318/"
}

The optimization progress report code can be seen multiple times as well if the main repo has changes while sub repos haven't changed.

{
    "child_tasks": [],
    "created_resources": [
        "/pulp/api/v3/repositories/rpm/rpm/be452987-051d-40ac-bcf3-6188678a3fce/versions/3/"
    ],
    "error": null,
    "finished_at": "2020-08-06T12:41:52.933645Z",
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
    "parent_task": null,
    "progress_reports": [
        {
            "code": "optimizing.sync",
            "done": 1,
            "message": "Optimizing Sync",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "optimizing.sync",
            "done": 1,
            "message": "Optimizing Sync",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "optimizing.sync",
            "done": 1,
            "message": "Optimizing Sync",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "optimizing.sync",
            "done": 1,
            "message": "Optimizing Sync",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "parsing.comps",
            "done": 3,
            "message": "Parsed Comps",
            "state": "completed",
            "suffix": null,
            "total": 3
        },
        {
            "code": "parsing.advisories",
            "done": 0,
            "message": "Parsed Advisories",
            "state": "completed",
            "suffix": null,
            "total": 0
        },
        {
            "code": "parsing.packages",
            "done": 1,
            "message": "Parsed Packages",
            "state": "completed",
            "suffix": null,
            "total": 1
        },
        {
            "code": "downloading.metadata",
            "done": 5,
            "message": "Downloading Metadata Files",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "associating.content",
            "done": 1,
            "message": "Associating Content",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "downloading.artifacts",
            "done": 1,
            "message": "Downloading Artifacts",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "unassociating.content",
            "done": 0,
            "message": "Un-Associating Content",
            "state": "completed",
            "suffix": null,
            "total": null
        }
    ],
    "pulp_created": "2020-08-06T12:40:49.725072Z",
    "pulp_href": "/pulp/api/v3/tasks/50cb4c4d-ca4a-448c-a3f3-2709b9495df5/",
    "reserved_resources_record": [
        "/pulp/api/v3/remotes/rpm/rpm/ce3e77e3-44eb-4039-a198-32da85ac6fee/",
        "/pulp/api/v3/repositories/rpm/rpm/be452987-051d-40ac-bcf3-6188678a3fce/"
    ],
    "started_at": "2020-08-06T12:40:49.845731Z",
    "state": "completed",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/4194635d-e1f1-4224-a687-0d6705181ec4/"
}

Not all images are created as Image instances for a distribution tree

Author: @goosemania (ttereshc)

Redmine Issue: 7175, https://pulp.plan.io/issues/7175

Motivation

It's very confusing to list images (see section images ) and see only some of them, while having other images (from stage2 of treeinfo) on the main model,

        {
            "pulp_href": "/pulp/api/v3/content/rpm/distribution_trees/b94bb596-4aa7-4920-938e-66360cbba652/",
            "header_version": "1.2",
            "release_name": "CentOS Linux",
            "release_short": "CentOS",
            "release_version": "8",
            "release_is_layered": false,
            "base_product_name": null,
            "base_product_short": null,
            "base_product_version": null,
            "arch": "x86_64",
            "build_timestamp": 1591654006.0,
            "instimage": null,                                    <------------ instimage
            "mainimage": "images/install.img",                    <------------ mainimage
            "discnum": null,
            "totaldiscs": null,
            "addons": [],
            "checksums": [
                {
                    "path": "images/pxeboot/vmlinuz",
                    "checksum": "sha256:151fb7b594cb096e25769688cc748eefafeea133894878271bf9c428dba2cc71"
                },
                {
                    "path": "images/pxeboot/initrd.img",
                    "checksum": "sha256:dbffac7f93e35b936dfecc73cbf47e79f7c304fbd84b5df3aaf896ed695ec28e"
                },
                {
                    "path": "images/install.img",
                    "checksum": "sha256:4807e94ec4bd55b02a0f217dd19d34a4824ab04466092dd3ff9ca4bb67e8c3b6"
                },
                {
                    "path": "images/efiboot.img",
                    "checksum": "sha256:39659e208d746e1afc88784f12d8bbd58b074aafca4a65b70be877e22afb3936"
                },
                {
                    "path": "images/boot.iso",
                    "checksum": "sha256:c67876a5602faa17f68b40ccf2628799b87454aa67700f0f57eec15c6ccdd98c"
                }
            ],
            "images": [
                {
                    "name": "kernel",
                    "path": "images/pxeboot/vmlinuz",
                    "platforms": "x86_64, xen",
                    "artifact": null
                },
                {
                    "name": "initrd",
                    "path": "images/pxeboot/initrd.img",
                    "platforms": "x86_64, xen",
                    "artifact": null
                },
                {
                    "name": "efiboot.img",
                    "path": "images/efiboot.img",
                    "platforms": "x86_64",
                    "artifact": null
                },
                {
                    "name": "boot.iso",
                    "path": "images/boot.iso",
                    "platforms": "x86_64",
                    "artifact": null
                }
            ],
            "variants": [
                {
                    "variant_id": "BaseOS",
                    "uid": "BaseOS",
                    "name": "BaseOS",
                    "type": "variant",
                    "packages": "Packages",
                    "source_packages": null,
                    "source_repository": null,
                    "debug_packages": null,
                    "debug_repository": null,
                    "identity": null
                },
                {
                    "variant_id": "AppStream",
                    "uid": "AppStream",
                    "name": "AppStream",
                    "type": "variant",
                    "packages": "AppStream/Packages",
                    "source_packages": null,
                    "source_repository": null,
                    "debug_packages": null,
                    "debug_repository": null,
                    "identity": null
                }
            ]
        }

To reproduce sync http://mirror.linux.duke.edu/pub/centos/8/BaseOS/x86_64/kickstart/

The images/install.img is not present among images but it's a part of a DistributionTree model itself. It is also present in the contentartifact table and is published correctly.

pulp=> select * from rpm_image;
               pulp_id                |         pulp_created          |       pulp_last_updated       |    name     |           path            |  platforms  
|         distribution_tree_id         
--------------------------------------+-------------------------------+-------------------------------+-------------+---------------------------+-------------
+--------------------------------------
 6fed6e01-494a-4c77-9e1c-c85c07c9e48e | 2020-07-20 18:08:35.447812+00 | 2020-07-20 18:08:35.447826+00 | boot.iso    | images/boot.iso           | x86_64      
| b94bb596-4aa7-4920-938e-66360cbba652
 593b55a2-9474-437c-83ce-34b87885fa55 | 2020-07-20 18:08:35.447857+00 | 2020-07-20 18:08:35.447865+00 | efiboot.img | images/efiboot.img        | x86_64      
| b94bb596-4aa7-4920-938e-66360cbba652
 82677b85-e707-4a28-affe-7a933cc48dbe | 2020-07-20 18:08:35.447889+00 | 2020-07-20 18:08:35.447896+00 | initrd      | images/pxeboot/initrd.img | x86_64, xen 
| b94bb596-4aa7-4920-938e-66360cbba652
 1d8d8515-3fc0-410d-ac02-6dd317dac708 | 2020-07-20 18:08:35.44792+00  | 2020-07-20 18:08:35.447927+00 | kernel      | images/pxeboot/vmlinuz    | x86_64, xen 
| b94bb596-4aa7-4920-938e-66360cbba652
(4 rows)

pulp=> select * from core_contentartifact where content_id='b94bb596-4aa7-4920-938e-66360cbba652';
               pulp_id                |         pulp_created          |       pulp_last_updated       |       relative_path       |             artifact_id   
           |              content_id              
--------------------------------------+-------------------------------+-------------------------------+---------------------------+---------------------------
-----------+--------------------------------------
 10ecfd6e-2046-4653-9313-508d4b112976 | 2020-07-20 18:08:35.414974+00 | 2020-07-20 18:08:35.41498+00  | .treeinfo                 | f8d285ea-c394-4da7-a0a4-40
e61b30f443 | b94bb596-4aa7-4920-938e-66360cbba652
 0bdfcf83-6d44-43aa-a299-1844d2ccee44 | 2020-07-20 18:08:35.414996+00 | 2020-07-20 18:08:35.415002+00 | images/boot.iso           |                           
           | b94bb596-4aa7-4920-938e-66360cbba652
 52cd8187-02e5-4c3a-a56e-7bbb12b064f0 | 2020-07-20 18:08:35.415016+00 | 2020-07-20 18:08:35.415022+00 | images/efiboot.img        |                           
           | b94bb596-4aa7-4920-938e-66360cbba652
 f960003a-9cb5-4d19-a4b2-a72a0bcaa07d | 2020-07-20 18:08:35.415036+00 | 2020-07-20 18:08:35.415042+00 | images/pxeboot/initrd.img |                           
           | b94bb596-4aa7-4920-938e-66360cbba652
 78d19586-663a-4de3-81d7-811d8490eaad | 2020-07-20 18:08:35.415056+00 | 2020-07-20 18:08:35.415062+00 | images/pxeboot/vmlinuz    |                           
           | b94bb596-4aa7-4920-938e-66360cbba652
 cd5c5408-f76e-482f-8cd3-8b941c412588 | 2020-07-20 18:08:35.415076+00 | 2020-07-20 18:08:35.415081+00 | images/install.img        |                           
           | b94bb596-4aa7-4920-938e-66360cbba652
(6 rows)

Proposal

  • Have all images created as instances of the Image model.
  • Allow platfroms to be null for images like mainimage which are not tight to a platform.
  • Remove mainimage and instimage from the distribution tree model

Data migration is needed to alter the schema and to create missing images.

RPM plugin needs dnf to test docs script

Author: @fao89 (fao89)

Redmine Issue: 6783, https://pulp.plan.io/issues/6783

Currently, it is not possible to test:
https://github.com/pulp/pulp_rpm/blob/master/docs/_scripts/docs_check_install_with_signed_repodata.sh
because it needs a dnf install to test the installation:
https://github.com/pulp/pulp_rpm/blob/master/docs/_scripts/install_from_signed_repository.sh
As travis runs on ubuntu, for executing this workflow we will need to run docs tests inside the single container

integrate RPM signing and re-signing

Author: carlgeorge (carlgeorge)

Redmine Issue: 1686, https://pulp.plan.io/issues/1686

I think it would be useful for pulp to be able to handle RPM signing.

  • import unsigned RPMs and sign them
  • import signed RPMs and sign them with a different key

If there is already a better alternative for this please let me know.

Thanks!

As an RPM user, I have containers

Author: @bmbouter (bmbouter)

Redmine Issue: 4203, https://pulp.plan.io/issues/4203

This machinery should run when a release tag is pushed indicating a new release, similar to how plugins publish to PyPI already. When activated it should:

1. Install the plugin on top of the generic, most recent core container.
2. start that container and run pulp-smash tests against it
3. If all the tests pass, publish the container to quay.io with credentials

The credentials need to be configured somehow in the config files.

Allow users to filter on package filename in /content/rpm/packages/ for pulp3

Author: jcabrera (jcabrera)

Redmine Issue: 4879, https://pulp.plan.io/issues/4879

Add the possibility to query the file name ("location_href"?) of a package in the contents.

This simplifies the use of jq as we get directly the good href with the use of a GET query

 http :24817/pulp/api/v3/content/rpm/packages/?filename=name-version-release.arch.rpm

I prefer the query version than using jq

http :24817/pulp/api/v3/content/rpm/packages/ | jq -r '.results[] | select( .location_href == "name-version-release.arch.rpm")'

There could be a limitation when there are to much contents because the pagination prevents to look for all contents.

Repomd compression behaves differently for S3

Author: @fao89 (fao89)

Redmine Issue: 6476, https://pulp.plan.io/issues/6476

on repomd we have:
primary.xml.gz
update_info.xml.gz
When testing xml.gz behaves like it is not compressed
On the other hand, it behaves as compressed on S3 testing
https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/tests/functional/api/test_publish.py#L51-L67

def read_xml_gz(content):
    """
    Read xml and xml.gz.
    Tests work normally but fails for S3 due '.gz'
    Why is it only compressed for S3?
    """
    with NamedTemporaryFile() as temp_file:
        temp_file.write(content)
        temp_file.seek(0)

        try:
            content_xml = gzip.open(temp_file.name).read()
        except OSError:
            # FIXME: fix this as in CI primary/update_info.xml has '.gz' but it is not gzipped
            content_xml = temp_file.read()
        return content_xml

When uploading an invalid rpm such as a non utf8 RPM, users should get a friendly error message

Author: kersom (kersom)

Redmine Issue: 4215, https://pulp.plan.io/issues/4215

When users upload an invalid RPM such as one that has non-utf8 characters in their headers, they should get a 4xx response and a user friendly error message. Currently they receive a 500.

Steps to reproduce.

http POST http://localhost:8000/pulp/api/v3/repositories/ name=foo

export REPO_HREF=$(http :8000/pulp/api/v3/repositories/ | jq -r '.results[] | select(.name == "foo") | ._href')

wget https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-non-utf-8/rpm-with-non-utf-8-1-1.fc25.noarch.rpm

http --form POST http://localhost:8000/pulp/api/v3/artifacts/ file@./rpm-with-non-utf-8-1-1.fc25.noarch.rpm

http POST http://localhost:8000/pulp/api/v3/content/rpm/packages/ relative_path=rpm-with-non-utf-8-1-1.fc25.noarch.rpm artifact="/pulp/api/v3/artifacts/1/" filename=rpm-with-non-utf8.rpm

Traceback

HTTP/1.1 500 Internal Server Error
Content-Length: 20215
Content-Type: text/plain; charset=utf-8
Date: Thu, 29 Nov 2018 15:41:48 GMT
Server: WSGIServer/0.2 CPython/3.7.1
Vary: Cookie
X-Frame-Options: SAMEORIGIN

UnicodeDecodeError at /pulp/api/v3/content/rpm/packages/
'utf-8' codec can't decode byte 0x80 in position 168: invalid start byte

Request Method: POST
Request URL: http://localhost:8000/pulp/api/v3/content/rpm/packages/
Django Version: 2.1.3
Python Executable: /home/vagrant/.virtualenvs/pulp/bin/python3
Python Version: 3.7.1
Python Path: ['/home/vagrant/.virtualenvs/pulp/bin', '/usr/lib64/python37.zip', '/usr/lib64/python3.7', '/usr/lib64/python3.7/lib-dynload', '/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages', '/home/vagrant/devel/pulp/pulpcore', '/home/vagrant/devel/pulp/plugin', '/home/vagrant/pulp_rpm', '/home/vagrant/.virtualenvs/pulp/lib/python3.7/site-packages']
Server time: Thu, 29 Nov 2018 15:41:48 +0000
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django_filters',
 'drf_yasg',
 'rest_framework',
 'pulpcore.app',
 'pulp_file.app.PulpFilePluginAppConfig',
 'pulp_rpm.app.PulpRpmPluginAppConfig',
 'crispy_forms',
 'django_extensions']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']

Traceback:

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/core/handlers/exception.py" in inner
  34.             response = get_response(request)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/core/handlers/base.py" in _get_response
  126.                 response = self.process_exception_by_middleware(e, request)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/core/handlers/base.py" in _get_response
  124.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
  54.         return view_func(*args, **kwargs)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/viewsets.py" in view
  116.             return self.dispatch(request, *args, **kwargs)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/views.py" in dispatch
  495.             response = self.handle_exception(exc)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/views.py" in handle_exception
  455.             self.raise_uncaught_exception(exc)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/views.py" in dispatch
  492.             response = handler(request, *args, **kwargs)

File "/usr/lib64/python3.7/contextlib.py" in inner
  74.                 return func(*args, **kwds)

File "/home/vagrant/pulp_rpm/pulp_rpm/app/viewsets.py" in create
  96.             package = Package.createrepo_to_dict(cr_pkginfo)

File "/home/vagrant/pulp_rpm/pulp_rpm/app/models.py" in createrepo_to_dict
  243.             'description': getattr(package, CREATEREPO_PACKAGE_ATTRS.DESCRIPTION) or '',

Exception Type: UnicodeDecodeError at /pulp/api/v3/content/rpm/packages/
Exception Value: 'utf-8' codec can't decode byte 0x80 in position 168: invalid start byte
Request information:
USER: admin

GET: No GET data

POST: No POST data

FILES: No FILES data

COOKIES: No cookie data

META:
CONTENT_LENGTH = '137'
CONTENT_TYPE = 'application/json'

[vagrant@p3 ~]$ cat /etc/redhat-release 
Fedora release 29 (Twenty Nine)

Document how to interact with advisories

Author: kersom (kersom)

Redmine Issue: 4149, https://pulp.plan.io/issues/4149

I used the auto generated docs in order to create an errata, and still not clear. Attempt was using the endpoint ` /pulp/api/v3/content/rpm/errata/` .

Add documentation about Errata in Pulp3, and how users can interact with Erratas in Pulp 3, what features are available, and so on.

[EPIC] Ursine RPM Copy dependencies on modular RPMS from Default Modules needs to be added

Author: bherring (bherring)

Redmine Issue: 5055, https://pulp.plan.io/issues/5055

History

There is currently a 'policy decision' that modules in the base repo (of rhel) don't rely on modular rpms, but that it's not an engineering limitation, and that fedora and rhel will likely do so at some point in the future. This is already a consideration for fedora and fedora-updates repos.

At least in Fedora, you can expect that there will be content in the updates-testing repo depending on content in updates-modular and/or fedora-modular.

This means Pulp will eventually need to support multiple source repos for copy.

Problem

As pulp is delivered today, Ursine RPM deps on modular RPMs are NOT covered by test.

Normal, non-modular (ursine) [0] RPMs can depend on module packages.

As an example, the postgres language RPMs depend on the default stream of the postgresql module.

When this is the case, the default module and all artifacts related to that module and the URSINE RPM are ALL copied to the target repo.

At this time, to account for this scenario, a hybrid repo containing URSINE and MODULAR RPMs with modules will be required.

Example

Recursive

Before

  • Copy Ursine RPM "zebra-1.0.rpm" from Repo A to B
  • Zebra requires modular RPM bar-1.0.rpm and has no other dependencies

default module: module-FOO: [foo-1.0.rpm, bar-1.0.rpm, baz-1.0.rpm]

repo A
  |
  |---- chicken-1.1.rpm
  |---- zebra-1.0.rpm
  |----module-FOO
  |----foo-1.0.rpm
  |----bar-1.0.rpm
  |----baz-1.0.rpm

repo B
  |
  |----bar-0.7.rpm

After Copy

Result of copying ursine RPM zebra-1.0.rpm from repo A to repo B:

repo B
 |
 |---- zebra-1.0.rpm
 |----module-FOO
 |----foo-1.0.rpm
 |----bar-1.0.rpm
 |----baz-1.0.rpm
 |----bar-0.7.rpm

All available artifacts are copied, always. There is no way to copy just module on its own,
if any of its artifacts are present in a source repo (repo A).

Solution

  • Add Ursine RPM permutations that exercise this case for recursive and non_recursive Ursine RPM copies in mixed repos with modular RPMs where default streams are and are not defined.

Note:

  • the entire module should be treated as a single content unit to the greatest extent possible. it should copy the module that provides both A and B, all if its artifacts, and, the module default
  • a module RPM should never be treated as just a normal RPM
  • If module has multiple streams, ONLY the artifacts associated with the default module and the default module are copied, correct?
    • only the artifacts associated with the default module stream should ever be considered in the first place. if there is no default stream for a module, then RPMs should not be able to depend on it

References

[0] - https://docs.fedoraproject.org/en-US/modularity/architecture/consuming/dnf-behavior/
[1] - https://docs.pulpproject.org/plugins/pulp_rpm/user-guide/features.html?highlight=modularity

Test that Pulp deals well with missing workers.

Author: @fao89 (fao89)

Redmine Issue: 6348, https://pulp.plan.io/issues/6348

1. Ensure there is only one Pulp worker.
2. Create a repository. Let its feed reference a large repository. (For example, EPEL.)
3. Start a sync. Immediately restart the ``pulp_workers`` service. (It's
   important that ``pulp_workers`` be restarted, not started and stopped.
   For details, see #2835 ) This should cause the first sync to abort.
4. Update the repository. Let its feed reference a small repository. (For
   example, ``https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-unsigned/``.)
5. Start a sync. Verify that it completes. If `Pulp #2835`_ still affects
   Pulp, then the worker will be broken, and the sync will never start.

Reference:
#2835

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.