Problem with proxy and mirroring some package repositories about packeton HOT 7 CLOSED

Seems problem in the docker configuration or cache. I can not reproduce on demo https://demo.packeton.org/proxies/amasty

It may happens if background worker and php-fpm uses different configuration or background worker was not restarted after updating the cache. Please make sure that config/packages/amasty.yaml available for container where packagist:run-workers is running

A similar issue was fixed in version 2.4.0 here. aa3ce9b

from packeton.

@vtsykun Apparently it was a problem with the repository from Amasty itself at the time of creating this issue. Its indeed working as intended.

from packeton.

@vtsykun Again we have same problem I can't get it to work with same amasty repository.

See the file: https://packages.epartment.nl/mirror/hapsastyepa/packages.json

I want to work. What did I wrong?

from packeton.

Hi @rbouma

Can you provide an example of the original metadata format from here?
https://composer.amasty.com/community/packages.json

Or made composer install test with amasty repository.

{
    "repositories": [
        {
            "type": "composer",
            "url": "https://composer.amasty.com/community"
        }
    ],
    "require": {
        "amasty/base": "*"
    }
}

Seems uid is empty in the original metadata

  [ErrorException]      
  Undefined index: uid 

2. About the first error "Service hapsastyepa not found"

Can you login into paketon worker docker container/instance and check the cache?

docker exec -it packeton_worker_name  bash
grep 'hapsastyepa' -r var/cache/prod/

Maybe cache haven't been cleared after a configuration change php bin/console cache:clear. php bin/console cache:clear --env=prod

from packeton.

Installed amasty/base via the official repo:

❯ cat composer.lock
{
    "_readme": [
        "This file locks the dependencies of your project to a known state",
        "Read more about it at https://getcomposer.org/doc/01-basic-usage.md#installing-dependencies",
        "This file is @generated automatically"
    ],
    "content-hash": "de3fc19d39017ecea7a9de890a45a632",
    "packages": [
        {
            "name": "amasty/base",
            "version": "1.14.7",
            "dist": {
                "type": "zip",
                "url": "https://composer.amasty.com/community/packages/amasty/base-1.14.7.zip"
            },
            "require": {
                "ext-dom": "*",
                "php": ">=7.3"
            },
            "type": "magento2-module",
            "autoload": {
                "files": [
                    "registration.php"
                ],
                "psr-4": {
                    "Amasty\\Base\\": ""
                }
            },
            "license": [
                "proprietary"
            ],
            "description": "Amasty Base"
        }
    ],
    "packages-dev": [],
    "aliases": [],
    "minimum-stability": "stable",
    "stability-flags": [],
    "prefer-stable": false,
    "prefer-lowest": false,
    "platform": [],
    "platform-dev": [],
    "plugin-api-version": "2.6.0"
}

See: https://raw.githubusercontent.com/rbouma/amasty-packages/main/packages.json For the full package metadata from amasty repo

For the second issue we run packeton on the server itself instead of running inside docker. But now I don't have this issue any more. Changed the name in the mirror config and refreshed the configs and seems to be working now.

But the mirroring still don't work due to uid.

probably the same error but when I add a package manually it has been found and added. But when I click on the package it gives an error.

See the screenshare:

Stacktrace:

ErrorException:
Warning: Undefined array key "version_normalized"

  at src/Composer/MetadataMinifier.php:28
  at Packeton\Composer\MetadataMinifier->Packeton\Composer\{closure}()
  at usort()
     (src/Composer/MetadataMinifier.php:28)
  at Packeton\Composer\MetadataMinifier->minify()
     (src/Controller/ProxiesController.php:111)
  at Packeton\Controller\ProxiesController->metadata()
     (vendor/symfony/http-kernel/HttpKernel.php:181)
  at Symfony\Component\HttpKernel\HttpKernel->handleRaw()
     (vendor/symfony/http-kernel/HttpKernel.php:76)
  at Symfony\Component\HttpKernel\HttpKernel->handle()
     (vendor/symfony/http-kernel/Kernel.php:197)
  at Symfony\Component\HttpKernel\Kernel->handle()
     (vendor/symfony/runtime/Runner/Symfony/HttpKernelRunner.php:35)
  at Symfony\Component\Runtime\Runner\Symfony\HttpKernelRunner->run()
     (vendor/autoload_runtime.php:29)
  at require_once('/home/packeton/domains/packages.epartment.nl/public_html/vendor/autoload_runtime.php')
     (public/index.php:5)          

from packeton.

Found out this is due its in lazy:true on both api v1 and api v2 mode
or lazy:false and api v2

In https://github.com/composer/composer/blob/4e5be9ee7d924d8efc58d676439b0c7bd18a9ce4/src/Composer/Repository/ComposerRepository.php#L1225-L1240

$this->lazyProvidersUrl is filled if v2 api is used then it used 'metadata-url' or in v1 with lazy is will use 'providers-lazy-url'

Then $hasProviders will return true on https://github.com/composer/composer/blob/4e5be9ee7d924d8efc58d676439b0c7bd18a9ce4/src/Composer/Repository/ComposerRepository.php#L484

Then it will not execute https://github.com/composer/composer/blob/4e5be9ee7d924d8efc58d676439b0c7bd18a9ce4/src/Composer/Repository/ComposerRepository.php#L486-L488 and continue with the code

Then it comes to this line:
https://github.com/composer/composer/blob/4e5be9ee7d924d8efc58d676439b0c7bd18a9ce4/src/Composer/Repository/ComposerRepository.php#L503

Which will execute a lot of code but in this case with uid fail this line is important:
https://github.com/composer/composer/blob/main/src/Composer/Repository/ComposerRepository.php#L932

The only thing i can find is https://github.com/rande/pkgmirror/blob/452f8747413a/mirror/composer/composer_structs.go#L35 where for package there is a definition that per package there needs to be a sort of unique identifier in lazy or v2 mode in combination with mirroring.

I have this now with Amasty and Swissup repositories for mirroring both doesn't have an uid.

solution?
disable lazy mode and use v1 api then it works.

from packeton.

The error has been fixed, you can try now.

For the second issue we run packeton on the server itself instead of running inside docker. But now I don't have this issue any more. Changed the name in the mirror config and refreshed the configs and seems to be working now.

Probably after update configs need to clear cache with command php bin/console cache:clear php bin/console cache:clear --env=prod and restart worker

from packeton.