Improve performance by avoiding loading asset package file for all versions about composer-asset-plugin HOT 22 CLOSED

The research is slow only in the first analysis of the repository, because the plugin looks for all available versions. After all the versions are cached and only the new versions are recovered (native system of the Composer VCS Repository).

I am always taking optimization, but the problem with your proposal, that is it is impossible to recover the dependencies, because they are shown only in the file bower.json or package.json.

If you have a solution for a performance optimization, while retaining the complete management of dependencies, I'm interested.

from composer-asset-plugin.

PS: the limit of Github API can be bypassed with a token of authentication.

from composer-asset-plugin.

Please have a look at the linked commit - the complete management of dependencies still works.

If you want I can try to create a pull request...

from composer-asset-plugin.

I watched your PR justly, is the least of it! :-)
On the other hand, I went next to the nsams/composer@53fc1cb#diff-0937c306b2bc60bd12540a6b8faa462fR151 which retrieves the file bower.json... mea culpa!

I think it would be much more interesting to directly improve the performance of Composer VCS system.

from composer-asset-plugin.

@nsams Given that it is in Composer\Repository\VcsRepository::initialize() that the Lazy Loading should be done, and that for assets, there is a class AssetVcsRepository that extends and override VcsRepository::initialize(), I think we can do it directly here for the assets.

from composer-asset-plugin.

@nsams I may be a solution for make the lazing loading, I'll try tomorrow.

from composer-asset-plugin.

@nsams I just finished the proof of concept, and it is achievable!

The compatibility is always kept with the VCS repositories, but it imports only the informations of asset package (JSON file) only when necessary.

More the dependencies are precise, least there will requests, but the requests below cannot be deleted:

• Retrieving the informations of asset package (JSON file) of root branch
• Retrieving the list of tags
• Retrieving the list of branches

You should feel a big improvement in performance.

from composer-asset-plugin.

Great to hear that!

My tries on that weren't that successful, but glad to hear you found a solution...

now i'm waiting eagerly for your push :D

from composer-asset-plugin.

@nsams I finish the cleaning up and tests, and it will be good!

from composer-asset-plugin.

It's done! (see b87d885)

from composer-asset-plugin.

Looking good. However you have the same problem my implementation has:
take bootstrap as example, it requires jquery >= 1.9.0. That results in jquery bower.json loads for all versions >= 1.9.0 (13 at the moment).

As far as I could see this is because of the Pool::$whitelist which is filled with required packages in RuleSetGenerator::whitelistFromPackage.
Now I have no idea what this whitelist is good for, so I stopped at this point.

from composer-asset-plugin.

Further testing showed that the lazy loading doesn't when updating already installed vendors. The bootstrap example (which depends on jquery) causes all (20) tags to load.

"require": {
    "bower-asset/bootstrap": "3.1.1"
}

from composer-asset-plugin.

This is what I stated above, more you specify the version, less there will be of request.

For have a single request for jquery, you can make:

{
   "require": {
        "bower-asset/bootstrap": "3.1.1",
        "bower-asset/jquery": "2.1.1"
    }
}

Output:

Loading composer repositories with package information
Installing dependencies (including require-dev)
Reading bower.json of bower-asset/bootstrap (v3.1.1)
Importing tag v3.1.1 (3.1.1.0)
Reading bower.json of bower-asset/jquery (2.1.1)
Importing tag 2.1.1 (2.1.1.0)
  - Installing bower-asset/jquery (2.1.1)
    Loading from cache
    Extracting archive

  - Installing bower-asset/bootstrap (v3.1.1)
    Loading from cache
    Extracting archive

Writing lock file
Generating autoload files

from composer-asset-plugin.

Ah, that helps. Not very convenient though.

Testing just showed me that removing a dependency an running composer update (which will delete the dependency from vendor) will load all versions of this page.

from composer-asset-plugin.

For the first point, I know, but I do not see how to change that.
For the second point, I have reproduces the behavior, I'll watch.

from composer-asset-plugin.

The behavior is always present for each update (and not only when you delete a dependency).
The problem is from the method Composer\DependencyResolver\RuleSetGenerator::whitelistFromPackage and Composer\DependencyResolver\RuleSetGenerator::addRulesForPackage

from composer-asset-plugin.

Do you know more about this whitelist? What does it do?

from composer-asset-plugin.

This is to include all dependencies in the queue, otherwise, the dependencies of dependencies are not resolved.

from composer-asset-plugin.

Yes, the Pool checks in computeWhatProvides if the package is in the whitelist, and if not it is skipped. So far I get it.

But why is this whitelist needed? which packages are not in the whitelist and must be skipped?

from composer-asset-plugin.

I wonder me the same question, I think it would be better to ask the question directly to creators of Composer.

from composer-asset-plugin.

2 issues on improving performance for Composer:

• For the Remote File System (see composer/composer#3282)
• For the Solver (see composer/composer#3283)

from composer-asset-plugin.

The issue #52 improves significantly the performance for the update.

from composer-asset-plugin.