Add support for installing module dependencies about r10k HOT 25 CLOSED

+100 for this feature. Definitely.

However, dependency resolution is a bit tricky, so this won't be coming right away, but this is a must-have feature.

from r10k.

After talking to a buddy this Tuesday, it is clear that not everybody wants this, so it should be optional.

from r10k.

If nothing else, I want this feature. :)

I actually talked with a few people who were thinking of implementing an independent dependency and versioning library. I don't know all the details and it could amount to vaporware, but if that picks up steam then it could be directly added in and r10k could get dependency resolution for free.

from r10k.

Sounds good.

from r10k.

I know https://github.com/openSUSE/libsolv aims to be a nice library for resolving dependencies, but I have no idea how usable it is for puppet modules. I do know I'd like something like this :).

from r10k.

@ekohl libsolv seems to be a relatively heavy-weight package considered module dependencies aren't that complicated.
It is written in C and requires CMake for building it, which would mean the r10k installation process would be way more complicated then.

Making use of usually already existing and Ruby-only solutions (gem, Bundler) for dependency resolution might make more sense.

from r10k.

@eliasp that certainly makes sense.

from r10k.

also, supporting librarian-puppet's Puppetfile.lock would be awesome.

from r10k.

IIRC, the version scheme for the Puppet Module Forge is semver 1 or 2 - which is not compatible with gem/bundler versions -- so another version abstraction might be necessary.

I encountered this problem on librarian-puppet and ended up writing a semver parser. Perhaps there is a better library out there to take care of this; but if not, maybe this will help: semantic_version.rb.

from r10k.

@kitchen I've been doing some research and it looks like implementing a Puppetfile.lock will be a necessity to make dependency resolution performant. I still have to nail down the exact details in how to balance ease of updates with dependency resolution, but Puppetfile.lock is in the cards.

Unfortunately it looks like the only standalone dependency resolution tool that's currently available is solve, which is an offline dependency resolution tool. That is, you have to have all the versions of all your libraries available up front, which isn't really an option for a tool like r10k.

One option could be to extract the existing dependency resolution from librarian, so that we can get online dependency resolution without having to bring in the entire librarian project.

from r10k.

Just to put it out there, for the future I'd love to have dependencies done in a better and more flexible way, like e.g similar to how npm does it, https://npmjs.org/doc/json.html#dependencies

I heard that Modulefile is going away at some point(?) to be replaced with a simple metadata.json - that could very well be an opportunity for PL to open up the 'standard' deps format (allow git refs!!)

from r10k.

Someone beat me to it : https://projects.puppetlabs.com/issues/22098

from r10k.

As an update to anyone watching this, I have some preliminary work on the core for dependency resolution, so there is forward movement on this.

from r10k.

Some quotes from the puppet-users group regarding the Modulefile -> metadata.json transition:

metadata.json (part of the puppet module build output) will become the single-source of metadata

The next major version of Puppet will include a greatly revamped Puppet Module Tool that (amongst other improvements) will make it easy to express and validate the new metadata directly in metadata.json.

There's a gem to interact with metadata.json.

To me it seems the sanest would be to support metadata.json with custom fields for non-forge sources, e.g.:

{
    "name": "puppetlabs/mysql",
    "version": "2.1.0",
    [...]
    "dependencies": [
        {
            "name": "foo/bar",
            "version_requirement": ">= 2.2.1",
            "source": "[email protected]/foo/bar.git" <--ADDED
        }
    ]
}

And maybe also allow this kind of declaration for environments instead of using a Puppetfile?

from r10k.

As puppet module install resolves dependencies, is there any option to run it from r10k without the --ignore-dependencies flag? I think it could be an easy solution at least for this kind of modules.

from r10k.

fyi, puppet ticket was moved to; https://tickets.puppetlabs.com/browse/PUP-1181
[PUP-1181] Modulefile dependency should support git repo as source - Puppet Labs Tickets

from r10k.

This would be a dreamed feature

from r10k.

Would love to have this feature as well ! As a Java developer, I am used to dependency management in Maven, and I think a lot can be learn by looking at how it works. Java might not be the most popular technology in the Puppet realm, but please let that aside for a moment. Dependency management is hard. Thought need to go in to things like version ranges, conflict resolution, overriding, optional dependencies, analysis tools, indirect dependencies, ...

I've been thinking a lot about writing a Maven plugin to manage my Puppet workflow (yes, it sounds like heresy). While that might not be everyone's favourite solution, it might start an interesting discussion that might end up in improvements in r10k. Let's see if I find some time to dig into it ...

from r10k.

I wrote a ruby script which fetches modules from forge and clones it into my own git repo. It also resolves dependencies. Maybe this is useful for anyone (let me know if you need more features - I can probably implement them):
https://github.com/bastelfreak/scripts/blob/master/forge_client.rb

from r10k.

I would more see a mechanism that would resolve both dependencies set in metadata.json & Puppetfile, and warn the user on conflicts or unsupported setups

from r10k.

+1 for adding r10k functionality to handle mod dependencies.

from r10k.

+1.
for me - r10k unusable without this feature

from r10k.

First off, we're incrementally moving GitHub issues for r10k from here to JIRA (see https://tickets.puppetlabs.com/browse/RK), so I'm closing out this issue in favor of RK-3.

Since it doesn't look like I've discussed it here, I'll briefly touch on why this has been so long coming.

To begin with, yes - dependency resolution would be very nice and it's quite painful to live without it. I hate the fact that right now users have to manually unroll dependencies. If this was something that we could just stuff into r10k and go from there then I would have done it long ago.

Unfortunately it's not as simple as grabbing a dependency resolver and slapping it into r10k. As @adreyer can attest to, the current dependency resolver in Puppet is pretty flawed - right now it can give incorrect results when resolving dependencies, which significantly degrades the usefulness of dependency resolution. In addition when a set of dependencies can't be installed together (two different modules require incompatible versions of a third module) it can take a long time for resolution to fail, and when it does you're given absolutely no information to figure out what went wrong and why. Lastly, the API of the current resolver in Puppet is in pretty poor shape, and integrating it into r10k would be enough hurt that it makes more sense to fix up the resolver before bringing it in to r10k.

I'm not as familiar with the librarian-puppet resolver as I am the Puppet resolver, but my understanding is that it suffers from similar problems.

In addition, adding dependency resolution to r10k in general is challenging. The expected approach of resolving dependencies when r10k runs on a Puppetfile, coupled with the fact that r10k is frequently run via a webhook or some remote method means that if dependency resolution fails it's hard to even see if r10k ran into issues while resolving dependencies. In addition r10k supports tracking Git branches for Puppet modules; this approach of continuously updating a package in this manner doesn't mesh well with dependency resolution.

All of these problems are solvable problems (aside from dependency resolution being NP complete; we're pretty hosed on that front.) However, it takes time and effort to solve these problems, and the the core developers on r10k have limited resources - there's only so many hours in a day and we have to determine which tickets provide the most amount of value to the most amount of users. This is a painful process and it does mean that we do have to let some issues slip, but trust me - this is on our radar, we want to solve it, and we will get to it as soon as we can.

In the mean time, if r10k doesn't meet your needs, then I am sorry - and I hope that you can find a tool that meets your needs. We do the best we can on r10k and we're trying to make the lives of our users easier, but if there's something better out there then please - by all means use what works best for you.

So again, sorry for the delay in getting this resolved, and thanks for your patience!

from r10k.

Closing in favor of RK-3.

from r10k.

Hello,

Anyone aware of using source as a *.tar.gz from a HTTP server in metadata.json or Puppetfile ?

Something like :

"dependencies": [
{
"name": "foo/bar",
"version_requirement": ">= 2.2.1",
** "source": "http://x.x.x.x/modules/abc.tar.gz" **
}

Is this possible ?

from r10k.