Installation of packages from outside sources about dotfiles-framework HOT 8 CLOSED

@gamesh411 Thanks for typing out the suggestion. I am not trying to reimplement apt or nixos, but I agree the lack of the ability to "splice" the repository with content from another user is a real lacking thing.

What do you think about the following:

1. We introduce a command like dotfiles --edit-sources which opens ~/.config/Dotfiles/sources.yaml in your $EDITOR (similarly to visudo 😉).
2. This file will contain the list of sources in the following format:

sources:
  - type: git repo
    repository: http://github.com/user/dotfiles-packages.git
    directory: additional-packages                   # Optional, defaults to "packages"

  - type: git repo
    name: official
    commit: 1234567abc                               # Commit to peg, optional.
    repository: http://github.com/whisperity/Dotfiles.git

  - type: local
    name: foobar                                     # Optional.
    directory: /home/user/MyDotfiles/foo/packages    # Required for local directories.

  # If you want to use the "packages/" directory from where dotfiles.py is running from.
  - type: default

• git repos specify the repository to clone and optionally the commit to check out, if you want the version to be pegged.
• git repos and locals can be optionally assigned a name. If a git repo is not named, it'll default to the user/repo name. If a local is not named, it'll not have a name.
• The default cannot be named, but automatically takes the "default" name.

3. The list itself is a precedence list. The list, by default, is the single element of type: default, which maps to the current behaviour.
4. When an --install or --list action is performed, Dotfiles will first perform the "update" operation. Update will, essentially, set up a system of directories (in an implementation-defined manner) under ~/.cache/Dotfiles (.cache, because they can be removed without a problem.)
   • git repos will be cloned into this location. If the clone already exists and is not pegged to a commit, git pull is performed. What happens afterwards is not our concern, the user should ensure the pull does not require authentication and whatnot...
   • local directories and the default (if configured) will be symlinked to their targets

Okay, now we have a data structure somewhere we can hopefully read into. Let's see...

When a package name pkg is to be resolved, the resolution will first try to see if pkg is already installed, and if not, go with the remotes in the source file in the order they are specified.

The --list output is extended with a new column, which shows the source's "name" (if there is a name given to it). By default, it will list the "topmost" (in the resolution priority) version of the package. For --list, the locally installed version will keep taking priority over any "remote", just as how it currently behaves.
--list will also take an optional argument, the "name" of the source, in which case only the packages (but in that case, all of the packages on that remote, irrespective of their "installedness" status), are listed.

Only one version of the same package can be installed at a time, from whichever source. Locally, the package name, irrespective of the source, identifies the package (e.g. when uninstalling).

However, to ensure that the user can still install a version of a package from another source, by specifying the remote name, i.e. even if, let's say, vim would resolve to some other repository, saying whisperity::vim or default::vim will use the version from that remote.

.* and .__ALL__ will keep matching all sub-packages of a package in every remote unless the remote-tag is prepended, in which case only the sub-packages of that particular remote are considered.

However, in the case of dependencies, the remote-tag cannot be given. So if a package depends on base and you have your version of base in an earlier remote, your base will be dependency-installed, and then everything rolls forward as usual. I do not wish to complicate matters even worse, it should be the downstream's responsibility to ensure compatibility with other packages, and for the users to list their sources in a way it keeps working. Especially because the "name" of the remote is a user-specific thing, you couldn't really script it.
(Otherwise, we'd have a whole mess of potentially Turing-complete stuff on our hands, together with weird crap like supporting a mechanism similar to #include_next, etc...)

from dotfiles-framework.

@whisperity I really like this proposal, good job at laying out a reasonable implementation plan!

One thing I keep getting back to is the optionality of the remote name. On one hand, it would be nice to ensure that all remotes are identified by a unique name. On the other hand, as you mentioned we definitely wouldn't want to go down that "resolve remote dependency chains in a context-sensitive manner" rabbit hole (the context being all the other remote specifications). Not necessarily having a name for the remote discourages that general path, which is nice in its own way.

In my general use-case, I would like to provide a base that is my own (my shell .rc files and aliases). If I specify a remote before the default one, which has a package named base, I would want it to be used everywhere where base is defined as a dependency. Is this the correct assumption according to this implementation plan?

from dotfiles-framework.

@gamesh411 Yes, that's the plan. A dependency saying base to be installed will just "recursively" start looking up base. The first "source packages" where base is found (if it is not installed yet) will install base from there.

from dotfiles-framework.

I've did a few changes to the things said above.

1. The "logical name" of the remote repository will necessarily have to be specified. It'll be the first thing asked by the wizard.
2. There will be no way of specifying packages with a namespace prefix, such as root::Package. The problem is, keeping these names in check while also handling the resolution to an "unqualified" name is just too much of a hassle.

When Dotfiles loads the known packages, it will load in the order of remotes configured. Package "namespaces" (empty directories that only contain subdirectories) will be merged together, but trees rooted at an actual package and their subpackages won't. This means that if repository A contains tools.fancy, vim, vim.foo and repository B contains tools.other, vim, vim.bar, if loaded in an "AB" order, the available package names will be: tools.fancy, tools.other, vim (from A) and vim.foo.

Alternatively, you can specify --source to the install command which will restrict to only installing from that source, i.e. the "current" behaviour.

These things are done in the first step when package names are fetched from the filesystem, and once this is done, only one copy of each package will be in memory.

from dotfiles-framework.

test$ ./dotfiles.py --source Local

| Source        | Package        | Description        |
| Local        | dummy        | A Dummy package        |
| Local        | test        |         |

This test depends on basic, however because the source list is restricted, it will not find the dependency and won't install.

test$ ./dotfiles.py --source Local test
KeyError: "Dependency 'basic' for 'test' was not found as a package."

from dotfiles-framework.

I think I've implemented this, @gamesh411, take a look at branch multiple-sources.

from dotfiles-framework.

It's marvellous! I'm so glad that I can extend this framework in a modular fashion. Thanks!

from dotfiles-framework.

@gamesh411 So it works? Thanks! But do be careful, I'll be doing a bit of changes to the schema with regards to the substitute environment variables action, so I'll only merge this afterwards.

from dotfiles-framework.