Packages definition are missing uri, description and license

We need a parameter that tells luet to:

• Uninstall forcefully a package
• Uninstall only a package without considering their deps (--nodeps)

This could be helpful for handle reboot of a repository with an upgrade with --force option.

works thus far in chroot.

luet-aarch64-linux.zip

Packages currently are handled as tars, but a compression mechanism that allows to switch between types would be prefered.

Migrate Viper global instance to specific instance more easy for tests integration.

The artifact mechanism needs to consume and generate checksums

Currently luet upgrade upgrades a system to the latest available packages, not taking care of downgrades of any sort. E.g. if a package is downgraded from a repository, the client will refuse to downgrade to that version, but will prefer the local installed one.

The idea is to implement a command like luet upgrade --purge/--sync for:

• permit downgrade of packages related to a removed repository (example: foo-1.0@removed-repo and foo-0.2.0@active-repo)

• permit downgrade of packages no more available from a specific repository for an error server side

• permit to upgrade of packages with the same version but with different build date (an option could be --sync-by-date)

Now constraints are too relaxed, we have to prefer higher versions over lower ones.

Some references about the subject:

• https://www.cs.ox.ac.uk/files/4014/maxclosed_orderencoding_v16_TR.pdf (4/5.)
• http://bach.istc.kobe-u.ac.jp/papers/pdf/cpai06.pdf
• http://bach.istc.kobe-u.ac.jp/papers/pdf/ciclops10slide.pdf (34)

ResolveDeps() needs also to expand the deps into packages that are actually into the world

Note, Expand is already implemented https://github.com/mudler/luet/blob/master/pkg/package/package.go#L222 - so it just needs to be plugged in.

So also entire unpacked layers after build steps can be reused as packages

Instead of loading the whole tree, we should just build a new one with the ones referenced in the repository manifest, which contains only the built artifacts.

In addition to supply, all repository params could be cool to use only the repository name and retrieve LuetRepository data from the configuration.

Currently recipes are hardcoding a db implementation while they should just consume it instead.

We should have a command to:

• Remove packages from system forcefully (delete only the entry from DB if necessary, with --force flag)
• Add packages in system forcefully (even accepting the metadata yaml as source, with a list of e.g. files that belongs to the package)

The tree and index files are treated specially. but we should at least have the checksum of the tree tarball

Instead of expecting a tree.tar, that should be specified as part of the repository.yaml

The archive generation needs to handle symlinks coming from layers and such.

If for some reason a package changes category or name luet must support this in a transparent way to the user.

We should handle reinstalls as well by cli, the mechanism is not exposed yet to the CLI.

Equivalent to: luet replace pack --for pack --nodeps

Currently install and build needs specific package versions (but the tree doesn't, as it expands already)

With luet 0.5 luet search --installed . shows always the same entry of the packages installed:

$ sudo LUET_NOLOCK=true luet install system/luet-develop system/container-diff
 Using config file: /etc/luet/.luet.yaml
 Parsing Repository Directory /etc/luet/repos.conf.d ...
🚧   Skip dir /etc/luet/repos.conf.d : open /etc/luet/repos.conf.d: no such file or directory
 Sync of the repository main in progress...
 Downloading http://127.0.0.1:8000/repository.yaml
 Downloaded repository.yaml of 0.02 MB ( 24.10 MiB/s )
 Downloading artifact tree.tar.gz from http://127.0.0.1:8000
 Downloaded tree.tar.gz of 0.00 MB ( 2.90 MiB/s )
 Copying file  /tmp/tree018908402/tree.tar.gz to /luet/var/luet/db/packages/tree.tar.gz
 Tree tarball for the repository main downloaded correctly.
 Decompress tree of the repository main...
 🏠  Repository main revision: 25 - 2020-02-08 17:02:12 +0100 CET
 Use artifact luet-develop-system-0.5.package.tar.gz from cache.
 Use artifact container-diff-system-0.15.0.package.tar.gz from cache.


$ sudo LUET_NOLOCK=true luet search --installed .
 Using config file: /etc/luet/.luet.yaml
 Parsing Repository Directory /etc/luet/repos.conf.d ...
🚧   Skip dir /etc/luet/repos.conf.d : open /etc/luet/repos.conf.d: no such file or directory
 📦  system luet-develop 0.5
 📦  system luet-develop 0.5

Using https://github.com/spf13/afero would add features such: image uncompression to ram, etc.

Integrate date/revision on Repository specs to permit show of sync status of the local repository.

Example:

$# luet repo list
[repo1] - enable - Sync 2019/12/30

As for now, it's easy to compose a new rootfs formed from different specfiles.
The PM can already assemble the image in a portable fashion.

E.g.

 luet build ..

 luet install --system-target /foo --system-db /foo/luet.db

The proposal is to combine in a 'wrapper around' way luet box:

• luet box create "foo" --packages <pkg1> --packages <pkg2>: It creates a
  new box from the given packages. It merely installs on a pre-defined folders
  and reassembles the image
• luet box delete "foo": delete the box
• luet box install "foo" <pkg1> <pkg2>: Install packages in the box
• luet box uninstall "foo" <pkg1> <pkg2>: Uninstall packages in the box
• luet box upgrade "foo": Upgrades packages in the box
• luet box run "foo" /my/command: Execute a command in a box
• luet box shell "foo": Opens up an interactive shell in the box

Note:
To achieve this there is only a functional limitation. The implementation must
provide also a way for finalizers to perform commands in the given
environment(box).

It would be helpful to avoid the forced use of submodules

We should add more examples and explanation of internals.

Also we need to add more docs about the usage from an user perspective

• #25
• #178
• #182

We do perform operations that can be skipped in local iterations (e.g. export of image if no changes are detected, etc)

Like for equo sec oscheck could be cool have a features that check the checksum of the files of a specific package

Build needs also optionally to compute the revdeps and compile them in parallel. We need to have a flag to build which enables this.

Compile should have a wrapper which calculate the revdeps of the packages that will be built, and compile them afterwards.

As per comment in:

We need to move out the resolvedeps step up when calling it in parallel, otherwise each thread will do the same job.

It would be handy to have some vars pre-populated (such as the package version) and some that the user can define in the build specs

It is not a real issue as the file is copied back and permissions restored, but we should just copy files and ensure folders have the same permissions

We have to expose as a flag to the search subcommand the -revdeps option. In this way we can let the user build a list of packages that depends on a specific one.
The feature is already present, each package returns its tree by calling

This issue is to expose the feature to the cli and allow for e.g. to : luet search --revdeps <A1>

It could be helpful to have a command for print dependencies tree for a specific package.

Note for future: We should change all the occurrences here

Originally posted by @mudler in https://github.com/_render_node/MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDM0NzA5OTEwOQ==/comments/review_comment

For trace installed/available packages we need a system from client side. The idea is to use a NoSQL/K-V database.
Here, the possibilities are:

• BoltDB (Few maintained)

• DGraph

• Badger

We need a parameter that tells luet to:

• Install forcefully a package
• Install only a package without considering their deps (--nodeps)
• Install only the package deps (--onlydeps)

Luet should track also the content of the files which are installed, storing a checksum internally. In this way we can build a command oscheck that:

• Can compare the checksum in the db with the system command
• Build a list of packages that can be reinstalled to recover the system

Integrate support of Viper/Cobra configuration for both client and build packages commands.

Format supported is YAML (or viper supported format):

Example of a configuration for the client:

logging:
   path: /var/log/luet/luet.log
   level: "info"

general:
   concurrency: 4
   # debug: false

# Environment options are usable
proxy:
    http: xxx
    https: xxxx
    ftp: xxx

cache_repositories:
    -  name: "Local cache node2"
       priority: 3
       mode: "ro"
       urls:
          -  https://mynode2.local/
       type: "http"
       # auth:

    -  name: "Local cache node1"
       priority: 2
       mode: "rw"
       urls:
          -  https://mynode1.local/
       type: "http"
       # auth:
   -  name: "Local dir"
      type: "dir"
      mode: rw
      priority: 1
      path: "/var/cache/luet/"

# Repositories where fetch tree and packages could be defined in the main
# configuration file or under a specific directory.
system_repositories:
    - name: "repo1"
      descr: "Repository 1"
      type: "disk"
      # Optional URL where retrieve API Repository server (that uses same auth credentials)
      api: "https://myrepo.server.org/repo1"
      path: "/mypath/repo1"
      priority: 1
   -  name: "repo2"
      descr: "Repository 2"
      type: "http"
      urls:
         -  "https://myrepo.org/repo2"
         -  "https://mirror.myrepo.org/repo2"
      priority: 2
      auth:
          # Token key (for example Mottainai Api key)
          token: "xxxxxxx"
          # Basic authentication
          #  username: "user"
          # password: "pwd"

List of repositories could be defined under the directory /etc/luet/repositories.d/ in yaml:
Example of repo1:

# cat /etc/luet/repositories.d/repo1.yaml

name: "repo1"
descr: "Repository 1"
type: "disk"
path: "/mypath/repo1"
# type: "http"
# urls:
#   - "https://myrepo/repo1"
priority: 1
enable: true
auth:
   # Token key (for example Mottainai Api key)
   token: "xxxxxxx"
   # Basic authentication
   #  username: "user"
   # password: "pwd"

Instead we should add all of them to the conflict list

e.g.

• https://github.com/olivere/jobqueue
• https://github.com/gocelery/gocelery

It would be handy to allow packages definition to source package content directly from the specified image. E.g. a simple yaml field in build.yaml indicating that unpacking the image and tarring it is enough.

Hereinafter, the list of features to implement:

• Local repositories

• HTTP/HTTPs repositories

• Integrate support of aggregate download per repository

• Integrate support of custom Authentication support (Basic, Token, etc.)

• Fetching packages from multiple sources (FTP, HTTP, Object Store, Directory)

• Support incremental download for package revisions

• Support Proxy Configuration

• Support --fetchonly option for prepare cache directory to use in offline env

• Mirror list for repository must be a separated file more easy to update through a package

• Integrity check / packages verification

We should have a way to package tree definitions, or alternatively track them from remote urls (git style).
In this way we can cache trees locally, sync them, and make the CLI experience a bit smoother by dropping the tree as a needed option (for create-repo, and build)

We should have a test suite (integration?) to test building packages from tree converted from Portage.

It is a really good use case to test building load, and performance regression for the SAT solver

ResolveDeps in tree.go is currently our bottleneck, it should walk partially tree (only of the needed packages) or (best approach imho) make the solver just consume the database instance to walk packages (by replacing completely World)

We need to optimize build dependencies which are shared between targets, maybe by also building them before and then start the concurrent builds. we cannot miss --nodeps and --depsonly on the build command while developing this feature.

• build --nodeps A B C
• build --onlydeps A B C
• build --shared A B C (builds the shared deps first, then builds the other concurrently, which probably should be default behavior that can be turned off)
• We should build first those that have no revdep