rancherfederal / hauler Goto Github PK

View Code? Open in Web Editor NEW

108.0 9.0 23.0 31.06 MB

Airgap Swiss Army Knife

Home Page: https://hauler.dev

License: Apache License 2.0

Shell 3.39% Go 94.39% Makefile 0.58% Dockerfile 0.56% Smarty 1.08%

oci containers airgap kubernetes

hauler's Introduction

Rancher Government Hauler

Airgap Swiss Army Knife

⚠️ Please Note: Hauler and the Hauler Documentation are recently Generally Available (GA).

Rancher Government Hauler simplifies the airgap experience without requiring operators to adopt a specific workflow. Hauler simplifies the airgapping process, by representing assets (images, charts, files, etc...) as content and collections to allow operators to easily fetch, store, package, and distribute these assets with declarative manifests or through the command line.

Hauler does this by storing contents and collections as OCI Artifacts and allows operators to serve contents and collections with an embedded registry and fileserver. Additionally, Hauler has the ability to store and inspect various non-image OCI Artifacts.

For more information, please review the Hauler Documentation!

Installation

Linux/Darwin

# installs latest release
curl -sfL https://get.hauler.dev | bash

Homebrew

# installs latest release
brew tap rancherfederal/homebrew-tap
brew install hauler

Windows

# coming soon

Acknowledgements

Hauler wouldn't be possible without the open-source community, but there are a few projects that stand out:

hauler's People

Contributors

Stargazers

Watchers

hauler's Issues

Bundle fails to create due to metadata.annotations being too long

Environmental Info:

Hauler Version:
v0.1.0-alpha3

System CPU architecture, OS, and Version:
CentOS 7
Linux airgap 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Describe the bug:
Large bundles pushed into the manifests folder cause the k3s auto-apply functionality to fail due to too long of annotations.

Steps To Reproduce:

mkdir cert-manager
cd cert-manager

curl -LO https://github.com/jetstack/cert-manager/releases/download/v1.4.0/cert-manager.crds.yaml
cat <<EOF > fleet.yaml
helm:
  repo: https://charts.jetstack.io
  chart: cert-manager
  version: v1.4.0
  releaseName: cert-manager
EOF

cd ..
cat <<EOF > hauler-package-ex.yaml
apiVersion: hauler.cattle.io/v1alpha1
kind: Package
metadata:
  name: hauler-package-ex
spec:
  driver:
    type: k3s
    version: v1.20.8+k3s1
  fleet:
    version: v0.3.5
  paths:
    - cert-manager
  images:
    - quay.io/jetstack/cert-manager-acmesolver:v1.4.0
EOF

hauler package build -c hauler-package-ex.yaml --name hauler-package-ex
hauler package run hauler-pkg.tar.zst

Expected behavior:
cert-manager bundle exists

Actual behavior:
cert-manager bundle does not exist due to error

Additional context / logs:
k3s journalctl:

Failed to process config: failed to process /var/lib/rancher/k3s/server/manifests/hauler/cert-manager.json: failed to create fleet-local/cert-manager fleet.cattle.io/v1alpha1, Kind=Bundle for  kube-system/cert-manager: Bundle.fleet.cattle.io \"cert-manager\" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

Allow directories to be passed to `Files`

Currently scaffolded but not implemented.

Ensure that when specifying a directory path (instead of a file) in content/Files that the directory is compressed and archived (*.tar.zst) before being pushed up.

Collect images from image list file

Collect files from directory, including subdirectories

`hauler store list` fails to find digested manifests with no corresponding tag

digested manifest discovery is not part of the distribution-spec, so we can only currently discover content that is tagged.

This means content stored via their digest is undiscoverable, and skipped in hauler store list and in hauler store copy.

Ultimately we need a better way of discovering all content within the store, but immediate need is to fix the early error warning for searching for non-existent tags for content that exists solely referenced by their digest.

User created `stores` should be declarative and ephemeral

Since the contents true data should be stored in a users cache directory (#61), stores should just be shallow links to the user wide cached blob storage.

Ideally when users create a new store, it is created within $PWD/store and manipulated from there. This should be overridable, and should allow users to create as many stores as they wish.

Add ability to print version of binary

Allow printing version of compiled executable to be printed, either via a flag or subcommand.

Add capability for user defined collections

Introduce the concepts of Collections to allow for user defined sets of known hauler content.

collections should allow for users to define their own api, and their own driver logic. drivers are user defined images that contain the business logic of how to fetch content. hauler will be responsible for running these images, and providing the inputs (sources) and processing the outputs (sinks).

Hauler should provide the source to the driver and expect a sink based on the following criteria:

driver contract:

source:

the users collection as a string to stdin

sink:

optional: an oci layout in a location specified by hauler
optional: an output.yaml containing known hauler content

Develop CI workflow for automated testing, builds, release creation.

Develop a workflow per the milestone description to complete automated build, tests, and releases.
The workflows has to also use Hauler to create the initial airgapped archives.

Tools:

Github Actions
GoReleaser
Renovate (maybe? supports go?)

Errors are not wrapped well

Provide context to returned errors to allow easier debugging when something goes wrong.

better docs explaining hauler's purpose and usecases

The current docs don't really do much outside of teach you some basic commands, and the README is lackluster.

Items below are some areas to address

more descriptive top level README
asciinema in top level README walking through basic hauler happy paths
detailed walkthrough for e2e usecase (fetch contents, install k3s, install stuff on k3s?)

Add oci image runner

In order to successfully complete #46, hauler needs to be able to run images.

TODO: Evaluate how we want to do this (abstract out to an existing container runtime, or embed one in hauler, or something else?)

Add `add` command to store to add specific content types imperatively

Until hauler store sync is fully functional, include a hauler store add <content> to allow users to add in specific content imperatively through the cli:

# file
hauler store add file https://get.k3s.io

# image
hauler store add image alpine:latest

# chart
hauler store add longhorn --repo "https://charts.longhorn.io"

bug: actual `chart` contents are not pulled with `thickchart`

Add better docs to README focusing on experience on connected vs disconnected systems

Please document the user experience better on the README. I would ideally like to see the example walk through the scenario of a) grabbing all the dependencies and b) then deploying those dependencies as a cluster. I don't think we do a great job of showing the value of hauler in the current docs form.

Develop CLI scaffolding for haulerctl and subcommands

Use Cobra and Viper to create a command/sub command standard for haulerctl

Populate registry with all images

Finalize beta cli ux

Lock in what functionality we should expose through the cli for beta release.

Deploy git server

Deploy docker registry

Ensure `Images` content functionality

update oras dependency to `1.0.0`

Populate file server with directory contents

Allow hauler's embedded registry to be configured when serving

Allow for a user defined config file to be passed to hauler when serving the registry with: hauler registry serve -c path/to/file.yaml

Develop copy package using oras

Use short lived oci layout for intermediate content store

intermediate oci content (before being pushed) should be stored temporarily on disk in oci layout format, this is to prepare for future where pluggable collections exist

Define `Driver` API

Define what the Driver api should look like, initial ideas below:

apiVersion: driver.hauler.cattle.io/v1alpha1
kind: Driver
metadata:
  name: bleh
spec:
  version: 0.1.0
  image:
    collect: my-collector-image:v1
    extract: my-extractor-image:v1

support multi platform images

current behavior defaults to the images default platform (defined in the image manifest).

Desired enhancements include:

allow for user defined Platform
add configuration option to use all images

Config file images not packaged

Environmental Info:

Hauler Version:
v0.1.0-alpha3

System CPU architecture, OS, and Version:
CentOS 7
Linux airgap 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Describe the bug:
Specifying images in a config file does not download the extra images into the generated package.

Steps To Reproduce:

cat <<EOF > hauler-package-images.yaml
apiVersion: hauler.cattle.io/v1alpha1
kind: Package
metadata:
  name: hauler-package-ex
spec:
  driver:
    type: k3s
    version: v1.20.8+k3s1
  fleet:
    version: v0.3.5
  images:
    - docker.io/library/busybox
EOF

hauler package build -c hauler-package-images.yaml

Expected behavior:
The container image is included in the package.

Actual behavior:
The container image is not included in the package.

Additional context / logs:

 INFO  Building package
 INFO  Config file 'hauler-package-images.yaml' specified, attempting to load existing package config
 INFO  Packaging 0 bundle(s)
 SUCCESS  Finished packaging 0 bundle(s) along with 0 autodetected image(s)
 INFO  Packaging 0 user defined images
 SUCCESS  Finished packaging 0 user defined images

Ensure content store's can be entirely declarative

Hauler can currently sync various types of declarative Content, but there isn't a guarantee that a store is entirely defined by a set of Contents. In other words, the store.Add() that sync uses is imperative, and is missing something like store.Remove().

Concurrent maps panic on hauler package build

Hauler Version:
v0.1.0-alpha3

System CPU architecture, OS, and Version:
Ubuntu 18.04

Describe the bug:

Steps To Reproduce:

101  cat <<EOF > hauler-package-ex.yaml
apiVersion: hauler.cattle.io/v1alpha1
kind: Package
metadata:
  name: hauler-package-ex
spec:
  driver:
    type: k3s
    version: v1.20.8+k3s1
  fleet:
    version: v0.3.5
EOF

  102  mkdir cert-manager
  103  cd cert-manager
  104  curl -LO https://github.com/jetstack/cert-manager/releases/download/v1.4.0/cert-manager.crds.yaml
  105  cat <<EOF > fleet.yaml
helm:
  repo: https://charts.jetstack.io
  chart: cert-manager
  version: v1.4.0
  releaseName: cert-manager
EOF

  106  hauler package build -p ./

Expected behavior:
build packages including bundles based on helm charts

Actual behavior:
see error below--fails to build bundle

Additional context / logs:

INFO  Building package                                                         
 INFO  No config file specified, strictly using cli arguments                                                                                                    
 INFO  Packaging 1 bundle(s)                                                    
 INFO  Creating bundle from path: ./                                            
fatal error: concurrent map writes                                              
fatal error: concurrent map writes                                                                                                                               
                                                                                
goroutine 12 [running]:                                                                                                                                          
runtime.throw(0x287bc3b, 0x15)                                                  
        /opt/hostedtoolcache/go/1.16.5/x64/src/runtime/panic.go:1117 +0x72 fp=0xc0009c8960 sp=0xc0009c8930 pc=0x4370d2                                           
runtime.mapassign_faststr(0x24efce0, 0xc000621590, 0x28571bf, 0x4, 0x0)         
        /opt/hostedtoolcache/go/1.16.5/x64/src/runtime/map_faststr.go:291 +0x3d8 fp=0xc0009c89c8 sp=0xc0009c8960 pc=0x4146b8
github.com/rancher/fleet/pkg/bundle.readContent(0x2c94f60, 0xc000a10fc0, 0xc0007ed758, 0xc000aff560, 0x20, 0xc000011780, 0x39, 0x0, 0x0, 0x0, ...)
        /home/runner/go/pkg/mod/github.com/rancher/[email protected]/pkg/bundle/resources.go:347 +0x351 fp=0xc0009c8c08 sp=0xc0009c89c8 pc=0x206d9b1                  
github.com/rancher/fleet/pkg/bundle.readDirectory(0x2c94f60, 0xc000a10fc0, 0xc0007ed758, 0x1, 0xc000418d70, 0x47, 0xc000817a80, 0x2, 0xc000011780, 0x39, ...)
        /home/runner/go/pkg/mod/github.com/rancher/[email protected]/pkg/bundle/resources.go:262 +0xb8 fp=0xc0009c8df0 sp=0xc0009c8c08 pc=0x206d078
github.com/rancher/fleet/pkg/bundle.readDirectories.func1(0x0, 0x0)            
        /home/runner/go/pkg/mod/github.com/rancher/[email protected]/pkg/bundle/resources.go:239 +0x158 fp=0xc0009c8f78 sp=0xc0009c8df0 pc=0x206f518
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000a94ab0, 0xc000a110c0)        
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x59 fp=0xc0009c8fd0 sp=0xc0009c8f78 pc=0x5efb99
runtime.goexit()                                                                
        /opt/hostedtoolcache/go/1.16.5/x64/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0009c8fd8 sp=0xc0009c8fd0 pc=0x46d561
created by golang.org/x/sync/errgroup.(*Group).Go                                                                                                                
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x66                                               
                                        
goroutine 1 [semacquire]:                                                                                                                                        
sync.runtime_Semacquire(0xc000a94ac0)                                           
        /opt/hostedtoolcache/go/1.16.5/x64/src/runtime/sema.go:56 +0x45                                                                                          
sync.(*WaitGroup).Wait(0xc000a94ab8)                                            
        /opt/hostedtoolcache/go/1.16.5/x64/src/sync/waitgroup.go:130 +0x65                                                                                       
golang.org/x/sync/errgroup.(*Group).Wait(0xc000a94ab0, 0xc000a110c0, 0xc000a10fc0)
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:40 +0x31                                               
github.com/rancher/fleet/pkg/bundle.readDirectories(0x2c94f60, 0xc000a11100, 0x1, 0xc000573c20, 0x2, 0x2, 0x0, 0x0, 0x0)                                         
        /home/runner/go/pkg/mod/github.com/rancher/[email protected]/pkg/bundle/resources.go:256 +0x3b9                                                               
github.com/rancher/fleet/pkg/bundle.readResources(0x2c94f60, 0xc000a11100, 0xc000a92000, 0x281ef01, 0xc000817a80, 0x2, 0x0, 0x0, 0x0, 0x0, ...)                  
        /home/runner/go/pkg/mod/github.com/rancher/[email protected]/pkg/bundle/resources.go:65 +0x248                                                                
github.com/rancher/fleet/pkg/bundle.read(0x2c94f60, 0xc000a11100, 0xc000817a80, 0x1, 0xc000817a80, 0x2, 0x2c3fb40, 0xc000a39ec0, 0xc000ae1200, 0xffffffff, ...)
        /home/runner/go/pkg/mod/github.com/rancher/[email protected]/pkg/bundle/read.go:131 +0x39c                                                 
github.com/rancher/fleet/pkg/bundle.Read(0x2c94f60, 0xc000a11100, 0xc000817a80, 0x1, 0xc000817a80, 0x2, 0x2c45320, 0xc000a7c3e0, 0xc000ae1200, 0x40d8248, ...)   
        /home/runner/go/pkg/mod/github.com/rancher/[email protected]/pkg/bundle/read.go:74 +0x150                                                   
github.com/rancher/fleet/pkg/bundle.Open(0x2c94f60, 0xc000a11100, 0xc000817a80, 0x1, 0xc000817a80, 0x2, 0x0, 0x0, 0xc0009a7200, 0x0, ...)
        /home/runner/go/pkg/mod/github.com/rancher/[email protected]/pkg/bundle/read.go:61 +0x266                                                                  
github.com/rancherfederal/hauler/pkg/packager.pkg.PackageBundles(0xc000a8e1a0, 0xc000564ca8, 0x15, 0x2ca1ef8, 0xc0001d5f90, 0x2c94f60, 0xc000a11100, 0xc0001d5f70
, 0x1, 0x1, ...)                                                                                                                                                 
        /home/runner/work/hauler/hauler/pkg/packager/packager.go:71 +0x2b9
github.com/rancherfederal/hauler/cmd/hauler/app.(*pkgBuildOpts).Run(0xc000a5ad20, 0x0, 0x0)                                                                      
        /home/runner/work/hauler/hauler/cmd/hauler/app/pkg_build.go:155 +0x47b                                                                                   
github.com/rancherfederal/hauler/cmd/hauler/app.NewPkgBuildCommand.func2(0xc000a88780, 0xc000a8e0e0, 0x0, 0x2, 0x0, 0x0)                                         
        /home/runner/work/hauler/hauler/cmd/hauler/app/pkg_build.go:62 +0x2a                                                                                     
github.com/spf13/cobra.(*Command).execute(0xc000a88780, 0xc000a8e0c0, 0x2, 0x2, 0xc000a88780, 0xc000a8e0c0)                                                              /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x472                                                                              
github.com/spf13/cobra.(*Command).ExecuteC(0xc0008bf680, 0x41ad01, 0x0, 0x0)    
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:960 +0x375                                                                              
github.com/spf13/cobra.(*Command).Execute(...)                                  
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:897    
main.main()                                                                     
        /home/runner/work/hauler/hauler/cmd/hauler/main.go:12 +0x2a                                                                                              
                                                                                
goroutine 6 [chan receive]:                                                                                                                                      
k8s.io/klog/v2.(*loggingT).flushDaemon(0x40fd920)                               
        /home/runner/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:1169 +0x8b                                                                                         
created by k8s.io/klog/v2.init.0                                                
        /home/runner/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:417 +0xdf                                                                                          
                                                                                                                                                                 
goroutine 8 [select]:                                                                                                                                            
go.opencensus.io/stats/view.(*worker).start(0xc0001181e0)                                                                                                        
        /home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:154 +0xcd                                                       
created by go.opencensus.io/stats/view.init.0                                  
        /home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:32 +0x57                                                         
                                                                                
goroutine 9 [select]:                                                                                                                                            
github.com/patrickmn/go-cache.(*janitor).Run(0xc000651990, 0xc0002a77c0)        
        /home/runner/go/pkg/mod/github.com/patrickmn/[email protected]+incompatible/cache.go:1079 +0xa9                   
created by github.com/patrickmn/go-cache.runJanitor                                                                                                              
        /home/runner/go/pkg/mod/github.com/patrickmn/[email protected]+incompatible/cache.go:1099 +0xa9                                                            
                                        
goroutine 19 [sleep]:                                                                                                                                            
time.Sleep(0x6fc23ac00)                                                         
        /opt/hostedtoolcache/go/1.16.5/x64/src/runtime/time.go:193 +0xd2                                                                                         
sigs.k8s.io/controller-runtime/pkg/log.init.0.func1()                           
        /home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/log/log.go:63 +0x45                                                                    
created by sigs.k8s.io/controller-runtime/pkg/log.init.0                                                                                                         
        /home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/log/log.go:62 +0x35                                                                    
                                                                                                                                                                 
goroutine 38 [IO wait]:                                                                                                                                          
internal/poll.runtime_pollWait(0x7f5a158bb670, 0x72, 0xffffffffffffffff)                                                                                         
        /opt/hostedtoolcache/go/1.16.5/x64/src/runtime/netpoll.go:222 +0x55                                                                                      
internal/poll.(*pollDesc).wait(0xc000945798, 0x72, 0x2500, 0x25c0, 0xffffffffffffffff)                                                                         
        /opt/hostedtoolcache/go/1.16.5/x64/src/internal/poll/fd_poll_runtime.go:87 +0x45                                                      
internal/poll.(*pollDesc).waitRead(...)                                                                                                                          
        /opt/hostedtoolcache/go/1.16.5/x64/src/internal/poll/fd_poll_runtime.go:92                                                             
internal/poll.(*FD).Read(0xc000945780, 0xc000e80000, 0x25c0, 0x25c0, 0x0, 0x0, 0x0)                                                      
        /opt/hostedtoolcache/go/1.16.5/x64/src/internal/poll/fd_unix.go:166 +0x1d5                                                                            
net.(*netFD).Read(0xc000945780, 0xc000e80000, 0x25c0, 0x25c0, 0x1c1a, 0xc000e809a1, 0x5)                                                                         
        /opt/hostedtoolcache/go/1.16.5/x64/src/net/fd_posix.go:55 +0x4f                                                                                          
net.(*conn).Read(0xc00000e0c0, 0xc000e80000, 0x25c0, 0x25c0, 0x0, 0x0, 0x0)
        /opt/hostedtoolcache/go/1.16.5/x64/src/net/net.go:183 +0x91                                                                                              
crypto/tls.(*atLeastReader).Read(0xc0005fc060, 0xc000e80000, 0x25c0, 0x25c0, 0x1c1a, 0x40fe800, 0xc000465680)                                                    
        /opt/hostedtoolcache/go/1.16.5/x64/src/crypto/tls/conn.go:776 +0x63                                                                                      
bytes.(*Buffer).ReadFrom(0xc0005c4278, 0x2c3fdc0, 0xc0005fc060, 0x40b785, 0x253fc60, 0x27b2b60)                                                                  
        /opt/hostedtoolcache/go/1.16.5/x64/src/bytes/buffer.go:204 +0xbe                                                                                         crypto/tls.(*Conn).readFromUntil(0xc0005c4000, 0x2c44fa0, 0xc00000e0c0, 0x5, 0xc00000e0c0, 0x9)                                                                  
        /opt/hostedtoolcache/go/1.16.5/x64/src/crypto/tls/conn.go:798 +0xf3     
crypto/tls.(*Conn).readRecordOrCCS(0xc0005c4000, 0x0, 0x0, 0xc00001dbb8)                                                                                         
        /opt/hostedtoolcache/go/1.16.5/x64/src/crypto/tls/conn.go:605 +0x115    
crypto/tls.(*Conn).readRecord(...)                                              
        /opt/hostedtoolcache/go/1.16.5/x64/src/crypto/tls/conn.go:573           
crypto/tls.(*Conn).Read(0xc0005c4000, 0xc000d7a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)                                                                               
        /opt/hostedtoolcache/go/1.16.5/x64/src/crypto/tls/conn.go:1276 +0x165   
bufio.(*Reader).Read(0xc000375320, 0xc000d3e9d8, 0x9, 0x9, 0x8117eb, 0xc00001dc78, 0x407005)                                                                     
        /opt/hostedtoolcache/go/1.16.5/x64/src/bufio/bufio.go:227 +0x222        
io.ReadAtLeast(0x2c3fae0, 0xc000375320, 0xc000d3e9d8, 0x9, 0x9, 0x9, 0xc000e4c010, 0x33d6e7c8fd600, 0x2)                                                         
        /opt/hostedtoolcache/go/1.16.5/x64/src/io/io.go:328 +0x87               
io.ReadFull(...)                                                                                                                                                 
        /opt/hostedtoolcache/go/1.16.5/x64/src/io/io.go:347                                                                                                      
net/http.http2readFrameHeader(0xc000d3e9d8, 0x9, 0x9, 0x2c3fae0, 0xc000375320, 0x0, 0x0, 0xc000e50000, 0x0)                                                      
        /opt/hostedtoolcache/go/1.16.5/x64/src/net/http/h2_bundle.go:1477 +0x89                                                                                  
net/http.(*http2Framer).ReadFrame(0xc000d3e9a0, 0xc000e50000, 0x0, 0x0, 0x0)                                                                                     
        /opt/hostedtoolcache/go/1.16.5/x64/src/net/http/h2_bundle.go:1735 +0xa5
net/http.(*http2clientConnReadLoop).run(0xc00001dfa8, 0x0, 0x0)                                                                                                  
        /opt/hostedtoolcache/go/1.16.5/x64/src/net/http/h2_bundle.go:8322 +0xd8 
net/http.(*http2ClientConn).readLoop(0xc0005ce180)                                                                                                               
        /opt/hostedtoolcache/go/1.16.5/x64/src/net/http/h2_bundle.go:8244 +0x6f 
created by net/http.(*http2Transport).newClientConn                                                                                                              
        /opt/hostedtoolcache/go/1.16.5/x64/src/net/http/h2_bundle.go:7208 +0x6c5                                                                                 
                                                                                                                                                                 
goroutine 11 [running]:                 
        goroutine running on other thread; stack unavailable                                                                                                     
created by golang.org/x/sync/errgroup.(*Group).Go                               
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x66                                               
                                                                                
goroutine 11 [running]:                                                                                                                                          
runtime.throw(0x287bc3b, 0x15)                                                                                                                                   
        /opt/hostedtoolcache/go/1.16.5/x64/src/runtime/panic.go:1117 +0x72 fp=0xc0009c4960 sp=0xc0009c4930 pc=0x4370d2                                           
runtime.mapassign_faststr(0x24efce0, 0xc000621590, 0x28571bf, 0x4, 0x0)                                                                                          
        /opt/hostedtoolcache/go/1.16.5/x64/src/runtime/map_faststr.go:291 +0x3d8 fp=0xc0009c49c8 sp=0xc0009c4960 pc=0x4146b8                                     
github.com/rancher/fleet/pkg/bundle.readContent(0x2c94f60, 0xc000a10fc0, 0xc0007ed758, 0xc000ca4420, 0x20, 0x285523c, 0x1, 0x0, 0x0, 0x0, ...)                   
        /home/runner/go/pkg/mod/github.com/rancher/[email protected]/pkg/bundle/resources.go:347 +0x351 fp=0xc0009c4c08 sp=0xc0009c49c8 pc=0x206d9b1                  
github.com/rancher/fleet/pkg/bundle.readDirectory(0x2c94f60, 0xc000a10fc0, 0xc0007ed758, 0x439d01, 0x285523c, 0x1, 0xc000817a80, 0x2, 0x285523c, 0x1, ...)     
        /home/runner/go/pkg/mod/github.com/rancher/[email protected]/pkg/bundle/resources.go:262 +0xb8 fp=0xc0009c4df0 sp=0xc0009c4c08 pc=0x206d078
github.com/rancher/fleet/pkg/bundle.readDirectories.func1(0x0, 0x0)                                                                                              
        /home/runner/go/pkg/mod/github.com/rancher/[email protected]/pkg/bundle/resources.go:239 +0x158 fp=0xc0009c4f78 sp=0xc0009c4df0 pc=0x206f518
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000a94ab0, 0xc000a11040)                                                                                         
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x59 fp=0xc0009c4fd0 sp=0xc0009c4f78 pc=0x5efb99
runtime.goexit()                                                                                                                                                 
        /opt/hostedtoolcache/go/1.16.5/x64/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0009c4fd8 sp=0xc0009c4fd0 pc=0x46d561                                         
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x6

transport oci layouts instead of registry backends

transporting content using the non-standard distribution backend provided a lot of convenience for serving, but it wastes the potential of the OCI Layout spec for all sorts of other things.

Evaluate transporting oci layouts instead, and then transferring into registry content at serve time or something similar

Populate git server with repos

Ensure `Platform` is configurable for `content`

hauler currently defaults to the contents default platform defined in the manifest index. This works sometimes, but making it user configurable would be ideal.

finalize mvp (`v1alpha1`) content api/features

Finalize embedded content api kinds: content.hauler.cattle.io/v1alpha1:

Serve git repositories

#10 Collect repos
#11 Deploy git server
#12 Populate git server with repos

Collect images from image archives

Ensure logs are present, useful, and coherent

Logs are all over the place as it stands today, ensure only callers are logging and that messages are precise and useful

Allow serving registry as a background process

a non blocking hauler store serve would be useful for many reasons, solve this with 2 options:

copy/pastable systemd unit file example
daemon flag to cli: hauler store serve -d

Serve files

#14 Collect files from directory, including sub directories
#15 Develop copy package with oras
#16 Populate file server with directory contents

Add command to list contents in hauler store

Need a convenient way for a user to quickly identify what's in a store

hauler store list

Add content blob caching

Blobs used by hauler should be cached by their digest and re-used whenever possible.

Initial implementation should be to use the users filesystem for caching, with a centralized location in $XDG_CACHE_DIR as the blob store. It should also rely on the contents digest. This can be improved in the future by some hybrid approach where the contents digest is also cached (since this initial method will still require the digest to be computed before the cache can be retrieved).

Ensure `Files` content functionality

Ensure Files content:

is adequately tested
functions as documented

System CPU architecture, OS, and Version:

Describe the bug:

hauler pkg run <> fails when there are 0 bundles (ie on a minimal k3s pkg install)

sudo dist/hauler_linux_amd64/hauler pkg run pkg.tar.zst
 INFO  Running from 'pkg.tar.zst'
 INFO  Beginning pre boot
Error: lstat /tmp/hauler3990812661/bundles: no such file or directory
2021/06/22 15:52:19 lstat /tmp/hauler3990812661/bundles: no such file or directory

Steps To Reproduce:

Expected behavior:

Actual behavior:

Additional context / logs:

Collect repos

Serve container images

#5 Collect images from image list file
#6 Collect images from image archives
#7 Deploy docker registry
#8 Populate registry with all images

rancherfederal / hauler Goto Github PK

hauler's Introduction

Rancher Government Hauler

Airgap Swiss Army Knife

Installation

Linux/Darwin

Homebrew

Windows

Acknowledgements

hauler's People

Contributors

Stargazers

Watchers

Forkers

hauler's Issues

Tools:

Recommend Projects

Recommend Topics

Recommend Org