robur-coop / albatross Goto Github PK

Albatross: orchestrate and manage MirageOS unikernels with Solo5

License: ISC License

OCaml 91.34% C 3.07% Shell 3.64% Nix 1.96%

mirageos unikernel virtual-machine ocaml security provisioning deployment orchestration

albatross's Introduction

Albatross: orchestrate and manage MirageOS unikernels with Solo5

The goal of albatross is robust deployment of MirageOS unikernels using Solo5. Resources managed by albatross are network interfaces of kind tap, which are connected to already existing bridges, block devices, memory, and CPU. Each unikernel is pinned (cpuset / taskset) to a specific core.

Albatross allows remote management. To deploy or destroy a unikernel, no shell access is necessary. The remote channel is a mutually authenticated (with X.509 certificates) TLS connection. Console output of the unikernels is stored in memory in a ring buffer, and accessible remotely. Monitoring data (CPU and memory usage) of the unikernels can be collected as well, and pushed into an Influx time series database.

Albatross consists of multiple processes, each running with the least privileges. Albatross can be run next to other orchestration systems; it does not assume to be the single instance on a dom0 which creates and destroys virtual machines. Resource policies can be dynamically configured for each administrative domain (similar to DNS, a hierarchical naming scheme), and are checked statically (to decrease while going down the tree) and dynamically when a new unikernel is to be deployed.

When a unikernel is deployed, albatross tries its best to keep it running, even when the physical hardware reboots, or albatross is restarted. When the unikernel exits, depending on configuration and its exit code, it is re-started. The current set of running unikernels is persisted on disk, though there is no dependency or order for restarting them.

The scope of albatross is to provide a minimal orchestration system that avoids the need for shell access on the dom0. This leads to mostly immutable - or only mutable via albatross - infrastructure. Further dissemination of albatross into virtual machines, and a communication interface for deploying and destroying unikernels, is being researched on.

Components

Albatross consists of a set of binaries. Several daemons, which communicate in a request-response style over Unix domain sockets, are run in the host system:

albatrossd: privileged to create and destroy unikernels
albatross-console: reads the console output of unikernels
albatross-stats: statistics gathering (rusage, ifstat, BHyve debug counters)
albatross-tls-endpoint: remote deployment via TLS (and possibly inetd)
albatross-influx: statistic reporting from albatross-stats to influx

The main daemon is the privileged albatrossd, which supervises unikernels. It opens a listening Unix domain socket, reads the persisted unikernel configuration, starts these unikernels, and awaits commands. Access can be regulated by Unix file permissions -- only those users who can write to that socket can send commands.

Albatross-console does not keep any persistent state, but a ring buffer of console output from each unikernel. These messages can be retrieved by a client as a stream of messages (history, and whenever a new line is output, it is sent to the interested client). Each unikernel output can only be read by a single client, to avoid amplification of traffic if lots of clients are connected. Albatrossd sends a message to albatross-console whenever a new unikernel is started, upon reception albatross-console opens and reads the fifo which the unikernel will write their standard output to.

Albatross-stats periodically gathers statistics (memory, CPU, network, hypervisor) from all running unikernels.

Albatross-tls-endpoint listens on a TCP port, or uses systemd socket activation, or via inetd, and proxies requests from remote clients to the respective daemons described above. It enforces client authentication, and uses the common names of the client certificate chain as the administrative domain. The policies are embedded in CA certificates, and the command is embedded in the leaf certificate.

The albatross-client is provided for both local and remote management. It executes the provided command, and can also prepare certificate signing requests (--csr) to send the certificate at a later point. It also includes functionality for generating an initial CA (and server certificate), and signing certificate signing requests.

Albatross over TLS

Albatross uses PKI to authenticate both client and server. Requests are signed by a certificate authority (CA) that is trusted by the server. CA can delegate resources using policies, which happens by creating an intermediate CA. Revokation is not implemented, as delegation happens without the server knowing about it.

This example shows how one can delegate part of the resources to a user. There are 4 entities:

server: runs the albatross TLS endpoint
CA: root certificate authority
intermediate CA: user certificate authority
client: requests initiator

Note: there are 4 entities but depending on the security model some can exist on the same machine. For example, when client and intermediate CA can be combined, requests are automatically signed using albatross-client --destination (see step 8).

Setup

This step-by-step guide shows how files are generated and to which entity they belong. Filename is in bold when it's created by the current step.

Generate the root CA certificate and server keypair

albatross-client generate ca db

description		server	CA	intermediate CA	client
private key		server.key	ca.key
public certificate		server.pem	cacert.pem

server: start the endpoint using the server keypair and the root CA certificate

albatross-tls-endpoint cacert.pem server.pem server.key

intermediate CA: we want to delegate part of the resources to a given user. The user generates a signing request to allow a memory of 1024MB to run 16 unikernels on CPU IDs 0 and 1.

albatross-client add_policy user 16 --mem 1024 --cpu 0 --cpu 1 --csr

description	server	CA	intermediate CA
private key	server.key	ca.key	user.key
public certificate	server.pem	cacert.pem
certificate signing request			user.req

CA: CA signs the user's request, which generates an intermediate CA certificate containing the restriction policies (limited memory, cpu), which in turn will be used to sign user requests.

albatross-client sign cacert.pem db ca.key user.req

description	server	CA	intermediate CA
private key	server.key	ca.key	user.key
public certificate	server.pem	cacert.pem	user.pem
certificate signing request			user.req

client: the client wants to create an unikernel, it has to wrap the request in a certificate signing request which will be submitted to the intermediate CA.

albatross-client create hello hello-key.hvt --csr [--arg='--hello=albatross-hi'] [--cpu=1]

description	server	CA	intermediate CA	client
private key	server.key	ca.key	user.key	hello.key
public certificate	server.pem	cacert.pem	user.pem
certificate signing request			user.req	hello.req

intermediate CA: the intermediate CA signs the request

albatross-client sign user.pem db user.key hello.req

description	server	CA	intermediate CA	client
private key	server.key	ca.key	user.key	hello.key
public certificate	server.pem	cacert.pem	user.pem	hello.pem
certificate signing request			user.req	hello.req

client: client sends the signed request to the server, albatross-client sign appended the intermediate CA certificate to hello.pem to form the full chain.

albatross-client certificate cacert.pem hello.pem hello.key --destination <REMOTE_IP:PORT>`

Steps 5, 6, and 7 can be done in a single command - if there's no requirement to retain the signing request and certificate, and the user keys are on the local machine.

albatross-client create hello hello.hvt --ca=user.pem --ca-key=user.pem --server-ca=cacert.pem --destination <REMOTE_IP:PORT> [--arg='--hello=albatross-hi'] [--cpu=1]

Installation

Binary packages are available for Debian, Ubuntu and FreeBSD. How to install.

For other operating systems / distributions, run opam install albatross.

Also read the blog article for the motivation behind albatross and an overview of its functionality.

albatross's People

Contributors

Stargazers

Watchers

Forkers

sg2342 bramford cfcs yomimono dinosaure magnuss nickbetteridge reynir kit-ty-kate smorimoto julow samoht ceonlabs thelortex dra27 firobe

albatross's Issues

exception

somehow on FreeBSD, the albatross-stat ran into Fatal error: exception Invalid_argument("List.combine"), which in turn resulted in albatrossd unable to send more data to the stats daemon (and thus, a failure without cleanup of tap devices etc.).

unikernel size - compression

at the moment, the DER encoded X.509 certificate (chain!) may occupy only 2^24-1 bytes (thanks to TLS). we could increase the possible image size by compressing / decompressing the ELF binary embedded into the certificate. this would require (a) new OID to do so (b) select a suitable compression algorithm (decompress comes to mind since it is in OCaml, and would not increase external dependencies).

an initial "benchmark" between gzip, bzip2, and xz showed for 3.7MB original image size, 1.2MB for xz, 1.3MB for bzip2, and 1.5MB for gzip. we should optimise for size, not for speed (maybe a gzip -9 or whatever is a fine solution)... we can have both vmm_req_vm and vmmd use the same decompress library :)

document and smooth the CA experience

related is #52 (for adjusting / decreasing the resources for a specific person)

@TheLortex tried to use the remote-tls endpoint and the albatross-client-remote-tls. Noteworthy things:

albatross-provision-ca sign of an intermediary (sub)CA is allowed even if there are no CPU -- which is slightly useless
it could as well check there to be at least one VM

Also, the albatross-client-remote-tls takes in a (client) certificate and the CA certificate -- but it expects an entire client CA chain and could locally verify that it's a good certificate before connecting to the remote (and transferring the entire unikernel). The "CA sign" could output a bundle instead of a single certifcate (though it may not know the entire chain... if there's more than one level of delegation involved).

support other VM types

the console output and statistics gathering is so useful that it'd be great (esp. with #53) to support other virtual machines to be started. for my own use, this would be: bhyve (both FreeBSD native and linux ones).

the semantics would be quite different since these virtual machines won't fit into a X.509 certificate -- instead the block device(s) should already exist and contain the operating system. some additional notes:

bhyve virtual machine:
- name
- #cpus
- #memory
- bridges
- block devices (not necessarily maintained by albatross)
- bhyveload or grub2-bhyve?
  - former: nothing more
  - latter: boot thingy (hd0, msdos1) -- and map "(hd0) /dev/zvol/data/mx"

--> cpu in bhyve now (no need for cpuset):
  -c [[cpus=]numcpus][,sockets=n][,cores=n][,threads=n]
  and -p vcpu:hostcpu

exit codes:
     0       rebooted
     1       powered off
     2       halted
     3       triple fault
     4       exited due to an error
~> a bit different from solo5 (but just executing bhyve prints help and returns 1 :/)

-> generate:
bhyveload -c stdio -m <mem> -d <boot=first-block> <name>
OR
grub-bhyve -m <map> -r hd0,msdos1 -M <mem> <name>
 (with hd0 and msdos1 hardcoded, and <map> written as (hd0) <first-block-device>)

the :0 is superfluous (the default)
tapN,mac=xx:xx:xx:xx:xx:xx

bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc
  -s <n>:0,virtio-net,tapN
  -s <m>:0,virtio-blk,/dev/zvol/data/block
  -l com1,stdio -c <#cpu> -m <mem> <name>

revise metrics wire format and data

The current ASN grammar:

stats DEFINITIONS ::=
BEGIN
statistics ::= SEQUENCE {

    resource-usage SEQUENCE {
      utime SEQUENCE {
        seconds OCTET STRING,
        microseconds INTEGER
      },
      stime SEQUENCE {
        seconds OCTET STRING,
        microseconds INTEGER
      },
      maxrss OCTET STRING,
      ixrss OCTET STRING,
      idrss OCTET STRING,
      isrss OCTET STRING,
      minflt OCTET STRING,
      majflt OCTET STRING,
      nswap OCTET STRING,
      inblock OCTET STRING,
      outblock OCTET STRING,
      msgsnd OCTET STRING,
      msgrcv OCTET STRING,
      nsignals OCTET STRING,
      nvcsw OCTET STRING,
      nivcsw OCTET STRING
    },

    ifdata SEQUENCE OF SEQUENCE {
      bridge UTF8String,
      flags INTEGER,
      send-length INTEGER,
      max-send-length INTEGER,
      send-drops INTEGER,
      mtu INTEGER,
      baudrate OCTET STRING,
      input-packets OCTET STRING,
      input-errors OCTET STRING,
      output-packets OCTET STRING,
      output-errors OCTET STRING,
      collisions OCTET STRING,
      input-bytes OCTET STRING,
      output-bytes OCTET STRING,
      input-mcast OCTET STRING,
      output-mcast OCTET STRING,
      input-dropped OCTET STRING,
      output-dropped OCTET STRING
    },

    vmm-stats [0] SEQUENCE OF SEQUENCE {
      key UTF8String,
      value OCTET STRING
    } OPTIONAL,

    kinfo-mem [1] IMPLICIT SEQUENCE {
      bsize OCTET STRING,
      rss OCTET STRING,
      tsize OCTET STRING,
      dsize OCTET STRING,
      ssize OCTET STRING,
      runtime OCTET STRING,
      cow INTEGER,
      start SEQUENCE {
        seconds OCTET STRING,
        microseconds INTEGER
      }
    } OPTIONAL

  }
END

notice

lots of data is not really of interest (and 0 - esp. on linux)
time is encoded as struct timeval (int * int64) -> use int64 instead (for utime/stime at least), start needs some thinking
int64 is encoded as octet string -> use integer (or int?) instead (depending on 32bit support or not, int may be insufficient)

what is interesting to gather from the host system: virtual and resident memory size, CPU usage (in percent and/or total), interface statistics

combine rusage and kinfo-mem, drop vmm-stats.

memory consumption of daemons

while albatross_log uses a ring buffer (as do albatross_console and others), it uses with a 65 MB big logfile easily >400MB memory. needs investigation and improvements.

clock skew trouble with vmmc_bistro

if the operator has a skewed clock in respect to the albatross server, vmmc_bistro will produce a signed certificate that is not yet / not anymore valid on the server side. there are no good ways to protect, but a workaround is to sign the certificate with a timestamp of now - 1m to allow for a clock skew of 1 minute that the server may be behind the client.

albatrossd did not clear up tap devices on shutdown

a service albatross_daemon stop killed all the unikernels, but left the tap devices in place. this was with 2d26a56. need to investigate (stress test) whether this is still an issue with more current albatross.

release checklist

Handle 404 in albatross-client update

We could explain that it seems the unikernel is not from that repository. Instead, we just print the http error.

$ albatross-client-local update --dryrun stub-resolver
albatross-client-local: [WARNING] received HTTP reply: ((version "HTTP/1.1") (status 404) (reason "Not Found") (headers 
                                  (("Content-Length" "1235")
                                  ("Content-Type" "text/html; charset=utf-8"))))
albatross-client-local: [ERROR] error in HTTP interaction: unexpected HTTP reply

debian packaging: should libnl-3-dev and libnl-route-3-dev be a dependency?

since albatross-stats depends on these packages, and both ubuntu and debian use the same package names, shouldn't they be part of the dependencies in the packaging/debian subfolder? //cc @reynir

License

Any idea of what the license for albatross will be?

provide a re-deploy of albatross story

sometimes albatross needs to be upgraded due to bugs or new features. this usually means either to manually destroy all the virtual machines, or sending a SIGHUP/SIGKILL to the process.

the latter doesn't properly clean up the used resources -- and neither approach allows a fresh albatross to restart all the virtual machines (the images are still on disk, while the certificates with the arguments aren't).

there are two options here:

cleanup TMPDIR on startup!
retain enough information "persistently" about running virtual machine to start them upon vmmd launch!

since both options are valuable in different scenarios, there should be a command line flag to vmmd. The unix domain sockets for console output etc. need to be wiped upfront in either case.

use solo5-elftool opam package instead of calling out to solo5-elftool

this removes the dependency in albatross-client-bistro -- it does not make a difference for the albatross daemon (which requires solo5-hvt/solo5-spt)

//cc @reynir

albatross.deb for ubuntu 22.04

I've been playing around with the debian packaging that @reynir has provided.

Firstly, just let me say that I'm no linux expert!

I cloned albatross, did a dune build
I edited 'albatross/packaging/debian/control' and updated %%VERSION%% to the current version - not too sure how to automate this
sh -ex packaging/debian/create_package.sh
sudo dpkg -i ./albatross.deb
sudo apt install -f
sudo systemctl enable albatross_daemon.service , etc

Deploying unikernels didn't work and I discovered that, for example, trying to deploy a hvt locally resulted in albatross-client-local trying to execute solo5-hvt in /var/lib/albatross, whereas the albatross binaries had been installed in /usr/bin.

I guess the packaging scripts are really meant for user customisation. Nevertheless, using albatross is a dream - thanks for making it available.

The following ubuntu netplan is for setting up a static host ip address with a service bridge for albatross - just in case it's useful for anyone

network:
  version: 2
  renderer: networkd

  ethernets:
    enpXXX:
      dhcp4: false
      dhcp6: false

  bridges:
    service:
      interfaces: [enpXXX]
      addresses: [10.0.0.XXX/24]
      # gateway4 is deprecated, use routes instead
      routes:
      - to: default
        via: 10.0.0.1
        metric: 100
        on-link: true
      mtu: 1500
      nameservers:
        addresses: [1.1.1.1,1.0.0.1]
      parameters:
        stp: true
        forward-delay: 4
      dhcp4: no
      dhcp6: no

where enpXXX is the exposed interface and 10.0.0.XXX/24 is the static IP for the host

specified bridge(s) does not match with the manifest

Greetings!

So, I haven't used albatross and unikernels for a long time - from reading the various mirage tutorials, I have made a bridge in ubuntu, pulled down static_website_tls from mirage-skeleton and built it (the whole flow of building cross-compiled unikernels has become a dream - qudos to all), built the debian version of albatross.deb and installed it. Swimmingly good so far.

When I tried a sudo albatross-client-local create --net=br0 --arg='--ipv4=10.0.0.27/24' my-hello-unikernel https.spt I got albatross-client-local: specified bridge(s) does not match with the manifest.

I can see that I could add something to the manifest.json in the unikernel/mirage dir, but I couldn't find (and didn't know where to look) for any pointers on how to tweek this.

There might be an issue with my netplan in ubuntu 22.04 - my host interface is enp0s1 with an ip of 10.0.0.27 and mac of cb:5c:8f:86:b0:d2.

  version: 2
  renderer: networkd
  ethernets:
    enp0s1:
      dhcp4: no
      dhcp6: no
      addresses:
        - 10.0.0.27/24
      routes:
        - to : default
          via: 10.0.0.1
      nameservers:
        addresses:
          - 1.1.1.1
          - 1.0.0.1
  bridges:
    br0:
      interfaces: [enp0s1]
      addresses: [10.0.0.27/24]
      routes:
        - to : default
          via: 10.0.0.1
      nameservers:
        addresses: [ 1.1.1.1, 1.0.0.1 ]
      macaddress: cb:5c:8f:86:b0:d2
      parameters:
        stp: true
        forward-delay: 4
      dhcp4: no
      dhcp6: no

However, sudo brctl show gives

bridge name	bridge id		STP enabled	interfaces
br0		8000.9c5c8e86b0da	yes		enp0s1

which suggests that there isn't an issue.

So, I'm a bit stuck on what to do next - any suggestions?

exit codes of client utilities

the client applications (bistro, local, remote_tls) should report:

0 on success
y on communication failure (i.e. access denied, ...)
z on command execution failure (i.e. info didn't list any unikernel, ..) - the z may be multiple concrete values if this makes sense

then they may be more useful from a scripting perspective (although they should not really be used in shell scripts, and instead the library/API should be used instead...)

Systemd: include network.target in After=

On a system where the service bridge is configured by NetworkManager albatross fails to restore VMs after a reboot due to the bridge not being up:

Jan 03 15:02:20 solsort albatrossd[1877]: albatrossd: [ERROR] failed to create [vm: stub-resolver]: interface service does not exist

Likely, adding network.target to the list in After=... in the systemd service files will fix this. I have added network.target on the machine and will confirm whether this fixed the issue (after I have done some upgrades).

finish implementing block device handling

it is not at all at the moment

Check manifest for devices

Using solo5-elftool query-manifest we can check if the user has passed exactly all the devices required by the unikernel when using the create command, and report an error if they don't match.

query manifest failure

I tried to send an unikernel to an albatross endpoint but I did not succeed. Tracking down the pb leads me to that situation, from the remote host I can create the .req, sign it and it fails to start the VM (the journalctl says that it fails to query abi), and locally provision-request does not creates the .req file and fails with a query manifest error message.

Strangely, solo5-elftool seems to correctly read the .hvt kernel. (The kernel is the static tls from mirage-skeleton, I have mirage 3.10.8 and ocaml switch is 4.13.1). I installed solo5-elftool and albatross with mirage but the system may still have some old installation remnants (I tried to remove them manually and which albatross-provision-request tells me that I launch the correct binary).

$ eval $(opam env)
$ albatross-provision-request create --net=service:br0 --arg=--ipv4-gateway=10.0.0.1 --arg=--ipv4=10.0.0.3/24 https https.hvt
albatross-provision-request: query manifest failure: Owee_buf.Invalid_format("No ELF magic number")
$ albatross-provision-request --version
version v1.4.1 protocol version 4
$ solo5-elftool --version
usage: solo5-elftool COMMAND ...
solo5-elftool version v0.6.9

COMMAND is:
    gen-manifest SOURCE OUTPUT:
        Generate application manifest from SOURCE, writing to OUTPUT.
    query-abi BINARY:
        Display the ABI target and version from BINARY.
    query-manifest BINARY:
        Display the application manifest from BINARY.
$ solo5-elftool query-manifest https.hvt
{
  "type": "solo5.manifest",
  "version": 1,
  "devices": [
    { "name": "service", "type": "NET_BASIC" }
  ]
}

startup dependencies

since albatross dumps all unikernels to disk, and restarts them on startup, it'd be nice to have control over the start ordering.

using names here would be quite a lot of hassle, but I think a single priority (similar to nice values, or your old /etc/rc.d/SYY-foo numbers): start in ascending order, by default put all unikernels at 100.

with this feature, a DNS resolver & DHCP server can be started early by setting its priority to a lower number.

Record start time and report it in `albatross-client info`

This issue can arguably be solved by setting up proper monitoring.

With the option --restart-on-fail unikernels are restarted when they exit for example if they run out of memory. If you monitor closely the console output (albatross-client console my-unikernel) or if you have proper monitoring setup you can figure out when this happens. Otherwise this information is easily lost. It would be nice to have reported in albatross-client info the start time of the unikernel or how long it's been running.

document and revise ASN.1 grammar

This is the ASN.1 grammar of the albatross 1.0.0 release.

The grammar used in client certificates (with a custom OID as key -- namely 1.3.6.1.4.1.49836.42) is (version is 4):

CertExtension DEFINITIONS ::=
BEGIN
CertExtension ::=
SEQUENCE {
  version INTEGER,
  command CHOICE {

    console [0] CHOICE {
      add [0] NULL,
      subscribe [1] CHOICE {
        since [0] UTCTime,
        count [1] INTEGER
      }
    },

    statistics [1] CHOICE {
      add [0] SEQUENCE {
        vmmdev UTF8String,
        pid INTEGER,
        network SEQUENCE OF SEQUENCE {
          bridge UTF8String,
          tap UTF8String
        }
      },
      remove [1] NULL,
      subscribe [2] NULL
    },

    log [2] CHOICE {
      subscribe-since [0] UTCTime,
      subscribe-count [1] INTEGER
    },

    unikernel [3] CHOICE {
      info [0] NULL,

      create-OLD [1] SEQUENCE {
        typ CHOICE {
          solo5 [0] NULL,
          placeholder [1] NULL
        },
        compressed BOOLEAN,
        image OCTET STRING,
        fail-behaviour CHOICE {
          quit [0] NULL,
          restart-exit-codes [1] SET OF INTEGER
        },
        cpuid INTEGER,
        memory INTEGER,
        blocks [0] SET OF UTF8String OPTIONAL,
        bridges [1] SET OF UTF8String OPTIONAL,
        arguments [2] SEQUENCE OF UTF8String OPTIONAL
      },

      force-create-OLD [2] SEQUENCE {
        typ CHOICE {
          solo5 [0] NULL,
          placeholder [1] NULL
        },
        compressed BOOLEAN,
        image OCTET STRING,
        fail-behaviour CHOICE {
          quit [0] NULL,
          restart-exit-codes [1] SET OF INTEGER
        },
        cpuid INTEGER,
        memory INTEGER,
        blocks [0] SET OF UTF8String OPTIONAL,
        bridges [1] SET OF UTF8String OPTIONAL,
        arguments [2] SEQUENCE OF UTF8String OPTIONAL
      },

      destroy [3] NULL,

      create [4] SEQUENCE {
        typ CHOICE {
          solo5 [0] NULL,
          placeholder [1] NULL
        },
        compressed BOOLEAN,
        image OCTET STRING,
        fail-behaviour CHOICE {
          quit [0] NULL,
          restart-exit-codes [1] SET OF INTEGER
        },
        cpuid INTEGER,
        memory INTEGER,
        blocks [0] SET OF UTF8String OPTIONAL,
        bridges [1] SEQUENCE OF SEQUENCE {
          netif UTF8String,
          bridge UTF8String OPTIONAL
        } OPTIONAL,
        arguments [2] SEQUENCE OF UTF8String OPTIONAL
      },

      force-create [5] SEQUENCE {
        typ CHOICE {
          solo5 [0] NULL,
          placeholder [1] NULL
        },
        compressed BOOLEAN,
        image OCTET STRING,
        fail-behaviour CHOICE {
          quit [0] NULL,
          restart-exit-codes [1] SET OF INTEGER
        },
        cpuid INTEGER,
        memory INTEGER,
        blocks [0] SET OF UTF8String OPTIONAL,
        bridges [1] SEQUENCE OF SEQUENCE {
          netif UTF8String,
          bridge UTF8String OPTIONAL
        } OPTIONAL,
        arguments [2] SEQUENCE OF UTF8String OPTIONAL
      },

      get [6] NULL,

      placeholder [7] NULL
    },

    policy [4] CHOICE {
      info [0] NULL,
      add [1] SEQUENCE {
        cpuids SEQUENCE OF INTEGER,
        vms INTEGER,
        memory INTEGER,
        block INTEGER OPTIONAL,
        bridges SEQUENCE OF UTF8String
      },
      remove [2] NULL
    },

    block [5] CHOICE {
      info [0] NULL,
      add [1] INTEGER,
      remove [2] NULL
    }
  }
}
END

The log data on disk:

LogEntry DEFINITIONS ::=
BEGIN
LogEntry ::=
SEQUENCE {
  version INTEGER,
  entry SEQUENCE {
    timestamp UTCTime,
    event CHOICE {
      startup [0] NULL,

      login [1] SEQUENCE {
        name SEQUENCE OF UTF8String,
        ip OCTET STRING,
        port INTEGER
      },

      logout [2] SEQUENCE {
        name SEQUENCE OF UTF8String,
        ip OCTET STRING,
        port INTEGER
      },

      unikernel-start-OLD [3] SEQUENCE {
        name SEQUENCE OF UTF8String,
        pid INTEGER,
        taps SEQUENCE OF UTF8String,
        block UTF8String OPTIONAL
      },

      unikernel-stop [4] SEQUENCE {
        name SEQUENCE OF UTF8String,
        pid INTEGER,
        status CHOICE {
          exit-code [0] INTEGER,
          signal [1] INTEGER,
          stopped [2] INTEGER
        }
      },

      hup [5] NULL,

      unikernel-start [6] SEQUENCE {
        name SEQUENCE OF UTF8String,
        pid INTEGER,
        taps SEQUENCE OF SEQUENCE {
          bridge UTF8String,
          tap UTF8String
        },
        blocks SEQUENCE OF SEQUENCE {
          name UTF8String,
          device UTF8String
        }
      },

      placeholder [7] NULL
    }
  }
}
END

The state file on disk:

State DEFINITIONS ::=
BEGIN
State ::=
CHOICE {
  unikernel-OLD1 [0] SEQUENCE OF SEQUENCE {
    name UTF8String,
    value SEQUENCE {
      typ CHOICE {
        solo5 [0] NULL,
        placeholder [1] NULL
      },
      compressed BOOLEAN,
      image OCTET STRING,
      fail-behaviour CHOICE {
        quit [0] NULL,
        restart-exit-codes [1] SET OF INTEGER
      },
      cpuid INTEGER,
      memory INTEGER,
      blocks [0] SET OF UTF8String OPTIONAL,
      bridges [1] SET OF UTF8String OPTIONAL,
      arguments [2] SEQUENCE OF UTF8String OPTIONAL
    }
  },

  unikernel-OLD0 [1] SEQUENCE OF SEQUENCE {
    name UTF8String,
    value SEQUENCE {
      cpu INTEGER,
      memory INTEGER,
      block UTF8String OPTIONAL,
      network-interfaces SEQUENCE OF UTF8String OPTIONAL,
      image CHOICE {
        hvt-amd64 [0] OCTET STRING,
        hvt-arm64 [1] OCTET STRING,
        hvt-amd64-compressed [2] OCTET STRING
      },
      arguments SEQUENCE OF UTF8String OPTIONAL
    }
  },

  unikernel [2] SEQUENCE OF SEQUENCE {
    name UTF8String,
    value SEQUENCE {
      typ CHOICE {
        solo5 [0] NULL,
        placeholder [1] NULL
      },
      compressed BOOLEAN,
      image OCTET STRING,
      fail-behaviour CHOICE {
        quit [0] NULL,
        restart-exit-codes [1] SET OF INTEGER
      },
      cpuid INTEGER,
      memory INTEGER,
      blocks [0] SET OF UTF8String OPTIONAL,
      bridges [1] SEQUENCE OF SEQUENCE {
        netif UTF8String,
        bridge UTF8String OPTIONAL
      } OPTIONAL,
      arguments [2] SEQUENCE OF UTF8String OPTIONAL
    }
  }
}
END

And finally the grammar what answer(s) to expect on the TLS connection (same is the communication between the daemons):

Wire DEFINITIONS ::=
BEGIN
Wire ::=
SEQUENCE {
  header SEQUENCE {
    version INTEGER,
    sequence OCTET STRING,
    name SEQUENCE OF UTF8String
  },
  payload CHOICE {

    command [0] CHOICE {

      console [0] CHOICE {
        add [0] NULL,
        subscribe [1] CHOICE {
          since [0] UTCTime,
          count [1] INTEGER
        }
      },

      statistics [1] CHOICE {
        add [0] SEQUENCE {
          vmmdev UTF8String,
          pid INTEGER,
          network SEQUENCE OF SEQUENCE {
            bridge UTF8String,
            tap UTF8String
          }
        },
        remove [1] NULL,
        subscribe [2] NULL
      },

      log [2] CHOICE {
        subscribe-since [0] UTCTime,
        subscribe-count [1] INTEGER
      },

      unikernel [3] CHOICE {
        info [0] NULL,
        create-OLD [1] SEQUENCE {
          typ CHOICE {
            solo5 [0] NULL,
            placeholder [1] NULL
          },
          compressed BOOLEAN,
          image OCTET STRING,
          fail-behaviour CHOICE {
            quit [0] NULL,
            restart-exit-codes [1] SET OF INTEGER
          },
          cpuid INTEGER,
          memory INTEGER,
          blocks [0] SET OF UTF8String OPTIONAL,
          bridges [1] SET OF UTF8String OPTIONAL,
          arguments [2] SEQUENCE OF UTF8String OPTIONAL
        },
        force-create-OLD [2] SEQUENCE {
          typ CHOICE {
            solo5 [0] NULL,
            placeholder [1] NULL
          },
          compressed BOOLEAN,
          image OCTET STRING,
          fail-behaviour CHOICE {
            quit [0] NULL,
            restart-exit-codes [1] SET OF INTEGER
          },
          cpuid INTEGER,
          memory INTEGER,
          blocks [0] SET OF UTF8String OPTIONAL,
          bridges [1] SET OF UTF8String OPTIONAL,
          arguments [2] SEQUENCE OF UTF8String OPTIONAL
        },
        destroy [3] NULL,
        create [4] SEQUENCE {
          typ CHOICE {
            solo5 [0] NULL,
            placeholder [1] NULL
          },
          compressed BOOLEAN,
          image OCTET STRING,
          fail-behaviour CHOICE {
            quit [0] NULL,
            restart-exit-codes [1] SET OF INTEGER
          },
          cpuid INTEGER,
          memory INTEGER,
          blocks [0] SET OF UTF8String OPTIONAL,
          bridges [1] SEQUENCE OF SEQUENCE {
            netif UTF8String,
            bridge UTF8String OPTIONAL
          } OPTIONAL,
          arguments [2] SEQUENCE OF UTF8String OPTIONAL
        },
        force-create [5] SEQUENCE {
          typ CHOICE {
            solo5 [0] NULL,
            placeholder [1] NULL
          },
          compressed BOOLEAN,
          image OCTET STRING,
          fail-behaviour CHOICE {
            quit [0] NULL,
            restart-exit-codes [1] SET OF INTEGER
          },
          cpuid INTEGER,
          memory INTEGER,
          blocks [0] SET OF UTF8String OPTIONAL,
          bridges [1] SEQUENCE OF SEQUENCE {
            netif UTF8String,
            bridge UTF8String OPTIONAL
          } OPTIONAL,
          arguments [2] SEQUENCE OF UTF8String OPTIONAL
        },
        get [6] NULL,
        placeholder [7] NULL
      },

      policy [4] CHOICE {
        info [0] NULL,
        add [1] SEQUENCE {
          cpuids SEQUENCE OF INTEGER,
          vms INTEGER,
          memory INTEGER,
          block INTEGER OPTIONAL,
          bridges SEQUENCE OF UTF8String
        },
        remove [2] NULL
      },

      block [5] CHOICE {
        info [0] NULL,
        add [1] INTEGER,
        remove [2] NULL
      }
    },

    reply [1] CHOICE {
      empty [0] NULL,

      string [1] UTF8String,

      policies [2] SEQUENCE OF SEQUENCE {
        name SEQUENCE OF UTF8String,
        policy SEQUENCE {
          cpuids SEQUENCE OF INTEGER,
          vms INTEGER,
          memory INTEGER,
          block INTEGER OPTIONAL,
          bridges SEQUENCE OF UTF8String
        }
      },

      unikernels [3] SEQUENCE OF SEQUENCE {
        name SEQUENCE OF UTF8String,
        config SEQUENCE {
          typ CHOICE {
            solo5 [0] NULL,
            placeholder [1] NULL
          },
          compressed BOOLEAN,
          image OCTET STRING,
          fail-behaviour CHOICE {
            quit [0] NULL,
            restart-exit-codes [1] SET OF INTEGER
          },
          cpuid INTEGER,
          memory INTEGER,
          blocks [0] SET OF UTF8String OPTIONAL,
          bridges [1] SEQUENCE OF SEQUENCE {
            netif UTF8String,
            bridge UTF8String OPTIONAL
          } OPTIONAL,
          arguments [2] SEQUENCE OF UTF8String OPTIONAL
        }
      },

      block-devices [4] SEQUENCE OF SEQUENCE {
        name SEQUENCE OF UTF8String,
        size INTEGER,
        active BOOLEAN
      }
    },

    failure [2] UTF8String,

    data [3] CHOICE {

      console [0] SEQUENCE {
        timestamp UTCTime,
        data UTF8String
      },

      statistics [1] SEQUENCE {
        resource-usage SEQUENCE {
          utime SEQUENCE {
            seconds OCTET STRING,
            microseconds INTEGER
          },
          stime SEQUENCE {
            seconds OCTET STRING,
            microseconds INTEGER
          },
          maxrss OCTET STRING,
          ixrss OCTET STRING,
          idrss OCTET STRING,
          isrss OCTET STRING,
          minflt OCTET STRING,
          majflt OCTET STRING,
          nswap OCTET STRING,
          inblock OCTET STRING,
          outblock OCTET STRING,
          msgsnd OCTET STRING,
          msgrcv OCTET STRING,
          nsignals OCTET STRING,
          nvcsw OCTET STRING,
          nivcsw OCTET STRING
        },
        ifdata SEQUENCE OF SEQUENCE {
          bridge UTF8String,
          flags INTEGER,
          send-length INTEGER,
          max-send-length INTEGER,
          send-drops INTEGER,
          mtu INTEGER,
          baudrate OCTET STRING,
          input-packets OCTET STRING,
          input-errors OCTET STRING,
          output-packets OCTET STRING,
          output-errors OCTET STRING,
          collisions OCTET STRING,
          input-bytes OCTET STRING,
          output-bytes OCTET STRING,
          input-mcast OCTET STRING,
          output-mcast OCTET STRING,
          input-dropped OCTET STRING,
          output-dropped OCTET STRING
        },
        vmm-stats [0] SEQUENCE OF SEQUENCE {
          key UTF8String,
          value OCTET STRING
        } OPTIONAL,
        kinfo-mem [1] IMPLICIT SEQUENCE {
          bsize OCTET STRING,
          rss OCTET STRING,
          tsize OCTET STRING,
          dsize OCTET STRING,
          ssize OCTET STRING,
          runtime OCTET STRING,
          cow INTEGER,
          start SEQUENCE {
            seconds OCTET STRING,
            microseconds INTEGER
          }
        } OPTIONAL
      },

      log [2] SEQUENCE {
        timestamp UTCTime,
        event CHOICE {
          startup [0] NULL,
          login [1] SEQUENCE {
            name SEQUENCE OF UTF8String,
            ip OCTET STRING,
            port INTEGER
          },
          logout [2] SEQUENCE {
            name SEQUENCE OF UTF8String,
            ip OCTET STRING,
            port INTEGER
          },
          unikernel-start-OLD [3] SEQUENCE {
            name SEQUENCE OF UTF8String,
            pid INTEGER,
            taps SEQUENCE OF UTF8String,
            block UTF8String OPTIONAL
          },
          unikernel-stop [4] SEQUENCE {
            name SEQUENCE OF UTF8String,
            pid INTEGER,
            status CHOICE {
              exit-code [0] INTEGER,
              signal [1] INTEGER,
              stopped [2] INTEGER
            }
          },
          hup [5] NULL,
          unikernel-start [6] SEQUENCE {
            name SEQUENCE OF UTF8String,
            pid INTEGER,
            taps SEQUENCE OF SEQUENCE {
              bridge UTF8String,
              tap UTF8String
            },
            blocks SEQUENCE OF SEQUENCE {
              name UTF8String,
              device UTF8String
            }
          },
          placeholder [7] NULL
        }
      }
    }
  }
}
END

improve block device support

sync them with network devices. so far, the policy contains "block device capacity" and "bridge names" that are allowed for the arc.

now, a unikernel_create gets a list of :, i.e. "--bridge external:public" or "--bridge internal:management" (or "--bridge public") -- meaning in order: acquire a tap device on bridge "public", and pass "--net:external=" to the tender; acquire a tap device on bridge "management" and pass "--net:internal=" to the tender; acquire a tap device on bridge "public" and pass "--net:public=" to the tender.

the block devices are all kept in "dbdir/block/", so block_add & remove take the arc (from ca chain / header) and create the specified device. this directory is read at startup. now, if a unikernel image 'test' requires "block device 'foo'" (in the com.example domain): dbdir/block/com.example.foo is used -- and passed to the tender -- there's no way to use block device "dbdir/block/com.example.bar" instead. and this is what should be cleaned up (similar to how bridges are passed by having a pair with a string option: "--block foo:bar" should be possible)

cleanup taps, fifo, sockets upon quit (vmmd and also vmm_console/vmm_log/vmm_stats_lwt)

unfortunately I lost debug output, but need to try to reproduce that... just had ~8 tap devices which weren't connected to processes, but have been created by albatross and not being destroyed.

implement statistics on linux

atm it is a stub returning nothing useful. it may be possible to retrieve the data in a structured way using some ioctl, otherwise go and parse /proc/

albatross-daemon fails to restore unikernels because fifo directory does not exist

albatrossd[499]: albatrossd: [ERROR] failed to create [vm: resolver]: file /run/albatross/fifo/resolver error in mkfifo: No such file or directory

The fifo directory is created (only) by the albatross_console.service systemd script. I am not sure why it is there and not in albatross_daemon.service.
https://github.com/roburio/albatross/blob/340f0e4771b57e50f4d7ff63d3115f353db50527/packaging/Linux/albatross_console.service#L15-L17

improve communication errors

atm there are (lwt) exceptions in Vmm_tls and Vmm_lwt, which instead should use the result type. a better recovery from errors is also appreciated (not: connect only once, but instead reconnect upon disconnection to be able to restart the individual processes separately)

add a restart command

to start a running unikernel from the beginning

locations of solo5 binaries

I got some (spurious?) errors when solo5-elftool and solo5-hvt are installed by the system in PATH (instead of dbdir). check that the code does not require them to be in dbdir.

Don't fill `syslog` with DEBUG logs from `albatross_influx`

Currently, albatross_influx runs with -vv which populate too fast my log:
https://github.com/roburio/albatross/blob/83dff4a3fe6a4a1486d935550136c59a35645c34/packaging/Linux/albatross_influx.service#L13

It will be nice to just delete this argument when we distribute albatross for systemd.

fix fd, mem leak in vmm_stats_stubs.c

https://github.com/hannesm/albatross/blob/e48d3b345701bfddc6674b3f6fd418f05400fbc7/stats/vmm_stats_stubs.c#L100

you should

close(vm_get_device_fd(ctx));free(ctx);

here

feature request: retrieve the SHA256 of a running unikernel

at the moment, there is info for a unikernel which returns the unikernel configuration without the unikernel image itself. and there is get that returns the entire configuration (including image).

in the light of reproducible builds etc. it'd be great to get the uniquely identified sha-256 hash of the running (and uncompressed) binary of a unikernel. this would ease inventory management. the hash could be computed upon unikernel startup (from the uncompressed data dumped to disk) to avoid re-computing the hash on every request.

I'd appreciate it being part of the info subcommand (meaning revising the Unikernel.config record and ASN grammar -- but note that it should not be provided by albatross_client_*, but instead computed by the server! so eventually the get and info subcommands should use some other data structure than the create/force-create subcommands)

boot loop + `--restart-on-fail` results in unremovable unikernels

Like a dolt, I sent a unikernel to albatross which boots up and immediately fails with the --restart-on-fail flag set. When I try to destroy the unikernel, I'm told it can't be destroyed because it's not running, but when I try to push another unikernel with the same name, I'm told there is already a unikernel running under that name.

I've removed --restart-on-fail from my default deploy-albatross target for the moment, since I have little faith in my ability to only push non-crashing unikernels.

Avoid daemons to override sockets of already executing daemons

To avoid accidental executions of albatross_console etc. (which will create and claim the socket). Eventually these programs should as well not start up if the socket is already there and opened (but that may lead to rather intricate semantics).

debian packaging: systemd service & installed binary path discrepancy

The debian package installs the daemon binaries in /usr/sbin/ while the systemd service files expects them to be in /usr/local/sbin/. The install.sh script in packaging/Linux/ meanwhile copies them to /usr/local/sbin/.

albatross_influx should cope with remote telegraf instance disconnecting

either reconnect TCP, or get rid of telegraf and use influx protocol via UDP (needs investigation and testing)

force deployment (quit old on demand)

at the moment, when i have unikernel yyy deployed, and i want to update it, i have to sign a certificate and upload this via vmm_client, which requires that after certificate transmission no yyy is running. if yyy is a production system, I'd like to keep it up as long as i can -- meaning I need to encode a force flag in the certificate which let's albatross kill the old yyy after the new one is received and about to start.

albatross_daemon keeps all unikernels in memory

since a Unikernel.config contains a image : Cstruct.t - this is kept in memory during operation. There used to be (in ancient times) code to replace this with the empty cstruct once no longer needed, but since Unikernel_get allows to download the image, this is no longer the case.

Now, the albatross_daemon process running a lot of unikernels unnecessarily keeps the unikernel images in memory. We could (a) compress or (b) re-apply the old methodology and have the get command retrieve it from disk before sending it to the client. This would be very welcome.

TLS endpoint: revocation

Allow uploading of revocation lists to revoke permissions for a host.

rework albatross-stats

There are some issues with albatross-stats:

it's an optional dependency of albatross-daemon (which expects the stats socket to be present or not)
there's quite some communication overhead between albatross-stats and albatross-influx
if albatross-stats goes away, all you can do is restart albatross-daemon

A solution from this mornings brainstorming:

revise the protocol, so an albatross-stats can connect to the albatross-daemon socket, and gets (a) running unikernels at connect-time (b) whenever a unikernel starts or stops, the albatross-stats connection is notified
merge albatross-influx into albatross-stats and avoid any re-marshalling of data. when there's at some point interest in prometheus or other wire formats, we can cope (but still only need a single stats collecting and reporting daemon)

This would as well solve #55 by just not encoding and decoding this information ;)

Provide an "info" subcommand that does not list all secrets (command line parameters)

Could be by default a more brief pretty-printer for Unikernel_info command (not printing command-line arguments), and in a verbose version the extended ones. Or a custom Unikernel_list command that only prints names and resource usages.

Albatross-tls-endpoint

Hi, thanks for your work on albatross, it's very helpful!
I've not updated albatross for a while until last week. I have now an issue starting albatross-tls-endpoint 1.5.3 (the daemon is started with systemd):

# albatross-tls-endpoint --version
version v1.5.3 protocol version 5
# albatross-tls-endpoint --port 1025 -v -v -v --verbosity=debug cacert.pem server.pem server.key
albatross-tls-endpoint: internal error, uncaught exception:
                        Unix.Unix_error(Unix.EINVAL, "bind", "")

This was the commands I used to have but something may have changed recently. With strace I have the following additional information:

...
stat("cacert.pem", {st_mode=S_IFREG|0644, st_size=396, ...}) = 0
stat("server.pem", {st_mode=S_IFREG|0644, st_size=518, ...}) = 0
stat("server.key", {st_mode=S_IFREG|0400, st_size=119, ...}) = 0
rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7ff7a2edb420}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
socket(AF_INET6, SOCK_STREAM, IPPROTO_IP) = 5
fcntl(5, F_GETFD)                       = 0
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
setsockopt(5, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
setsockopt(5, SOL_IPV6, IPV6_V6ONLY, [0], 4) = 0
bind(5, {sa_family=AF_INET, sin_port=htons(1025), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EINVAL (Invalid argument)
write(2, "albatross-tls-endpoint: internal"..., 150albatross-tls-endpoint: internal error, uncaught exception:
                        Unix.Unix_error(Unix.EINVAL, "bind", "")
                        
) = 150
rt_sigaction(SIGCHLD, {sa_handler=0x5652b8785300, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7ff7a2edb420}, NULL, 8) = 0
rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7ff7a2edb420}, NULL, 8) = 0
sigaltstack({ss_sp=0x28, ss_flags=SS_DISABLE, ss_size=140735027670208}, {ss_sp=0x5652b945eb50, ss_flags=0, ss_size=8192}) = 0
exit_group(125)                         = ?
+++ exited with 125 +++

The manual page for bind states that EINVAL is returned when the port is already binded (I checked that with netstat -talp and it's not), or addrlen is wrong, or addr is not a valid address for this ocket's domain.. The addrlen is 16 which suggest that bind try to use an IPv6 whereas sin_addr is an IPv4. Is there a way to specify which IP type address I want to use?

vmm_stats: expose to some mrtg or other graphing system

maybe via munin? also need to actually compute the diff between two runs then

process handling unstable

sometimes i see zombie processes as child processes of albatross' vmmd.

other times, I also see in vmm_client info some vms which are no longer running, destroy says ok, but it keeps in the data structure (and thus I can't re-deploy it)

robur-coop / albatross Goto Github PK

albatross's Introduction

Albatross: orchestrate and manage MirageOS unikernels with Solo5

Components

Albatross over TLS

Setup

Installation

albatross's People

Contributors

Stargazers

Watchers

Forkers

albatross's Issues

Recommend Projects

Recommend Topics

Recommend Org