hadfl / kvmadm Goto Github PK

when having plenty of kvms it might be useful to cascade the start. pre_start_cmd gives the option to run a script or command before kvmadm starts the qemu process. once pre_start_cmd terminates kvmadm will start the qemu process

When Creating a kvm inside a zone using

kvmadm import test2.json

I get the following output:

 
# /opt/kvmadm/bin/kvmadm import test2.json
-> zvol rpool/zones/foo/kvm0 does not exist. creating it...
-> vnic 'vnicfoo0' does not exist. creating it...
-> vnic 'kvmfoo0' does not exist. creating it...
/zones/foo must not be group readable.
/zones/foo must not be group executable.
/zones/foo must not be world readable.
/zones/foo must not be world executable.
/zones/foo: changing permissions to 0700.
   Publisher: Using omnios (http://pkg.omniti.com/omnios/r151014/ ).
       Image: Preparing at /zones/foo/root.

       Cache: Using /var/pkg/publisher.
Sanity Check: Looking for 'entire' incorporation.
  Installing: Packages (output follows)
Packages to install: 386
Mediators to change:   1
 Services to change:   4

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                            386/386   36684/36684  316.0/316.0    0B/s

PHASE                                          ITEMS
Installing new actions                   59892/59892
Updating package state database                 Done
Updating package cache                           0/0
Updating image state                            Done
Creating fast lookup database                   Done

        Note: Man pages can be obtained by installing pkg:/system/manual
 Postinstall: Copying SMF seed repository ... done.
        Done: Installation completed in 72.379 seconds.

  Next Steps: Boot the zone, then log into the zone console (zlogin -C)
              to complete the configuration process.
setting up system/kvm within zone. this might take a while...
svccfg: Pattern 'svc:/system/kvm' doesn't match any instances or services
svccfg: No such property group "manifestfiles".
ERROR: cannot delete property group from svc:/system/kvm
Do you want to retry [Y/n]?

I do retry, but the configuration looks fine, so I continue:
after that it produces no error

setting up SMF instance within zone. this might take a while...

and exits

Here is the test2.json:

{
   "foo" : {
      "disk" : [
         {
            "index" : "0",
            "boot" : "true",
            "disk_path" : "rpool/zones/foo/kvm0",
            "model" : "virtio",
            "disk_size" : "50G",
            "cache" : "none"
         },
         {
            "index" : "1",
            "media" : "cdrom",
            "disk_path" : "/export/iso/CentOS-7-x86_64-Minimal-1511.iso",
            "model" : "ide",
            "cache" : "none"
         }
      ],
      "serial" : [
         {
            "index" : "0",
            "serial_name" : "console"
         }
      ],
      "nic" : [
         {
            "index" : "0",
            "over" : "switch0",
            "nic_name" : "vnicfoo0",
            "model" : "virtio"
         },
         {
            "index" : "1",
            "over" : "e1000g0",
            "nic_name" : "kvmfoo0",
            "model" : "virtio"
         }
      ],
      "zone" : {
         "zonepath" : "/zones/foo",
         "bootargs" : "",
         "hostid" : "",
         "brand" : "lipkg",
         "fs-allowed" : "",
         "scheduling-class" : "",
         "pool" : "",
         "autoboot" : "false",
         "limitpriv" : "default",
         "ip-type" : "exclusive",
         "fs" : [
            {
               "special" : "/etc/opt/kvmadm/myvncpw",
               "options" : "[ro,nodevices]",
               "type" : "lofs",
               "dir" : "/etc/opt/kvmadm/myvncpw"
            }
         ]
      },
      "vnc" : "socket",
      "uuid" : "ed228d46-7862-4a43-93dd-c0ad975fa930",
      "hpet" : "false",
      "boot_order" : "cd",
      "vcpus" : "4",
      "ram" : "2048",
      "time_base" : "utc"
   }
}

Starting the kvm

/opt/kvmadm/bin/kvmadm start foo

produces no error, but also no result. No qemu-system-x86_64 process is running inside the zone.

Examining the service manifest does not look complete if i compare it to another server, where the kvmadm installation is working

# svccfg export kvm
<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='manifest' name='export'>
  <service name='system/kvm' type='service' version='0'>
    <instance name='foo' enabled='true'>
      <property_group name='kvmadm' type='application'>
        <propval name='disk0_cache' type='astring' value='none'/>
        <propval name='disk0_boot' type='boolean' value='true'/>
        <propval name='disk0_disk_size' type='astring' value='50G'/>
        <propval name='disk0_disk_path' type='astring' value='rpool/zones/foo/kvm0'/>
        <propval name='disk0_model' type='astring' value='virtio'/>
        <propval name='disk0_index' type='count' value='0'/>
        <propval name='disk1_cache' type='astring' value='none'/>
        <propval name='disk1_disk_path' type='astring' value='/export/iso/CentOS-7-x86_64-Minimal-1511.iso'/>
        <propval name='disk1_index' type='count' value='1'/>
        <propval name='disk1_model' type='astring' value='ide'/>
        <propval name='disk1_media' type='astring' value='cdrom'/>
        <propval name='nic0_index' type='count' value='0'/>
        <propval name='nic0_nic_name' type='astring' value='vnicfoo0'/>
        <propval name='nic0_over' type='astring' value='switch0'/>
        <propval name='nic0_model' type='astring' value='virtio'/>
        <propval name='nic1_over' type='astring' value='e1000g0'/>
        <propval name='nic1_index' type='count' value='1'/>
        <propval name='nic1_model' type='astring' value='virtio'/>
        <propval name='nic1_nic_name' type='astring' value='kvmfoo0'/>
        <propval name='serial0_serial_name' type='astring' value='console'/>
        <propval name='serial0_index' type='count' value='0'/>
        <propval name='vcpus' type='count' value='4'/>
        <propval name='vnc' type='astring' value='socket'/>
        <propval name='hpet' type='boolean' value='false'/>
        <propval name='ram' type='count' value='2048'/>
        <propval name='boot_order' type='astring' value='cd'/>
        <propval name='uuid' type='astring' value='ed228d46-7862-4a43-93dd-c0ad975fa930'/>
        <propval name='time_base' type='astring' value='utc'/>
      </property_group>
      <property_group name='general' type='framework'>
        <propval name='complete' type='astring' value='foo'/>
      </property_group>
    </instance>
  </service>
</service_bundle>

trying to create a vm (apparently that needs its own zone) on openindiana hipster, but there is no brand named 'lipkg', only the default 'ipkg' and solaris10 etc.

'lipkg' is omnios's brand for linked images - tried to use 'ipkg' instead but that wants to install the whole set of packages - hardly needed for a zone/vm running a different os.

man page didn't help a lot. am i missing the documentation on this?

thanks,
bamdad

kvmadm list vm1 > vm2.json
change a few things
kvmadm create -f vm2.json vm2
changes applied to vm1

At the very least: warn if overwriting on create.
But It's probably better to have the 2nd parameters over write the vmname when passing a file to create.

A debug option that would just print the qemu command instead of running it.
Should help with debugging some oddities.

A field in the config that is directly passed to qemu as is.

Can be great for the odd one of tweaks a vm needs.

I hope someone there will recognize the problem after reviewing this error received when building
kvmadm on openinidana/hipster
root # /usr/bin/gmake install
Making install in thirdparty
gmake[1]: Entering directory '/usr/local/src/vcs/git/kvmadm/thirdparty'
GEN all-local
gmake[2]: Entering directory '/usr/local/src/vcs/git/kvmadm/thirdparty'
GEN touch
Found Illumos::Zones 0.1.4 which doesn't satisfy == 0.1.2.
Found Data::Processor 0.4.3 which doesn't satisfy == 0.4.2.
gmake[2]: *** [Makefile:359: touch] Error 123
gmake[2]: Leaving directory '/usr/local/src/vcs/git/kvmadm/thirdparty'
gmake[1]: *** [Makefile:356: all-local] Error 2
gmake[1]: Leaving directory '/usr/local/src/vcs/git/kvmadm/thirdparty'
gmake: *** [Makefile:404: install-recursive] Error 1

If a vm does not support acpi, the service can get stuck and make the host shutdown stop.
Having a timeout of say 90 seconds and then killing the process would solve this.

man kvmadm shows the manpage properly formatted
kvmadm man does not format it properly

(Using OmniOS)

Been playing with it a bit, some things seem to be missing that are rather useful. I'm opening a few feature requests :) But very nice work so far!

Allow for sockets for vnc perhaps providing a path and option 'sock' parameter to make it use a default path.

-vnc unix:/var/kvm/{vmname}_vnc.sock

To export the socket for clients to use (I usually forward this over ssh to a different box)

socat tcp-listen:5900,reuseaddr,fork unix-client:/var/kvm/{vmname}_vnc.sock

cpu type does not take tweaked cpus
e.g. qemu64,+ssse3,+sse4.1,+sse4.2 (selectively exposing sse)

boot false does not work.

kvmadm list usgw
..
"boot" : "false",
"disk_path" : "/dev/zvol/rdsk/fast/vm-volatile/us-gw/swap",

kvmadm show-cmd usgw
..
-drive file=/dev/zvol/rdsk/fast/vm-volatile/us-gw/swap,if=virtio,media=disk,index=1,cache=none,boot=on

The import using file test01.json (see below) failed, because rpool/zones was not mounted to /zones but to /rpool, i.e. not to where it was expected.
In the first place the zonepath did not exist actually, but kvmadm offered to correct it.
However, this also failed due to the wrong path (non-existing /zones directory).
After that kvm complained about a missing repository.db in /zones/test01/root/...

This problem still existed when trying to import test01.json again.

{
   "test01" : {
      "disk" : [
         {
            "index" : "0",
            "boot" : "true",
            "disk_path" : "rpool/zones/test01/kvm0",
            "model" : "virtio",
            "disk_size" : "50G",
            "cache" : "none"
         },
         {
            "index" : "1",
            "media" : "cdrom",
            "disk_path" : "/export/iso/CentOS-7-x86_64-Minimal-1511.iso",
            "model" : "ide",
            "cache" : "none"
         }
      ],
      "serial" : [
         {
            "index" : "0",
            "serial_name" : "console"
         }
      ],
      "nic" : [
         {
            "index" : "0",
            "over" : "e1000g0",
            "nic_name" : "kvmtest01",
            "model" : "e1000"
         },
         {
            "index" : "1",
            "over" : "switch0",
            "nic_name" : "vnictest01",
            "model" : "e1000"
         }
      ],
      "zone" : {
         "zonepath" : "/zones/test01",
         "bootargs" : "",
         "hostid" : "",
         "brand" : "lipkg",
         "fs-allowed" : "",
         "scheduling-class" : "",
         "pool" : "",
         "autoboot" : "false",
         "limitpriv" : "default",
         "ip-type" : "exclusive",
         "fs" : [
            {
               "special" : "/etc/opt/kvmadm/myvncpw",
               "options" : "[ro,nodevices]",
               "type" : "lofs",
               "dir" : "/etc/opt/kvmadm/myvncpw"
            }
         ]
      },
      "vnc" : "socket",
      "uuid" : "ed228d46-7862-4a43-93dd-c0ad975fa930",
      "hpet" : "false",
      "boot_order" : "cd",
      "vcpus" : "4",
      "vnc_pw_file" : "/etc/opt/kvmadm/myvncpw",
      "time_base" : "utc",
      "ram" : "2048"
   }
}

Here the error:

root@tron:/root# /opt/kvmadm-0.11.0/bin/kvmadm import test01.json ->
zvol rpool/zones/test01/kvm0 does not exist. creating it...
svc.configd: Fatal error: /zones/test01/root/etc/svc/repository.db:
integrity check failed. Details in /etc/svc/volatile/db_errors
svccfg: Repository server failed (exit 102).
setting up system/kvm within zone. this might take a while...
svccfg: Pattern 'svc:/system/kvm' doesn't match any instances or services
svc.configd: Fatal error: /zones/test01/root/etc/svc/repository.db:
integrity check failed. Details in /etc/svc/volatile/db_errors
svccfg: Repository server failed (exit 102).
svc.configd: Fatal error: /zones/test01/root/etc/svc/repository.db:
integrity check failed. Details in /etc/svc/volatile/db_errors
svccfg: Repository server failed (exit 102).
ERROR: cannot add 'system/kvm'
Do you want to retry [Y/n]? y
svc.configd: Fatal error: /zones/test01/root/etc/svc/repository.db:
integrity check failed. Details in /etc/svc/volatile/db_errors
svccfg: Repository server failed (exit 102).
setting up system/kvm within zone. this might take a while...
svccfg: Pattern 'svc:/system/kvm' doesn't match any instances or services
svc.configd: Fatal error: /zones/test01/root/etc/svc/repository.db:
integrity check failed. Details in /etc/svc/volatile/db_errors
svccfg: Repository server failed (exit 102).
svc.configd: Fatal error: /zones/test01/root/etc/svc/repository.db:
integrity check failed. Details in /etc/svc/volatile/db_errors
svccfg: Repository server failed (exit 102).
ERROR: cannot add 'system/kvm'
Do you want to retry [Y/n]? n

kvmadm list also produced the same error message.

It's probably a good idea to add cache=none for zvol backed disks.

Enabling usb and adding a usb pointer device of type tablet -usb -usbdevice tablet seems to improve the mouse pointer experience over vnc for some operating systems. Mostly windows xp from my experience.

From what I can tell $nic->{index} is used to set the vlan of the nic. This is a bit confusing.

I some vm's can have multiple nics in the same/multiple vlans.

There seems to be some reference to it in the code though, but there is no way to actually set it.

Usage: masking certain CPU options/compatibility

It would be nice to have a console, monitor and vnc command to connect to the serial socket, monitor socket and optionally vnc socket.

I suck at perl but I wrote a quick hackup for ksh93.

#!/bin/sh

## paths
MINICOM=/opt/obd/bin/minicom
NETCAT=/usr/bin/nc
SOCAT=/opt/obd/bin/socat

KVM_RUN_PATH=/var/run/kvm

## init
# check our paths
HAVECON=1
HAVEVNC=1
[ ! -e ${MINICOM} ] && [ ! -e ${NETCAT} ] && HAVECON=0
[ ! -e ${SOCAT} ] && HAVEVNC=0

# actions
case $1 in
  console)
    VM=$2

    if [ ${HAVECON} -eq 0 ]; then
       echo "ERROR: Please install minicom or netcat!"
       exit 1
    fi

    if [ -z ${VM} ]; then
      echo "ERROR: Please provide a kvm_name!"
      exit 2
    fi

    SOCK=
    [ -S "$KVM_RUN_PATH/$VM.ttyA" ] && SOCK="$KVM_RUN_PATH/$VM.ttyA"
    [ -S "$KVM_RUN_PATH/$VM.ttya" ] && SOCK="$KVM_RUN_PATH/$VM.ttya"
    [ -S "$KVM_RUN_PATH/$VM.com0" ] && SOCK="$KVM_RUN_PATH/$VM.com0"
    [ -S "$KVM_RUN_PATH/$VM.console" ] && SOCK="$KVM_RUN_PATH/$VM.console"


    if [ -z ${SOCK} ]; then
      echo "ERROR: Could not find a socket named console, ttya or com0 for $VM!"
      exit 3
    else
      if [ -e ${MINICOM} ]; then
        ${MINICOM} -8 -m -c on -D unix\#${SOCK}
      else
        ${NETCAT} -U ${SOCK}
      fi
    fi
  ;;
  monitor)
    VM=$2

    if [ ${HAVECON} -eq 0 ]; then
       echo "ERROR: Please install minicom or netcat!"
       exit 1
    fi

    if [ -z ${VM} ]; then
      echo "ERROR: Please provide a kvm_name!"
      exit 2
    fi

    SOCK=
    [ -S "$KVM_RUN_PATH/$VM.monitor" ] && SOCK="$KVM_RUN_PATH/$VM.monitor"


    if [ -z ${SOCK} ]; then
      echo "ERROR: Could not find a socket named monitor for $VM!"
      exit 3
    else
      if [ -e ${MINICOM} ]; then
        ${MINICOM} -8 -m -c on -D unix\#${SOCK}
      else
        ${NETCAT} -U ${SOCK}
      fi
    fi
  ;;
  vnc)
    VM=$2
    PORT=$3

    [ -z ${PORT} ] && PORT=1

    if [ ${HAVEVNC} -eq 0 ]; then
      echo "ERROR: Please install socat!"
      exit 1
    fi

    if [ -z ${VM} ]; then
      echo "ERROR: Please provide a kvm_name!"
      exit 2
    fi

    SOCK=
    [ -S "$KVM_RUN_PATH/$VM.vnc" ] && SOCK="$KVM_RUN_PATH/$VM.vnc"


    if [ -z ${SOCK} ]; then
      echo "ERROR: Could not find a socket named vnc for $VM!\nThe KVM could not have a display configured or already be listening for connection."
      exit 3
    else
      echo "Listening on $(expr 5900 + ${PORT}) for ${VM} ..."
      ${SOCAT} tcp-listen:$(expr 5900 + ${PORT}),reuseaddr,fork unix-client:/var/run/kvm/dyon.vnc
    fi
  ;;
  *)
    echo "Usage: $(basename $0) <command> <kvm_name> [port]"
    echo "  console :: try to open serial console for <kvm_name>"
    echo "  vnc     :: try to open vnc port for <kvm_name>"
    echo "  monitor :: try to open qemu monitor for <kvm_name>"
  ;;
esac

exit 0

I'm using OmniOS v11 r151014.
After successful installation

./configure --prefix=/opt/kvmadm-0.11.0  --enable-svcinstall=/var/svc/manifest/site
gmake
gmake install

I did

svccfg import -v /var/svc/manifest/site/system-kvm.xml

which returned no errors.

However the service did not register correctly, i.e.

svcs -a | grep kvm

does not return any results.

instead of an error message about the missing argument

reaper# kvmadm import
Use of uninitialized value $fileName in open at /opt/bin/kvmadm line 122.
Use of uninitialized value $fileName in concatenation (.) or string at /opt/bin/kvmadm line 122.
ERROR: open file :No such file or directory
zsh: exit 2     kvmadm import

Please make it possible to add serial ports to a vm. (I see a monitor socket is already created 👍 )

-chardev socket,id=serial0,path=/vms/hosts/${VMNAME}/run/console.sock,server,nowait \
-serial chardev:serial0 \
-chardev socket,id=serial1,path=/vms/hosts/${VMNAME}/run/ttyb.sock,server,nowait \
-serial chardev:serial1 \

I usually add one or two, serial0 is very handy to provide a console on. Again you can connect to those using socat or minicom or other tools.

minicom -8 -m -c on -D unix\#/vms/hosts/${VMNAME}/run/console.sock

See title :)

Hi

We use 128k blocksize for ssd disks ... it would be great if this could be configured in the
kvmadm config property.

"disk_blocksize" : "128K", or "volblocksize" : "128K"

support custom templates for creating new kvm instances

add support to kvmadm to manage KVMs running in individual zones

• a 'clone' option will clone an existing KVM instance (i.e. cloning zone root dataset if KVM is set up as zone)

The vnc option should not be manditory. Certain OS'ses like OpenBSD/FreeBSD and linux are prefectly happy with no vga display but just a serial port.

If no vnc is present it should fallback to -nographic or maybe -nographic and a single serial port.

By default -no-hpet is used and can't be removed. Having a toggle property set to false by default would be better.

Hi

  {
        "index" : "1",
        "disk_path" : "/dev/zvol/rdsk/fast/vm/satellite/z-drive",
        "model" : "virtio",
        "disk_size" : "31G"
     },

if the disk has no boot value there is an error in the log file:
Use of uninitialized value in string eq at /opt/oep/bin/../lib/KVMadm/Config.pm line 331.
there should be a default value for boot.

..
. ($disk->{boot} eq 'true' ? ',boot=on' : ''));
..

cheers Manuel

hi

I did try the host name us-gw and kvmadm ends up with the error

[ Mar 31 15:04:39 Executing start method ("/opt/oep/bin/system-kvm start"). ]
Use of uninitialized value $kvmName in concatenation (.) or string at /opt/oep/bin/../lib/KVMadm/Config.pm line 243.

As per qemu's help
-smp n[,maxcpus=cpus][,cores=cores][,threads=threads][,sockets=sockets]

lstopo from the hwloc package confirms qemu is not parsing it correctly as set by kvmadm.

kvmadm does not set the first mandatory 'n' portion.
-smp sockets=1,cores=1,threads=2 vs -smp 2,sockets=1,cores=2,threads=1

It's impossible to set the vcpu config option with the correct string:
ERROR: value '2,sockets=1,cores=1,threads=2' for property 'vcpus' not correct.

To calculate the correct number: socket_cores_threads
It's probably better to calculate it vs having the user specify it.

Hey,

It looks like it's not possible to pass complex CPU topology options.

e.g. -smp sockets=1,cores=4,threads=2 for 8 vcpu's under windows but still requiring only a single socket license ;)

http://wiki.qemu.org/download/qemu-doc.html#vnc_005fsec_005fcertificate

Also add a vnc_cert_file option to enable TLS?

Hi, is there any reason why a Windows VM shutting down from within Windows leaves the specific svc in maintenance state?
I always need to disable the specific svc to enable it again.

Gabriele

It would be nice to (offer) to delete vnics automatically created when a vm is delete.

When trying to execute kvmadm, it fails with the following errors:

root@omnios-test:/opt/kvmadm-0.9.0-rc2# bin/kvmadm
Global symbol "@MON_INFO" requires explicit package name at bin/kvmadm line 264.
Global symbol "@MON_INFO" requires explicit package name at bin/kvmadm line 267.
Execution of bin/kvmadm aborted due to compilation errors.

root@omnios-test:/opt/kvmadm-0.9.0-rc2# perl --version

This is perl 5, version 16, subversion 1 (v5.16.1) built for i86pc-solaris-thread-multi-64int
[..]

This is on a fresh OmniOS r151014 install, only added gnu-make package.

When deleting the VM "test2" with purging everything the following error messages are produced:

kvmadm delete --purge=vnic,zvol,zone test2
Are you sure you want to uninstall zone test2 (y/[n])? y
Are you sure you want to delete zone test2 (y/[n])? y
cannot open 'rpool/zones/test2/kvm0': dataset does not exist
ERROR: cannot destroy zvol 'rpool/zones/test2/kvm0'

Obviously the zone deletion also deletes the image file, which kvmadm does not recognize.
The KVM image is within the zone path, so the ZFS gets destroyed as well when the zone is purged:

rpool/zones/test2             10.8G   373G    19K  /zones/test2
rpool/zones/test2/ROOT         515M   373G    19K  legacy
rpool/zones/test2/ROOT/zbe     515M   373G   515M  legacy
rpool/zones/test2/kvm0        10.3G   384G     8K  -

Recommended solution: Delete each zone's dataset separately.
Also, it might be possible that only the zone might be purged, not KVM image.

"make install" / "gmake install" fails with the following message:

if [ "xyes" = "xyes" ]; then svccfg import /opt/kvmadm-0.9.0-rc2/share/smf/system-kvm.xml; fi
svccfg: Unable to stat file /opt/kvmadm-0.9.0-rc2/share/smf/system-kvm.xml.  No such file or directory

/opt/kvmadm-0.9.0-rc2 only contains bin and lib directories.
This is on a fresh OmniOS r151014 install, only added gnu-make package.

Having volumes with 8k volblocksize on 4k drives is rather inefficient. It can lead to up to 50% space loss ... since blocks are compressed, they will always be less than 8k but often more then 4k ... with 4k drives the allocation size is 4k.

We found (not excessively tested) that things look much better with 128k volblock size for such volumes. Space use is similar to the 8k volblocks on 512 disk blocks ...

There will probably be a performance penalty for small read/writes ... but we also found the large reads at least are up to twice as fast as with 8k volblocks (probably due to the lower amount of wasted space)

see also openzfs/zfs/issues/548

Also pass this to the vnic creation to have it create with the specified MTU.

I have a number of zones on my server which are shutdown. Running kvmadm list produces errors as it tries to iterate the FMRIs within the zones. Since they are not even mounted, the repository is not accessible. Can the recent svcs -Z be used instead?

reaper# kvmadm list
svc.configd: Fatal error: /data/zone/hosting/root/etc/svc/repository.db: integrity check failed. Details in /etc/svc/volatile/db_errors
svccfg: Repository server failed (exit 102).
... repeated for each zone that is not running ...

nic_tag is used in SmartOS to indicate which nic it should create the vnic for the zone/vm. Rather confusing. So perhaps call it nic_name?

PS: I added kvmadm to my OmniOS repository: http://omnios.blackdot.be, this should make it more accessable for some people.

Woops just remembered (by running into it) why I use minicom for the vt100 emulation.

1. passwords are echo'ed using kvmcli's console
2. terminal PS1 strings are garbled ( at least for smartos )
3. ctrl+xyz does not seem to be forwarded, ctrl+c terms the connection but not say a running command.

Can't find any resource by google.
Can I have an example?

kvmadm should ignore lx zones if present

When creating an incorrect json and answering 'Y' to retry, it reverts to the template!