summary

I have several incus servers, and want to create containers from one central place (i.e. my laptop). therefore it needs to work via https and incus API. I also need to use my PKI so that I can issue certificates to other users.

On Incus server, I placed CA cert in /var/lib/incus/server.ca and using incus config trust add-certificate command. See relevant links below.

I stored client cert/key and CA (client.crt, client.key, client.ca) on my laptop in ~/.config/incus/.

curl tests pass, see below.

issue

I think certificates and PKI are setup correctly, and deployed into incus server correctly. curl test works with these certs,

however does not work when I use terraform.

I tried concatenating (cat) client.ca into client.crt, still works for curl but still does not work for terraform.

incus_instance.instance: Creating...
╷
│ Error: Failed to retrieve Incus InstanceServer
│ 
│   with incus_instance.instance,
│   on main.tf line 1, in resource "incus_instance" "instance":
│    1: resource "incus_instance" "instance" {
│ 
│ Unable to create server client for remote "test-11": Unable to authenticate with remote server: not authorized

terraform code:

# provider.tf
terraform {
  required_providers {
    incus = {
      source = "lxc/incus"
      version = "0.1.1"
    }
  }
}

provider "incus" {
  generate_client_certificates = false
  accept_remote_certificate    = true
  remote {
    name = "test-11"
    scheme = "https"
    address = "192.168.1.11"
    default = true
  }
}

# main.tf
resource "incus_instance" "instance" {
  name = "testytest"
  image = "images:debian/bookworm/cloud"
  profiles = ["default"]
}

other relevant logs

terraform can access certs

$ inotifywait -m -e open ~/.config/incus/*
Setting up watches.
Watches established.
/home/invizus/.config/incus/servercerts/ OPEN test-11.crt
/home/invizus/.config/incus/client.crt OPEN 
/home/invizus/.config/incus/client.ca OPEN 
/home/invizus/.config/incus/client.key OPEN

curl works:

$ curl -s -k --cert ~/.config/incus/client.crt --cacert ~/.config/incus/client.ca \
--key ~/.config/incus/client.key https://192.168.1.11:8443/1.0 -X GET | jq .metadata.auth
"trusted"

Update: Just FYI curl works only when concatenating CA into client cert.

relevant links

https://discuss.linuxcontainers.org/t/how-to-add-a-certificate-to-incus-remotely/19549

https://linuxcontainers.org/incus/docs/main/authentication/#using-a-pki-system

Our Terraform provider should be published in the Terraform Registry, see Publishing Providers.

Open questions:

• @stgraber: To publish the package, we need to grant the Terraform Registry access to the lxc GitHub organization. We also need to create a signing key that will be used to publish the package. How would you like to proceed here?
• How should we organize the versioning and changelog from now on? Incus started with 0.0.1 and so we could do the same here, but maybe start with 1.0.0?

Hello !

In order to continue the improve the provider, having support of network forward feature could be nice.

Thanks !

I think we want to perform the following renaming to have things be consistent:

• incus_volume => incus_storage_volume
• incus_volume_copy => incus_storage_volume_copy
• incus_snapshot => incus_instance_snapshot
• incus_publish_image => incus_image_publish
• incus_cached_image => incus_image

We're luckily early enough in Incus and this provider that we can likely get away with a rename followed by a minor version bump.

Hi, first of all I'd like to thank all the maintainers for this provider, it's awesome.
Recently, I've been trying to automate the creation of a kubernetes cluster (k3s, RKE2, k0s and others) on Incus VMs and everything works well enough except for when I try to run terraform apply a second time after installing a kubernetes distribution:

Kubernetes components create a bunch of networks, bridges and so on and these addresses show up in the Incus webui as well as in the CLI. Only problem is, the ipv4_address and ipv6_address exported values from my incus_instance resources change from a private IPv4 on the same nework I run terraform from to a private IP only accessible from the networks created by Kubernetes.

Example:

Upon running a second terraform apply, the IPv4 changes

Changes to Outputs:
  ~ instance-ip = "10.127.0.41" -> "10.42.0.0"
null_resource.kubeconfig: Destroying... [id=5068605674949049145]
null_resource.kubeconfig: Destruction complete after 0s
null_resource.get-kubeconfig: Creating...
null_resource.get-kubeconfig: Provisioning with 'local-exec'...
null_resource.get-kubeconfig (local-exec): Executing: ["/bin/sh" "-c" "ssh -o 'StrictHostKeyChecking no' [email protected] sudo cat /etc/rancher/rke2/rke2.yaml > temp-kubeconfig"]
null_resource.get-kubeconfig: Still creating... [10s elapsed]
...
[Times out eventually because 10.42.0.0 is not reachable from my computer]

+-----------+---------+-----------------------+--------------------------------+-----------------+-----------+
|   NAME    |  STATE  |         IPV4          |              IPV6              |      TYPE       | SNAPSHOTS |
+-----------+---------+-----------------------+--------------------------------+-----------------+-----------+
| instance0 | RUNNING | 10.42.0.0 (flannel.1) | 2001::e91 (enp5s0) | VIRTUAL-MACHINE | 0         |
|           |         | 10.127.0.41 (enp5s0)  |                                |                 |           |
+-----------+---------+-----------------------+--------------------------------+-----------------+-----------+

In this example, the address I'm trying to reach instance0 from is 10.127.0.41 and that's what ipv4_address returns at first, but upon installing RKE2, ipv4_address returns 10.42.0.0 which is not accessible from my computer.

Now what I think is happening is something is grabbing the first address (index 0) from a list of IP addresses given by Incus agent and rolls with it. It's not a problem when the VM/container only has one address, but could become incorrect when it has more than one.

Unfortunately, it seems that that address is not always the one we're interested in.
Maybe ipv4_addresses could be changed to a map of interface names to IPs? enp5s0 -> 10.127.0.41, flannel.1 -> 10.42.0.0, and so on?

OK. The typo is one thing (occurs four times).

The real issue is that I'm getting this error for a simple example. I have added an alias to a local image. Then
this example:

resource "incus_instance" "c9" {
  name  = "c9"
  image = "local:centos/9-Stream/cloud/vm"
  type = "virtual-machine"

  config = {
    "boot.autostart" = true
    "security.secureboot" = false
  }

  limits = {
    cpu = 2
  }
  profiles = ["default", "user-kees-centos", "config-centos"]
}

The user is member of the incus group, not incus-admin.

incus_instance.c9: Creating...
╷
│ Error: Failed to retireve image info for instance "c9"
│ 
│   with incus_instance.c9,
│   on main.tf line 1, in resource "incus_instance" "c9":
│    1: resource "incus_instance" "c9" {
│ 
│ Image not found
╵

If I run this with an incus-admin user it succeeds.

terraform {
  required_providers {
    incus = {
      source = "lxc/incus"
      version = "~> 0.1.1"
    }
  }
  required_version = ">= 1.8"
}

provider "incus" {}

resource "incus_project" "project" {
  name = "myproject"

  config = {
    "features.profiles" = true
  }
}

resource "incus_profile" "default" {
  name = "default"
  project = incus_project.project.name

  device {
    type = "disk"
    name = "root"

    properties = {
      pool = "default"
      path = "/"
    }
  }
}

returns the next error

│ Error: Failed to create profile "default"
│ 
│   with incus_profile.default,
│   on main.tf line 21, in resource "incus_profile" "default":
│   21: resource "incus_profile" "default" {
│ 
│ Error inserting "default" into database: The profile already exists

The error is expected because incus always creates a default profile.
Of course, we can change the name of the profile and working without the default profile but in this case we cannot track the "default" dynamically created.
Import would not work because the profile is not known before the project creation.
I can remind that other providers (aws) uses data source with the default.
Would data for default profile (like data profile_default) fix the problem?

We should add a contribution guideline to the repository, inspired by CONTRIBUTING.md of the incus project.

Why it is helpful

• Standardization: Establishes a consistent process for submitting contributions.
• Clarity for contributors: Provides a clear roadmap for effective contributions.
• Improved collaboration: Promotes a collaborative and inclusive environment.
• Improve code quality: Enforces coding standards and best practices.
• Growing the community: Attracts diverse contributors, fostering community growth.

terraform {
  required_providers {
    incus = {
      source = "lxc/incus"
      version = "~> 0.1.1"
    }
  }
  required_version = ">= 1.8"
}

provider "incus" {}

resource "incus_network" "net" {
  name = "a-bridged-network"

  type = "bridge"

  config = {
    "ipv4.address" = "auto"
    "ipv4.nat" = "true"

    "ipv6.address" = "auto"
    "ipv6.nat" = "true"
  }
}

gives the next errors.

incus_network.net: Creating...
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to incus_network.net, provider "provider[\"registry.terraform.io/lxc/incus\"]" produced an unexpected new value: .config["ipv4.address"]: was
│ cty.StringVal("auto"), but now cty.StringVal("10.58.213.1/24").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to incus_network.net, provider "provider[\"registry.terraform.io/lxc/incus\"]" produced an unexpected new value: .config["ipv6.address"]: was
│ cty.StringVal("auto"), but now cty.StringVal("fd42:1c94:e5e9:4d18::1/64").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

though it is expected on incus side because the keyes ipv{4,6}.address are generated if we don't override them (auto as initial value)

It would be nice to have support for incus exec in the incus_instance resource.

I migrate my lxd infrastructure to incus. I use terraform to create network, profile and instance.
Some instance have a defined MAC address. With lxd provider, I use to configure this MAC address like this:

resource "incus_instance" "machine" {
...
  config =  {
            "volatile.eth0.hwaddr" = "00:16:3e:73:dc:45"
  }
...
}

But with incus provider I get this message: Config key cannot have "volatile." or "image." prefix. Got: "volatile.eth0.hwaddr".

Is some body can tell me how can I define this option?
thx

This error pops up if a user is "just" in the incus group, not in the incus-admin group.

The code in determineIncusDir checks if /var/lib/incus/unix.socket is writable, which it isn't for non admin users.

A workaround is to set envvar INCUS_SOCKET=/var/lib/incus/unix.socket.user

Hi,

I was looking at this project referenced at [1] and noticed that there's no way to manage remotes. Something along the lines of

resource "incus_remote" "remote" {
  name = "myremote"
  url = "https://example.invalid/"
}

Regards

1: https://discuss.linuxcontainers.org/t/ansible-guidance-receiving-errors-when-using-lxd-container-plugin/18734

It looks like there's still quite a bit of legacy wording around containers rather than the more generic terminology of instances (as apply to both containers and VMs).

In general we should be using instance everywhere and so only use container or virtual-machine in the cases where the particular feature only applies to just one of the two types.

I've noticed OpenTofu getting pretty confused about a volume.zfs.remove_snapshots=true config key appearing on a volume that's defined in terraform. That's because the config key was set on the parent storage pool and so propagated to the new volume.

That's normal and we definitely don't want terraform to try to rectify this by deleting and re-creating the volume as it attempted here :)

Instances can be created with an empty image, this is particularly useful for VMs.

Given:

resource "incus_storage_bucket" "this" {
  name = "bucket"

  pool = "default"

  config = {
    "size" = "100MiB"
  }
}

╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to incus_storage_bucket.this, provider "provider[\"registry.terraform.io/lxc/incus\"]"
│ produced an unexpected new value: .config: new element "block.filesystem" has appeared.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to incus_storage_bucket.this, provider "provider[\"registry.terraform.io/lxc/incus\"]"
│ produced an unexpected new value: .config: new element "block.mount_options" has appeared.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Workaround:

resource "incus_storage_bucket" "this" {
  name = "bucket"

  pool = "default"

  config = {
    "size" = "100MiB"
    "block.filesystem" : "ext4"
    "block.mount_options" : "discard"
  }
}

I have created a *.tf file, which contains creation of networks and some containers. When I try to create/destroy the infrastructure I get failed creating DHCP static allocation... or network currently in use... Issue is present for all dependencies on creates and destroy, projects, profiles, network, etc.

OpenTofu is now GA and uses its own registry where we should publish our provider: https://github.com/opentofu/registry/

I have not created a ~/.config/incus/config.yml and my config looks like this:

provider "incus" {
  generate_client_certificates = true
  accept_remote_certificate    = true

  remote {
    name    = "demeter"
    scheme  = "https"
    address = "192.168.123.15"
    token = "<redacted>"
    default = true
  }

  //config_dir = "/Users/jmaguire/.config/incus"
}

Running terraform apply creates a new keypair saved to $HOME/.config/incus directory structure in the local directory. Additionally, if I move the config directory to my home directory, it's ignored. This is because config.yml does not exist:

Note that os.ExpandEnv is only applied to the complete config.yml file name, not the configDir var assigned to the new config struct.

❯ terraform version
Terraform v1.6.6
on darwin_arm64
+ provider registry.terraform.io/lxc/incus v0.0.2