xtruder / kubenix Goto Github PK

Hi!

I'm pretty new to NixOS/Nix and I can't say for certain I'll be able to use this project. It would be awesome to give detailed instructions for installation/usage or even host a full example living in a different repo to highlight how it's used.

Thank you!

Description

Kubenix is in stage of refactoring due all the things we discovered at @GateHubNet while we were using kubenix for complex production deployments. We made kubenix public even before, so someone could reuse ideas, but it was still highly specialized tool for our needs. This refactoring will make kubenix generally available, but will also introduce breaking changes.

For work in progress branch please look into: https://github.com/xtruder/kubenix/tree/kubenix-2.0

New features

• Improved and reimplemented submodule system with versioning support
• Pre-generated kubernetes module definitions with explicit API versioning support
• Pre-generated istio module definitions
• Support for k8s 1.11,1.12,1.13
• Validation for io.k8s.apimachinery.pkg.api.resource.Quantity and similar types
• Helm integration
• Initial testing framework
• Improved unit tests
• Improved e2e tests
• CI integration (travisci and hercules-ci)
• More examples
• Usage documentation
• Submodule documentation
• Generated documentation for all kubernetes options
• Legacy kubenix 1.0 support

Coming from a kubernetes background and have little exposure to Nix, what would Kubenix do for me? The term 'resource builder' is not really evident to me.

The issue I'm seeing seems only happens when using a module in a separate file, so it might be me misunderstanding modules. Or maybe it's a bug in kubenix because I'm not doing anything unusual I think.

Here's a repro for the error I'm seeing:

# ==> repro.nix <==
{
  nixpkgs ? import <nixpkgs> {},
  kubenix ? import (nixpkgs.fetchgit {
    url = "https://github.com/xtruder/kubenix/";
    rev = "611059a329493a77ec0e862fcce4671cd3768f32";
    sha256 = "1lmmzb087ahmx2mdjarbi52a9424qczhzqbxrvcrg11cbmv9b191";
  }) {}
}:
rec {
  config = (kubenix.evalModules {
    modules = [
      ./module.nix { inherit kubenix; }
    ];
  }).config;

  generated = config.kubernetes.generated;
  result = config.kubernetes.result;
}

# ==> module.nix <==
{ config, lib, pkgs, kubenix, ... }:

with kubenix.lib;

let exampleYaml = pkgs.writeText "namespace.yaml" ''
apiVersion: v1
kind: Namespace
metadata:
  name: default
'';
in
{
  imports = with kubenix.modules; [
    k8s
  ];

  kubernetes.imports = [
    exampleYaml
  ];
}

If I try to build that kubernetes object, I get:

$ nix-build repro.nix
error: The option `kubenix.buildResources' defined in `<unknown-file>' does not exist.
(use '--show-trace' to show detailed location information)

I think it's related to modules because if I shove it all in one file, like so, then it builds correctly:

{
  nixpkgs ? import <nixpkgs> {},
  kubenix ? import (nixpkgs.fetchgit {
    url = "https://github.com/xtruder/kubenix/";
    rev = "611059a329493a77ec0e862fcce4671cd3768f32";
    sha256 = "1lmmzb087ahmx2mdjarbi52a9424qczhzqbxrvcrg11cbmv9b191";
  }) {}
}:
let exampleYaml = nixpkgs.writeText "namespace.yaml" ''
apiVersion: v1
kind: Namespace
metadata:
  name: default
'';
in
rec {
  config = (kubenix.evalModules {
    modules = [
      {
        imports = with kubenix.modules; [
          k8s
        ];

        kubernetes.imports = [
          exampleYaml
        ];
      }
    ];
  }).config;

  generated = config.kubernetes.generated;
  result = config.kubernetes.result;
}

One other thing to note: after poking through the stacktrace, I realized the inscrutable error may have actually been trying to say "kubenix.project wasn't set".

If I delete the two lines in modules/k8s.nix that reference config.kubenix.project (

My totally uneducated understanding is that kubenix.evalModules is breaking the default for project = "kubenix" somehow, and when I don't split the module into a separate file, I don't see this issue because I guess I'm using the kubenix from the outer scope anyway? Yeah, I'm not totally sure.

If you specify default secret claim name in the service deployment module, then the defined claim name does not overwrite the default claim name.

I think I am seeing flakiness with the kube node getting spun up during test runs. I don't know enough to give more information, the output of nix build -f ./. test-script rapidly flashes incomplete information. It looks like its repeatedly running ("kubectl get node kube.my.xzy | grep -w Ready"). Any guidance to getting more info on this?

I see that there's support for loading resources from yaml files, however I've wanted to support using kustomize which basically generates a stream of yaml documents - this isn't supported by remarshal but is supported by yq via jq:s "-s" option so I've used that to enable loading of kustomizations.

Anyway, I've been toying around with switching completely from kustomize to kubenix but encountered some problems when trying to deploy cert-manager which defines its own types that I need to use from kubenix... it doesn't seem obvious to me how one would load such types from yaml such that they become available within kubenix.

Since there's very little docs atm I guess I might be missing something here but my question is if it is possible to load crd:s from yaml such that they become available as custom types within kubenix or if it is something you've thought about at all?

I really like what you're doing here, it's starting to look really awesome! Thanks for this!

hi @xtruder

kubenix is brilliant, it makes writing kubernetes resources so much more pleasant than helm.

Follows, a dump of my experience with the library:

No derivation

It would be nice if the output was just a nix object. With nix-instanciate --eval --strict --json it should be possible to generate the same output but also not have anything written in the /nix/store. This is important when generating secret resources for example since the store is world-readable.

Fixed by #7

Resource names and pluralization

It took me a while to figure out that resources are defined automatically from the swagger spec. Ingres is under "ingresses" and "PersistentVolume" is under "persistentVolumes".

Not a lot of docs

I had to dig quite a bit to figure out how everything comes together. For example there is an option to choose the kubernetes schema under config.kubernetes.version = "1.11" and which default is "1.9".

If I hadn't been super comfortable with the nix language I would have probably given up.

TODO: submit PR to improve the doc

Hey!

It seems that 1.18 is the last supported version of kubenix... would be great if this project gets continued support.

Issue

vscode-vim extension does not work for some reason, when installed using nix

The k8s ecosystem is starting to catch up with it's deficiencies over nix.

https://kubectl.docs.kubernetes.io/guides/app_deployment/publishing_bases/

This means, it wold be tremendously useful if kubenix could hook into White Box Application concept, yet allow users to customize according to accustomed nix's powers.

Let's say we have "io.k8s.api.core.v1.ResourceRequirements" submodule from https://github.com/xtruder/kubenix/blob/master/modules/generated/v1.18.nix
And we want to override a type for limits & requests from (types.nullOr (types.attrsOf types.str)) to (types.nullOr (types.attrsOf types.oneOf [types.int types.str]));

How could we write this override?

This allows your deployment to refer to a specific configmap, and also means that if the configmap changes that the deployment gets redeployed. Also if deployment fails then the previous configmap still exists on rollback.

e.g.:

  kubernetes.resources.deployments.nginx = {
    spec = {
      replicas = 10;
      selector.matchLabels.app = "nginx";
      template = {
        metadata.labels.app = "nginx";
        spec = {
          securityContext.fsGroup = 1000;
          containers.nginx = {
            image = config.docker.images.nginx.path;
            imagePullPolicy = "IfNotPresent";
            volumeMounts."/etc/nginx".name = "config";
          };
          volumes.config.configMap.name = config.kubernetes.resources.nginx-config.path;
        };
      };
    };
  };

  kubernetes.resources.configMaps.nginx-config.data."nginx.conf" = ''
    user nginx nginx;
    daemon off;
    error_log /dev/stdout info;
    pid /dev/null;
    events {}
    http {
      access_log /dev/stdout;
      server {
        listen 80;
        index index.html;
        location / {
          root /var/lib/html;
        }
      }
    }
  '';

And in this example config.kubernetes.resources.nginx-config.path will evaluate to:

"zyy26wn23ac3ivl8lkc6wv2bdyjz99h1-nginx-config"

named after the nix-store path with the /nix/store part stripped.

code-server info:
code-server: v3.12.0
VS Code: v1.60.0
Commit: 4cd55f94c0a72f05c18cea070e10b969996614d2
Date: 2021-09-15T19:34:37Z
Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36

I made a ClusterRoleBinding object and tried to send it to a 1.11.5 cluster. I get this error:

error: unable to recognize no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1alpha1"

This is reproducible across four major k8s vendors, all using 1.11.x clusters. On all clusters, the same fix works:

sed -e s/alpha/beta/ </nix/store/….json >fixed.json
kubectl apply -f fixed.json