Code Monkey home page Code Monkey logo

Comments (8)

sebgl avatar sebgl commented on May 17, 2024 2

After discussion with @nkvoll on zoom, we agreed on:

apiVersion: deployments.k8s.elastic.co/v1alpha1
kind: Stack
metadata:
    labels:
        controller-tools.k8s.io: "1.0"
    name: stack-sample
spec:
    version: "6.4.2"
    elasticsearch:
        setVmMaxMapCount: true
        topologies:
        # master only
        - nodeTypes:
            master: true
            data: false
            ingest: false
          resources:
            limits:
                mem: 1Gi
                disk: 1GB
                cpu: 800m
          nodeCount: 1
        # data only
        - nodeTypes:
            master: false
            data: true
            ingest: false
          resources:
            limits:
              mem: 8Gi
              disk: 1TB
              cpu: 800m
          nodeCount: 3

from cloud-on-k8s.

sebgl avatar sebgl commented on May 17, 2024 1

I think if we introduce that later on, we can keep backward compatibility.
Ie. if a "DNT" reference is specified, use it, otherwise, use the given settings.
Anyway I tend to agree we don't need it for now.

from cloud-on-k8s.

nkvoll avatar nkvoll commented on May 17, 2024 1

Something like this?

apiVersion: deployments.k8s.elastic.co/v1alpha1
kind: Stack
metadata:
  labels:
    controller-tools.k8s.io: "1.0"
  name: stack-sample
spec:
  version: "6.4.2"
  elasticsearch:
    setVmMaxMapCount: true
    nodes:
      # only one of flat/toplogies allowed
      # flat caters to the simplest of cases
      flat:
        replicas: 1
      # topologies caters to the DNT-style, more advanced setups
      topologies:
        # master only
        - nodeTypes:
            master: true
            data: false
            ingest: false
          resources:
            limits:
              mem: 1Gi
              disk: 1GB
              cpu: 800m
          replicas: 1
        # data only
        - nodeTypes:
            master: false
            data: true
            ingest: false
          resources:
            limits:
              mem: 8Gi
              disk: 1TB
              cpu: 800m
          replicas: 3

I nested resources.limits slightly (matching the pod api), nested nodeTypes so they are namespaced a little better, used replicas instead of count, as replicas are more commonly used terminology in K8sland

from cloud-on-k8s.

sebgl avatar sebgl commented on May 17, 2024

A proposal on the CRD format:

Add an array under nodes, one item per "node type".
A node type represents the node topology (master/data/ingest, capacity, disk, etc.) and the number of nodes to create for that topology.

apiVersion: deployments.k8s.elastic.co/v1alpha1
kind: Stack
metadata:
  labels:
    controller-tools.k8s.io: "1.0"
  name: stack-sample
spec:
  version: "6.4.2"
  elasticsearch:
    setVmMaxMapCount: true
    nodes:
      # master only
      - master: true
        data: false
        ingest: false
        ram: 1GB
        disk: 10GB
        count: 1
      # data only
      - master: false
        data: true
        ingest: false
        ram: 8GB
        disk: 1TB
        count: 3  # create 3 nodes like that
      # ingest only
      - master: false
        ingest: true
        data: false
        count: 3
      # coordinator
      - master: false
        ingest: false
        data: false
        count: 1
      # default to master+data+ingest
      - count: 1

Default values:

count: 1
ram: 1GB
master: true
data: true
ingest: true
disk: 10GB

from cloud-on-k8s.

marclop avatar marclop commented on May 17, 2024

Proposal looks good, the only thing I'd think about is keeping a consistency between the units that we've got in the stack and the ones we've got in our CRD. See https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-memory.

Maybe in the future it'd be worth declaring other CRDs with the ES DNT configs?

from cloud-on-k8s.

sebgl avatar sebgl commented on May 17, 2024

Oh yes, better use the k8s standard units 👍

Maybe in the future it'd be worth declaring other CRDs with the ES DNT configs?

You mean we have a CRD dedicated to represent a node type, and we reference this node type in the stack CRD?

# dnt.yaml
name: my-node-type
data: true
master: true
# stack.yaml
nodes:
    - dnt: my-node-type
      count: 3

I guess it's nice when you want to manage many clusters. And it's too complex when you want to manage a few clusters only.

from cloud-on-k8s.

marclop avatar marclop commented on May 17, 2024

Yeah I think it's a nice way to reduce your declarations in your stack and make them easier to read. And it's like what you say, if you've got multiple stacks you want to kinda have this centrally managed definition. But it can be done later on

from cloud-on-k8s.

nkvoll avatar nkvoll commented on May 17, 2024

@sebgl I believe this can be closed now. Reopen otherwise :)

from cloud-on-k8s.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.