tritondatacenter / triton-kubernetes Goto Github PK

Not sure where to ask that, but which storage it should use on kubernetes for volumes? could not find something related to manta for that.

master/slave terminology should not be used given its negative historical and cultural connotations. Primary/replica or leader/follower are better terms to use.

$ triton ls
SHORTID   NAME               IMG                              STATE    FLAGS  AGE
224569d6  clstokes-master-1  ubuntu-certified-16.04@20180109  running  K      17h
afb6ee91  etcd-1             ubuntu-certified-16.04@20180109  running  K      17h
56e1d04a  node-2             ubuntu-certified-16.04@20180109  running  K      17h
55d0624b  node-1             ubuntu-certified-16.04@20180109  running  K      17h
96d4a1de  node-4             ubuntu-certified-16.04@20180109  running  K      16h
f815fb11  node-3             ubuntu-certified-16.04@20180109  running  K      16h

Currently (as of 45cccbf), the components on Triton do not have the firewall enabled on any of the machines (master, node-*, etcd-*, etc).

Firewalls should be enabled and connectivity restricted to only necessary ports and CIDR blocks for all environments and components - Triton, AWS, GCP, etc.

Related to #57.

There should be some sort of a LB (i.e. nginx servers) on top of the cluster managers.

When you hit the quota there is no way to roll back or continue with the setup. The only fix I found is to delete everything and go none HA. I think the limit is 32 GB Dram in one data center. There should be an easy roll-back option to create maybe one less worker node.

Environment

DC: Joyent us-sw-1
Running Env: OS X

Commands

$ ./triton-kubernetes.sh -c
Using terraform ...

Name your Global Cluster Manager: (global-cluster)
Do you want to set up the Global Cluster Manager in HA mode? (yes | no) yes
Number of cluster manager nodes for global-cluster Global Cluster Manager: (2)
From below options:
Joyent-SDC-Public
Joyent-SDC-Private
Both
Which Triton networks should be used for this environment: (Joyent-SDC-Public) Both
From below packages:

Here is where the application exited.

While trying to launch a GKE cluster, triton-kubernetes presents a list of Kubernetes versions to choose from. Is this list obtained from the Google-API?
It seems that no supported Kubernetes version is presented. Resulting in an error message like
google_container_cluster.primary: googleapi: Error 400: master version "1.8.12-gke.0" is unsupported., badRequest

If that list is obtained from the Google-API, it seems odd, that it only presents unsupported Kubernetes versions.

Deploying a pair of managers in an on premise Triton installation works like charm. triton-kubernetes talks to the api to learn images, packages and networks.

Deploying a cluster (and nodes for it) does not work because the network "Joyent-SDC-Public" is defined somewhere statically and cannot be changed during the "question and answer" phase.
And changing various occurrences of "Joyent-SDC-Public" in various terraform *tf files did not help either.

The error message is:

Error: Error refreshing state: 1 error(s) occurred:

• module.cluster_triton_cl2.data.triton_network.networks: 1 error(s) occurred:

• module.cluster_triton_cl2.data.triton_network.networks: data.triton_network.networks: No Networks found by name "Joyent-SDC-Public"

It would be nice, if the network variable could be changed for every time, it is required by Terraform to create the respective VMs.

Currently (as of 45cccbf), the Rancher UI is open to anonymous users. Local Authentication should be enabled by default and ideally with a random password created at installation to provide a relatively secure environment out-of-the-box.

When looking at the CLI help output for triton-kubernetes create it is unclear that the next string be one of the following strings: manager, cluster or node.

I suggest that we change the wording such that it is clear that you need to type in one of those words.

Hi,

Seeing failures when using a key that has a passphrase; key is ~/.ssh/id_rsa and it is available in ssh-agent under Mac OS X High Sierra.

Using a key w/o a passphrase works as expected.

Looking at the Triton Provider docs for terraform it looks like this should work - so I'm not exactly sure why this is failling for me.

Log is here:

setup.sh                                                                                                                                                  sh  utf-8[unix]   43% ☰  322/736  : 19  ☲ [58]trailing 
/network
  * module.kubemaster1.triton_machine.master: "networks": [DEPRECATED] Networks is deprecated, please use `nic`
  * module.kubemaster2.triton_machine.master: "networks": [DEPRECATED] Networks is deprecated, please use `nic`
  * module.kubemasterdb.triton_machine.mysqldb: "networks": [DEPRECATED] Networks is deprecated, please use `nic`
  * module.kubenode1.triton_machine.host: "networks": [DEPRECATED] Networks is deprecated, please use `nic`
  * module.kubenode2.triton_machine.host: "networks": [DEPRECATED] Networks is deprecated, please use `nic`
  * module.kubenodeetcd1.triton_machine.k8setcd: "networks": [DEPRECATED] Networks is deprecated, please use `nic`
  * module.kubenodeetcd2.triton_machine.k8setcd: "networks": [DEPRECATED] Networks is deprecated, please use `nic`
  * module.kubenodeetcd3.triton_machine.k8setcd: "networks": [DEPRECATED] Networks is deprecated, please use `nic`
  * module.kubenodesrvs1.triton_machine.k8sha: "networks": [DEPRECATED] Networks is deprecated, please use `nic`
  * module.kubenodesrvs2.triton_machine.k8sha: "networks": [DEPRECATED] Networks is deprecated, please use `nic`
  * module.kubenodesrvs3.triton_machine.k8sha: "networks": [DEPRECATED] Networks is deprecated, please use `nic`

Error running plan: 11 error(s) occurred:

* module.kubenodeetcd3.provider.triton: Error Creating SSH Private Key Signer: Error parsing private key: asn1: structure error: tags don't match (16 vs {class:1 tag:15 length:112 isCompound:true}) {optional:false explicit:false application:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} pkcs1PrivateKey @2
* module.kubemasterdb.provider.triton: Error Creating SSH Private Key Signer: Error parsing private key: asn1: structure error: tags don't match (16 vs {class:1 tag:15 length:112 isCompound:true}) {optional:false explicit:false application:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} pkcs1PrivateKey @2
* module.kubenodesrvs1.provider.triton: Error Creating SSH Private Key Signer: Error parsing private key: asn1: structure error: tags don't match (16 vs {class:1 tag:15 length:112 isCompound:true}) {optional:false explicit:false application:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} pkcs1PrivateKey @2
* module.kubemaster2.provider.triton: Error Creating SSH Private Key Signer: Error parsing private key: asn1: structure error: tags don't match (16 vs {class:1 tag:15 length:112 isCompound:true}) {optional:false explicit:false application:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} pkcs1PrivateKey @2
* module.kubenode2.provider.triton: Error Creating SSH Private Key Signer: Error parsing private key: asn1: structure error: tags don't match (16 vs {class:1 tag:15 length:112 isCompound:true}) {optional:false explicit:false application:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} pkcs1PrivateKey @2
* module.kubenodeetcd2.provider.triton: Error Creating SSH Private Key Signer: Error parsing private key: asn1: structure error: tags don't match (16 vs {class:1 tag:15 length:112 isCompound:true}) {optional:false explicit:false application:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} pkcs1PrivateKey @2
* module.kubenode1.provider.triton: Error Creating SSH Private Key Signer: Error parsing private key: asn1: structure error: tags don't match (16 vs {class:1 tag:15 length:112 isCompound:true}) {optional:false explicit:false application:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} pkcs1PrivateKey @2
* module.kubenodeetcd1.provider.triton: Error Creating SSH Private Key Signer: Error parsing private key: asn1: structure error: tags don't match (16 vs {class:1 tag:15 length:112 isCompound:true}) {optional:false explicit:false application:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} pkcs1PrivateKey @2
* module.kubemaster1.provider.triton: Error Creating SSH Private Key Signer: Error parsing private key: asn1: structure error: tags don't match (16 vs {class:1 tag:15 length:112 isCompound:true}) {optional:false explicit:false application:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} pkcs1PrivateKey @2
* module.kubenodesrvs3.provider.triton: Error Creating SSH Private Key Signer: Error parsing private key: asn1: structure error: tags don't match (16 vs {class:1 tag:15 length:112 isCompound:true}) {optional:false explicit:false application:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} pkcs1PrivateKey @2
* module.kubenodesrvs2.provider.triton: Error Creating SSH Private Key Signer: Error parsing private key: asn1: structure error: tags don't match (16 vs {class:1 tag:15 length:112 isCompound:true}) {optional:false explicit:false application:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} pkcs1PrivateKey @2

Happy to test as needed.

Cheers,

Jay

Old links are still mentioned in the extensive documentation. We should probably search and replace these from https://github.com/fayazg/tritonK8ssupervisor.git => https://github.com/joyent/k8s-triton-supervisor.git.

Right now, when ha=true, the number of cluster managers is hardcoded to 2 master instances. You should allow the user to specify the number of cluster master instances.

When running the installer for rancher or for kubernetes, you are given a network choice:

From below options:
Joyent-SDC-Public
Joyent-SDC-Private
Both

The Joyent-SDC-Public network is used for internet facing traffic and isn't appropriate for intra-node communication. It unneededly uses up IPs from our scare IP pool and puts network traffic on external NICs when it isn't necessary. Additionally, many users may not even want to attach external NICs for security reasons.

The Joyent-SDC-Private network suffers from some serious flaws as reported in OPS-3341. Essentially, communication between different IP pools is impossible AND IPs from this network are accessible from other tenants within Joyent's infrastructure, so this isn't a viable internal network. Additionally, users will be charged for traffic over the external NIC.

Provide a way for the user to add/remove a compute node to/from an environment (that has HA configuration).

Create pre-compiles build package for OSX for go-cli version

For AWS, the user must specify the image ID in the conf file. But for Triton, Azure and GCP, the script uses hardcoded image ID (for ubuntu-16.04), and thus this may cause issue if:
a) The ubuntu-16.04 image has different ID's on different datacenters/regions/locations.
b) The image is not available on the datacenter/region/location used by the user.
c) The image is not accessible (due to lack of permission) by the user.

Some cloud providers require unique names for certain components and this can cause collisions and failed deployments if users enter commonly used names for their cluster names. The deployment should add a random prefix to components on deployment. This value should be consistent across all components and providers for the same global cluster.

e.g.

$ triton ls
SHORTID   NAME               IMG                              STATE    FLAGS  AGE
224569d6  fda-312-clstokes-master-1  ubuntu-certified-16.04@20180109  running  K      17h
afb6ee91  fda-312-etcd-1             ubuntu-certified-16.04@20180109  running  K      17h
56e1d04a  fda-312-node-2             ubuntu-certified-16.04@20180109  running  K      17h
55d0624b  fda-312-node-1             ubuntu-certified-16.04@20180109  running  K      17h
96d4a1de  fda-312-node-4             ubuntu-certified-16.04@20180109  running  K      16h
f815fb11  fda-312-node-3             ubuntu-certified-16.04@20180109  running  K      16h

Users will want to know what network connectivity is required between the various components within an environment and between the environments and the cluster manager(s). We should document the following:

• What ports are required to be open and between what components?
• What network protocols are in use between the components?
• That TLS is in use between components that traverse the public internet and ideally for internal connectivity as well.

Hi *,
after the webcast I wanted to check the latest release:
root@e3b75fc5-3621-cdd8-f9dd-c9acbf4a37e9:~# SOURCE_URL=/root/triton-kubernetes/ SOURCE_REF=master ./triton-kubernetes_linux-amd64 version
no version set for this build... triton-kubernetes v0.9.0-pre2 (local)

triton-kubernetes create manager exits with the following error:

Error: Error applying plan:

2 error(s) occurred:

• module.cluster-manager.module.rancher_access_key.data.external.shell: data.external.shell: can't find external program "ruby"
• module.cluster-manager.module.rancher_secret_key.data.external.shell: data.external.shell: can't find external program "ruby"

And that's true - ruby is not installed in the ubuntu-Image. I have tried two of those:
b2da7f6e ubuntu-certified-16.04 20171122 P linux zvol 2017-11-28
d42c37f4 ubuntu-certified-16.04 20180109 P linux zvol 2018-01-10

I have searched through the repository to find the script/location where additional packages are installed - but I could not find that location.

Then I tried a local install of ruby (on the box I executed triton-kubernetes on) and the next try worked ok. So maybe you should add "ruby" to the prerequisites :-)

While trying to create cluster on baremetal, there is Host Prefix field that need to be given. When given values like - worker, control etc., this will be taken up a worker-1/control-1 as the host name. So after the setup the rancher tries to connect to host worker-1/control-1 and that fails.

Not sure what Host prefix can be given to fix this issue.

Create pre-compiles build package for Linux 64 Bit for go-cli version

Cluster destroyed using triton-kubernetes does not remove the cluster config from the rancher dashboard. Have to remove the config from rancher dashboard manually to again re-use the cluster name.

Create pre-compiles build package for windows for go-cli version

1. I created one Kubernetes environment called "azure-test" in the "westus2" data center.
2. Copy template_azure_environment.conf to template_azure_environment2.conf
3. Changed "azure_location=westus" and "name=azure-test2".
4. Execute the k8s-triton-supervisor script create the second environment, and got the following errors:

1 error(s) occurred:

* module.azure-test2.azurerm_resource_group.resource_group: 1 error(s) occurred:

* azurerm_resource_group.resource_group: Error creating resource group: resources.GroupsClient#CreateOrUpdate: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="InvalidResourceGroupLocation" Message="Invalid resource group location 'westus'. The Resource group already exists in location 'westus2'."

Now that triton-kubernetes is in active development and we are often building off of the master branch, it is difficult to know exactly what version you are running without a git hash.

When using the go-cli, I can't store my terraform data in Manta:

$ ./triton-kubernetes create manager
✔ Backend Provider: Manta
✔ Triton Account Name (usually your email): <redacted>
✔ Triton Key Path: ~/.ssh/id_rsa
✔ Triton URL: https://us-east-1.api.joyent.com
create manager called
✔ Cluster Manager Name: demo-1
✔ Highly Available: Yes
✔ How many master nodes: 2
✔ Triton Account Name (usually your email): <redacted>
✔ Triton Key Path: ~/.ssh/id_rsa
✔ Triton URL: https://us-east-1.api.joyent.com
✔ Triton Networks: Joyent-SDC-Public
  Attach another? No
✔ Triton Image: ubuntu-certified-16.04@20180109
✔ Triton SSH User: root
✔ Rancher Master Triton Machine Package: k4-highcpu-kvm-3.75G
Downloading modules...
Get: git::https://github.com/joyent/triton-kubernetes.git?ref=go-cli

Initializing the backend...

Error configuring the backend "manta": Failed to configure remote backend "manta": Error getting Manta credentials: Error constructing authentication for <redacted>: invalid private key data:

Please update the configuration in your Terraform files to fix this error
then run this command again.

exit status 1

Here's what I see

 $ triton-kubernetes create manager
✔ Backend Provider: Local
create manager called
✔ Cluster Manager Name: test
✔ Highly Available: Yes
✔ How many master nodes: 2
✔ Private Registry: None
✔ Rancher Server Image: Default
✔ Rancher Agent Image: Default
✔ Triton Key Path: ~/.ssh/id_rsa_triton
Private key file does not match public key fingerprint

I can create docker containers via triton-docker and peform other tasks with the configured Triton profile. Also, I ran the following file to check if the keys match:

PRIVKEY=~/.ssh/id_rsa_triton
TESTKEY=~/.ssh/id_rsa_triton.pub
diff <( ssh-keygen -y -e -f "$PRIVKEY" ) <( ssh-keygen -y -e -f "$TESTKEY" )

which output nothing (as it should for matching keys). I also verified the public key I used on my Triton account matches the public key I have locally.

So, if anyone can help me figure out how to get past this, I'd appreciate it. I really would like to use k8s on Triton...

Thanks.

Is there any particular reason you are not using Overlay / Overlay2?

OverlayFS only works with two layers. This means that performance should be better than AUFS, which can suffer noticeable latencies when searching for files in images with many layers. This advantage applies to both overlay and overlay2 drivers. overlayfs2 will be slightly less performant than overlayfs on initial read, because it has to look through more layers, but it caches the results so this is only a small penalty.

Create silent install option, which allows to pass config files with -f option

Currently,

The shell scripts to bootstrap rancher / k8s are based around package names and other things that may not exist in a private triton implementation.

Perhaps it would make sense to have default variables for JPC that are easily overwritten in a .vars or in env vars. I hard coded package names / network names etc for my private cloud.

For deploying clusters in GCP or GKE a json file with the respective credentals is needed. I haven't found an example file in the docs, yet. Could you provide one?

If I run ./triton-kubernetes.sh without jq installed, I get this:

$ ./triton-kubernetes.sh
error: jq is not in your PATH ...
       Make sure it is in your PATH and run this command again.

But jq is not listed as a pre-requisite in the README.md. It's mentioned in passing when it is explicitly used in a suggested AWS cli command, but that's it. It also doesn't tell me how to get it.

For a Triton install, I was getting this error from Terraform:

Error: output 'masters': unknown module referenced: create_rancher

Error: output 'masters': reference to undefined module "create_rancher"

I think it's because in:
triton-kubernetes/terraform/create-rancher.tf

Line 7: module "create_rancher-example" {

doesn't match the module called in lines 34-35:

output "masters" {
value = "${module.create_rancher.masters}"

When I changed line 7 to create_rancher the Terraform section completed without errors.

Hope that helps.

In the go-cli branch, there is no documentation or Makefile about how to build the golang utility.

For security reason, Rancher should require authentication with default username=admin or something along that line. The user can provide the password in the conf file.

All my hosts register into rancher, but the kuberenetes stack never reaches full healthy.

I am using ubuntu certified 16.04 instances, and 1 network. My packages are decently sized.

Is there any configuration I am missing?

1. First I created an environment called "kn-azure-test" that is not HA.
2. Copied template_azure_enviornment.conf file into template_azure_template2.conf
3. Modified it for a second environment called "kn2-azure-test" that is HA.
4. When I tried to execute k8s-triton-supervisor to add the second environment, I got the following error:

Error: Error applying plan:
1 error(s) occurred:

module.kn2-azure-test.azurerm_virtual_network.vnet: 1 error(s) occurred:
azurerm_virtual_network.vnet: network.VirtualNetworksClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="InUseSubnetCannotBeDeleted" Message="Subnet k8s-subnet is in use by /subscriptions/76ac1da8-ead0-4824-a5d8-b0a997c4b90e/resourceGroups/k8s/providers/Microsoft.Network/networkInterfaces/kn-azure-test-compute-1/ipConfigurations/testconfiguration1 and cannot be deleted." Details=[]

Looks like it is trying to delete the "k8s-subnet" resource if it exists.