azure / azure-capi-cli-extension Goto Github PK
View Code? Open in Web Editor NEWKubernetes Cluster API support in the Azure CLI
License: MIT License
Kubernetes Cluster API support in the Azure CLI
License: MIT License
Currently, we are checking the existence of azure prerequired environment variables and raising an error to the user in an individual manner. Meaning the first missing variable will throw an error to the user.
This update aims to improve user experience by telling user about all missing variables at once, instead of the user needing to restart multiple times due to other missing azure env variables.
We could validate all required environment variables and raise an error with all missing variables at once.
See #64 (comment)
az capi create -n test-win -l westus2 -w
Command group 'capi' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Create the Kubernetes cluster "test-win" in the Azure resource group "test-win"? (y/N): y
✓ Cluster API is ready
Do you want to use kind-capi-manager as the management cluster? (y/n): y
Could not generate workload cluster configuration. 'AZURE_SSH_PUBLIC_KEY' is undefined
This is on the main branch
If you have a fresh machine and you don't have docker installed, choose kind
as your local management cluster you can end up in a situation where the az capi create
command can fail:
Where do you want to create a management cluster?
[1] azure - a management cluster in the Azure cloud
[2] local - a local Docker container-based management cluster
[3] exit - don't create a management cluster
Please enter a choice [Default choice(1)]: 2
Please name the management cluster [Default capi-manager]:
Download and install kind? (y/N): y
✓ Downloaded kind
Couldn't create kind management cluster
root@igorpc:~/.azure# kind
kind creates and manages local Kubernetes clusters using Docker container 'nodes'
Usage:
kind [command]
Available Commands:
build Build one of [node-image]
completion Output shell completion code for the specified shell (bash, zsh or fish)
create Creates one of [cluster]
delete Deletes one of [cluster]
export Exports one of [kubeconfig, logs]
get Gets one of [clusters, nodes, kubeconfig]
help Help about any command
load Loads images into nodes
version Prints the kind CLI version
Flags:
-h, --help help for kind
--loglevel string DEPRECATED: see -v instead
-q, --quiet silence all stderr output
-v, --verbosity int32 info log verbosity
--version version for kind
Use "kind [command] --help" for more information about a command.
#:~/.azure# kind get clusters
ERROR: failed to list clusters: command "docker ps -a --filter label=io.x-k8s.kind.cluster --format '{{.Label "io.x-k8s.kind.cluster"}}'" failed with error: exit status 1
I don't think we should install docker but a check and pointer to install docker would be nice.
It would be lovely if we could get a third-party tool set up to monitor our unit tests and report on code coverage.
I think -v
is too often used with --verbose
and thus this is confusing; let's just not have a short option for that flag in az capi create
.
Rather than a flat file, it would be more user-friendly and informative to write out separate files for resource types, contained within a cluster folder. This is also a better practice for storing CAPI manifests in version control, IMHO.
We also need to add AzureClusterIdentity and identityRef so we don't use manager identity anymore, this can be a follow up though
Originally posted by @CecileRobertMichon in #61 (review)
The templates are old and need to be updated to match current CAPI stuff.
It's not currently using the short-description field, and it should use the long-description field as a helpful pointer, like:
"You can use https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-azure/main/templates/cluster-template.yaml as a starting point for a custom template"
We could provide an option to cache out the artifacts that we fetch from clusterctl init
first time around.
User scenario:
User's environment has strict rules around internet access so something we could do is cache the artifacts locally in a known location.
In some environments, users won't have access to /usr/local/bin, eg:
cecile@Azure:~$ az capi create -n cerobert-test -g test
Command group 'capi' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
wrote manifest file to cerobert-test.yaml
Do you want to create this Kubernetes cluster "cerobert-test" in the Azure resource group "test"? (y/N): y
Downloading client to "/usr/local/bin/clusterctl" from "https://github.com/kubernetes-sigs/cluster-api/releases/latest/download/clusterctl-linux-amd64"
Connection error while attempting to download client ([Errno 13] Permission denied: '/usr/local/bin/clusterctl')
When creating the management cluster, az capi
always creates and uses a resource group named "capi-manager." This should be able to be changed.
We should have more clarity on the current goals of this project, since there are multiple things it attempts to solve. This should be stated more clearly in a design doc to help contributors get oriented and to provide feedback on whether the goals are worthwhile.
These commands should require user confirmation before proceeding.
Currently we deploy calico but the kubeproxy image is missing
azure-capi-cli-extension/src/capi/azext_capi/custom.py
Lines 438 to 442 in 86d2687
We can use clusterctl to grab https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/main/templates/addons/windows/calico/kube-proxy-windows.yaml and replace the kubernetes version with version configured to create the cluster
There are several manually written loops that retry commands until success. This code could maybe be simplified and made more consistent with a decorator or helper func that I know is available in a couple of different python libraries.
When I follow the instructions to run in Codespaces, I run into an error when executing azdev test
. Below is the output from executing the instructions from my Codespaces terminal.
I would expect to follow the instructions and be able to execute the project tests from the Codespaces terminal.
vscode ➜ /workspaces/azure-capi-cli-extension (foo ✗) $ python3 -m venv env
n -m pip install -U pip
python -m pip install -r requirements.txt
azdev setup --repo . --ext capi --verbose
vscode ➜ /workspaces/azure-capi-cli-extension (foo ✗) $ source env/bin/activate
(env) vscode ➜ /workspaces/azure-capi-cli-extension (foo ✗) $
(env) vscode ➜ /workspaces/azure-capi-cli-extension (foo ✗) $ python -m pip install -U pip
Requirement already satisfied: pip in ./env/lib/python3.6/site-packages (21.0.1)
(env) vscode ➜ /workspaces/azure-capi-cli-extension (foo ✗) $ python -m pip install -r requirements.txt
Requirement already satisfied: azdev in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (0.1.29)
Requirement already satisfied: black in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 2)) (20.8b1)
Requirement already satisfied: Jinja2 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 3)) (2.11.3)
Requirement already satisfied: MarkupSafe in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 4)) (1.1.1)
Requirement already satisfied: sphinx==1.6.7 in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (1.6.7)
Requirement already satisfied: flake8 in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (3.9.0)
Requirement already satisfied: tox in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (3.23.0)
Requirement already satisfied: pytest-xdist in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (2.2.1)
Requirement already satisfied: pytest>=5.0.0 in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (6.2.2)
Requirement already satisfied: pyyaml in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (5.4.1)
Requirement already satisfied: azure-storage-blob<2.0.0,>=1.3.1 in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (1.5.0)
Requirement already satisfied: mock in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (4.0.3)
Requirement already satisfied: wheel==0.30.0 in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (0.30.0)
Requirement already satisfied: isort==4.3.21 in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (4.3.21)
Requirement already satisfied: docutils in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (0.16)
Requirement already satisfied: knack in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (0.7.2)
Requirement already satisfied: pylint==2.3.0 in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (2.3.0)
Requirement already satisfied: gitpython in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (3.1.14)
Requirement already satisfied: requests in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (2.25.1)
Requirement already satisfied: mccabe in ./env/lib/python3.6/site-packages (from pylint==2.3.0->azdev->-r requirements.txt (line 1)) (0.6.1)
Requirement already satisfied: astroid>=2.2.0 in ./env/lib/python3.6/site-packages (from pylint==2.3.0->azdev->-r requirements.txt (line 1)) (2.5.1)
Requirement already satisfied: snowballstemmer>=1.1 in ./env/lib/python3.6/site-packages (from sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (2.1.0)
Requirement already satisfied: Pygments>=2.0 in ./env/lib/python3.6/site-packages (from sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (2.8.1)
Requirement already satisfied: sphinxcontrib-websupport in ./env/lib/python3.6/site-packages (from sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (1.2.4)
Requirement already satisfied: alabaster<0.8,>=0.7 in ./env/lib/python3.6/site-packages (from sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (0.7.12)
Requirement already satisfied: six>=1.5 in ./env/lib/python3.6/site-packages (from sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (1.15.0)
Requirement already satisfied: setuptools in ./env/lib/python3.6/site-packages (from sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (40.6.2)
Requirement already satisfied: babel!=2.0,>=1.3 in ./env/lib/python3.6/site-packages (from sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (2.9.0)
Requirement already satisfied: imagesize in ./env/lib/python3.6/site-packages (from sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (1.2.0)
Requirement already satisfied: typed-ast<1.5,>=1.4.0 in ./env/lib/python3.6/site-packages (from astroid>=2.2.0->pylint==2.3.0->azdev->-r requirements.txt (line 1)) (1.4.2)
Requirement already satisfied: wrapt<1.13,>=1.11 in ./env/lib/python3.6/site-packages (from astroid>=2.2.0->pylint==2.3.0->azdev->-r requirements.txt (line 1)) (1.12.1)
Requirement already satisfied: lazy-object-proxy>=1.4.0 in ./env/lib/python3.6/site-packages (from astroid>=2.2.0->pylint==2.3.0->azdev->-r requirements.txt (line 1)) (1.5.2)
Requirement already satisfied: azure-common>=1.1.5 in ./env/lib/python3.6/site-packages (from azure-storage-blob<2.0.0,>=1.3.1->azdev->-r requirements.txt (line 1)) (1.1.26)
Requirement already satisfied: azure-storage-common~=1.4 in ./env/lib/python3.6/site-packages (from azure-storage-blob<2.0.0,>=1.3.1->azdev->-r requirements.txt (line 1)) (1.4.2)
Requirement already satisfied: python-dateutil in ./env/lib/python3.6/site-packages (from azure-storage-common~=1.4->azure-storage-blob<2.0.0,>=1.3.1->azdev->-r requirements.txt (line 1)) (2.8.1)
Requirement already satisfied: cryptography in ./env/lib/python3.6/site-packages (from azure-storage-common~=1.4->azure-storage-blob<2.0.0,>=1.3.1->azdev->-r requirements.txt (line 1)) (3.4.6)
Requirement already satisfied: pytz>=2015.7 in ./env/lib/python3.6/site-packages (from babel!=2.0,>=1.3->sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (2021.1)
Requirement already satisfied: py>=1.8.2 in ./env/lib/python3.6/site-packages (from pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (1.10.0)
Requirement already satisfied: toml in ./env/lib/python3.6/site-packages (from pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (0.10.2)
Requirement already satisfied: iniconfig in ./env/lib/python3.6/site-packages (from pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (1.1.1)
Requirement already satisfied: attrs>=19.2.0 in ./env/lib/python3.6/site-packages (from pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (20.3.0)
Requirement already satisfied: pluggy<1.0.0a1,>=0.12 in ./env/lib/python3.6/site-packages (from pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (0.13.1)
Requirement already satisfied: importlib-metadata>=0.12 in ./env/lib/python3.6/site-packages (from pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (3.7.3)
Requirement already satisfied: packaging in ./env/lib/python3.6/site-packages (from pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (20.9)
Requirement already satisfied: zipp>=0.5 in ./env/lib/python3.6/site-packages (from importlib-metadata>=0.12->pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (3.4.1)
Requirement already satisfied: typing-extensions>=3.6.4 in ./env/lib/python3.6/site-packages (from importlib-metadata>=0.12->pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (3.7.4.3)
Requirement already satisfied: chardet<5,>=3.0.2 in ./env/lib/python3.6/site-packages (from requests->azdev->-r requirements.txt (line 1)) (4.0.0)
Requirement already satisfied: certifi>=2017.4.17 in ./env/lib/python3.6/site-packages (from requests->azdev->-r requirements.txt (line 1)) (2020.12.5)
Requirement already satisfied: idna<3,>=2.5 in ./env/lib/python3.6/site-packages (from requests->azdev->-r requirements.txt (line 1)) (2.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./env/lib/python3.6/site-packages (from requests->azdev->-r requirements.txt (line 1)) (1.26.4)
Requirement already satisfied: regex>=2020.1.8 in ./env/lib/python3.6/site-packages (from black->-r requirements.txt (line 2)) (2021.3.17)
Requirement already satisfied: click>=7.1.2 in ./env/lib/python3.6/site-packages (from black->-r requirements.txt (line 2)) (7.1.2)
Requirement already satisfied: pathspec<1,>=0.6 in ./env/lib/python3.6/site-packages (from black->-r requirements.txt (line 2)) (0.8.1)
Requirement already satisfied: mypy-extensions>=0.4.3 in ./env/lib/python3.6/site-packages (from black->-r requirements.txt (line 2)) (0.4.3)
Requirement already satisfied: dataclasses>=0.6 in ./env/lib/python3.6/site-packages (from black->-r requirements.txt (line 2)) (0.8)
Requirement already satisfied: appdirs in ./env/lib/python3.6/site-packages (from black->-r requirements.txt (line 2)) (1.4.4)
Requirement already satisfied: cffi>=1.12 in ./env/lib/python3.6/site-packages (from cryptography->azure-storage-common~=1.4->azure-storage-blob<2.0.0,>=1.3.1->azdev->-r requirements.txt (line 1)) (1.14.5)
Requirement already satisfied: pycparser in ./env/lib/python3.6/site-packages (from cffi>=1.12->cryptography->azure-storage-common~=1.4->azure-storage-blob<2.0.0,>=1.3.1->azdev->-r requirements.txt (line 1)) (2.20)
Requirement already satisfied: pyflakes<2.4.0,>=2.3.0 in ./env/lib/python3.6/site-packages (from flake8->azdev->-r requirements.txt (line 1)) (2.3.0)
Requirement already satisfied: pycodestyle<2.8.0,>=2.7.0 in ./env/lib/python3.6/site-packages (from flake8->azdev->-r requirements.txt (line 1)) (2.7.0)
Requirement already satisfied: gitdb<5,>=4.0.1 in ./env/lib/python3.6/site-packages (from gitpython->azdev->-r requirements.txt (line 1)) (4.0.5)
Requirement already satisfied: smmap<4,>=3.0.1 in ./env/lib/python3.6/site-packages (from gitdb<5,>=4.0.1->gitpython->azdev->-r requirements.txt (line 1)) (3.0.5)
Requirement already satisfied: tabulate in ./env/lib/python3.6/site-packages (from knack->azdev->-r requirements.txt (line 1)) (0.8.9)
Requirement already satisfied: colorama in ./env/lib/python3.6/site-packages (from knack->azdev->-r requirements.txt (line 1)) (0.4.4)
Requirement already satisfied: jmespath in ./env/lib/python3.6/site-packages (from knack->azdev->-r requirements.txt (line 1)) (0.10.0)
Requirement already satisfied: argcomplete in ./env/lib/python3.6/site-packages (from knack->azdev->-r requirements.txt (line 1)) (1.12.2)
Requirement already satisfied: pyparsing>=2.0.2 in ./env/lib/python3.6/site-packages (from packaging->pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (2.4.7)
Requirement already satisfied: pytest-forked in ./env/lib/python3.6/site-packages (from pytest-xdist->azdev->-r requirements.txt (line 1)) (1.3.0)
Requirement already satisfied: execnet>=1.1 in ./env/lib/python3.6/site-packages (from pytest-xdist->azdev->-r requirements.txt (line 1)) (1.8.0)
Requirement already satisfied: apipkg>=1.4 in ./env/lib/python3.6/site-packages (from execnet>=1.1->pytest-xdist->azdev->-r requirements.txt (line 1)) (1.5)
Requirement already satisfied: sphinxcontrib-serializinghtml in ./env/lib/python3.6/site-packages (from sphinxcontrib-websupport->sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (1.1.4)
Requirement already satisfied: virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0 in ./env/lib/python3.6/site-packages (from tox->azdev->-r requirements.txt (line 1)) (20.4.3)
Requirement already satisfied: filelock>=3.0.0 in ./env/lib/python3.6/site-packages (from tox->azdev->-r requirements.txt (line 1)) (3.0.12)
Requirement already satisfied: distlib<1,>=0.3.1 in ./env/lib/python3.6/site-packages (from virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0->tox->azdev->-r requirements.txt (line 1)) (0.3.1)
Requirement already satisfied: importlib-resources>=1.0 in ./env/lib/python3.6/site-packages (from virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0->tox->azdev->-r requirements.txt (line 1)) (5.1.2)
. --ext capi --verboseces/azure-capi-cli-extension (foo ✗) $ azdev setup --repo
=======================
| Azure CLI Dev Setup |
=======================
Azure CLI:
PyPI
Azure CLI extension repos:
/workspaces/azure-capi-cli-extension
Azure CLI extensions:
/workspaces/azure-capi-cli-extension/src/capi
Installing packages
=====================
Upgrading pip...
Running: /usr/local/py-utils/bin/python -m pip install --upgrade pip
Installing `azure-cli`...
Running: /usr/local/py-utils/bin/python -m pip install --upgrade azure-cli
Installing `azure-cli-testsdk`...
Running: /usr/local/py-utils/bin/python -m pip install git+https://github.com/Azure/azure-cli@master#subdirectory=src/azure-cli-testsdk
Adding extension '/workspaces/azure-capi-cli-extension/src/capi'...
Running: /usr/local/py-utils/bin/python -m pip install -e /workspaces/azure-capi-cli-extension/src/capi
Elapsed time: 0 min 58 sec
Finished dev setup!
=====================
(env) vscode ➜ /workspaces/azure-capi-cli-extension (foo ✗) $ azdev test
=============
| Run Tests |
=============
No module named 'azure.cli'
Traceback (most recent call last):
File "/workspaces/azure-capi-cli-extension/env/lib/python3.6/site-packages/knack/cli.py", line 215, in invoke
cmd_result = self.invocation.execute(args)
File "/workspaces/azure-capi-cli-extension/env/lib/python3.6/site-packages/knack/invocation.py", line 225, in execute
cmd_result = parsed_args.func(params)
File "/workspaces/azure-capi-cli-extension/env/lib/python3.6/site-packages/knack/commands.py", line 148, in __call__
return self.handler(*args, **kwargs)
File "/workspaces/azure-capi-cli-extension/env/lib/python3.6/site-packages/knack/commands.py", line 255, in _command_handler
result = op(client, **command_args) if client else op(**command_args)
File "/workspaces/azure-capi-cli-extension/env/lib/python3.6/site-packages/azdev/operations/testtool/__init__.py", line 46, in run_tests
path_table = get_path_table()
File "/workspaces/azure-capi-cli-extension/env/lib/python3.6/site-packages/azdev/utilities/path.py", line 193, in get_path_table
from azure.cli.core.extension import EXTENSIONS_DIR # pylint: disable=import-error
ModuleNotFoundError: No module named 'azure.cli'
--name
could be optional if we add a random name generator.
At the moment we have a default hardcoded location, for our resource group creation but we will need to update to allow user to set specific locations
Originally posted by @CecileRobertMichon in #76 (comment)
"Please specify a location so a resource group can be created." doesn't say how to set the location, it should include the -l/--location flags to point the user in the right direction.
It would be extremely helpful if az capi create
could give the user more of a clue when it's done, perhaps by printing out something like:
Reference "my-cluster.kubeconfig" to access your new workload cluster. For example:
kubectl --kubeconfig=my-cluster.kubeconfig get nodes
or
Use this command to ensure your workload cluster is deployed:
I installed the v0.0.4 extension but ran into this error:
% az extension list
% az extension add --source https://github.com/Azure/azure-capi-cli-extension/releases/download/v0.0.4/capi-0.0.4-py2.py3-none-any.whl --upgrade
Are you sure you want to install this extension? (y/n): y
The installed extension 'capi' is in preview.
% az capi -h
The command failed with an unexpected error. Here is the traceback:
...
import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/Users/matt/.azure/cliextensions/capi/azext_capi/custom.py", line 30, in <module>
from .helpers.generic import add_kubeconfig_to_command, has_kind_prefix
ModuleNotFoundError: No module named 'azext_capi.helpers'
I think this is because there is no __init__.py
in the new helpers directory, so Python doesn't recognize it as a package. Maybe it worked in development because it was seen as a namespace package?
Currently we follow the monolithic manifest referenced in the CAPI Quick Start for installing CNI the Azure way. An alternative approach using a ClusterResourceSet on the management cluster and annotations on workload clusters might be more forward-looking and flexible.
The process to release az capi
is simple, but needs to be documented somewhere. Having a RELEASING.md file in the project root is a common convention for this.
az capi list
Command group 'capi' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Unable to connect to the server: dial tcp: lookup myaksclust-myresourcegroup-740677-0fe9d6b3.hcp.eastus.azmk8s.io on 168.63.129.16:53: no such host
Command '['kubectl', 'get', 'clusters', '-o', 'json']' returned non-zero exit status 1.
az capi create
runs quite a few separate commands under the covers, most of which print their native output to stdout
(and stderr
for the most part). This output can be a bit confusing since it includes suggestions in the middle of the workflow such as
Set kubectl context to "kind-capi-manager-d31i47"
You can now use your cluster with:
kubectl cluster-info --context kind-capi-manager-d31i47
Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
and
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Ideally the az capi
extension would hide all this output (by default, not with the --verbose
flag) in favor of simpler status messages with progress bars, as in the Waiting for CAPI components to be running
case.
As a user I would like to have an easy way to tell az capi
to clean up everything that it has created while running az capi create
:
This command/option should be able to determine to:
Most output formats are well supported in az capi
:
$ az capi show -n testcluster2 -o table
Command group 'capi' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Name Phase Created Namespace
------------ ----------- -------------------- -----------
testcluster2 Provisioned 2021-04-07T21:37:55Z default
$ az capi list -o yaml | head
https://aka.ms/CLI_refstatus
apiVersion: v1
items:
- apiVersion: cluster.x-k8s.io/v1alpha3
kind: Cluster
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"cluster.x-k8s.io/v1alpha3","kind":"Cluster","metadata":{"annotations":{},"labels":{"cni":"calico"},"name":"testcluster2","namespace":"default"},"spec":{"clusterNetwork":{"pods":{"cidrBlocks":["192.168.0.0/16"]}},"controlPlaneRef":{"apiVersion":"controlplane.cluster.x-k8s.io/v1alpha3","kind":"KubeadmControlPlane","name":"testcluster2-control-plane"},"infrastructureRef":{"apiVersion":"infrastructure.cluster.x-k8s.io/v1alpha3","kind":"AzureCluster","name":"testcluster2"}}}
'
creationTimestamp: '2021-04-07T21:37:55Z'
But not the tab-separated-values format:
$ az capi show -n testcluster2 -o tsv
cluster.x-k8s.io/v1alpha3 Cluster
$ az capi list -o tsv
v1 1 List
While creating the table formats is relatively straightforward, I don't see how to handle tsv
customization. But the current default output isn't useful.
Getting started with git-flow can be daunting. We should help folks get started with CAPI / CAPZ by setting them up with a batteries included git-flow project(s) using Flux v2.
Perhaps, we can start with a proposal for how we can blend https://toolkit.fluxcd.io/get-started/ with az capi init
.
I'm imagining a command to init a repo, setup a ./clusters/my-mgmt-cluster
path with a kustomize set of CAPI / CAPZ resources, deploy a management cluster with Flux components and point to the new repo.
If the default ~/.kube/config
file is empty or a stub, the az aks get-credentials
command that az capi
calls under the covers will raise an error.
The workaround is to delete that kubeconfig file and try again, but az capi
should find a workaround for this so it doesn't break the flow. We should also submit an issue and/or fix upstream to azure-cli.
Haven't verified that the basic workflow runs on Windows natively yet, especially the file download helpers.
TODOs:
At the moment when we run capi create
we will try to see if there is a cluster within default context and will use it as our management cluster, and if cluster doesn't have CAPI components instead it will install them
We should update it to look for cluster in current context, however we need to ask for confirmation of user if they desire to use the found cluster as the management cluster or if they want to create one instead. The goal is to prevent unexpected use of previous existing cluster without informing the user first
Currently, we "shell out" to the kubectl
binary as well as some others. Instead, we could use the python-kubernetes client library directly to accomplish the same tasks, which would avoid the fiddly process of downloading the binary and putting in the search PATH.
Steps to repro:
az capi create -g test -n my-cluster
az capi create -g test -n my-cluster-2
The second cluster fails to create with error:
E0325 22:19:33.874459 1 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to reconcile cluster services: failed to reconcile virtual network: failed to create virtual network test-2-vnet: network.VirtualNetworksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code=\"LocationNotAvailableForResourceType\" Message=\"The provided location 'None' is not available for resource type 'Microsoft.Network/virtualNetworks'. List of available regions for the resource type is 'westus,eastus,northeurope,westeurope,eastasia,southeastasia,northcentralus,southcentralus,centralus,eastus2,japaneast,japanwest,brazilsouth,australiaeast,australiasoutheast,centralindia,southindia,westindia,canadacentral,canadaeast,westcentralus,westus2,ukwest,uksouth,koreacentral,koreasouth,francecentral,australiacentral,southafricanorth,uaenorth,switzerlandnorth,germanywestcentral,norwayeast'.\"" "controller"="azurecluster" "name"="test-2" "namespace"="default"
The default behavior for Jinja2 templating is to render a missing variable as an empty string ""
. Now that we don't have any defaults in the templates, we should change that behavior to raise an error.
Raising an error right away is more useful. A mistaken empty string seems likely to cause errors later that are harder to debug.
az extension add --source https://github.com/Azure/azure-capi-cli-extension/releases/download/v0.0.3/capi-0.0.3-py2.py3-none-any.whl --upgrade
az capi create -n test-win -l westus2
Command group 'capi' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
✓ Generated workload cluster configuration at "test-win.yaml"
Do you want to create this Kubernetes cluster "test-win" in the Azure resource group "test-win"? (y/N): y
Connection error while attempting to download client ([Errno 13] Permission denied: '/usr/local/bin/clusterctl')
We have often observed that the Spinner doesn't animate in Windows Terminal shells. (Maybe has something to do with zsh
on Windows?)
The saved JSON in the scenario tests still uses v1alpha3 models and should be updated.
Add support for user custom defined templates on az capi create
We do output a kubeconfig during cluster creation to the local folder but if for some reason the cluster templates were applied but the cluster didn't get created then you might not get to that step. You might also have removed that file and need an easy way to grab the clusterconfig for use. We could make the output of that file optional.
clusterctl can do this via clusterctl get kubeconfig clustername
maybe something like az capi show -n <clustername> -kubeconfig
?
We provide other features similar to this with az capi show
and az capi list
This would look something like the following, including documenting the requirements for creating a service principal:
#sp for capz controller manager
az ad sp create-for-rbac --name capz-sp
az ad sp list --display-name "jjsclusterapitest" --query "[].objectId" -o tsv
#learn more at https://capz.sigs.k8s.io/topics/identities-use-cases.html#capz-controller
# sub access
az role assignment create --assignee <sp-objectid> --role Contributor --scope subscriptions/<subid>
# resource group access
# requires you to create RG up front
az role assignment create --assignee <sp-objectid> --role Contributor --scope subscriptions/<sub>/resourceGroups/<rgname>
Then to create cluster
If on windows use WSL.
export AZURE_CLIENT_ID=<fill in>
export AZURE_CLIENT_SECRET=<fill in>
export AZURE_SUBSCRIPTION_ID=<fill in>
export AZURE_TENANT_ID=<fill in>
# download the attached file and add the extesion
az extension add -s <url>
# this will walk you through the set up steps, including installing stuff then drop a kubeconfig locally
az capi create -n win1 -l westus2 -k 1.23.5
To connect to the cluster, after the extension is done:
export KUBECONFIG=win1.kubeconfig
kubectl get nodes (this might take some time for all nodes to join cluster ~5 mins)
az capi
tries to be parallel to or compatible with the Quick Start and CAPI/CAPZ docs, so we shouldn't require the --location
argument if the user already has AZURE_LOCATION
set in their environment.
% az capi create -l southcentralus -n test1
Command group 'capi' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
(ResourceGroupNotFound) Resource group 'test1' could not be found.
Code: ResourceGroupNotFound
Message: Resource group 'test1' could not be found.
The intention is that if the "test1" resource group doesn't exist, we create it for the user just-in-time. But I think the exception handling has broken over time, or else we've broken something in making AKS the default for the management cluster.
It will be beneficial to restructure custom.py
into smaller part for better readability and maintenance
Originally posted by @CecileRobertMichon in #86 (comment)
The setup steps needed for kubeadm on windows are very different if the container runtime is docker vs containerd.
AZ CAPI should support containerd
This aims to facilitate the user to set custom default variable values to run az capi commands.
Case scenario:
A user knows he/she will be creating workload cluster always on a specific location. Instead of explicitly running every time:
az capi create -n <cluster-name> -l <desired-location>
Instead the user should be able to set a default location and be able to run
az capi create -n <cluster-name>
Possible Approaches:
For some important variables there are some defaults in python code and templates. There should only be one source of truth.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.