azure / azure-capi-cli-extension Goto Github PK

Kubernetes Cluster API support in the Azure CLI

License: MIT License

Python 87.57% Dockerfile 0.89% Shell 11.07% Jinja 0.47%

azure-capi-cli-extension's Introduction

Kubernetes Cluster API extension for Azure CLI

The Kubernetes Cluster API extension for Azure CLI helps you create, evolve, and maintain Kubernetes clusters on Azure in a familiar, declarative way. Add this extension to your Azure CLI to harness the power and flexibility of Cluster API (CAPI) in just a few az capi commands.

How to Use

Install az, the command-line interface to the Microsoft Azure cloud
Use az extension add with the latest release
Run az capi -h to get an overview of the commands available to you

Developer Setup

Developing this Azure CLI extension requires a virtual environment with Python 3.7 or later, several required libraries, and the azdev tool.

You can jump into development right now on the web or on your workstation with GitHub Codespaces, or you can set up a local environment.

GitHub Codespaces

From the GitHub homepage for this project, click the big green "Code" button and choose "Open with Codespaces." After some time to prepare the environment, you'll be presented with the web-based version of Visual Studio Code with this project's source code ready to hack on.

You can also use codespaces for local development. After opening the codespace as described above, click the "Open in Visual Studio Code" button on the environment preparation screen.

NOTE: when the Codespace runs for the first time, the create-dev-env.sh script will still be running. After a few minutes, the virtual environment will be configured and ready.

Local Environment

Create a virtual environment for Python 3.7 or later, activate it, install required libraries, and tell the azdev tool about our "capi" extension:

./create-dev-env.sh

The script may take several minutes to complete, so please be patient.

Linting and Testing

You can lint and test your code with these commands:

source ./env/bin/activate

azdev linter
azdev style
azdev test

Submitting Pull Requests

To add a feature or change an existing one, please begin by submitting a markdown document that briefly describes your proposal. This will allow others to review and suggest improvements before you move forward with implementation.

Since this extension hopes to become an official one and eventually to merge upstream, pull requests should follow the azure-cli guidelines.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

azure-capi-cli-extension's People

Contributors

Stargazers

Watchers

Forkers

devigned mboersma jorgearteiro dannybrito isabella232 jsturtevant cecilerobertmichon bennycortese ranakan19 willie-yao marosset bingbing8 nawazkh sourcecodecheck step-security-bot mikesecurity

azure-capi-cli-extension's Issues

"capi-manager" resource group name is hard-coded

When creating the management cluster, az capi always creates and uses a resource group named "capi-manager." This should be able to be changed.

Remove shortcut for --vlan-name flag

I think -v is too often used with --verbose and thus this is confusing; let's just not have a short option for that flag in az capi create.

Use a third-party library for retrying commands

There are several manually written loops that retry commands until success. This code could maybe be simplified and made more consistent with a decorator or helper func that I know is available in a couple of different python libraries.

Write out cluster resources in a structured hierarchy

Rather than a flat file, it would be more user-friendly and informative to write out separate files for resource types, contained within a cluster folder. This is also a better practice for storing CAPI manifests in version control, IMHO.

`az capi delete` and `management delete` need confirmation and `--yes` flag

These commands should require user confirmation before proceeding.

Location is None if rg already exists

Steps to repro:

Create a cluster in a new rg az capi create -g test -n my-cluster
Create a second cluster reusing the same rg az capi create -g test -n my-cluster-2

The second cluster fails to create with error:

E0325 22:19:33.874459       1 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to reconcile cluster services: failed to reconcile virtual network: failed to create virtual network test-2-vnet: network.VirtualNetworksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code=\"LocationNotAvailableForResourceType\" Message=\"The provided location 'None' is not available for resource type 'Microsoft.Network/virtualNetworks'. List of available regions for the resource type is 'westus,eastus,northeurope,westeurope,eastasia,southeastasia,northcentralus,southcentralus,centralus,eastus2,japaneast,japanwest,brazilsouth,australiaeast,australiasoutheast,centralindia,southindia,westindia,canadacentral,canadaeast,westcentralus,westus2,ukwest,uksouth,koreacentral,koreasouth,francecentral,australiacentral,southafricanorth,uaenorth,switzerlandnorth,germanywestcentral,norwayeast'.\"" "controller"="azurecluster" "name"="test-2" "namespace"="default"

Commands should support useful "tsv" output

Most output formats are well supported in az capi:

$ az capi show -n testcluster2 -o table
Command group 'capi' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Name          Phase        Created               Namespace
------------  -----------  --------------------  -----------
testcluster2  Provisioned  2021-04-07T21:37:55Z  default

$ az capi list -o yaml | head
https://aka.ms/CLI_refstatus
apiVersion: v1
items:
- apiVersion: cluster.x-k8s.io/v1alpha3
  kind: Cluster
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"cluster.x-k8s.io/v1alpha3","kind":"Cluster","metadata":{"annotations":{},"labels":{"cni":"calico"},"name":"testcluster2","namespace":"default"},"spec":{"clusterNetwork":{"pods":{"cidrBlocks":["192.168.0.0/16"]}},"controlPlaneRef":{"apiVersion":"controlplane.cluster.x-k8s.io/v1alpha3","kind":"KubeadmControlPlane","name":"testcluster2-control-plane"},"infrastructureRef":{"apiVersion":"infrastructure.cluster.x-k8s.io/v1alpha3","kind":"AzureCluster","name":"testcluster2"}}}

        '
    creationTimestamp: '2021-04-07T21:37:55Z'

But not the tab-separated-values format:

$ az capi show -n testcluster2 -o tsv
cluster.x-k8s.io/v1alpha3	Cluster		
	
$ az capi list -o tsv
v1	1	List

While creating the table formats is relatively straightforward, I don't see how to handle tsv customization. But the current default output isn't useful.

Update templates from v1alpha3 to v1beta1

The templates are old and need to be updated to match current CAPI stuff.

Binaries output path should be configurable

In some environments, users won't have access to /usr/local/bin, eg:

cecile@Azure:~$ az capi create -n cerobert-test -g test
Command group 'capi' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
wrote manifest file to cerobert-test.yaml
Do you want to create this Kubernetes cluster "cerobert-test" in the Azure resource group "test"? (y/N): y
Downloading client to "/usr/local/bin/clusterctl" from "https://github.com/kubernetes-sigs/cluster-api/releases/latest/download/clusterctl-linux-amd64"
Connection error while attempting to download client ([Errno 13] Permission denied: '/usr/local/bin/clusterctl')

Should show "next steps" when `az capi create` finishes

It would be extremely helpful if az capi create could give the user more of a clue when it's done, perhaps by printing out something like:

Reference "my-cluster.kubeconfig" to access your new workload cluster. For example:
kubectl --kubeconfig=my-cluster.kubeconfig get nodes

Use this command to ensure your workload cluster is deployed:

Project needs a RELEASING.md

The process to release az capi is simple, but needs to be documented somewhere. Having a RELEASING.md file in the project root is a common convention for this.

On linux (including wsl2) the install to /usr/local/bin/clusterctl fails with permission denied

 az extension add --source https://github.com/Azure/azure-capi-cli-extension/releases/download/v0.0.3/capi-0.0.3-py2.py3-none-any.whl --upgrade   

 az capi create -n test-win -l westus2
Command group 'capi' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus  
✓ Generated workload cluster configuration at "test-win.yaml" 
Do you want to create this Kubernetes cluster "test-win" in the Azure resource group "test-win"? (y/N): y 
Connection error while attempting to download client ([Errno 13] Permission denied: '/usr/local/bin/clusterctl')

Add a clean up command

As a user I would like to have an easy way to tell az capi to clean up everything that it has created while running az capi create:

This command/option should be able to determine to:

Delete all created workload clusters
Delete the management cluster
Delete any template/file/kubeconfig that had created

Update readme with instruction on creating a cluster step by step

This would look something like the following, including documenting the requirements for creating a service principal:

#sp for capz controller manager
az ad sp create-for-rbac --name capz-sp
az ad sp list --display-name "jjsclusterapitest" --query "[].objectId" -o tsv

#learn more at https://capz.sigs.k8s.io/topics/identities-use-cases.html#capz-controller
# sub access 
az role assignment create --assignee <sp-objectid> --role Contributor --scope subscriptions/<subid>

# resource group access
# requires you to create RG up front 
az role assignment create --assignee <sp-objectid> --role Contributor --scope subscriptions/<sub>/resourceGroups/<rgname>

Then to create cluster

If on windows use WSL.

export AZURE_CLIENT_ID=<fill in>
export AZURE_CLIENT_SECRET=<fill in>
export AZURE_SUBSCRIPTION_ID=<fill in>
export AZURE_TENANT_ID=<fill in>

# download the attached file and add the extesion

az extension add -s <url>

# this will walk you through the set up steps, including installing stuff then drop a kubeconfig locally
az capi create -n win1 -l westus2 -k 1.23.5

To connect to the cluster, after the extension is done:

export KUBECONFIG=win1.kubeconfig

kubectl get nodes (this might take some time for all nodes to join cluster ~5 mins)

`AZURE_LOCATION` should be used as the default for `--location`

az capi tries to be parallel to or compatible with the Quick Start and CAPI/CAPZ docs, so we shouldn't require the --location argument if the user already has AZURE_LOCATION set in their environment.

Defaults in python code as well as templates

For some important variables there are some defaults in python code and templates. There should only be one source of truth.

v0.0.4 extension raises ModuleNotFoundError

I installed the v0.0.4 extension but ran into this error:

% az extension list          

% az extension add --source https://github.com/Azure/azure-capi-cli-extension/releases/download/v0.0.4/capi-0.0.4-py2.py3-none-any.whl --upgrade
Are you sure you want to install this extension? (y/n): y
The installed extension 'capi' is in preview.
% az capi -h
The command failed with an unexpected error. Here is the traceback:
...
import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/matt/.azure/cliextensions/capi/azext_capi/custom.py", line 30, in <module>
    from .helpers.generic import add_kubeconfig_to_command, has_kind_prefix
ModuleNotFoundError: No module named 'azext_capi.helpers'

I think this is because there is no __init__.py in the new helpers directory, so Python doesn't recognize it as a package. Maybe it worked in development because it was seen as a namespace package?

Templates should raise an error when a variable is missing

The default behavior for Jinja2 templating is to render a missing variable as an empty string "". Now that we don't have any defaults in the templates, we should change that behavior to raise an error.

Raising an error right away is more useful. A mistaken empty string seems likely to cause errors later that are harder to debug.

Investigate using python-kubernetes client library

Currently, we "shell out" to the kubectl binary as well as some others. Instead, we could use the python-kubernetes client library directly to accomplish the same tasks, which would avoid the fiddly process of downloading the binary and putting in the search PATH.

Filter stdout/stderr by default for a cleaner UX

az capi create runs quite a few separate commands under the covers, most of which print their native output to stdout (and stderr for the most part). This output can be a bit confusing since it includes suggestions in the middle of the workflow such as

Set kubectl context to "kind-capi-manager-d31i47"
You can now use your cluster with:

kubectl cluster-info --context kind-capi-manager-d31i47

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

and

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Ideally the az capi extension would hide all this output (by default, not with the --verbose flag) in favor of simpler status messages with progress bars, as in the Waiting for CAPI components to be running case.

Getting started with Codespaces

Actual

When I follow the instructions to run in Codespaces, I run into an error when executing azdev test. Below is the output from executing the instructions from my Codespaces terminal.

Expectation

I would expect to follow the instructions and be able to execute the project tests from the Codespaces terminal.

Output from Codespaces

vscode ➜ /workspaces/azure-capi-cli-extension (foo ✗) $ python3 -m venv env
n -m pip install -U pip
python -m pip install -r requirements.txt
azdev setup --repo . --ext capi --verbose
vscode ➜ /workspaces/azure-capi-cli-extension (foo ✗) $ source env/bin/activate
(env) vscode ➜ /workspaces/azure-capi-cli-extension (foo ✗) $ 
(env) vscode ➜ /workspaces/azure-capi-cli-extension (foo ✗) $ python -m pip install -U pip
Requirement already satisfied: pip in ./env/lib/python3.6/site-packages (21.0.1)
(env) vscode ➜ /workspaces/azure-capi-cli-extension (foo ✗) $ python -m pip install -r requirements.txt
Requirement already satisfied: azdev in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (0.1.29)
Requirement already satisfied: black in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 2)) (20.8b1)
Requirement already satisfied: Jinja2 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 3)) (2.11.3)
Requirement already satisfied: MarkupSafe in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 4)) (1.1.1)
Requirement already satisfied: sphinx==1.6.7 in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (1.6.7)
Requirement already satisfied: flake8 in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (3.9.0)
Requirement already satisfied: tox in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (3.23.0)
Requirement already satisfied: pytest-xdist in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (2.2.1)
Requirement already satisfied: pytest>=5.0.0 in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (6.2.2)
Requirement already satisfied: pyyaml in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (5.4.1)
Requirement already satisfied: azure-storage-blob<2.0.0,>=1.3.1 in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (1.5.0)
Requirement already satisfied: mock in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (4.0.3)
Requirement already satisfied: wheel==0.30.0 in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (0.30.0)
Requirement already satisfied: isort==4.3.21 in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (4.3.21)
Requirement already satisfied: docutils in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (0.16)
Requirement already satisfied: knack in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (0.7.2)
Requirement already satisfied: pylint==2.3.0 in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (2.3.0)
Requirement already satisfied: gitpython in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (3.1.14)
Requirement already satisfied: requests in ./env/lib/python3.6/site-packages (from azdev->-r requirements.txt (line 1)) (2.25.1)
Requirement already satisfied: mccabe in ./env/lib/python3.6/site-packages (from pylint==2.3.0->azdev->-r requirements.txt (line 1)) (0.6.1)
Requirement already satisfied: astroid>=2.2.0 in ./env/lib/python3.6/site-packages (from pylint==2.3.0->azdev->-r requirements.txt (line 1)) (2.5.1)
Requirement already satisfied: snowballstemmer>=1.1 in ./env/lib/python3.6/site-packages (from sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (2.1.0)
Requirement already satisfied: Pygments>=2.0 in ./env/lib/python3.6/site-packages (from sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (2.8.1)
Requirement already satisfied: sphinxcontrib-websupport in ./env/lib/python3.6/site-packages (from sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (1.2.4)
Requirement already satisfied: alabaster<0.8,>=0.7 in ./env/lib/python3.6/site-packages (from sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (0.7.12)
Requirement already satisfied: six>=1.5 in ./env/lib/python3.6/site-packages (from sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (1.15.0)
Requirement already satisfied: setuptools in ./env/lib/python3.6/site-packages (from sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (40.6.2)
Requirement already satisfied: babel!=2.0,>=1.3 in ./env/lib/python3.6/site-packages (from sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (2.9.0)
Requirement already satisfied: imagesize in ./env/lib/python3.6/site-packages (from sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (1.2.0)
Requirement already satisfied: typed-ast<1.5,>=1.4.0 in ./env/lib/python3.6/site-packages (from astroid>=2.2.0->pylint==2.3.0->azdev->-r requirements.txt (line 1)) (1.4.2)
Requirement already satisfied: wrapt<1.13,>=1.11 in ./env/lib/python3.6/site-packages (from astroid>=2.2.0->pylint==2.3.0->azdev->-r requirements.txt (line 1)) (1.12.1)
Requirement already satisfied: lazy-object-proxy>=1.4.0 in ./env/lib/python3.6/site-packages (from astroid>=2.2.0->pylint==2.3.0->azdev->-r requirements.txt (line 1)) (1.5.2)
Requirement already satisfied: azure-common>=1.1.5 in ./env/lib/python3.6/site-packages (from azure-storage-blob<2.0.0,>=1.3.1->azdev->-r requirements.txt (line 1)) (1.1.26)
Requirement already satisfied: azure-storage-common~=1.4 in ./env/lib/python3.6/site-packages (from azure-storage-blob<2.0.0,>=1.3.1->azdev->-r requirements.txt (line 1)) (1.4.2)
Requirement already satisfied: python-dateutil in ./env/lib/python3.6/site-packages (from azure-storage-common~=1.4->azure-storage-blob<2.0.0,>=1.3.1->azdev->-r requirements.txt (line 1)) (2.8.1)
Requirement already satisfied: cryptography in ./env/lib/python3.6/site-packages (from azure-storage-common~=1.4->azure-storage-blob<2.0.0,>=1.3.1->azdev->-r requirements.txt (line 1)) (3.4.6)
Requirement already satisfied: pytz>=2015.7 in ./env/lib/python3.6/site-packages (from babel!=2.0,>=1.3->sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (2021.1)
Requirement already satisfied: py>=1.8.2 in ./env/lib/python3.6/site-packages (from pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (1.10.0)
Requirement already satisfied: toml in ./env/lib/python3.6/site-packages (from pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (0.10.2)
Requirement already satisfied: iniconfig in ./env/lib/python3.6/site-packages (from pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (1.1.1)
Requirement already satisfied: attrs>=19.2.0 in ./env/lib/python3.6/site-packages (from pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (20.3.0)
Requirement already satisfied: pluggy<1.0.0a1,>=0.12 in ./env/lib/python3.6/site-packages (from pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (0.13.1)
Requirement already satisfied: importlib-metadata>=0.12 in ./env/lib/python3.6/site-packages (from pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (3.7.3)
Requirement already satisfied: packaging in ./env/lib/python3.6/site-packages (from pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (20.9)
Requirement already satisfied: zipp>=0.5 in ./env/lib/python3.6/site-packages (from importlib-metadata>=0.12->pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (3.4.1)
Requirement already satisfied: typing-extensions>=3.6.4 in ./env/lib/python3.6/site-packages (from importlib-metadata>=0.12->pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (3.7.4.3)
Requirement already satisfied: chardet<5,>=3.0.2 in ./env/lib/python3.6/site-packages (from requests->azdev->-r requirements.txt (line 1)) (4.0.0)
Requirement already satisfied: certifi>=2017.4.17 in ./env/lib/python3.6/site-packages (from requests->azdev->-r requirements.txt (line 1)) (2020.12.5)
Requirement already satisfied: idna<3,>=2.5 in ./env/lib/python3.6/site-packages (from requests->azdev->-r requirements.txt (line 1)) (2.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./env/lib/python3.6/site-packages (from requests->azdev->-r requirements.txt (line 1)) (1.26.4)
Requirement already satisfied: regex>=2020.1.8 in ./env/lib/python3.6/site-packages (from black->-r requirements.txt (line 2)) (2021.3.17)
Requirement already satisfied: click>=7.1.2 in ./env/lib/python3.6/site-packages (from black->-r requirements.txt (line 2)) (7.1.2)
Requirement already satisfied: pathspec<1,>=0.6 in ./env/lib/python3.6/site-packages (from black->-r requirements.txt (line 2)) (0.8.1)
Requirement already satisfied: mypy-extensions>=0.4.3 in ./env/lib/python3.6/site-packages (from black->-r requirements.txt (line 2)) (0.4.3)
Requirement already satisfied: dataclasses>=0.6 in ./env/lib/python3.6/site-packages (from black->-r requirements.txt (line 2)) (0.8)
Requirement already satisfied: appdirs in ./env/lib/python3.6/site-packages (from black->-r requirements.txt (line 2)) (1.4.4)
Requirement already satisfied: cffi>=1.12 in ./env/lib/python3.6/site-packages (from cryptography->azure-storage-common~=1.4->azure-storage-blob<2.0.0,>=1.3.1->azdev->-r requirements.txt (line 1)) (1.14.5)
Requirement already satisfied: pycparser in ./env/lib/python3.6/site-packages (from cffi>=1.12->cryptography->azure-storage-common~=1.4->azure-storage-blob<2.0.0,>=1.3.1->azdev->-r requirements.txt (line 1)) (2.20)
Requirement already satisfied: pyflakes<2.4.0,>=2.3.0 in ./env/lib/python3.6/site-packages (from flake8->azdev->-r requirements.txt (line 1)) (2.3.0)
Requirement already satisfied: pycodestyle<2.8.0,>=2.7.0 in ./env/lib/python3.6/site-packages (from flake8->azdev->-r requirements.txt (line 1)) (2.7.0)
Requirement already satisfied: gitdb<5,>=4.0.1 in ./env/lib/python3.6/site-packages (from gitpython->azdev->-r requirements.txt (line 1)) (4.0.5)
Requirement already satisfied: smmap<4,>=3.0.1 in ./env/lib/python3.6/site-packages (from gitdb<5,>=4.0.1->gitpython->azdev->-r requirements.txt (line 1)) (3.0.5)
Requirement already satisfied: tabulate in ./env/lib/python3.6/site-packages (from knack->azdev->-r requirements.txt (line 1)) (0.8.9)
Requirement already satisfied: colorama in ./env/lib/python3.6/site-packages (from knack->azdev->-r requirements.txt (line 1)) (0.4.4)
Requirement already satisfied: jmespath in ./env/lib/python3.6/site-packages (from knack->azdev->-r requirements.txt (line 1)) (0.10.0)
Requirement already satisfied: argcomplete in ./env/lib/python3.6/site-packages (from knack->azdev->-r requirements.txt (line 1)) (1.12.2)
Requirement already satisfied: pyparsing>=2.0.2 in ./env/lib/python3.6/site-packages (from packaging->pytest>=5.0.0->azdev->-r requirements.txt (line 1)) (2.4.7)
Requirement already satisfied: pytest-forked in ./env/lib/python3.6/site-packages (from pytest-xdist->azdev->-r requirements.txt (line 1)) (1.3.0)
Requirement already satisfied: execnet>=1.1 in ./env/lib/python3.6/site-packages (from pytest-xdist->azdev->-r requirements.txt (line 1)) (1.8.0)
Requirement already satisfied: apipkg>=1.4 in ./env/lib/python3.6/site-packages (from execnet>=1.1->pytest-xdist->azdev->-r requirements.txt (line 1)) (1.5)
Requirement already satisfied: sphinxcontrib-serializinghtml in ./env/lib/python3.6/site-packages (from sphinxcontrib-websupport->sphinx==1.6.7->azdev->-r requirements.txt (line 1)) (1.1.4)
Requirement already satisfied: virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0 in ./env/lib/python3.6/site-packages (from tox->azdev->-r requirements.txt (line 1)) (20.4.3)
Requirement already satisfied: filelock>=3.0.0 in ./env/lib/python3.6/site-packages (from tox->azdev->-r requirements.txt (line 1)) (3.0.12)
Requirement already satisfied: distlib<1,>=0.3.1 in ./env/lib/python3.6/site-packages (from virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0->tox->azdev->-r requirements.txt (line 1)) (0.3.1)
Requirement already satisfied: importlib-resources>=1.0 in ./env/lib/python3.6/site-packages (from virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0->tox->azdev->-r requirements.txt (line 1)) (5.1.2)
 . --ext capi --verboseces/azure-capi-cli-extension (foo ✗) $ azdev setup --repo 
=======================
| Azure CLI Dev Setup |
=======================
Azure CLI:
    PyPI
Azure CLI extension repos:
    /workspaces/azure-capi-cli-extension
Azure CLI extensions:
    /workspaces/azure-capi-cli-extension/src/capi
 Installing packages 
=====================
Upgrading pip...
Running: /usr/local/py-utils/bin/python -m pip install --upgrade pip
Installing `azure-cli`...
Running: /usr/local/py-utils/bin/python -m pip install --upgrade azure-cli
Installing `azure-cli-testsdk`...
Running: /usr/local/py-utils/bin/python -m pip install git+https://github.com/Azure/azure-cli@master#subdirectory=src/azure-cli-testsdk
Adding extension '/workspaces/azure-capi-cli-extension/src/capi'...
Running: /usr/local/py-utils/bin/python -m pip install -e /workspaces/azure-capi-cli-extension/src/capi
Elapsed time: 0 min 58 sec
 Finished dev setup! 
=====================
(env) vscode ➜ /workspaces/azure-capi-cli-extension (foo ✗) $ azdev test
=============
| Run Tests |
=============
No module named 'azure.cli'
Traceback (most recent call last):
  File "/workspaces/azure-capi-cli-extension/env/lib/python3.6/site-packages/knack/cli.py", line 215, in invoke
    cmd_result = self.invocation.execute(args)
  File "/workspaces/azure-capi-cli-extension/env/lib/python3.6/site-packages/knack/invocation.py", line 225, in execute
    cmd_result = parsed_args.func(params)
  File "/workspaces/azure-capi-cli-extension/env/lib/python3.6/site-packages/knack/commands.py", line 148, in __call__
    return self.handler(*args, **kwargs)
  File "/workspaces/azure-capi-cli-extension/env/lib/python3.6/site-packages/knack/commands.py", line 255, in _command_handler
    result = op(client, **command_args) if client else op(**command_args)
  File "/workspaces/azure-capi-cli-extension/env/lib/python3.6/site-packages/azdev/operations/testtool/__init__.py", line 46, in run_tests
    path_table = get_path_table()
  File "/workspaces/azure-capi-cli-extension/env/lib/python3.6/site-packages/azdev/utilities/path.py", line 193, in get_path_table
    from azure.cli.core.extension import EXTENSIONS_DIR  # pylint: disable=import-error
ModuleNotFoundError: No module named 'azure.cli'

Add a code coverage badge

It would be lovely if we could get a third-party tool set up to monitor our unit tests and report on code coverage.

Create a design document with clearer goals

We should have more clarity on the current goals of this project, since there are multiple things it attempts to solve. This should be stated more clearly in a design doc to help contributors get oriented and to provide feedback on whether the goals are worthwhile.

Determine which management cluster should be used in capi create

At the moment when we run capi create we will try to see if there is a cluster within default context and will use it as our management cluster, and if cluster doesn't have CAPI components instead it will install them

We should update it to look for cluster in current context, however we need to ask for confirmation of user if they desire to use the found cluster as the management cluster or if they want to create one instead. The goal is to prevent unexpected use of previous existing cluster without informing the user first

Update KubeadmConfig templates for windows

The setup steps needed for kubeadm on windows are very different if the container runtime is docker vs containerd.
AZ CAPI should support containerd

Bootstrap git-flow project with Flux v2

Getting started with git-flow can be daunting. We should help folks get started with CAPI / CAPZ by setting them up with a batteries included git-flow project(s) using Flux v2.

Perhaps, we can start with a proposal for how we can blend https://toolkit.fluxcd.io/get-started/ with az capi init.

I'm imagining a command to init a repo, setup a ./clusters/my-mgmt-cluster path with a kustomize set of CAPI / CAPZ resources, deploy a management cluster with Flux components and point to the new repo.

b64 variable check in check_prereqs() may not be needed

See #64 (comment)

Allow users to set defaults values to run commands

This aims to facilitate the user to set custom default variable values to run az capi commands.

Case scenario:
A user knows he/she will be creating workload cluster always on a specific location. Instead of explicitly running every time:

az capi create -n <cluster-name> -l <desired-location>

Instead the user should be able to set a default location and be able to run

az capi create -n <cluster-name>

Possible Approaches:

See if possible to leverage az config to implement defaults in capi extension
Create our own configuration command

Windows clusters should apply the kube-proxy image as well

Currently we deploy calico but the kubeproxy image is missing

azure-capi-cli-extension/src/capi/azext_capi/custom.py

Lines 438 to 442 in 86d2687

    
           calico_manifest = "https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-azure/main/templates/addons/windows/calico/calico.yaml"  # pylint: disable=line-too-long 
        
           spinner_enter_message = "Deploying Windows Calico support" 
        
           spinner_exit_message = "✓ Deployed Windows Calico support to worload cluster" 
        
           error_message = "Couldn't install Windows Calico support after waiting 5 minutes." 
        
           apply_calico_manifest(cmd, calico_manifest, workload_cfg, spinner_enter_message,

We can use clusterctl to grab https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/main/templates/addons/windows/calico/kube-proxy-windows.yaml and replace the kubernetes version with version configured to create the cluster

Resource group should be created for the user

% az capi create -l southcentralus -n test1
Command group 'capi' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
(ResourceGroupNotFound) Resource group 'test1' could not be found.
Code: ResourceGroupNotFound
Message: Resource group 'test1' could not be found.

The intention is that if the "test1" resource group doesn't exist, we create it for the user just-in-time. But I think the exception handling has broken over time, or else we've broken something in making AKS the default for the management cluster.

Allow use of local artifacts for clusterctl init

We could provide an option to cache out the artifacts that we fetch from clusterctl init first time around.

User scenario:

User's environment has strict rules around internet access so something we could do is cache the artifacts locally in a known location.

Replace CNI implementation with ClusterResourceSet

Currently we follow the monolithic manifest referenced in the CAPI Quick Start for installing CNI the Azure way. An alternative approach using a ClusterResourceSet on the management cluster and annotations on workload clusters might be more forward-looking and flexible.

Refactor/Restructure Custom File into smaller parts

It will be beneficial to restructure custom.py into smaller part for better readability and maintenance

Originally posted by @CecileRobertMichon in #86 (comment)

Restructure helper methods
Restructure command methods

Need to test on Windows

Haven't verified that the basic workflow runs on Windows natively yet, especially the file download helpers.

Error: failed to find "clusters" key in existing kubeconfig

If the default ~/.kube/config file is empty or a stub, the az aks get-credentials command that az capi calls under the covers will raise an error.

The workaround is to delete that kubeconfig file and try again, but az capi should find a workaround for this so it doesn't break the flow. We should also submit an issue and/or fix upstream to azure-cli.

Allow specifying location for resource group when creating AKS management cluster

At the moment we have a default hardcoded location, for our resource group creation but we will need to update to allow user to set specific locations

Originally posted by @CecileRobertMichon in #76 (comment)

Spinner doesn't spin on Windows Terminal

We have often observed that the Spinner doesn't animate in Windows Terminal shells. (Maybe has something to do with zsh on Windows?)

Add command to get a kubeconfig for a cluster

request

We do output a kubeconfig during cluster creation to the local folder but if for some reason the cluster templates were applied but the cluster didn't get created then you might not get to that step. You might also have removed that file and need an easy way to grab the clusterconfig for use. We could make the output of that file optional.

clusterctl can do this via clusterctl get kubeconfig clustername

maybe something like az capi show -n <clustername> -kubeconfig?

other context

We provide other features similar to this with az capi show and az capi list

Scenario tests still refer to v1alpha3

The saved JSON in the scenario tests still uses v1alpha3 models and should be updated.

`--template` argument help could be improved

It's not currently using the short-description field, and it should use the long-description field as a helpful pointer, like:
"You can use https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-azure/main/templates/cluster-template.yaml as a starting point for a custom template"

Epic: Use az capi to create kubernetes clusters for testing

TODOs:

#85
#98
#49
default to Azure management cluster vs. local management cluster
CNI / addons via ClusterResourceSet (+ turn on feature flag by default)
provide a convenience for configuring the node OS (either allow passing in script to run on worker/control plane nodes or some easy way to apply a Daemonset on the workload cluster) #97
de-duplicate template composition, improve composition of templates (individual CRDs) in CAPZ so we can rely on those fully

Before running kind should check that docker is installed

Issue

If you have a fresh machine and you don't have docker installed, choose kind as your local management cluster you can end up in a situation where the az capi create command can fail:

Where do you want to create a management cluster?



[1] azure - a management cluster in the Azure cloud
[2] local - a local Docker container-based management cluster
[3] exit - don't create a management cluster
Please enter a choice [Default choice(1)]: 2
Please name the management cluster [Default capi-manager]:
Download and install kind? (y/N): y
✓ Downloaded kind
Couldn't create kind management cluster
root@igorpc:~/.azure# kind
kind creates and manages local Kubernetes clusters using Docker container 'nodes'



Usage:
kind [command]



Available Commands:
build Build one of [node-image]
completion Output shell completion code for the specified shell (bash, zsh or fish)
create Creates one of [cluster]
delete Deletes one of [cluster]
export Exports one of [kubeconfig, logs]
get Gets one of [clusters, nodes, kubeconfig]
help Help about any command
load Loads images into nodes
version Prints the kind CLI version



Flags:
-h, --help help for kind
--loglevel string DEPRECATED: see -v instead
-q, --quiet silence all stderr output
-v, --verbosity int32 info log verbosity
--version version for kind



Use "kind [command] --help" for more information about a command.
#:~/.azure# kind get clusters
ERROR: failed to list clusters: command "docker ps -a --filter label=io.x-k8s.kind.cluster --format '{{.Label "io.x-k8s.kind.cluster"}}'" failed with error: exit status 1

possible solution

I don't think we should install docker but a check and pointer to install docker would be nice.

Add AzureClusterIdentity and identityRef in templates

We also need to add AzureClusterIdentity and identityRef so we don't use manager identity anymore, this can be a follow up though

Originally posted by @CecileRobertMichon in #61 (review)

az capi list should tell me I don't have a management cluster

az capi list
Command group 'capi' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Unable to connect to the server: dial tcp: lookup myaksclust-myresourcegroup-740677-0fe9d6b3.hcp.eastus.azmk8s.io on 168.63.129.16:53: no such host
Command '['kubectl', 'get', 'clusters', '-o', 'json']' returned non-zero exit status 1.

We could validate all required environment variables and raise an error with all missing variables at once.

pass in URI/file to workload template

Add support for user custom defined templates on az capi create

Creating a windows cluster results in error with missing env `AZURE_SSH_PUBLIC_KEY`

 az capi create -n test-win -l westus2 -w
 Command group 'capi' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus 
Create the Kubernetes cluster "test-win" in the Azure resource group "test-win"? (y/N): y
 ✓ Cluster API is ready                                                                                                         
Do you want to use kind-capi-manager as the management cluster? (y/n): y                                                       
Could not generate workload cluster configuration. 'AZURE_SSH_PUBLIC_KEY' is undefined

This is on the main branch

	calico_manifest = "https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-azure/main/templates/addons/windows/calico/calico.yaml" # pylint: disable=line-too-long
	spinner_enter_message = "Deploying Windows Calico support"
	spinner_exit_message = "✓ Deployed Windows Calico support to worload cluster"
	error_message = "Couldn't install Windows Calico support after waiting 5 minutes."
	apply_calico_manifest(cmd, calico_manifest, workload_cfg, spinner_enter_message,