ListPodIds(podns, podname) has no retries. There exists a race between MIC discovery of new pods and associating them with Azure IDs. Addting retry with wait in NMI on the listing pod id's with reduce the possibility of this to happen.

Looks like the tutorial doesn't say that the script: 1-create-azure-id.sh should be run anymore. Is this step no longer necessary ?

Here

is lookup of podns podname from ip address. If there are terminated pods with same ip address as alive pod it might fail on lookup of assignment. Its a stopper for us.

When using a cluster with vmss agent pools created by acs-engine, getting the following error from nmi logs:

adal: Refresh request failed. Status Code = '400'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"Identity not found\"}" req.method=GET req.path=/host/token/ req.remote=127.0.0.1

and this from the mic logs:

E0718 00:40:42.975094       1 mic.go:385] compute.VirtualMachinesClient#Get: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The Resource 'Microsoft.Compute/virtualMachines/k8s-agentpool-32078657-vmss000000' under resource group 'xxxx' was not found."

Hello can you provide the docker file for the demo image.
I want to create my own and I don't know what type of ADAL you have installed on it.

Thanks

e2e tests to validate pod->nmi->mic e2e scenario's

1. Service prinicipal
2. MSI
3. Flex volume

PodCIDR is not present on default configuration of acs-engine (using Azure CNI) and advanced networking configuration of AKS (uses Azure CNI instead of kubenet)

Deploying will result in: (for acs-engine)

time="2018-06-06T18:52:10Z" level=info msg="starting nmi process"
time="2018-06-06T18:52:10Z" level=info msg="Listening on port 2579"
time="2018-06-06T18:52:10Z" level=info msg="node: k8s-agentpool-35881120-vmss000000"
time="2018-06-06T18:52:10Z" level=fatal msg="podcidr is nil or empty, nodename: k8s-agentpool-35881120-vmss000000"

Changing the resource in the request does not give a different token.

These two requests return the same token, with the same Audience.

GET http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.azure.com/

GET http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://storage.azure.com/

Root cause may be the resource is hard coded. See line 25 in \pkg\nmi\server\server.go

const (
	azureResourceName                  = "https://management.azure.com/"
	iptableUpdateTimeIntervalInSeconds = 10
)

Then on line 182

case aadpodid.UserAssignedMSI:
	logger.Infof("matched identityType:%v clientid:%s resource:%s", idType, clientID, azureResourceName)
	return auth.GetServicePrincipalTokenFromMSIWithUserAssignedID(clientID, azureResourceName)

First, this is a really good add-on, the missing piece between Azure and Kubernetes :)
If I understood the docs currently, each pod with the labels mentioned in the Selector field under identity binding will be able to request a token using MSI. Isn't that a security issue? Anyone with access to the cluster can deploy an app with the required label, and impersonate a pod.
After reading the docs, it seems that you can explicitly (maybe in the future?) to specify which pods to select. Is this in order to mitigate this threat?

I deployed a fresh new cluster using ACS engine. Kubernetes version is 1.11.3. AAD pod identity is installed via ACS engine. When looking on the MIC logs I noticed the following error:

W1021 05:36:24.866744       1 mic.go:26] --kubeconfig not passed will use InClusterConfig
I1021 05:36:24.866843       1 mic.go:29] kubeconfig () cloudconfig (/etc/kubernetes/azure.json)
I1021 05:36:24.867013       1 mic.go:52] Starting to create the pod identity client
I1021 05:36:25.361861       1 pod.go:80] Pod cache synchronized. Took 493.679245ms
I1021 05:36:25.361890       1 pod.go:86] Pod watcher started !!
I1021 05:36:25.361918       1 crd.go:156] CRD watchers started
I1021 05:36:25.361925       1 mic.go:42] AAD Pod identity controller initialized!!
I1021 05:36:25.361947       1 mic.go:124] Sync thread started
E1024 22:22:08.731546       1 crd.go:225] the server was unable to return a response in the time allotted, but may still be processing the request (get azureassignedidentities.aadpodidentity.k8s.io)
E1024 22:22:58.183328       1 reflector.go:322] github.com/Azure/aad-pod-identity/pkg/crd/crd.go:155: Failed to watch *v1.AzureIdentity: the server has asked for the client to provide credentials (get azureidentities.aadpodidentity.k8s.io)
E1024 22:23:08.824020       1 crd.go:213] the server was unable to return a response in the time allotted, but may still be processing the request (get azureidentitybindings.aadpodidentity.k8s.io)
E1024 22:23:57.560417       1 reflector.go:322] github.com/Azure/aad-pod-identity/pkg/crd/crd.go:154: Failed to watch *v1.AzureIdentityBinding: the server has asked for the client to provide credentials (get azureidentitybindings.aadpodidentity.k8s.io)
E1024 22:23:59.186012       1 reflector.go:205] github.com/Azure/aad-pod-identity/pkg/crd/crd.go:155: Failed to list *v1.AzureIdentity: the server was unable to return a response in the time allotted, but may still be processing the request (get azureidentities.aadpodidentity.k8s.io)
E1024 22:24:08.877544       1 crd.go:213] the server was unable to return a response in the time allotted, but may still be processing the request (get azureidentitybindings.aadpodidentity.k8s.io)
E1024 22:28:08.270368       1 crd.go:213] the server was unable to return a response in the time allotted, but may still be processing the request (get azureidentitybindings.aadpodidentity.k8s.io)
E1024 23:13:04.608050       1 crd.go:225] the server was unable to return a response in the time allotted, but may still be processing the request (get azureassignedidentities.aadpodidentity.k8s.io)
E1024 23:13:40.311224       1 reflector.go:322] github.com/Azure/aad-pod-identity/vendor/k8s.io/client-go/informers/factory.go:87: Failed to watch *v1.Pod: the server has asked for the client to provide credentials (get pods)
E1024 23:14:04.612716       1 crd.go:213] the server was unable to return a response in the time allotted, but may still be processing the request (get azureidentitybindings.aadpodidentity.k8s.io)
E1024 23:14:41.313998       1 reflector.go:205] github.com/Azure/aad-pod-identity/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.Pod: the server was unable to return a response in the time allotted, but may still be processing the request (get pods)
E1024 23:16:09.715401       1 crd.go:213] the server was unable to return a response in the time allotted, but may still be processing the request (get azureidentitybindings.aadpodidentity.k8s.io)
E1024 23:20:34.304179       1 crd.go:225] the server was unable to return a response in the time allotted, but may still be processing the request (get azureassignedidentities.aadpodidentity.k8s.io)
E1024 23:31:20.487486       1 crd.go:213] the server was unable to return a response in the time allotted, but may still be processing the request (get azureidentitybindings.aadpodidentity.k8s.io)
E1024 23:32:20.489676       1 crd.go:213] the server was unable to return a response in the time allotted, but may still be processing the request (get azureidentitybindings.aadpodidentity.k8s.io)
E1024 23:32:38.783960       1 reflector.go:322] github.com/Azure/aad-pod-identity/pkg/crd/crd.go:155: Failed to watch *v1.AzureIdentity: the server has asked for the client to provide credentials (get azureidentities.aadpodidentity.k8s.io)
E1024 23:33:20.491873       1 crd.go:213] the server was unable to return a response in the time allotted, but may still be processing the request (get azureidentitybindings.aadpodidentity.k8s.io)
E1024 23:33:39.793512       1 reflector.go:205] github.com/Azure/aad-pod-identity/pkg/crd/crd.go:155: Failed to list *v1.AzureIdentity: the server was unable to return a response in the time allotted, but may still be processing the request (get azureidentities.aadpodidentity.k8s.io)
E1025 13:56:47.884019       1 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: no kind "Status" is registered for version "v1"
W1025 13:56:47.884077       1 reflector.go:341] github.com/Azure/aad-pod-identity/pkg/crd/crd.go:154: watch of *v1.AzureIdentityBinding ended with: very short watch: github.com/Azure/aad-pod-identity/pkg/crd/crd.go:154: Unexpected watch close - watch lasted less than a second and no items received
E1025 14:01:16.761536       1 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: no kind "Status" is registered for version "v1"
W1025 14:01:16.761636       1 reflector.go:341] github.com/Azure/aad-pod-identity/pkg/crd/crd.go:155: watch of *v1.AzureIdentity ended with: very short watch: github.com/Azure/aad-pod-identity/pkg/crd/crd.go:155: Unexpected watch close - watch lasted less than a second and no items received

I noticed this error after trying to use AAD in a pod on the cluster and failed.
Is this error expected?

The current e2e tests are always executed against docker images already published, instead of using the ones generated as part of the build process. Impairing the ability to fully test changes either locally or as part of a build pipeline.

Steps to Reproduce

1. Clone repo and get dependencies

git clone [email protected]:Azure/aad-pod-identity.git
go get -v -t -d ./...

2. Configuration Setup
   As per test\e2e\README.md, including set REGISTRY=my-docker-hub-registry.

3. Break the code
   Remove the RUN command from all the Dockerfiles in the project. Things such as iptables, curl and bash will not be installed into the target images.

4. Kick-off build/e2e process

make build
make image
make push
make e2e

Result

The tests pass, although they should not:

Ran 4 of 4 Specs in 92.214 seconds
SUCCESS! -- 4 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestAADPodIdentity (92.11s)
PASS
ok  	github.com/Azure/aad-pod-identity/test/e2e	92.218s

Images deployed into the cluster:

chewong/identityvalidator:1.0
mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.2
mcr.microsoft.com/k8s/aad-pod-identity/mic:1.2

Proposed changes

• Template all the yaml files used to deploy components for the E2E tests. Add placeholders for registry and docker version tag.
• Change Makefile to be able to consume version ids from environment variables.
• Change e2e code to apply registry and version tags before deploy component on the cluster.

Any thoughts on this @rite2nikhil @khenidak @ritazh ?

Raising this issue to track the review comments from #100

• Add the retry sleep interval as option.
• Add UT for the call.

Raising an issue to keep track of release criterion checklist:

Release criterion:

• Feature complete for requirements for the release.
• No known blocker bugs.
• CI/CD – UTs with good code coverage
• CI/CD – e2e tests with good code coverage.
• Peer code review
• Images in the right repositories (not individual docker accounts)
• Review of log messages to see if there are appropriate ones to figure out if we can efficiently debug customer issues quickly.
• Peer team bug bash
• Documentation sufficient for customers to run effectively (document any manual steps involved).
• Restart/Reboot scenarios handled deterministically.
• Customer feedback incorporation from previews with priority and bugs marked appropriately.

Privacy

• Review logs to check if any credentials or private information from being printed.

Security

• Ensure the RBAC rules are sufficiently restrictive.

Dependent product enablement

• Add as add-on to acs-engine
• Ensure acs-engine images are coming from the right repositories.
• Add as AKS add-on.

Does this sample work on a non-default namespace? I am getting a 403 error from the demo pod if I do that. The AD Binding seems to be not working. I have set the namespace to the demo application, the infra(AzureIdentity and AzureIdentityBinding). Is there anything else we would need to do?

See kubernetes/kubernetes#46372 for reference and explainations

In short, when iptables is used from within a container, it should always use the -w flag (taken care of by the iptables go package used here) and the container should mount /run/xtables.lock to allow the lock to operate.

Implementation here : kubernetes/kubernetes@0a1b7d9

/cc @khenidak

attempting to run the demo app will result in a bunch of 403 errors as it tries to access a preset subscription, see output{1}, the demo app sets these values as defaults https://github.com/Azure/aad-pod-identity/blob/master/cmd/demo/main.go#L20

Easiest would be to update the deployment resource by setting a "command" and "argument" attribute to override how the container is started and set the appropriate flags there.
https://github.com/Azure/aad-pod-identity/blob/master/deploy/demo/deployment.yaml#L15

Readme, should be updated accordingly.

{1} demoapp output
4-c2x24 time="2018-05-31T16:06:01Z" level=error msg="failed list all vm azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/c1089427-83d3-4286-9f35-5af546a6eb67/resourceGroups/MC_nbhatia-eu-01_eu-1_eastus/providers/Microsoft.Compute/virtualMachines?api-version=2017-12-01: StatusCode=403 -- Original Error: adal: Refresh request failed. Status Code = '403'. Response body: No AzureAssignedIdentity found for pod:default/demo-c6dc8f664-c2x24\n" podip=10.244.1.9 podname=demo-c6dc8f664-c2x24 podnamespace=demo-c6dc8f664-c2x24 time="2018-05-31T16:06:01Z" level=error msg="failed to refresh ServicePrincipalTokenFromMSI using the MSI VM extension, msiEndpoint(http://169.254.169.254/metadata/identity/oauth2/token)" podip=10.244.1.9 podname=demo-c6dc8f664-c2x24 podnamespace=demo-c6dc8f664-c2x24 time="2018-05-31T16:06:01Z" level=error msg="testMSIEndpoint failed, <nil>" podip=10.244.1.9 podname=demo-c6dc8f664-c2x24 podnamespace=demo-c6dc8f664-c2x24 time="2018-05-31T16:06:01Z" level=error msg="failed list all vm azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/c1089427-83d3-4286-9f35-5af546a6eb67/resourceGroups/MC_nbhatia-eu-01_eu-1_eastus/providers/Microsoft.Compute/virtualMachines?api-version=2017-12-01: StatusCode=403 -- Original Error: adal: Refresh request failed. Status Code = '403'. Response body: No AzureAssignedIdentity found for pod:default/demo-c6dc8f664-c2x24\n" podip=10.244.1.9 podname=demo-c6dc8f664-c2x24 podnamespace=demo-c6dc8f664-c2x24 time="2018-05-31T16:06:04Z" level=error msg="failed to refresh ServicePrincipalTokenFromMSI using the MSI VM extension, msiEndpoint(http://169.254.169.254/metadata/identity/oauth2/token)" podip=10.244.1.9 podname=demo-c6dc8f664-c2x24 podnamespace=demo-c6dc8f664-c2x24 time="2018-05-31T16:06:04Z" level=error msg="testMSIEndpoint failed, <nil>" podip=10.244.1.9 podname=demo-c6dc8f664-c2x24 podnamespace=demo-c6dc8f664-c2x24 time="2018-05-31T16:06:04Z" level=error msg="failed list all vm azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/c1089427-83d3-4286-9f35-5af546a6eb67/resourceGroups/MC_nbhatia-eu-01_eu-1_eastus/providers/Microsoft.Compute/virtualMachines?api-version=2017-12-01: StatusCode=403 -- Original Error: adal: Refresh request failed. Status Code = '403'. Response body: No AzureAssignedIdentity found for pod:default/demo-c6dc8f664-c2x24\n" podip=10.244.1.9 podname=demo-c6dc8f664-c2x24 podnamespace=demo-c6dc8f664-c2x24 ^C

• Dev
  • Create azureIdentity azureIdentityBinding and azureAssignedIdentity CRDs
  • Create exported go types that represents the CRDs
  • Support service principal as azureIdentity with secretRef to kubernetes secret
  • MI Controller
    • Assign EMSI Identity to Node control loop
    • Bind pod <-> identity control loop
    • Use labels for the match with binding
    • Reflect Binding changes - remove the corresponding assigned identity.
    • Handle, container deletion when controller is down.
    • Events when an identity is assigned or deleted.
    • Create a script which creates a deployment.
    • Make it easy for the admin to create binding.
  • Node Managed Identity
    • Abstract Azure Managed Identity endpoint
    • Assert requests based on source ip -> pod -> azureAssignedIdentity
    • Support service principals
    • Respond to pod calls (only pod Cidr).
    • Make IP Table look check in a loop
• Deployment
  • Create containers for MI Controller and Node Managed Identity
  • Create spec for deployment to kubernetes clusters
  • Redirect traffic for managed id endpoint
• Docs
  • Expand design docs
  • User manual
  • Create demo application
• Test
  • Unit tests
  • CI/CD

After deploying this solution, one of the Node Managed Identity pods is reporting the following:

level=info msg="matched identityType:1 tenantid: clientid:(REDACTED)" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=(REDACTED)
level=error msg="failed to get service principal token for pod:default/akv-test-azure-ad-deployment-5b95bff488-zdzdz, adal: Refresh request failed. Status Code = '404'. Response body: " req.method=GET req.path=/metadata/identity/oauth2/token req.remote=(REDACTED)
level=info msg="Status (403) took 2800821427 ns" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=(REDACTED)

For the nikhilbh/nmi:1.0 image used for the nmi deployment, I'm seeing the following error (Kubernetes):

container_linux.go:247: starting container process caused "exec: "--host-ip=x.x.x.x": executable file not found in $PATH"

Note: the IP address is populated, I just removed it from the error message

Running this container locally on my host, I can see that the nmi bin is indeed not located in /bin.

docker run --rm -it nikhilbh/nmi:1.0 ls -la /bin

Looking at the Dockerfile for this image, it looks like it should be present there.

I tried to deploy a pod with an ASP.NET Core API. It is not possible to get an access token using AzureServiceTokenProvider:
System.AggregateException: One or more errors occurred. (Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: . Exception Message: Tried the following 3 methods to get an access token, but none of them worked.
Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: . Exception Message: Tried to get token using Managed Service Identity. Unable to connect to the Managed Service Identity (MSI) endpoint. Please check that you are running on an Azure resource that has MSI setup.
Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: . Exception Message: Tried to get token using Visual Studio. Access token could not be acquired. Environment variable LOCALAPPDATA not set.
Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: . Exception Message: Tried to get token using Azure CLI. Access token could not be acquired. /bin/bash: az: No such file or directory

) ---> Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProviderException: Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: . Exception Message: Tried the following 3 methods to get an access token, but none of them worked.
Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: . Exception Message: Tried to get token using Managed Service Identity. Unable to connect to the Managed Service Identity (MSI) endpoint. Please check that you are running on an Azure resource that has MSI setup.
Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: . Exception Message: Tried to get token using Visual Studio. Access token could not be acquired. Environment variable LOCALAPPDATA not set.
Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: . Exception Message: Tried to get token using Azure CLI. Access token could not be acquired. /bin/bash: az: No such file or directory

at Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProvider.GetAccessTokenAsyncImpl(String authority, String resource, String scope)
at Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProvider.GetAccessTokenAsync(String resource, String tenantId)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at aspnetcoreaadpodidentity.Controllers.EchoController.GetMsiEndpoint() in /app/Controllers/EchoController.cs:line 55
---> (Inner Exception #0) Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProviderException: Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: . Exception Message: Tried the following 3 methods to get an access token, but none of them worked.
Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: . Exception Message: Tried to get token using Managed Service Identity. Unable to connect to the Managed Service Identity (MSI) endpoint. Please check that you are running on an Azure resource that has MSI setup.
Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: . Exception Message: Tried to get token using Visual Studio. Access token could not be acquired. Environment variable LOCALAPPDATA not set.
Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: . Exception Message: Tried to get token using Azure CLI. Access token could not be acquired. /bin/bash: az: No such file or directory

Azure Dev Spaces (AZDS) is a tool in preview for remote debugging and development in AKS. See this getting started page

AZDS uses an init container and proxy container, which seem to interfere with the operation of NMI pods.

Steps to reproduce:

1. Deploy AAD Pods infrastructure as normal.

2. Enable Dev Spaces

az aks use-dev-spaces -g RESOURCEGROUP -n K8SNAME

Result: MIC pods work fine, but NMI pods crash. Have been unable to get the logs so far.

$ kubectl describe pod  nmi-b8chg
Name:           nmi-b8chg
Namespace:      default
Node:           aks-agentpool-40778490-1/10.240.0.4
Start Time:     Thu, 14 Jun 2018 10:03:10 +1000
Labels:         component=nmi
                controller-revision-hash=4155098794
                pod-template-generation=1
                tier=node
Annotations:    <none>
Status:         Running
IP:             10.240.0.4
Controlled By:  DaemonSet/nmi
Init Containers:
  mindaro-proxy-init:
    Container ID:   docker://8ec99e0277974e48222fd5c39dfbdf163c838e89ad44545447b641497a733c06
    Image:          mindaro/azds-proxy-init:20180531.2
    Image ID:       docker-pullable://mindaro/azds-proxy-init@sha256:a429bb0b0a06f1a9b4442dad94b39b8eb4b0b5b8211c2304ccda287ffbc46879
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 14 Jun 2018 10:03:13 +1000
      Finished:     Thu, 14 Jun 2018 10:03:14 +1000
    Ready:          True
    Restart Count:  1
    Environment:
      BASE_NAMESPACES:
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-rsnk8 (ro)
Containers:
  nmi:
    Container ID:  docker://f9de876f6e138e648bd5ce7bd03784541e81f7b56720af24181f40e52aec1d49
    Image:         nikhilbh/nmi:1.0
    Image ID:      docker-pullable://nikhilbh/nmi@sha256:d24f3c8beb84f4b72f9b48e725615703229cd098b598c68c2bf47a57385cbdf1
    Port:          <none>
    Host Port:     <none>
    Args:
      --host-ip=$(HOST_IP)
      --node=$(NODE_NAME)
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 14 Jun 2018 10:08:58 +1000
      Finished:     Thu, 14 Jun 2018 10:08:58 +1000
    Ready:          False
    Restart Count:  6
    Environment:
      HOST_IP:     (v1:status.podIP)
      NODE_NAME:   (v1:spec.nodeName)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-rsnk8 (ro)
  mindaro-proxy:
    Container ID:   docker://b4409ead04bcfa494ab5da9a0db8276e97f4887689e41e0d7d936f7915496261
    Image:          mindaro/azds-proxy:20180531.2
    Image ID:       docker-pullable://mindaro/azds-proxy@sha256:0721d607bc54d98e287fdabd91c50a8c8b97be3d82946b939ef6cbb82527d55d
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Thu, 14 Jun 2018 10:08:58 +1000
      Finished:     Thu, 14 Jun 2018 10:08:58 +1000
    Ready:          False
    Restart Count:  6
    Environment:
      NAMESPACE:                 default
      TRACE_ENDPOINT:            https://azds-3b85dd70-9e28-40ec-9969-c36fc7d71bde.eus.azds.io/api/traces
      TRACE_VERSION_KEY_HEADER:  x-azds-version-key
      TRACE_VERSION_KEY:         0.1
      TRACE_AUTH_KEY_HEADER:     x-azds-auth-key
      TRACE_AUTH_KEY:           XXXXX
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-rsnk8 (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
Volumes:
  default-token-rsnk8:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-rsnk8
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/unreachable:NoExecute
Events:
  Type     Reason                 Age                From                               Message
  ----     ------                 ----               ----                               -------
  Normal   SuccessfulMountVolume  5m                 kubelet, aks-agentpool-40778490-1  MountVolume.SetUp succeeded for volume "default-token-rsnk8"
  Normal   Pulled                 5m (x2 over 5m)    kubelet, aks-agentpool-40778490-1  Container image "mindaro/azds-proxy-init:20180531.2" already present on machine
  Normal   Created                5m (x2 over 5m)    kubelet, aks-agentpool-40778490-1  Created container
  Normal   Started                5m (x2 over 5m)    kubelet, aks-agentpool-40778490-1  Started container
  Normal   Created                5m (x2 over 5m)    kubelet, aks-agentpool-40778490-1  Created container
  Normal   Pulled                 5m (x2 over 5m)    kubelet, aks-agentpool-40778490-1  Successfully pulled image "nikhilbh/nmi:1.0"
  Normal   Pulling                5m (x2 over 5m)    kubelet, aks-agentpool-40778490-1  pulling image "nikhilbh/nmi:1.0"
  Normal   Started                5m (x2 over 5m)    kubelet, aks-agentpool-40778490-1  Started container
  Normal   Pulled                 5m (x2 over 5m)    kubelet, aks-agentpool-40778490-1  Container image "mindaro/azds-proxy:20180531.2" already present on machine
  Normal   Created                5m (x2 over 5m)    kubelet, aks-agentpool-40778490-1  Created container
  Normal   Started                5m (x2 over 5m)    kubelet, aks-agentpool-40778490-1  Started container
  Warning  BackOff                5m (x2 over 5m)    kubelet, aks-agentpool-40778490-1  Back-off restarting failed container
  Warning  BackOff                54s (x27 over 5m)  kubelet, aks-agentpool-40778490-1  Back-off restarting failed container

The docs state you can also use an SP as an identity. is there an example of this. my reason for the question is its not clear what should be the ResourceID. a SP doesn't have one as far as i know.
https://github.com/Azure/aad-pod-identity#install-user-azure-identity-on-k8s-cluster

Let me premise this by saying I followed the README and got into this position. The root cause may be a bug, but more likely is missing or misleading documentation.

ENV:

• AKS
• eastus
• 1 node

Deployed the Infra and the demo, plus the bindings, as described in the README.

Looks like the MIC pod is unable to assign the identity.

Cannot perform action Microsoft.ManagedIdentity/userAssignedIdentities/assign/action.

computer:~$ kubectl logs -f mic-64ddcf5f65-h4hft
I0606 05:34:42.676870       1 mic.go:28] kubeconfig (/etc/kubernetes/kubeconfig/kubeconfig) cloudconfig (/etc/kubernetes/azure.json)
I0606 05:34:42.678773       1 mic.go:36] Starting to create the pod identity client
I0606 05:34:42.682479       1 mic.go:82] Pod watcher started !!
I0606 05:34:42.682533       1 mic.go:84] CRD watcher started
I0606 05:34:42.682558       1 mic.go:42] AAD Pod identity controller initialized!!
I0606 05:34:42.682584       1 mic.go:140] Sync thread started
I0606 05:34:44.956703       1 crd.go:123] Got id podid to assign
I0606 05:34:44.956798       1 crd.go:142] Creating assigned Id: demo-5788d95785-pwk7g-default-podid
I0606 05:34:44.965067       1 cloudprovider.go:170] Find aks-nodepool1-15831963-0 in resource group: MC_k8s-test_clusterFrank_eastus
E0606 05:34:46.288109       1 cloudprovider.go:147] compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=403 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client 'd6f4dd69-b093-45cd-833d-298be532685d' with object id 'd6f4dd69-b093-45cd-833d-298be532685d' has permission to perform action 'Microsoft.Compute/virtualMachines/write' on scope '/subscriptions/c5760548-23c2-4223-b41e-5d68a8320a0c/resourceGroups/MC_k8s-test_clusterFrank_eastus/providers/Microsoft.Compute/virtualMachines/aks-nodepool1-15831963-0'; however, it does not have permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptions/c5760548-23c2-4223-b41e-5d68a8320a0c/resourcegroups/k8s-test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/podId'."
E0606 05:34:46.288154       1 mic.go:365] compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=403 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client 'd6f4dd69-b093-45cd-833d-298be532685d' with object id 'd6f4dd69-b093-45cd-833d-298be532685d' has permission to perform action 'Microsoft.Compute/virtualMachines/write' on scope '/subscriptions/c5760548-23c2-4223-b41e-5d68a8320a0c/resourceGroups/MC_k8s-test_clusterFrank_eastus/providers/Microsoft.Compute/virtualMachines/aks-nodepool1-15831963-0'; however, it does not have permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptions/c5760548-23c2-4223-b41e-5d68a8320a0c/resourcegroups/k8s-test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/podId'."
I0606 05:34:46.288231       1 mic.go:259] Sync took: 1.356004792s

Do these permissions need to be set somewhere?

Currently Helm package does not offer lot of value over just providing the deployment-rbac.yaml. It should provide the needed files to work as a Helm repository and function like normal Helm package. Similar to https://github.com/Azure/application-gateway-kubernetes-ingress/tree/master/helm

I have a weird situation coming up now where one of my pods can't get an identity. It gets an error 403 error {"error":"invalid_request","error_description":"Identity not found"}

Running a kubectl get AzureAssignedIdentity I do see an identity in there however looking at the mic logs I see:

1 crd.go:174] Got id token-issuer-identity to assign
I0727 15:49:47.694207 1 crd.go:194] Creating assigned Id: token-issuer-9fcdfdcfc-4g6dv-default-token-issuer-identity
I0727 15:49:47.928282 1 cloudprovider.go:147] Find aks-agentpool-41851317-1 in resource group: MC_dev-kitt_dev-kitt_eastus
E0727 15:49:49.137022 1 vm.go:42] compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidParameter" Message="ResourceIdentity identityId 'XXXRedactedXXX' is not a valid reference to a Microsoft.ManagedIdentity identity." Target="resourceIdentity.identityIds"
E0727 15:49:49.137065 1 mic.go:385] compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidParameter" Message="ResourceIdentity identityId 'XXXRedactedXXX' is not a valid reference to a Microsoft.ManagedIdentity identity." Target="resourceIdentity.identityIds"

What I don't understand though is the resource identity clearly appears when I run an az identity list. I've copied the resource id from the id in that result and they match. I've also confirmed the client id is the same. Everything is set up the same for this deployment as the others where everything works except for a different Azure identity is being used.

What would cause the mic to be unable to find the identity even though I can find it?

Any suggestions on how to troubleshoot this?

Thanks!

My team has a k8s environment with microservices distributed across various namespaces. Based on our observations, it seems like the AzureIdentity and AzureIdentityBinding both need to be applied to the default namespace where the MIC and NMI pods are running, as well as into the namespaces of the microservices that are using the pod identity. Is this the correct approach? Or, are there simpler ways to configure our service?

We've observed our mic controller periodically crashes with a panic. In looking at the logs after we see this stack trace:

panic: runtime error: index out of range

goroutine 38 [running]:
github.com/Azure/aad-pod-identity/pkg/cloudprovider.(*Client).RemoveUserMSI(0xc4202c8380, 0xc4208f58c0, 0xb3, 0xc420711240, 0x18, 0x0, 0x0)
/home/nikhil/go/src/github.com/Azure/aad-pod-identity/pkg/cloudprovider/cloudprovider.go:111 +0xafd
github.com/Azure/aad-pod-identity/pkg/mic.(*Client).RemoveAssignedIDsWithDeps(0xc4202be690, 0xc4205ecc48, 0x0, 0xc420710a20, 0x18)
/home/nikhil/go/src/github.com/Azure/aad-pod-identity/pkg/mic/mic.go:401 +0x156
github.com/Azure/aad-pod-identity/pkg/mic.(*Client).Sync(0xc4202be690, 0xc4200689c0)
/home/nikhil/go/src/github.com/Azure/aad-pod-identity/pkg/mic/mic.go:223 +0x1c2a
created by github.com/Azure/aad-pod-identity/pkg/mic.(*Client).Start
/home/nikhil/go/src/github.com/Azure/aad-pod-identity/pkg/mic/mic.go:94 +0x16b

Looking at the code it appears on line 111 in cloudprovider.go it tries to assign a value in an empty slice. From my read of the code there is no way this will ever work so if you get into this if block it will crash as the slice hasn't been initialized anywhere. I'm wondering if what you really want to do is use the append function which can auto grow the slice.

I don't have enough context to understand when we do get into this versus when we don't or what the code is actually trying to accomplish. We can observe things work sometimes but we do crash regularly enough and see intermittent issues that this must be a case that occurs regularly needs to be fixed.

Expected the aad metadata rules to be deleted from worker nodes' ip tables after deleting the CRDs (i.e. kubectl delete -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml).
Command executed in worker nodes: sudo iptable -L -t nat

Ref: #106 #45 and other

Today, Identities are hardcoded to default namespace. and pods are matched using selector irrespective of pod.namespace. The is causing the following:

1. Users would like to compartmentalize application resources at the namespace level. This includes the identity.
2. This model currently does not allow matching identity.namespace == pod.namespace

Proposed change:

1. Remove the hardcoded selector in https://github.com/Azure/aad-pod-identity/blob/master/pkg/crd/crd.go#L128 (along with any reference to default namespace in listXXX calls)
2. Add annotation to azureIdentity to force matching to current namespace only. i.e true = (pod.namespace == azureIdentity.namespace) When the annotation is not found then we match in any namespace.

@kkmsft @rite2nikhil @ritazh Thoughts?

Adding a validation web hook

• AzureIdentity is created
• AzureIdentityBinding is created

README mentions that it works on acs-engine but I believe RBAC is enabled by default in acs-engine and deployment results in

time="2018-06-06T18:38:24Z" level=fatal msg="nodes \"k8s-agentpool-35881120-vmss000000\" is forbidden: User \"system:serviceaccount:default:default\" cannot get nodes at the cluster scope"

Deployment should have a service account that has permission to get nodes.

Namespace parameter is ignored when looking up CRD type "azureassignedidentities" in the method GetUserAssignedMS. only referenced when a error occurs.
this results in the aad-pod-identity being only useable in the default namespace
[1]
https://github.com/Azure/aad-pod-identity/blob/master/pkg/crd/crd.go#L190

Our k8s cluster is experiencing issues where services will intermittently fail to authenticate with AAD. Closer inspection of the virtual machine that hosted the failing service will show that the user-assigned managed identity is missing. Manually re-adding the managed identity onto the virtual machine resolves the issue. However, it is unclear how individual virtual machines in our cluster are getting into this broken state.

Hi,

There are multiple scenarios where you might want to give different identities to different containers in a pod.

For example, if you have an init-container that does some work that requires special permissions, you don't necessarily want to give that permission to the containers in the pod.

Is it possible to assign the identity at a container level? is it something we can support?

Idan

Please create an image based on alpine or similar small docker image. hyperkube is an overkill

@ritazh @kkmsft @rite2nikhil

We are experiencing a getting tokens from the nmi because the first call directly after pod startup fails:
level=error msg="missing podname for podip:10.244.0.103, not found"

This error is similiar to what has been reported in issue #84

After a short period of time the next call to the nmi works fine so I believe that a retry on the func (c *KubeClient) GetPodName(podip string) would solve the problem.

I set up an AKS cluster with AAD integration as described here:

https://docs.microsoft.com/en-us/azure/aks/aad-integration

This allows us to secure manual administrator access to the cluster but we also need to allow a management process running in a pod on the cluster access to submit other pods to the cluster. We were looking to do this by running the management pod with a pod identity/MSI that has access to the cluster via AAD group. I setup pod identity as described here https://github.com/Azure/aad-pod-identity#deploy-the-azure-aad-identity-infra and hooked up my management pod with the AzureIdentityBinding. Within the pod I login as the identity and retrieve the kubeconfig for my cluster:

az login --identity
az aks get-credentials --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME

And then run a python script to submit the child pod to the cluster:

from kubernetes import client, config, watch

config.load_kube_config()
kub_client = client.CoreV1Api()

spec = {'apiVersion' : 'v1',
        'kind' : 'Pod',
        'metadata' : {'name' : 'testpod'},
        'spec' : {'containers' : [{'image' : 'ubuntu:16.04',
                                   'command' : ['/bin/bash', '-c', 'sleep 15m'],
                                   'name' : 'testcontainer'}]}}

kub_client.create_namespaced_pod('default', spec)

The create_namespaced_pod call fails with this error:

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User "system:anonymous" cannot create pods in the namespace "default"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

Should we be able to non-interactively access a pod that has AAD integration enabled?

I tried the same setup on a cluster without AAD integration and the child pod is able to be submitted as expected utilizing the pod identity MSI.

Hi,

I just want to be sure that I get the functionality right...

So aad-pod-identity enables me to have a separate identity per application which is cool.
MIC uses cluster service principal to access Azure resources (including Managed Identities) so I assume that cluster service principal (secret) is still accessible on all the nodes correct?
Having access to that I could access all other identities and escalate.

Do I miss something?

Thanks!

I have a MIC pod deployed.

$ kubectl describe pod mic-64ddcf5f65-nz86s
Name:           mic-64ddcf5f65-nz86s
Namespace:      default
Node:           aks-agentpool-26209738-0/10.240.0.4
Start Time:     Thu, 14 Jun 2018 15:25:05 +1000
Labels:         component=mic
                pod-template-hash=2088791921
Annotations:    <none>
Status:         Running
IP:             10.244.0.25
Controlled By:  ReplicaSet/mic-64ddcf5f65
Containers:
  mic:
    Container ID:  docker://3c9f9570b87c0448dcdf21df67d42a0bce4b6fc44e1e0053c4eae2ad36701a33
    Image:         nikhilbh/mic:1.0
    Image ID:      docker-pullable://nikhilbh/mic@sha256:aacf21946398d7d9d7536e0cb185bae3ac4d92d1a8d80e95ce6b39b1f8642ba3
    Port:          <none>
    Host Port:     <none>
    Args:
      --kubeconfig=/etc/kubernetes/kubeconfig/kubeconfig
      --cloudconfig=/etc/kubernetes/azure.json
      --logtostderr
    State:          Running
      Started:      Thu, 14 Jun 2018 15:25:15 +1000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/kubernetes/azure.json from k8s-azure-file (ro)
      /etc/kubernetes/certs from certificates (ro)
      /etc/kubernetes/kubeconfig from kubeconfig (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-w5gz9 (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          True
  PodScheduled   True
Volumes:
  kubeconfig:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet
    HostPathType:
  certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/certs
    HostPathType:
  k8s-azure-file:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/azure.json
    HostPathType:
  default-token-w5gz9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-w5gz9
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason                 Age   From                               Message
  ----    ------                 ----  ----                               -------
  Normal  Scheduled              6m    default-scheduler                  Successfully assigned mic-64ddcf5f65-nz86s to aks-agentpool-26209738-0
  Normal  SuccessfulMountVolume  6m    kubelet, aks-agentpool-26209738-0  MountVolume.SetUp succeeded for volume "k8s-azure-file"
  Normal  SuccessfulMountVolume  6m    kubelet, aks-agentpool-26209738-0  MountVolume.SetUp succeeded for volume "certificates"
  Normal  SuccessfulMountVolume  6m    kubelet, aks-agentpool-26209738-0  MountVolume.SetUp succeeded for volume "kubeconfig"
  Normal  SuccessfulMountVolume  6m    kubelet, aks-agentpool-26209738-0  MountVolume.SetUp succeeded for volume "default-token-w5gz9"
  Normal  Pulling                6m    kubelet, aks-agentpool-26209738-0  pulling image "nikhilbh/mic:1.0"
  Normal  Pulled                 6m    kubelet, aks-agentpool-26209738-0  Successfully pulled image "nikhilbh/mic:1.0"
  Normal  Created                6m    kubelet, aks-agentpool-26209738-0  Created container
  Normal  Started                6m    kubelet, aks-agentpool-26209738-0  Started container

Also a Identity and binding

$ kubectl describe azureidentitybinding
Name:         rianid-binding
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"aadpodidentity.k8s.io/v1","kind":"AzureIdentityBinding","metadata":{"annotations":{},"name":"rianid-binding","namespace":"default"},"spe...
API Version:  aadpodidentity.k8s.io/v1
Kind:         AzureIdentityBinding
Metadata:
  Cluster Name:
  Creation Timestamp:  2018-06-14T05:25:44Z
  Resource Version:    40188
  Self Link:           /apis/aadpodidentity.k8s.io/v1/namespaces/default/azureidentitybindings/rianid-binding
  UID:                 6322f0c9-6f93-11e8-ba7a-7ee6c52fd58c
Spec:
  Azure Identity:  rianid
  Selector:        tester
Events:            <none>

$ kubectl describe azureidentity
Name:         rianid
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"aadpodidentity.k8s.io/v1","kind":"AzureIdentity","metadata":{"annotations":{},"name":"rianid","namespace":"default"},"spec":{"ClientID":...
API Version:  aadpodidentity.k8s.io/v1
Kind:         AzureIdentity
Metadata:
  Cluster Name:
  Creation Timestamp:  2018-06-14T05:25:40Z
  Resource Version:    40181
  Self Link:           /apis/aadpodidentity.k8s.io/v1/namespaces/default/azureidentities/rianid
  UID:                 60bba85e-6f93-11e8-ba7a-7ee6c52fd58c
Spec:
  Client ID:    67404446-b013-4a8f-ba7f-6222575f9983
  Resource ID:  /subscriptions/e39a92b5-b9a4-43d1-97a3-c31c819a583a/resourcegroups/MC_istiotest_jordosydneyaks_australiaeast/providers/Microsoft.ManagedIdentity/userAssignedIdentities/rianid
  Type:         0
Events:         <none>

But looking at the MIC logs, it hasn't done what I expected, or anything useful for that matter.

$ kubectl logs -f mic-64ddcf5f65-nz86s
I0614 05:25:15.083355       1 mic.go:28] kubeconfig (/etc/kubernetes/kubeconfig/kubeconfig) cloudconfig (/etc/kubernetes/azure.json)
I0614 05:25:15.085260       1 mic.go:36] Starting to create the pod identity client
I0614 05:25:15.089427       1 mic.go:82] Pod watcher started !!
I0614 05:25:15.089478       1 mic.go:84] CRD watcher started
I0614 05:25:15.089492       1 mic.go:42] AAD Pod identity controller initialized!!
I0614 05:25:15.089509       1 mic.go:140] Sync thread started
[>WAITING HERE FOR 10 MINUTES<]

There are also events showing from this command, could that be related?

$ kubectl get events --field-selector=involvedObject.kind=azureidentitybinding
No resources found.

I am trying to use the kubernetes-keyvault-flexvol plugin on a k8s cluster deployed using acs-engine. The user assigned identity binding is created and the user assigned identity (UAI) is assigned to the node the pod is running on. When kubernetes-keyvault-flexvol tries to access the secret, I receive the following error in the nmi pod.

ERROR: logging before flag.Parse: E1008 19:57:57.185131       1 crd.go:250] unknown (get azureassignedidentities.aadpodidentity.k8s.io)
time="2018-10-08T19:57:57Z" level=error msg="no AzureAssignedIdentity found for pod:default/nginx-flex-kv-podid, unknown (get azureassignedidentities.aadpodidentity.k8s.io)" req.method=GET req.path=/host/token/ req.remote=127.0.0.1

The pod does not successfully spin up and shows the following error: MountVolume.SetUp failed for volume "test" : mount command failed, status: Failure, reason: /etc/kubernetes/volumeplugins/azure~kv/azurekeyvault-flexvolume failed, failed to get key vault

Repro steps:

acs-engine version

$ acs-engine version
Version: v0.22.3
GitCommit: ae61d3bfa
GitTreeState: clean

Set the following environment variables

SUB_ID=[AZURE_SUB_ID]
TENANT_ID=[AZURE_TENANT_ID]
ACS_RG=[RESOURCE_GROUP_NAME]
KV_RG="$ACS_RG-kv-rg"
KV_NAME="$ACS_RG-kv"
KV_SECRET_NAME=foo
UAI_NAME="$KV_NAME-uai"

kubernetes.json

Leave servicePrincipalProfile empty and let acs-engine generate it. Disable flex volumes addons and deploy manually in next steps

{
    "apiVersion": "vlabs",
    "properties": {
        "orchestratorProfile": {
            "orchestratorType": "Kubernetes",
            "orchestratorRelease": "1.11",
            "kubernetesConfig": {
                "addons": [
                    {
                        "name": "keyvault-flexvolume",
                        "enabled": false
                    },
                    {
                        "name": "blobfuse-flexvolume",
                        "enabled": false
                    },
                    {
                        "name": "smb-flexvolume",
                        "enabled": false
                    }
                ]
            }
        },
        "masterProfile": {
            "count": 1,
            "dnsPrefix": "",
            "vmSize": "Standard_D2_v2"
        },
        "agentPoolProfiles": [
            {
                "name": "agentpool1",
                "count": 1,
                "vmSize": "Standard_D2_v2",
                "availabilityProfile": "AvailabilitySet"
            }
        ],
        "linuxProfile": {
            "adminUsername": "azureuser",
            "ssh": {
                "publicKeys": [
                    {
                        "keyData": ""
                    }
                ]
            }
        },
        "servicePrincipalProfile": {
            "clientId": "",
            "secret": ""
        }
    }
}

Deploy

# Create k8s cluster with acs-engine
acs-engine deploy --subscription-id $SUB_ID \
--dns-prefix $ACS_RG \
--api-model kubernetes.json \
--location eastus \
--output-directory _output/$ACS_RG

# Get service principal ID for acs-engine cluster
CLUSTER_SP_ID=$(az ad sp list --display-name $ACS_RG --query "[?appDisplayName=='$ACS_RG'] | [0].appId" -o tsv)

# Create key vault in a separate resource group
az group create -n $KV_RG -l eastus
az keyvault create -g $KV_RG -n $KV_NAME
KV_ID=$(az keyvault show -g $KV_RG -n $KV_NAME --query id -o tsv)

# Create test secret
az keyvault secret set --vault-name $KV_NAME --name $KV_SECRET_NAME --value bar
KV_SECRET_VERSION=$(az keyvault secret show --vault-name $KV_NAME --name $KV_SECRET_NAME --query id -o tsv | awk '{n=split($0,a,"/"); print a[n]}')

# Create user assigned identity in the key vault rg
az identity create -g $KV_RG -n $UAI_NAME
UAI_ID=$(az identity show -g $KV_RG -n $UAI_NAME --query id -o tsv)
UAI_CLIENT_ID=$(az identity show -g $KV_RG -n $UAI_NAME --query clientId -o tsv)
UAI_PRINCIPAL_ID=$(az identity show -g $KV_RG -n $UAI_NAME --query principalId -o tsv)

# Assign UAI the Reader Role on key vault
az role assignment create --role Reader --assignee $UAI_PRINCIPAL_ID --scope $KV_ID

# set policy to access secrets in your keyvault
az keyvault set-policy -n $KV_NAME --secret-permissions get list --spn $UAI_CLIENT_ID

# Let cluster service principal operate on behalf of the user assigned identity
az role assignment create --role "Managed Identity Operator" --assignee $CLUSTER_SP_ID --scope $UAI_ID

# Deploy aad-pod-identity infrastructure
kubectl create -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment.yaml

# Deploy KeyVault Flex volume infrastructure
kubectl create -f https://raw.githubusercontent.com/Azure/kubernetes-keyvault-flexvol/master/deployment/kv-flexvol-installer.yaml

# Deploy aad identity and binding
AAD_POD_IDENTITY_FILE_NAME=aadpodidentity.yaml

cat <<EOT >> $AAD_POD_IDENTITY_FILE_NAME
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentity
metadata:
 name: $UAI_NAME
spec:
 type: 0
 ResourceID: $UAI_ID
 ClientID: $UAI_CLIENT_ID
---
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
 name: $UAI_NAME-binding
spec:
 AzureIdentity: $UAI_NAME
 Selector: $UAI_NAME-binding-selector
EOT

kubectl apply -f $AAD_POD_IDENTITY_FILE_NAME
kubectl get azureidentity
kubectl get azureidentitybinding

# Deploy test app that needs test secret
DEPLOYMENT_FILE_NAME=deployment.yaml

cat <<EOT >> $DEPLOYMENT_FILE_NAME
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: nginx-flex-kv-podid
    aadpodidbinding: "$UAI_NAME-binding-selector"
  name: nginx-flex-kv-podid
spec:
  containers:
  - name: nginx-flex-kv-podid
    image: nginx
    volumeMounts:
    - name: test
      mountPath: /kvmnt
      readOnly: true
  volumes:
  - name: test
    flexVolume:
      driver: "azure/kv"
      options:
        usepodidentity: "true"
        keyvaultname: "$KV_NAME"
        keyvaultobjectname: "$KV_SECRET_NAME"
        keyvaultobjecttype: secret # OPTIONS: secret, key, cert
        keyvaultobjectversion: "$KV_SECRET_VERSION"
        resourcegroup: "$KV_RG"
        subscriptionid: "$SUB_ID"
        tenantid: "$TENANT_ID"
EOT

kubectl apply -f $DEPLOYMENT_FILE_NAME

/ cc @kkmsft @ritazh

Refer the #69 to improving the logging and error return for this issue.

If the same user identity is set on a node from provisioning software (like acs-engine), there is a chance that aad-pod-identity would remove it if the same id used (in the removal code path of the identity). This bug is raised in order to fix this issue.

Azure Identities/AAD were not meant to just control access to azure own resources. But arbitrary applications running on Azure/other clouds/on-prem as well. Users can register applications on AAD and use AAD tokens to access them. AAD authentication can be proxied to on-prem AD Proxy or others.

Proposal:

• Create a different demo where the target application is not is not ARM or Azure resource but a custom arbitrary web api.
• Add documentation with links to how Azure Identities can be used generally example: https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-v1-dotnet-webapi

Enable installation via helm

Since the switch to alpine the Dockerfiles for both NMI and MIC are missing the package gcompat.