We can't use project level IAMPolicy for our current use case since we need a more patch style behavior for adding policy bindings as discussed earlier with Google.

The suggested solution was introducing more fine grained resources, such as IAMPolicyBinding and IAMPolicyMember. This would be useful for other resource types as well since it is likely that we'd like to declare access to the same resource in multiple source code locations, e.g. a bucket might be declared together with the rest of the manifests for an application that writes to it but we also want to declare a viewer policy binding together with the manifests for another application that reads from it.

I also wonder how Project level IAMPolicy is meant to be used is since it removes "bootstrap type" policy bindings such as "Compute Engine default service account" that you get automatically when creating the project and even any policy bindings used to give Config Connector access to the project. It gets even worse if you at any point delete the IAMPolicy as that leaves you with no Project level access at all. I guess another way of putting it is that a difference between a Project IAMPolicy and an IAMPolicy for some other resource (especially if managed via Config Connector) is that you are very likely to irrevocably overwrite existing stuff for the former.

We need a resource to be able to create/delete cloudsql users (with passwords ideally settable via a secret) so that applications can connect to the database without needing a service account and the cloudsql-proxy configured and running.

Spec:
  Bucket Ref:
    Name:  test
  Entity:  [email protected]
  Role:    OWNER
Status:
  Conditions:
    Last Transition Time:  2019-09-13T08:47:28Z
    Message:               Delete call failed: googleapi: Error 400: Unknown user email address: [email protected], invalid
    Reason:                DeleteFailed
    Status:                False
    Type:                  Ready
  Email:                   [email protected]
  Id:                      test/[email protected]

The [email protected] ServiceAccount has been deleted, so the StorageBucketAccessControl becomes useless at this point, but it cannot be deleted.

would be nice to be able create users with custom grants.

Backend service requires a named port. There's a separate issue opened on supporting instance groups , this one is track support for named ports configuration as part of supporting instance groups.

I can create an IAM service account and policies, but I can't use it as part of an application without extra work. We should allow the user to provide a namespace / secret to store the credentials json file that can be created for a service account.

see example here - https://github.com/paulczar/gcp-cloud-compute-operator/blob/master/pkg/controller/serviceaccountkey/serviceaccountkey_controller.go

Declarative way to provision global address is part of enabling fully declarative multi-cluster ingress scenario. Adding this issue to track adding this support.

would be nice to be able to customize the Instance, like:

• connectivity type ( Private/ Public IP)
• associated networking ( Private IP )
• authorized networks ( Public IP )

Today it is not possible to use Config Connector to provision firewall rules for default network, which is the primary place of where you would want to configure them. This is also a blocker for multi-cluster ingress scenario, see more details for in Firewall Rule section here: https://www.keeprunning.io/2019/09/08/kubernetes-multi-cluster-ingress-with-config-connector-part3/

We're exploring Config Connector for a multi-tenant cluster, but I saw this limitation:

To function correctly, the GCP project and the destination Kubernetes Namespace must have the same name.

Is it possible to use Config Connector for many namespaces within the same GCP Project?

This may or may not be related to #26

would be nice to have

Tracking support for GlobalForwarding rule that is blocking multi-cluster ingress scenario

TLDR: This is expected that secondary instance cannot be created before primary, but also expected that the controller will not attempt to create the instance, until primary creation is complete and not issue calls that will result in errors.

Try to create 2 SQL instances:

apiVersion: sql.cnrm.cloud.google.com/v1alpha2
kind: SQLInstance
metadata:
  labels:
    label-one: "value-one"
  name: mysql-instance-sample-with-replication-primary
spec:
  databaseVersion: MYSQL_5_7
  region: us-central1
  settings:
    tier: db-f1-micro
    backupConfiguration:
        binaryLogEnabled: true
        enabled: true
        startTime: "18:00"
    ipConfiguration:
      requireSsl: true
    locationPreference:
        zone: us-central1-b
---
apiVersion: sql.cnrm.cloud.google.com/v1alpha2
kind: SQLInstance
metadata:
  labels:
    label-one: "value-one"
  name: mysql-instance-sample-with-replication-secondary
spec:
  databaseVersion: MYSQL_5_7
  region: us-central1
  masterInstanceName: mysql-instance-sample-with-replication-primary
  mysqlReplicaConfiguration:
    connectRetryInterval: 30
  settings:
    tier: db-f1-micro
    ipConfiguration:
      requireSsl: true
    locationPreference:
        zone: us-central1-c

The first instance is created successfully, the second instance is not created and in the event log I can see the following errors, first this one: Could not read response as json for google api sqladmin-v1beta4: reading the error from the request is not implemented, which seems to be common and then another one with more details:

 jsonPayload: {
  controller:  "sqlinstance-controller"   
  error:  "error syncing resource: Create call failed: googleapi: Error 409: Operation failed because another operation was already in progress., operationInProgress"   
  level:  "error"   
  logger:  "kubebuilder.controller"   
  msg:  "Reconciler error"   
  request:  "default/mysql-instance-sample-with-replication-secondary"   
  stacktrace:  "cnrm.googlesource.com/cnrm/vendor/github.com/go-logr/zapr.(*zapLogger).Error
	/go/src/cnrm.googlesource.com/cnrm/vendor/github.com/go-logr/zapr/zapr.go:128
cnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/src/cnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217
cnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
	/go/src/cnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158
cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
	/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.Until
	/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"   
  ts:  1563976375.8204267   
 }

Note that eventually, with the subsequent reconciliation cycle, the second instance is created.
This is expected that secondary instance cannot be created before primary, but also expected that the controller will not attempt to create the instance, until primary creation is complete.

would be nice to be able to create DNS records with kcc.

We added a gcp category, but adding resources to all as well would be nice. Will need to double-check on the latest guidance, since we'd be sharing that space with a bunch of other resources. I think we'll need to filter out certain types that may not fit well, like potentially IAMServiceAccountKey.

To use Config Connector in production we would need it to be installed in a multi-tenant cluster and then we need a way of authorizing Config Connector's GCP SA to create resources in our GCP project but only in specific k8s namespaces.

One suggestion here is to be able to use a member syntax similar to the one used for Workload Identities. For example, say Config Connector uses the GCP SA [email protected] and I have mapped the k8s namespace ns1 to the GCP project gcp_proj_1, then I would add a project IAM policy binding making [email protected][ns1] owner in gcp_proj_1.

I would like to know when support for IAM Conditions is added

It would make a lot of sense to be able to reference a k8s secrets, e.g. as the password of an SQLUser.

Following the documentation at https://cloud.google.com/config-connector/docs/how-to/install-upgrade-uninstall#installing_kcc and issuing

curl -X GET -sLO \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  --location-trusted \
  https://us-central1-cnrm-eap.cloudfunctions.net/download/latest/infra/install-bundle.tar.gz

returns a borken link:

<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: cnrm-eap/0.1.7/infra/install-bundle.tar.gz</Details></Error>

The musicians seemed to be taking a while to deploy and the delay is primarily during the creation of the Cloud SQL instance.

The delay is okay, what am not clear about is whether the driver errors am seeing in the web container are as a result of the delay or if something else is going on.

Note:
I renamed my SQL instance to validate the issue; don't be perplexed by the instance name, musicians-demo2

$ kubectl logs musicians-686886cfd7-wz5sr -c web 
2019/06/18 20:55:57 DB_PASSWORD environment variable unspecified or ''
[mysql] 2019/06/18 20:55:57 packets.go:36: unexpected EOF
[mysql] 2019/06/18 20:55:57 packets.go:36: unexpected EOF
[mysql] 2019/06/18 20:55:57 packets.go:36: unexpected EOF
2019/06/18 20:55:57 Failed to create a database: driver: bad connection
$ gcloud sql instances list
NAME             DATABASE_VERSION  LOCATION       TIER              PRIMARY_ADDRESS  PRIVATE_ADDRESS  STATUS
musicians-demo2  MYSQL_5_7         us-central1-a  db-n1-standard-1  35.226.xxx.xxx   -                RUNNABLE
$ 

In addition, a look at the controller logs seems to confirm my initial suspicions. It does appear the controller is not able to sufficiently determine the state of a Cloud SQL instance.

E0619 15:48:30.874918       1 resource_processor_impl.go:434] Could not read response as json for google api sqladmin-v1beta4: reading the error from the request is not implemented
I0619 15:48:30.884195       1 dynamic_controller.go:101] [SQLInstance default/musicians-demo]: successfully finished reconcile

Lastly, deleting the sql instance is taking a while to complete. Would this be a good task to perform asynchronously to avoid holding the users prompt? Especially given the operator does not return any status from GCP regarding the task.

In one of our current use cases we set up a SQLInstance with an SQLDatabase and we then have a service that will fail startup until the SQLDatabase is available. The SQLInstance takes around 3 minutes to be fully initialized. At that point it seems like cnrm-controller-manager reconcile has started to back off so it takes a couple of minutes before it attempts to create the database and once that happens the k8s pod startup crash loop backoff has increased a fair bit so it takes another couple of minutes before there is another attempt to start the pod.

Unfortunately k8s does not provide a way of configuring the back-off for starting pods so in the end it takes ~10 minutes to start the service when it could have been less than 4 minutes. If cnrm-controller-manager could do a reconcile immediately after a resource with dependent resources enters the "Ready" state this would be much less noticeable.

Requested in multiple customer conversations, so opening GH issue to allow subscribing for updates to get notified when support is implemented

Adding some categories to the Config Connector CRDs would make them nicer to work with using kubectl. I'm thinking the category "all" makes sense as you probably want to see them when doing kubectl get all, but maybe also a category like "gcp" that allows you to list all GCP resources.

The main README.md file should incorporate the instructions from the Samples/App README.md file.

Prerequisites
Before trying any of the samples you must run through the following prerequisites to get your environment configured.

In addition, it is assumed your system has the following.

cnrm
gcloud
kustomize
Create a GKE cluster with CNRM support.

The sample app tries to create a storage bucket storagebucket-sample but bucket names need to be globally unique. This will cause issues with anyone following the guide.

In user guide:

kubectl apply -f samples/apps/bookstore/config/release-configuration.yaml

The file does not exist

@kibbles-n-bytes

As part of configuring UrlMap I was attempting to link existing backend service, however that reference was not recognized, even though backend service was already configured:

kind: ComputeURLMap
metadata:
  name: node-app-url-map
spec:
  defaultServiceRef:
    name: node-app-backend-service
    kind: ComputeBackendService

As part of configuring multi-cluster ingress we need to be able to reference instance groups in a backend service. For that we need to support creating node pools and referencing instance groups from them.
For details please see Node Pools and Instance Groups here: https://www.keeprunning.io/2019/09/05/kubernetes-multi-cluster-ingress-with-config-connector-part2/

By default, the cnrm-controller-manager pod requests 40Mi of memory and limits it to 60Mi. This results in the pods almost always being OOMKilled. At 256Mi the pod looks stable.

We need a way of specifying the name of the underlying GCP resource so that it is different from the k8s resource name. In many cases this not critical but for some resource types the idea of the 1:1 relationship between k8s namespace and GCP project breaks down, e.g. SQLUser - if you have multiple SQLInstance resources in the same namespace there seems to be no way of setting the root password for more than one of them.

There's a few use cases for SQLInstance Resources I don't see in the documentation. How could I accomplish these using Config Connector?
• Creating a new backup run on demand
• Export data from a Cloud SQL instance to a Cloud Storage bucket as a SQL dump or CSV file.
• Failover the instance to its failover replica instance.
• Restart a Cloud SQL instance.

Or would I need to use the regular GCP API?

I'm trying to use config-connector with a k8s cluster associated with projectA to manage resources in projectB. Both are part of the same organization. So far things have worked except for IAMServiceAccountKey.

IAMServiceAccount works to create the service account in projectB, but keys cannot be created. My theory is that IAMServiceAccountKey is either ignoring or not fully specifying the project requested within k8s during the API call.

The IAMServiceAccountKey fails with this error:

Update call failed: Error creating service account key: googleapi:
        Error 403: Permission iam.serviceAccountKeys.create is required to perform
        this operation on service account projects/-/serviceAccounts/[email protected].,
        forbidden'

The part that looks fishy is the projectA in the account email. It should be projectB. [email protected] does not exist.

If I:

1. Manually create a test-sa service account via the IAM console in projectA
2. kubectl delete -n projectB IAMServiceAccountKey test-sa-credentials
3. Re-apply the yaml without change
   The last two steps might not be required, but it causes the reconciliation to trigger immediately.

Then the reconciliation succeeds on projectA. This means to me that the call to the IAM api is using projectA (or nothing and it's defaulting to the calling project) instead of projectB.

The controller I'm running is tag 36026d3 aka sha256:c98d253d88c4f85fe3df4ee4154804b17343fdf0bd65e7ecdeba9bf399ece44d
This is the version configured in the current install-bundle.tar.gz

The updated IAMServiceAccount looks correct to me with these values:
status:
email: [email protected]
name: projects/projectB/serviceAccounts/[email protected]

Here is the yaml I'm testing with. test-sa.yaml.txt

After 0.0.6 release: going through the installation steps, running kubectl apply on the CRD directory and getting partial success with servicemapping resources failing to install:

....
unable to recognize "install-bundle/bigquery-servicemapping.yaml": no matches for kind "ServiceMapping" in version "core.cnrm.cloud.google.com/v1alpha1"
unable to recognize "install-bundle/compute-servicemapping.yaml": no matches for kind "ServiceMapping" in version "core.cnrm.cloud.google.com/v1alpha1"
unable to recognize "install-bundle/iam-servicemapping.yaml": no matches for kind "ServiceMapping" in version "core.cnrm.cloud.google.com/v1alpha1"
unable to recognize "install-bundle/pubsub-servicemapping.yaml": no matches for kind "ServiceMapping" in version "core.cnrm.cloud.google.com/v1alpha1"
unable to recognize "install-bundle/redis-servicemapping.yaml": no matches for kind "ServiceMapping" in version "core.cnrm.cloud.google.com/v1alpha1"
unable to recognize "install-bundle/spanner-servicemapping.yaml": no matches for kind "ServiceMapping" in version "core.cnrm.cloud.google.com/v1alpha1"
unable to recognize "install-bundle/storage-servicemapping.yaml": no matches for kind "ServiceMapping" in version "core.cnrm.cloud.google.com/v1alpha1"

CC @kibbles-n-bytes @xiaobaitusi

Hey guys,

looking at the controller gcr.io/cnrm-eap/controller:a70be8d log, I see the following when adding a pubsub topic:

2019/10/04 04:20:20 [PubSubTopic test/test]: starting reconcile                                                                                                                                          │
│ 2019/10/04 04:20:20 [DEBUG] matching ID playground/test to regex (?P<project>[^/]+)/(?P<name>[^/]+).                                                                                       │
│ 2019/10/04 04:20:20 [DEBUG] importing project = playground                                                                                                                                 │
│ 2019/10/04 04:20:20 [DEBUG] importing name = test                                                                                                                                                        │
│ 2019/10/04 04:20:20 [DEBUG] Waiting for state to become: [success]                                                                                                                                       │
│ 2019/10/04 04:20:22 [WARN] Removing PubsubTopic "projects/playground/topics/test" because it's gone                                                                                        │
│ 2019/10/04 04:20:22 [PubSubTopic test/test]: creating/updating underlying resource                                                                                                                       │
│ 2019/10/04 04:20:22 [DEBUG] Creating new Topic: map[string]interface {}{"labels":map[string]string{"managed-by-cnrm":"true"}}                                                                            │
│ 2019/10/04 04:20:22 [DEBUG] Waiting for state to become: [success]                                                                                                                                       │
│ 2019/10/04 04:20:25 [DEBUG] Finished creating Topic "projects/playground/topics/test": map[string]interface {}{"labels":map[string]interface {}{"managed-by-cnrm":"true"}, "name":"project │
│ s/playground/topics/test"}                                                                                                                                                                 │
│ 2019/10/04 04:20:25 [DEBUG] Waiting for state to become: [success]                                                                                                                                       │
│ 2019/10/04 04:20:26 [PubSubTopic test/test]: successfully finished reconcile  

and I was wondering what this means:

2019/10/04 04:20:22 [WARN] Removing PubsubTopic "projects/playground/topics/test" because it's gone 

It doesn't look like its possible to use KCC to create GCP projects today; is that accurate?

In which case filing this issue to track adding support for that.

StorageBucketAccessControl is able to manage but there is no parameter for defaultObjectAcl list.

IAMServiceAccountKey creates a Secret with key.json key. Many helm packages needs spesific key for google service account key such as google-credentials.json or credentials.json. I think the option of specifying the file name will be quite useful.

• Example : [external-dns ] needs (https://github.com/helm/charts/blob/master/stable/external-dns/templates/deployment.yaml#L223) credentials.json filename .

It is awesome that provisioning of various Google cloud managed services is getting Kubernetes-native support through CRDs.

Given that application developers can consume multiple managed services, they will end up working with multiple Custom Resources. We have been developing guidelines for simplifying creation of such multi-Custom Resource stacks. Particularly, https://github.com/cloud-ark/kubeplus/blob/master/Guidelines.md#12-add-platform-as-code-annotations-on-your-crd-yaml is focused on making it easy for application developers to discover static usage information and dynamic runtime information about Custom Resources.

I was wondering if it would be possible to add the annotations mentioned in the above guidelines to config-connector's CRD definitions? Especially, the 'composition' annotation can help with generating runtime composition tree of Kubernetes resources that are created for handling a particular Custom Resource instance.

would be nice to have a way to run custom scripts once the sql database is created, for:

• DB schema initialization

• Enable Config Connector 0.0.7 on K8s cluster
• Create pubsubtopic resource
• Delete K8s cluster

Expected: pubsubtopic resource is deleted when the cluster is deleted
Actual: pubsubtopic resource is left behind orphaned

If this is expected behavior, I think it needs to be documented in the user guide. Currently the default behavior that is described, without abandon annotation is that underlying GCP resources are deleted when K8s resources are deleted.

On initial inspection it appears that only MySQL instances/dbs are in-scope here. Timeline/plans for Postgres support?

I have taken all the steps mentioned in Prerequisite

when I execute apply command as mentioned at Bookstore Sample, some deployment works fine, but for some, I get the following errors:
kubectl apply -f samples/apps/bookstore/config/release-configuration.yaml

Errors:

service "booksfe" created
service "inventory" created
service "purchases" created
service "users" created
deployment.extensions "booksfe" created
deployment.extensions "inventory" created
deployment.extensions "purchases" created
deployment.extensions "users" created
unable to recognize "samples/apps/bookstore/config/release-configuration.yaml": no matches for kind "PubSubSubscription" in version "pubsub.cnrm.cloud.google.com/v1alpha2"
unable to recognize "samples/apps/bookstore/config/release-configuration.yaml": no matches for kind "PubSubTopic" in version "pubsub.cnrm.cloud.google.com/v1alpha2"
unable to recognize "samples/apps/bookstore/config/release-configuration.yaml": no matches for kind "SpannerInstance" in version "spanner.cnrm.cloud.google.com/v1alpha2"

It is great so we can manage gcp resource via these CRDs, one question here is, because all these operators are not open source, is it possible to customize, for example, after creating MemoryStore instances, we also want assign a DNS record for each instance, how to achieve this?

I am following the how to guide for Cloud SQL with MYSQL in the README.md

But it seems I am missing some secret that’s preventing the musicians pod from starting:

musicians-prod-7b7cd9bd89-27wnr 0/2 ContainerCreating 0 17m

$ kubectl describe pod musicians-prod-7b7cd9bd89-27wnr
…
 FailedMount  43s (x18 over 21m)  kubelet, gke-traffic-director-clu-default-pool-ba1b383a-sldg  MountVolume.SetUp failed for volume "google-cloud-key" : secrets "gcp-key" not found
…

currently the KCC resource name is used to name the GCP resources, which introduces the k8s naming constraints. Would be nice to be able to use for ex capital letters in GCP resource names like cloudsql database.

The strategy of using k8s resource labels as GCP resource labels seems flawed since the key and value spaces are not compatible. This means that it is possible to add valid k8s labels that will fail GCP resource creation. One suggestion is to completely separate the two and instead define GCP resource labels as part of the spec

it would be nice to make it simpler to set up the resources needed to use Workload Identities. Currently you need a fair bit of boilerplate - a k8s SA, a GCP SA and a GCP IAMPolicy that all reference each other in a non trivial manner.

Even though the we set forceDestroy: true in the spec deletion fails. In the cnrm-controller-manager logs we can see that it's failing to set the corresponding flag for the delete request.

1. Make sure redis api is not enabled (or disable is by calling gcloud services disable redis.googleapis.com)
2. Create redis instance
   kubectl create -f k8s-config-connector/samples/resources/redis_v1alpha2_redisinstance.yaml
3. Enable redis api
   gcloud services enable redis.googleapis.com
4. Verify redis instance was created
   kubectl get redisinstance
gcloud redis instances list --region=us-central1
5. Attempt to describe it
   kubectl describe redisinstance redisinstance-sample
   Expected: no error

Actual:

$ kubectl get redisinstance      
NAME                   AGE      
redisinstance-sample   18h      
$ kubectl describe redisinstance redistinstance-sample      
Error from server (NotFound): redisinstances.redis.cnrm.cloud.google.com "redistinstance-sample" not found