kubepug / kubepug Goto Github PK

View Code? Open in Web Editor NEW

1.4K 13.0 68.0 16.25 MB

Kubernetes PreUpGrade (Checker)

Home Page: https://kubepug.xyz/

License: Apache License 2.0

Go 97.94% Makefile 1.23% Shell 0.38% Dockerfile 0.45%

kubernetes kubectl-plugins kubernetes-plugin

kubepug's Introduction

Deprecations AKA KubePug - Pre UpGrade (Checker)

KubePug/Deprecations is intended to be a kubectl plugin, which:

Downloads a generated data.json file containing API deprecation information for a specified release of Kubernetes
Scans a running Kubernetes cluster to determine if any objects will be affected by deprication
Displays affected objects to the user

Features

Can run against a Kubernetes cluster, using kubeconfig or the current cluster
Can run against a different set of manifest/files
Allows specifying the target Kubernetes version to be validated
Provides the replacement API that should be used
Informs the version that the API was deprecated or deleted, based on the target cluster version

How to use it as a krew plugin

Just run kubectl krew install deprecations

How to use it with Helm

If you want to verify the generated manifests by Helm, you can run the program as following:

helm template -f values.yaml .0 | kubepug --k8s-version v1.22.0 --input-file=-

Change the arguments in kubepug program (and Helm template!) as desired!

How to Use it as a standalone program

Download the correct version from Releases page.

After that, the command can be used just as kubectl, but with the following flags:

$ kubepug --help
[...]
Flags:
      --as-uid string            UID to impersonate for the operation.
      --cluster string           The name of the kubeconfig cluster to use
      --context string           The name of the kubeconfig context to use
      --database string          Sets the generated database location. Can be remote file or local (default "https://kubepug.xyz/data/data.json")
      --disable-compression      If true, opt-out of response compression for all requests to the server
      --error-on-deleted         If a deleted object is found, the program will exit with return code 1 instead of 0. Defaults to false
      --error-on-deprecated      If a deprecated object is found, the program will exit with return code 1 instead of 0. Defaults to false
      --filename string          Name of the file the results will be saved to, if empty it will display to stdout
      --format string            Format in which the list will be displayed [stdout, plain, json, yaml] (default "stdout")
  -h, --help                     help for kubepug
      --input-file string        Location of a file or directory containing k8s manifests to be analized
      --k8s-version string       Which kubernetes release version (https://github.com/kubernetes/kubernetes/releases) should be used to validate objects. Defaults to master (default "master")
      --kubeconfig string        Path to the kubeconfig file to use for CLI requests.
      --tls-server-name string   Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used
  -v, --verbosity string         Log level: debug, info, warn, error, fatal, panic (default "warning")

Alternatively you can install using go install

$ go install github.com/kubepug/kubepug@latest

Checking a Kubernetes Cluster

You can check the status of a running cluster with the following command.

$ kubepug --k8s-version=v1.22 # Will verify the current context against v1.22 version
[...]
RESULTS:
Deprecated APIs:
PodSecurityPolicy found in policy/v1beta1
	 ├─ Deprecated at: 1.21
	 ├─ PodSecurityPolicy governs the ability to make requests that affect the Security Contextthat will be applied to a pod and container.Deprecated in 1.21.
		-> OBJECT: restrictive namespace: default

Deleted APIs:
	 APIs REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELY!!
Ingress found in extensions/v1beta1
	 ├─ Deleted at: 1.22
	 ├─ Replacement: networking.k8s.io/v1/Ingress
	 ├─ Ingress is a collection of rules that allow inbound connections to reach theendpoints defined by a backend. An Ingress can be configured to give servicesexternally-reachable urls, load balance traffic, terminate SSL, offer namebased virtual hosting etc.DEPRECATED - This group version of Ingress is deprecated by networking.k8s.io/v1beta1 Ingress. See the release notes for more information.
		-> OBJECT: bla namespace: blabla

Putting Kubepug in your CI / Checking input files

You can verify files with the following:

$ kubepug --input-file=./deployment/ --error-on-deleted --error-on-deprecated

With the command above

The data.json from https://kubepug.xyz/data/data.json will be used
All YAML files (excluding subdirectories) will be verified
The program will exit with an error if deprecated or deleted objects are found.

Air-gapped environment

This happens when you have a secure environment that does not have an internet connectivity.

The data.json file is generated every hour, based on the latest stable version of Kubernetes API. You can download it from https://kubepug.xyz/data/data.json and move it to a safe location.

Then run kubepug pointing to the location of this file:

kubepug --k8s-version=v1.22 --database=location/of/your/data.json

Building your own data.json file

Steps to follow:

Clone/Download this repository, and build the container on generator/ directory

git clone https://github.com/kubepug/kubepug
docker build -t generator -f generator/Dockerfile generator

Generate the data.json

docker run generator > data.json

Generator uses the latest stable Kubernetes API version, if you want the latest dev version you should run as:

docker run -e VERSION=master generator > data.json

Securely move the json file to your Air-Gapped environment, to the folder of your choosing. This folder will be used by kubepug.
Execute kubepug with the option database, like this

kubepug --k8s-version=v1.22 --database=location/of/your/data.json

Example of Usage in CI with Github Actions

name: Sample CI Workflow
# This workflow is triggered on pushes to the repository.
on: [push]
env:
  HELM_VERSION: "v3.9.0"
  K8S_TARGET_VERSION: "v1.22.0"

jobs:
 api-deprecations-test:
    runs-on: ubuntu-latest
    steps:
      - name: Check-out repo
        uses: actions/checkout@v2

      - uses: azure/setup-helm@v1
        with:
          version: $HELM_VERSION
        id: install

      - uses: cpanato/[email protected]

      - name: Run Kubepug with your Helm Charts Repository
        run: |
          find charts -mindepth 1 -maxdepth 1 -type d | xargs -t -n1 -I% /bin/bash -c 'helm template % --api-versions ${K8S_TARGET_VERSION} | kubepug --error-on-deprecated --error-on-deleted --k8s-version ${K8S_TARGET_VERSION} --input-file /dev/stdin'

Screenshot

References

As I've used this project to learn Go and also some Kubernetes client-go some parts of this plugin are based in Caio Begotti's Pod-Tree, Ahmet Balkan kubectl-tree and Bitnami Kubecfg

Logo based in Mão vetor criado por freepik - br.freepik.com

kubepug's People

Contributors

Stargazers

Watchers

Forkers

chentex haoqing0110 ramirezramirez125 sandeepk17 harshanarayana benhemp ig0rsky everesio mattmattox igaskin samarsinghal 0xack13 mteodori isgasho dm-dv janpieper devtron-labs adambro xorilog cpanato magnologan letty5411 mshiekh forbearing eduardflorinescu rbnis sinesiobittencourt mpsanj jxt518 fourfifthscode dysosmus avinashsi marcelocyreno wolfsrudel pascal-h-kim hobti01 gg-big-org kjm0001 charlottemach doytsujin andre161292 ahmedqazimuhammadjamil ahinvinith iamhritik iq-scm shepardo ameukam shadowarmor kranthirachakonda michael-bowen-sc warmchang viniciussalmeida diogouchoas ripulhandoo adityabhatia lucasgrvarela arahja fleezesd clebs austinpray-mixpanel bth2 eng-aacm doker78 skbankjoy isaacnboyd deorerohit127 kiranbavireddy sri-gitops

kubepug's Issues

Improve YAML parser

Yaml parser halts when a file with multiple documents got only one invalid document in the middle.

kubectl also behaves like that, so it's not something in the go-yaml library that would solve this BUT I can try:

got an idea of maybe convert the whole file to the struct and then deal with the invalid entries in the struct.

Create json printer

Create an optional json printer instead of using the tree printing

args[0] basename

Usage:
  /Users/ahmetb/.krew/bin/kubectl-deprecations [flags]

Examples:
/Users/ahmetb/.krew/bin/kubectl-deprecations

looks like this when installed via krew.

Consider doing something like

if filepath.Basename(args[0]) == "kubectl-deprecations" // or .exe
  show "kubectl deprecations"

Improve performance

Since the new library refactor it seems pug is pretty slower, as the following:

# before refactoring
real    0m2,050s
user    0m0,453s
sys     0m0,091s

# After refactoring
real    0m30,541s
user    0m0,866s
sys     0m0,126s

This needs to be investigated before cutting a new release

Rename binary to kubectl-pug

Thanks for writing this plugin!

It would be awesome if the binary can be renamed kubectl-pug in future releases, as per:

https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/#installing-kubectl-plugins
A plugin is nothing more than a standalone executable file, whose name begins with kubectl-. To install a plugin, simply move its executable file to anywhere on your PATH.

Convert the map[string]interface{} loops to unstructured methods from k8s

Kubernetes already provides a unstructured method to walk through interfaces. Instead of repeating loops, change to this methods

can't have a plugin without specifying file operations

kubectl krew install deprecations
Updated the local copy of plugin index.
F0429 16:49:52.796455   13476 root.go:52] failed to load plugin "deprecations" from the index: platform ({Head: URI:https://github.com/rikatz/kubepug/releases/download/v0.2.6/kubepug_darwin_amd64.tar.gz Sha256:d583c80786e1c0c67aa65b8824680ae5f3f229cf80c5d9c52a5efc0a20a494e0 Selector:&LabelSelector{MatchLabels:map[string]string{arch: amd64,os: darwin,},MatchExpressions:[],} Files:[] Bin:kubepug}) is badly constructed: can't have a plugin without specifying file operations

Add link to the release page in the documentation

In the readme file, on How to use it as standalone program, it says

Download the correct version from Releases page.

There is no mention of where this release page is, so the user does not know from where it needs to download the software.

Acceptance criteria:
the user see a link to the URL pointing to the release page on the words "Releases page"
When clicking on the link, the user's browser open a new tab and the release page is displayed.

Would really appreciate, Thx !

Incorrect usage of --api-versions helm flag in README

See helm/helm#9088 (comment)

Basically --api-versions is supposed to have an array of the specific API names like they are returned from .Capabilities.APIVersions, not a K8S version as in the kubepug README

Ideally there should be an equivalent --k8s-version flag in helm too like in kubepug, but this is not implemented at the moment, so the only workaround I can think of is to collect somehow the full list of API names for a specific k8s version to pass to the existing --api-versions helm flag

Unable to run KubePug against OpenShift cluster

Hi, I am running KubePug against 3.11 Openshift cluster ( kubepug --k8s-version=v1.22.0) and I am seeing below error:
Failed communicating with k8s while discovering the object preferred name and gv. Error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request, servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request"

Is there any thing I need to take care?

Feature: GitHub Action

A GitHub action to simplify usage in CI would be great.

Just did a quick test with the following:

  api-deprecations:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v1

      - name: kubepug
        run: |
          mkdir -p ~/bin
          curl -sSL https://github.com/rikatz/kubepug/releases/download/v1.1.0/kubepug_linux_amd64.tar.gz | tar xvfz - --overwrite -C ~/bin/
          curl -sSL https://get.helm.sh/helm-v3.2.4-linux-amd64.tar.gz | tar xvfz - -C ~/bin/ --wildcards --strip 1 '*/helm'
          
          find charts -mindepth 1 -maxdepth 1 -type d | xargs -t -n1 -I% /bin/bash -c '~/bin/helm template % | ~/bin/kubepug --error-on-deprecated --error-on-deleted --k8s-version v1.16.0 --input-file /dev/stdin'

I probably should provide the kubernetes version to helm as well.

Turn pug into a krew plugin

Self descriptive title :P

False positive detected deprecation ?

Hi and thanks for this useful krew plugin :)

I just installed it and when I test it on my 1.16.6 cluster, I am not sure to understand the output.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.6", GitCommit:"72c30166b2105cd7d3350f2c28a219e6abcd79eb", GitTreeState:"clean", BuildDate:"2020-01-18T23:23:21Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl krew list                
PLUGIN             VERSION
deprecations       v1.0.1

The plugin indicates multiple deprecated version, here is an extract of the output :

$ kubectl deprecations --kubeconfig=~/.kube/config

RESULTS:
Deprecated APIs:

RoleBinding found in rbac.authorization.k8s.io/v1beta1
	 ├─ RoleBinding references a role, but does not contain it.  It can reference a Role in the same namespace or a ClusterRole in the global namespace. It adds who information via Subjects and namespace information by which namespace it exists in.  RoleBindings in a given namespace only have effect in that namespace. Deprecated in v1.17 in favor of rbac.authorization.k8s.io/v1 RoleBinding, and will no longer be served in v1.22.
		-> OBJECT: cert-manager-webhook:dynamic-serving namespace: cert-manager

[...]

ClusterRoleBinding found in rbac.authorization.k8s.io/v1beta1
	 ├─ ClusterRoleBinding references a ClusterRole, but not contain it.  It can reference a ClusterRole in the global namespace, and adds who information via Subject. Deprecated in v1.17 in favor of rbac.authorization.k8s.io/v1 ClusterRoleBinding, and will no longer be served in v1.22.
		-> GLOBAL: calico-kube-controllers 
		-> GLOBAL: calico-node 
[...]

But when I display the apiVersion of some of the detected objects, it seems to already be in the right API version :

$ kubectl get rolebindings.rbac.authorization.k8s.io -n cert-manager cert-manager-webhook:dynamic-serving -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
[...]


$ kubectl get clusterrolebindings.rbac.authorization.k8s.io calico-kube-controllers -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
[...]

$ kubectl get clusterrolebindings.rbac.authorization.k8s.io calico-node -o yaml            
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding

Did I misunderstand something ?
Thank you

Usage question

We are looking at upgrading our Cluster from v1.14.9 to v1.18.0. It's not clear to me whether I should pull the v1.14 swagger json or the v1.18 json.

E2E tests

It would be easier for me to check the behavior of kubepug if we got some nice e2e tests

Well known versions:

v1.14
v1.16
v1.18

The input data

Some old Ingress, Network Policy, Deployment objects
Some CRDs file
Some apiextension objects

The test matrix

Files

Test the content of the directory directly, verifying if the json output is the same as expected: Are the deleted APIs there? Are the deprecated APIs there?

Kubernetes Cluster

Create clusters in the well known versions above with KinD, and check if all the outputs (including the built-in from KinD) are the same as expected

Performance

For now, it would be good to at least check if kubepug runs inside a desired time window, if it's not timing out

New features

Prometheus scrapping, format exporting, etc could also be tested in some tests evolutions

New k8s client-go

We need to check if updating to newer client-go will break anything :)

JSON Parser

As per #65 it would be nice to also parse json files. Don't know if people still uses it as input for kubernetes, but who knows 🤷‍♂️

So this is an improvement

Inexistent objects in swagger aren't verified - API Walker

To the future me:

When you have an API removed from swagger.json it cannot be verified against the API Server, because the normal flow is to check all the deprecated apis from swagger.json and check their existence. If something doesn't exists, we cannot check :(

The solution here is to walk across all the existent resources in the API Server and check if they still exists in swagger.json.

If don't, check if there's some object inside that resource and notify that Object might disappear / stop working in future releases

So the flow will be (using discovery and dynamicclient)

Get all the GroupVersion items
For each GV item, get resources
For each resource, check if its definition exists in the swagger JSON
If that resource doesn't exists in the JSON, check if there're objects inside that Resource
If there're objects, show the objects alerting about the deprecation of that version

alert the user about the nonexistence of that API in the future k8s release

Binary assets missing for 1.2.0+

The instructions for using KubePug as part of a CI pipeline suggests downloading:

https://github.com/rikatz/kubepug/releases/latest/download/kubepug_linux_amd64.tar.gz

This redirects to:

https://github.com/rikatz/kubepug/releases/download/v1.2.1/kubepug_linux_amd64.tar.gz

which gives a 404. I suspect this is because the binaries have not been generated for 1.2.0 or 1.2.1. The following works:

https://github.com/rikatz/kubepug/releases/download/v1.1.3/kubepug_linux_amd64.tar.gz

Can't install plugin on Apple Silicon based mac

Looks like the plugin might be build for intel only:

kubectl krew install deprecations
Updated the local copy of plugin index.
Updated the local copy of plugin index "kvaps".
Installing plugin: deprecations
W0829 13:50:54.381681   11235 install.go:164] failed to install plugin "deprecations": plugin "deprecations" does not offer installation for this platform
F0829 13:50:54.381735   11235 root.go:79] failed to install some plugins: [deprecations]: plugin "deprecations" does not offer installation for this platform

kubepug is not finding deprecated APIs

@rikatz With implementing #82 we're not able to see deprecations anymore. Thinking about the implementation I was wondering how is kubepug able to identify deprecations as in my mind the Kube API will return whatever object you're requesting but not the original source, is that assumption correct?

I was wondering if you could read from metadata.annotations."kubectl.kubernetes.io/last-applied-configuration" instead and use this as the source for analysis.

Thoughts?

doesn't recognize CRDs

I see #9 was in v0.2.0 but when I just installed every single Istio and Knative CRD/objects showed as deprecated.

$ kubectl deprecations
MutatingWebhookConfiguration found in admissionregistration.k8s.io/v1beta1
	 ├─ MutatingWebhookConfiguration describes the configuration of and admission webhook that accept or reject and may change the object. Deprecated in v1.16, planned for removal in v1.19. Use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration instead.
		-> GLOBAL: istio-sidecar-injector
		-> GLOBAL: pod-ready.config.common-webhooks.networking.gke.io
		-> GLOBAL: webhook.eventing.knative.dev
		-> GLOBAL: webhook.serving.knative.dev

ValidatingWebhookConfiguration found in admissionregistration.k8s.io/v1beta1
	 ├─ ValidatingWebhookConfiguration describes the configuration of and admission webhook that accept or reject and object without changing it. Deprecated in v1.16, planned for removal in v1.19. Use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration instead.
		-> GLOBAL: istio-galley

Role found in rbac.authorization.k8s.io/v1beta1
	 ├─ Role is a namespaced, logical grouping of PolicyRules that can be referenced as a unit by a RoleBinding. Deprecated in v1.17 in favor of rbac.authorization.k8s.io/v1 Role, and will no longer be served in v1.20.
		-> Object: istio-ingress-sds namespace: gke-system
		-> Object: istio-ingressgateway-sds namespace: istio-system
		-> Object: system:controller:bootstrap-signer namespace: kube-public
		-> Object: cloud-provider namespace: kube-system
		-> Object: extension-apiserver-authentication-reader namespace: kube-system
		-> Object: gce:cloud-provider namespace: kube-system
		-> Object: system::leader-locking-kube-controller-manager namespace: kube-system
		-> Object: system::leader-locking-kube-scheduler namespace: kube-system
		-> Object: system:controller:bootstrap-signer namespace: kube-system
		-> Object: system:controller:cloud-provider namespace: kube-system
		-> Object: system:controller:token-cleaner namespace: kube-system
		-> Object: system:fluentd-gcp-scaler namespace: kube-system
		-> Object: system:pod-nanny namespace: kube-system

ClusterRole found in rbac.authorization.k8s.io/v1beta1
	 ├─ ClusterRole is a cluster level, logical grouping of PolicyRules that can be referenced as a unit by a RoleBinding or ClusterRoleBinding. Deprecated in v1.17 in favor of rbac.authorization.k8s.io/v1 ClusterRole, and will no longer be served in v1.20.
		-> GLOBAL: addressable-resolver
		-> GLOBAL: admin
		-> GLOBAL: broker-addressable-resolver
		-> GLOBAL: channel-addressable-resolver
		-> GLOBAL: channelable-manipulator
		-> GLOBAL: cloud-provider
		-> GLOBAL: cluster-admin
		-> GLOBAL: cluster-local-gateway-gke-system
		-> GLOBAL: cluster-local-gateway-istio-system
		-> GLOBAL: edit
		-> GLOBAL: eventing-broker-filter
		-> GLOBAL: eventing-broker-ingress
		-> GLOBAL: eventing-config-reader
		-> GLOBAL: eventing-sources-source-observer
		-> GLOBAL: gce:beta:kubelet-certificate-bootstrap
		-> GLOBAL: gce:beta:kubelet-certificate-rotation
		-> GLOBAL: gce:cloud-provider
		-> GLOBAL: imc-addressable-resolver
		-> GLOBAL: imc-channelable-manipulator
		-> GLOBAL: imc-controller
		-> GLOBAL: imc-dispatcher
		-> GLOBAL: istio-citadel-istio-system
		-> GLOBAL: istio-cleanup-secrets-istio-system
		-> GLOBAL: istio-galley-istio-system
		-> GLOBAL: istio-ingress-gke-system
		-> GLOBAL: istio-ingressgateway-istio-system
		-> GLOBAL: istio-init-gke-system
		-> GLOBAL: istio-init-istio-system
		-> GLOBAL: istio-mixer-istio-system
		-> GLOBAL: istio-pilot-gke-system
		-> GLOBAL: istio-pilot-istio-system
		-> GLOBAL: istio-reader
		-> GLOBAL: istio-security-post-install-istio-system
		-> GLOBAL: istio-sidecar-injector-istio-system
		-> GLOBAL: knative-eventing-controller
		-> GLOBAL: knative-eventing-namespaced-admin
		-> GLOBAL: knative-eventing-namespaced-edit
		-> GLOBAL: knative-eventing-namespaced-view
		-> GLOBAL: knative-eventing-source-controller
		-> GLOBAL: knative-eventing-sources-namespaced-admin
		-> GLOBAL: knative-eventing-webhook
		-> GLOBAL: knative-messaging-namespaced-admin
		-> GLOBAL: knative-serving-admin
		-> GLOBAL: knative-serving-cloudrun
		-> GLOBAL: knative-serving-core
		-> GLOBAL: knative-serving-istio
		-> GLOBAL: knative-serving-namespaced-admin
		-> GLOBAL: kubelet-api-admin
		-> GLOBAL: messaging-addressable-resolver
		-> GLOBAL: promsd-istio-system
		-> GLOBAL: read-updateinfo
		-> GLOBAL: service-addressable-resolver
		-> GLOBAL: serving-addressable-resolver
		-> GLOBAL: source-observer
		-> GLOBAL: stackdriver:fluentd-gcp
		-> GLOBAL: stackdriver:metadata-agent
		-> GLOBAL: system:aggregate-to-admin
		-> GLOBAL: system:aggregate-to-edit
		-> GLOBAL: system:aggregate-to-view
		-> GLOBAL: system:auth-delegator
		-> GLOBAL: system:aws-cloud-provider
		-> GLOBAL: system:basic-user
		-> GLOBAL: system:certificates.k8s.io:certificatesigningrequests:nodeclient
		-> GLOBAL: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
		-> GLOBAL: system:controller:attachdetach-controller
		-> GLOBAL: system:controller:certificate-controller
		-> GLOBAL: system:controller:clusterrole-aggregation-controller
		-> GLOBAL: system:controller:cronjob-controller
		-> GLOBAL: system:controller:daemon-set-controller
		-> GLOBAL: system:controller:deployment-controller
		-> GLOBAL: system:controller:disruption-controller
		-> GLOBAL: system:controller:endpoint-controller
		-> GLOBAL: system:controller:expand-controller
		-> GLOBAL: system:controller:generic-garbage-collector
		-> GLOBAL: system:controller:horizontal-pod-autoscaler
		-> GLOBAL: system:controller:job-controller
		-> GLOBAL: system:controller:namespace-controller
		-> GLOBAL: system:controller:node-controller
		-> GLOBAL: system:controller:persistent-volume-binder
		-> GLOBAL: system:controller:pod-garbage-collector
		-> GLOBAL: system:controller:pv-protection-controller
		-> GLOBAL: system:controller:pvc-protection-controller
		-> GLOBAL: system:controller:replicaset-controller
		-> GLOBAL: system:controller:replication-controller
		-> GLOBAL: system:controller:resourcequota-controller
		-> GLOBAL: system:controller:route-controller
		-> GLOBAL: system:controller:service-account-controller
		-> GLOBAL: system:controller:service-controller
		-> GLOBAL: system:controller:statefulset-controller
		-> GLOBAL: system:controller:ttl-controller
		-> GLOBAL: system:csi-external-attacher
		-> GLOBAL: system:csi-external-provisioner
		-> GLOBAL: system:discovery
		-> GLOBAL: system:heapster
		-> GLOBAL: system:kube-aggregator
		-> GLOBAL: system:kube-controller-manager
		-> GLOBAL: system:kube-dns
		-> GLOBAL: system:kube-dns-autoscaler
		-> GLOBAL: system:kube-scheduler
		-> GLOBAL: system:kubelet-api-admin
		-> GLOBAL: system:metrics-server
		-> GLOBAL: system:node
		-> GLOBAL: system:node-bootstrapper
		-> GLOBAL: system:node-problem-detector
		-> GLOBAL: system:node-proxier
		-> GLOBAL: system:persistent-volume-provisioner
		-> GLOBAL: system:public-info-viewer
		-> GLOBAL: system:volume-scheduler
		-> GLOBAL: view

PriorityClass found in scheduling.k8s.io/v1beta1
	 ├─ DEPRECATED - This group version of PriorityClass is deprecated by scheduling.k8s.io/v1/PriorityClass. PriorityClass defines mapping from a priority class name to the priority integer value. The value can be any valid integer.
		-> GLOBAL: system-cluster-critical
		-> GLOBAL: system-node-critical

ClusterRoleBinding found in rbac.authorization.k8s.io/v1beta1
	 ├─ ClusterRoleBinding references a ClusterRole, but not contain it.  It can reference a ClusterRole in the global namespace, and adds who information via Subject. Deprecated in v1.17 in favor of rbac.authorization.k8s.io/v1 ClusterRoleBinding, and will no longer be served in v1.20.
		-> GLOBAL: ahmet-is-admin
		-> GLOBAL: cluster-admin
		-> GLOBAL: cluster-admin-binding
		-> GLOBAL: cluster-autoscaler-updateinfo
		-> GLOBAL: cluster-local-gateway-gke-system
		-> GLOBAL: cluster-local-gateway-istio-system
		-> GLOBAL: event-exporter-rb
		-> GLOBAL: eventing-controller
		-> GLOBAL: eventing-controller-manipulator
		-> GLOBAL: eventing-controller-resolver
		-> GLOBAL: eventing-controller-source-observer
		-> GLOBAL: eventing-source-controller
		-> GLOBAL: eventing-source-controller-resolver
		-> GLOBAL: eventing-webhook
		-> GLOBAL: gce:beta:kubelet-certificate-bootstrap
		-> GLOBAL: gce:beta:kubelet-certificate-rotation
		-> GLOBAL: gce:cloud-provider
		-> GLOBAL: heapster-binding
		-> GLOBAL: imc-controller
		-> GLOBAL: imc-dispatcher
		-> GLOBAL: istio-citadel-istio-system
		-> GLOBAL: istio-cleanup-secrets-istio-system
		-> GLOBAL: istio-galley-admin-role-binding-istio-system
		-> GLOBAL: istio-ingress-gke-system
		-> GLOBAL: istio-ingressgateway-istio-system
		-> GLOBAL: istio-init-admin-role-binding-gke-system
		-> GLOBAL: istio-init-admin-role-binding-istio-system
		-> GLOBAL: istio-mixer-admin-role-binding-istio-system
		-> GLOBAL: istio-multi
		-> GLOBAL: istio-pilot-gke-system
		-> GLOBAL: istio-pilot-istio-system
		-> GLOBAL: istio-security-post-install-role-binding-istio-system
		-> GLOBAL: istio-sidecar-injector-admin-role-binding-istio-system
		-> GLOBAL: jeff-is-admin
		-> GLOBAL: knative-serving-controller-admin
		-> GLOBAL: kube-apiserver-kubelet-api-admin
		-> GLOBAL: kubelet-bootstrap
		-> GLOBAL: kubelet-bootstrap-certificate-bootstrap
		-> GLOBAL: kubelet-bootstrap-node-bootstrapper
		-> GLOBAL: kubelet-cluster-admin
		-> GLOBAL: metrics-server:system:auth-delegator
		-> GLOBAL: npd-binding
		-> GLOBAL: promsd-istio-system
		-> GLOBAL: stackdriver:fluentd-gcp
		-> GLOBAL: stackdriver:metadata-agent
		-> GLOBAL: sundar-is-admin
		-> GLOBAL: system:aws-cloud-provider
		-> GLOBAL: system:basic-user
		-> GLOBAL: system:controller:attachdetach-controller
		-> GLOBAL: system:controller:certificate-controller
		-> GLOBAL: system:controller:clusterrole-aggregation-controller
		-> GLOBAL: system:controller:cronjob-controller
		-> GLOBAL: system:controller:daemon-set-controller
		-> GLOBAL: system:controller:deployment-controller
		-> GLOBAL: system:controller:disruption-controller
		-> GLOBAL: system:controller:endpoint-controller
		-> GLOBAL: system:controller:expand-controller
		-> GLOBAL: system:controller:generic-garbage-collector
		-> GLOBAL: system:controller:horizontal-pod-autoscaler
		-> GLOBAL: system:controller:job-controller
		-> GLOBAL: system:controller:namespace-controller
		-> GLOBAL: system:controller:node-controller
		-> GLOBAL: system:controller:persistent-volume-binder
		-> GLOBAL: system:controller:pod-garbage-collector
		-> GLOBAL: system:controller:pv-protection-controller
		-> GLOBAL: system:controller:pvc-protection-controller
		-> GLOBAL: system:controller:replicaset-controller
		-> GLOBAL: system:controller:replication-controller
		-> GLOBAL: system:controller:resourcequota-controller
		-> GLOBAL: system:controller:route-controller
		-> GLOBAL: system:controller:service-account-controller
		-> GLOBAL: system:controller:service-controller
		-> GLOBAL: system:controller:statefulset-controller
		-> GLOBAL: system:controller:ttl-controller
		-> GLOBAL: system:discovery
		-> GLOBAL: system:kube-controller-manager
		-> GLOBAL: system:kube-dns
		-> GLOBAL: system:kube-dns-autoscaler
		-> GLOBAL: system:kube-scheduler
		-> GLOBAL: system:metrics-server
		-> GLOBAL: system:node
		-> GLOBAL: system:node-proxier
		-> GLOBAL: system:public-info-viewer
		-> GLOBAL: system:volume-scheduler

RoleBinding found in rbac.authorization.k8s.io/v1beta1
	 ├─ RoleBinding references a role, but does not contain it.  It can reference a Role in the same namespace or a ClusterRole in the global namespace. It adds who information via Subjects and namespace information by which namespace it exists in.  RoleBindings in a given namespace only have effect in that namespace. Deprecated in v1.17 in favor of rbac.authorization.k8s.io/v1 RoleBinding, and will no longer be served in v1.20.
		-> Object: eventing-broker-filter namespace: event-example
		-> Object: eventing-broker-ingress namespace: event-example
		-> Object: istio-ingress-sds namespace: gke-system
		-> Object: istio-ingressgateway-sds namespace: istio-system
		-> Object: eventing-broker-filter-event-example namespace: knative-eventing
		-> Object: eventing-broker-ingress-event-example namespace: knative-eventing
		-> Object: system:controller:bootstrap-signer namespace: kube-public
		-> Object: fluentd-gcp-scaler-binding namespace: kube-system
		-> Object: gce:cloud-provider namespace: kube-system
		-> Object: heapster-binding namespace: kube-system
		-> Object: metrics-server-auth-reader namespace: kube-system
		-> Object: system::extension-apiserver-authentication-reader namespace: kube-system
		-> Object: system::leader-locking-kube-controller-manager namespace: kube-system
		-> Object: system::leader-locking-kube-scheduler namespace: kube-system
		-> Object: system:controller:bootstrap-signer namespace: kube-system
		-> Object: system:controller:cloud-provider namespace: kube-system
		-> Object: system:controller:token-cleaner namespace: kube-system

CustomResourceDefinition found in apiextensions.k8s.io/v1beta1
	 ├─ CustomResourceDefinition represents a resource that should be exposed on the API server.  Its name MUST be in the format <.spec.name>.<.spec.group>. Deprecated in v1.16, planned for removal in v1.19. Use apiextensions.k8s.io/v1 CustomResourceDefinition instead.
		-> GLOBAL: adapters.config.istio.io
		-> GLOBAL: apikeys.config.istio.io
		-> GLOBAL: apiserversources.sources.eventing.knative.dev
		-> GLOBAL: attributemanifests.config.istio.io
		-> GLOBAL: authorizations.config.istio.io
		-> GLOBAL: backendconfigs.cloud.google.com
		-> GLOBAL: brokers.eventing.knative.dev
		-> GLOBAL: bypasses.config.istio.io
		-> GLOBAL: certificates.certmanager.k8s.io
		-> GLOBAL: certificates.networking.internal.knative.dev
		-> GLOBAL: challenges.certmanager.k8s.io
		-> GLOBAL: channels.messaging.knative.dev
		-> GLOBAL: checknothings.config.istio.io
		-> GLOBAL: circonuses.config.istio.io
		-> GLOBAL: cloudwatches.config.istio.io
		-> GLOBAL: clusteringresses.networking.internal.knative.dev
		-> GLOBAL: clusterissuers.certmanager.k8s.io
		-> GLOBAL: clusterrbacconfigs.rbac.istio.io
		-> GLOBAL: configurations.serving.knative.dev
		-> GLOBAL: containersources.sources.eventing.knative.dev
		-> GLOBAL: cronjobsources.sources.eventing.knative.dev
		-> GLOBAL: deniers.config.istio.io
		-> GLOBAL: destinationrules.networking.istio.io
		-> GLOBAL: dogstatsds.config.istio.io
		-> GLOBAL: domainmappings.domains.cloudrun.com
		-> GLOBAL: edges.config.istio.io
		-> GLOBAL: envoyfilters.networking.istio.io
		-> GLOBAL: eventtypes.eventing.knative.dev
		-> GLOBAL: fluentds.config.istio.io
		-> GLOBAL: gateways.networking.istio.io
		-> GLOBAL: handlers.config.istio.io
		-> GLOBAL: httpapispecbindings.config.istio.io
		-> GLOBAL: httpapispecs.config.istio.io
		-> GLOBAL: images.caching.internal.knative.dev
		-> GLOBAL: ingresses.networking.internal.knative.dev
		-> GLOBAL: inmemorychannels.messaging.knative.dev
		-> GLOBAL: instances.config.istio.io
		-> GLOBAL: issuers.certmanager.k8s.io
		-> GLOBAL: kubernetesenvs.config.istio.io
		-> GLOBAL: kuberneteses.config.istio.io
		-> GLOBAL: listcheckers.config.istio.io
		-> GLOBAL: listentries.config.istio.io
		-> GLOBAL: logentries.config.istio.io
		-> GLOBAL: managedcertificates.networking.gke.io
		-> GLOBAL: memquotas.config.istio.io
		-> GLOBAL: meshpolicies.authentication.istio.io
		-> GLOBAL: metrics.autoscaling.internal.knative.dev
		-> GLOBAL: metrics.config.istio.io
		-> GLOBAL: noops.config.istio.io
		-> GLOBAL: opas.config.istio.io
		-> GLOBAL: orders.certmanager.k8s.io
		-> GLOBAL: parallels.messaging.knative.dev
		-> GLOBAL: podautoscalers.autoscaling.internal.knative.dev
		-> GLOBAL: policies.authentication.istio.io
		-> GLOBAL: prometheuses.config.istio.io
		-> GLOBAL: quotas.config.istio.io
		-> GLOBAL: quotaspecbindings.config.istio.io
		-> GLOBAL: quotaspecs.config.istio.io
		-> GLOBAL: rbacconfigs.rbac.istio.io
		-> GLOBAL: rbacs.config.istio.io
		-> GLOBAL: redisquotas.config.istio.io
		-> GLOBAL: reportnothings.config.istio.io
		-> GLOBAL: revisions.serving.knative.dev
		-> GLOBAL: routes.serving.knative.dev
		-> GLOBAL: rules.config.istio.io
		-> GLOBAL: scalingpolicies.scalingpolicy.kope.io
		-> GLOBAL: sequences.messaging.knative.dev
		-> GLOBAL: serverlessservices.networking.internal.knative.dev
		-> GLOBAL: serviceentries.networking.istio.io
		-> GLOBAL: servicerolebindings.rbac.istio.io
		-> GLOBAL: serviceroles.rbac.istio.io
		-> GLOBAL: services.serving.knative.dev
		-> GLOBAL: sidecars.networking.istio.io
		-> GLOBAL: signalfxs.config.istio.io
		-> GLOBAL: solarwindses.config.istio.io
		-> GLOBAL: stackdrivers.config.istio.io
		-> GLOBAL: statsds.config.istio.io
		-> GLOBAL: stdios.config.istio.io
		-> GLOBAL: subscriptions.messaging.knative.dev
		-> GLOBAL: templates.config.istio.io
		-> GLOBAL: tracespans.config.istio.io
		-> GLOBAL: triggers.eventing.knative.dev
		-> GLOBAL: updateinfos.nodemanagement.gke.io
		-> GLOBAL: virtualservices.networking.istio.io
		-> GLOBAL: zipkins.config.istio.io

DaemonSet found in extensions/v1beta1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: daemonset-hold-image namespace: kube-system
		-> Object: fluentd-gcp-v3.1.1 namespace: kube-system
		-> Object: metadata-proxy-v0.1 namespace: kube-system
		-> Object: nvidia-gpu-device-plugin namespace: kube-system
		-> Object: prometheus-to-sd namespace: kube-system

Deployment found in extensions/v1beta1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping-r7f8r-deployment namespace: default
		-> Object: grpc-ping-rc2jr-deployment namespace: default
		-> Object: hello-lbsxh-3-deployment namespace: default
		-> Object: hello-tdpmb-1-deployment namespace: default
		-> Object: hello-tjxhq-2-deployment namespace: default
		-> Object: nginx namespace: default
		-> Object: default-broker-filter namespace: event-example
		-> Object: default-broker-ingress namespace: event-example
		-> Object: cluster-local-gateway namespace: gke-system
		-> Object: istio-ingress namespace: gke-system
		-> Object: istio-pilot namespace: gke-system
		-> Object: istio-citadel namespace: istio-system
		-> Object: istio-galley namespace: istio-system
		-> Object: istio-ingressgateway namespace: istio-system
		-> Object: istio-pilot namespace: istio-system
		-> Object: istio-policy namespace: istio-system
		-> Object: istio-sidecar-injector namespace: istio-system
		-> Object: istio-telemetry namespace: istio-system
		-> Object: promsd namespace: istio-system
		-> Object: eventing-controller namespace: knative-eventing
		-> Object: eventing-webhook namespace: knative-eventing
		-> Object: imc-controller namespace: knative-eventing
		-> Object: imc-dispatcher namespace: knative-eventing
		-> Object: sources-controller namespace: knative-eventing
		-> Object: activator namespace: knative-serving
		-> Object: autoscaler namespace: knative-serving
		-> Object: controller namespace: knative-serving
		-> Object: webhook namespace: knative-serving
		-> Object: event-exporter-v0.2.5 namespace: kube-system
		-> Object: fluentd-gcp-scaler namespace: kube-system
		-> Object: heapster namespace: kube-system
		-> Object: kube-dns namespace: kube-system
		-> Object: kube-dns-autoscaler namespace: kube-system
		-> Object: l7-default-backend namespace: kube-system
		-> Object: metrics-server-v0.3.1 namespace: kube-system
		-> Object: stackdriver-metadata-agent-cluster-level namespace: kube-system

ReplicaSet found in extensions/v1beta1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping-r7f8r-deployment-54d688d777 namespace: default
		-> Object: grpc-ping-rc2jr-deployment-5f9c4fbf49 namespace: default
		-> Object: hello-lbsxh-3-deployment-6c867cf8dd namespace: default
		-> Object: hello-tdpmb-1-deployment-5fcc78b9ff namespace: default
		-> Object: hello-tjxhq-2-deployment-6d7b48d64c namespace: default
		-> Object: nginx-697b9dcc86 namespace: default
		-> Object: nginx-7c6cfbddbc namespace: default
		-> Object: default-broker-filter-9fbc5c667 namespace: event-example
		-> Object: default-broker-ingress-7948d9b8fc namespace: event-example
		-> Object: cluster-local-gateway-86db848cf5 namespace: gke-system
		-> Object: istio-ingress-5b4d9c44dd namespace: gke-system
		-> Object: istio-pilot-996485bd namespace: gke-system
		-> Object: istio-citadel-5448cbc9f8 namespace: istio-system
		-> Object: istio-citadel-5949896b4b namespace: istio-system
		-> Object: istio-citadel-5c88f4f47d namespace: istio-system
		-> Object: istio-galley-5bf8f944b7 namespace: istio-system
		-> Object: istio-ingressgateway-57449dd8bc namespace: istio-system
		-> Object: istio-ingressgateway-7c96766d85 namespace: istio-system
		-> Object: istio-ingressgateway-7f6db9967c namespace: istio-system
		-> Object: istio-pilot-56b86b87b6 namespace: istio-system
		-> Object: istio-pilot-56df8c9cf6 namespace: istio-system
		-> Object: istio-pilot-75654d4d9 namespace: istio-system
		-> Object: istio-policy-59dd46fb7d namespace: istio-system
		-> Object: istio-policy-68775985b8 namespace: istio-system
		-> Object: istio-policy-99fd7f7f5 namespace: istio-system
		-> Object: istio-sidecar-injector-5b5454d777 namespace: istio-system
		-> Object: istio-sidecar-injector-6cf9dc4549 namespace: istio-system
		-> Object: istio-sidecar-injector-bc79fc549 namespace: istio-system
		-> Object: istio-telemetry-5965f9d996 namespace: istio-system
		-> Object: istio-telemetry-699ffd959d namespace: istio-system
		-> Object: istio-telemetry-cdf9c6d7 namespace: istio-system
		-> Object: promsd-76f8d4cff8 namespace: istio-system
		-> Object: promsd-f876c8c55 namespace: istio-system
		-> Object: eventing-controller-7d75cd8598 namespace: knative-eventing
		-> Object: eventing-webhook-5cb89d8974 namespace: knative-eventing
		-> Object: imc-controller-654d689bc9 namespace: knative-eventing
		-> Object: imc-dispatcher-794f546c85 namespace: knative-eventing
		-> Object: sources-controller-67788d5b86 namespace: knative-eventing
		-> Object: activator-5477b7ff7 namespace: knative-serving
		-> Object: activator-6865899867 namespace: knative-serving
		-> Object: activator-74488ccb4c namespace: knative-serving
		-> Object: activator-7688586ccb namespace: knative-serving
		-> Object: autoscaler-5c56954884 namespace: knative-serving
		-> Object: autoscaler-7688987947 namespace: knative-serving
		-> Object: autoscaler-7c95566b65 namespace: knative-serving
		-> Object: autoscaler-85f5b489d7 namespace: knative-serving
		-> Object: controller-55bb588dbd namespace: knative-serving
		-> Object: controller-5ff95479f5 namespace: knative-serving
		-> Object: controller-78fd6cb64 namespace: knative-serving
		-> Object: controller-845759bc78 namespace: knative-serving
		-> Object: webhook-5dcf9dc6b5 namespace: knative-serving
		-> Object: webhook-64645549f6 namespace: knative-serving
		-> Object: webhook-759c4676bd namespace: knative-serving
		-> Object: webhook-77856f76fc namespace: knative-serving
		-> Object: event-exporter-v0.2.5-7d99d74cf8 namespace: kube-system
		-> Object: fluentd-gcp-scaler-54ccb89d5 namespace: kube-system
		-> Object: fluentd-gcp-scaler-55bdf597c namespace: kube-system
		-> Object: heapster-5f4c566b6d namespace: kube-system
		-> Object: heapster-64df54d744 namespace: kube-system
		-> Object: heapster-666fffdc58 namespace: kube-system
		-> Object: heapster-846cc4b8c6 namespace: kube-system
		-> Object: kube-dns-5877696fb4 namespace: kube-system
		-> Object: kube-dns-6987857fdb namespace: kube-system
		-> Object: kube-dns-autoscaler-85f8bdb54 namespace: kube-system
		-> Object: kube-dns-autoscaler-bb58c6784 namespace: kube-system
		-> Object: l7-default-backend-fd59995cd namespace: kube-system
		-> Object: metrics-server-v0.3.1-57c75779f namespace: kube-system
		-> Object: metrics-server-v0.3.1-c4cddd5f5 namespace: kube-system
		-> Object: stackdriver-metadata-agent-cluster-level-74d55f7669 namespace: kube-system
		-> Object: stackdriver-metadata-agent-cluster-level-8d8677c5d namespace: kube-system
		-> Object: stackdriver-metadata-agent-cluster-level-bb9d8bb64 namespace: kube-system

ControllerRevision found in apps/v1beta2
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: daemonset-hold-image-5668b66bcd namespace: kube-system
		-> Object: fluentd-gcp-v3.1.1-57756ccc47 namespace: kube-system
		-> Object: fluentd-gcp-v3.1.1-595d677f66 namespace: kube-system
		-> Object: fluentd-gcp-v3.1.1-5c8c4c9864 namespace: kube-system
		-> Object: fluentd-gcp-v3.1.1-789d446b4c namespace: kube-system
		-> Object: fluentd-gcp-v3.1.1-864d84b477 namespace: kube-system
		-> Object: metadata-proxy-v0.1-5b4567fdff namespace: kube-system
		-> Object: metadata-proxy-v0.1-7d7b4bb744 namespace: kube-system
		-> Object: nvidia-gpu-device-plugin-64d8cbf474 namespace: kube-system
		-> Object: nvidia-gpu-device-plugin-65b49677fc namespace: kube-system
		-> Object: nvidia-gpu-device-plugin-6c79557f9b namespace: kube-system
		-> Object: prometheus-to-sd-5b58787fb5 namespace: kube-system
		-> Object: prometheus-to-sd-776c6cbcb namespace: kube-system
		-> Object: prometheus-to-sd-99cd5649b namespace: kube-system
		-> Object: prometheus-to-sd-9dd499d6d namespace: kube-system

DaemonSet found in apps/v1beta2
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: daemonset-hold-image namespace: kube-system
		-> Object: fluentd-gcp-v3.1.1 namespace: kube-system
		-> Object: metadata-proxy-v0.1 namespace: kube-system
		-> Object: nvidia-gpu-device-plugin namespace: kube-system
		-> Object: prometheus-to-sd namespace: kube-system

Deployment found in apps/v1beta2
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping-r7f8r-deployment namespace: default
		-> Object: grpc-ping-rc2jr-deployment namespace: default
		-> Object: hello-lbsxh-3-deployment namespace: default
		-> Object: hello-tdpmb-1-deployment namespace: default
		-> Object: hello-tjxhq-2-deployment namespace: default
		-> Object: nginx namespace: default
		-> Object: default-broker-filter namespace: event-example
		-> Object: default-broker-ingress namespace: event-example
		-> Object: cluster-local-gateway namespace: gke-system
		-> Object: istio-ingress namespace: gke-system
		-> Object: istio-pilot namespace: gke-system
		-> Object: istio-citadel namespace: istio-system
		-> Object: istio-galley namespace: istio-system
		-> Object: istio-ingressgateway namespace: istio-system
		-> Object: istio-pilot namespace: istio-system
		-> Object: istio-policy namespace: istio-system
		-> Object: istio-sidecar-injector namespace: istio-system
		-> Object: istio-telemetry namespace: istio-system
		-> Object: promsd namespace: istio-system
		-> Object: eventing-controller namespace: knative-eventing
		-> Object: eventing-webhook namespace: knative-eventing
		-> Object: imc-controller namespace: knative-eventing
		-> Object: imc-dispatcher namespace: knative-eventing
		-> Object: sources-controller namespace: knative-eventing
		-> Object: activator namespace: knative-serving
		-> Object: autoscaler namespace: knative-serving
		-> Object: controller namespace: knative-serving
		-> Object: webhook namespace: knative-serving
		-> Object: event-exporter-v0.2.5 namespace: kube-system
		-> Object: fluentd-gcp-scaler namespace: kube-system
		-> Object: heapster namespace: kube-system
		-> Object: kube-dns namespace: kube-system
		-> Object: kube-dns-autoscaler namespace: kube-system
		-> Object: l7-default-backend namespace: kube-system
		-> Object: metrics-server-v0.3.1 namespace: kube-system
		-> Object: stackdriver-metadata-agent-cluster-level namespace: kube-system

ReplicaSet found in apps/v1beta2
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping-r7f8r-deployment-54d688d777 namespace: default
		-> Object: grpc-ping-rc2jr-deployment-5f9c4fbf49 namespace: default
		-> Object: hello-lbsxh-3-deployment-6c867cf8dd namespace: default
		-> Object: hello-tdpmb-1-deployment-5fcc78b9ff namespace: default
		-> Object: hello-tjxhq-2-deployment-6d7b48d64c namespace: default
		-> Object: nginx-697b9dcc86 namespace: default
		-> Object: nginx-7c6cfbddbc namespace: default
		-> Object: default-broker-filter-9fbc5c667 namespace: event-example
		-> Object: default-broker-ingress-7948d9b8fc namespace: event-example
		-> Object: cluster-local-gateway-86db848cf5 namespace: gke-system
		-> Object: istio-ingress-5b4d9c44dd namespace: gke-system
		-> Object: istio-pilot-996485bd namespace: gke-system
		-> Object: istio-citadel-5448cbc9f8 namespace: istio-system
		-> Object: istio-citadel-5949896b4b namespace: istio-system
		-> Object: istio-citadel-5c88f4f47d namespace: istio-system
		-> Object: istio-galley-5bf8f944b7 namespace: istio-system
		-> Object: istio-ingressgateway-57449dd8bc namespace: istio-system
		-> Object: istio-ingressgateway-7c96766d85 namespace: istio-system
		-> Object: istio-ingressgateway-7f6db9967c namespace: istio-system
		-> Object: istio-pilot-56b86b87b6 namespace: istio-system
		-> Object: istio-pilot-56df8c9cf6 namespace: istio-system
		-> Object: istio-pilot-75654d4d9 namespace: istio-system
		-> Object: istio-policy-59dd46fb7d namespace: istio-system
		-> Object: istio-policy-68775985b8 namespace: istio-system
		-> Object: istio-policy-99fd7f7f5 namespace: istio-system
		-> Object: istio-sidecar-injector-5b5454d777 namespace: istio-system
		-> Object: istio-sidecar-injector-6cf9dc4549 namespace: istio-system
		-> Object: istio-sidecar-injector-bc79fc549 namespace: istio-system
		-> Object: istio-telemetry-5965f9d996 namespace: istio-system
		-> Object: istio-telemetry-699ffd959d namespace: istio-system
		-> Object: istio-telemetry-cdf9c6d7 namespace: istio-system
		-> Object: promsd-76f8d4cff8 namespace: istio-system
		-> Object: promsd-f876c8c55 namespace: istio-system
		-> Object: eventing-controller-7d75cd8598 namespace: knative-eventing
		-> Object: eventing-webhook-5cb89d8974 namespace: knative-eventing
		-> Object: imc-controller-654d689bc9 namespace: knative-eventing
		-> Object: imc-dispatcher-794f546c85 namespace: knative-eventing
		-> Object: sources-controller-67788d5b86 namespace: knative-eventing
		-> Object: activator-5477b7ff7 namespace: knative-serving
		-> Object: activator-6865899867 namespace: knative-serving
		-> Object: activator-74488ccb4c namespace: knative-serving
		-> Object: activator-7688586ccb namespace: knative-serving
		-> Object: autoscaler-5c56954884 namespace: knative-serving
		-> Object: autoscaler-7688987947 namespace: knative-serving
		-> Object: autoscaler-7c95566b65 namespace: knative-serving
		-> Object: autoscaler-85f5b489d7 namespace: knative-serving
		-> Object: controller-55bb588dbd namespace: knative-serving
		-> Object: controller-5ff95479f5 namespace: knative-serving
		-> Object: controller-78fd6cb64 namespace: knative-serving
		-> Object: controller-845759bc78 namespace: knative-serving
		-> Object: webhook-5dcf9dc6b5 namespace: knative-serving
		-> Object: webhook-64645549f6 namespace: knative-serving
		-> Object: webhook-759c4676bd namespace: knative-serving
		-> Object: webhook-77856f76fc namespace: knative-serving
		-> Object: event-exporter-v0.2.5-7d99d74cf8 namespace: kube-system
		-> Object: fluentd-gcp-scaler-54ccb89d5 namespace: kube-system
		-> Object: fluentd-gcp-scaler-55bdf597c namespace: kube-system
		-> Object: heapster-5f4c566b6d namespace: kube-system
		-> Object: heapster-64df54d744 namespace: kube-system
		-> Object: heapster-666fffdc58 namespace: kube-system
		-> Object: heapster-846cc4b8c6 namespace: kube-system
		-> Object: kube-dns-5877696fb4 namespace: kube-system
		-> Object: kube-dns-6987857fdb namespace: kube-system
		-> Object: kube-dns-autoscaler-85f8bdb54 namespace: kube-system
		-> Object: kube-dns-autoscaler-bb58c6784 namespace: kube-system
		-> Object: l7-default-backend-fd59995cd namespace: kube-system
		-> Object: metrics-server-v0.3.1-57c75779f namespace: kube-system
		-> Object: metrics-server-v0.3.1-c4cddd5f5 namespace: kube-system
		-> Object: stackdriver-metadata-agent-cluster-level-74d55f7669 namespace: kube-system
		-> Object: stackdriver-metadata-agent-cluster-level-8d8677c5d namespace: kube-system
		-> Object: stackdriver-metadata-agent-cluster-level-bb9d8bb64 namespace: kube-system

ControllerRevision found in apps/v1beta1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: daemonset-hold-image-5668b66bcd namespace: kube-system
		-> Object: fluentd-gcp-v3.1.1-57756ccc47 namespace: kube-system
		-> Object: fluentd-gcp-v3.1.1-595d677f66 namespace: kube-system
		-> Object: fluentd-gcp-v3.1.1-5c8c4c9864 namespace: kube-system
		-> Object: fluentd-gcp-v3.1.1-789d446b4c namespace: kube-system
		-> Object: fluentd-gcp-v3.1.1-864d84b477 namespace: kube-system
		-> Object: metadata-proxy-v0.1-5b4567fdff namespace: kube-system
		-> Object: metadata-proxy-v0.1-7d7b4bb744 namespace: kube-system
		-> Object: nvidia-gpu-device-plugin-64d8cbf474 namespace: kube-system
		-> Object: nvidia-gpu-device-plugin-65b49677fc namespace: kube-system
		-> Object: nvidia-gpu-device-plugin-6c79557f9b namespace: kube-system
		-> Object: prometheus-to-sd-5b58787fb5 namespace: kube-system
		-> Object: prometheus-to-sd-776c6cbcb namespace: kube-system
		-> Object: prometheus-to-sd-99cd5649b namespace: kube-system
		-> Object: prometheus-to-sd-9dd499d6d namespace: kube-system

Deployment found in apps/v1beta1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping-r7f8r-deployment namespace: default
		-> Object: grpc-ping-rc2jr-deployment namespace: default
		-> Object: hello-lbsxh-3-deployment namespace: default
		-> Object: hello-tdpmb-1-deployment namespace: default
		-> Object: hello-tjxhq-2-deployment namespace: default
		-> Object: nginx namespace: default
		-> Object: default-broker-filter namespace: event-example
		-> Object: default-broker-ingress namespace: event-example
		-> Object: cluster-local-gateway namespace: gke-system
		-> Object: istio-ingress namespace: gke-system
		-> Object: istio-pilot namespace: gke-system
		-> Object: istio-citadel namespace: istio-system
		-> Object: istio-galley namespace: istio-system
		-> Object: istio-ingressgateway namespace: istio-system
		-> Object: istio-pilot namespace: istio-system
		-> Object: istio-policy namespace: istio-system
		-> Object: istio-sidecar-injector namespace: istio-system
		-> Object: istio-telemetry namespace: istio-system
		-> Object: promsd namespace: istio-system
		-> Object: eventing-controller namespace: knative-eventing
		-> Object: eventing-webhook namespace: knative-eventing
		-> Object: imc-controller namespace: knative-eventing
		-> Object: imc-dispatcher namespace: knative-eventing
		-> Object: sources-controller namespace: knative-eventing
		-> Object: activator namespace: knative-serving
		-> Object: autoscaler namespace: knative-serving
		-> Object: controller namespace: knative-serving
		-> Object: webhook namespace: knative-serving
		-> Object: event-exporter-v0.2.5 namespace: kube-system
		-> Object: fluentd-gcp-scaler namespace: kube-system
		-> Object: heapster namespace: kube-system
		-> Object: kube-dns namespace: kube-system
		-> Object: kube-dns-autoscaler namespace: kube-system
		-> Object: l7-default-backend namespace: kube-system
		-> Object: metrics-server-v0.3.1 namespace: kube-system
		-> Object: stackdriver-metadata-agent-cluster-level namespace: kube-system

Route found in serving.knative.dev/v1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping namespace: default
		-> Object: hello namespace: default

Configuration found in serving.knative.dev/v1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping namespace: default
		-> Object: hello namespace: default

Service found in serving.knative.dev/v1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping namespace: default
		-> Object: hello namespace: default

Revision found in serving.knative.dev/v1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping-r7f8r namespace: default
		-> Object: grpc-ping-rc2jr namespace: default
		-> Object: hello-lbsxh-3 namespace: default
		-> Object: hello-tdpmb-1 namespace: default
		-> Object: hello-tjxhq-2 namespace: default

Service found in serving.knative.dev/v1beta1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping namespace: default
		-> Object: hello namespace: default

Revision found in serving.knative.dev/v1beta1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping-r7f8r namespace: default
		-> Object: grpc-ping-rc2jr namespace: default
		-> Object: hello-lbsxh-3 namespace: default
		-> Object: hello-tdpmb-1 namespace: default
		-> Object: hello-tjxhq-2 namespace: default

Configuration found in serving.knative.dev/v1beta1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping namespace: default
		-> Object: hello namespace: default

Route found in serving.knative.dev/v1beta1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping namespace: default
		-> Object: hello namespace: default

Route found in serving.knative.dev/v1alpha1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping namespace: default
		-> Object: hello namespace: default

Configuration found in serving.knative.dev/v1alpha1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping namespace: default
		-> Object: hello namespace: default

Service found in serving.knative.dev/v1alpha1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping namespace: default
		-> Object: hello namespace: default

Revision found in serving.knative.dev/v1alpha1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping-r7f8r namespace: default
		-> Object: grpc-ping-rc2jr namespace: default
		-> Object: hello-lbsxh-3 namespace: default
		-> Object: hello-tdpmb-1 namespace: default
		-> Object: hello-tjxhq-2 namespace: default

MeshPolicy found in authentication.istio.io/v1alpha1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> GLOBAL: default

PodAutoscaler found in autoscaling.internal.knative.dev/v1alpha1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping-r7f8r namespace: default
		-> Object: grpc-ping-rc2jr namespace: default
		-> Object: hello-lbsxh-3 namespace: default
		-> Object: hello-tdpmb-1 namespace: default
		-> Object: hello-tjxhq-2 namespace: default

Metric found in autoscaling.internal.knative.dev/v1alpha1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping-r7f8r namespace: default
		-> Object: grpc-ping-rc2jr namespace: default
		-> Object: hello-lbsxh-3 namespace: default
		-> Object: hello-tdpmb-1 namespace: default
		-> Object: hello-tjxhq-2 namespace: default

Image found in caching.internal.knative.dev/v1alpha1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping-r7f8r-cache namespace: default
		-> Object: grpc-ping-rc2jr-cache namespace: default
		-> Object: hello-lbsxh-3-cache namespace: default
		-> Object: hello-tdpmb-1-cache namespace: default
		-> Object: hello-tjxhq-2-cache namespace: default
		-> Object: queue-proxy namespace: knative-serving

Broker found in eventing.knative.dev/v1alpha1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: default namespace: event-example

Trigger found in eventing.knative.dev/v1alpha1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: hello-consumer namespace: event-example

InMemoryChannel found in messaging.knative.dev/v1alpha1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: default-kne-ingress namespace: event-example
		-> Object: default-kne-trigger namespace: event-example

Subscription found in messaging.knative.dev/v1alpha1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: default-hello-consumer-e7e99d36-17aa-11ea-8e23-42010a800016 namespace: event-example
		-> Object: internal-ingress-default-a92403a9-17a7-11ea-8e23-42010a800016 namespace: event-example

Ingress found in networking.internal.knative.dev/v1alpha1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping namespace: default
		-> Object: hello namespace: default

ServerlessService found in networking.internal.knative.dev/v1alpha1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping-r7f8r namespace: default
		-> Object: grpc-ping-rc2jr namespace: default
		-> Object: hello-lbsxh-3 namespace: default
		-> Object: hello-tdpmb-1 namespace: default
		-> Object: hello-tjxhq-2 namespace: default

tracespan found in config.istio.io/v1alpha2
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: stackdriver-span namespace: istio-system

handler found in config.istio.io/v1alpha2
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: kubernetesenv namespace: istio-system

attributemanifest found in config.istio.io/v1alpha2
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: istioproxy namespace: istio-system
		-> Object: kubernetes namespace: istio-system

kubernetes found in config.istio.io/v1alpha2
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: attributes namespace: istio-system

rule found in config.istio.io/v1alpha2
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: edgetosd namespace: istio-system
		-> Object: kubeattrgenrulerule namespace: istio-system
		-> Object: stackdriver-client namespace: istio-system
		-> Object: stackdriver-log namespace: istio-system
		-> Object: stackdriver-log-tcp namespace: istio-system
		-> Object: stackdriver-mixer namespace: istio-system
		-> Object: stackdriver-server namespace: istio-system
		-> Object: stackdriver-tcp-client namespace: istio-system
		-> Object: stackdriver-tcp-server namespace: istio-system
		-> Object: stackdriver-tracing-rule namespace: istio-system
		-> Object: tcpkubeattrgenrulerule namespace: istio-system

logentry found in config.istio.io/v1alpha2
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: server-accesslog-stackdriver namespace: istio-system
		-> Object: server-tcp-accesslog-stackdriver namespace: istio-system

stackdriver found in config.istio.io/v1alpha2
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: handler namespace: istio-system

metric found in config.istio.io/v1alpha2
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: client-received-bytes-count namespace: istio-system
		-> Object: client-request-bytes namespace: istio-system
		-> Object: client-request-count namespace: istio-system
		-> Object: client-response-bytes namespace: istio-system
		-> Object: client-roundtrip-latencies namespace: istio-system
		-> Object: client-sent-bytes-count namespace: istio-system
		-> Object: mixer-request-count namespace: istio-system
		-> Object: mixer-request-latency namespace: istio-system
		-> Object: server-received-bytes-count namespace: istio-system
		-> Object: server-request-bytes namespace: istio-system
		-> Object: server-request-count namespace: istio-system
		-> Object: server-response-bytes namespace: istio-system
		-> Object: server-response-latencies namespace: istio-system
		-> Object: server-sent-bytes-count namespace: istio-system

edge found in config.istio.io/v1alpha2
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: default namespace: istio-system

Gateway found in networking.istio.io/v1alpha3
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: cluster-local-gateway namespace: knative-serving
		-> Object: gke-system-gateway namespace: knative-serving
		-> Object: knative-ingress-gateway namespace: knative-serving

VirtualService found in networking.istio.io/v1alpha3
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: grpc-ping namespace: default
		-> Object: grpc-ping-mesh namespace: default
		-> Object: hello namespace: default
		-> Object: hello-mesh namespace: default

DestinationRule found in networking.istio.io/v1alpha3
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: istio-policy namespace: istio-system
		-> Object: istio-telemetry namespace: istio-system

NodeMetrics found in metrics.k8s.io/v1beta1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> GLOBAL: gke-gke-cluster-default-pool-ab457210-vbn1
		-> GLOBAL: gke-gke-cluster-default-pool-ab457210-z71m
		-> GLOBAL: gke-gke-cluster-default-pool-ab457210-37z1
		-> GLOBAL: gke-gke-cluster-default-pool-ab457210-dzg8
		-> GLOBAL: gke-gke-cluster-default-pool-ab457210-sh7q

PodMetrics found in metrics.k8s.io/v1beta1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: fluentd-gcp-v3.1.1-ltxjb namespace: kube-system
		-> Object: grpc-ping-r7f8r-deployment-54d688d777-njswn namespace: default
		-> Object: event-exporter-v0.2.5-7d99d74cf8-pmxf6 namespace: kube-system
		-> Object: promsd-76f8d4cff8-6bzxk namespace: istio-system
		-> Object: kube-dns-autoscaler-85f8bdb54-tq8q6 namespace: kube-system
		-> Object: prometheus-to-sd-jt4zm namespace: kube-system
		-> Object: istio-ingress-5b4d9c44dd-z4tv7 namespace: gke-system
		-> Object: istio-sidecar-injector-6cf9dc4549-j6ktm namespace: istio-system
		-> Object: nginx-697b9dcc86-gmxfh namespace: default
		-> Object: grpc-ping-r7f8r-deployment-54d688d777-ghmd9 namespace: default
		-> Object: nginx-697b9dcc86-tm2dt namespace: default
		-> Object: fluentd-gcp-scaler-54ccb89d5-xsbnh namespace: kube-system
		-> Object: controller-55bb588dbd-ztjds namespace: knative-serving
		-> Object: prometheus-to-sd-msvsg namespace: kube-system
		-> Object: kube-proxy-gke-gke-cluster-default-pool-ab457210-37z1 namespace: kube-system
		-> Object: eventing-controller-7d75cd8598-wfs6w namespace: knative-eventing
		-> Object: kube-dns-5877696fb4-j4cdd namespace: kube-system
		-> Object: cluster-local-gateway-86db848cf5-pllhj namespace: gke-system
		-> Object: webhook-5dcf9dc6b5-5xkj8 namespace: knative-serving
		-> Object: daemonset-hold-image-8mqn7 namespace: kube-system
		-> Object: grpc-ping-r7f8r-deployment-54d688d777-sw5rj namespace: default
		-> Object: grpc-ping-r7f8r-deployment-54d688d777-hfbhn namespace: default
		-> Object: istio-galley-5bf8f944b7-dxrr7 namespace: istio-system
		-> Object: prometheus-to-sd-n96wf namespace: kube-system
		-> Object: istio-ingressgateway-57449dd8bc-tw6hf namespace: istio-system
		-> Object: eventing-webhook-5cb89d8974-4ztxb namespace: knative-eventing
		-> Object: imc-controller-654d689bc9-m5fgv namespace: knative-eventing
		-> Object: kube-proxy-gke-gke-cluster-default-pool-ab457210-sh7q namespace: kube-system
		-> Object: prometheus-to-sd-5cl6p namespace: kube-system
		-> Object: fluentd-gcp-v3.1.1-4plmf namespace: kube-system
		-> Object: stackdriver-metadata-agent-cluster-level-74d55f7669-xd6xh namespace: kube-system
		-> Object: daemonset-hold-image-hvkkk namespace: kube-system
		-> Object: istio-policy-59dd46fb7d-xnj6b namespace: istio-system
		-> Object: autoscaler-85f5b489d7-t55nt namespace: knative-serving
		-> Object: istio-pilot-56b86b87b6-hcj4d namespace: istio-system
		-> Object: kube-dns-5877696fb4-c4bnt namespace: kube-system
		-> Object: istio-citadel-5448cbc9f8-rjtm4 namespace: istio-system
		-> Object: istio-telemetry-699ffd959d-wd27n namespace: istio-system
		-> Object: nginx-697b9dcc86-gpvwh namespace: default
		-> Object: sources-controller-67788d5b86-j6jf8 namespace: knative-eventing
		-> Object: daemonset-hold-image-q6hpq namespace: kube-system
		-> Object: fluentd-gcp-v3.1.1-zk656 namespace: kube-system
		-> Object: imc-dispatcher-794f546c85-mz7d8 namespace: knative-eventing
		-> Object: activator-7688586ccb-gl9qx namespace: knative-serving
		-> Object: kube-proxy-gke-gke-cluster-default-pool-ab457210-dzg8 namespace: kube-system
		-> Object: prometheus-to-sd-psrt2 namespace: kube-system
		-> Object: fluentd-gcp-v3.1.1-9r62t namespace: kube-system
		-> Object: metrics-server-v0.3.1-57c75779f-vf72x namespace: kube-system
		-> Object: kube-proxy-gke-gke-cluster-default-pool-ab457210-vbn1 namespace: kube-system
		-> Object: heapster-846cc4b8c6-h9lr4 namespace: kube-system
		-> Object: kube-proxy-gke-gke-cluster-default-pool-ab457210-z71m namespace: kube-system
		-> Object: default-broker-ingress-7948d9b8fc-85rr6 namespace: event-example
		-> Object: fluentd-gcp-v3.1.1-6w9rl namespace: kube-system
		-> Object: daemonset-hold-image-d4vgx namespace: kube-system
		-> Object: daemonset-hold-image-qwvm7 namespace: kube-system
		-> Object: default-broker-filter-9fbc5c667-bwr2b namespace: event-example
		-> Object: istio-pilot-996485bd-m4cs4 namespace: gke-system
		-> Object: l7-default-backend-fd59995cd-gm7t2 namespace: kube-system

Walker marking all objects deprecated

API Walker is marking every object (even the in core v1) as deprecated.

Needs to check what's and why this is happening

swagger url incorrect

While trying to check deprecations I am receiving error:

Error: could not download the swagger file https://raw.githubusercontent.com/kubernetes/kubernetes/1.20.11/api/openapi-spec/swagger.json
time="2021-09-24T15:46:54-04:00" level=error msg="An error has ocurred: could not download the swagger file https://raw.githubusercontent.com/kubernetes/kubernetes/1.20.11/api/openapi-spec/swagger.json"

The issue here is that the url is incorrect.

The correct path is:

https://raw.githubusercontent.com/kubernetes/kubernetes/v1.20.11/api/openapi-spec/swagger.json

The v before the version is missing

in the downloader.go utility it shows:

https://github.com/rikatz/kubepug/blob/9f4a2a2737c7d4755600a9bd2defb76180e5dbf7/pkg/utils/downloader.go#L13-L14

and these are used later:
https://github.com/rikatz/kubepug/blob/9f4a2a2737c7d4755600a9bd2defb76180e5dbf7/pkg/utils/downloader.go#L62

Not sure when it was changed but the current urls all have the v before the version number.

Workaround:
I have used the following to get around this, basically manually downloading and naming per the convention used in the downloader.go utility.
https://github.com/rikatz/kubepug/blob/9f4a2a2737c7d4755600a9bd2defb76180e5dbf7/pkg/utils/downloader.go#L57

basically it is swagger-<k8s version>.json

k_swag_version="1.20.11"
mkdir ~/kube_api_swagger_files
curl https://raw.githubusercontent.com/kubernetes/kubernetes/v${k_swag_version}/api/openapi-spec/swagger.json --output ~/kube_api_swagger_files/swagger-${k_swag_version}.json

kubectl deprecations --k8s-version $k_swag_version --swagger-dir ~/kube_api_swagger_files --input-file kubernetes.yaml

exit 1 if upgrade would be a breaking change

let's say I upgrade to 1.18 ... then I don't need to fix v1.19 deprecations

	 ├─ MutatingWebhookConfiguration describes the configuration of and admission webhook that accept or reject and may change the object. 
Deprecated in v1.16, planned for removal in v1.19. Use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration instead.

so it would be nice if I can get a exit 1 for "this will break in 1.18" so I can automate this task nicely against all clusters

Refactor methods

The whole kubepug runs around the DeprecatedAPIs object.

So instead of passing this map to every function, creating a type and methods for that would be probably better

kubepug reports an invalid Deprecation

Running kubepug against swagger version 1.14.9 against a 1.14.5 cluster I get list of deprecations against my deployments complaining that my deployments are found in extensions/v1beta1 which is simply not true. ALL of my deployment yamls specify apps/v1 as the apiVersion. Are you running kubectl under the covers because kubectl makes the same mistake when I do:

kubectl get deployments -o yam --v==8

In this case kubectl uses the extensions/v1beta1 API

My own java REST api uses the apps/v1 api when getting a list of deployments and all my deployments com back with that API. Any idea what's going on here?

Change the converter to create a swagger to structured data mapper

Swagger from Kubernetes uses a lot of different informations.

It would be nice if we could:

from .definitions extract only fields containing x-kubernetes-group-version-kind
Create a struct that could be used across all the program:

type KubernetesResource struct { 
   group: Group,
   version: version,
   kind: Kind,
   name: pluralName,
   description: description,
   deprecated: true
}


// the string identifier from the map shoud be "GV.Resource"?
var AllResources map[string]KubernetesResource

This map can be used in the flow:

-> Get all deprecated and check if they exist in the API Server (for, if v.deprecated)
-> Walk through all the APIServer (#1) and verify if this Kubernetes object exists in the map

parseable output format --json or so ?

we have lots of teams on our cluster, so having a way to get back "these 2 deployments are deprecated" in json would enable us to lookup up which team owns them and then send out notifications

Feature: Use YAML instead of Kubernetes as validation source

The tool currently checks objects in Kubernetes, which is great. I would also like to add it to our CI/CD to check manifests before they make it to Kubernetes. In that regards, I'd like to either pipe YAML/JSON into the tool and it provides the same output including error codes for the pipeline to fail.

Let me know your thoughts.

Deprecated resource being shown in multiple API groups

Thank you for your work on this project, seems very useful!

I installed kubepug just now and I'm seeing resources being flagged as deprecated in multiple API groups which creates a lot of noise. An example of what I mean:

On my v1.15 cluster, kubectl deprecations --k8s-version=v1.16.0 returns:

RESULTS:
Deprecated APIs:

Deployment found in apps/v1beta1
	 ├─ DEPRECATED - This group version of Deployment is deprecated by apps/v1beta2/Deployment. See the release notes for more information. Deployment enables declarative updates for Pods and ReplicaSets.
		-> OBJECT: cert-manager namespace: cert-manager
		-> OBJECT: cert-manager-cainjector namespace: cert-manager
		-> OBJECT: cert-manager-webhook namespace: cert-manager

[...snip...]

Deployment found in apps/v1beta2
	 ├─ DEPRECATED - This group version of Deployment is deprecated by apps/v1/Deployment. See the release notes for more information. Deployment enables declarative updates for Pods and ReplicaSets.
		-> OBJECT: cert-manager namespace: cert-manager
		-> OBJECT: cert-manager-cainjector namespace: cert-manager
		-> OBJECT: cert-manager-webhook namespace: cert-manager

[...snip...]

Deployment found in extensions/v1beta1
	 ├─ DEPRECATED - This group version of Deployment is deprecated by apps/v1beta2/Deployment. See the release notes for more information. Deployment enables declarative updates for Pods and ReplicaSets.
		-> OBJECT: cert-manager namespace: cert-manager
		-> OBJECT: cert-manager-cainjector namespace: cert-manager
		-> OBJECT: cert-manager-webhook namespace: cert-manager

This is happening for all resource groups (i.e Replicasets, Ingress, etc), not just Deployments.

I'm guessing this is related to the fact that the k8s API server will automatically marshall between apiversions as can be seen when specifying the apiversion with kubectl get as seen below. Note that these are the same resource, just a different representation.

kubectl -n cert-manager get deployment.v1beta1.extensions -o yaml

[...snip...]
- apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    name: cert-manager
    namespace: cert-manager
[...snip...]

versus:
kubectl -n cert-manager get deployment.apps -o yaml

[...snip...]
- apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: cert-manager
    namespace: cert-manager
[...snip...]

Rewrite the program logics to be more k8s cluster decoupled

While dealing with #65 I've remembered that we need to connect to a Kubernetes Cluster to convert an Object Kind (singular) to Object Name (plural) because the dynamic client-go needs it to List the current Resources of the cluster:

https://github.com/rikatz/kubepug/blob/v1.0.1/pkg/kubepug/k8sdeprecated.go#L43-L44

But, what happens is that we don't really need this information until we decide to swipe a Kubernetes Cluster looking for deprecated objects. In case of input files the current Object Kind is already used.

DeprecatedAPIs uses a key/value structure because the Walker uses here the Name, but still this can be converted to the Kind, as the APIResource object contains both of them.

So some steps will be taken to ship a better decoupling:

Isolate the part of the code that downloads the swagger.json and maybe put the conversion of the Swagger to a map of APIs, which the key will be group/version/kind and will contain the current structure already existing in here
Create a interface that will run the parts of the code that might compare any input with the KubernetesAPI map, generating a set of output
The interface may contain two Methods: ListDeprecated and ListDeleted
When the input is a *genericclioptions.ConfigFlags it will populate each KubernetesAPI map with the missing object resource name (plural), to be used later by the function of ListDeprecated of the K8s cluster. ListDeleted (Walker) will keep the same logic, only changing the map that it compares from object.ResourceName to object.Kind
When the input is a file (yaml, json, whatever) we can convert it to whatever the program understand (or have different types and a new interface) and simply compare it with the Map, so if the object is found but flagged as deprecated this might be added to the Deprecated Results. If it's not found, it might be in the Deleted. Still, we can in this pass ignore some common group/versions related to CRDs, but this might generate some fake results, because if we don't have access to the Kubernetes Cluster it's not possible to check wether the group/version of some input object is part of a CRD or it's something that really doesn't exists.

Add unit tests

Add some unit tests

Ignore CRD

Pug should ignore CRD:

Ex.:

BackupStorageLocation found in ark.heptio.com/v1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: default namespace: heptio-ark

Backup found in ark.heptio.com/v1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: dsr-cluster-producao-20200106000001 namespace: heptio-ark
		-> Object: dsr-cluster-producao-20200106120002 namespace: heptio-ark
		-> Object: dsr-cluster-producao-20200107000003 namespace: heptio-ark
		-> Object: dsr-cluster-producao-20200107120003 namespace: heptio-ark
		-> Object: dsr-cluster-producao-20200108000004 namespace: heptio-ark
		-> Object: dsr-cluster-producao-20200108120005 namespace: heptio-ark

Replace all the 'panics' with something more user friendly

Maybe using "log" package?

Ignore CRD

Pug should ignore CRD:

Ex.:

BackupStorageLocation found in ark.heptio.com/v1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: default namespace: heptio-ark

Backup found in ark.heptio.com/v1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: dsr-cluster-producao-20200106000001 namespace: heptio-ark
		-> Object: dsr-cluster-producao-20200106120002 namespace: heptio-ark
		-> Object: dsr-cluster-producao-20200107000003 namespace: heptio-ark
		-> Object: dsr-cluster-producao-20200107120003 namespace: heptio-ark
		-> Object: dsr-cluster-producao-20200108000004 namespace: heptio-ark
		-> Object: dsr-cluster-producao-20200108120005 namespace: heptio-ark

don't write to current dir

Consider just creating a temp dir please.

Failed to create the K8s client while listing Deprecated objects

Hi I just installed the latest version of kubepug

GitVersion:    devel
GitCommit:     42529d7c573cbc2f5cb8d4810b715b29b76e0289
GitTreeState:  clean
BuildDate:     '2021-11-29T01:19:53Z'
GoVersion:     go1.17.3
Compiler:      gc
Platform:      linux/amd64

When I try to use it on my kubernetes cluster I get this error

> kubepug --k8s-version=v1.22.2
time="2021-11-29T09:13:06+01:00" level=fatal msg="Failed to create the K8s client while listing Deprecated objects"

It was working before I install the latest version, my cluster is operational and here is the version

> kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:28:09Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.14-gke.1900", GitCommit:"abc4e63ae76afef74b341d2dba1892471781604f", GitTreeState:"clean", BuildDate:"2021-09-07T09:21:04Z", GoVersion:"go1.15.15b5", Compiler:"gc", Platform:"linux/amd64"}

Feature request prometheus metrics for deprecation warnings

Opening this github issue as a feature request to instrument kubepug with prometheus metrics. This would allow other subsystems to consume metrics for monitoring/alerting purposes. I realize that this would likely change the operational behavior of kubepug from a cli that is run adhoc to one that is run persistently. Feel free to close this issue, if this request is out of scope of kubepug.

Happy to help with a PR here, if this is something of interest to the core maintainers.

Regression in 1.1.0

Updated to 1.1.0 and now it doesn't find the deprecations anymore.

Dependabot can't parse your go.mod

Dependabot couldn't parse the go.mod found at /go.mod.

The error Dependabot encountered was:

go: github.com/spf13/[email protected] requires
	gopkg.in/[email protected] requires
	gopkg.in/[email protected]: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /opt/go/gopath/pkg/mod/cache/vcs/9241c28341fcedca6a799ab7a465dd6924dc5d94044cbfabb75778817250adfc: exit status 128:
	fatal: The remote end hung up unexpectedly

View the update logs.

Aggregated apiserver APIs are marked as removed

Looks like the APIs of aggregated apiservers like metrics-server are checked against the official swagger and reported removed, which is false:

NodeMetrics found in metrics.k8s.io/v1beta1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> GLOBAL: demomaster1test 
		-> GLOBAL: demoworker1test 
		-> GLOBAL: demoworker2test 
		-> GLOBAL: demoworker3test 
		-> GLOBAL: demoworker4test 

PodMetrics found in metrics.k8s.io/v1beta1
	 ├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELLY!!
		-> Object: nginx-ingress-default-backend-576b86996d-94pqh namespace: ingress
		-> Object: kubernetes-dashboard-proxy-654cb7d867-rhmpn namespace: kube-system
		-> Object: prometheus-node-exporter-lhmzc namespace: prometheus
		-> Object: kube-proxy-b69sk namespace: kube-system
		-> Object: canal-pjxlk namespace: kube-system
                 ...

# kubectl get apiservices.apiregistration.k8s.io 
NAME                                   SERVICE                      AVAILABLE   AGE
...
v1.coordination.k8s.io                 Local                        True        94d
v1.crd.projectcalico.org               Local                        True        90m
v1.scheduling.k8s.io                   Local                        True        94d
v1.storage.k8s.io                      Local                        True        238d
v1beta1.extensions                     Local                        True        238d
v1beta1.metrics.k8s.io                 kube-system/metrics-server   True        238d
v1beta1.networking.k8s.io              Local                        True        94d
...

Output kubepug's version

Is there a way to output kubepug's own version? To see if I need to upgrade the Pre UpGrade (Checker) ;-)

Kind List not supported

Templates of kind list always comes back with an "deleted api" report.

Will happen if you run kubepug against yamls files like in the exemple for ResourseQuotas https://kubernetes.io/docs/concepts/policy/resource-quotas/

Client 1.19 now support warning deprecated apis

As per Jordan Liggitt (wont mark him here to not spam the GH box) the future client-go 1.19 now supports warning about deprecated APIs (and also the server side).

This probably will simplify a lot of things here :)

Opening the issue to keep a track and implement as soon as possible!

kubernetes/website#21581

kubernetes/kubernetes#73032

Invalid VerticalPodAutoscaler deprecation

Hello, currently I use this plugin to validate local k8s manifest files which works pretty well. I realised last updates probably introduced an issue with k8s v1.18.0 + autoscaling.k8s.io/v1beta2 VPA.

It says that k8s v1.18.0 removed autoscaling.k8s.io/v1beta2, but actually it supposed to be delete next k8s versions.

kubectl-deprecations --k8s-version v1.18.0 --input-file datadog-v7/agent-vpa.yaml --error-on-deleted

RESULTS:
Deprecated APIs:

Deleted APIs:

VerticalPodAutoscaler found in autoscaling.k8s.io/v1beta2
├─ API REMOVED FROM THE CURRENT VERSION AND SHOULD BE MIGRATED IMMEDIATELY!!
-> OBJECT: datadog-agent namespace: kube-system location: datadog-v7/agent-vpa.yaml

Error: found 1 Deleted APIs and 0 Deprecated APIs
time="2021-11-30T01:08:16Z" level=error msg="An error has occurred: found 1 Deleted APIs and 0 Deprecated APIs"

Update README.md

We got some brand new arguments, a better application that have multiple inputs and outputs so why not update the README.md and maybe also create a page for this? ;)

"Found invalid yaml" warnings on valid YAML with --- in a value

I'm seeing "Found invalid yaml" warnings on input that contains --- in a value, e.g.

apiVersion: v1
kind: ConfigMap
metadata:
  name: foo
data:
  FOO: '---'

$ kubepug --k8s-version v1.23.0 --input-file foo.yaml --error-on-deleted --error-on-deprecated
time="2022-01-11T16:06:23Z" level=warning msg="Found invalid yaml: error converting YAML to JSON: yaml: line 4: found unexpected end of stream. Skipping to next"
time="2022-01-11T16:06:23Z" level=warning msg="Found invalid yaml: error converting YAML to JSON: yaml: line 6: mapping values are not allowed in this context. Skipping to next"

This is valid YAML:

$ yamllint foo.yaml ; echo $?
0

(Related: When used in a multi-document YAML file the line numbers are the same; it appears they're relative to the individual components of the multi-document YAML file, but the component that's reporting the warnings isn't identified.)

kubepug doesn't find deprecated batch/v1beta1 CronJob API

kubepug -k8s-version v1.21.0 --input-file deprecated-cronjob.yaml --error-on-deleted --error-on-deprecated passes for this file (using v1.21.0, v1.22.0, and v1.23.0) but when deploying it to a v1.21.0 cluster I get the warning Warning: batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: foo
spec:
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: foo
            image: ...
          restartPolicy: Never
  schedule: 0 0 0 0 0

Can't get "--input-file" to work

Hello,

I tried to use the --input-file option with a single file, a folder or things like "cat *.yaml" but it never works.

I always get :

level=warning msg="Found invalid yaml: ./FILENAME: yaml: did not find expected node content. Skipping to next"

For exemple with the gitlab-runner helm charts :

helm pull gitlab/gitlab-runner --version 0.23.0
tar xzvf gitlab-runner-0.23.0.tgz 
kubepug --k8s-version=v1.15.5 --input-file=./gitlab-runner/templates
time="2020-12-07T12:07:09+01:00" level=warning msg="Found invalid yaml: ./gitlab-runner/templates//NOTES.txt: yaml: did not find expected node content. Skipping to next"
time="2020-12-07T12:07:09+01:00" level=warning msg="Found invalid yaml: ./gitlab-runner/templates//_cache.tpl: yaml: did not find expected node content. Skipping to next"
time="2020-12-07T12:07:09+01:00" level=warning msg="Found invalid yaml: ./gitlab-runner/templates//_env_vars.tpl: yaml: did not find expected node content. Skipping to next"
time="2020-12-07T12:07:09+01:00" level=warning msg="Found invalid yaml: ./gitlab-runner/templates//_helpers.tpl: yaml: did not find expected alphabetic or numeric character. Skipping to next"
time="2020-12-07T12:07:09+01:00" level=warning msg="Found invalid yaml: ./gitlab-runner/templates//configmap.yaml: yaml: line 70: could not find expected ':'. Skipping to next"
time="2020-12-07T12:07:09+01:00" level=warning msg="Found invalid yaml: ./gitlab-runner/templates//deployment.yaml: yaml: line 22: could not find expected ':'. Skipping to next"
time="2020-12-07T12:07:09+01:00" level=warning msg="Found invalid yaml: ./gitlab-runner/templates//hpa.yaml: yaml: did not find expected node content. Skipping to next"
time="2020-12-07T12:07:09+01:00" level=warning msg="Found invalid yaml: ./gitlab-runner/templates//role-binding.yaml: yaml: did not find expected node content. Skipping to next"
time="2020-12-07T12:07:09+01:00" level=warning msg="Found invalid yaml: ./gitlab-runner/templates//role.yaml: yaml: did not find expected node content. Skipping to next"
time="2020-12-07T12:07:09+01:00" level=warning msg="Found invalid yaml: ./gitlab-runner/templates//secrets.yaml: yaml: did not find expected node content. Skipping to next"
time="2020-12-07T12:07:09+01:00" level=warning msg="Found invalid yaml: ./gitlab-runner/templates//service-account.yaml: yaml: did not find expected node content. Skipping to next"
RESULTS:
Deprecated APIs:


Deleted APIs:

What am I doing wrong ?

I'm trying this option because i don't manage to have the kubepug command working with serviceaccount that can only access one specific namespace (but if it can be done I will take this way).

time="2020-12-07T10:27:52Z" level=fatal msg="Failed to list objects in the cluster. Permission denied! Please check if you have the proper authorization"

Regards,

Johan

Support offline swagger files

In some environments, internet is inaccessible. It would be great if kubepug had a command line option to support a local file as input.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.