doitintl / iris3 Goto Github PK

View Code? Open in Web Editor NEW

70.0 6.0 22.0 851 KB

An upgraded and improved version of the Iris automatic GCP-labeling project

License: MIT License

Shell 18.64% Python 81.36%

google-cloud google-cloud-platform cost-control cloud-cost labeling set-labels bigquery cloud-storage pubsub gcp

iris3's People

Contributors

Stargazers

Watchers

Forkers

thesassyking mithun2507 syllogy climacell containersolutions rafaelzanoni rodrigorbonfim dannielmts fjhuertal sc-developer code-monkeys frossi-git devmarcoslima won21kr geekflyer usaptos aptos-labs maria980000 djktno

iris3's Issues

Don't automatically label all resources

I installed it with the default settings, but there are no labeled resources. I also tried the manual trigger but it didn't work.
I have provided some information below
Thanks.

./deploy.sh

2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; Sent do_label message for acoustic-gizmo-400701 , Subscriptions
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; Finished publisha attempt   iris_schedulelabeling_topic: {"project_id": "acoustic-gizmo-400701", "plugin": "Topics"}
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; Sent do_label message for acoustic-gizmo-400701 , Topics
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; Finished publisha attempt   iris_schedulelabeling_topic: {"project_id": "<project-id>", "plugin": "Bigquery"}
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; Sent do_label message for <project-id> , Bigquery
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; Finished publisha attempt   iris_schedulelabeling_topic: {"project_id": "<project-id>", "plugin": "Buckets"}
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; Sent do_label message for <project-id> , Buckets
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; Finished publisha attempt   iris_schedulelabeling_topic: {"project_id": "<project-id>", "plugin": "Cloudsql"}
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; Sent do_label message for <project-id> , Cloudsql
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; Finished publisha attempt   iris_schedulelabeling_topic: {"project_id": "<project-id>", "plugin": "Disks"}
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; Sent do_label message for <project-id> , Disks
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; Finished publisha attempt   iris_schedulelabeling_topic: {"project_id": "<project-id>", "plugin": "Instances"}
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; Sent do_label message for <project-id> , Instances
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; Finished publisha attempt   iris_schedulelabeling_topic: {"project_id": "<project-id>", "plugin": "Snapshots"}
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; Sent do_label message for <project-id> , Snapshots
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; Finished publisha attempt   iris_schedulelabeling_topic: {"project_id": "<project-id>", "plugin": "Subscriptions"}
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; Sent do_label message for <project-id> , Subscriptions
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; Finished publisha attempt   iris_schedulelabeling_topic: {"project_id": "<project-id>", "plugin": "Topics"}
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; Sent do_label message for <project-id> , Topics
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; schedule() sent 16 messages to label 2 projects
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; GAEInst x100no {'schedule': 1}; end schedule; RAM 398m; Libs:[pubsub_v1,resourcemanager_v3];
2024-01-10 06:42:25 iris3[20240110t044551]  INFO; iris_auto_label; Trace: 2ede21;o=1; /schedule; timing: schedule(): 4350 ms

config.yaml
# Copy this to make your config.yaml;
# You can also make  config-dev.yaml and config-test.yaml

# The default values as-is should work out-of-the-box.

# Keys:

# projects: Only resources in these projects will get labeled.
# But if the value is empty, *all* projects in the organization are included.
projects: []
#  - projects/<project-id>

# plugins: Only these plugins are enabled.
# For example, add some of these to tiris3/uninstall_scriptshe list:
#     bigquery, buckets, disks,  cloudsql, instances, snapshots, subscriptions, topics
# But if the value is empty, *all* plugins are enabled.
plugins: []

# iris_prefix plus underscore is prefixed to the key of each label that is added.
#    If empty string is used (the default), then no prefix and no underscore is addded.
iris_prefix: iris_auto_label

# specific_prefixes gives a prefix per resource type, instead of iris_prefix.
# The default is that there is no override.
# About the meaning of null (missing) values vs empty-string:
#  - A null (missing) value, as in the Buckets ex ample below,
#   will not replace the general iris_prefix.
#  -  Empty string as "", as in the Bigquery example below, overrides
#   the iris_prefix, so that you get labels with no prefix.
# For an example, see the comment below.
specific_prefixes: {}

# Example:
# specific_prefixes:
#   Buckets:
#   Bigquery: ""
#   Cloudsql: sql
#   Disks: gcedisk
#   Instances: gce
#   Snapshots:
#   Subscriptions:
#   Topics:

# If from_project is True, then for each resource we are labeling, copy the labels from its project onto it.
# The default is False.
from_project: True

# If label_all_on_cron is False (the default), then to save money,
#  only resources of certain types get labeled on cron: those whose plugins either
#    - return True on relabel_on_cron() (like Disks)
#    - or return False in is_labeled_on_creation() (like Cloud SQL)
# If it isTrue, then all resource types will be labeled on every Cloud Scheduler cycle.
#   This is useful for labeling existing resources when you first launch Iris3.

label_all_on_cron: True

# Optionally change this token before first deployment for added security in
# communication between PubSub and the Iris App on App Engine.
# You could even re-generate a new token per deployment.
# Note that this token-based approach is not very secure, though it  was once recommended by Google.
# However, so long as the GCP project running the Iris3 AppEngine service is otherwise secure,
# this token protects against unwanted invocations of labeling.
pubsub_verification_token: 0b0a30cde7e3489f0a9cd74bb51c514d

# If running (i.e., in AppEngine) in a project with one of the strings in the name,
# then *scheduled* labelings (those created by the Cloud Scheduler)
# will fail if more than 3 projects are enabled.
# This is intended to prevent a situation where you forget to limit the scope of
# labeling in development (see "projects" key above),
# and so accidentally labeling your entire org with test-labels.
test_or_dev_project_markers:
  - playground
  - test
  - dev
  - qa

Cannot handle more than 1000 objects of a given type in cron-based labeling

This possible never worked, even in Iris2. After you execute a batch object, it still holds the old requests; it is not cleared. Reusing a batch object means adding more and more objects to it.

Fix:

Recreate self._batch object after executing the batch
Reduce batch limit to 990 to avoid off-by-one errors (since the GCE API has an internal limit of 1000)

Fix in branch nogke-and-over1k-obj

Diff here

master...nogke-and-over1k-obj#diff-08655555cc53ca79554ba180d1ae7e6ebe5c9b0a25e1772ce248fdd8b83dab34R97

Error line 101 Deploy.sh

Hello, in the IRIS3 project, deploy.sh has an error on line 101, to correct the folder, replace the gcloud service list with the command below:
change:
done < <(gcloud services list | tail -n +2)

to:
done < <(gcloud services list --format="value(config.name,config.title)")

allow user to create the custom role

A flag that can disable deploy.sh from attempting to create the custom role (the one named iris3), so that the user can pass in the name of an existing role

Feature request: Label each disk with the name of the VM that it is attached to, if any

The current code adds only these labels: the zone of the disk, its name, its region, whether it is attached

Similar to this, need to a method _gcp_instance() to class Disks, and pull that info from the creation-log data which is passed in gcp_object , and they’d get a label like iris_instance

Note: Disk-Instance-Attachment is a mutable value, unlike most values that Iris works with. However, disks are exceptional in that Iris will by default relabel them on each scheduled run in order to capture changes.

Uninstall script

To uninstall Iris, delete

On the org level (see _deploy-org.sh), delete
- Custom role iris3 along with the policy binding granting this role to the built-in App Engine service account [email protected]
- Log sinks iris_sink
On the project level (see _deploy-project.sh), delete
- The Cloud Scheduler job
- The GAE service, from CLI or the GCP Console.
The policy bindings allowing the PubSub service account to access label_one and do_label subscriptions and iris_deadletter_topic Topic
The PubSub topics
- iris_schedulelabeling_topic
- iris_deadletter_topic
and the associated PubSub subscriptions
- iris_deadletter
- do_label
- label_one

If I missed something, please write it here in the issue report.

If you write a script for this, please submit a PR.

AlloyDB resource support

Adding AlloyDB resource support.

Question: Addition of custom labels in the roadmap?

Hello,

Are you considering adding the ability to add custom labels to projects and/or resources?

e.g.:

environment = dev
owner = "joe smith"

etc.

Thanks!

Add iris_pd_attached label to resources of persistent disk type

Is your feature request related to a problem? Please describe.
Google billing export doesn't have information on whether the persistent disk is attached or detached. Therefore, it's hard to identify unattached disks when looking thru billing records.

Describe the solution you'd like
Add iris_pd_attached label key to persistent disk resources. The value can be True or False, signaling whether the disk is attached or not.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Error cannot import name 'json' from 'itsdangerous'

A fresh test installation of iris logs the following error, after creating a VM instance that should be tagged:

ImportError: cannot import name 'json' from 'itsdangerous' (/layers/google.python.pip/pip/lib/python3.8/site-packages/itsdangerous/__init__.py)

at .<module> ( [/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/json/__init__.py:15](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Flayers%2Fgoogle.python.pip%2Fpip%2Flib%2Fpython3.8%2Fsite-packages%2Fflask%2Fjson%2F__init__.py&line=15&project=iris-test-xxxxxx) )
at .<module> ( [/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/__init__.py:19](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Flayers%2Fgoogle.python.pip%2Fpip%2Flib%2Fpython3.8%2Fsite-packages%2Fflask%2F__init__.py&line=19&project=iris-test-xxxxxx) )
at .<module> ( [/srv/main.py:11](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fsrv%2Fmain.py&line=11&project=iris-test-xxxxxx) )
at ._call_with_frames_removed ( [<frozen importlib._bootstrap>:219](https://console.cloud.google.com/debug?referrer=fromlog&file=%3Cfrozen%20importlib._bootstrap%3E&line=219&project=iris-test-xxxxxx) )
at .exec_module ( [<frozen importlib._bootstrap_external>:843](https://console.cloud.google.com/debug?referrer=fromlog&file=%3Cfrozen%20importlib._bootstrap_external%3E&line=843&project=iris-test-xxxxxx) )
at ._load_unlocked ( [<frozen importlib._bootstrap>:671](https://console.cloud.google.com/debug?referrer=fromlog&file=%3Cfrozen%20importlib._bootstrap%3E&line=671&project=iris-test-xxxxxx) )
at ._find_and_load_unlocked ( [<frozen importlib._bootstrap>:975](https://console.cloud.google.com/debug?referrer=fromlog&file=%3Cfrozen%20importlib._bootstrap%3E&line=975&project=iris-test-xxxxxx) )
at ._find_and_load ( [<frozen importlib._bootstrap>:991](https://console.cloud.google.com/debug?referrer=fromlog&file=%3Cfrozen%20importlib._bootstrap%3E&line=991&project=iris-test-xxxxxx) )
at ._gcd_import ( [<frozen importlib._bootstrap>:1014](https://console.cloud.google.com/debug?referrer=fromlog&file=%3Cfrozen%20importlib._bootstrap%3E&line=1014&project=iris-test-xxxxxx) )
at .import_module ( [/opt/python3.8/lib/python3.8/importlib/__init__.py:127](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fopt%2Fpython3.8%2Flib%2Fpython3.8%2Fimportlib%2F__init__.py&line=127&project=iris-test-xxxxxx) )
at .import_app ( [/layers/google.python.pip/pip/lib/python3.8/site-packages/gunicorn/util.py:359](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Flayers%2Fgoogle.python.pip%2Fpip%2Flib%2Fpython3.8%2Fsite-packages%2Fgunicorn%2Futil.py&line=359&project=iris-test-xxxxxx) )
at .load_wsgiapp ( [/layers/google.python.pip/pip/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py:48](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Flayers%2Fgoogle.python.pip%2Fpip%2Flib%2Fpython3.8%2Fsite-packages%2Fgunicorn%2Fapp%2Fwsgiapp.py&line=48&project=iris-test-xxxxxx) )
at .load ( [/layers/google.python.pip/pip/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py:58](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Flayers%2Fgoogle.python.pip%2Fpip%2Flib%2Fpython3.8%2Fsite-packages%2Fgunicorn%2Fapp%2Fwsgiapp.py&line=58&project=iris-test-xxxxxx) )
at .wsgi ( [/layers/google.python.pip/pip/lib/python3.8/site-packages/gunicorn/app/base.py:67](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Flayers%2Fgoogle.python.pip%2Fpip%2Flib%2Fpython3.8%2Fsite-packages%2Fgunicorn%2Fapp%2Fbase.py&line=67&project=iris-test-xxxxxx) )
at .load_wsgi ( [/layers/google.python.pip/pip/lib/python3.8/site-packages/gunicorn/workers/base.py:146](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Flayers%2Fgoogle.python.pip%2Fpip%2Flib%2Fpython3.8%2Fsite-packages%2Fgunicorn%2Fworkers%2Fbase.py&line=146&project=iris-test-xxxxxx) )
at .init_process ( [/layers/google.python.pip/pip/lib/python3.8/site-packages/gunicorn/workers/base.py:134](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Flayers%2Fgoogle.python.pip%2Fpip%2Flib%2Fpython3.8%2Fsite-packages%2Fgunicorn%2Fworkers%2Fbase.py&line=134&project=iris-test-xxxxxx) )
at .init_process ( [/layers/google.python.pip/pip/lib/python3.8/site-packages/gunicorn/workers/gthread.py:92](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Flayers%2Fgoogle.python.pip%2Fpip%2Flib%2Fpython3.8%2Fsite-packages%2Fgunicorn%2Fworkers%2Fgthread.py&line=92&project=iris-test-xxxxxx) )
at .spawn_worker ( [/layers/google.python.pip/pip/lib/python3.8/site-packages/gunicorn/arbiter.py:589](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Flayers%2Fgoogle.python.pip%2Fpip%2Flib%2Fpython3.8%2Fsite-packages%2Fgunicorn%2Farbiter.py&line=589&project=iris-test-xxxxxx) )

Googling for the problem I found similar answers:
https://itsmycode.com/importerror-cannot-import-name-json-from-itsdangerous
https://stackoverflow.com/questions/71189819/python-docker-importerror-cannot-import-name-json-from-itsdangerous

I tried updating requirements.txt with:

Flask==1.1.4
markupsafe==2.0.1

but I had versions conflicts:

ERROR: Cannot install -r requirements.txt (line 1) and -r requirements.txt (line 9) because these package versions have conflicting dependencies.

The conflict is caused by:
    flask 1.1.4 depends on click<8.0 and >=5.1
    black 22.1.0 depends on click>=8.0.0

Then I tried updating to the latest Flask version:

Flask==2.0.3

This last attempt was successful, the application is deployed correctly and no more errors are logged.
VM instances are tagged correctly, but i haven't tested other services.

In Yaml, use [] not {} to indicate empty-list

In Yaml, use [] not {} to indicate empty-list. See config.yaml

Iris3 deployment stops with error "TypeError: Expected maxsize to be an integer or None"

I tried deploying iris3 but got an error at the end. Here is the whole output of my attempt to deploy it (./deploy.sh iris3-306714):

+ set -u
+ set -e
+ [[ 5.0.3(1)-release == 3. ]]
++ date +%s
+ START=1615232733
+ ROLEID=iris3
+ LOGS_TOPIC=iris_logs_topic
+ SCHEDULELABELING_TOPIC=iris_schedulelabeling_topic
+ LOG_SINK=iris_log
+ DO_LABEL_SUBSCRIPTION=do_label
+ LABEL_ONE_SUBSCRIPTION=label_one
+ REGION=us-central
+ GAE_REGION_ABBREV=uc
+ [[ 1 -eq 0 ]]
+ PROJECTID=iris3-306714
+ shift
+ CRON_ONLY=
+ getopts c opt
+ gcloud projects describe iris3-306714
createTime: '2021-03-05T14:58:31.398Z'
lifecycleState: ACTIVE
name: Iris3
parent:
  id: '273839554717'
  type: organization
projectId: iris3-306714
projectNumber: '969185023301'
+ echo 'Project ID iris3-306714'
Project ID iris3-306714
+ gcloud config set project iris3-306714
Updated property [core/project].
++ grep service: app.yaml
++ awk '{print $2}'
+ GAE_SVC=iris3
++ grep ' PUBSUB_VERIFICATION_TOKEN:' app.yaml
++ awk '{print $2}'
+ PUBSUB_VERIFICATION_TOKEN=2a343f4c1b76512039fe763412756c4fbb30c
+ LABEL_ONE_SUBSCRIPTION_ENDPOINT='https://iris3-dot-iris3-306714.uc.r.appspot.com/label_one?token=2a343f4c1b76512039fe763412756c4fbb30c'
+ DO_LABEL_SUBSCRIPTION_ENDPOINT='https://iris3-dot-iris3-306714.uc.r.appspot.com/do_label?token=2a343f4c1b76512039fe763412756c4fbb30c'
+ declare -A enabled_services
+ read -r svc _
++ gcloud services list
++ tail -n +2
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ enabled_services["$svc"]=yes
+ read -r svc _
+ required_svcs=(cloudresourcemanager.googleapis.com pubsub.googleapis.com compute.googleapis.com bigtable.googleapis.com bigtableadmin.googleapis.com storage-component.googleapis.com sql-component.googleapis.com sqladmin.googleapis.com)
+ for svc in "${required_svcs[@]}"
+ '[' _ ']'
+ for svc in "${required_svcs[@]}"
+ '[' _ ']'
+ for svc in "${required_svcs[@]}"
+ '[' _ ']'
+ for svc in "${required_svcs[@]}"
+ '[' _ ']'
+ for svc in "${required_svcs[@]}"
+ '[' _ ']'
+ for svc in "${required_svcs[@]}"
+ '[' _ ']'
+ for svc in "${required_svcs[@]}"
+ '[' _ ']'
+ for svc in "${required_svcs[@]}"
+ '[' _ ']'
++ grep -A 1 organization
++ tail -n 1
+++ gcloud auth print-access-token
++ cut '-d"' -f4
++ tr -d ' '
++ curl -X POST -H 'Authorization: Bearer "ya29.a0AfH6SMD-cDqjFIEodSZFf80R6DJz-vnF8HkVn12dOXipMWGwoSmLVHFGFsY_v_nPf890UYQgl-LT0hj_yLSReiTFBfvrG86uX2_7h_yRIqg5QSDzZnEAExGdHQYgPu-gIXDxjUihAoXx3HepSZGkhx6xyim61RlmbHbtoMk9EA9TPjQytebAMCM35q5F4e15y-YLfxf0wa5pHwKvzzs17aQg8rIigVo-xoDoB0AjqPQWtPdLyNpvUU7tRW_JYRNVfB488Po"' -H 'Content-Type: application/json; charset=utf-8' https://cloudresourcemanager.googleapis.com/v1/projects/iris3-306714:getAncestry
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0^M100   226    0   226    0     0   1506      0 --:--:-- --:--:-- --:--:--  1506
+ ORGID=273839554717
+ gcloud app describe
+ gcloud iam roles describe iris3 --organization 273839554717
description: Iris auto tagging service for google cloud
etag: BwW9C6c_6TM=
includedPermissions:
- bigquery.datasets.get
- bigquery.datasets.update
- bigquery.tables.delete
- bigquery.tables.get
- bigquery.tables.list
- bigquery.tables.update
- bigtable.clusters.get
- bigtable.clusters.list
- bigtable.clusters.update
- bigtable.instances.get
- bigtable.instances.list
- bigtable.instances.update
- bigtable.tables.get
- bigtable.tables.list
- cloudsql.instances.get
- cloudsql.instances.list
- cloudsql.instances.update
- compute.disks.get
- compute.disks.list
- compute.disks.setLabels
- compute.instances.get
- compute.instances.list
- compute.instances.setLabels
- compute.instances.setTags
- compute.projects.get
- compute.regions.get
- compute.regions.list
- compute.snapshots.get
- compute.snapshots.list
- compute.snapshots.setLabels
- compute.zones.get
- compute.zones.list
- pubsub.subscriptions.get
- pubsub.subscriptions.list
- pubsub.subscriptions.update
- pubsub.topics.get
- pubsub.topics.list
- pubsub.topics.update
- pubsub.topics.updateTag
- resourcemanager.folders.get
- resourcemanager.folders.list
- resourcemanager.organizations.get
- resourcemanager.projects.get
- resourcemanager.projects.list
- serviceusage.services.list
- storage.buckets.get
- storage.buckets.list
- storage.buckets.update
name: organizations/273839554717/roles/iris3
stage: GA
title: Iris3
+ gcloud iam roles update -q iris3 --organization 273839554717 --file roles.yaml
description: Iris auto tagging service for google cloud
etag: BwW9C620cr4=
includedPermissions:
- bigquery.datasets.get
- bigquery.datasets.update
- bigquery.tables.delete
- bigquery.tables.get
- bigquery.tables.list
- bigquery.tables.update
- bigtable.clusters.get
- bigtable.clusters.list
- bigtable.clusters.update
- bigtable.instances.get
- bigtable.instances.list
- bigtable.instances.update
- bigtable.tables.get
- bigtable.tables.list
- cloudsql.instances.get
- cloudsql.instances.list
- cloudsql.instances.update
- compute.disks.get
- compute.disks.list
- compute.disks.setLabels
- compute.instances.get
- compute.instances.list
- compute.instances.setLabels
- compute.instances.setTags
- compute.projects.get
- compute.regions.get
- compute.regions.list
- compute.snapshots.get
- compute.snapshots.list
- compute.snapshots.setLabels
- compute.zones.get
- compute.zones.list
- pubsub.subscriptions.get
- pubsub.subscriptions.list
- pubsub.subscriptions.update
- pubsub.topics.get
- pubsub.topics.list
- pubsub.topics.update
- pubsub.topics.updateTag
- resourcemanager.folders.get
- resourcemanager.folders.list
- resourcemanager.organizations.get
- resourcemanager.projects.get
- resourcemanager.projects.list
- serviceusage.services.list
- storage.buckets.get
- storage.buckets.list
- storage.buckets.update
name: organizations/273839554717/roles/iris3
stage: GA
title: Iris3
+ gcloud organizations add-iam-policy-binding 273839554717 --member serviceAccount:[email protected] --role organizations/273839554717/roles/iris3 --condition=None
Updated IAM policy for organization [273839554717].
bindings:
- members:
  - serviceAccount:[email protected]
  role: organizations/273839554717/roles/iris3
- members:
  - domain:webinit.net
  role: roles/billing.creator
- members:
  - domain:webinit.net
  role: roles/iam.organizationRoleAdmin
- members:
  - domain:webinit.net
  role: roles/iam.securityAdmin
- members:
  - domain:webinit.net
  role: roles/logging.configWriter
- members:
  - user:[email protected]
  role: roles/resourcemanager.organizationAdmin
- members:
  - domain:webinit.net
  role: roles/resourcemanager.projectCreator
etag: BwW9C63WzDs=
version: 1
+ gcloud pubsub topics describe iris_schedulelabeling_topic --project=iris3-306714
name: projects/iris3-306714/topics/iris_schedulelabeling_topic
+ gcloud pubsub subscriptions describe do_label --project=iris3-306714
ackDeadlineSeconds: 300
expirationPolicy:
  ttl: 2678400s
messageRetentionDuration: 604800s
name: projects/iris3-306714/subscriptions/do_label
pushConfig:
  pushEndpoint: https://iris3-dot-iris3-306714.uc.r.appspot.com/do_label?token=2a343f4c1b76512039fe763412756c4fbb30c
topic: projects/iris3-306714/topics/iris_schedulelabeling_topic
+ [[ '' == \t\r\u\e ]]
+ gcloud pubsub topics describe iris_logs_topic --project=iris3-306714
name: projects/iris3-306714/topics/iris_logs_topic
+ gcloud pubsub subscriptions describe label_one --project=iris3-306714
ackDeadlineSeconds: 300
expirationPolicy:
  ttl: 2678400s
messageRetentionDuration: 604800s
name: projects/iris3-306714/subscriptions/label_one
pushConfig:
  pushEndpoint: https://iris3-dot-iris3-306714.uc.r.appspot.com/label_one?token=2a343f4c1b76512039fe763412756c4fbb30c
topic: projects/iris3-306714/topics/iris_logs_topic
+ log_filter=("")
+ export PYTHONPATH=.
+ PYTHONPATH=.
++ python3 ./util/print_included_projects.py
Traceback (most recent call last):
  File "./util/print_included_projects.py", line 1, in <module>
    from util.config_utils import included_projects
  File "/home/andreas_berger/iris3/util/config_utils.py", line 33, in <module>
    def __load_config() -> typing.Dict:
  File "/usr/lib/python3.7/functools.py", line 477, in lru_cache
    raise TypeError('Expected maxsize to be an integer or None')
TypeError: Expected maxsize to be an integer or None
+ included_projects_line=

Did I miss anything or am I using the wrong python version?

Log the fact that a resource was successfully labeled, after the fact

Today, filter logging on "will label_one" to see all labeling attempts with details on the object.
Some GCP resource types, like Topics, and Subscriptions for example, get a log line after they are successfully labeled
For other resource types, like GCE Instances, batch labeling is used. This is asynchronous, and may label up to 1000 objects at a time, and so the resource are not logged after labeling. If there is an error, that is indicated, though without stating the resource
Feature requests
-- log the specific resource that was successfully labeled
-- log the specific resource for which labeling failed
However, note that Iris may be labeling a lot of objects and we do not want to clog up the logs.

Iris Labeller fails when it is unable to find a ephemeral GCP resource

After deploying the labeller on GCP we ran into issue multiple times where the labeller would crash when it cant find a temporary or a ephemeral resource that no longer exists
"nit__.py", line 494, in api_request\n raise exceptions.from_http_response(response)\ngoogle.api_core.exceptions.NotFound: 404 GET https://bigquery.googleapis.com/bigquery/v2/projects/analytics-test-345723/datasets/indexer_testnet_gcp_apps/tables/_sdc_current_token_pending_claims_staging?prettyPrint=false: Not found: Table analytics-test-345723:indexer_testnet_gcp_apps._sdc_current_token_pending_claims_staging","
Is there a way to avoid labelling resources temp in nature ?

How to run ./deploy.sh as in getting python version error

Hi,
When running ./deploy.sh in google cloud shell. I'm getting below error
"This script requires Python 3.8 or higher!
You are using Python 3.7."

How to update cloud shell to 3.7 or I have to run this script somewhere else?

Labeling GKE Node causes re-creation of Node

Fix: Do not label instances or disks that are part of GKE.

Known errors in Google API Client Library

We see lots of errors on the SSL layer. These are not security errors but connection failures. See "broken pipe"[1] and “socket.timeout”[2] on the google-api-python-client (Google API Client for Python) issue tracker. This results from the implementation of httplib2.

Issue [3] tracks the discussion about a fix, but Google does not plan any fix to the (Google API Client for Python) , as it would require a significant rewrite.

Instead, Google recommends using the Cloud Client Libraries. Unfortunately, these lack batch capability [4] and are radically slower.

[1] googleapis/google-api-python-client#218
[2] https://github.com/googleapis/google-api-python-client/search?q=socket+timeout&type=issues
[3] googleapis/google-api-python-client#1118
[4] https://stackoverflow.com/questions/75712871/

deploy.sh should read pubsub token from config.yaml with yaml-parsing

As-is, if you comment out the old pubsub-token line and add a new one, the first one will be taken by a simple grep.

No resources are being labeled

I have a project that I've deployed to and the only change I've made from the default is to update the config.yaml to specify a single other specific project to get labeled for testing purposes that just has two buckets.

There are no errors when running deploy.sh -c
Neither bucket in my target project receives any labels.

I'm manually going into the Cloud Scheduler and selecting the Force a job run option
I see logs in the app engine showing:

textPayload: "INFO [] [Trace: bc6a996b4...5461;o=1] schedule() sent messages to label 1 projects, 1 messages"
textPayload: "INFO [] [Trace: bc6a996b4...5461;o=1] Time schedule(): 1162 ms"

A message also ends up in the iris_deadletter subscription

Any suggestions or method of debugging to help troubleshoot this better?

Feature to label all resources on Iris deployment

Add a feature that labels all resources--but only once.

Today you can do that with Cloud Scheduler, but after the first mass-labeling, we usually do not want to do it again because then we are spending money on reading the same labels daily.

This could be done by using curl on a new endpoint /label_all_resources at the end of the deployment script.

This endpoint would just do the same as /schedule but on all resource types.

Socket error in Google Discovery API

An occasional socket failure in getting info on a project
see known issue in Google Client here
Recommended: switch to this library for Discovery https://github.com/googleapis/google-cloud-python

Scheduled labeling does not work in projects in folders unless one specifies projects.

There is a failure to label where all these are true:

The resource is in a project is in a folder (i.e, the bug does not occur if the project is directly in the org)
and
The labeling is scheduled (by Cloud Scheduler) (i.e., the bug does not occur with labeling that occurs on creation of resources).
and
the projects config key (in config.yaml is empty projects: [] (i.e., the bug does not occur if you have projects: ['myproject']

Looking at the code for listing projects, we see that if projects are not explicitly listed, all_projects() finds the projects with list_projects(parent=org_name), which lists only the "projects that are direct children of the specified folder or organization resource".

(As to why this issue was not earlier caught: It is probably because most labeling occurs on-creation, not with Cloud Scheduler. So, this issue would only be seen for projects in a folder, and for existing resources labeled for the first time after Iris has been launched, or for Cloud SQL, or for disks whose attachment state has changed.)

The solution would be to use a combination of list_folders and the above-mentioned list_projects to recursively walk the organization. It might make sense to parallelize the tree-walking algo. It might be best to return a Generator and have get_enabled_projects return a generator to better support cases with a huge number of projetcs, though if so, sorting will need to be removed -- and if we really have such a huge number of projects, we probably have bigger headaches.