Comments (3)
nolar
commented on July 22, 2024
2
Kopf is designed to work "out of the box" for the majority of cases, i.e. of typically small resources — without any special configuration or external dependencies.
For this, it uses annotations to store its per-object state. However, if Kubernetes prohibits storing big values, Kopf can do nothing here. I am open to suggestions though.
What an operator developer can do, is to configure their own state storage (2 storages of 2 types, actually), as documented here:
- https://kopf.readthedocs.io/en/stable/configuration/#handling-progress
- https://kopf.readthedocs.io/en/stable/configuration/#change-detection
It can be e.g. a MySQL/Postgres/any other AWS RDS database, or even a filesystem (if the volume is persisted and shared across multiple instanced of Kopf or on restarts). All of this requires some extra configuration to start an operator and therefore is not the default "out of the box" setup.
There is also the Kopf-provided storage to store the state in the status field — but since a few years ago, it requires that the CRD schema explicitly declares those fields as allowed to store any arbitrary value with x-kubernetes-preserve-unknown-fields: true. So it is not the default either (but it was back then). See the blue note section in the link above.
from kopf.
Ramblurr
commented on July 22, 2024
wow this is unfortunate, literally the first lines of code I write with kopf and I hit this error:
import kopf
import logging
@kopf.on.create("secret")
def create_fn(body, **kwargs):
logging.info(f"A handler is called with body")
kopf run -n database kopftest.py
[2024-02-16 09:26:51,554] kopf._core.engines.a [INFO ] Initial authentication has been initiated.
[2024-02-16 09:26:51,556] kopf.activities.auth [INFO ] Activity 'login_with_kubeconfig' succeeded.
[2024-02-16 09:26:51,557] kopf._core.engines.a [INFO ] Initial authentication has finished.
[2024-02-16 09:26:51,735] root [INFO ] A handler is called with body
[2024-02-16 09:26:51,735] kopf.objects [INFO ] [database/sh.helm.release.v1.crunchy-postgres-operator.v1] Handler 'create_fn' succeeded.
[2024-02-16 09:26:51,736] kopf.objects [INFO ] [database/sh.helm.release.v1.crunchy-postgres-operator.v1] Creation is processed: 1 succeeded; 0 failed.
[2024-02-16 09:26:51,780] kopf.objects [ERROR ] [database/sh.helm.release.v1.crunchy-postgres-operator.v1] Throttling for 1 seconds due to an unexpected error: APIClientError('Secret "sh.helm.release.v1.crunchy-postgres-operator.v1" is invalid: metadata.annotations: Too long: must have at most 262144 bytes', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'Secret "sh.helm.release.v1.crunchy-postgres-operator.v1" is invalid: metadata.annotations: Too long: must have at most 262144 bytes', 'reason': 'Invalid', 'details': {'name': 'sh.helm.release.v1.crunchy-postgres-operator.v1', 'kind': 'Secret', 'causes': [{'reason': 'FieldValueTooLong', 'message': 'Too long: must have at most 262144 bytes', 'field': 'metadata.annotations'}, {'reason': 'FieldValueTooLong', 'message': 'Too long: must have at most 262144 bytes', 'field': 'metadata.annotations'}, {'reason': 'FieldValueTooLong', 'message': 'Too long: must have at most 262144 bytes', 'field': 'metadata.annotations'}, {'reason': 'FieldValueTooLong', 'message': 'Too long: must have at most 262144 bytes', 'field': 'metadata.annotations'}]}, 'code': 422})
Traceback (most recent call last):
File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_cogs/clients/errors.py", line 148, in check_response
response.raise_for_status()
File "/nix/store/xrgwijximlapr1rkwwwnirf510h3kvc3-python3.11-aiohttp-3.9.1/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1059, in raise_for_status
raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 422, message='Unprocessable Entity', url=URL('https://10.9.9.120:6443/api/v1/namespaces/database/secrets/sh.helm.release.v1.crunchy-postgres-operator.v1')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_core/actions/throttlers.py", line 44, in throttled
yield should_run
File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_core/reactor/processing.py", line 130, in process_resource_event
applied = await application.apply(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_core/actions/application.py", line 60, in apply
await patch_and_check(
File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_core/actions/application.py", line 131, in patch_and_check
resulting_body = await patching.patch_obj(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_cogs/clients/patching.py", line 47, in patch_obj
patched_body = await api.patch(
^^^^^^^^^^^^^^^^
File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_cogs/clients/api.py", line 155, in patch
response = await request(
^^^^^^^^^^^^^^
File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_cogs/clients/auth.py", line 45, in wrapper
return await fn(*args, **kwargs, context=context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_cogs/clients/api.py", line 85, in request
await errors.check_response(response) # but do not parse it!
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_cogs/clients/errors.py", line 150, in check_response
raise cls(payload, status=response.status) from e
kopf._cogs.clients.errors.APIClientError: ('Secret "sh.helm.release.v1.crunchy-postgres-operator.v1" is invalid: metadata.annotations: Too long: must have at most 262144 bytes', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'Secret "sh.helm.release.v1.crunchy-postgres-operator.v1" is invalid: metadata.annotations: Too long: must have at most 262144 bytes', 'reason': 'Invalid', 'details': {'name': 'sh.helm.release.v1.crunchy-postgres-operator.v1', 'kind': 'Secret', 'causes': [{'reason': 'FieldValueTooLong', 'message': 'Too long: must have at most 262144 bytes', 'field': 'metadata.annotations'}, {'reason': 'FieldValueTooLong', 'message': 'Too long: must have at most 262144 bytes', 'field': 'metadata.annotations'}, {'reason': 'FieldValueTooLong', 'message': 'Too long: must have at most 262144 bytes', 'field': 'metadata.annotations'}, {'reason': 'FieldValueTooLong', 'message': 'Too long: must have at most 262144 bytes', 'field': 'metadata.annotations'}]}, 'code': 422})
[2024-02-16 09:26:52,781] kopf.objects [INFO ] [database/sh.helm.release.v1.crunchy-postgres-operator.v1] Throttling is over. Switching back to normal operations.
^C[2024-02-16 09:26:52,830] kopf._core.reactor.r [INFO ] Signal SIGINT is received. Operator is stopping.
from kopf.
baracoder
commented on July 22, 2024
I have a problem https://github.com/telekom-mms/trivy-dojo-report-operator which uses kopf and works with VulnerabilityReport resources that can get very big. But the reports have a field report.updateTimestamp which would be sufficient to check for changes. Is it possible to configure, which fields get persisted for change detection?
Edit: Would it work if a hash of the resources is saved and used for comparison?
from kopf.
Related Issues (20)
- Flakey behavior of on.create handler not reacting to CRO creation event. HOT 1
- Automating Kopf Operator Code Updates: Seeking a Convenient Solution to Avoid Tedious Run Commands HOT 1
- Controlling Kopf Operator Behavior to Prevent Unwanted Object Creation HOT 8
- Using named import in kopf
- Namespace deletion is stuck when using namespace selector startup mode HOT 1
- Infinite watch-streams stopping immediately with no obvious reasons why. The watched resources do then spin up.
- Liveness probe stops working HOT 2
- Kubernetes client not configured in startup handler HOT 3
- Admission Controller Path
- `FieldSpec` and `resolve` does not support list access
- Attach to container start event
- Behaviour of timer with finalizer
- Finalisers race condition
- Finalizers from other controllers conflicting with kopf finalizer
- Kopf does not restart after 429 too many requests error HOT 1
- How to avoid Errors accessing metrics API and CRDs HOT 1
- add cwd to sys.path HOT 4
- Allow kopf.adopt to Communicate Resource Readiness to ArgoCD HOT 4
- Object stuck in loop with inconsistent status updates and handler failure HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kopf.