Comments (4)
Task naming is available since Python 3.8. Kopf still supports Python 3.7. I’m eagerly waiting for the end-of-life for 3.7 as a milestone, which happens somewhen this summer 2023 if I remember correctly.
from kopf.
Hi. You are welcome!
keeping reference to all kopf-spawned tasks
Exactly this is a problem — it is difficult to achieve. First, Kopf spawns tasks for its own needs somewhere deep in the call stack, some of them are "fire-and-forget" style (daemons, timers). Second, regular handlers can spawn new tasks — this was the case before daemons/timers arrived, so users had to spawn their own tasks explicitly in the resume/creation handlers.
All in all, at the end, when the operator exits, it must ensure the graceful termination of all spawned tasks/daemons/timers; here, "ensure" means 2 steps: cancellation + awaiting (actually, 2 cancellation attempts: soft & hard, each with waiting), plus also some logging on the problems if some tasks do not end and are "abandoned".
Since it is difficult, the task tracking is instead delegated to the event loop, which already has this functionality.
So, Kopf considers all tasks spawned after its startup as its own — for simplicity.
For the advanced cases if someone wants to run it all in a shared event loop or orchestrate the tasks themselves, there is kopf.spawn_tasks()
as the official Kopf API. You can replace the kopf.operator()
call with the call to kopf.spawn_tasks()
and then orchestrate them somehow the same way as in kopf.run_tasks()
. The logic is simple: wait for the spawned tasks infinitely; once any single one of them exits, terminate others as gracefully as possible — nothing more. What "other tasks" mean in a shared loop — it is for you to decide.
set some custom attribute…
The official asyncio API does not provide this capability, so you cannot be sure that setting arbitrary attributes on tasks is possible. Tasks can be slotted or C-level objects with no __dict__
. Or they can be tasks from other event loop implementations — e.g. uvloop — which also cannot accept arbitrary attributes. So, such tracking must be done in a centralized place, such as an operator-scoped weakref.WeakSet
or alike.
it seemed obvious to run some web app with embedded operator on the same event loop
That is a somewhat debatable and rather controversial point of view. I personally would keep separate components of an application in separate threads, each having its event loop. Cross-thread communication is rather safe & fast and does not add any extra overhead (as inter-process communication does, for example). Since both apps are i/o-bound, GIL is also not a problem. But I prefer not to argue about how other people do this in their apps — because "whatever works, works".
At the end of the day, such task-tracking capabilities can be considered and added, but that would require a huge refactoring and a lot of effort with little or no benefit for the majority of use cases (compared to possible workarounds: multi-threading or using kopf.spawn_tasks()
).
Sadly, I am not able to dedicate enough time to this project nowadays, so it remains in its current state (only some major bugs are fixed).
from kopf.
Thanks for reply!
So, Kopf considers all tasks spawned after its startup as its own — for simplicity.
I see, that is understandable.
I have come up with another idea - instead of keep references to all tasks in one place (as you mentioned that would require heavy refactoring), name them in specific way, i.e. with prefix kopf.
. That would obviously work only for kopf tasks and might be extended to tasks created via kopf.spawn_tasks
or similar. If user of kopf spawns task using different mechanism - no way to manage it. I know, sounds brittle but I wanted to share the idea nonetheless.
For the advanced cases if someone wants to run it all in a shared event loop or orchestrate the tasks themselves, there is kopf.spawn_tasks() as the official Kopf API. You can replace the kopf.operator() call with the call to kopf.spawn_tasks() and then orchestrate them somehow the same way as in kopf.run_tasks(). The logic is simple: wait for the spawned tasks infinitely; once any single one of them exits, terminate others as gracefully as possible — nothing more. What "other tasks" mean in a shared loop — it is for you to decide.
That sounds very interesting, I'll need to look into this. Maybe it could be documented under section I want to embed kopf but still use shared event loop
?
That is a somewhat debatable and rather controversial point of view. I personally would keep separate components of an application in separate threads, each having its event loop. Cross-thread communication is rather safe & fast and does not add any extra overhead (as inter-process communication does, for example). Since both apps are i/o-bound, GIL is also not a problem. But I prefer not to argue about how other people do this in their apps — because "whatever works, works".
That's a matter of preference. For me one thing was obvious, for someone else - something entirely different. Different use-cases means different requirements, etc. Still, would be nice have ability to run kopf in shared event loop. Will try that and see how it works - maybe even craft a PR with documentation, who knows.
Thanks for pointers!
from kopf.
For anyone seeing this thread, there are some docs for this now: https://kopf.readthedocs.io/en/stable/embedding/
from kopf.
Related Issues (20)
- Flakey behavior of on.create handler not reacting to CRO creation event. HOT 1
- Automating Kopf Operator Code Updates: Seeking a Convenient Solution to Avoid Tedious Run Commands HOT 1
- Controlling Kopf Operator Behavior to Prevent Unwanted Object Creation HOT 8
- Using named import in kopf
- Handle large resource spec being annotated to `last-handled-configuration` HOT 3
- Namespace deletion is stuck when using namespace selector startup mode HOT 1
- Infinite watch-streams stopping immediately with no obvious reasons why. The watched resources do then spin up.
- Liveness probe stops working HOT 2
- Kubernetes client not configured in startup handler HOT 3
- Admission Controller Path
- `FieldSpec` and `resolve` does not support list access
- Attach to container start event
- Behaviour of timer with finalizer
- Finalisers race condition
- Finalizers from other controllers conflicting with kopf finalizer
- Kopf does not restart after 429 too many requests error HOT 1
- How to avoid Errors accessing metrics API and CRDs HOT 1
- add cwd to sys.path HOT 4
- Allow kopf.adopt to Communicate Resource Readiness to ArgoCD HOT 4
- Object stuck in loop with inconsistent status updates and handler failure HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kopf.