Comments (9)
Do you think the check should happen at the add-dependency level (ie as soon as you create a cycle when adding a dependency, it fails to save it) or when saving the whole DAG? (or both)
from dagobah.
I did some thinking on this last week and realized that there are a few complications here.
tl;dr: We need to implement a better consistency model before we can disallow saving bad DAGs. I'll go create an issue for that.
First, some facts:
- The state of a particular DAG can be stored in three places at any one time: a user's browser, the running
dagobahd
, and the database. It should be our goal to keep these as consistent as possible. - Currently, DAG updates are done through small atomic updates, e.g. "add a dependency between nodes 1 and 2." There is not currently a way to send a complete DAG through the API and overwrite whatever is in the database.
Atomic updates will not be sufficient if we are going to disallow saving bad DAGs. This is what the interaction will look like:
- User loads the DAG in their browser. All three states are consistent.
- User adds a dependency that does not invalidate the DAG. This can go through the atomic update path. All three states are consistent.
- User adds a dependency that invalidates the DAG. This tries to go through the atomic update path but is rejected by the server. The bad dependency stays in the browser DAG so the user can complete whatever restructuring they are trying to do. The browser now diverges from the server and database. The browser marks its DAG as dirty (divergent).
- Updates while the browser DAG is dirty cannot use the atomic updates (since state diverges) and instead must POST the entire browser DAG to the server. This is a very destructive operation. Any update any other user has made in this time will be blown away.
Dagobah's consistency model is already bad, but this was mitigated partially by the nature of atomic updating. I'm not comfortable with full-DAG POSTs unless we have a better consistency model. Once we have a consistency model, this method should be fine.
from dagobah.
In step 3, what is wrong with removing the dependency when the server rejects it? Or not updating the front end model until the server has confirmed it's validity?
from dagobah.
You don't want to remove the dependency from the front-end because the user purposefully put it there. She's probably in the process of making some more changes to make the DAG valid again. Refusing to save the incremental changes along the way could be very annoying.
from dagobah.
I mean, is it that much of a problem to ask them to fix the cycle before allowing them to add that connection? What about turning that arrow red (or something) in the front end?
from dagobah.
That would be possible, though we would then need to maintain some sort of sync queue in the frontend with changes that have been made in the browser but not successfully saved on the server. We would then need to try to sync everything in the queue whenever something else is changed. This could probably be made easier by having some DAG validation logic in the frontend as well.
This does seem like it might be simpler. I think that approach is safer and could be merged before the consistency model, though I still want to add that in at some point.
from dagobah.
I just don't see what's so bad about disallowing adding a dependency if there's a cycle, how often would this case happen? And wouldn't immediate feedback be useful?
from dagobah.
I suppose it won't be all that annoying, since the only action that could cause an inconsistency is adding an edge. It would be more annoying if a node addition/deletion got backed out, but that won't happen. Yeah, we can do it that way. So basically:
- User adds an edge in the browser, tries to tell the API to add it.
- API freaks out and returns a 400.
- Browser tells the user it would invalidate the DAG and deletes the edge.
from dagobah.
Yeah, that was my thought process. Although #88 is a nice-to-have for potential simultaneous editing.
The other issue is if we do put in a PR for this, if users upgrade, it wont detect any existing cyclic DAGs, but I guess that would mean they're not in use anyway because they won't run with cycles.
from dagobah.
Related Issues (20)
- error when start dagobah
- Installation issue, missing gpost HOT 1
- dagobahd not working on Mac OS X HOT 1
- get_status method for job HOT 2
- Changing code to use py-dag as a package instead of the internal file HOT 2
- RuntimeError: can't start new thread
- How to run and debug the whole project? HOT 1
- Is it distributed?
- Timezone cron / scheduler
- Updating config file, changes not reflected HOT 6
- Run Dagobah from folder HOT 1
- Address jump HOT 6
- [CLOSED] reformat code
- [CLOSED] Reformat code
- [CLOSED] Reformat code
- [CLOSED] update setup requirements
- [CLOSED] add amazon ses service in emails
- [CLOSED] Close SSH connection for remote tasks
- [CLOSED] UnicodeEncodeError while sending basic email
- Fail to install dagobah HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dagobah.