Comments (4)
I wrote a small custom step decorator for myself to tag each task with its parent tasks. I had to do this because I couldn't find any mechanism built into the Metaflow API to track task lineage (do correct me if I'm wrong but it seems easy to check which Step a task came from with .parent
, but it doesn't seem possible to check which task spawned the current task?)
def _track_dag(step_func: Callable) -> Callable:
"""Decorator on steps to update self.parent_tasks and self.curr_task so as to enable
visualization of the DAG during analysis
"""
@wraps(step_func)
def wrapper(*args: Any, **kwargs: Any) -> None:
self_ = args[0]
curr_task: Optional[str] = self_.curr_task if hasattr(self_, "curr_task") else None
if "inputs" in signature(step_func).parameters:
inputs = args[1] if len(args) > 1 else kwargs["inputs"]
self_.parent_tasks = [i.curr_task for i in inputs]
else:
self_.parent_tasks = [curr_task] if curr_task else []
self_.curr_task = current.pathspec
return step_func(*args, **kwargs)
return wrapper
So my flows look like:
class MyFlow(FlowSpec):
@step
@_track_dag
def step(self):
pass
...
And when I'm looking at my run in a notebook, I have another helper that uses PyGraphViz to use the parent pointers and display the DAG, along with some custom coloring to show status of the tasks:
from IPython.display import Image
import pygraphviz as pgv
class MetaflowUtils:
@staticmethod
def show_graph(run):
G = pgv.AGraph(directed=True)
for s in list(run):
for t in list(s):
color = None
if not t.finished:
color = 'black'
elif t.successful:
color = 'green'
else:
color = 'red'
G.add_node(parse_pathspec(t.data.curr_task), color=color)
for s in list(run):
for t in list(s):
for p in t.data.parent_tasks:
G.add_edge(parse_pathspec(p), parse_pathspec(t.data.curr_task))
return Image(G.draw(format='png', prog='dot'))
MetaflowUtils.show_graph(run)
Is this perhaps close to what you're looking for @SteNicholas ?
from metaflow.
There's some related built-in functionality in Metaflow, which requires graphviz:
$ python playlist.py output-dot | dot -Tpng -o playlist.png
The format can be changed, e.g. to SVG, if you're having issues with PNG dependencies.
from metaflow.
@SteNicholas Can you elaborate on your ask? You should be able to stand-up custom dashboards using the metaflow.client
in a Jupyter notebook easily. A very basic example is in the docs. The client gives you access to the flow state and all the data artifacts. In my experience, dashboards and visualizations depend on a particular business context and visualizing via a Notebook is a very good first step.
from metaflow.
@jaychia Yes, I will close this issue.
from metaflow.
Related Issues (20)
- Question: Access hash of a card while being populated HOT 2
- Conda environment being treated as disabled, and not appending environment to PATH.
- Metaflow crashes on AWS Batch if folder called `metaflow` is present in the working directory HOT 5
- Cardview on WSL error HOT 2
- S3 access denied even if I have full access to S3
- Certain flows failing on Argo Workflows =>3.5.0 HOT 1
- Metaflow job completion or exit handlers?
- run.finished not set when using AWS Step Functions and there's an error
- setting METAFLOW_OTEL_ENDPOINT when running in ECS fargate, not Kubernetes HOT 1
- add __repr__ methods to Parameter
- create contributing guide
- "Service token file does not exist" error when deploying flow to Argo from CI HOT 1
- argo-workflows create --only-json doesn't export the cron workflow configuration
- Using `tags` as a Parameter name breaks flow. HOT 1
- Add option to batch decorator to increase ephemeralStorage on Fargate
- `--package-suffixes` omits dotfiles HOT 1
- Is it possible to run metaflow steps in custom docker containers on local?
- Opentelemetry configuration not carrying over to Batch
- Add a priority class option for the kubernetes flow decorator HOT 1
- Reduce the number of reserved parameter names
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from metaflow.