Design document is not publicly accessible about marquez HOT 2 CLOSED

@teabot : It's great to hear that you're interested in learning more about Marquez! The design doc linked in the README was originally authored internally at WeWork, then shared publicly. If you shoot me an email at [email protected] I'd be happy to get you access to the doc. And given that the project is in the early development phase, it'd be great to get your feedback as well. Note that there are gaps in the design doc and sections that need to be revisited, but updates to the doc will happen more regularly.

We are currently working on building metadata collection as a core requirement into all jobs (streaming or batch) at WeWork. Are immediate focus is to integrate Marquez with Airflow in order to capture the job (=task) runtime arguments, input/output datasets and state (RUNNING, COMPLETED, etc). This will help define both the Job API and Dataset API.

We have an internal roadmap and milestones for Marquez at WeWork, but our goal is to be transparent about the project and it's direction. I just opened issue #74

from marquez.

@teabot : We have made our design public! see https://marquezproject.github.io/marquez

from marquez.