Support branching sections about livebook HOT 12 CLOSED

Branches may be a better metaphor for this too!

from livebook.

I have another suggestion then: what about just allowing users to put cells straight to the document right below the h1 on *.livemd? Those then could be used as the shared context and all sections could be independent processes that would only copy the evaluation context from there.

That's actually the model we wanted to go with initially, but in the end we concluded that a more intuitive default behavior is for the sections to be linear (so they can stay small and focused). Also, one important detail is that copying evaluation context (or more specifically bindings) could lead to very inefficient memory usage. For example imagine someone would load 1GB dataset in a shared section, then every other section has the penalty of copying all of that data, so you may even run our of memory (and from the user perspective there's nothing wrong with adding more sections). So we think it's more reasonable to provide linear evaluation by default and be able to create an aside section (branching out), so that only this section copies the evaluation context.

FWIW we did consider multiple approaches, the one with nested headings as well ^^

from livebook.

A couple more ideas:

1. Maybe we should call them "sticky sections"
2. We should also transfer the process dictionary from the sticky section to the new sessions. This will be required for things like the new Nx.default_backend that stores configuration in the process dictionary. Also for things like Logger and so on

from livebook.

Sticky sounds nice, I also thought about "leading section" or "template section". I think we should allow just one, as otherwise sections would have to link to the sticky section they use and that will likely become messy really quick. As far as I understand we would put installs, imports and config there, so one should do the trick.

You mentioned bindings in the original comment, but as we recently discussed it's probably better to ignore bindings, so that users don't load a huge file and end up copying the data to all further sections.

from livebook.

Just to clarify the issue. We decided to make the sections evaluation linear by default to provide intuitive flow. The plan is to add aside sections - such section will copy evaluation context from the last cell of the previous normal section and will be detached from the main evaluation flow (i.e. it will be evaluated in a separate process and will have no impact on subsequent sections). This can be thought of as starting a git branch ^^ The use case is that there may be a long running computation in the main flow (like training a NN), so it would be nice to be able to create an aside section and play around with the data at the same time.

from livebook.

+1 for "Aside Sections" . I've been using Livebook for a week or so as my primary way to do exploratory Elixir coding and it's quite frustrating when you just want to test a snippet in a separate cell and it leads to a complete re-evaluation of lots of cells. BTW: thanks all for this amazing tool!

from livebook.

What about making the sections a nested tree structure? Where we could then mark the parent nodes as async making them a parallel execution line.

from livebook.

Trees are likely more nesting than we need. We want to try something simpler.

from livebook.

My view as a naiive user: the absolute simplest is have all sections be independent. That was actually how I half expected it to work. But allowing a section to be marked independent isn't that hard to understand either so that would be great too. Anyway -- really love this project. I'm a long-time user of ipython/jupyter notebooks and I have to say that the fact that you can connect livebook to a running node, or situate your node in a mix project, gives livebook a dimension that jupyter does not have. Cheers.

from livebook.

I have another suggestion then: what about just allowing users to put cells straight to the document right below the h1 on *.livemd? Those then could be used as the shared context and all sections could be independent processes that would only copy the evaluation context from there.

from livebook.

So # defines the shared evaluation, ## defines new sections (that gets executed in parallel, on it's own process) and we could introduce ### as subsection (that get executed sequentially, inside the section process), for organization purposes.

from livebook.

The drawback I see on it though is that then we would be forcing some kind of document structure and tying it to it's actual execution. Maybe we would want to decouple the document and its execution strategy somehow, like executing sequentially as we do now by default (as it is the most intuitive way IMO), and defining the execution steps on somekind of markdown configuration headers, something like this:

[
  "Section 1": [:"Section 1.1", :"Section 1.2"]
]
---
# File

## Section 1

``elixir
a = 1
``

## Section 1.1

``elixir
b = a * 2
``

## Section 1.2

``elixir
b = a * 3
``

from livebook.