Places to look at and extend:

• Implementation for the local filesystem: https://github.com/rstudio/learnr/blob/51b4fcb26f128bc7f8617bd1b04b097ba8a0034c/R/storage.R#L342
• Need to extend learnr to accept stepper "submissions": https://github.com/rstudio/learnr/blob/51b4fcb26f128bc7f8617bd1b04b097ba8a0034c/R/storage.R#L120

We'd like to be able to go back to the original state of things. A Reset button to go back to original state in explorable code

It would be ideal for instructor to be able to write a chunk that's just for the code:

Setup code:

```{python nba_stepper_setup, include=FALSE}
import pandas as pd
import numpy as np
nba = pd.read_csv(r.nba_data_path)
column_names = {
  'Date': 'date',
  'Start (ET)': 'start',
  'Unamed: 2': 'box',
  'Visitor/Neutral': 'away_team',
  'PTS': 'away_points',
  'Home/Neutral': 'home_team',
  'PTS.1': 'home_points',
  'Unamed: 7': 'n_ot'
}
```

Code to step through:

```{python nba_code, include=FALSE, eval=FALSE}
nba = (nba.rename(columns=column_names)
  .dropna(thresh=4)
  [['date', 'away_team', 'away_points', 'home_team', 'home_points']]
  .assign(date=lambda x: pd.to_datetime(x['date'], format='%a, %b %d, %Y'))
  .set_index('date', append=True)
  .rename_axis(["game_id", "date"])
  .sort_index()
)
```

Stepper:

```{r nba_stepper, echo=FALSE}
stepper(
  setup_label = "nba_stepper_setup",
  code_label = "nba_code",
  explanations = list(
    "The column names of `nba` are quite messy so, we first rename column names with `rename` using our earlier `column_names` dict.",
    "Then, we drop any rows that contains more than 4 NAs."
  )
)
```

TODOs:

• Setup/Code: the setup code and the stepping code
• Explanations: thinking of list structure corresponding to relevant lines
• Stepper: a Shiny stepper widget that renders the widget

Ideas:

• https://docs.jsplumbtoolkit.com/toolkit/current/articles/connectors.html
• https://github.com/musclesoft/jquery-connections

Read: be able to plug in stepper widget that's based on particular knitr chunks.

Explaining method chaining:

tumble_after(
    broke(
        fell_down(
            fetch(went_up(jack_jill, "hill"), "water"),
            jack),
        "crown"),
    "jill"
)

In R:

jack_jill %>%
    went_up("hill") %>%
    fetch("water") %>%
    fell_down("jack") %>%
    broke("crown") %>%
    tumble_after("jill")

In Python:

# assume you owned JackAndJill object:
jack_jill = JackAndJill()
(jack_jill.went_up('hill')
    .fetch('water')
    .fell_down('jack')
    .broke('crown')
    .tumble_after('jill')
)

# but if you don't like in the case of pandas, need to use `pipe`
jack_jill = pd.DataFrame()
(jack_jill.pipe(went_up, 'hill')
    .pipe(fetch, 'water')
    .pipe(fell_down, 'jack')
    .pipe(broke, 'crown')
    .pipe(tumble_after, 'jill')
)

src: https://tomaugspurger.github.io/method-chaining

munging:

arrays = [np.array(['bar', 'bar', 'baz', 'baz', 'foo', 'foo', 'qux', 'qux']),
         np.array(['one', 'two', 'one', 'two', 'one', 'two', 'one', 'two'])]

s = pd.Series(np.random.randn(8), index=arrays)
s

bar  one   -0.861849
     two   -2.104569
baz  one   -0.494929
     two    1.071804
foo  one    0.721555
     two   -0.706771
qux  one   -1.039575
     two    0.271860
dtype: float64

df = pd.DataFrame(np.random.randn(8, 4), index=arrays)
df
                0         1         2         3
bar one -0.424972  0.567020  0.276232 -1.087401
    two -0.673690  0.113648 -1.478427  0.524988
baz one  0.404705  0.577046 -1.715002 -1.039268
    two -0.370647 -1.157892 -1.344312  0.844885
foo one  1.075770 -0.109050  1.643563 -1.469388
    two  0.357021 -0.674600 -1.776904 -0.968914
qux one -1.294524  0.413738  0.276662 -0.472035
    two -0.013960 -0.362543 -0.006154 -0.923061

df = pd.DataFrame(np.random.randn(3, 8), index=['A', 'B', 'C'], columns=index)
df

first        bar                 baz                 foo                 qux          
second       one       two       one       two       one       two       one       two
A       0.895717  0.805244 -1.206412  2.565646  1.431256  1.340309 -1.170299 -0.226169
B       0.410835  0.813850  0.132003 -0.827317 -0.076467 -1.187678  1.130127 -1.436737
C      -1.413681  1.607920  1.024180  0.569605  0.875906 -2.211372  0.974466 -2.006747

src: https://pandas.pydata.org/pandas-docs/stable/user_guide/advanced.html#creating-a-multiindex-hierarchical-index-object

split-apply-combine:

df = pd.DataFrame({'CID':[2,2,3],
                   'FE':[5,5,6],
                   'FID':[1,7,9]})

print (df)
   CID  FE  FID
0    2   5    1
1    2   5    7
2    3   6    9

df = df.groupby(by=['CID','FE'])['FID']
       .count()
       .unstack()
       .reset_index()
       .rename_axis(None, axis=1)

print (df)    
   CID    5    6
0    2  2.0  NaN
1    3  NaN  1.0

src: https://stackoverflow.com/questions/44023770/pandas-getting-rid-of-the-multiindex

As before, for the new stepper cadence, we need to render the strings that may have markdown formatted text via stepper_text

Places to look at:

• encode/decode: https://github.com/rundel/learnrhash/blob/master/R/encode.R
• extracting from question/submission: https://github.com/rundel/learnrhash/blob/master/R/extract.R (we would a extract_stepper for extracting data from hash)

Ideas:

• https://docs.python.org/3.6/library/bdb.html
• https://docs.python.org/3.6/library/pdb.html

Make tidylog like summary but enhanced with annotations and better text/info.

TODO:

• capture tidylog verb summary
• annotate [row x col] change text
• generate the data tidylog(data) summary by default (if indeed the line is the data line)
• helper function to color text with data change semantics
• annotate column change text
• annotate standalone row or column number change

The idea behind this is that instructors need to highlight certain parts of the text explanation, the code snippet, or the output. For example, if there are X, Y, Z distinct terms to highlight between code (action) <-> output (effect) <-> explanation. It would be useful to accentuate the decoration for X, while muting the decorations Y, and Z. This would help highlight the parts that matter when user activates those related things.

For e.g. could we just get the last_value somehow instead of doing extra work of parsing text, eval'ing getting last line etc?

Places to look:

• https://github.com/rstudio/reticulate/blob/a8701771fe759161c76a8e4a81d4ccc506269de1/R/utils.R#L130

Currently, the knitr custom hook for pretty printing Python dataframes is pretty disorganized and fragile. I propose the following:

• resolve #19 and #7
• use the new solution to detect type of object for last expression
• add tests

Some things I'd like to handle/include in pprints:

• MultiIndex (useful for (un)stack and pivot/pivot_table agg operations); multiple row names

e.g. this block of code eventually sets index which makes the df Multindex with other columns raised up (as if grouped).
.rename_axis(["game_id", "date"] introduces two named rownames

 nba = (nba.rename(columns=column_names)
    .dropna(thresh=4)
    [['date', 'away_team', 'away_points', 'home_team', 'home_points']]
    .assign(date=lambda x: pd.to_datetime(x['date'], format='%a, %b %d, %Y'))
    .set_index('date', append=True)
    .rename_axis([\"game_id\", \"date\"])
    .sort_index()
  )

• grouped dfs
• meta info like # rows etc
• retain NaN for empty values
• convert data properly
  • NAs
  • ints / doubles / floats / strings
  • pd.datetime64

See if you can make headway with hooking into python engine for plotly output

It feels odd to have to write the explanation text in strings especially with the lack of distinguishing code text bits from the narrative bits.

We could instead write knitr chunks that uses the block2 and contains information for callouts and other things like "starting expression".

Todos:

• reorder
• add/remove or enable/disable
• summary box
• dim/brighten on enable/disable
• data prompt tooltip
• code text annotation
• update summary box on on rearrange/enable/disable
• update pipes on rearrange/enable/disable
• update outputs on action on rearrange/enable/disable

Upon detection of language engine, we should have the same functionality of parse/execute/store outputs, and stepping to work with R as well.

It could also be easier to do this because R is easier to introspect and no worries about output rendering as much.

Ideas:

• https://caleydo.org/tools/taco/
• https://dl.acm.org/doi/10.1109/32.177365

It would be nice to be able to customize the text within explanation summaries.

E.g.:

"The column names of `nba` are quite messy so, we first rename column names with `rename` using our earlier `column_names` dict."

<div id="rename-hint">
**Note:** dictionaries (dict) are closest to R's environment structure in which we have keys to value mappings. This is a very common data structure in Python that can be handy for renaming columns such as above.
</div>

One can imagine us embedded custom elements within the explanations, so seems like a good idea to support that from the get-go.

For e.g. in

diamonds %>%
  select(-everything()) %>%
  group_by(color) 

we get this:

This might be specific to group_by because it is attempting to take a row 0 tibble and attempting to group and failing. The error message strangely does not ripple up to frontend either even though I do see it in console log:

$ :List of 4
  ..$ line  : int 3
  ..$ code  : chr "\tgroup_by(color) %>%"
  ..$ change: chr "error"
  ..$ err   : chr "<strong>Error:</strong> Must group by variables found in `.data`.\n* Column `color` is not found."

The {reactable} has some nice hooks for JS / css styling and might be more performant than {kableExtra}, so consider using it for our tables: https://glin.github.io/reactable/articles/conditional-styling.html

reactable also would allow us to style header on data diff tables: https://glin.github.io/reactable/articles/examples.html#header-rendering

Switch to the new way of producing summary based on a function that takes in variable number of character or callout objects. This will also end up cleaning up the summary input on the stepper via explain since we won't have to use R inline function. Instead, we use the functions directly very similar to callout_ for dataframes.

Currently, we are using an extremely hacky function get_last_line which is not going to be robust.

Instead, let's use ast + astunparse to get the very last ast.Expr to evaluate as the "last result"

This will help to fix the following issues:

• #10 (use a small python script to return last Expr from code string)
• #6 (we won't need to manually handle comments etc.)

Use three functions for isolating data within a table:

• select()
• filter()
• arrange()

Use three functions for deriving new data from a table:

• summarise()
• group_by()
• mutate()

source: RStudio Cloud Primers: Work with data

The app does not gracefully handle invalid orders, for e.g.:

  select(...) %>%
diamonds

For this to work, we need to make sure that we highlight the code text in the Stepper shiny module. There are 2 viable options:

• use the prism.js library with Prism.highlightAll()
• even better would be to somehow make use of highlight.js (but this is not working currently)

Be able to detect columns used in a verb so we can allow data transparency of them in UI as well.

Check out https://github.com/r-lib/downlit.

Allow user to invoke the help page for a particular function within the code in Unravel.

TODO

• basic mockup of JS + R
• dynamically create verb editors
• user input 🎉
• dynamically render row/col
• handle updates after toggle updates
• handle updates after reorder updates

POC for these work:

• codemirror editor in Shiny
• codemirror with gutter markers in Shiny
• linked highlights for text -> code
• linked highlights for code -> text
• function to specify these callouts
• final integration into stepper

With Pagination on tables, we have too many next/previous going on, so remove it for stepper case.

Ex:

```{python ex-nba-load, exercise=TRUE, exercise.setup="pew_setup", opts.label="pprinter"}
nba
#

```

Cover:

• #
• """ docstrings

Rn, I'm a bit iffy on this. The pro is we work on similar looking presentation, but con is we are sorta "lying" in a sense of true repr.

I think we could for supporting this for consistent look and feel of dataframes, except we still can be honest about repr for components of the data (indexes, NaNs, etc.)

Some challenges:

• groupby object repr

TODO for reactable:

• columns highlighting
• column labels highlighting

This is to make it easy to write in 2 different styles depending on author's desire of compact vs spread versions of stepper.

Stack:

Warning: Error in UseMethod: no applicable method for 'mutate_' applied to an object of class "c('pandas.core.frame.DataFrame', 'pandas.core.generic.NDFrame', 'pandas.core.base.PandasObject', 'pandas.core.accessor.DirNamesMixin', 'pandas.core.base.SelectionMixin', 'pandas.core.indexing.IndexingMixin', 'python.builtin.object')"
  95: mutate_
  94: mutate.default
  92: function_list[[i]]
  91: freduce
  90: _fseq
  89: eval
  88: eval
  86: %>%
  85: python_df [/Users/nischal/Documents/rstudio/DataTutor/R/utils.R#64]
  84: kable_pandas [/Users/nischal/Documents/rstudio/DataTutor/R/kable_pandas.R#32]
  83: df_kable [/Users/nischal/Documents/rstudio/DataTutor/R/stepper.R#305]
  82: output$nba_stepper-line_table [/Users/nischal/Documents/rstudio/DataTutor/R/stepper.R#410]
   3: <Anonymous>
   1: rmarkdown::run

I believe this was happening because of the hacky solution to get the "last expression/value" and should be updated with a more robust way to get that either via reticulate or python script.

Ex:

column_names = {'Date': 'date', 'Start (ET)': 'start',
  'Unamed: 2': 'box', 'Visitor/Neutral': 'away_team',
  'PTS': 'away_points', 'Home/Neutral': 'home_team',
  'PTS.1': 'home_points', 'Unamed: 7': 'n_ot'}
nba = (
  nba.rename(columns=column_names)
  .dropna(thresh=4)
  [['date', 'away_team', 'away_points', 'home_team', 'home_points']]
  .assign(date=lambda x: pd.to_datetime(x['date'], format='%a, %b %d, %Y'))
  .set_index('date', append=True)
  .rename_axis(["game_id", "date"])
  .sort_index()
)
nba

This could be solved with the python version of Daff, but it looked a bit clunky: https://github.com/paulfitz/daff/blob/a2a013cef4c842bef1e8babacdc99fc6e5a1be07/scripts/example.py

The problem with the way text summary is placed in the center is that the size of the area is fixed. This is a problem because what if text takes up a lot of space? It would overflow. In the dynamic case, it's also annoying because it would keep pushing the output downwards.

An alternate layout would be code and output are right on top of each other and almost fused as one entity (see this and this). The output df can then be fixed for dataframe outputs and other types of outputs can also be fixed.