Vega-Lite 4.0 adds a number of additional transforms, including density, loess, pivot, quantile, and regression. These can't be fully supported until Altair 4.0 comes out, but we could start implementing them before that.

We should support transforms specified as part of encodings. For example, we could create a helper function that would take a spec like this (vega editor):

{
  "data": { "url": "data/population.json"},
  "mark": "bar",
  "encoding": {
    "y": {
      "field": "age", "type": "ordinal"
    },
    "x": {
      "aggregate": "sum",
      "field": "people",
      "type": "quantitative",
      "axis": {"title": "population"}
    }
  }
}

and convert it to something like this (vega editor):

{
  "data": {"url": "data/population.json"},
  "transform": [
    {
      "aggregate": [{"op": "sum", "field": "people", "as": "people"}],
      "groupby": ["age"]
    }
  ],
  "mark": "bar",
  "encoding": {
    "y": {"field": "age", "type": "ordinal"},
    "x": {
      "field": "people",
      "type": "quantitative",
      "axis": {"title": "population"}
    }
  }
}

Before passing it to the transform functionality.

I wanted to reach out, since we have been working on a similar project over at https://github.com/Quansight/jupyterlab-omnisci.

The goal there is to let users create Altair charts and have the heavy lifting transparently executed on a database.

To get a feel for it, you can open the notebooks/Ibis + Altair + Extraction.ipynb notebook in Binder. If you run the cells, the graphs should appear.

We are using Ibis to build up the SQL expression. We are building it to execute on an OmniSci database, but most of the work should translate to any other Ibis backend.

Currently, we update the Vega Lite spec to take out the transforms and map them to Ibis. So we are implementing a very limited version of what you have here, targeting Ibis instead of Pandas, and using the extracted transforms in the VL spec.

However, our next goal is to support interactions, so that after a user interacts, a new query is computed and run. To do this, we are looking to switch from processing the Vega Lite spec to using the underlying Vega spec or graph. The idea being, we take the initial Altair chart, generate Vega Lite, convert to Vega, then pre-process the Vega spec to turn some of the transforms into a custom transform that will run the query using Ibis back on the kernel. We are tracking that here: https://github.com/Quansight/jupyterlab-omnisci/issues/54

On the Python side, that would involve somehow taking an existing Vega dataflow graph or Vega spec and understanding how those operations map to Ibis expressions. It seems that task shares a lot in common with what you have implemented here.

Like I said, although this work initially targets OmniSci, and their database is particular suited to computing these types of analytic queries, I hope that the general approach will be useful generally for using Altair in Python with other data sources on the kernel, like Pandas dataframes or other databases.

I would be happy to collaborate on any part of this that you would like or get your feedback on your general approach and understand if you have thoughts on how to support this kind of use case on top of Altair.

Also, thank you for helping to maintain this repo!

It's a treat to be able to use the UX in Altair to create large scale visualizations.

import pandas as pd
import numpy as np
import altair as alt

data = { 'ColA': {('A', 'A-1'): 'w',
                 ('A', 'A-2'): 'w',
                 ('A', 'A-3'): 'w',
                 ('B', 'B-1'): 'q',
                 ('B', 'B-2'): 'q',
                 ('B', 'B-3'): 'r',
                 ('C', 'C-1'): 'w',
                 ('C', 'C-2'): 'q',
                 ('C', 'C-3'): 'q',
                 ('C', 'C-4'): 'r'},
        'ColB': {('A', 'A-1'): 'r',
                 ('A', 'A-2'): 'w',
                 ('A', 'A-3'): 'w',
                 ('B', 'B-1'): 'q',
                 ('B', 'B-2'): 'q',
                 ('B', 'B-3'): 'e',
                 ('C', 'C-1'): 'e',
                 ('C', 'C-2'): 'q',
                 ('C', 'C-3'): 'r',
                 ('C', 'C-4'): 'w'} 
        }
                 
df = pd.DataFrame(data).reset_index( drop = True )

mychart = alt.Chart(df).transform_fold(
    [r'ColA', 'ColB'], as_=['column', 'value'] 
).mark_bar().encode(
    x=alt.X('value:N', sort=['r', 'q', 'e', 'w']),
    y=alt.Y('count():Q', scale=alt.Scale(domain=[0, len(df.index)])),
    column='column:N'
)

from altair_transform import extract_data
data = extract_data(mychart)
data.head()

generates the error:

altair-transform/altair_transform/core/fold.py in visit_fold(transform, df)
      9     transform = transform.to_dict()
     10     fold = transform["fold"]
---> 11     var_name, value_name = transform._get("as", ("key", "value"))
     12     value_vars = [c for c in df.columns if c in fold]
     13     id_vars = [c for c in df.columns if c not in fold]


AttributeError: 'dict' object has no attribute '_get'

import pandas as pd
import altair as alt
from altair_transform import transform_chart

df = pd.read_json('https://vega.github.io/vega-datasets/data/iris.json')

error_bars = alt.Chart(df).mark_errorbar(extent='ci').encode(
  x=alt.X('petalLength:Q', scale=alt.Scale(zero=False)),
  y=alt.Y('species:N')
)

transform_chart(error_bars)

The transformation does not take place but NotImplementedError is also not raised (which would be expected based on the Limitations section in the README).

I'm trying out altair-transform, and when I run the second example, I still get the MaxRowsError even with transform_chart(chart). Running extract_data(chart) doesn't work either; it gives me the input data unchanged.

Strangely, the first example works: extract_data(line) returns the aggregated dataframe.

altair v3.2.0
altair-transform v0.1.0

import pandas as pd

source = pd.DataFrame({"A": [1,2,3,4], "B": pd.date_range("2023-06-1", periods=4, freq="D")})

from altair_transform import extract_data

# Works:
data = extract_data(alt.Chart(source).encode(x="A:Q", y="B:Q"))

# AttributeError: module 'pandas' has no attribute 'Int64Index'
data = extract_data(alt.Chart(source).encode(x="A:Q", y="date(B):Q"))

pd.Int64Index was removed in Pandas v2.0 (https://pandas.pydata.org/pandas-docs/version/1.5/reference/api/pandas.Int64Index.html) in favor of pd.NumericIndex

For example, this should work:

import altair as alt

chart = alt.Chart(
    {"sequence": {"start": -4, "stop": 4, "step": 0.1, "as": "x"}}
).transform_calculate(
    y="densityNormal(datum.x, 0, 1)"
).mark_line().encode(
    x='x:Q',
    y='y:Q'
)

from altair_transform import extract_data
extract_data(chart)

Currently the altair object transform only supports simple chart specifications. With a bit of work, we could also support layered, faceted, concatenated, and repeated charts as well.

Vega has some fairly sophisticated logic around choosing bin boundaries. Currently altair-transform does not do a good job of duplicating that logic.