Similar to Elm's online editor. http://elm-lang.org/examples/buttons

• Parse forall x, P(x) and exists x, P(x) in Substance

• Handle the binding structure of nesting quantifiers

• Design a quantifier pattern-matching mechanism in Style

• Find common diagrams that show the visual semantics of forall or exists

• Write the visual semantics in Style programs

• Design interaction techniques for the semantics (e.g. forall lets you drag a point and exists does not)

• Implication: ?

This GitHub issue tracker has too many "bucket list" and "nice to have, one day" research ideas as issues. It's useful to track and label these ideas for future students, but it's hard to find the real issues among them. Therefore, I plan to make a separate repository or README to store the bucket list.

...and rename the fields of objects accordingly to be much more concise and consistent. https://ghc.haskell.org/trac/ghc/wiki/Records/OverloadedRecordFields/DuplicateRecordFields

With constraint at(A, 0, 0), the set sizes blow up and then NaN. Only happens in snap, not gloss. Programs: https://www.dropbox.com/s/pqhphxgrn31wp98/Screenshot%202017-10-23%2017.57.20.png?dl=0

Better to just support one frontend.

In principle, the exterior point method requires the initial state to start in the exterior of the feasible region; that is, at least one of the constraints should be violated. See this overview of the EP method.

In practice, it doesn't seem to matter for our small examples (so this issue isn't really a bug yet). Currently we sample an initial state according to the initRng seed and it may or may not satisfy all the constraints (it's more likely not to if there are more constraints).

But we should do the principled thing eventually, and provide a boolean constraint function with each penalty-version constraint function, so sampleConstrainedState can use it according to the Style program.

See project proposal here (internal link).

• Clarify what properties a visual primitive (e.g. Arrow, Circle) has.
• Document the existing types' attributes (e.g. a circle's radius) and which properties are "added" (e.g. arrow's start and end).
• Figure out how to make them work together.

This issue has two goals:

• Improve and showcase the generality of our system's rendering frontend.
• Generate more beautiful and useful visualizations for particular domains that are well-suited to 3D diagrams.

Some first steps:

• To ramp up, first add a 3D Style and renderer for existing domains we've implemented, e.g. set theory and linear algebra. Step number one is to create a version of venn.sty where the circles are now semi-transparent spheres.
• The Substance programs should largely stay the same. The Style programs will change to use 3D types/functions/objectives and to specify a renderer. For example, any sampling mechanisms that used types like a 2D vector or matrix should be generalized.

More specifically:

• Add host language datatypes that support optimizing in 3D
  • Possibly split out and generalize rendering functions for each shape
• Write objectives and constraints for optimizing in 3D
• Write styles for 3D (e.g. bold outlines, soft shadows, contour lines, 3D surfaces, lighting)
• Output the optimized world as a 3D scene in some format for some renderer. Perhaps output COLLADA (an XML-based schema for 3D scenes), POVRAY, or the pbrt input format
• Optimize and project labels onto the rendered image

Later:

• Add new Substance domains that benefit from 3D diagrams and scenes, such as geometry, topology, meshes, raytracing, and optimization, and repeat the steps above
• Write documentation, examples, and tests. Especially document how to generally extend the system with a new renderer (3D or not).
• Possible research questions: optimizing curves on surfaces, creating illustrative rendering styles (à la Blender Freestyle), sampling 3D diagrams

Stretch goal:

• Output to VR?! Generate something like the Venn Diagram Museum

To look into:

• What concrete examples/classes of well-known, compelling, and useful 3D diagrams could we aim to create?
• What renderers could we use, and with what output formats? What are the pros and cons of each?
• What other related work is there?
• What research questions might we explore?

More internal discussion, and links to related work, can be found here.

If an optimization involves a computation, e.g. objective repel(f(x), g(y), 10), and the parameters are varying, let the optimization/autodiff differentiate through f and g by having the user provide df/dx and dg/dy.

Goal

Overview

To accomplish the cartesian coordinate system visualization for function, we need the following new geometries:

• Bezier Curves
• Dotted Lines
• Shaded region

Important notes

• I did not implement any syntax for specifying the actual path for curve. Now the only way to do so is to change initCurve function, where I hard coded a list of points for testing purposes
• @hypotext do let me know if you want some syntax for specifying path, say

  Curve { path = [(0, 0), (100, 100)] }
  

• The packing and unpacking gets tricky. What I did was to flatten the entire path array, mark all of them as Fixed, and pack them up again. Not sure if that fits @hypotext 's assumption in computation.
• Because of the limitation of Located interface, I had to do some computation to properly do translation of all points on a path. I awkwardly returned the center of the bbox for getX and getY. We should change Located very soon. #22 requires a total rewrite on this, so I guess we can deal with it soon.

Bezier Curves

• API: A bezier curve is simply a list of points ([(a, a)]
• SVG standard path string: there is a specific format that SVG (and thus Snap.svg) follows.
• The frontend will translate the list of points (passed in from the backend as 2D array) to a path format string based on the following assumptions
  • We are generating Cubic Bezier curves, not quadratic ones
  • The first point is the starting point, which gets a special command M
  • The second, third, and fourth point are the control points for the first Bezier curve, which get a command C
  • The rest of the points are even-numbered. Every two of them forms another Bezier curve connecting to the previous one. The reason why we only need two more point can be found here. Essentially we want the combined curve to be smooth, and SVG automatically figures out how to connect the neighboring curves
  • Example: [(10, 100), (50, 0), (60, 0), (100, 100), (250, 250), (300, 100)] gets translated into "M 10 100 C 50 0, 60 0, 100 100 S 250 250, 300 100"

Dotted lines

• Any Curve type geometry with a path of only two points is a line
• Added String literal support to the parser so that we can specify properties in the way similar to CSS. For instance, Curve { style = "dashed" } where a lookup for style is hardcoded in Runtime.hs in initCurve function. The righthand side can be arbitrary strings here. It is up to the frontend to decide what to render.

Some screenshots

• Sample program:

-- Substance
Set A, B
f: A -> B

-- Style

Map f x y {
    -- Shape of the curve determined by the hard coded path
    shape = Curve {
        style = "dashed" -- if unspecified or set to anything else, a solid line appears
    }
}

I ran ghc Runtime.hs to compile the system and got the following error message:

$ ghc Runtime.hs
[2 of 2] Compiling Main             ( Runtime.hs, Runtime.o )

Runtime.hs:1231:42: error:
    • Could not deduce (Floating a0)
      from the context: (Floating a2,
                         Show a3,
                         Ord a3,
                         Floating a3,
                         Real a3)
        bound by the type signature for:
                   objFnUnconstrained :: (Floating a2, Show a3, Ord a3, Floating a3,
                                          Real a3) =>
                                         [a3] -> [a3] -> a3
        at Runtime.hs:1231:42-63
      The type variable ‘a0’ is ambiguous
    • In the ambiguity check for ‘objFnUnconstrained’
      To defer the ambiguity check to use sites, enable AllowAmbiguousTypes
      In the type signature:
        objFnUnconstrained :: Floating a => ObjFn2 a
      In an equation for ‘objFnPenalty’:
          objFnPenalty weight
            = combineObjfns objFnUnconstrained weight
            where
                objFnUnconstrained :: Floating a => ObjFn2 a
                objFnUnconstrained = centerObjs

• Enable layouts like grids, graphs, radial, etc. without optimization (just compute and place the objects)
• Add keywords to compute non-optimized relative positions like "above", "below", "left of", "right of", "on", "in", "aligned with", "around"
• Look at diagrams and xy-pic for inspiration

[Potential ramp-up task for Lily]

Style writers and implementers need abstractions for optimizing the relationships between any pair of objects, rather then having to write special-purpose graphics code for every new objective/constraint function and every n-ary combination of graphical primitives.

Therefore, to improve the generality of our optimizer and the user experience for Style implementers, do the following:

For any object and any point in the plane, add the ability to compute a signed distance function from that point to the object (positive if outside, negative if inside). Additionally, compute level sets of distances for use in margins.

Test how well your implementation works with the existing optimizer and how well it works with the existing Styles. Specifically, test the following two important applications of signed distance:

• Improve our label layout (and any objective functions that act on text), now that you can calculate the signed distance to the outline of the text, rather than just a bounding box.
• Optimize objects composed of Bezier curves by writing new objective functions that use the signed distance to Bezier curves.

This issue depends on issue #75.

Currently only the gloss frontend outputs debug messages.

• Reduce number of changes needed to add a new object: replace Floats with polymorphism and automatically derive all the typeclasses (Located, etc.).
• Reduce number of changes needed to add a new objective function or computation (e.g. registering its type, adding the function name to the dictionary). This can be done by pre-generating the registration code before the Penrose system is compiled.
• Reduce number of changes needed to add a new computation: generate or remove computeOn so people don't have to implement typechecking.

Give examples of how to add/modify new objects, functions, computations.

A lot of these can be done using Template Haskell's quoting and reification features: https://wiki.haskell.org/A_practical_Template_Haskell_Tutorial#Reification

Candidates: Elm, Purescript, Typescript, Javascript, maybe using d3, Svelte, Snap, React.

Figure out the state of web support in the following:

• Sophisticated static type systems and typechecking
• Autodifferentiation
• Optimization, linear algebra, and numerical computation
• Parser combinators and generators
• DSL definition, design, metaprogramming, reflection, language workbenches
• Property-based testing and unit testing
• UI libraries for prodirect manipulation

Figure out which parts of the backend to leave in Haskell.

After the optimization is done in snap with autostep, sometimes the diagrams will disappear:

Also, sometimes the diagram will disappear after autostep is changed from on to off.

Maybe the server is dropping frames?

I've sometimes been able to get the disappearing problem to stop by restarting the Haskell server.

This happens on branch shapes, with either of the following two commands: ./Main snap sub/cart-test.sub sty/cart-test.sty or ./Main snap sub/twopoints.sub sty/twopoints.sty

...perhaps with an emacs or Atom mode.

• Allow users to define Style modules (of objectives, constraints, computations, and visual semantics) and import specific ones.
• Add import keywords to Style parser.
• Make Style library compilation independent from compilation of the Penrose system. (Dynamically link libraries in Haskell?)

The system seems to have problems optimizing certain constraints. Running the following command has several problems. ./Main snap sub/twosets.sub sty/venn_comp.sty

• constraint contains(y, x) does not work well in two scenarios: 1. with any objective on the labels, or 2. if the constraint on containment starts off violated.
• In principle, the exterior point method requires constraints to be violated in the initial state (that is, the "point" lies in the exterior). Setting constraintFlag = True means that the Subset constraint is initially satisfied, which defeats the purpose. Even if the flag is false, the system does not guarantee that at least one constraint is violated, as the rng seed might be set such that the constraints happen to be satisfied.
• For some reason, if the x and y coordinates of the two sets are set to varying, then they are NaNing after one step. Only those coordinates NaN out first; the radii are fine. If the x and y coordinates are set to fixed, then there aren't problems.
• A lot of things could possibly be causing this problem. Autodiff? Initial changes made to implement the computational graph? The objective function strictSubset being written incorrectly? (Seems to work though.) A problem with circle packing and unpacking?

Temporarily resolved. To fully resolve:

• If I square both sides, will strictSubset still work as a valid penalty?
• Check if state starts in the exterior now
• Why did distance work sometimes originally?
• Document the solution with avoiding divide by zero
• Why is the optimization so slow? Might have to do with starting in exterior vs. interior
• Test the full subset program
• Regression-test the existing programs that might have NaNs
• Add test suite

...instead of just one diagram at a time.

With commit 5a705a on shapes, the Bézier curve does not show up in snap, in Chrome, on a Mac. It seems to be caused by a Chrome extension that I have, since it works in incognito and in Safari, and the CB is successfully sent in JSON.

Chrome version: Version 62.0.3202.89 (Official Build) (64-bit)
Programs: cart-test.sub cart-test.sty

e.g. linear algebra and mesh diagrams, or see L. Kuper's comment on Twitter.

The following program does not parse and I believe it should:

Set A
Point B
In B A

Main: sub/venn_comp_pt.sub:3:1:
unexpected 'I'
expecting alphanumeric character, end of input, letter, or newline

CallStack (from HasCallStack):
  error, called at ./Substance.hs:341:18 in main:Substance

Like Python and Mathematica, and with sensible defaults. Example:

Proposed solution

• Allow LaTeX strings to be rendered in labels, e.g. `Label X $\sum_i^n\frac{x^2}{10}$
  
• A labeling language, so Substance variables can be named in ASCII and given TeX labels, e.g.

Set A, B
f: A -> B
AutoLabel A, B
Label f $\sum_i^n \frac{x^2}{10}$

data SubStmt = {- all other Substance stmts -} | LabelDecl String String | AutoLabel [String]

Set A, B
AutoLabel A, B

f: A -> B
Label f $\sum_i^n \frac{x^2}{10}$

Set A {
  component_1 = Circle {
      text = concat(A.text, "_1")
  }
  component_2 = Square {
      text = concat(A.text, "_2")
  }
}

TBD.

All function comments should be illustrated by and accompanied by concrete examples.

For example, "f gives the head of a reversed list" is better written as "On an example list y = [1,2,3], f reverses the list to [3,2,1] and returns the head element [3]."

Even better, these concrete examples should just be written as unit tests (#8). Better readability and usability.

Also, make all assumptions explicit, e.g. "f assumes that the input list y is nonempty." Ideally, enforce them via invariants: "f [] = error 'empty list'" or better.

All function declarations should have type annotations, unless typeclasses and polymorphism make it too unwieldy.

Try to avoid single-letter variable names unless it's i, j, (index), n (number), or a specific parameter without meaning (e.g. a in line search).

In where clauses, declare things in definition order, like you would in a series of let clauses.

Standardize where vs. let usage. (I prefer let, since it enforces an order that is more readable.)

In particular, it could be nice to check the properties of exhaustiveness and non-redundancy (in analogy to pattern-matching).

Checking exhaustiveness “statically” would entail showing that a Style can “visualize anything writable in this DSL.”
Checking exhaustiveness “dynamically” would entail showing that a Style will “visualize everything in this particular Substance program” (nothing has been accidentally left out).
Checking non-redundancy “statically” would entail checking that no selectors are the same or equivalent.

See PFPL, 1st ed, chapter 13.

Right now all points in the curve are fixed. Can we make them a part of the optimization parameters?

In frontend, show toolbar with some of the following:

• Value of state
• Energy over time
• Gradient norm over time
• Exterior point method weight and iteration
• Unconstrained optimization progress
• Line search progress
• Debug values for pre-NaN fixing
• The (projected) energy landscape of the optimization problem, sample diagrams in valleys and peaks, and paths along the energy landscape
• Line search parameters
• Are there other quantities we can visualize to help debug the opt? e.g. Tensorboard and Edward

With adjustable parameters, e.g. a and b in line search.

Notes:

• Probably the constraints are what's making the opt problem hard, so focus on those
• Show energy or loss (or other metrics) on the diagrams themselves, as they change over time
• For example, highlight the areas on the diagrams where the constraints are the most violated, and highlight directions of steepest ascent/descent for various objective/constraint functions

(Could the design of optimization visualization/debugging UIs be a research question in itself??)

See graphics class and DDG course notes.

If we implemented our own PRG (or family of PRGs) with the autodiff, depending on the kind of PRG, we might be able to differentiate with respect to the PRG's parameters to minimize a loss. (Being able to differentiate with respect to the PRG's input would probably break the "pseudorandom" property.) That means we could, say, generate new "random" points (or other mathematical objects) that improve label legibility in a smaller bounding box.

In most diagrams, there is on the order of tens of parameters to optimize, and the system visibly takes a few seconds to lay it out. Figure out where the bottleneck is: JSON message-passing? optimization? something else?

For example, ellipses take an especially long time to optimize: ./Main snap sub/composition.sub sty/composition.sty

Now

• Get original injection/bijection/surjection example working
• Fix diagram aesthetics (axis length, bbox margins, Bezier curve positioning and scaling WRT other objects)
• Compute curve bbox in terms of optimized objects, rather than hardcoding. Requires scheduling.
• Show generated points on curve
• Sample new *jection on refresh?
• Clean up lookup of functions in initCircle
• Generalize beyond circles in computeInner
• Apply computations on resample, accounting for state order (currently they are not applied)
• Add post-optimization computation application for both renderers
• Test with more thorough examples
• Extend parser to allow computation in any Expr? What would that entail for the rest of the system?
• Pass randomness around through computations in Runtime

Fixed/varying

• Fix the fact that all circle radii are hardcoded to be fixed for this example
• Come up with a better computation/optimization model
• Generalize "Fixed" and "Varying" to be specific to an object (by name), not by type
• In Style language, allow user to specify fixed or varying

Soon

• Automatically look up and add all computations for each attribute of an object (currently it's manually looked up for each attribute e.g. "radius" for circle)--and don't hardcode expecting a computation or a variable, allow defaults
• In Style, resolve / look up vars in function params (currently they are passed as-is to Runtime, ignoring pattern-matching)
• Bind results of computation to a name in Style; give the resulting object a visual semantics
• Compute dependency graph/scheduling (when are computations evaluated?)
• Genericize computeRadius function, allow attributes (or references to attributes?) to be passed in to computations

Later

• Come up with more principled input/output typechecking and attributes in applyComputation
• Implement scales
• Fix style design: subset, % of radii, (like CSS%), why are the sets tangent
• Fix color shadowing
• Support ~ and * syntax in Style
• Implement vector addition and multiplication examples
• Smart-convert between float and int in Style parsing

One day

• Use Template Haskell reification instead of "poor man's reflection"
• In Style, support quoting Haskell and antiquoting into Style
• Multiple function calls and function composition: typechecking?

Add language constructs for specifying the staging, or order, or a diagram, as it might be used in a talk, animation, or presentation.

Comment from Keenan: Would also be good to infer, as much as possible, a default ordering (e.g., based on logical dependencies).

Also we need to allow users to specify diagram display / layering order (so shapes with transparency don't display on top of labels, and so on).

I've started documenting the test infrastructure and the working and not-working program pair examples in the Penrose wiki.

Would be nice to have code coverage stats. stack test --coverage doesn't yet work.

TODO:

• account for label size set in Snap
• add rest of examples as regression/unit tests
• remove gloss dependency
• test the optimization and computation more thoroughly

Eventually:

• figure out how to test js files

The initial exterior point weight seems very high. But the error doesn't happen in snap.

With the debug output in the computation branch, it seems that Style programs are being analyzed and parsed multiple times during runtime, each time a diagram is re-sampled. This is unnecessary as programs won't change during runtime.

GUI changes to the diagram (location, duplication, etc.) should be inferred/applied/stored in the Substance program, and vice versa. Depends on #17.

We currently support lines that look something like this:
Vector v, w
Scalar u
InnerProduct u v w

We want to support a line like this: u := <v, w>

We need:

• to infer the type of u, v, w (and/or do typechecking)
• to allow simultaneous declarations and constraints in one line
• to allow the definition of syntax sugar for innerProduct
• to allow Style writers to pattern-match on the sugared form(s)
• to figure out how to deal with nested operations in both the parser and the pattern-matcher, e.g. u := <v, w> + det(x) + 4 (and figure out what we want to allow the user to write)

See Keenan's linear algebra spec for examples.

There are a lot of inline TODOs and FIXMEs. Let's document the reasons behind them, remove them, or fix them.

What needs to be changed across Substance, Style, Runtime, etc. if an implementer simply wants to add a statement like Scalar c := innerProduct(v, w) and style it in a custom manner?

The first step here is to replace some of the parameters in the computations/objectives with negative numbers (see the sample programs in the wiki), see what breaks, and then fix the parser.

Possibly following this guide: https://www.schoolofhaskell.com/school/to-infinity-and-beyond/pick-of-the-week/parsing-floats-with-parsec#minus

Learn the Substance program from a sketch or series of strokes, similar to this paper: https://arxiv.org/abs/1707.09627

...to make sure the diagram fits well and is centered.

• Should the user have to write scales (or be able to write scales)?
• Or should there be an automatic conversion layer?
• Where should that layer be?
• Should the user have access to the canvas size / bounding box (or be able to set it)?

Create GUI capabilities that are aware of optimization objectives and constraints. Cinderella and CAD tools are examples of tools that are constraint-aware. The novelty is in designing a UI for Penrose that is also objective-aware. Some interesting problems:

• an object (e.g. label) could be easier or harder to drag to a certain place, according to the overall objective function
• highlight the background with colors (green, red, etc.) to show local minima for different objectives (e.g. labeling vs. repelling) and the overall objective
• the objective function is a function of many variables, so figure out a way to project and display the optimization landscape (with the diagrams in various locations in the landscape) in an intelligent manner (e.g. t-SNE, PCA)