idyll-lang / idyll Goto Github PK

Create a simple project from scratch and run yarn link "idyll-default-components" (or npm link if you prefer). When you try to run Idyll you will get an error something like this.

Can you reproduce @mathisonian?

ReferenceError: Unknown plugin "transform-runtime" specified in "/Users/bclinkinbeard/Code/idyll-lang/idyll-default-components/node_modules/regenerator-transform/package.json" at 0, attempted to resolve relative to "/Users/bclinkinbeard/Code/idyll-lang/idyll-default-components/node_modules/regenerator-transform" (While processing preset: "/Users/bclinkinbeard/Code/idyll-lang/idyll-default-components/node_modules/babel-preset-es2015/lib/index.js")
    at /Users/bclinkinbeard/Code/scratch/waypoints/node_modules/babel-core/lib/transformation/file/options/option-manager.js:180:17
    at Array.map (native)
    at Function.normalisePlugins (/Users/bclinkinbeard/Code/scratch/waypoints/node_modules/babel-core/lib/transformation/file/options/option-manager.js:158:20)
    at OptionManager.mergeOptions (/Users/bclinkinbeard/Code/scratch/waypoints/node_modules/babel-core/lib/transformation/file/options/option-manager.js:234:36)
    at OptionManager.init (/Users/bclinkinbeard/Code/scratch/waypoints/node_modules/babel-core/lib/transformation/file/options/option-manager.js:368:12)
    at compile (/Users/bclinkinbeard/Code/scratch/waypoints/node_modules/babel-register/lib/node.js:103:45)
    at loader (/Users/bclinkinbeard/Code/scratch/waypoints/node_modules/babel-register/lib/node.js:144:14)
    at Object.require.extensions.(anonymous function) [as .js] (/Users/bclinkinbeard/Code/scratch/waypoints/node_modules/babel-register/lib/node.js:154:7)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)
    at Function.Module._load (module.js:458:3)
    at Module.require (module.js:513:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/Users/bclinkinbeard/Code/idyll-lang/idyll-default-components/node_modules/babel-plugin-transform-regenerator/lib/index.js:3:18)
    at Module._compile (module.js:569:30)
    at loader (/Users/bclinkinbeard/Code/scratch/waypoints/node_modules/babel-register/lib/node.js:144:5)
    at Object.require.extensions.(anonymous function) [as .js] (/Users/bclinkinbeard/Code/scratch/waypoints/node_modules/babel-register/lib/node.js:154:7)

The documentation states that this combination of flags should build a bundle:

idyll my-file.idl --build > bundle.js

But when I run this on my project...

./node_modules/idyll/bin/idyll.js index.idl --build > bundle.js

... I just get the stdout echoed to the file.

Spellcheck:
No misspellings found.

Is the documentation out of date, or am I doing something wrong?

Unless I'm mistaken, this line returns a new callback each time it's invoked here to generate new props for each component. The result is that it triggers a shallow prop comparison mismatch and causes components to always trigger PureComponent's shouldComponentUpdate check.

The solution would seem to be to either rework the pattern to not require a thunk or to customize idyll-component's shouldComponentUpdate method to avoid checking the identity of __handleUpdateProps, which always changes.

The getMeta function should be updated to handle the case where no meta component is defined

Right now if you create a new custom component, you have to stop and restart the server to get it to show up. Thanks @rreusser for raising this

Can we have the ability to create dashboards.

a great feature of budo is that it shows errors directly in the browser so you know when something isn't compiling and it doesn't get lost in the terminal. idyll should do this too.

Right now the name of the output file is hard-coded as index.html. I'm just going to go out on a limb and propose:

1. idyll can optionally (--batch?) look for all .idl files in the current directory.
2. It would look for _filename.html and then _index.html and then the internal _index.html when looking for a template for filename.idl so that you could customize if desired.
3. If there's more than one idyll file, idyll creates the ast for each one first (could store in .idyll/filename.ast.json instead of .idyll/ast.json) so that it can create a common bundle for all the components (can punt on component source bundle splitting for now, right?)
4. The ast json is inlined into the html file so that each file still only has to include index.js. Or else it could be included separately so that you could do fancy history API transitions or something. But at that point, why are you using idyll instead of just writing a React app. Or at that point you could be using idyll-compiler and the InteractiveDocument component within your own react app.

The use-case of this would be, say, a project with multiple pages. Since they would all have the same component bundle you couldn't perfectly optimize the bundle for each page, but it would at least facilitate a MIMO pattern.

Some aliases use camel case, while others uses kebab

  .alias({
    m: 'components',
    c: 'css',
    d: 'datasets',
    q: 'defaultComponents',
    f: 'inputFile',
    s: 'inputString',
    l: 'layout',
    n: 'no-minify',
    o: 'output',
    r: 'no-ssr',
    t: 'template',
    e: 'theme',
    w: 'watch'
  })

All errors in the pipeline are just logged to the console. We probably need an error event so API clients can react accordingly.

Add in automatic CSV -> JSON conversion, we might have to expose some additional options for the CSV parser.

[data name:"myCSVData" source:"data.csv" /]

NODE_ENV=production should be set automatically during a build, but this shouldn't be set if watch is true: the production flag will cause valuable debugging information to become inaccessible.

Currently the meta component is just used to populate meta variables in the HTML header, but it still adds a <meta> tag to the body of the HTML.

It would be better if the component populated those variables, but didn't add anything to the body

Because it is transforming things globally, rather than only transforming client side code for SSR

SSR seems to create some weird loading flash behavior with certain components. Sometimes this seems worse than not rendering on the server at all.

This would be better as an option rather than always on

Hey! I wish I discovered this before writing out all the logic the flame maps demo page in React. It would have simplified my life a lot...

Anyway, I noticed that if you turn JS off on any of the demos, the entire webpage remains blank. That's because of React, I suppose. I had the same thing, and I fixed it by manually duplicating all the text parts of my website inside the <noscript> tag of the index page. Since it's text only, and not that big of a webpage, it doesn't add much overhead.

I know the whole point of Idyll is to make an interactive website, but some form of graceful degradation would be nice I think.

It would be nice if (like markdown), you could just open up a single file and start working. The minimal project for me looks like:

project-dir
├─┬ components
│ └── default_components
├── package.json
└── index.idl

(It's a little unhappy that _index.html doesn't exist, but it still seems to render.) Seems like it would maybe be nice (and not fundamentally difficult?) to eliminate the need for components and package.json so that you really could just open up a file and start, then add pieces as needed. Presumably it would just override built-in defaults with more pieces as specified. This is probably more a matter of configuration than anything else.

The benefit to me is that it's easier to start from scratch and feels more like a 'language' + 'document' when it's just a file and pulls in external resources as needed, rather than the fundamental unit being a carefully organized collection of files. (= framework?)

Especially since #50, looks like shouldn't be too bad. Thoughts/opinions?

A good suggestion from @ejb

If someone wants to use a custom theme, it would be very useful to be able to specify a local CSS file, e.g.

idyll index.idl --theme ../my-custom-theme.css

The same logic could be applied to layout.

This issue is to discuss and prioritize upcoming work. I think/hope this will make things sufficiently polished to warrant a v2 release. My hope is that between myself and @mathisonian (and anyone else who wants to help!) we can complete these items by the end of August.

• Move to a monorepo structure using Yarn workspaces (announcement post preview)
• Clean up and refine idyll-interactive-document, specifically the walkers
• Improve tests
• Re-enable Travis for the monorepo
• Curate and add to the list of default components (will create a separate issue to discuss in detail)
• Remove the need for idyll-component and improve support for plain React components
• Ensure editor code is production-ready
• Clean up compiler related to issues mentioned below
• Adopt React 16
• Fill out and update READMEs for all packages
• Update site docs
• Ensure site has at least basic sponsorship info
• Getting idyll-editor publishing to idyll-lang.github.io/editor
• Deprecate old packages (idyll-interactive-document, idyll-component, generator?)

If you change a component, the server doesn't automatically pull in the new code for SSR, even though it should. This can lead to things being slightly out of sync if you are changing custom component code

It would be nice to make the transforms passed to browserify configurable, or at least let people add more in addition to the ones that idyll includes by default.

It seems like sometimes editing a CSS file isn't triggering a reload properly (or else the sheet is being re-generated)

I usually capitalize component files if they use class, which I think is a pretty common convention. Our logic for finding files does not currently support this but it probably should.

I've determined that expect.js is the reason the tests hang when trying to compare the JS bundles. Node's native assert does the same comparison with no delay whatsoever, as does Jest. Since Idyll is creating React projects it seems like Jest would be the best choice for a testing library as it may enable more elaborate testing in the future.

If you're amenable to this change I can have a PR ready in the next couple of days.

The README shouldn't be displaying docs if it isn't automatically kept in sync with the website. See #22 and #23

Right now Idyll just expects you to use the ./build directory when you build a version of a project. This should be an option instead

Can you support plot.ly out of the box

https://edwardtufte.github.io/tufte-css/

idyll-lang/idyll-compiler#14 adds support to capture a language associated with a code block, but this information isn't used right now.

Idyll should automatically do syntax highlighting if you specify a language.

hi,
I did a fresh install of idyll and yo. Ran yo idyll and npm start and got:

Spellcheck:
var: did you mean vary
Error: Cannot find module 'c:\git\demo-idyll\c' from 'c:\git\demo-idyll'

anyone else got this problem?

Is it possible to support getting the data and updates from the backend.

[Loader src:"true" /] and [Loader src:1 /] both work, but [Loader src:true /] fails with a message like Error parsing input at line 14, column 72. Is there a way to pass boolean props to component that I'm perhaps missing?

Currently the github theme scopes all of its styles within .idyll-root, making it somewhat a pain to override things (e.g. you have to write .idyll-root a {...} instead of just a {...} to change simple link styles)

To me this feels like one of the last major features missing.

Idyll components are react components are npm modules, so why can't I just npm install a third-party idyll component and use it in my project?

Here are a couple possible solutions:

1. Keywords, automatic scanning

Scenario:

1. A user publishes an idyll component to npm, they name the package: idyll-component-a. In the package.json they make sure to include the following:

"keywords": ["idyll-component"],
...
"idyll": {
    "componentName": "componentA"
}

2. Another user has a local idyll project and runs npm install --save idyll-component-a. In their index.idl file they write:

[componentA /]

3. The idyll compiler picks up on this component because of the keyword tag, and everything works as expected.

2. Explicit listing

Scenario:

1. A user publishes an idyll component to npm, they name the package: idyll-component-a. It doesn't have any special configuration or keywords.

2. Another user has a local idyll project and runs npm install --save idyll-component-a. In their package.json file they list it explicitly:

{
  "idyll": {
    "installedComponents": {
      "componentA": "idyll-component-a"
    }        
  }
}

In their index.idl file they write:

[componentA /]

and everything works as expected.

#14 covers making render-to-static complete, but I think we now need to back up and take a look at making it work at all. Currently the rendered output is mostly just a bare script tag. For load speed and SEO (does google run scripts now? Is that still a thing?) and just because it's a really nice feature that should hopefully be very easy, it seems like it would be nice to actually render to string.

Currently, the require('__IDYLL_DATA__') strategy makes it rather difficult to fake it within the main idyll process. Are those necessary or can we simply make those inputs to the functions that need them rather than requires?

Prior art: remark

Goal: Table of contents. References. Other similar things.

Proposed interface for plugins that are applied to the AST after parsing and before use:

module.exports = {
  name: 'tableOfContents',
  plugin: function (ast) {
    return {headings: ...}
  }
};

Presumably it can do the cowboy thing and even modify the AST. Like to add anchors to the headings. The data could get registered globally as:

{pluginData: {tableOfContents: {headings: ...}}}

So that you could write:

[TableOfContents/]

Which would know enough to grab the tableOfContents data and render it to a list. This would also work great for references. You could write:

As seen in [Reference author:"Smith"/]

And then write at the end of the document

[References/]

And again…………… magic?

The other component to this would be a couple visitor style helper functions to make it easier to traverse the AST. Presumably to use these you'd write idyll index.idl --plugin table-of-contents which would have to get fancy about finding and requiring it via node_modules. Seems possible though.

i is in our list of valid html tags, but it is throwing errors when used. [em] is doesn't throw errors.

Can we have ability to have spreadsheet (E.g. Google docs) and dynamic presentation like functionality (E.g. https://plot.ly/powerpoint-online/, http://formidable.com/open-source/spectacle/)

Also might be interested ing checking out:

• https://github.com/stencila/stencila
• http://orgmode.org/
• https://en.wikipedia.org/wiki/Computable_Document_Format
• http://escher-jl.org/
• https://juliaplots.github.io/
• https://www.scribus.net/
• https://www.lyx.org/

It's about time that we added a much more expansive test suite, given all the different ways in which Idyll may be used, and all the different options available. I think the first step for this is just to figure out what the structure looks like for having multiple projects in the tests folder (I believe @bclinkinbeard set this up so it should be fairly straightforward), then start testing different situations.

Specifically, one thing I think we really need are different tests for when Idyll is included in the local node_modules, vs when it is used from the command line vs when it is used programatically.

• idyll
  • command line options work
    • themes
    • component paths
    • minification and SSR
    • accepts strings or filepaths
    • watch mode
  • components
    • it finds custom components
    • it finds default components
    • it works with components from npm
    • it works with dot notation
  • server side rendering
    • variables render correctly
    • datasets render correctly
    • derived variables render correctly
• idyll-components
  • all components mount properly
  • all components mount properly outside of the browser (SSR)
  • what component specific tests??
• idyll-document -
  • expressions work on initial mount and after updates
    • objects
    • strings
    • variables
    • derived
    • datasets
  • basic AST renders properly
  • variable updating works
• compiler - tests are already pretty good there

The use of Markdown syntax is a wonderful way to leverage people's existing familiarity. However, there are some gaps in coverage; examples included below. Some should be easy to address, others perhaps less so. At minimum, it seems important that the documentation eventually note any discrepancies, which seem likely to trip up new users.

Missing Constructs

• Italic/bold styling using underscore (_italic_, __bold__).
• Unordered lists initiated by hyphen (- List item)
• Ordered lists (1. List item)
• Syntax highlighted code blocks (```json)

Inconsistent Constructs

• Using an asterisk as the first character of a line in Idyll results in an unordered list. Other Markdown flavors will produce italicized text if a line of text consists solely of *italic*, and bold text if a line consists solely of **bold**.
• Hyperlinks [text](url) collide with Idyll's component syntax. Trying to include a Markdown hyperlink can result in an Idyll compilation error, which could be jarring for some. Perhaps the use of parentheses and lack of closing bracket /] is syntactically unique here? If so, standard links might be supported without modification to component definitions.

This remove most of the client side code from this module, which I think probably makes sense. It would also be nice to be testing the component independently to make sure the dataflow, ast parsing, etc. works as expected

As discussed on gitter, from @rreusser:

It would be nice to be able to index into components, to allow for sharing scope for related components.

For example:

[Slideshow.Container]
  [Slideshow.Slide]x[/Slideshow.Slide]
  [Slideshow.Slide]x[/Slideshow.Slide]
  [Slideshow.Slide]x[/Slideshow.Slide]
[/Slideshow.Container]

In this case, the Slideshow.Container component should resolve to require('slideshow').Container

Now that the client side component is stand-alone, it should be much easier to add tests at least for that portion of things

The meta component does not seem to be working currently

Not sure why exactly - possibly because new browsersync instances are being created across builds

The current API looks like this in JS: idyll(inputPath, opts, cb) where inputPath is the absolute path to your .idl file, opts is an options object, and cb is a callback function that will receive the build artifacts once #39 lands.

The CLI is of course very similar, since it calls the JS API, looking something like this: idyll index.idl --layout centered --css styles.css --build, where inputPath is the first argument and everything else is translated to a property on the options object.

Of note:

• the CLI accepts relative paths
• there is no analog to the callback when using the CLI

Resolution of inputPath should definitely move into the JS part so inputPath can be a relative path regardless of how Idyll is being used.

Questions

• Should the CLI return a stream or something to support piping into something else? Seems low priority.
• How should we support passing in Idyll source, rather than forcing the reading of a .idl file from disk?

One approach that seems fairly simple would be to support the omission of inputPath entirely, and in that scenario look for a inputSrc (or similar) property on opts. This is similar to how Browserify allows you to specify opts.entries instead of providing files as the first argument to the browserify() function.

Thoughts @mathisonian @rreusser et al?

As mentioned in the other issue and on HN: I have been working with custom plots on canvas in React, not knowing about this project.

Maybe some of the ideas in my approach are useful, so I thought I'd share. With a bit of work, this might be turned into a blank canvas that is relatively easy to write custom drawing code for - which may be simpler than writing a custom React component.

In my use-case, the canvas is a static image that only reacts to changes in passed props, window sizes, or self-contained animations with requestAnimationFrame, but with a few changes this also work for more interactive stuff.

(Aside: in most contexts, the approach in idyll's regl example probably works better (one fixed position full-screen behind-everything-else canvas). However, I'm dealing with scientists who often print out websites to annotate them, so having separate canvases with their own renders is a more sensible solution for me)

I needed a canvas component that:

• has crisp pixels (as in, one pixel is one pixel, no zooming and smearing) regardless of browser zoom settings or display density (retina screens for example)
• automatically fills its containing div to pixel-perfection
• plays nicely with flexbox and other CSS-based sizes
• dynamically resizes when its containing div resizes

The only thing that seems to work consistently across browsers is mounting the canvas after everything else. Basically, first mount the divs that will contain the canvas, using CSS to make them the appropriate size, and then after that has happened, mount the canvas at the size matching the div:

	mountedView(view) {
		// Scaling lets us adjust the painter function for
		// high density displays and zoomed browsers.
		// Painter functions decide how to use scaling
		// on a case-by-case basis.
		if (view) {
			const pixelScale = this.props.pixelScale || 1;
			const ratio = window.devicePixelRatio || 1;
			const width = (view.clientWidth * ratio) | 0;
			const height = (view.clientHeight * ratio) | 0;
			this.setState({ view, width, height, ratio, pixelScale });
		}
	}

	render() {
		// The way canvas interacts with CSS layouting is a bit buggy
		// and inconsistent across browsers. To make it dependent on
		// the layout of the parent container, we only render it after
		// mounting view, that is: after CSS layouting is done.
		const canvas = this.state && this.state.view ? (
			<canvas
				ref={(cv) => { this.canvas = cv; }}
				width={this.state.width}
				height={this.state.height}
				style={{
					width: '100%',
					height: '100%'
				}}
			/>
		) : null;

		return (
			<div
				ref={this.mountedView}
				style={this.props.style}
			>
				{canvas}
			</div>
		);
	}

There's a bit more to it than that, but that is the basic idea.

On top of that, the component is wrapped in a higher-order component that detects when the screen resizes, replacing the old canvas.

To draw on it, I pass a paint function as a prop

	draw() {
		const { canvas, props, state } = this;
		if (canvas && props.paint) {
			let context = canvas.getContext('2d');
			// store width, height and ratio in context for paint functions
			context.width = state.width;
			context.height = state.height;
			context.pixelRatio = state.ratio;
			context.pixelScale = state.pixelScale;
			// should we clear the canvas every redraw?
			if (props.clear) {
				if (props.bgColor) {
					context.fillStyle = props.bgColor;
					context.fillRect(0, 0, context.width, context.height);
				} else {
					context.clearRect(0, 0, context.width, context.height);
				}
			}
			props.paint(context);
		}
	}

The actual painting code is simply a function takes a drawing context as an argument, and knows the sizes and scales of the pixels. Which lets me more-or-less decouple the logic from the canvas element entirely.

BTW, canvas interactivity like detecting the mouse would require a more complicated API, perhaps similar to p5js' instance mode. In fact, it's probably even easier to just wrap p5js entirely, which could make writing embedded p5js sketches relatively easy.

The files required in components.js will rarely change during normal content development workflows. I think we should be able to bundle them on their own, and mark them as external in the main bundle, and get much faster bundle times.