Proposal for mdxast about ideas HOT 17 CLOSED

Hey @johnotander thanks for getting started on this. I've been thinking about this a bit for the last few days and wanted to leave some thoughts.

1. There's quite a few cases where having some sort of nesting is nice. You could imagine having a custom blog post with something like:

<Post title="my cool blog post">

  yada yada **this is cool**.

  <Sidenote>
    This is a sidenote or sidebar or just another section you want to fill in. 

    It would be nice to be just markdown.
  </Sidenote>

</Post>

This is definitely more complicated than the original AST, but I think the rule could be something like if there's only one inner markdown node, unwrap the <p> tag surrounding it.

It's also not a showstopper if the compiler gets it wrong because you can override styles with something like this (e.g. .post > p { display: inline }).

2. I don't think frontmatter is necessary

As demonstrated above, I think it could just be a custom tag if markdown children are possible.

3. I don't think the imports are necessary either

As a matter of simplifying the AST a bit, I think you can implement all these resolvers programmatically and it'd be a bit more generic, since there'd be no difference between HTML elements produced by markdown and these custom elements.

You have that with the components: { ... } in https://github.com/c8r/markdown#options. I'm also thinking of something like this: https://github.com/developit/preact-markup#custom-elements-via-components or have one resolve function that all element nodes pass through.

4. Scoped <style> support

It'd be cool to have scoped style in there, so you can customize specific blog posts, something like:

yada yada **this is cool**.

<style>
  p { background: green; }
</style>

from ideas.

As to supporting a full syntax for jsx, it was my next step anyway for remark-jsx

❤️

When a component has children, the user most probably wants to interleave markdown and "jsx". It makes sense for example to write this Click here

We should definitely consult some potential users and get their thoughts. I personally think it'd be a reasonable constraint to not interleave within any block of JSX, especially considering that the same can be achieved with:

<BlinkyButton><b>Click here</b></BlinkyButton>

I worry that interleaving isn't deterministic. Consider the following scenario:

<BlinkyButton>
  Click here

  <Icon name="arrow" />
</BlinkyButton>

Does this result in

<button class="blinky">Click here <img src="/icons/arrow.svg" /></button>

or

<button class="blinky">
  <p>Click here</p>
  <p><img src="/icons/arrow.svg" /></p>
</button>

I'm not sure there's really an easy way to determine what should and should not be parsed as markdown. In JSX/HTML repeated whitespace isn't syntactically significant. Though it is in markdown.

I initially thought we could work around that by determining whether an element is phrasing content or not, but with custom elements that isn't possible to know ahead of time. MyComponent could be a div, paragraph, kbd, or all three.

Perhaps a syntax to "reopen" md parsing?

Another thing we can consider is a <markdown> tag similarly to what mdxc uses:

<div>
  # Markdown {"doesn't".toUpperCase()} work here.
  <markdown>
    *But does work here*.
  </markdown>
</div>

we should discuss this further and find a middle ground we can all stand on together

Absolutely!

from ideas.

I began spiking out a quick potential proposal based on mdast: mdx-js/mdx#49

from ideas.

/CC @fazouane-marouane, you’ve been working on something like this right? https://github.com/fazouane-marouane/remark-jsx

from ideas.

Hi @johnotander! 👋 Nice to meet you :)

Replies

Unsurprisingly, the lion share of the work will exist on the parsing side.

probably 😉

Interleaving

How *should* that be handled though? 🤔

Code blocks

You can change that by passing new tags in.

Aside

Re: https://github.com/c8r/markdown/blob/master/MDXAST_SPECIFICATION.md

Maybe add a section on how this differs / corresponds with mdast/hast?

Future

Sounds like a plan! How do you want to proceed? For one, you can post this in the relevant Gitter communities maybe?!

from ideas.

Hi @johnotander! 👋 Nice to meet you :)

You too! And thanks for all your hard work on these unist projects (and others). I've learned a lot from them and used them on numerous occasions. Brilliant work ❤️.

Interleaving

How should that be handled though? 🤔

I think that's TBD at this point. But, I'm personally leaning towards not supporting interleaving for a few reasons.

1. It will be complex and seemingly difficult to do predictably, and likely drastic require syntax changes similar to other experiments like reactdown
2. It can be supported by transclusion via the import syntax

Maybe add a section on how this differs / corresponds with mdast/hast?

Done

Sounds like a plan! How do you want to proceed? For one, you can post this in the relevant Gitter communities maybe?!

I will! Before I do too much I'd definitely like to hear thoughts from @fazouane-marouane. remark-jsx is definitely the closest implementation to what I'm hoping to be achieved. In fact it's exactly what I'm hoping for but with a more formalized and battle tested parser which might likely fit well directly in that project.

from ideas.

Hello there

I'm glad the project found such interest. Yes, I'm all for formalisation and for a battle-tested parser.

I think that users of "react in markdown" fall into two camps. Those who want nesting/interleaving and those who want exactly the jsx experience. I'd be interested in having both visions aligned in the most intuitive way. Discussions is definitely the way to go.

Maybe we can:

1. Have the mdx spec, so those specific code blocks won't need to worry about nesting or markdown and can be plain jsx code.
2. And have the possibility to use naked react components with a more html-ish syntax so it can support nesting like any other html tag in plain markdown. In the packages I worked on, an attribute's value can be on multiple lines. An attribute can have a html namespace "js:" prepended to its name to tell that it is a special and that it needs compiling/interpreting. This is tested but not documented for now.

What are your thoughts on this?

from ideas.

Hello there

👋 Happy to hear you chiming in @fazouane-marouane <3

You're exactly right about the usage of React in Markdown falling into two camps. The MDX proposal definitely falls into the "pure jsx" camp and has different goals than something more interleaving. There's a good chance these projects might need to exist independently. Handling pure jsx and interleaving in the same parser most likely proves to be too difficult.

I also don't want you to feel like that's something you and your project need to be "shoehorned" into. It's a bit of a departure from your current (amazing) implementation. Though, it's where I would like to arrive for the MDX/MDXAST (if it ends up being possible).

I'd love to be able to completely nuke the notion of .jsx code blocks. I only added those to the spec as a potential stop gap in case parsing complex MDX was too difficult to implement swiftly.

Super stoked that you have implementations for all the mini bugs I've already raised on remark-jsx. This will be immensely useful on building out our c8r/markdown library and make it prod ready.

In the long term, though, I'd love to have a pure JSX implementation that can essentially declare anything that's possible in JSX with no troubles. This will require it's own implementation layer (loaders, transpilers, etc.).

I understand that this might not make sense to all folks hoping to use React in Markdown, but I think that MDX is a solid superset of Markdown that allows JSX to be introduced incrementally to a project while still allowing less technical folks to contribute to content (without new syntax).

tl;dr

The initial question this initially boils down to is whether remark-jsx wants to support MDX or if we want to collaborate on a fork like remark-mdx. There are two potential user bases to target, both of which are a bit different, so that's ultimately up to you. I think it's about what makes you happy <3.

from ideas.

As to supporting a full syntax for jsx, it was my next step anyway for remark-jsx 😄. Yes I'd like to support it as long as the parser still supports nesting.

When a component has children, the user most probably wants to interleave markdown and "jsx". It makes sense for example to write this <BlinkyButton>*Click here*</BlinkyButton> and expect bold text inside the button. We can keep the jsx block idea to disable nesting if that's what the user wanted.

So, we should discuss this further and find a middle ground we can all stand on together. I'm all for more jsx in remark-jsx!

from ideas.

FWIW, regarding interleaving, from an authoring experience, I'd not expect markdown syntax within JSX code blocks to render as markdown, just like it doesn't work within an HTML block in regular markdown and just like I wouldn't expect JSX expressions to work in markdown blocks.

// example
<div>**I wouldn't expect this to work in HTML**</div>

# And I wouldn't expect {props.description} to work in markdown

from ideas.

@jxnblk @johnotander Yes actually you do make a point. So nesting after all is foreign to html in markdown in the first place 🤔 .

The <markdown> special tag might be our middle ground. I like it 😊 It will solve most nesting issues, actually.

In that case, I have some comments/suggestions for the mdx spec.

1. I think that imports should be optional and be declared at the top of the file.
2. Having a global props object to pass may be a good thing.
3. We should think of what means this in the context of a mdx file. Is it the global object? null? the component that represents the page?
4. The spec should also specify that non-whitelisted component, if a whitelist was provided to the parser, will not make it to the AST and will be passed as a raw token/pure text.
5. Maybe we can also specify what a safe mode for mdx will be. This is important for when you have no control over the content creation (like in a comment section of a forum or sth).

(2), (4) and (5) may seem implementation-specific but I'd rather discuss it sooner.

What are your thoughts on this?

from ideas.

The <markdown> special tag might be our middle ground. I like it 😊 It will solve most nesting issues, actually.

That definitely works for me : )

I think that imports should be optional and be declared at the top of the file.

Sgtm 👍, I'll explicitly define that in the MDX spec.

Having a global props object to pass may be a good thing.

Yeah, I've been thinking about this a bit myself as well. Just not sure of what the syntax might be as I would personally prefer to avoid frontmatter for MDX. I was thinking a potential implementation would be from the wrapping element.

file.mdx

import { Hero } from './ui'

<Hero cta={props.cta} />

* other
* stuff

The app/page/etc

App.js

import Doc from './file.mdx'

export default () =>
  <Doc cta='Neat!' />

We should think of what means this in the context of a mdx file. Is it the global object? null? the component that represents the page?

Interesting, yeah I'm definitely not sure what that might be off the top of my head. Will need to do some thinking.

The spec should also specify that non-whitelisted component, if a whitelist was provided to the parser, will not make it to the AST and will be passed as a raw token/pure text.

Agreed, great idea, I'll add that in. Will also add the caveat that imports will add to the whitelist.

Maybe we can also specify what a safe mode for mdx will be. This is important for when you have no control over the content creation (like in a comment section of a forum or sth)

Yeah for sure, prolly separate from the MDXAST itself, but we should have a story there. Hopefully even an implementation : ). Though I'm not sure offhand how we will achieve that quite yet. More thinking to do!

from ideas.

Yeah, I've been thinking about this a bit myself as well. Just not sure of what the syntax might be as I would personally prefer to avoid frontmatter for MDX. I was thinking a potential implementation would be from the wrapping element.

That maybe something to discuss with Gatby/Phenomic users, for example. They seem to be very fond of frontmatter in markdown.

Agreed, great idea, I'll add that in. Will also add the caveat that imports will add to the whitelist.

Oh yes of course! People who want whitelists should maybe whitelist the component's provenance (the path in the imports).

We should think of what means this in the context of a mdx file. Is it the global object? null? the component that represents the page?

Interesting, yeah I'm definitely not sure what that might be off the top of my head. Will need to do some thinking.

I was thinking about the global props and the meaning of this. If this denotes the page's component, then we can just use this.props which may IMO seem natural.

I'm glad we agree on the most/all parts 👍 I'll read the spec in the next few days and get back with my feedback if any, and try to prototype the parser on the weekend 😄

from ideas.

That maybe something to discuss with Gatby/Phenomic users, for example. They seem to be very fond of frontmatter in markdown.

Yeah, this is true. Let's add it (optionally similar to imports), after all this is AST specific so we may as well add it in to support more potential use cases.

I was thinking about the global props and the meaning of this. If this denotes the page's component, then we can just use this.props which may IMO seem natural.

Yeah, fair point and something we should ponder. I think that this should either be in the context of the document itself (as a component) or null with props in scope like a functional component. But we can ponder that for the moment.

I'm glad we agree on the most/all parts 👍

Yeah, I think our headspace is in the same place so super happy to collab : )

I'll read the spec in the next few days and get back with my feedback if any, and try to prototype the parser on the weekend 😄

Please don't hesitate to provide any feedback ❤️

from ideas.

Played around a little bit with generating the MDXAST by parsing code blocks with babylon. Got it to work for a trivial use case, but then ran into issues with the special <markdown> element because babylon parses all contents of that element as well.

mdx-js/mdx#52

from ideas.

@johno Is there still discussion needed around this issue, or can we close it? And: should we link from unist to MDXAST?

from ideas.

@matthewmueller Thanks for your thoughts on this! Definitely helpful for folks to weight in.

1. There's quite a few cases where having some sort of nesting is nice.

We opted to not support nesting/interleaving. Primary motivations for this decision is based on the non-determinism in parsing and because one can use components once they open up a JSX block. Perhaps it's something we will revisit in the future.

2. I don't think frontmatter is necessary

Yeah, we've gone ahead and removed that from the spec!

3. I don't think the imports are necessary either

We believe the imports are useful for the context when an MDX file is the entrypoint for a page, like a blogpost. This allows for custom components, like a video embed element or an interactive graph to be imported only on the posts that need it without having to wrap the MDX as it's own import and render it with new scope.

4. Scoped <style> support

That's a really good idea. That's currently supported via JSX (using a template string and dangerously setting HTML), but that's definitely something we can bring in to be more native.

@wooorm I think we can close it now, and yeah we could def link to the MDXAST spec from unist.

from ideas.