Question: Recommendations on how to handle dark mode about shiki HOT 42 CLOSED

Excited to try out Shiki in Astro!

You can see in the linked RFC above (and in our Discord server) there was a fair bit of pushback from our community on the idea of sending 2x the necessary HTML down to the client. For context, Astro is pretty oriented around the goal of "ship as little code as needed" so the current workaround is a bit of a non-starter for us.

I've created a theme based on the min theme that uses CSS variables instead of hardcoded colors. The idea is that this theme would be useful for someone who only wants to use the Shiki tokenizer, but then wants to handle styling themselves. This would also support light & dark mode entirely in CSS, however it would be up to that user to define those CSS variables themselves. It won't be as detailed as some of the other themes here.

Is there interest in adding this theme to Shiki for others? We'll just ship it inside of Astro otherwise, but I can see how this would make a valuable dark mode story for all Shiki users. A larger audience would also help improve the theme over time.

{
    "name": "custom",
    "type": "light",
    "colors": {
      "editor.foreground": "var(--code-foreground)",
      "editor.background": "var(--code-background)"
    },
    "tokenColors": [
      {
        "settings": {
          "foreground": "var(--code-token-default)"
        }
      },
      {
        "scope": [
          "keyword.operator.accessor",
          "meta.group.braces.round.function.arguments",
          "meta.template.expression",
          "markup.fenced_code meta.embedded.block"
        ],
        "settings": {
          "foreground": "var(--code-token-default)"
        }
      },
      {
        "scope": "emphasis",
        "settings": {
          "fontStyle": "italic"
        }
      },
      {
        "scope": ["strong", "markup.heading.markdown", "markup.bold.markdown"],
        "settings": {
          "fontStyle": "bold"
        }
      },
      {
        "scope": ["markup.italic.markdown"],
        "settings": {
          "fontStyle": "italic"
        }
      },
      {
        "scope": "meta.link.inline.markdown",
        "settings": {
          "fontStyle": "underline",
          "foreground": "var(--code-token-constant)"
        }
      },
      {
        "scope": ["string", "markup.fenced_code", "markup.inline"],
        "settings": {
          "foreground": "var(--code-token-string)"
        }
      },
      {
        "scope": ["comment", "string.quoted.docstring.multi"],
        "settings": {
          "foreground": "var(--code-token-comment)"
        }
      },
      {
        "scope": [
          "constant.numeric",
          "constant.language",
          "constant.other.placeholder",
          "constant.character.format.placeholder",
          "variable.language.this",
          "variable.other.object",
          "variable.other.class",
          "variable.other.constant",
          "meta.property-name",
          "meta.property-value",
          "support"
        ],
        "settings": {
          "foreground": "var(--code-token-constant)"
        }
      },
      {
        "scope": [
          "keyword",
          "storage.modifier",
          "storage.type",
          "storage.control.clojure",
          "entity.name.function.clojure",
          "entity.name.tag.yaml",
          "support.function.node",
          "support.type.property-name.json",
          "punctuation.separator.key-value",
          "punctuation.definition.template-expression"
        ],
        "settings": {
          "foreground": "var(--code-token-keyword)"
        }
      },
      {
        "scope": "variable.parameter.function",
        "settings": {
          "foreground": "var(--code-token-parameter)"
        }
      },
      {
        "scope": [
          "support.function",
          "entity.name.type",
          "entity.other.inherited-class",
          "meta.function-call",
          "meta.instance.constructor",
          "entity.other.attribute-name",
          "entity.name.function",
          "constant.keyword.clojure"
        ],
        "settings": {
          "foreground": "var(--code-token-function)"
        }
      },
      {
        "scope": [
          "entity.name.tag",
          "string.quoted",
          "string.regexp",
          "string.interpolated",
          "string.template",
          "string.unquoted.plain.out.yaml",
          "keyword.other.template"
        ],
        "settings": {
          "foreground": "var(--code-token-string-expression)"
        }
      },
      {
        "scope": "token.info-token",
        "settings": {
          "foreground": "var(--code-token-info)"
        }
      },
      {
        "scope": "token.warn-token",
        "settings": {
          "foreground": "var(--code-token-warn)"
        }
      },
      {
        "scope": "token.error-token",
        "settings": {
          "foreground": "var(--code-token-warn)"
        }
      },
      {
        "scope": "token.debug-token",
        "settings": {
          "foreground": "var(--code-token-debug)"
        }
      },
      {
        "scope": ["strong", "markup.heading.markdown", "markup.bold.markdown"],
        "settings": {
          "foreground": "var(--code-token-strong)"
        }
      },
      {
        "scope": [
          "punctuation.definition.arguments",
          "punctuation.definition.dict",
          "punctuation.separator",
          "meta.function-call.arguments"
        ],
        "settings": {
          "foreground": "var(--code-token-punctuation)"
        }
      },
      {
        "name": "[Custom] Markdown links",
        "scope": ["markup.underline.link", "punctuation.definition.metadata.markdown"],
        "settings": {
          "foreground": "var(--code-token-link)"
        }
      },
      {
        "name": "[Custom] Markdown list",
        "scope": ["beginning.punctuation.definition.list.markdown"],
        "settings": {
          "foreground": "var(--code-token-string)"
        }
      },
      {
        "name": "[Custom] Markdown punctuation definition brackets",
        "scope": [
          "punctuation.definition.string.begin.markdown",
          "punctuation.definition.string.end.markdown",
          "string.other.link.title.markdown",
          "string.other.link.description.markdown"
        ],
        "settings": {
          "foreground": "var(--code-token-keyword)"
        }
      }
    ]
  }

  :root {
    --code-foreground: #123456;
    --code-background: #ABCDEF;
    --code-token-default: #123456;
    --code-token-constant: #123456;
    --code-token-string: #123456;
    --code-token-comment: #123456;
    --code-token-keyword: #123456;
    --code-token-parameter: #123456;
    --code-token-function: #123456;
    --code-token-string-expression: #123456;
    --code-token-info: #123456;
    --code-token-warn: #123456;
    --code-token-warn: #123456;
    --code-token-debug: #123456;
    --code-token-strong: #123456;
    --code-token-punctuation: #123456;
    --code-token-link: #123456;
  }
@media (prefers-color-scheme: dark) {

:root {

--code-foreground: #ABCDEF;

--code-background: #123456;

--code-token-default: #ABCDEF;

--code-token-constant: #ABCDEF;

--code-token-string: #ABCDEF;

--code-token-comment: #ABCDEF;

--code-token-keyword: #ABCDEF;

--code-token-parameter: #ABCDEF;

--code-token-function: #ABCDEF;

--code-token-string-expression: #ABCDEF;

--code-token-info: #ABCDEF;

--code-token-warn: #ABCDEF;

--code-token-warn: #ABCDEF;

--code-token-debug: #ABCDEF;

--code-token-strong: #ABCDEF;

--code-token-punctuation: #ABCDEF;

--code-token-link: #ABCDEF;

}

}

from shiki.

I think I've found a pretty good solution. I'm editing rehype-shiki to support a darkTheme option.
That option will make the rehype plugin generate 2 separate code blocks using shiki, one for light mode and one for dark mode with a class that reflects that (ex: syntax-dark).

Then you just add the following css to hide it based on a media query. You could also do class based dark theming, it's up to the user.

.syntax-dark {
  display: none;
}

@media (prefers-color-scheme: dark) {
  .syntax-light {
    display: none;
  }

  .syntax-dark {
    display: block;
  }
}

from shiki.

Since nobody mentioned this already: using inline html style attributes is somewhat undesirable, since it requires setting style-src: 'unsafe-inline' Content Security Policy source.

As far as I understand, the two approaches proposed in this thread rely on HTML classes and CSS variables, respectively. Since they both require the use of a separate CSS stylesheet, I'd argue that using HTML classes is better since it doesn't have the aforementioned CSP issue.

Am I missing something? Please let me know!

Thanks for your awesome work on Shiki :D

from shiki.

I think maybe I found the most balanced way of doing so. You can learn more about the details at antfu/shikiji#5

Basically, it generated inline CSS variables like:

<span style="color:#1976D2;--shiki-dark:#D8DEE9">console</span>

That can be overridden with a short CSS snippet:

@media (prefers-color-scheme: dark) {
  .shiki span {
    color: var(--shiki-dark) !important;
  }
}

live preview

This way we will have a perfectly working light mode output, while being able to switch to dark mode conditionally, without duplicating the content.

Would be happy to port it back to Shiki if you think this approach makes sense.

from shiki.

Yep - that's a reasonable workaround to not have this problem. That's totally doable for people, but I'm not planning on making that design compromise

from shiki.

Hi, upon closer inspection I found out that we can get token.explanation when using highlighter.codeToThemedTokens. I think we can implement dark/light mode support purely in user space without modification in shiki. CMIIW

I'm not really familiar with how vscode textmate parses grammar and the resulting type so here's my assumption. We can find what scope matches inside token.explanation[0].scopes

type Scope = {
  scopeName: string;
  themeMatches: Array<ThemeMatch>
}

const token = highlighter.codeToThemedTokens(code, lang)

const matchingScope: Scope = token.explanation[0].scopes.find(scope => {
  return scope.themeMatches.length > 0
});

If empty we can use --fallback-color, or other convention

if (!matchingScope) {
  return "--fallback-color"
}

Even though technically we can generate CSS variable using scope.scopeName, I think it's better to use themeMatch to reduce the number of generated CSS variable.

I found that 0 index is the one that's being used if there's more than 1 themeMatches

type Theme = {
  name: string;
  scope: Array<string>;
}

const matchingTheme: Theme = matchingScope.themeMatches[0]

// we can then generate CSS variable from either name or scope
// remove whitespace, convert invalid character, etc
const cssVar = convertToCSSVariable(matchingTheme.name)

// or use scope
// join all scope, compute hash
const cssVar = convertToCSSVariable(matchingTheme.scope);

return cssVar

Then we can generate HTML tag and use CSS variable fallback.

const cssVar = generateCSSVariable(token);
const html += `<span style="color: var(${cssVar}, ${token.color})">${token.content}</span>`

from shiki.

Currently renderers have no access to theme data. I plan to do an API change:

ThemedTokenizer: theme + lang => IThemedTokens[][]
Renderer: IThemedTokens[][] + theme => HTML / SVG / HTML+CSS

This way a renderer has all needed info to output HTML+CSS (with meaningful class names).
We could have HTMLRenderer, SVGRenderer and HTMLCSSRenderer, etc.

HTMLCSSRenderer could generate a CSS mapping such as:

.support-function {
  color: #fff;
}

And give each token its matching scope as class.

Would this work for you?

from shiki.

Yes, I think we should ship this theme in Shiki - perhaps simply called css-variables.

The CSS will need to go into shiki docs somewhere. I'm not sure where @octref feels WRT a user-facing docs site (e.g. like the shiki-twoslash one, but for now I think this can be put in the docs folder of this repo.

from shiki.

I think I've done it, not only dark mode but also multiple themes. Inspired by rehype-pretty-code and Fatih Kalifa’s blog post.

from shiki.

We are currently inverting and colour hue shifting on the TypeScript website, and I think that's probably enough for us - microsoft/TypeScript-Website#1536

from shiki.

Hah, nice idea

I keep feeling like all of our answers live outside of 'shiki' and in whatever shiki -> x rendering tool you're using

from shiki.

I built a playground with dark mode. You can give it a try at https://shiki-play.matsu.io. Source code is at https://github.com/shikijs/shiki-playground

Here are my thoughts on different approaches:

• Outputting two HTML blocks is actually the simplest way as @hipstersmoothie suggested.
• Most themes have a corresponding dark/light theme. I think it's easier for 99% of users to just use the provided counterpart theme, instead of messing with CSS variables and coming up with other colors.

Outputting semantic HTML (meaningful class names) with CSS

Given a source, a grammar and a theme, you can get the matching scope that makes a token a specific color. For example, here the token is #9ecbff because it's string.

So we need a renderer that does this:

• For each token with a matching scope foo.bar that determines its color
• Give it a class foo-bar (convert all scopes to classes would be too verbose)
• Output CSS, matching each scope to a single color.

Note to myself - on the conversion:

If a theme colorizes string and string.quoted differently, I should generate classes like string and string-quoted, but never string quoted. The CSS should use selectors like .string-quoted, not .string.quoted. Textmate themes have different order and specificity than CSS, so I'd want to avoid getting tangled in the conversion.

API wise, something like this:

interface SemanticHTMLCSSRenderer {
  generateSemanticHTML(code: string, lang: Lang): string
  generateCSSFromTheme(theme: Theme): string
}

from shiki.

I'm trying out the css-variables theme and I see several issues with this approach:

No clear API, functionality is triggered by a theme name ('css-variables')
Limited to only 12 colors
No control over color names. I'm forced to use colors like shiki-token-punctuation for something semantically different to make use of all 12 colors.

Instead of the hacky and limited css-variables theme, why not just enable the user to supply their own color remap here:

Seems to me that by doing this it would unlock full theming via CSS Variables and resolve all dark-mode issues, including my problems listed above. Users would be able to swap out any colors they'd like for a CSS variable instead.

from shiki.

As an innocent user, I thought I could do this on my-theme.json

{
  "tokenColors": [
    {
      "scope": ["comment"],
      "settings": {
        "foreground": "var(--color-muted)",
        "fontStyle": "italic"
      }
    },
    {
      "scope": ["constant"],
      "settings": {
        "foreground": "var(--color-pine)"
      }
    },
    {
      "scope": [
        "constant.numeric",
        "constant.language",
        "constant.charcter.escape"
      ],
      "settings": {
        "foreground": "var(--color-rose)"
      }
    },
  ]
}

So when Shiki processing, it will inject that value to token style <span style="color: var(--color-muted)">variable</span> 😓

from shiki.

@pveyes solved this by switching the output to use css variables:
https://fatihkalifa.com/typescript-twoslash

from shiki.

https://github.com/anotherglitchinthematrix/monochrome could be a good test.

I could generate 10 different variations and make a slider demo.

from shiki.

@orta's approach is an okay solution but I don't think it works for every theme. Works well on the TS website though. My ideal API would be the following:

const highlighter = await shiki.getHighlighter({
  theme: 'github-light',
  darkTheme: 'github-dark',
  langs: [...BUNDLED_LANGUAGES, ...langs]
})

As a user I want to be able to pick the theme that best fits my website for light/dark mode.

from shiki.

Hey @FredKSchott, great work! While I'm late to review the PR, since we are still pre 1.0 I'd like to get the API right. IMO hiding this functionality behind a special theme is not as good as having an explicit API.

I think what you have is a good start. What I also want to see are:

• A standalone API in addition to codeToHtml. Adding option to codeToHtml is not good since theme is uesless in that case. Not sure what's a good name. Maybe codeToParameterizedHtml or codeToHtmlWithCssVariables?
• A way for users to get tokens with groups, similar to codeToThemedTokens but outputting token group instead of color

from shiki.

@orta I think shikiji handles it correctly. For tokens without colors, inherit will be rendered:

<code>
  <span class="line">
    <span style="color:#1976D2;--shiki-dark:#D8DEE9">I</span>
    <span style="color:#6F42C1;--shiki-dark:inherit"><</span>
    <span style="color:#6F42C1;--shiki-dark:#88C0D0">3</span>
  </span>
</code>

Also it would be cases some theme output larger token, like:

<code>
  <span class="line">
    <span style="color:#1976D2;">I < 3</span>
  </span>
</code>

In that case, shikiji will break those tokens into the common set of two themes.

So far I think it covers most of the edge cases.

from shiki.

Does your first idea mean using a single style that looks good both in light and dark mode?

Also, I'd like to help implementing this, is it something lower in complexity?

from shiki.

I think of it as being:

• Shiki returns "#ffeeaa" for a token
• My app makes a sha of it: "#ffeeaa" -> "asd12"
• My app sets the class of the token to be "asd12"
• We keep track of all colors which were assigned and spit out a single css file for all those colors
• That CSS file can use prefers-color-scheme: dark to manually set colors for the dark theme

from shiki.

It is not an elegant solution but code containers could be dark even in light theme. Check Dan Abramov's overreacted.io (https://overreacted.io/goodbye-clean-code/, for example). It looks good for me to read, both in light and dark theme. That seems the easiest possible solution. At least could bring some fresh to the eyes at night until the ideal solution could come up.

from shiki.

Yes, I understand. I'm not proposing defining as design choice, but it is a better solution than the current one. I believe devs feel more comfortable reading dark code during the day than light code at night.

from shiki.

I don't think it is practical enough to support CSS properties-backed dark and light theme, given the size of grammar and sheer number of color tokens. What I'd suggest is to customize some color tokens (like background-color and foreground-color) through CSS properties and toggle them through a feature query. You'll have to find a theme which has got tokens that work nice enough in light and dark mode (which is annoyingly hard). I was able to do that with Prism (an example here), given its very limited set of tokens but gave up on Shiki.

from shiki.

So, dark mode alone wasn't quite sufficient for me. I wanted the color switching with themes that @pveyes built, but without being locked into a specific theme / set of themes. I'd love to have readable class names, but after messing around with scopes as suggested above, was unable to come up with a good solution and have settled for just generating hl-1, hl-2...

Leaving this here as it might be useful to someone - https://gist.github.com/Gerrit0/275a4b8ffee4fa133fd075f5edeb3cda. TypeDoc will use a similar approach in the rebuilt themes.

from shiki.

I need to read up dark mode a little bit first, so this won't cut it for 0.2.0.

from shiki.

That would be awesome. For the CSS mapping - how static would these classes be across different themes? For my use case, I want to be able to generate links for some identifiers. Right now, to get highlighting I'm generating a string that looks like TypeScript, getting the tokens from that, and matching the text of tokens against the identifiers I expect. It would be neat to avoid the extra pass to Shiki. https://github.com/TypeStrong/typedoc/blob/library-mode/src/lib/renderer/default-templates.tsx#L539-L591

from shiki.

how static would these classes be across different themes

Each token can have multiple scopes, and it's up to the theme to decide which one it would colorize.
In the API, we could also allow multiple themes, and write the matching scopeNames by each theme into each token.
So a token could look like <span class="a b">, and the output CSS would include:

.github-dark {
  .a { }
}
.github-light {
  .a { }
}

Although that would take a larger refactor. ThemedTokenizer really becomes Tokenizer without theming, and there's a ThemeMatcher (in renderer) that assigns color to each token. That would also mean we need to fork vscode-textmate.

I want to be able to generate links for some identifiers.

That's a separate feature, isn't it?

from shiki.

I've been doing something like this on my new work-in-progress blog using gatsby-remark-vscode, which is kinda similar, but comes with it's own set of issues - so i've been looking into using shiki instead.

What is done there - and what works really well, is @orta's second idea: Matching two seperate themes together. One is applied in light mode, the other in dark mode.

from shiki.

In the API, we could also allow multiple themes, and write the matching scopeNames by each theme into each token.

This sounds like what I'm looking for.

That's a separate feature, isn't it?

Kind of - I already have it working by using the tokens myself, I'm just looking for a better way of choosing the CSS classes + CSS generation for each token. Really it's just a reason why I will still use tokens, not the codeToHtml, even if codeToHtml supports light/dark mode.

from shiki.

My gut says the simplest API is we allow theme to be either a string or string[], then give each codeblock a css class with the theme name in it. Then people can use the CSS @hipstersmoothie mentioned.

from shiki.

I also joined the club of "render many times" with remark-shiki-typescript microsoft/TypeScript-Website#1831

It can render multiple copies of the code, and then it's on the user to write the CSS which removes the specific theme

Light / Dark Modes

If you pass more than one theme into themes then a codeblock will render for each theme into >
your HTML. This means that you can use CSS display: none on the one which shouldn't be seen.

const jsx = await mdx(content, {
  filepath: "file/path/file.mdx",
  remarkPlugins: [[remarkShikiTwoslash, { themes: ["dark-plus", "light-plus"] }]],
})

@media (prefers-color-scheme: light) {
  .shiki.dark-plus {
    display: none;
  }
}

@media (prefers-color-scheme: dark) {
  .shiki.light-plus {
    display: none;
  }
}

from shiki.

For the record, I made markdown-it-shiki using the similar approach of rendering twice:

https://github.com/antfu/markdown-it-shiki#dark-mode

from shiki.

Great, PR added: #212

from shiki.

#212 is great! Is is possible to get required CSS for specific theme?

from shiki.

@Enter-tainer Take a look at docs here: https://github.com/shikijs/shiki/blob/main/docs/themes.md#theming-with-css-variables

from shiki.

I'm a bit wary about having this as a separate API codepath, interested to see how it turns out. Having a different API means that higher level abstractions like remark-shiki would need to be aware of this and expose new config vars for what is essentially a custom theme output

from shiki.

@orta That's a valid concern. However I'm thinking about additional use cases like this. So far what we have are:

code + theme + grammar = themedTokens
themedTokens + renderer = html/svg/etc...

This assumes user want the tokens already colorized, and the only way for them to tweak the coloring pipeline is to write a theme. What we could also have is:

code + grammar = groupedTokens

Essentially what highlight.js does. People who want to write their own colorizer can use this API. It's also not that much – just mapping functions/keywords etc to a color and you are done.

I also feel codeToHtml should output directly embeddable HTML, not "given an option and then you need to fill in some CSS" kind of HTML.

from shiki.

Also the COLOR_REPLACEMENT map even contains errors, there is no #3 entry...

from shiki.

Yeah, I think this answer is the right one - and should probably be canonically the answer

Something I'm not certain about (from a high level) is what might happen if the two themes have different support for coloring source attributes, e.g.

I < 3

might generate something like

  <code>
    <span class="line">
      <span style="color:#1976D2;--shiki-dark:#D8DEE9">I</span>
      <span style="color:#6F42C1;--shiki-dark:#ECEFF4"><</span>
      <span style="color:#6F42C1;--shiki-dark:#88C0D0">3</span>
    </span>
 </code>

Is it possible that there could be inconsistencies in the theme in terms of the tokens <=> colors they support, making something like:

  <code>
    <span class="line">
      <span style="color:#1976D2;--shiki-dark:#D8DEE9">I</span>
      <span style="color:#6F42C1;"><</span>
      <span style="color:#6F42C1;--shiki-dark:#88C0D0">3</span>
    </span>
 </code>

possibly happen?

from shiki.

Yes, it's absolutely possible for that to happen. TypeDoc's implementation of this uses a single class for each color, and overrides what the class does, so I've seen this quite a lot when debugging, even when highlighting with "sister" themes like Light Plus/Dark Plus

from shiki.

👍🏻

from shiki.