zikaari / monaco-editor-textmate Goto Github PK

Steps to reproduce

1. Create an instance of monaco-editor
2. Make the language as html.
3. Go to facebook.com and get their HTML source.
4. Add the HTML source in the editor.
5. The editor is stuck and not usable.

First, thank you so much for the work on this project! I haven't been able to get any other solutions working for TextMate grammars besides this library.

I'm running into a weird issue that I was able to trace to the 0.30.0 of monaco-editor. Moving between both of the versions will produce the following inconsistencies when highlighting identifiers in certain cases:

Before 0.29.1

After 0.30.0

I'm not entirely sure how all of the highlighting is working and wish I could provide more information. It's not clear either why things started breaking between those two versions as the changes seem unrelated.

This repo showcases the issue in a previous commit here. I've fixed the version (managed by @monaco-editor/loader) to get around the issue for now.

In this project readme you state it is recommended to use this webpack plugin to disable the included language things.

you're advised to use Webpack with monaco-editor-webpack-plugin which allows you to control which of "built-in" languages should monaco-editor use/bundle, leaving the rest.

But what if I'm not using webpack? Is there a way to do this using the monaco api? Even if its messy/ requires some async grossness?

Hello,

In monaco-editor, each language has a configuration (type of comments, foldind, etc.). So if you are in html mode, the "toggle comment" key wraps the code in <!-- -->, if you are in css, the comments become /* */, etc.

This behavior stops working correctly if we have languages with embedded modes (ex. html with <style> or <script>). In that case, all the comments become <!-- -->, no matter if you are in one of the tags mentioned.

Is there a call you can make on the fly (during tokenization) to change the configuration, based on the language type of the current line that we are tokenizing?

Hello,
I am trying to use monaco-editor-textmate with @monaco-editor/react, monaco-vscode-textmate-theme-converter, and monaco-textmate to support vscode themes in monaco editor however I am struggling to understand the code sample on the usage section. Would it be possible to create a full working example or possible add some more documentation for someone less familiar with these specific topics? I have a small demo project I created for the sole purpose of getting this working here: https://github.com/ChristopherHButler/vscode-themes-in-monaco It is not working yet but pull requests are very welcome!
Thanks for your time and consideration :)

Would it be possible to update to the newest monaco-editor version?
I always receive an npm-warning in my project because I use the newest version and this library needs monaco 0.14.0.

First I would like to thank you for developing this plugin. This is really helping me out to apply better syntax highlight in Monaco editor.

I had an issue where syntax highlight was not applying when I was switching the language of the same editor using monaco.editor.setModelLanguage option.

My grammar load file looks like below, I am setting grammer for required languages

import { loadWASM } from 'onigasm';
import { wireTmGrammars } from 'monaco-editor-textmate';
import { LanguageRegistry } from 'monaco-textmate-languages';
import * as monaco from 'monaco-editor';

import theme from './themes/DarkPlus.json';
monaco.editor.defineTheme('darkplus', theme);

(async () => {
  // See https://www.npmjs.com/package/onigasm#light-it-up
  await loadWASM(`/monaco/onigasm.wasm`);

  const registry = new LanguageRegistry({
    basePath: '/',
    textFetcher: async (uri) => {
      return (await fetch(uri)).text();
    }
  });

  // map of monaco "language id's" to TextMate scopeNames
  const grammars = new Map();

  grammars.set('css', 'source.css');
  grammars.set('less', 'source.less');
  grammars.set('scss', 'source.scss');

  grammars.set('html', 'text.html.basic');
  grammars.set('pug', 'text.pug');

  grammars.set('typescript', 'source.ts');
  grammars.set('javascript', 'source.js');

  await wireTmGrammars(monaco, registry, grammars);
})();

In Monaco editor component, I am instantiating like below (default language:css)

    this.editor = monaco.editor.create(current, {
      ...DEFAULT_EDITOR_PROPS,
      value: value || '',
      readOnly: readonly,
      language
    });

It works for the default language properly

However when I switch the language of same editor using

    let model = this.editor.getModel();
    let currentLang = model._languageIdentifier.language;

    if (currentLang !== language) {
      monaco.editor.setModelLanguage(model, language);
    }

It turns off the syntax highlight

I thought earlier that it could be scss grammer loading issue, however it was not because instantiating editor with default "scss" language does load properly

Please let me know if I am missing anything.

Hi! Thank you for your wonderful plugin.
The plugin brokes auto-closing brackets and quotes.

Editor options enabled:

autoClosingBrackets: true
autoClosingQuotes: true,
autoClosingOvertype: true,

Working Example

Do you plan supporting EncodedTokensProvider?

According to document, i excluded languages from MonacoEditorWebpackPlugin.
But looks like TMGrammers are not loaded for that languages and i am getting following error

Uncaught (in promise) Error: Cannot set tokens provider for unknown language javascript
    at Object.setTokensProvider (standaloneLanguages.js:231)
    at _callee$ (index.js:77)
    at tryCatch (runtime.js:45)
    at Generator.invoke [as _invoke] (runtime.js:264)
    at Generator.prototype.<computed> [as next] (runtime.js:98)
    at asyncGeneratorStep (asyncToGenerator.js:3)
    at _next (asyncToGenerator.js:25)

Code

async function liftOff(monaco) {
  const registry = new Registry({
    getGrammarDefinition: async scopeName => {
      console.log("scopeName :", scopeName);
      return {
        format: "json",
        content: await (await fetch(`/grammers/${scopeName}.tmLanguage.json`)).text(),
      };
    },
  });

  // map of monaco "language id's" to TextMate scopeNames
  const grammars = new Map();
  grammars.set("python", "source.python");
  grammars.set("javascript", "source.js");

  await wireTmGrammars(monaco, registry, grammars);
}

How i am changing language ?

monaco.editor.setModelLanguage(editorRef.current.getModel(), "javascript");

In the tokenizer tokenize method, you convert an array of scopes to just the last scope.

In our case, this is good enough for 99% of the cases, but it fails on punctuation.

Would it be possible to add a callback on this last bit of code, where you return the result of the tokenization, so that we could make our own conversion ? Maybe just use a default callback that does what you are currently doing, but leave use the choice to change it :)

Maybe just add a callback on this last piece:

                        return {
                            endState: new TokenizerState(res.ruleStack),
                            tokens: res.tokens.map(token => ({
                                ...token,
                                // TODO: At the moment, monaco-editor doesn't seem to accept array of scopes
                                scopes: token.scopes[token.scopes.length - 1]
                            })),
                        }

Like:

                        return callback({
                            endState: new TokenizerState(res.ruleStack),
                            tokens: res.tokens
                        })

This would allow us to return the last scope except punctuation, for instance. Or maybe format the scopes in the way monaco is expecting it (as it seems that vscode supports scope arrays, using the Developer: Inspect TM Scopes command). I have no idea how it works yet though.

Hey there!

I find myself struggling with replacing the terrible JavaScript syntax highlighting that comes bundled with Monaco.

My starting point was this:
https://github.com/microsoft/monaco-editor-samples/tree/master/browser-esm-webpack-monaco-plugin

I only included "typescript" and "css" as languages in the build. Including or excluding typescript doesn't make a difference.
I wired up onigasm, monaco-textmate, and monaco-editor-textmate as per your example.

The JavaScript textmate grammar I got from here:
https://github.com/textmate/javascript.tmbundle/tree/master/Syntaxes

And this was the result:

This ... can't be right... right?

I'd appreciate any pointers as to where I messed up.

Many thanks in advance!

Here's the sourcecode:

package.json

{
	"name": "monaco",
	"version": "1.0.0",
	"description": "",
	"main": "index.js",
	"private": true,
	"scripts": {
		"test": "echo \"Error: no test specified\" && exit 1",
		"build": "webpack --progress"
	},
	"keywords": [],
	"author": "",
	"devDependencies": {
		"css-loader": "^3.4.0",
		"file-loader": "^5.0.2",
		"monaco-editor": "^0.19.0",
		"monaco-editor-webpack-plugin": "^1.8.0",
		"style-loader": "^1.1.1",
		"webpack": "^4.41.4",
		"webpack-cli": "^3.3.10"
	},
	"dependencies": {
		"monaco-editor-textmate": "^2.2.1",
		"monaco-textmate": "^3.0.1",
		"onigasm": "^2.2.4"
	}
}

webpack.config.js

const path = require("path");
const MonacoWebpackPlugin = require("monaco-editor-webpack-plugin");

module.exports = {
	mode: process.env.NODE_ENV,
	entry: "./index.js",
	output: {
		path: path.resolve(__dirname, "dist"),
		filename: "[name].bundle.js",
	},
	module: {
		rules: [{
			test: /\.css$/,
			use: ["style-loader", "css-loader",],
		}, {
			test: /\.ttf$/,
			use: ['file-loader']
		}],
	},
	plugins: [
		new MonacoWebpackPlugin({
			languages: ["typescript", "css"],
		})
	]
};

index.js

import * as monaco from "monaco-editor/esm/vs/editor/editor.api";
import { loadWASM } from 'onigasm' // peer dependency of 'monaco-textmate'
import { Registry } from 'monaco-textmate' // peer dependency
import { wireTmGrammars } from 'monaco-editor-textmate'

// create div to avoid needing a HtmlWebpackPlugin template
const div = document.createElement('div');
div.id = 'root';
div.style = 'width:800px; height:600px; border:1px solid #ccc;';

document.body.appendChild(div);

(async function(){
	await loadWASM('onigasm/lib/onigasm.wasm') // See https://www.npmjs.com/package/onigasm#light-it-up

	const registry = new Registry({
		getGrammarDefinition: async (scopeName) => {
			return {
				format: 'plist',
				content: await (await fetch('static/grammars/JavaScript.plist')).text()
			}
		}
	})
	
	// map of monaco "language id's" to TextMate scopeNames
	const grammars = new Map()
	grammars.set('javascript', 'source.js')
	//grammars.set('typescript', 'source.ts')
	
	monaco.languages.register({id: 'javascript'});
	await wireTmGrammars(monaco, registry, grammars)
	
	monaco.editor.create(
		document.getElementById('root'),
		{
			value: [
				'var abc = 123;',
				'abc++;',
				'console.log(abc);'
			].join('\n'),
			language: 'javascript',
			theme: 'vs-dark'
		}
	);
	
})();

I don't know if it is such a good idea, but I was trying to define my own tokenizer using your package, but the TokenizerState class is not exported. I was just hoping I could import it instead of having to write something that would do exactly what you already did :)

(I don't know how the copyrights would play here, so I would prefer having you exporting the class)

Hi, I have successfully wired monaco editor with textmate as the instructions.
I did not get any red console in my browser until I tried to set editor's model with thousand lines of string (more than 2k lines).

I thought the errors which was thrown did not affect the usage of monaco editor. The editor could be used well with no problem.
But maybe someone knows how to solve this. thanks.

The packages I use:

"onigasm": "^2.2.5",
"monaco-editor": "^0.32.1",
"monaco-editor-textmate": "^3.0.0",
"monaco-textmate": "^3.0.1"

Here is a GIF of this where I am reloading the page... sometimes the syntax highlighting works, sometimes not.

In order to get the syntax highlighting to work 100% of the time, I had to make a minor adjustment to the example:

setTimeout(() => {
    wireTmGrammars(monaco, registry, grammars, editor)
}, 1)

I guess this pushes the wireTmGrammars behind whatever internal monaco operations it depends on.
Note: I'm using monaco-editor version 0.21.2 with this fork of monaco-editor-textmate: #16

When using webpack, your package is bundled with this piece of code:

                    tokens: res.tokens.map(token => ({
                        ...token,
                        // TODO: At the moment, monaco-editor doesn't seem to accept array of scopes
                        scopes: token.scopes[token.scopes.length - 1]
                    })),

But the ...token notation is not working in the Electron renderer, even Firefox (ESR 52.8.1) has trouble with it.

It seems to be only working on Chrome.

The browser crashed while I was writing 60k compressed js code into the monaco-editor. When I commented out the highlighted logic, the compressed code appeared normally

Please help, I am getting this error:

Uncaught Error: editor._themeService.getTheme is not a function

TypeError: editor._themeService.getTheme is not a function
at Object.exports.TMToMonacoToken (tm-to-monaco-token.js:37)
at eval (index.js:45)
at Array.map ()
at Object.tokenize (index.js:42)
at TokenizationSupport2Adapter.tokenize2 (standaloneLanguages.js:208)
at safeTokenize (textModelTokens.js:372)
at TextModelTokenization._updateTokensUntilLine (textModelTokens.js:296)
at TextModelTokenization._tokenizeOneInvalidLine (textModelTokens.js:282)
at TextModelTokenization._revalidateTokensNow (textModelTokens.js:233)
at Object.eval [as callback] (textModelTokens.js:220)
at eval (errors.js:20)

index.js

// import * as monaco from "monaco-editor";
// or
import * as monaco from "monaco-editor/esm/vs/editor/editor.api";
// if shipping only a subset of the features & languages is desired

import { loadWASM } from "onigasm"; // peer dependency of 'monaco-textmate'
import { Registry } from "monaco-textmate"; // peer dependency
import { wireTmGrammars } from "monaco-editor-textmate";

async function liftOff() {
  await loadWASM(`https://cdn.jsdelivr.net/npm/[email protected]/lib/onigasm.wasm`); // See https://www.npmjs.com/package/onigasm#light-it-up

  const registry = new Registry({
    getGrammarDefinition: async (scopeName) => {
      console.log(scopeName);

      return {
        format: "plist",
        content: await (
          await fetch(
            `https://raw.githubusercontent.com/textmate/javascript.tmbundle/master/info.plist`
          )
        ).text(),
      };
    },
  });

  // map of monaco "language id's" to TextMate scopeNames
  const grammars = new Map();
  grammars.set("javascript", "source.js");

  monaco.editor.defineTheme("vs-code-theme-converted", {
    // Theme data
  });

  const editor = monaco.editor.create(document.getElementById("container"), {
    value: [`console.log('Hello')`].join("\n"),
    language: "javascript", // this won't work out of the box, see below for more info,
    theme: "vs-code-theme-converted", // very important, see comment above
  });

  setTimeout(() => {
    wireTmGrammars(monaco, registry, grammars, editor);
  }, 3000);
}

liftOff();

webpack.config.js

const MonacoWebpackPlugin = require("monaco-editor-webpack-plugin");
const path = require("path");

module.exports = {
  entry: "./index.js",
  output: {
    path: path.resolve(__dirname, "dist"),
    filename: "app.js",
  },
  module: {
    rules: [
      {
        test: /\.css$/,
        use: ["style-loader", "css-loader"],
      },
      {
        test: /\.ttf$/,
        use: ["file-loader"],
      },
    ],
  },
  plugins: [
    new MonacoWebpackPlugin({
      languages: ["javascript"],
    }),
  ],
  mode: "development",
  resolve: { fallback: { path: require.resolve("path-browserify") } },
};

According to this line in wireTmGrammars, this library (or Monaco in general, I suppose) currently doesn't support multiple scopes.

Will there be support in the future?

Is there currently a way to work around it?

Having a setup with Rust and WASM, I do have the problem that I can't pass in the monaco parameter to the wireGrammar function. There is no type/instance for the namespace.

However, using an existing wrapper for the monco editor, I already have the API available for calling monaco.languages.setTokensProvider. If I would have an instance to set.

Unfortunately, this is hidden in the wireTmGrammars. It would be great if the API could be extended, to offer some createProvider function, which creates the provider instance, but not setting it.

Something like:

export function createProvider(grammar: IGrammar, editor?: monacoNsps.editor.ICodeEditor) : TokensProvider {
    return {
        getInitialState: () => new TokenizerState(INITIAL),
        tokenize: (line: string, state: TokenizerState) => {
            const res = grammar.tokenizeLine(line, state.ruleStack)
            return {
                endState: new TokenizerState(res.ruleStack),
                tokens: res.tokens.map(token => ({
                    ...token,
                    // TODO: At the moment, monaco-editor doesn't seem to accept array of scopes
                    scopes: editor ? TMToMonacoToken(editor, token.scopes) : token.scopes[token.scopes.length - 1]
                })),
            }
        }
    }
}

export function wireTmGrammars(monaco: typeof monacoNsps, registry: Registry, languages: Map<string, string>, editor?: monacoNsps.editor.ICodeEditor) {
    return Promise.all(
        Array.from(languages.keys())
            .map(async (languageId) => {
                const grammar = await registry.loadGrammar(languages.get(languageId))
                const provider = createProvider(grammar, editor);
                monaco.languages.setTokensProvider(languageId, provider)
            })
    )
}

This would allow one to re-use as much code as possible, but performing the task of setting the tokens provider manually.

As said here: zikaari/monaco-textmate#4, there is no dist directory in the latest version of this library, so node can't find the module.

I just quickly looked at the code, but what is the use case for the grammars map? Is it to have different extension names that could use the same grammar? Could it be optional?

    const grammars = new Map()
    grammars.set('css', 'CSS')

    await wireTmGrammars(monaco, registry, grammars)

Editor option maxTokenizationLineLength is not work. Parsing long text lines can consume a lot of CPU resources, leading to browser page fake death. Under normal circumstances, long text lines should not be parsed.