Your codestyle is all over the place, and contributers have no idea what style to use. Make an odd eslint config please :)

Since a preprocessor is not a standard part of a processor (e.g. not all processors use a preprocessor), we should embrace the plugin-based architecture and make a plugin out of it. Or several!

Because streams are inherently asynchronous, the synchronous pipeline has to be rewritten into a proper asynchronous flow.

I tried to set up a quick way to test the new lexer (#12), but this runs stages in erroneous order.

You did this to yourself ❤️ #4

The lexer works but I cannot remember what anything does. That's bad. Please rewrite your horrible mess you dummy.

Theres no documentation. :osrs-loader

For testing, we're processing small files. When the language will be used in bigger projects, it will not scale if it has to load and save everything into memory. Streams were introduced exactly for this use case, so we should use them.

When the lexer has gone through all defined grammars but finds none, it throws an error. Currently it just says that the unidentifiable lexeme and the entire string after it are all unidentifiable. The lexeme should be cut off at a point where actually no grammar can be found.

The lexer now logs the correct line when a lexical error was found, but that line is just white. Syntax colouring should help identify the error better.

The old architecture does not allow scaling in any way. All features must be hacked into the code. Even the cleanest code is a chore to traverse if all you want to do is add a feature, or just update a certain part of the compiler. By transitioning to a more plugin-based architecture, we explicitly design all parts of the language as plugins, allowing for easy additions and removals, as well as a inherently easy-to-maintain structure.

There are some lexemes that benefit from having a very specific type, but it would alse be very useful for developing to adress a generalised group of specifik tokens, ergo:

const allowedTypes = [
  "integer number",
  "floating point number",
  "single line comment",
  "multi line comment"
];
//Check something against allowedTypes

Could (and should) be shortened by generalising them as such:

const allowedTypes = [
  "number",
  "comment"
];
//Check something against allowedTypes

We need to think about wether it's useful to implement this as a lexer feature, or if we sould just check if the type includes the shortened version (as a whole word).

Perhaps in a way like this:

const Lexer = require("./Lexer");
const lexer = new Lexer();
lexer.rule("super,specific,type", /regex/);
/* > LexicalToken: {
  types: ["super", "specific", "type"],
  type: "super.specific.type",
  lexeme: ...
} */

This would require every token to either have a new property which is an array of all levels of specificity separated by a delimiter, or just store the raw string as a type and splitting by delimitter on demand.

When an error occurs, the cause of the error will be stripped out of the input, and will be used for error logging. This normally works, but in some cases it will freeze the lexer. No idea why yet.

This does not freeze

const 💩 = 13;

This does freeze

const 💩 💩 = 13;

It seems this is caused by having any other unrecognised lexeme after any recognised tokens, such as the whitespace in the aforementioned example.

As of now, you can only assign a single token that complies to a single rule, but regular expressions can give back countless matches. We should look into a syntax for assigning multiple tokens with one regex, e.g:

const Lexer = require("./Lexer");
const lexer = new Lexer();
lexer.rule([
  "builtin math",
  "operator",
  "builtin math method"
], /\b(Math)(.?)(floor|round|ceil)\b/);

The preprocessor does not work yet. In order for it to work, it must recognise the beginning and ending of preprocessor directives from tokens generated by the lexer.

The old architecture does not allow scaling in any way. All features must be hacked into the code. Even the cleanest code is a chore to traverse if all you want to do is add a feature, or just update a certain part of the compiler. By transitioning to a more plugin-based architecture, we explicitly design all parts of the language as plugins, allowing for easy additions and removals, as well as a inherently easy-to-maintain structure.

In order for some plugins (such as the preprocessor) to work, they must be able to access the token stream instead of an AST. There should be a way for a user to tell a plugin what stage to hook into.