Since bugs in the Ruby LSP can be disruptive of developer's workflows, we should have a stable release branch, which includes only changes that we are confident can be pushed forward.

Investigate and prototype a mechanism for doing regular stable releases from a stable branch. Tapioca has something similar and can serve as an example.

The ruby-lsp is really just an executable that we execute as the server. Although we will recommend adding the gem to the bundle, we could still fallback to the global ruby-lsp installation and let developers get the benefits just by doing gem install ruby-lsp.

This would allow developers to get the benefits from ruby-lsp even in projects where they may not have control over the bundle, like folks working on open source projects or even single file scripts.

This regexp that seems to mess things up:

ExpectedStderrList = [
      %r(
        -e:(?:1:)?\s\[BUG\]\sSegmentation\sfault.*\n
      )x,
      %r(
        #{ Regexp.quote(NO_JIT_DESCRIPTION) }\n\n
      )x,
      %r(
        (?:--\s(?:.+\n)*\n)?
        --\sControl\sframe\sinformation\s-+\n
        (?:(?:c:.*\n)|(?:^\s+.+\n))*
        \n
      )x,
      %r(
        (?:
        --\sRuby\slevel\sbacktrace\sinformation\s----------------------------------------\n
        (?:-e:1:in\s\`(?:block\sin\s)?<main>\'\n)*
        -e:1:in\s\`kill\'\n
        \n
        )?
      )x,
      %r(
        (?:--\sMachine(?:.+\n)*\n)?
      )x,
      %r(
        (?:
          --\sC\slevel\sbacktrace\sinformation\s-------------------------------------------\n
          (?:Un(?:expected|supported|known)\s.*\n)*
          (?:(?:.*\s)?\[0x\h+\].*\n|.*:\d+\n)*\n
        )?
      )x,
      %r(
        (?:--\sOther\sruntime\sinformation\s-+\n
          (?:.*\n)*
        )?
      )x,
    ]

Highlight Before the RegExp

Highlight After the RegExp

Things like [] or method parameters are highlighted incorrectly.

Although our LSP won't offer first class completion like Sorbet's LSP (since it would require complete typechecking), we can still offer some simple convenience auto completes.

Some ideas, but not an exhaustive list:

• on typing do, offer to complete with two options |arg| \n end or just end
• on proc, lambda or -> offer to complete with { |arg| } or just {}
• on if, offer else, elsif and complete the end
• offer to add end on other keywords, such as while, until, for
• on require or require_relative, offer a filtered list of possible files

Documentation: https://microsoft.github.io/language-server-protocol/specification#textDocument_completion

In addition, we can investigate how IRB completion works and evaluate if we can somehow use it from inside the Ruby LSP.

We can use a similar mechanism of #199 and find possible references if we store those in the index. These will not always be correct as type checking is needed to provide accurate results.

Documentation: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_references

Because we're checking if the node type matches exactly, we are not highlighting some legitimate occurrences of symbols. For example,

class Foo; end

Foo # <-- clicking here will not highlight the Foo in the class definition, but it should

The same thing happens for other types of definitions, like methods and modules.

It would be nice to have our yard documentation published as a static website and linked from our README. Investigate the best practices around this and add a GitHub action to publish the website when something is merged to main.

To generate the documentation, simply run

bundle exec rake yard

Keeping the server gem up to date is essential to make sure the latest features and bug fixes are being used to provide the best experience possible.

We can use bundle outdated to check if a newer version of the ruby-lsp gem exists, prompt users to upgrade and provide a button to run bundle update ruby-lsp from inside the editor.

This roadmap provides a rough idea of the next efforts for the Ruby LSP.

Items may be changed or reordered and are not guaranteed to be implemented.

Coming up next

• 🐛 #165
• ✨ #207
• ✨ #60
• ✨ #58
• ✨ #55
• 🛠 #1475
• 🛠 #1481

Planned

• ✨ #54
• ✨ #110
• ✨ #61
• ✨ #57
• 🛠 #205
• 🛠 #238

Desired

The following are features that are desired for the Ruby LSP, but are not yet planned - likely because they are already covered by the Sorbet LSP, turning them into nice to haves.

• ✨ #199
• ✨ #59
• ✨ #77
• ✨ #200
• ✨ #201
• ✨ #202
• ✨ #203
• ✨ #204
• ✨ #208

SyntaxTree comes with an RBS formatter. We can register the ruby-lsp as a formatter for .rbs files and take advantage of it to offer auto-formatting.

Inlay hints display "invisible" information to the user to assist with the understanding of the code. For example:

class User < ApplicationRecord
  (self.)has_many :posts
#^ inlay hint displaying the receiver (doesn't actually exist in the real code)
end

We can also display the name of positional arguments as inlay hints.

Documentation: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_inlayHint

Evaluated the advantages and disadvantages to add Sorbet to the ruby-lsp, so that we can decide on whether we want to do it or not.

If the decision is to add Sorbet, add it using our default tooling.

The code lens request allows us to attach a command to a piece of code. The main use case we want to support is running and debugging tests. We can add two commands to each discovered test (run | debug).

Additionally, we can populate VS Code's test interface using the same request. By using a middleware, we can retrieve test cases discovered, populate the test interface and provide similar options to run or debug examples.

Documentation: https://microsoft.github.io/language-server-protocol/specification#textDocument_codeLens

It's easier to see and understand documented features visually, rather than explain LSP requests in text.

Explore the possibility of adding GIFs to our requests documentation. E.g.:

• Are they displayed properly when generated from YARD?
• Where should GIFs be stored?

If #199 works and is successful in providing decent results for finding possible method definitions, we can use the same mechanism to provide signature help. This request shows the method signature for the invocation at the given cursor position, displaying arguments.

Note: there may be conflicts between possible methods. It's likely that we'll only be able to provide signature help if only a single result is found in the index.

Documentation: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_signatureHelp

Motivation

When contributing to projects without ruby-lsp installed (e.g. Ruby stdlibs), it's easy to commit the gem "ruby-lsp", require: false by mistake. And it's distracting to keep reminding oneself to avoid it.

So even though it's generally better to start ruby-lsp with bundle exec, I hope there can be an config to let me install and run it globally (disabled by default).

We currently only support shadowenv as a Ruby version manager, which means the server won't activate properly when using alternatives.

Add support for RVM, rbenv, asdf, chruby and ruby-installer (Windows).

Let me start by saying I am so excited that ruby tooling has gotten some love lately.
Are there a plans to add support for other editors (most notably) Neovim?
Neovim since version 0.5 natively supports Language Server Protocol and there is a significant subset of already supported LSP.

We allow users to configure the desired features from the VS Code plugin. However, it currently requires starting the LSP for the configuration changes to take effect.

We can use the didChangeConfiguration notification to verify if a feature has been enabled or disabled and then notify the client by unregistering or registering a capability without requiring a full restart.

Documentation: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#workspace_didChangeConfiguration

Problem

It seems like the tests for features such as semantic highlighting, folding, and document symbols are growing.
They are also more or less using the same Ruby source code as a test. This is not ideal because:

• Repetitive test code that may be wrong or have missing cases
• Causes a large and growing test file as we cover more cases
• May be a growing problem as we implement more features/LSP requests

Potential Solution

@Morriar and I were talking about potentially using syntax_tree's Node fixtures (which is currently being used to test nodes in syntax_tree) to test each of the LSP feature.

If we can all use the same fixtures, there will be one source of truth and we don't need to worry about the validity of the tests.

Questions

• Do you agree it's a problem?
• Is it worth solving?
• If so, what other ways are there to solve this problem?
• How can we import syntax_tree's fixtures to use them?

Thanks for the awesome gem. LSP server for ruby is a must have feature 🙂 Is there any forks with vim/neovim support? Just curious. 😀

Please point me out if there are any. Thanks and looking forward to hear from the community.

Full support for renaming things, like method names and constants is pretty difficult and should be left for the Sorbet LSP. However, our LSP can support renaming some simple things that don't require complete type checking, like:

• local variables
• method/block/lambda arguments

Remember that our LSP doesn't parse every file in the code base, so our renames are limited to the current file.

Documentation: https://microsoft.github.io/language-server-protocol/specification#textDocument_rename

Improve our existing semantic highlighting to address more complex Ruby syntax.

• Method and block arguments
• Punctuation (e.g.: the namespace separator :: or the pipes in a block argument are sometimes highlighted incorrectly without semantic support)
• self in front of singleton method definitions and the accompanying dot

We can also enhance the highlighting of syntax that is related to a specific tool, such as Sorbet or Rails. For example,

• Sorbet signature declarations
• Other Sorbet inline type annotations, such as T.let

Additionally, the specification supports some performance related improvements we can explore. There are two existing options (maybe implementing both is a good idea too)

• Implementing semantic tokens delta and refresh. This means we have to maintain in memory (inside our Document store) the current semantic tokens for a file and only return the difference between the previous response and the current one (indicating which tokens were deleted and which ones were added)
• Implement semantic tokens for a range. This one is considerably easier than delta and refresh, it's just a matter of returning only the tokens for the given visible range of the document to avoid flickering. In short, it's a new request endpoint where we could simply re-use our semantic tokens implementation and filter tokens that are outside of range

Documentation: https://microsoft.github.io/language-server-protocol/specification#textDocument_semanticTokens

Currently Ruby LSP folds ranges by including the closing delimiter into the folded range. For example, the following Ruby code:

def foo
  42
end

def bar
  "hello"
end

folds to:

def foo ⋯

def bar ⋯

However, that introduces an ergonomic problem if I want to add a new line after the definition of the foo method, for example. If I place the cursor at the end of the def foo⋯ line and press Enter, I start adding a line inside the foo method (above 42). There is actually no way to add a line right after the def foo block without starting to type on the empty line between the foo and bar methods.

I propose that we start folding ranges by leaving the closing delimiters unfolded, so that the original code gets folded as:

def foo ⋯
end

def bar ⋯
end

This way, if I want to add a new line inside the foo method, I put the cursor at the end of the def foo line and press Enter. If I want to add a new line after the foo method, I put the cursor at the end of the matching end of the foo method and press Enter. Thus, this folding style is more flexible for editing code in folded form.

Expected

When opening a file that's ignored in .rubocop.yml

eg.

AllCops:
  Exclude:
    - ignored.rb

It should hide the errors

Actual

It runs Rubocop on that file, and displays any errors it finds.

DM me on Slack for an example repo to reproduce on :)

Currently, we have no high level tests for the handler. Which is inconvenient as we can break it while changing requests. Add tests for the handler and make sure current features are covered.

Basically, we would want to do this through the CLI, which declares how we handle requests. The challenge is figuring out how to write to and read from the right pipes in order to trigger requests and see the responses.

For example,

# This begins the main loop with all registered requests using our handler
RubyLsp::Cli.start

# How do we send requests and read responses in tests?
some_pipe.write("textDocument/documentSymbol", params)

result = other_pipe.read
assert_equal(blah, result)

Because the handler/cli is an infinite loop, do not forget to shut it down on every test example.

After configuring BBEdit 14 (any version) to use ruby-lsp as the preferred language server, I'm finding during testing that the server does not exit when the editor sends an exit protocol notification. This causes the application to hang when quitting in cases where the user has opened a Ruby file (which lazy-starts the language server).

Server version: 0.2.0
Ruby version: 2.7.3
macOS 12.4 (21F79)

The repro on this is fairly straightforward, as follows:

1. Download and install BBEdit 14.5.1 (macOS 10.15.4 or later required).
2. Start the application.
3. Configure BBEdit to use ruby-lsp as the preferred language server (see documentation for how to do this).
4. Open any Ruby file.
5. Quit the application.

Expected:
=> Application quits immediately

Actual:
=> Application hangs waiting for ruby-lsp to exit.

The troubleshooting section of the LSP documentation contains a workaround for ill-behaved servers (the fact that it's necessary tells you how often the issue comes up, so it isn't just you :-) ); but the server should quit immediately (or at least as rapidly as possible) upon receiving an exit notification.

When we introduce new features for the ruby-lsp, we always need to start the plugin on a specific branch of ruby-lsp to check if things are working as expected.

Could we have some automation that prepares a web instance of VS Code already on the right branch? This would make manual testing much more convenient.

This may be possible with codespaces, which support defining extensions to be installed.

If a project does not use RuboCop, we can still fallback and offer auto-formatting through SyntaxTree. The interface used to format the file was introduced here.

As a part of this work, investigate if the best solution is to remove rubocop from our dependencies, check if it's a part of the application's bundle and then decide which auto-formatting to use.

Alternatively, we can keep rubocop as a dependency and check for a .rubocop.yml file.

Issue

Not a great user experience when people start a new Ruby file and highlighting glitches

Note that there is an obvious change in colour because I want to point out the swap to semantic highlighting to the default VS Code highlighting. Perhaps this won't be an issue is the themes are similar in colour and that could be a solution.

Background / Reason

syntax_tree does not provide nodes if it detects there is a parse error. The current behaviour for the LSP when it encounters a parse error is to reuse the old nodes in memory (containing a valid syntax). This works for now, but thinking about the future, when users create a new file and starts editing them, this means that the LSP features won't work until there is a fully valid state. It's not a great user experience as the semantic highlighting kind of glitches in and out of the default syntax highlighting.

Perhaps a good enough solution is to ensure the semantic highlighting theme and the fall-back theme is similar in colour so it won't be a jarring, drastic difference.

Additional context from Slack...

@kddnewton :

The current Ruby parser is not very error tolerant. If you check out how SyntaxTree.parse is implemented, you can get some nodes back in some cases which might make it slightly better, but it’s still not going to be great.

The real solution here is to rewrite the parser, which is a much bigger issue. Unfortunately not a huge amount can be done from the SyntaxTree standpoint until the underlying parser is improved.

@paracycle :

currently, the closest one is the Sorbet parser, since they had to do a bunch of manual changes to their parser to support error cases. they needed that to support autocomplete and other editor tooling.

Range formatting is a request very similar to formatting, but it applies only to the selected range and not the entire document.

Currently, trying to format ranges with either RuboCop or SyntaxTree fails, because both believe they are always looking at whole files instead of ranges.

For RuboCop, this will be difficult to overcome, given that trying to format a range in the middle of the document will even add # frozen_string_literal comments.

However, we may be able to use SyntaxTree for this if we add the ability of informing the formatter about the current indentation (or nesting) level. In the current behaviour, the code inside the range will be considered as a whole file and everything is pushed to indentation level zero.

Documentation: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_rangeFormatting

We currently provide examples of supported features as gifs and written code examples. While gifs better demonstrate the functionality we've implemented, they are not accessible to people with slow internet or those with low/no vision, seizure disorders, and other conditions.

This raised the question of how accessible the code examples are to begin with. Can people with screen readers easily understand what the LSP features are meant to do?

More investigation is required.

As we're adding new features, it will be good to provide the ability of configuring the ruby-lsp and turning things on or off as desired.

Since each developer can configure VS Code differently, we should add configuration to this plugin that is passed as options to the server, which then decides which capabilities to broadcast based on the configuration.

The selection range request returns suggestions of ranges the user might want to select based on cursor position. Typically, these match the same response for folding ranges, but we can add other things if we wish.

Documentation: https://microsoft.github.io/language-server-protocol/specification#textDocument_selectionRange

Semantic highlighting is already provided by the Ruby LSP server. However, if the server crashes or if the gem is not installed, we don't have a grammar file to fallback to and provide at least a decent amount of highlighting.

Explore adding a grammar file that we can contribute from this extension.

Definition of done

• Discover whether we need to define the complete grammar or if it's possible to just enhance the one bundled in VS Code
• Discover whether the grammar allows us to improve the basic highlighting
• Estimate the effort for building the grammar
• Create tickets to build the grammar

The document highlight request highlights a range of interest depending on the position of the cursor.

For example, we can highlight all occurrences of the same local variable. Or highlight the instance variable associated to an attr_reader.

Documentation: https://microsoft.github.io/language-server-protocol/specification#textDocument_documentHighlight

Code actions provide the ability to define any type of refactor that developers can manually apply. Some examples:

• Extract to method: select a portion of the code and have it extracted into a private method with the right parameters
• Make a method private
• Move method to parent class
• Change a block from do - end to {}
• Change a single line invocation to multiline
• Change if to unless
• Change while to until
• Wrap class in module
• Add attr_reader, attr_writer or attr_accessor for a given instance variable

Documentation: https://microsoft.github.io/language-server-protocol/specification#textDocument_codeAction

It was reported by a user for the rubocop-lsp that this code showed extra diagnostics on VS Code, that did not match the CLI output.

def my_method
  if a
    do_thing_0
  else
    if b
      do_thing_1
      do_thing_2
    end
  end
end

On the CLI:

• Only one violation for Style/IfInsideElse

On VS Code:

• One violation for Style/IfInsideElse
• An extra violation for Layout/IndentationConsistency on line 6
• For some reason, disabling one fixed both # rubocop:disable Style/IfInsideElse

Add a test to our diagnostics requests with this example and only return the Style/IfInsideElse violation.

Basic document highlight #87 followed what currently exists for Ruby, which is just highlighting simple occurrences of the same text without doing scope resolution.

Advanced document highlight means that only occurrences of the same entity should be highlighted. This means that if some name is shadowed, we still only highlight the correct instances of the entity.

Example

def foo
  a = 1 # <<- clicking on `a` here should highlight only the last `a` on this method, but not the block argument, which shadows the variable name

  [1, 2, 3].each do |a| # <<- not highlighted
    puts a # <<- not highlighted
  end

  a + 2 # <<- highlighted
end

The document link request allows us to attach a link to a piece of code. This is very useful to directing people to documentation. Some examples of what we can do:

• When looking at Sorbet things, link to the Sorbet documentation
• When looking at Rails things, link to the Rails documentation
• When looking at GraphQL things, link to the GraphQL documentation
• When hovering gems on the Gemfile, link to the official repo

Note: remember that if we want to add any internal resources, we'll have to allow this to be configurable somehow.

Idea: since we don't do type checking, we can't be absolutely sure that a method comes from Rails/Sorbet/GraphQL. As a best effort, we can check the parent class and other method invocations within the file to make a best guess before linking to the documentation.

Documentation: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_documentLink

After adding gem source locations to RBIs generated by Tapioca in Shopify/tapioca#1025, we can allow jumping to those locations through the document link request.

Documentation related functionality for document link will be implemented separately in #58.

It would help developers to better visualize meta-programming, if we could somehow provide a preview of what the result is going to be. Something like

class Foo
  ["bar", "baz", "bat"]. each do |name|
    # Meta-programming preview
    # def name
    #  process(name).to_sym
    # end
    define_method(name) { process(name).to_sym }
  end

  def process(string)
    # ...
  end
end

This could either be a part of inlay hints or a custom request, depending on how the visualization happens.

Despite not being able to provide a fully correct go to definition implementation (which requires type checking), the Ruby LSP could provide a simpler version of it.

One possible idea is to index all existing classes and methods in the code base and then providing go to definition results for methods that match the same name. Despite being naive, it's better than no support at all.

Ideas for exploration

• The indexer can run in a separate process (maybe a Ractor?) to not block the LSP start up
• The indexer can cache indices from files that haven't been modified and re-use those to speed up the initial start up
• The indexer can try to be a bit smart about argument count. In this case, it would find method definitions by searching not only the name, but argument count. Need to be careful about optionals and splats

Note: the Ruby LSP should not provide any go to definition at all if Sorbet is present, since Sorbet does proper type checking and returns correct go to definition results.

We have received reports of RuboCop formatting hanging indefinitely. Notice that it does not seem like the issue is performance on large files, which is a separate issue.

It seems that somehow the LSP either gets stuck or returns something invalid to VS Code (which in turn makes the "Running Ruby LSP formatter" dialogue never disappear).

Both reports we have received mentioned that there's nothing in the output tab, other than the default message of "Starting the Ruby LSP...".

Suggestion

We should consider making format on save false in the extension pack, while we work on improving the situation.

Suggestion on how to debug

The difficult part of this is reproducing the issue, given that we do not know what causes it. Some possible steps:

• Make sure "ruby lsp.trace.server": "messages" is turned on in .vscode/settings.json
• When formatting hangs, check the output tab and see if any other requests are still being made. If you manage to get requests going for any file, then it's unlikely that the server is stuck - which probably means we returned something invalid to VS Code (or didn't return anything when it was expecting something)

Definition of done

This ticket is considered closed when the root cause of the bug (hanging formatting) is identified and addressed.

If we add the indexer in #199, then it knows which classes, modules, constants and methods are defined in a workspace, which allows us to provide the workspace symbols request.

This request is very similar to document symbols, but instead of being scoped to a single file, it allows fuzzy searching for symbols across the entire project.

Documentation: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#workspace_symbol

What is the problem we're trying to solve?

While working on this PR, I noticed that IntegrationTest does not surface any errors found while reading the response in #read_response.

Why is this a problem we need to solve?

Better exception reporting and therefore ✨ PRODUCTIVITY ✨

The problem is that, when there is an error, the :result key in the response is nil, and that causes the test to fail without a useful error message. For example, this is a test exception where the response object is nil:

Minitest::UnexpectedError:         NoMethodError: undefined method `dig' for nil:NilClass

            assert(true, response.dig(:capabilities, :textDocumentSync, :openClose))
                                 ^^^^
            /Users/mapleong/src/github.com/Shopify/ruby-lsp/test/integration_test.rb:258:in `initialize_lsp'
            /Users/mapleong/src/github.com/Shopify/ruby-lsp/test/integration_test.rb:74:in `test_semantic_highlighting'

The reason why it's nil is because the response object's :result key is nil, but for a different reason than the one stated above. We would have to investigate it deeper after seeing this message.

However, if we just had the error message from this JSON surface, the problem would be immediately obvious:

  { "method": "telemetry/event",
    "params": { "request": "initialize",
                "lspVersion": "0.1.0",
                "requestTime": 0.001909999642521143, "errorClass": "NameError", "errorMessage": "uninitialized constant RubyLsp::Requests::SemanticHighlighting::Set\n\n      SPECIAL_RUBY_METHODS = T.let(Set[\n                                   ^^^", }, "jsonrpc": "2.0", }

How do you propose we solve it?

Raise the errors immediately if there is one, and show the underlying error message.

If #199 works and is successful in providing decent results for finding possible method definitions, we can use the same mechanism to provide hover. This request shows documentation/information about methods and constant when hovering with the cursor.

Note: there may be conflicts between possible methods and constants. It's likely that we'll only be able to provide hover if only a single result is found in the index.

Documentation: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_hover

Some developers are left wondering why some violations aren't automatically fixed on formatting. Sometimes, it's just because those violations are not auto-correctable (or because their auto-correct is considered unsafe).

We should include some text in our diagnostics to call attention to that, so that devs know it's expected.

For a document like the following:

module Foo
end

module Foo::Bar
end

class Foo::Bar::Baz
end

the LSP server shows the following in the outline:

- Foo (module)
- Bar (module)
- Baz (class)

which is not very helpful since it loses all the nesting.

It should either show:

- Foo (module)
- Foo::Bar (module)
- Foo::Bar::Baz (class)

or

- Foo (module)
+- Bar (module)
 +- Baz (class)