Bug report

• I confirm this is a bug with Supabase, not with my own application.
• I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug

I can't build the project hence I can't start developing. I tried running git submodule update --init --recursive, but I'm getting this error:

fatal: Unable to find current revision in submodule path 'libpg_query'

I can't seem to pull the libpg_query submodule

To Reproduce

Steps to reproduce the behavior, please provide code snippets or a repository:

1. Clone the repo
2. Run git submodule update --init --recursive on the shell.
3. See error

Expected behavior

I expect the submodule to be pulled so that the build process can continue.

Screenshots

If applicable, add screenshots to help explain your problem.

System information

• OS: Windows 10

Postgres parses sql strings such as execute 'select 1'; and function bodies, e.g.

CREATE FUNCTION dup(in int, out f1 int, out f2 text)
    AS $$ SELECT $1, CAST($1 AS text) || ' is text' $$
    LANGUAGE SQL;

as string constants. To improve dx, especially for function bodies, we should pass the string back into the SourceParser.

A few things to consider here:

• refactor SourceFileParser to SourceParser, and pub fn parse_source_file(&mut self, text: &str) to pub fn parse_source(&mut self, text &str, at_offset: Option<i32>), similar to how the statement parser is designed (done)
• we have to find a way to make a distinction between a sql string, and a normal string. I would propose to just parse any string constant, and if does not return an error use replace it with the sub-tree.

Personally I'd use it in its unfinished state as it's better than anything in there already, but whatever you wish. This would be a good idea when it is 'stable' I think

we are still using logos in the statement_parser, and its quite the overkill. we can switch to a regex-based solution similar to source_parser and drop the dependency.

Bug report

during a cargo xtask install i get the following error

error[E0432]: unresolved import `crate::codegen::SyntaxKind`
 --> crates/parser/src/lexer.rs:8:5
  |
8 | use crate::codegen::SyntaxKind;
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^ no `SyntaxKind` in `codegen`

error[E0432]: unresolved imports `crate::codegen::get_nodes`, `crate::codegen::Node`, `crate::codegen::SyntaxKind`
 --> crates/parser/src/parse/libpg_query_node.rs:4:15
  |
4 |     codegen::{get_nodes, Node, SyntaxKind},
  |               ^^^^^^^^^  ^^^^  ^^^^^^^^^^ no `SyntaxKind` in `codegen`
  |               |          |
  |               |          no `Node` in `codegen`
  |               no `get_nodes` in `codegen`
  |
  = help: consider importing one of these items instead:
          cstree::util::NodeOrToken::Node
          petgraph::data::Element::Node
          petgraph::graph::Node
          pg_query::Node
          pg_query::protobuf::node::Node
          pg_query_proto_parser::FieldType::Node
          pg_query_proto_parser::Node

error[E0432]: unresolved import `crate::codegen::SyntaxKind`
 --> crates/parser/src/parse/source.rs:1:5
  |
1 | use crate::codegen::SyntaxKind;
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^ no `SyntaxKind` in `codegen`

error[E0432]: unresolved import `crate::codegen::SyntaxKind`
 --> crates/parser/src/parse/statement.rs:6:5
  |
6 | use crate::codegen::SyntaxKind;
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^ no `SyntaxKind` in `codegen`

error[E0432]: unresolved import `crate::codegen::SyntaxKind`
 --> crates/parser/src/parse/statement_start.rs:4:5
  |
4 | use crate::codegen::SyntaxKind;
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^ no `SyntaxKind` in `codegen`

error[E0432]: unresolved import `crate::codegen::SyntaxKind`
  --> crates/parser/src/parser.rs:11:5
   |
11 | use crate::codegen::SyntaxKind;
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^ no `SyntaxKind` in `codegen`

error[E0432]: unresolved import `crate::codegen::SyntaxKind`
 --> crates/parser/src/sibling_token.rs:1:5
  |
1 | use crate::codegen::SyntaxKind;
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^ no `SyntaxKind` in `codegen`

error[E0432]: unresolved import `crate::codegen::SyntaxKind`
 --> crates/parser/src/syntax_node.rs:9:5
  |
9 | use crate::codegen::SyntaxKind;
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^ no `SyntaxKind` in `codegen`

error[E0432]: unresolved import `crate::codegen::SyntaxKind`
  --> crates/parser/src/lib.rs:32:9
   |
32 | pub use crate::codegen::SyntaxKind;
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^ no `SyntaxKind` in `codegen`

error[E0412]: cannot find type `TokenProperty` in module `crate::codegen`
   --> crates/parser/src/parse/libpg_query_node.rs:416:35
    |
416 | fn cmp_tokens(p: &crate::codegen::TokenProperty, token: &crate::lexer::Token) -> bool {
    |                                   ^^^^^^^^^^^^^ not found in `crate::codegen`

System information

• OS: WSL2 ubuntu 22.04

Bug report

• I confirm this is a bug with Supabase, not with my own application.
• I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug

Edit: Even though this may not be a bug, I reopened from the comment from @kiwicopple - if this is unwanted, feel free to transform this into a discussion under supabase/postgres_lsp discussions

Not sure if this is actually a bug, but it would be great to have support for all features in SQL inside JavaScript and TypeScript template literals

First of all, thanks for all of the effort that Supabase is putting into the PostgreSQL community, including tooling, really amazing 🙌

I recently encountered Postgres Language Server via the recent tweet from @kiwicopple and it looks amazing!

I have been chasing the "holy grail" of tooling for SQL template literals in JavaScript + TypeScript for a while now:

• syntax highlighting (works, with the VS Code extension SQL tagged template literals (syntax only))
• linting (works, using SafeQL)
• formatting, some options
  • prettier-plugin-embed and prettier-plugin-sql - uses sql-formatter
  • kind of works, with eslint-plugin-sql and eslint-plugin-unicorn, but very opinionated / not so configurable - uses pg-formatter
  • @potygen/prettier-plugin-pgsql by @ivank - uses custom Prettier AST formatter

Wasn't even looking for inline documentation yet, but that's a great addition too!

It seems that the plan will be for Postgres Language Server to support all of these use cases, which is amazing!

I would love to have everything handled by one highly-battle-tested tool, which could maybe also deal with some of the shortcomings from the tools listed above.

However, for Postgres Language Server to work for any of our projects, it would need a feature which I'm not sure it currently has (can't tell, from a quick glance at the docs and the code):

• Support for tagged template literals in JavaScript and TypeScript

There is precedent for SQL tagged template literals in JS+TS, some of the more popular ones being:

• Postgres.js
• @vercel/postgres

Eg. I would love for all of the features of Postgres Language Server to be available in the following code block:

import { createClient } from '@vercel/postgres';
 
async function queryPosts() {
  const client = createClient();
  await client.connect();
 
  try {
    const likes = 100;
    const { rows, fields } =
      await client.sql`SELECT * FROM posts WHERE likes > ${likes};`;
  } finally {
    await client.end();
  }
}

Or this code block:

// db.js
import postgres from 'postgres'

const sql = postgres({ /* options */ }) // will use psql environment variables

async function getUsersOver(age) {
  const users = await sql`
    select
      name,
      age
    from users
    where age > ${ age }
  `
  // users = Result [{ name: "Walter", age: 80 }, { name: 'Murray', age: 68 }, ...]
  return users
}


async function insertUser({ name, age }) {
  const users = await sql`
    insert into users
      (name, age)
    values
      (${ name }, ${ age })
    returning name, age
  `
  // users = Result [{ name: "Murray", age: 68 }]
  return users
}

To Reproduce

Steps to reproduce the behavior, please provide code snippets or a repository:

1. Attempt to use Postgres Language Server with SQL inside JavaScript and TypeScript template literals
2. No support

Expected behavior

SQL inside JavaScript and TypeScript template literals should also have all features of the language server

Screenshots

--

System information

• OS: [e.g. macOS, Windows]
• Browser (if applies) [e.g. chrome, safari]
• Version of supabase-js: [e.g. 6.0.2]
• Version of Node.js: [e.g. 10.10.0]

Additional context

--

Bug report

Describe the bug

I am not sure yet whether this is a bug, but we should add tests for transaction statements to see how they are parsed by lipg_query. They should be parsed as a single statement. We may have to adapt the source_parser regex for that.

Expected behavior

Transactions are beneath a single ast node, and a nested cst node.

Bug report

• I confirm this is a bug with Supabase, not with my own application.
• I confirm I have searched the Docs, GitHub Discussions, and Discord.

FYI to the maintainers, the above bullet points for this issue template still reference Supabase directly, not postgres_lsp. 😄

Describe the bug

I was trying to follow the "building from source" section of the docs:

• I ran git submodule update --init --recursive and verified the submodule was populated correctly.
• Then I ran cargo xtask install but it eventually errored; see output below.

❯ cargo xtask install
...
...
...
$ cargo install --path crates/postgres_lsp --locked --force
...
...
...
error[E0554]: `#![feature]` may not be used on the stable release channel
  --> crates/parser/src/lib.rs:18:12
   |
18 | #![feature(lazy_cell, is_sorted)]
   |            ^^^^^^^^^

error[E0554]: `#![feature]` may not be used on the stable release channel
  --> crates/parser/src/lib.rs:18:23
   |
18 | #![feature(lazy_cell, is_sorted)]
   |                       ^^^^^^^^^

For more information about this error, try `rustc --explain E0554`.
error: could not compile `parser` (lib) due to 2 previous errors
warning: build failed, waiting for other jobs to finish...
error: failed to compile `postgres_lsp v0.0.0 (/Users/noah/Git/postgres_lsp/crates/postgres_lsp)`, intermediate artifacts can be found at `/Users/noah/Git/postgres_lsp/target`
Error: install server

Caused by:
    command exited with non-zero code `cargo install --path crates/postgres_lsp --locked --force`: 101

I'm relatively unfamiliar with Rust and its tooling; I found an SO thread online but the suggested fix is to just install the nightly version of Rust/cargo? That feels odd to me, so I'll wait for maintainer feedback. 😅

System information

• macOS Sonoma 14.1.1
• asdf package manager with asdf-rust plugin

❯ asdf plugin list --refs --urls
╭────┬────────────┬─────────────────────────────────────────────────────┬────────┬─────────╮
│  # │    name    │                     repository                      │ branch │   ref   │
├────┼────────────┼─────────────────────────────────────────────────────┼────────┼─────────┤
│  0 │ nodejs     │ https://github.com/asdf-vm/asdf-nodejs.git          │ master │ 150ec5d │
│  1 │ rust       │ https://github.com/code-lever/asdf-rust.git         │ master │ 95acf4f │
╰────┴────────────┴─────────────────────────────────────────────────────┴────────┴─────────╯

❯ rustc --version
rustc 1.71.1 (eb26296b5 2023-08-03)

❯ cargo --version
cargo 1.71.1 (7f1d04c00 2023-07-29)

❯ node --version
v18.15.0

Additional context

I ended up here because I was reminded about postgres_lsp from a related HN thread posted today.

Bug report

• I confirm this is a bug with Supabase, not with my own application.
• I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug

A clear and concise description of what the bug is.

Building from source and using documentation's config in neovimv0.9.5

To Reproduce

Steps to reproduce the behavior, please provide code snippets or a repository:

1. Build the lsp from source.
2. Create a file for the config and source it

Expected behavior

The lsp to run

Screenshots

N/A

System information

• OS: Ubuntu 22.04

Additional context

Was having issues building because cargo couldn't find libclang, so I install libclang 14 and it allowed me to build.

Chore

Describe the chore

Bump pg_query to a version which includes pganalyze/pg_query.rs#28. This will fix https://github.com/supabase/postgres_lsp/security/dependabot/1

Additional context

N/A

Planned Work

We already had it in place but dropped it because it's unnecessary now that the parser panics on invalid statements. but turns out it's actually quite useful when reviewing pull requests for #51. we should add it back using insta as before. we should be able to simply copy it from the git history of the statement_parser_test file.

Improve documentation

Link

https://github.com/supabase/postgres_lsp/blob/d4e971eb4e526e568d44a17fc878807069d515ea/README.md#neovim

Describe the problem

Postgres Language Server installation appears to be dependent on LSP Zero.
I would like to know how to install it without dependencies.

Describe the improvement

Avoid reliance on LSP Zero.

Additional context

Some of the statements in the valid statements tests are not recognised as such. Especially CREATE POLICY, GRANT and REVOKE are never recognised as statements and thereby never past to the libpg_query parser. Take the following statements to test against.

-- 0027
GRANT UPDATE ON accounts TO Joe;
-- 0028
REVOKE ALL ON accounts FROM PUBLIC;
-- 0029
GRANT SELECT (col1), UPDATE (col1) ON mytable TO miriam_rw;

This has to be fixed in the statement parser.

Right now, the SourceFileParser cuts a sql source into individual statements and comments. These are then passed into the StatementParser.

The lexer is regex-based. Right now, it mainly tries to cut at ;. This approach does not work with nested statements, especially create function statements that contain multiple statements in their parser. Unfortunately, due to upstream limitations, the regular expressions required to exclude ; that are within dollar-quoted strings do not work. We will have to find a way to cut the source into separate statements, even for a source that contain create function statements.

We recently finished the implementation of the core parser algorithm. Details can be found in this blog post.

While a lot of work has been automated with procedural macros, we still need to provide the keywords for a node in get_node_properties manually. For example, a SelectStmt node has the select keyword as a property, and if there is a from_clause, a from keyword. This is implemented as

fn custom_handlers(node: &Node) -> TokenStream {
    match node.name.as_str() {
	"SelectStmt" => quote! {
	        tokens.push(TokenProperty::from(Token::Select));
	        if n.from_clause.len() > 0 {
	               tokens.push(TokenProperty::from(Token::From));
	        }
	        ...
	},
        ...
}

Requirements

Setup a rust development environment:

1. Install rustup
2. Install nightly rust via rustup-init

rustup-init --default-toolchain nightly -y

3. Fork the repository

Run your first test

To make it as easy as possible for contributors to get into it, we prepared a simple test suite to quickly assert the tokens for a node.

1. open the file crates/parser/src/codegen.rs. you will find a bunch of tests there, e.g. test_simple_select. In each test, the test_get_node_properties helper is called . The parameters are a valid sql input str, the kind of the node we want to test for, and a Vec<TokenProperty> with the properties we expect.
2. in a terminal, cd into crates/parser
3. now, run a test with debugging enabled RUST_LOG=DEBUG cargo test test_simple_select. you will see both the entire graph, as well as the node you are testing against within the logs. The test should pass, since we are expecting only the Select keyword.

Become familiar with the codebase

A lot of logic is generated using macros, which makes it harder to find the definition of struct, enums and functions. Most important in this case are SyntaxKind and TokenProperty.

SyntaxKind is an enum that holds a variant for every node and token that Postgres supports, and some custom ones that we need for parsing such as Eof (end of file). To figure out what variants exactly, you can look into the generated source of pg_query.rs to find out how nodes and tokens are defined. This is also useful to figure out what properties each node has. Open the file crates/parser/src/parse/libpg_query_node.rs and go to the definition of use pg_query::NodeEnum;. The file contains all nodes and tokens. Search for enum Token to find the latter.

TokenProperty is a struct that defines a property of a node. It is defined in the codemod for get_node_properties and holds either a string value or a kind or both. You can create a TokenProperty from a various types, e.g. TokenProperty::from(SyntaxKind::SelectStmt).

Add your own test

Let's say you want to add a test for the SelectStmt node to contain the from property when a from clause is used. We choose a simple select 1 from contact; statement as our input. Since there are many nodes within this statement, we have to pass the kind of the node that we want to test: SyntaxKind::SelectStmt. We expect two properties: Select, and From. The test now looks like:

    #[test]
    fn test_select_with_from() {
        test_get_node_properties(
            "select 1 from contact;",
            SyntaxKind::SelectStmt,
            vec![
                TokenProperty::from(SyntaxKind::Select),
                TokenProperty::from(SyntaxKind::From),
            ],
        )
    }

Note that SyntaxKind contains all nodes and token types. In this case, the implementation has already been done and the test should pass. If your sample contains a case that is not yet implemented, go to crates/codegen/src/get_node_properties and add the missing implementation to custom_handlers. You should be able to figure out what properties are relevant from the node definition generated by pg_query.rs (see above).

Any help is highly appreciated. Please let us know if you have problems setting everything up. We are very happy to support and improve this guide to maximise efficiency for contributors.

Sibling tokens are tokens that should end up at the same depth in the CST, such as ( and ). Right now, that is not the case. Take for example the following statement:

ALTER TABLE products ALTER COLUMN price TYPE numeric(10,2);

The brackets of numeric are not on the same level:

[email protected]
        [email protected] "ALTER"
        [email protected] " "
        [email protected] "TABLE"
        [email protected] " "
        [email protected]
          [email protected] "products"
        [email protected] " "
        [email protected]
          [email protected] "ALTER"
          [email protected] " "
          [email protected] "COLUMN"
          [email protected] " "
          [email protected] "price"
          [email protected] " "
          [email protected] "TYPE"
          [email protected] " "
          [email protected]
            [email protected]
              [email protected] "numeric"
              [email protected] "("  <-- this is deeper than
              [email protected]
                [email protected] "10"
              [email protected] ","
              [email protected]
                [email protected] "2"
        [email protected] ")" <-- this
        [email protected] ";"

To fix this, we have to keep track the depth at which tokens are "opened". If the closing token is encountered, we apply it at the same depth as its sibling.

A SiblingToken struct already exists from past implementations. It should be reused.