Code Monkey home page Code Monkey logo

look-at-me-sideways's People

Contributors

andrew-nocera-looker avatar arthurbellis avatar dependabot[bot] avatar drstrangelooker avatar esya avatar fabio-looker avatar iserko avatar jamescurtin avatar jeremytchang avatar josephaxisa avatar kmccauley138 avatar villoro avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

look-at-me-sideways's Issues

False positive on F1 rule with a derived table

LAMS is reporting a false positive on the F1 rule for a derived table using parameters.

Detailed error: linkback references another view, if linkback_type, via {% if linkback_type._parameter_value == "Client" %}

The full dimension code is:

  dimension: linkback {
    sql: TRUE ;;
    html:
      {% if linkback_type._parameter_value == "Client" %}
      <p>Thank you for logging in!</p>
      <p><a href="https://<redacted>/client/{{ client_id._parameter_value | replace:'$','-' }}">Return to dashboard</a></p>
      {% endif %}
    ;;
  }

LAMS not currently compatible with enterprise github accounts hosted on-prem?

I'm looking to implement LAMS for a large logistics firm with an enterprise Github account hosted on-prem. Currently the account/repo environment variables passed to the LAMS docker container during build assume the user's github account is hosted by github (https://github.com/<ACCOUNT_NAME>/<REPO_NAME>/). How can I help make it possible to integrate LAMS with github accounts not hosted by github (https://github.<ENTERPRISE_NAME>.com/<ACCOUNT_NAME>/<REPO_NAME>/)? Is it just the look_at_me_sideways_config.xml file that needs to change? Is it reasonable that enterprise github accounts would follow the same url format https://github.<ENTERPRISE_NAME>.com/ ?

Question about rule F3

Hi

I have a question about the style rule F3. Is there any explanation of why this rule is valid? I have checked it in the latest version of Looker and it doesn't look like that looker is using COUNT(*) syntax anymore.

Is it possible that this rule is obsolete? or it is connected to specific database?

Thank you
Adrian

Wrong implementation of K1 rule

You're using naming convention to check both K1 and K2 at the same time, while K1 should be checked basing on if there's a dimension that has a primary key attribute set to yes

Rule Exceptions Syntax not working

I tried the global rule exception syntax in manifest file, But I still get errors related to those rules. Am I missing anything?

Tried below syntaxes:
1)

 # LAMS
# rule_exemptions: {
#  K4: "Rule not Required"
# }
 # LAMS
# rule_exemptions: {
#  F2: "Rule not Required"
# }
# LAMS
# rule_exemptions: {
#  k4,F2,F3,T1,T2: "Rule not Required"
# }

Please help me implement these exemptions.

I am getting the below error

error: {
toString: [Function: toString],
message: 'Expected "#", "expr", "html", "sql", "}", [ \t\n\r], or [\-+_a-zA-Z0-9.] but "," found.',
expected: [Array],
found: '[',
location: [Object],
name: 'SyntaxError',
context: ''
},
_file_path: 'manifest.lkml',
_file_rel: '',
_file_name: '',
_file_type: 'manifest'
},

Can't execute lams

Hello,

I created a Dockerfile with the following:

FROM alpine
RUN apk update
RUN apk add --update nodejs nodejs-npm git openssh-client vim-nox

WORKDIR /root/
RUN git clone [email protected]:my_repo/my_looker.git
RUN ls -l /root/my_looker

WORKDIR /root/
RUN git clone https://github.com/looker-open-source/look-at-me-sideways.git
RUN ls -l /root/look-at-me-sideways
RUN ls -l /root

WORKDIR /root/look-at-me-sideways
RUN npm install

WORKDIR /root/my_looker
RUN node /root/look-at-me-sideways

Then i run the docker and:

~ # pwd
/root
~ # ls -l
total 8
drwxr-xr-x    1 root     root          4096 Sep  6 16:01 look-at-me-sideways
drwxr-xr-x    4 root     root          4096 Sep  6 16:00 my_looker
~ #

All looks ok, but then:

~ # cd my_looker/
~/my_looker # node ../look-at-me-sideways/
~/my_looker #

But it's not doing anything...

Help is greatly appreciated!

Share GitLab CI configuration

Sharing my Gitlab CI setup (which was a quick project);
Created a simple Dockerfile to wrap look-at-me-sideways

FROM alpine:edge

WORKDIR /opt/lams

RUN apk --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/ upgrade && \
 apk --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/ add \
 nodejs nodejs-npm

RUN  npm i -g @looker/look-at-me-sideways --unsafe-perm

CMD ["/bin/sh"]

This image is stored in the registry; (registry.example.com/look-at-me-sideways-bot)

Created a gitlab-ci.yml file in the LookML repo

stages:
  - πŸ‘€ Look at me sideways

Lint:
  stage: πŸ‘€ Look at me sideways
  image:
    name: registry.example.com/look-at-me-sideways-bot
    entrypoint: [""]
  script:
  - lams --reporting=no --output-to-cli
  artifacts:
    expire_in: 2 weeks
    when: always
    paths:
      - issues.md
  when: on_success
  allow_failure: true

On every commit, lams will run an saves the issues.md in the Gitlab CI artifacts.

New Rule? PK information also in sql of dimension

If a PK dimension, in addition to following the naming convention, were to use the following sql comment:

dimension: pk1_user_id {
  hidden: yes
  sql: ${TABLE}.id /*pk1*/ ;;
}

Then dev users looking to troubleshoot a query could quick evaluate the cardinality of its joins by looking at the SQL alone.

Not sure on this one, it may be a bit of a stretch in terms of which audience will see which information, but I found myself trying to debug a difference in a query while developing and not having this in the SQL query meant I would have to go look up each join in the explore to double-check their logic.

Issue with custom rule JSON path

For the last few days I have been applying some custom rules to my LookML code and I've come across an issue.

When trying to use the match $.model..view.[dimension,measure].*, it returns an error saying "Error in custom rule definition", which doesn't happen when taking the last section out, or referencing to just either a dimension or a measure. I could split the rules into the same one separated by dimension and measure, but that also didn't return any errors or warnings (the view had mistakes so I could test it). But it also didn't return the initial "Error in custom rule definition" error.

Is there another way to write the path to all dimensions and measures? Or a way to get around this?

P1: Parsing Error

image

We have upgraded our looker instance to 22.0.18 and I am getting P1: Parsing error: Expected "#", "\"", [ \t\n\r], or [\-+_a-zA-Z0-9.] but "]" found in the files that we didn't change in months.

Lams could not be used, SyntaxError: Unexpected token

Hi, is the library failing for anyone else?

This is my first time using it and I'm already running into problems when I'm trying to run it.

Commands i run:

sudo npm install -g @looker/look-at-me-sideways
lams

error message on lams command:

/usr/local/lib/node_modules/@looker/look-at-me-sideways/cli.js:22
                .map(([k, v])=>[k.replace(/[-_][a-zA-Z-0-9]/g, (s)=>s.slice(1).toUpperCase()), v])
                      ^

SyntaxError: Unexpected token [
    at exports.runInThisContext (vm.js:53:16)
    at Module._compile (module.js:374:25)
    at Object.Module._extensions..js (module.js:417:10)
    at Module.load (module.js:344:32)
    at Function.Module._load (module.js:301:12)
    at Function.Module.runMain (module.js:442:10)
    at startup (node.js:136:18)
    at node.js:966:3

Seems that the latest deployment itself has a failing CI build, could that be reason why? 77b2ba0

Add glob `ignore` argument, and default to include `node_modules` folder

We have added LAMS to our project and have found that some reported issues were due to views that don't even exist in our codebase, digging into it I discovered that those views come from the lookml-parser dependency that includes them to be linted with the project (i.e. https://github.com/fabio-looker/node-lookml-parser/blob/9cf49326ece2bd9834535bf77cec1087554ccbfe/test-projects/002-ndt/ndt.view.lkml)

image

Also when created a new project and LAMS is added it fails to run because of those same views that the parser is including, it raises the following error ⬇️

TypeError: Cannot read property 'match' of undefined
    at pkNamingConvention (/home/marcelolx/looker/node_modules/@looker/look-at-me-sideways/rules/k1-2-3-4.js:28:23)
    at Array.filter (<anonymous>)
    at module.exports (/home/marcelolx/looker/node_modules/@looker/look-at-me-sideways/rules/k1-2-3-4.js:38:61)
    at module.exports (/home/marcelolx/looker/node_modules/@looker/look-at-me-sideways/index.js:111:17)
    at async /home/marcelolx/looker/node_modules/@looker/look-at-me-sideways/cli.js:26:2
error Command failed with exit code 1.

And when checking why it fails I can see that it fails on a dimension from the view that only exists in lookml-parser

{
  '$strings': [
    '@$type',
    ':',
    ' ',
    '@$name',
    ' ',
    '{',
    [ 'hidden', 'hidden', ':', ' ', '@' ],
    '}'
  ],
  '$type': 'dimension',
  '$name': 'user_id',
  hidden: true,
  '$view': 'user_90_day_facts'
}

I have created a project if that helps https://github.com/marcelolx/looker

  • checkout
  • yarn install
  • yarn lint

Cannot read properties of undefined (reading 'url')

TypeError: Cannot read properties of undefined (reading 'url')
    at /c/dev/work/node/node-v16.13.2-linux-x64/lib/node_modules/@looker/look-at-me-sideways/rules/f1.js:44:59
    at Array.map (<anonymous>)
    at module.exports (/c/dev/work/node/node-v16.13.2-linux-x64/lib/node_modules/@looker/look-at-me-sideways/rules/f1.js:44:46)
    at module.exports (/c/dev/work/node/node-v16.13.2-linux-x64/lib/node_modules/@looker/look-at-me-sideways/index.js:131:26)
    at async /c/dev/work/node/node-v16.13.2-linux-x64/lib/node_modules/@looker/look-at-me-sideways/cli.js:28:2

Running into an issue when lams tries to parse a dimensions link property. Specifically

dimension: key {
    label: "Key"
    #group_label: "Jira"
    type: string
    sql: ${TABLE}.key ;;
    link: {
      url: "http://@{COMPANY_DOMAIN_JIRA}.atlassian.net/browse/{{ value }}"
      label: "View in Jira"
    }
  }

If I comment this out lams runs fine.

False positive(s) for rule E2 when constraining a primary key on a single-quoted string.

LAMS is reporting false positive(s) for PKs missing in a SQL join statement where most PKs are used in equality constraints with matching PKs and one PK is constrained against a string surrounded in single quotes.

Sample failing input:

explore: test_explore {
			join: join_view {
					relationship: one_to_one
					type: inner
					sql_on:
					\${test_explore.pk1_id} = \${join_view.pk2_id} AND \${join_view.pk2_category} = 'TEST' ;;
			}
}

Sample LAMS response:

<tr>
<td>β›”</td>
<td>model:test_model&#47;explore:test_explore&#47;join:join_view </td>
<td>test_explore's PK pk1_id is not used in an equality constraint in join_view join</td>
</tr>
<tr>
<td>β›”</td>
<td>model:test_model&#47;explore:test_explore&#47;join:join_view </td>
<td>join_view's PK pk2_id is not used in an equality constraint in join_view join</td>
</tr>
<tr>
<td>β›”</td>
<td>model:test_model&#47;explore:test_explore&#47;join:join_view </td>
<td>join_view's PK pk2_category is not used in an equality constraint in join_view join</td>
</tr>

Sample failing input 2:

explore: test_explore {
			join: join_view {
					relationship: one_to_many
					type: inner
					sql_on:
					\${test_explore.pk1_id} = \${join_view.pk2_id} AND \${join_view.pk2_category} = 'TEST' ;;
			}
}

Sample LAMS response 2:

<tr>
<td>β›”</td>
<td>model:test_model&#47;explore:test_explore&#47;join:join_view </td>
<td>test_explore's PK pk1_id is not used in an equality constraint in join_view join</td>
</tr>

Introduce/Support Foreign Keys?

We have multiple joins which use the PK field of a given view and the Foreign Key to generate the correct join but LAMS keep saying in those scenarios that no PKs dimensions used for a given view in a given join, an example is the following explore

include: "/views/items.view"
include: "/views/organizations.view"
include: "/views/addresses.view"

explore: invoices {
  view_name: items

  join: organizations {
    type: inner
    relationship: one_to_one
    sql_on: ${organizations.pk1_organization_id} = ${items.organization_id};;
  }

  join: addresses {
    view_label: "Organization Address"
    type:  inner
    relationship: one_to_one
    sql_on: ${organizations.address_id} = ${addresses.pk1_address_id};;
  }
}

For those two joins, it says that for example in the addresses join we didn't use a PK for the organizations view but we don't want to use the PK in this case since we want to join using the address_id on the organization and not his Primary Key.

I think that this could be fixed if LAMS would support/introduce Foreign Key as it does with PrimaryKey (name convention?) and here https://github.com/looker-open-source/look-at-me-sideways/blob/master/rules/e2.js#L80 where it checks if the name matches the PK regex it could also check if not matches the PK regex check if it matches the Foreign Key regex.


Maybe we're missing something in our LookML files but as far as I could understand LAMS always expect an one_to_one join to be between two PKs instead of one PK and an FK.

Rule F3 and drill_fields

Do i still need to specify a filter in the case of something like

  measure: count {
    type: count
    drill_fields: [detail*]
  }

  # ----- Sets of fields for drilling ------
  set: detail {
    fields: [
      pk1_app_foo_relation_id,
      bar_company_code_name,
      foo_detail_name,
      pt_current_term_name,
      company_name,
      bar_name,
      name,
      foo.count
    ]
  }
}

`primary_key` field seems to be ignored. Leads to inconsistencies in errors K1-2

Hello, I came across some unexpected behavior while running LAMS on my code.
I traced it back to what I think are some issues in the code, but I'm not familiar with js (or with LookML for that matter), so I'm not 100% sure.

See the the test.lkml below;

view: my_view {
  sql_table_name: my_db.table ;;
  dimension: something {
    primary_key: no
    hidden: yes
    type: number
    sql: ${TABLE}.id ;;
  }
}

Following the definitions of K1 and K2 as per docs/rules.html

  • βœ”οΈRunning LAMS on this raises a K1... No Primary Key Dimensions found in my_view, which is the expected result.
  • βœ”οΈ Setting primary_key: yes and renaming the dimension to pk1_something raises no error, which is the expected result.
  • ❌ Only setting primary_key: yes still leads to a K1 error, while I would expect it to be a K2.
  • ❌ Only renaming the dimension to pk1_something raises no error, while I would expect a K1

Looking at the code, it looks like the primary_key field is never actually looked at.
In rules/k1-2-3-4.js, primary keys are implicitly defined as fields matching the pkNamingConvention filter, even when $dim.primary_key is false.

(The regex used in pkNamingConvention is itself not consistent with the documentation, see issue #63.)

(If you are open to external PRs, I may have some time for this one)

Output errors to a file & provide a custom manifest.lkml

Hi Team,
Thanks for creating LAMS, it's a wonderful tool that we are planning to use. I had 2 suggestions:

  1. Can we output the errors to a file? I know we can use --output-to-cli but that does not seem to output all the error, instead it caps out at around 300 errors and displays xx more errors at the end. Our use case is that we want to display all the errors at pipeline failures.
  2. Can we provide a manifest.lkml file that is not in the looker project repository? The use case is that we want to have a central manifest.lkml for all our Looker projects that define all our custom linting rules.
    Let me know if we have alternatives for the above two.

Thanks

Custom rule to have description mandatory for all explores

Hi @fabio-looker ,
Currently I have put an exception for rule [F4.]Non-hidden fields should have descriptions. in the manifest file.
I would like to implement a rule to make it mandatory to include description for all explores in every model.
Can I declare this custom rule in manifest file? If yes, would you please help me with the syntax?
If no, then can it be done using javascript ? Also, please help me with that syntax too.

Thanks,
Priyanka

Should this break rule F1?

We have a view from a derived table

view: currency_selector {
  derived_table: {
    sql:
      SELECT 'USD' AS currency
      UNION ALL
      SELECT 'EUR' AS currency
      UNION ALL
      SELECT 'GBP' AS currency
      UNION ALL
      SELECT 'AUD' AS currency
      UNION ALL
      SELECT 'CAD' AS currency;;
  }

  dimension: currency {
    type: string
    hidden: yes
    sql: ${TABLE}.currency ;;
  }

  parameter: currency_selector {
    suggest_dimension: currency
  }


}

which we are using to dynamically populate currency types and symbols in another view(s)

view: payment_ledger {
...
  dimension: general_currency {
    type: number
    sql: CASE WHEN {% parameter currency_selector.currency_selector %} = 'EUR' THEN ${amount_eur}
              WHEN {% parameter currency_selector.currency_selector %} = 'USD' THEN ${amount_usd}
              WHEN {% parameter currency_selector.currency_selector %} = 'GBP' THEN ${amount_gbp}
         ELSE ${amount_usd}
         END;;
    html: {{ currency_symbol._value }}{{ rendered_value }} ;;
    value_format_name: decimal_2
  }

  dimension: currency_symbol{
    type: string
    hidden: yes
    sql: CASE
        WHEN {% parameter currency_selector.currency_selector %} = 'EUR' THEN '€'
        WHEN {% parameter currency_selector.currency_selector %} = 'GBP' THEN 'Β£'
        WHEN {% parameter currency_selector.currency_selector %} = 'USD' THEN '$'
        END;;
  }

  measure: total_other_actual {
    sql:  ${general_currency} ;;
    type:  sum
    value_format_name: big_money
    filters: [currency: "-USD, -EUR, -GBP", ledger_entry_type_description: "Actual"]
    html: {{ currency_symbol._value }}{{ rendered_value }} ;;
  }

would this quality as a tightly coupled view per F1? is there a better way to do this? thanks for the clarification on the last issue btw.

Investigate CI failure

Github Actions is failing with the following error message:

npm ERR! cipm can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.

This is perhaps a new behavior in npm/github actions, but is certainly triggered by my rather creati ve setup npm-shrinkwrap setup...

Disable date in issues.md

We use LAMS with at some repositories where we use pull requests (PR). Each time we close a PR we get merge conflicts with the existing ones caused by the file issues.md.

The problem is that each branch is writting the line with the date and that causes the conflicts. The line is:

<p style="text-align:right;color:#cccs">
Generated Mon, 22 Jun 2020 15:39:09 GMT
</p>

It is possible to configure it so that LAMS does not write the date on the issues.md? I think that the best solution will be to have a parameter to disable the date.

How can you refer to two different paths in an expression?

I am writing a few custom rules that fit our requirements and one of them needs to refer to two paths. I want to check if the type is yesno and if it is, then check if the dimension name ends with a _flag.
I tried changing the path inside the expression but that doesn't work either. Any tips on how to write this?

Exemption for custom rules

Is it possible to set granular exemptions for custom rules? I tried something like this, but it didn't seem to work:

    # LAMS
    # rule_exemptions: {
    #  my_custom_rule: "2018-11-12 - I can't use datagroups for this super special reason and Bob said it's ok"
    # }

Parsing error

hi @fabio-looker -

we're getting this parsing error

Parsing error: Object.fromEntries is not a function

and i can't tell where it's coming from.

im not seeing any errors on the CLI output itself..

Run lams --reporting=save-yes --source **/{views/core/catalog_items/dim_catalog_items.view.lkml,manifest.lkml} || echo "ERROR=true" >> $GITHUB_ENV
Parsing project...
> Issues occurred during parsing (containing files will not be considered. See messages for details.)
> Parsing done!
Checking rules... 
> E1
> E2
> F1
> F2
> F3
> F[4](https://github.com/DrizlyInc/data-looker/runs/5142238622?check_suite_focus=true#step:7:4)
> K1-2-3-4
> T1
> T2-[10](https://github.com/DrizlyInc/data-looker/runs/5142238622?check_suite_focus=true#step:7:10)
> Rules done!
Writing issues.md...
> issues.md done

Screen Shot 2022-02-10 at 9 41 28 AM

any thoughts on what could be going wrong here?

LAMS Erroring

@fabio-looker - apologies for a somewhat direct message here, but I've spent some time digging here and can't seem to find what's going / don't have the relevant expertise. Our last successful lint was 8 days ago (1/20), our next run wasn't until 1/26, but that run and all subsequent have failed with an error that looks like the one below.

I didn't see any new recent lams commits, but based on the error I suspected some issue with the the Object's value's not being available, which I thought might be parser related. I see a commit 7 days ago in lookml-parser. It looks harmless. I re-installed lams locally and confirmed the same error was occurring with a fresh install. I downloaded [email protected] and replaced the contents of the lams lookml-parser directory locally (which had 6.4.1) with the 6.4.0 contents. Unfortunately, this didn't solve the problem. I tried 6.3.0 as well, and rather than an error in F1, it passed through to K1-2-3-4 until erroring with a similar issue.

I'm not quite sure where to go from here, but things seem a bit broken. We rely on some custom lint rules so I hope you can point me in the right direction!

Fresh lams install and 6.4.0 parser error:
image

6.3.0 parser error:
image

New Rule? Equality constraint type checking

Possible enhancement...

As I recently discovered, for strictly typed dialects like BigQuery, updating the type of a column/field can be very difficult since you don't know where that field is referenced in join constraints.

So for example, the following could throw an error:

explore: a {
  join: b {sql_on: ${b.pk1_b_id} = ${a.b_id} ;;}
}
view: a { dimension: b_id {type:string}}
view: b { dimension: pk1_b_id {type:number}}

As stated, this would the first linter-enabled rule that is not strictly locally scoped. (Contrast with PK rules which are mediated by a naming convention, so they are implementable as two separate rules, one which checks local conformance to the naming convention, and another which checks local conformance given the name).

Alternately, we could encourage defensive application of the new LookML type coercion operator (::type), if that operator does not apply unnecessary conversions when the field is already of the desired type, like so:

explore: a {
  join: b {sql_on: ${b.pk1_b_id::number} = ${a.b_id::number} ;;}
}
view: a { dimension: b_id {type:string}}
view: b { dimension: pk1_b_id {type:number}}

I'm not sure which I would recommend, but wanted to create a place to discuss

Add support for folders

Looker 6.16 now allows a folder structure to be present. LAMS currently does not see files that are in subfolders.

GitHub Action Update

Hello,

As of 11/16/2020 GitHub disabled the use of set-env commands (link). This causes LAMS set up through GitHub actions to have issues running properly.

Documentation for LAMS GitHub Actions here

This can be resolved here by updating:
run: lams --reporting=... || echo "::set-env name=ERROR::true"

to
run: lams --reporting=save-no || echo "ERROR=true" >> $GITHUB_ENV

Documentation from GitHub for setting environment variables: here

native Looker validator complains about manifest.lkml

Looker doesn't seem to like the manifest.lkml file :(

Screenshot 2021-04-05 at 16 11 19

name: "XXX"

rule: filter_one {
  description: "yadda yadda"
  match: "$.model.*.explore.*"
  expr_rule: (!== ::match:access_filter undefined) ;; # error is here
}

Is there any way around this?

Short circuit rule if project-level exemption

If an exemption exists at the entire project level, we could check for that at the beginning of our rules files and return early, thus eliminating some looping and improving performance for those cases

Error incorrectly fires if multiple CTEs use --- separator

Thanks @fabio-looker for LAMS, my team at mercari.com has found it really useful to help us write maintainable SQL and LookML.

Ran into an interesting case that my team couldn't figure out until we debugged LAMS. :-)

We expected this to pass the linter:

  WITH
  
  step_1 AS (
    SELECT
      account_id AS pk1_account_id
      ---
    , COUNT(*)
    FROM users
    GROUP BY 1
  )
  
  , step_2 AS (
    SELECT
      pk2_tenant_id
    , pk2_user_id
      ---
    , MAX(start) as last_login
    FROM sessions
    GROUP BY 1,2
  )

  SELECT "foo" as bar

but got an error saying that the second CTE had mismatching primary key field counts:

Primary Key columns in " SELECT pk2_ten" in foo declare 2 column(s), but there are 1

It turned out this works if we use trailing commas, so that's a fine workaround:

  WITH
  
  step_1 AS (
    SELECT
      account_id AS pk1_account_id,
      ---
      COUNT(*)
    FROM users
    GROUP BY 1
  )
  
  , step_2 AS (
    SELECT
      pk2_tenant_id,
      pk2_user_id,
      ---
      MAX(start) as last_login
    FROM sessions
    GROUP BY 1,2
  )

  SELECT "foo" as bar

But the root cause seems to be that the rule parser is only replacing --- once, so it misses the second one. I put together a quick diff w/ a test and will submit a pull request.

How to exempt the `T2` rule?

Thanks for releasing v2! I am trying to introduce LAMS in CI.

First, I tried to exempt all the current errors, but I just couldn't exempt the T2 rule.

Workaround?

According to the source code, I need to exempt all T2~T10. When I exempted all of them, I was able to exempt the error as expected. Is this the intended behavior?

for (let rule of ruleIds) {
rules[rule] = {
globallyExempt: getExemption(project.manifest, rule),
matches: 0,
exemptions: 0,
errors: 0,
};
if (rules[rule].globallyExempt) {
messages.push({
rule, level: 'info', location: 'project',
exempt: rules[rule].globallyExempt,
path: `/projects/${project.name}/files/manifest.lkml`,
description: `Project-level exemption: ${rules[rule].globallyExempt}`,
});
} else {
allExempted = false;
}
}
if (allExempted) {
return {messages};
}

manifest.lkml:

# LAMS
# rule_exemptions: {
#  T2:  "We may have to deal with this later."
#  T3:  "We may have to deal with this later."
#  T4:  "We may have to deal with this later."
#  T5:  "We may have to deal with this later."
#  T6:  "We may have to deal with this later."
#  T7:  "We may have to deal with this later."
#  T8:  "We may have to deal with this later."
#  T9:  "We may have to deal with this later."
#  T10: "We may have to deal with this later."
# }

Exemption per error

If I exempt them in manifest.lkml, I don't notice that the number of errors increases afterward. Therefore, I tried the per-error exemption, but it did not work.

I think the reason is that !exempt('') always returns true. Is it possible to have a per-error exemption for T2 rule?

if (!exempt('')) {
if (actualPkCount === 0) {
rulesInMatch.T2.errored = true;
messages.push({
location, path, rule: 'T2', level: 'error', exempt: exempt('T2'),
description: `No Primary Key columns/selectAliases found in ${view.$name}`,
});
continue;
}
}

Custom rule for type check doesn't work

I currently have 2 custom rules in my manifest file. One works which is the production connection check:

# rule: prod_connection {
#  description: "Models must not use sandbox to deploy"
#  match: "$.model.*"
#  expr_rule: ( $boolean($match "sandbox^" ::match:connection) ) ;;
# }

But the second one doesn't work. It is supposed to check if the type is available, but I tried writing it as the example here:

# rule: types_required {
#	match: "$.model.*.view.*['dimension','measure'].*"
#	description: "Fields must specify a type"
#	expr_rule: ( !== ::match:type undefined ) ;;
# }

But this doesn't work. To make sure that the lookml code isn't the issue, I tried changing the rule slightly since I'm sure I have dimensions and measures that are not type "string":

# rule: fields_must_have_type {
#  description: "All dimensions and measures must have a type defined"
#  match: "$.model.*.view.*['dimension','measure'].*"
#  expr_rule: ( !== ::match:type "string" ) ;;
# }

Do you have any insights on why this rule is not working?
Thank you

Support for extension/refinement

Extended views are currently treated as separate views even when they are inheriting the underlying lookml code from the base view... i.e. if the base view has a description defined, the extended view doesn't require it and the explore will surface the description but lams will still throw an error.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.