databrickslabs / pylint-plugin Goto Github PK

In the current version of Spark Connect, the Spark Connect implementations of the Column and DataFrame classes aren't inherited from the corresponding classes, so the following checks will fail:

from pyspark.sql import DataFrame, Column

if isinstance(cl, Column):
  ...

if isinstance(df, DataFrame):
  ....

Making it harder to port code to UC Shared clusters and serverless notebooks/workflows...

Is there a way to customize what is marked as incompatible with UC and/or a way to mark a line to be skipped during the check?

One of our tasks imports boto3 to remove files from S3, but it uses secrets set on the cluster to create the session, so it's totally compatible. Come to think of it, I'm not actually sure what legacy behavior this is checking for, was boto3 used for hive metastore operations?

PS: Love the idea of this project, I was recently bit several times by inadvertent global spark variables and have been checking things manually since then, but I'll be adding this to my CI pipelines for this going forward.

Suggest rewrites with df.transform(x)

iterators cannot be consumed twice, which may lead to bugs if iterators are passed as arguments

• Put
• Mount
• Head

Strings like 'foo ' 'bar' resolve to 'foo bar', but are very confusing to read:

'The use of default dbfs: references is deprecated: ' '/mnt/things/e/f/g',

Passing an empty collection is better than passing an optional collection

%pip install pytest
%pip install databricks-labs-pylint...whl
%sh pylint ...

S3, boto3, etc

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

It would be good to be consistent across detection and remediation recommendations between the newly released https://pypi.org/project/databricks-labs-pylint/ and the UCX.

Proposed Solution

Review pylint & UCX detection capabilities & recommendations, research with Product and Field Eng and change assessment.md, detection queries & views or identify changes to the databricks-labs-pylint project.

Additional Context

No response

Allow list of good locations

Less readable:

self.reader(table_query)
.options(**self._get_jdbc_reader_options(jdbc_reader_options) | self._get_timestamp_options())

More readable:

options = self._get_jdbc_reader_options(jdbc_reader_options) | self._get_timestamp_options()
self.reader(table_query).options(**options)

• Data security mode not set