inspera / blackbricks Goto Github PK
View Code? Open in Web Editor NEWBlack for Databricks notebooks
License: MIT License
blackbricks's People
Contributors
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Stargazers
Watchers
blackbricks's Issues
Troubleshooting failed parse
When running blackbricks, version 0.6.7 on a file that cannot be parsed, instead of handling the error I am getting this error.
Traceback (most recent call last):
File "C:\Users\xyz\.conda\envs\abc123\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\xyz\.conda\envs\abc123\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\xyz\.conda\envs\abc123\Scripts\blackbricks.exe\__main__.py", line 7, in <module>
File "C:\Users\xyz\.conda\envs\abc123\lib\site-packages\typer\main.py", line 214, in __call__
return get_command(self)(*args, **kwargs)
File "C:\Users\xyz\.conda\envs\abc123\lib\site-packages\click\core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "C:\Users\xyz\.conda\envs\abc123\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "C:\Users\xyz\.conda\envs\abc123\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\xyz\.conda\envs\abc123\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "C:\Users\xyz\.conda\envs\abc123\lib\site-packages\typer\main.py", line 497, in wrapper
return callback(**use_params) # type: ignore
File "C:\Users\xyz\.conda\envs\abc123\lib\site-packages\blackbricks\cli.py", line 183, in main
n_changed_files = process_files(
File "C:\Users\xyz\.conda\envs\abc123\lib\site-packages\blackbricks\cli.py", line 25, in process_files
content = file_.content
File "C:\Users\xyz\.conda\envs\abc123\lib\site-packages\blackbricks\files.py", line 31, in content
return f.read()
File "C:\Users\xyz\.conda\envs\abc123\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]Is there some verbose command that can be used to troubleshoot which file is causing the issue in the repo?
Can this error be handled safely as part of returning the reader?
blackbricks expects username and password lines in ~/.databrickscfg
My ~/.databrickscfg neither has a username = <something>, nor a password = <something> line. It was created via databricks configure --token as described in the Databricks docs and only features a host = <something> and a token = <something> line in the DEFAULT profile.
Apparently blackbricks expects the username and password lines to be present in the .databrickscfg profile as it crashes in their absence when trying to regex match the username and the password.
Format a string of sql commands contained as text in a spark.sql command.
When I run blackbricks on a notebook that has a spark.sql command with a test string like the code in the picture, it brings the top line down and over to the left but the rest of the code is left untouched. It would be nice if the rest of the sql command was formatted and brought over to the same indentation.
Could the entire sql string be parsed using parsesql and then indented?
Only include .py files
Hi,
It seems that blackbricks right now runs on every file (I get the following error, see below).
InvalidInput: Cannot parse: 1:0: !/dbfs/databricks/init_scripts/install_prod_artifacts.sh
Is it possible in any way to only run blackbricks on .py files?
Question: Is it possible to relax the typer version?
By fixing the typer version to ^0.6.1 I cannot use the latest version of dbx, which requires typer==0.70 (https://github.com/databrickslabs/dbx/blob/main/setup.py). This in turn results in a critical dependabot alert (CVE-2023-0286) for the cryptography module...
cannot parse pip install commands in the notebook
I have this in my beginning of the databricks notebook to run the pip install
# Databricks notebook source
%pip install pyyaml # noqa: E999
# COMMAND ----------
dbutils.library.restartPython()
After running blackbricks via pre-commit, it fail on the pip install command. I also try without % but still not working.
error message
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /Users/arkliu/.cache/pre-commit/repoc8laulmt/py_env-python3/lib/python3.9/si │
│ te-packages/blackbricks/cli.py:197 in main │
│ │
│ 194 │ else: │
│ 195 │ │ files = [LocalFile(fname) for fname in resolve_filepaths(filen │
│ 196 │ │
│ ❱ 197 │ n_changed_files = process_files( │
│ 198 │ │ files, │
│ 199 │ │ format_config=FormatConfig(line_length=line_length, sql_upper= │
│ 200 │ │ diff=diff, │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ check = False │ │
│ │ databricks_profile = 'DEFAULT' │ │
│ │ diff = False │ │
│ │ filenames = [] │ │
│ │ files = [ │ │
│ │ │ LocalFile( │ │
│ │ │ │ │ │
│ │ path='/Users/arkliu/github/my_project_path/de-high… │ │
│ │ │ ), │ │
│ │ │ LocalFile( │ │
│ │ │ │ │ │
│ │ path='/Users/arkliu/github/my_project_path/de-gene… │ │
│ │ │ ) │ │
│ │ ] │ │
│ │ line_length = 88 │ │
│ │ remote_filenames = False │ │
│ │ sql_upper = True │ │
│ │ version = None │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /Users/arkliu/.cache/pre-commit/repoc8laulmt/py_env-python3/lib/python3.9/si │
│ te-packages/blackbricks/cli.py:45 in process_files │
│ │
│ 42 │ │ │ continue │
│ 43 │ │ │
│ 44 │ │ n_notebooks += 1 │
│ ❱ 45 │ │ output = format_str(content, config=format_config) │
│ 46 │ │ │
│ 47 │ │ no_change &= output == content │
│ 48 │ │ n_changed_files += output != content │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ check = False │ │
│ │ content = '# Databricks notebook source\npip install pyyaml # │ │
│ │ noqa: E999\n\n# COMMAND -------'+5193 │ │
│ │ diff = False │ │
│ │ file_ = LocalFile( │ │
│ │ │ │ │
│ │ path='/Users/arkliu/github/my_project_path/de-general… │ │
│ │ ) │ │
│ │ files = [ │ │
│ │ │ LocalFile( │ │
│ │ │ │ │ │
│ │ path='/Users/arkliu/github/my_project_path/de-highly-… │ │
│ │ │ ), │ │
│ │ │ LocalFile( │ │
│ │ │ │ │ │
│ │ path='/Users/arkliu/github/my_project_path/de-general… │ │
│ │ │ ) │ │
│ │ ] │ │
│ │ format_config = FormatConfig(line_length=88, sql_upper=True) │ │
│ │ n_changed_files = 0 │ │
│ │ n_notebooks = 2 │ │
│ │ no_change = True │ │
│ │ output = '# Databricks notebook source\nimport dlt\nimport │ │
│ │ pyspark.sql.functions as F\n\n# COM'+1989 │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /Users/arkliu/.cache/pre-commit/repoc8laulmt/py_env-python3/lib/python3.9/si │
│ te-packages/blackbricks/blackbricks.py:49 in format_str │
│ │
│ 46 │ │ │ output_cells.append(cell) # Generic magic cell - output a │
│ 47 │ │ else: │
│ 48 │ │ │ output_cells.append( │
│ ❱ 49 │ │ │ │ black.format_str( │
│ 50 │ │ │ │ │ cell, mode=black.FileMode(line_length=config.line_ │
│ 51 │ │ │ │ ) │
│ 52 │ │ │ ) │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ cell = 'pip install pyyaml # noqa: E999' │ │
│ │ cells = [ │ │
│ │ │ '\npip install pyyaml # noqa: E999\n\n', │ │
│ │ │ '\n\ndbutils.library.restartPython()\n\n', │ │
│ │ │ '\n\nimport json\nimport logging\nfrom pprint import │ │
│ │ pprint\n\nimport requests\nimport y'+5, │ │
│ │ │ "\n\n# TODO: replace Ark's token with the token from │ │
│ │ machine\n# TODO: move the secre"+329, │ │
│ │ │ '\n\ndatabricks_oauth_base_url = │ │
│ │ "https://accounts.cloud.databricks.com/oidc/accoun'+169, │ │
│ │ │ '\n\n# │ │
│ │ https://docs.databricks.com/en/dev-tools/authentication-… │ │
│ │ │ '\n\n# get user setting\nresponse = requests.get(\n │ │
│ │ url=f"{databricks_account_leve'+243, │ │
│ │ │ '\n\n# get group setting\nresponse = requests.get(\n │ │
│ │ url=f"{databricks_account_lev'+316, │ │
│ │ │ '\n\n# get YAML setting from Github\nres = │ │
│ │ requests.get(\n url="https://api.github'+567, │ │
│ │ │ '\n\n# find members delta for group defined in the │ │
│ │ YAML file\nfor key, value in grou'+1889, │ │
│ │ │ ... +2 │ │
│ │ ] │ │
│ │ config = FormatConfig(line_length=88, sql_upper=True) │ │
│ │ content = '\npip install pyyaml # noqa: E999\n\n# COMMAND │ │
│ │ ----------\n\ndbutils.library.restart'+5165 │ │
│ │ output_cells = [] │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /Users/arkliu/github/my_project_path/src/black/__init__.py:1079 in │
│ format_str │
│ │
│ [Errno 2] No such file or directory: │
│ '/Users/arkliu/github/my_project_path/src/black/__init__.py' │
│ │
│ /Users/arkliu/github/my_project_path/src/black/__init__.py:1089 in │
│ _format_str_once │
│ │
│ [Errno 2] No such file or directory: │
│ '/Users/arkliu/github/my_project_path/src/black/__init__.py' │
│ │
│ /Users/arkliu/github/my_project_path/src/black/parsing.py:89 in │
│ lib2to3_parse │
│ │
│ [Errno 2] No such file or directory: │
│ '/Users/arkliu/github/my_project_path/src/black/parsing.py' │
╰──────────────────────────────────────────────────────────────────────────────╯
InvalidInput: Cannot parse: 1:4: pip install pyyaml # noqa: E999
I do read the doc but cannot find anything that allows me to skip or bypass this line.
The error message is InvalidInput: Cannot parse suggests this package just will not work with a notebook with pip install.
If support ( or skip ) pip install command line is align with the roadmap, I can try to find time to contribute to it.
Add `isort` to be executed by blackbrick
Hi again, here is another feature request 😃
I think blackbrick can benefit from also executing isort for sorting Python imports to a best practice.
isort can be executed in Python on a string of code, e.g. from a notebook cell by using isort.code("import b\nimport a\n").
What is the opinion on adding more formatters than just black and sqlparser?
Blackbricks forces indentation of two spaces, even when user/notebook setting set to four spaces
CI/CD integration ?
Hi folks!
I didn't find any information related to "run blackbricks automatically in your CI/CD piepline".
Any ideas or experiences out there?
I was thinking about a Git hook ... pre-push would be fine from my point of view.
Dear @bsamseth, should my question be wrong here, please feel free to correct me and point me to the right location!
Dejan Hrubenja [email protected] , Mercedes-Benz Tech Innovation GmbH
https://www.mercedes-benz-techinnovation.com/en/imprint/
TypeError: __init__() got an unexpected keyword argument 'no_args_is_help'
blackbricks --remote /Users/[email protected]/Alcanna/Generate_Sale_Payment_Line
Traceback (most recent call last):
File "/Users/henokyemam/opt/anaconda3/envs/experiment/bin/blackbricks", line 8, in
sys.exit(app())
File "/Users/henokyemam/opt/anaconda3/envs/experiment/lib/python3.6/site-packages/typer/main.py", line 214, in call
return get_command(self)(*args, **kwargs)
File "/Users/henokyemam/opt/anaconda3/envs/experiment/lib/python3.6/site-packages/typer/main.py", line 239, in get_command
click_command = get_command_from_info(typer_instance.registered_commands[0])
File "/Users/henokyemam/opt/anaconda3/envs/experiment/lib/python3.6/site-packages/typer/main.py", line 442, in get_command_from_info
deprecated=command_info.deprecated,
TypeError: init() got an unexpected keyword argument 'no_args_is_help'
Format SQL notebooks
Blackbricks is currently designed to only format Python notebooks (text files with a # Databricks notebook source header). I'd like to implement formatting of SQL notebooks (-- Databricks notebooks source header), as well. I will try to write some new tests and will definitely clarify this change in the docs.
What do you think? Does this align with your vision for Blackbricks?
Shell commands throw parse error
Cells with shell commands throw a parse error in build 0.6.4.
To reproduce this issue create a single cell in a notebook with the following contents !ls and run blackbricks on it.
Here's a snippet of the error:
File "/usr/local/opt/pyenv/versions/3.8.12/envs/test/lib/python3.8/site-packages/blackbricks/cli.py", line 184, in main
n_changed_files = process_files(
File "/usr/local/opt/pyenv/versions/3.8.12/envs/test/lib/python3.8/site-packages/blackbricks/cli.py", line 32, in process_files
output = format_str(content, config=format_config)
File "/usr/local/opt/pyenv/versions/3.8.12/envs/test/lib/python3.8/site-packages/blackbricks/blackbricks.py", line 111, in format_str
black.format_str(
File "src/black/__init__.py", line 1131, in format_str
File "src/black/__init__.py", line 1141, in _format_str_once
File "src/black/parsing.py", line 128, in lib2to3_parse
black.parsing.InvalidInput: Cannot parse: 1:0: !ls
Discussion: sqlparse vs sqlfluff
Hi, first of all, thanks for developing blackbricks, and enabling better software practices for notebooks 😄
I have been interested in Python code formatting packages for the last couple of years and have used some in different small projects.
Currently, I'm developing a "data platform" for my company which mainly builds on top of Databricks. As part of this platform, I implemented pre-commit and CI validation of our code formats, including Python and SQL script and in Databricks notebooks.
After investigating the area of different open-source SQL formatters, I decided to use sqlfluff for my company SQL scripts and blackbricks for our Databricks notebooks, which (as you know) uses sqlparser.
My question is: Why use sqlparse instead of sqlfluff?
sqlfluff procs:
- The community is bigger, as seen by the activity score and stars difference, see a comparison here: https://www.libhunt.com/compare-sqlfluff-vs-sqlparse. A bigger community results in more contributions, and more tests of use cases and edge cases.
- As sqlfluff covers multiple SQL dialects, thus blackbrick can also better support different dialects, as Databricks often operate different SQL engines.
- Sqlfluff already has many of the features sqlparser does not have yet, e.g. configuration via config file; #31 (comment)
- Sqlfluff has better documentation, possibly due to a bigger community.
sqlfluff cons:
- As the community and code base is larger than sqlparser, changes can be more complex to add and more time costly.
- As sqlfluff covers multiple SQL dialects, thus a decision and solution has to be developed to fully "automate" the "Format notebooks" action in databricks notebooks.
Solution direction?
I think that sqlfluff can be substituted with sqlparser, using it as python package and running sqlfluff.fix(my_bad_query, dialect="databricks"), see this example https://github.com/sqlfluff/sqlfluff/blob/a20005a6647dbd7fc4ac228897db5b0d776defa7/examples/01_basic_api_usage.py#L26.
To decide the dialect I see 3 options blackbricks has:
- assume the dialect always is "databricks", "SparkSQL", or "ANSI". ANSI will probably not provide any errors, but has to be tested.
- develop some code in sqlfluff or blackbricks that with an heuristic guess the dialect.
- blackblicks enables the user via a comment in SQL cells and code snippets to specify the dialect. Got inspiration from sqlfluff' inline config, see https://docs.sqlfluff.com/en/stable/configuration.html#in-file-configuration-directives
I'm looking forward to discussing this 😃
AttributeError: module 'black' has no attribute 'is_docstring'
when i tried in a new virtualenv with black==21.4b0 and blackbricks, version 0.6.2
$ blackbricks --check /Users/natarajan.chakrapani/repos/prodigy-databricks/notebooks/ETL/jdbc_query_to_lake.py
Traceback (most recent call last):
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/bin/blackbricks", line 8, in <module>
sys.exit(app())
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/typer/main.py", line 214, in __call__
return get_command(self)(*args, **kwargs)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/typer/main.py", line 497, in wrapper
return callback(**use_params) # type: ignore
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/blackbricks/cli.py", line 192, in main
check=check,
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/blackbricks/cli.py", line 32, in process_files
output = format_str(content, config=format_config)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/blackbricks/blackbricks.py", line 112, in format_str
cell, mode=black.FileMode(line_length=config.line_length)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/black/__init__.py", line 832, in format_str
for current_line in lines.visit(src_node):
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/black/nodes.py", line 164, in visit
yield from self.visit_default(node)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/black/linegen.py", line 101, in visit_default
yield from super().visit_default(node)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/black/nodes.py", line 170, in visit_default
yield from self.visit(child)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/black/nodes.py", line 162, in visit
yield from visitf(node)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/black/linegen.py", line 171, in visit_simple_stmt
yield from self.visit_default(node)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/black/linegen.py", line 101, in visit_default
yield from super().visit_default(node)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/black/nodes.py", line 170, in visit_default
yield from self.visit(child)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/black/nodes.py", line 164, in visit
yield from self.visit_default(node)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/black/linegen.py", line 101, in visit_default
yield from super().visit_default(node)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/black/nodes.py", line 170, in visit_default
yield from self.visit(child)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/black/nodes.py", line 164, in visit
yield from self.visit_default(node)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/black/linegen.py", line 101, in visit_default
yield from super().visit_default(node)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/black/nodes.py", line 170, in visit_default
yield from self.visit(child)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/black/nodes.py", line 164, in visit
yield from self.visit_default(node)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/black/linegen.py", line 101, in visit_default
yield from super().visit_default(node)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/black/nodes.py", line 170, in visit_default
yield from self.visit(child)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/black/nodes.py", line 162, in visit
yield from visitf(node)
File "/Users/natarajan.chakrapani/repos/prodigy-databricks/test/lib/python3.7/site-packages/blackbricks/blackbricks.py", line 57, in patch_visit_STRING
if black.is_docstring(leaf) and "\\\n" not in leaf.value:
AttributeError: module 'black' has no attribute 'is_docstring'However
if i i tried in a new virtualenv with a previous black version black==20.8b1 and blackbricks, version 0.6.2 this works
$ blackbricks --check /Users/natarajan.chakrapani/repos/prodigy-databricks/notebooks/ETL/jdbc_query_to_lake.py
would reformat /Users/natarajan.chakrapani/repos/prodigy-databricks/notebooks/ETL/jdbc_query_to_lake.py
All done!
1 files would be reformattedFR: Enable blackbricks to use most of sqlparse options
Relates to:
- Metadata-Version: 2.1
- Name: blackbricks
- Version: 1.0.1
AS-IS: (in blackbricks.py)
sqlparse.format(
"\n".join(sql_lines), reindent=True, keyword_case=sql_keyword_case
- reindent
- is set to True always when sqlparse is called by blackbricks
- keyword_case
- defaults to keyword_case=upper, and can be influenced using --no-sql-upper … in this case keyword_case=lower will be used
TO-BE: (requested feature)
-
Improve blackbricks to be able to set/use (nearly) all sqlparse options. For this, please:
- implement additional sqlparse-options which can be used in blackbricks
- pass them to sqlparse
- add default value(s) for these sqlparse options (similar to: --sql-upper is default (upper(KEYWORDS), but one can optionally overwrite this using --no-sql-upper (lower(KEYWORDS)) in order to ensure downwards comaptibility
- so ... nobody using blackricks already would need to change anything, because blackbricks would set DEFAULT sqlparse options by default!
- BTW, I've already checked sqlparse for DEFAULT values ... and I've put them into the list beneath as <DEFAULT_VALUE> is DEFAULT in sqlparse
-
Here's the list of desired/commonly-used options when formatting SQL source code:
- --sql-identifier-case
- Allow: upper, lower, capitalize
- default = n/a
- None is DEFAULT in sqlparse?!?
- --sql-strip_comments
- Allow: True, False
- default = False
- False is DEFAULT in sqlparse
- --sql-reindent
- Allow: True, False
- default = False
- False is DEFAULT in sqlparse
- --sql-reindent_aligned
- Allow: True, False
- default = False
- False is DEFAULT in sqlparse
- --sql-use_space_around_operators
- Allow: True, False
- default = False
- False is DEFAULT in sqlparse
- --sql-indent_tabs
- Allow: True, False
- default = False
- False is DEFAULT in sqlparse
- --sql-indent_width
- Allow: smallint or 1..8 or 1..256
- default = 2
- 2 is DEFAULT in sqlparse
- --sql-wrap_after
- Allow: smallint or 1..8 or 1..256
- default = 0
- 0 is DEFAULT in sqlparse
- --sql-comma_first
- Allow: True, False
- default = False
- False is DEFAULT in sqlparse
- --sql-identifier-case
Dejan Hrubenja [email protected] Mercedes-Benz Tech Innovation GmbH
https://github.com/mercedes-benz/foss/blob/master/PROVIDER_INFORMATION.md
Titles for %sql commands are not correctly parsed
Hi! I'm using Azure Databricks and I give most of my commands a title.
Also the sql commands.
Unfortunately those titles are not correctly parsed.
I am not sure what is causing this.. in the example below you can see that the title was "Merge Silver view into Gold".
But it's happening for all the sql commands.
Please see the screenshot below.
Add isort for import cells
I've been using blackbricks extensively with my remote Databricks notebooks, and I would like to add isort so it can also reformat cells with imports.
Any hints about how to proceed?
No newline at eol in formatted notebooks
We use the new black formatting in databricks notebooks which adds a newline at the end of each notebook. We also use blackbricks in our CI/CD pipeline to check for formatting issues. To make sure that our CI/CD passes we tend to checkout our changes locally just to run blackbricks locally before we run our pipeline, which is quite tedious.
A potential fix #36
Can you run blackbricks while using databricks in the browser?
For those without a CLI setup, could you run this in the browser version of databricks?
My thought would be blackbricks.format_notebook('notebook_path', ['notebook_dest']) with ['notebook_dest'] being an optional argument with the default being notebook_path to just update the current notebook.
AttributeError: module 'black' has no attribute 'get_string_prefix'
Hi,
Closely related to this issue.
Would need to change black.get_string_prefix to black.strings.get_string_prefix in line 60 of blackbricks.py
Might be worth checking if any other issues may arise from recent black restructuring.
Would be great to have this updated as we are using your library quite extensively, great work, thanks!
Remove empty cells when run format
Hi, I have a feature request.
I think that blackbrick should remove empty cells. Meaning remove cells containing:
- only whitespaces
- only the format magic specification e.g. "MAGIC %sql" or "MAGIC %python" ect.
Empty cells are often added by mistake to the end of a notebook when one just spams Shift+Enter to run all cells indevidually.
I suppose this can be implemeted "simply" in
blackbricks/blackbricks/blackbricks.py
Lines 47 to 50 in 6be4e1f
output_cells if it do not satisfy the conditions stated above.
Looking forward to hear your opinion.
Combine black and blackbricks?
Hi,
I have a large directory with a combination of regular python files and some databricks notebooks.
Do you have any example of how to run black on the normal python files and blackbricks on the databricks notebooks?
Would be nice to have a small option in blackbricks to run both :)
Cheers,
black.InvalidInput: Cannot parse
I'm trying to format a notebook and it's failing. Could it be because I've used markdown cells?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.