datathirstltd / dbfs-explorer Goto Github PK

View Code? Open in Web Editor NEW

65.0 6.0 26.0 7.08 MB

DBFS Explorer for Databricks

Home Page: https://datathirst.net/projects/dbfs-explorer/

JavaScript 53.69% Vue 45.80% EJS 0.52%

dbfs-explorer's Introduction

DEPRECATED

This code is no longer maintained.

dbfs_explorer

DBFS EXPLORER

Build Setup

# install dependencies
npm install

# serve with hot reload at localhost:9080
npm run dev

# build electron application for production
npm run build


# lint all JS/Vue component files in `src/`
npm run lint

dbfs-explorer's People

Contributors

Stargazers

Watchers

dbfs-explorer's Issues

Drag and drop on Windows fails

I'm unable to upload files by dragging in from Windows explorer. It works fine on a mac from Finder.

Unable to Paste Token

On a Mac you cannot paste a token into the Bearer token field on the first authentication screen. For some reason paste seems to be disabled.

Cannot find latest-linux-yml

When running v0.1.3 built on Linux, I get the following error:

Checking for update
Error: Error: Cannot find latest-linux.yml in the latest release artifacts (https://github.com/DataThirstLtd/DBFS-Explorer/releases/download/v0.1.3/latest-linux.yml): HttpError: 404 Not Found
"method: GET url: https://github.com/DataThirstLtd/DBFS-Explorer/releases/download/v0.1.3/latest-linux.yml\n\nPlease double check that your authentication token is correct. Due to security reasons actual status maybe not reported, but 404.\n"

If I use the download button from the menu bar it popups the select folder dialog - and the downloads screen but does not actually down the file.
If I right click and download it works.
If I use the download button after downloading a file by right click it works.
Only seems to be a problem when you first open the application.

Not able to open app in Mac

Its showing macOS cannot verify that this app is free from malware.

Also, tried to allow app as open anyway from security & privacy. Yet giving same error.

Also tried as ctrl + click and select open option from menu. Getting same error.

Implement Main Screen

Add base64 support, mainly, for images

native support for snappy.parquet compression

Love this tool already. Would love to see support snappy compression into flat text.

Thanks so much

Implement activity screen extendable from bottom toolbar.

Preview Play audio type files

Able to play audio type files prior to downloading it.

Auto updater

Implement Auto update GitHub provider.

Application shortcuts

Navigator:

Able to select all files and folders. Suggested Shortcut key: CmdOrCtrl+A
Able to view properties from the selected file(s). Suggested Shortcut key: CmdOrCtrl+I
Able to move around files/folders using keyboard arrow keys.
Able to open selected folder using keyboard enter ↵ key.
Able to delete selected file/folder using keyboard Delete DEL key.
Able to create new folder on current working directory. CmdOrCtrl+N

Application Level

Able to Toggle Download/Upload dialog. Suggested Shortcut key: CmdOrCtrl+J

Add pool based thread library

Edit files

It'd be great to be able to edit files and automatically reupload them
For example, you right click on a file, select "edit", then it gets downloaded and a notepad is opened. You edit the file, save it and automatically (or after pressing some button) it gets reuploaded to the dbfs

Convert base-64 files into regular files after download

Refresh feauture

It would be great to have 'Refresh' feature to refresh current view.

Support AWS URL

Can you add an option to select AWS or Azure on the login screen. This needs to affect the URL for the API.

If they select AWS then the URL is .cloud.databricks.com
If they select Azure then the URL is .azuredatabricks.net

Maybe display the domain ".cloud.databricks.com" or ".azuredatabricks.net" after the first textbox as a hint to the user as to what they should enter.

Last request then we are done!

Add download upload activity

Preview image type files

Able to preview image type files without downloading.

Download issues

while downloading a file it always downloads half of the file lets say 6mb of 29mb and says completed

list view

It would be great if the tool had a list-view. We have many files in dbfs that have the same prefix and so the names all look the same in the UI. List view also makes it quicker to browse directorie with a lot of files

Windows Build Failing

Hey @saikksub - I'm getting this error when I run the build on Windows. Have some new Python dependancies been added? I'm not sure where to look to fix this. The mac build runs fine.

Failed to execute 'C:\Program Files\nodejs\node.exe C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js configure --fallback-to-build --module=F:\Builds\_work\8\s\node_modules\sqlite3\lib\binding\electron-v5.0-win32-x64\node_sqlite3.node --module_name=node_sqlite3 --module_path=F:\Builds\_work\8\s\node_modules\sqlite3\lib\binding\electron-v5.0-win32-x64 --napi_version=3 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=electron-v5.0' (1)

Error output:
node-pre-gyp WARN Using request for node-pre-gyp https download 
node-pre-gyp WARN Tried to download(403): https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v4.0.6/electron-v5.0-win32-x64.tar.gz 
node-pre-gyp WARN Pre-built binaries not found for [email protected] and [email protected] (electron-v5.0 ABI, unknown) (falling back to source compile with node-gyp) 
gyp ERR! configure error 
gyp ERR! stack Error: Command failed: C:\ProgramData\Miniconda3\python.EXE -c import sys; print "%s.%s.%s" % sys.version_info[:3];
gyp ERR! stack   File "<string>", line 1
gyp ERR! stack     import sys; print "%s.%s.%s" % sys.version_info[:3];
gyp ERR! stack                                ^
gyp ERR! stack SyntaxError: invalid syntax
gyp ERR! stack 
gyp ERR! stack     at ChildProcess.exithandler (child_process.js:289:12)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at maybeClose (internal/child_process.js:962:16)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:251:5)
gyp ERR! System Windows_NT 10.0.14393
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "configure" "--fallback-to-build" "--module=F:\\Builds\\_work\\8\\s\\node_modules\\sqlite3\\lib\\binding\\electron-v5.0-win32-x64\\node_sqlite3.node" "--module_name=node_sqlite3" "--module_path=F:\\Builds\\_work\\8\\s\\node_modules\\sqlite3\\lib\\binding\\electron-v5.0-win32-x64" "--napi_version=3" "--node_abi_napi=napi" "--napi_build_version=0" "--node_napi_label=electron-v5.0"
gyp ERR! cwd F:\Builds\_work\8\s\node_modules\sqlite3
gyp ERR! node -v v10.13.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 
node-pre-gyp ERR! build error 
node-pre-gyp ERR! stack Error: Failed to execute 'C:\Program Files\nodejs\node.exe C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js configure --fallback-to-build --module=F:\Builds\_work\8\s\node_modules\sqlite3\lib\binding\electron-v5.0-win32-x64\node_sqlite3.node --module_name=node_sqlite3 --module_path=F:\Builds\_work\8\s\node_modules\sqlite3\lib\binding\electron-v5.0-win32-x64 --napi_version=3 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=electron-v5.0' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (F:\Builds\_work\8\s\node_modules\node-pre-gyp\lib\util\compile.js:83:29)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:962:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:251:5)
node-pre-gyp ERR! System Windows_NT 10.0.14393
node-pre-gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "F:\\Builds\\_work\\8\\s\\node_modules\\node-pre-gyp\\bin\\node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd F:\Builds\_work\8\s\node_modules\sqlite3
node-pre-gyp ERR! node -v v10.13.0
node-pre-gyp ERR! node-pre-gyp -v v0.11.0
node-pre-gyp ERR! not ok 
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-pre-gyp install --fallback-to-build`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

Multi-process base64 offset downloads

Current Implementation

Download base64 files with offset synchronously.

Proposed Implementation

Calculate offset index points of a target file and download each offset using Multi-process/threads.