dbcli / athenacli Goto Github PK

View Code? Open in Web Editor NEW

206.0 206.0 31.0 1019 KB

AthenaCLI is a CLI tool for AWS Athena service that can do auto-completion and syntax highlighting.

Home Page: https://athenacli.readthedocs.io/

License: BSD 3-Clause "New" or "Revised" License

Python 99.87% Dockerfile 0.13%

athena-cli autocompletion aws-athena aws-cli bigdata cli command-line python syntax-highlighting

athenacli's People

Contributors

Stargazers

Watchers

athenacli's Issues

When creating the config file, ask user for mandatory details in that session itself

Current behavior:

When the user runs the program, a check is made to see if a configuration exists.
If not, a new configuration file is generated at ~/.athenacli/athenaclirc.
The user has to manually go and update this file for at least the mandatory parameters:
i. aws_access_key_id
ii. aws_secret_access_key
iii. region
iv. s3_staging_dir

Proposed behavior:

When the user runs the program for the first time, a check is made to see if a configuration exists.
If not, the user is asked to (optionally?) provide the parameters mentioned above.
After the user provides the above parameters, the configuration file is generated at ~/.athenacli/athenaclirc with the above parameters filled with the user's input.

Example requested: how to escape `where`

I'd like to be able to execute the following query with -e, without writing a sql file. What's the best way to do this? Because I need both " and ' I'm unsure how to escape in the context of -e 'my query string here'.

select *
from "AwsDataCatalog"."database"."table"
where "column" = 'value'

Naively executing this just returns a confusing error (it's mistaking my value for a column):

COLUMN_NOT_FOUND: line 1:190: Column 'value' cannot be resolved or requester is not authorized to access requested resources

Homebrew option

I'd love to install the athenacli using homebrew instead of pip so it doesn't clutter global dependencies

brew install athenacli

`sqlparse.compat` is not a module in `sqlparse` `0.4.x`

Trying to start athenacli when installed with the latest sqlparse leads to the following error:

from sqlparse.compat import text_type
ModuleNotFoundError: No module named 'sqlparse.compat'

Create a Homebrew formula for athenacli

Add prompt_toolkit 2.0 support

IAM roles adjustments for security

Hi,
I would like to limit an ec2 instance to be able to use athenacli and view only specific resources.
To be more specific, I would like the following:

when executing \l within athenacli, it would display only a set of allowed databases (And of course be able to use only them)
When using that same database, and executing \dt, it would display only a set of allowed tables within that database (And of course be able to query, and \dt <table> to only them)

I'm guessing that I could filter that using an appropriate IAM role for the instance.
Do you have maybe an example of how to do such a thing?

Thanks

Ctrl-C to abort query

I wanted to cancel an executing query, and tried ctrl-c. However, this quit the CLI altogether. I suggest ctrl-c during a query should abort it without quitting.

Table name starting with a digit causes the code completion to fail

# athenacli aleksei_ds_parity
us-east-1:aleksei_ds_parity> Exception in thread completion_refresh:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/athenacli/completion_refresher.py", line 71, in _bg_refresh
    refresher(completer, executor)
  File "/usr/local/lib/python3.9/site-packages/athenacli/completion_refresher.py", line 113, in refresh_tablespletions...                                                                                                                                                                                                                                                                                                                              
    completer.extend_columns(executor.table_columns(), kind='tables')
  File "/usr/local/lib/python3.9/site-packages/athenacli/completer.py", line 124, in extend_columns
    metadata[self.dbname][relname].append(column)
KeyError: '"2a"'

Probably doesn't matter - but I am running it against Athena:

pip install athenacli

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SESSION_TOKEN=...
export AWS_DEFAULT_REGION=us-east-1
export AWS_ATHENA_S3_STAGING_DIR=s3://my-bucket/athena
export AWS_ATHENA_WORK_GROUP=primary

athenacli aleksei_ds_parity

Add more tests

I haven't found an easy way to test Athena locally. It seems moto is a good way to test it, but it doesn't support Athena now, maybe we can spend some time to make moto support Athena if nobody takes that. Anyway, I'd appreciate it if you have any ideas about testing Athena locally.

No row count information

Because PyAthena's cursor doesn't support rowcount information. We can fetch all the rows and count them.

Add Dockerfile

Running athenacli without configuration changes causes visible Exception

When I don't set the values of the AWS configuration parameters, there is an exception thrown in the console causing the CLI to exit.

Perhaps it would be better to have an exception handling routine to handle this scenario. Maybe something similar to what we are trying to do in #15

At the very least, it would be great to avoid the exception by showing an error message.

Error when typing a query

I started writing this query:

SELECT d.* FROM doodads d JOIN trinkets t on d.

`JOIN` doesn't appear

JOIN did not auto-complete as an option (I only saw JSON)

Unhandled Exception

When I type the . at the end (I'm trying to join on d.trinket_id = doodad.id) I get the following output:

Unhandled exception in event loop:
  File "/home/admin/.local/lib/python3.7/site-packages/prompt_toolkit/eventloop/coroutine.py", line 92, in step_next
    new_f = coroutine.throw(exc)
  File "/home/admin/.local/lib/python3.7/site-packages/prompt_toolkit/buffer.py", line 1654, in new_coroutine
    yield From(coroutine(*a, **kw))
  File "/home/admin/.local/lib/python3.7/site-packages/prompt_toolkit/eventloop/coroutine.py", line 92, in step_next
    new_f = coroutine.throw(exc)
  File "/home/admin/.local/lib/python3.7/site-packages/prompt_toolkit/buffer.py", line 1506, in async_completer
    cancel=lambda: not proceed()))
  File "/home/admin/.local/lib/python3.7/site-packages/prompt_toolkit/eventloop/coroutine.py", line 88, in step_next
    new_f = coroutine.send(None)
  File "/home/admin/.local/lib/python3.7/site-packages/prompt_toolkit/eventloop/async_generator.py", line 117, in consume_async_generator
    item = iterator.send(send)
  File "/home/admin/.local/lib/python3.7/site-packages/prompt_toolkit/completion/base.py", line 176, in get_completions_async
    for item in self.get_completions(document, complete_event):
  File "/home/admin/.local/lib/python3.7/site-packages/athenacli/completer.py", line 210, in get_completions
    suggestions = suggest_type(document.text, document.text_before_cursor)
  File "/home/admin/.local/lib/python3.7/site-packages/athenacli/packages/completion_engine.py", line 121, in suggest_type
    full_text, identifier)
  File "/home/admin/.local/lib/python3.7/site-packages/athenacli/packages/completion_engine.py", line 293, in suggest_based_on_last_token
    tables = tuple(t for t in tables if identifies(parent, t))
  File "/home/admin/.local/lib/python3.7/site-packages/athenacli/packages/completion_engine.py", line 293, in <genexpr>
    tables = tuple(t for t in tables if identifies(parent, t))

Exception identifies() missing 2 required positional arguments: 'table' and 'alias'

I also tried this without the table aliases, e.g.:

SELECT doodads.* FROM doodads JOIN trinkets on doodads.

and got the same "unhandled exception" error when typing the . at the end.

Support role assumption & MFA prompts from profile without AWS_SESSION... env vars

In our environment, we often rely on profiles without the AWS_SESSION... environment variables, which is apparently the "new way" of doing temporary tokens in AWS-land.

So boto3 tries to prompt for my MFA, at which point athenacli gets a bit confused: it appears that athenacli and boto3 are competing for control of the terminal. I get prompted several times, and get some really lovely stack traces:

Enter MFA code for arn:aws:iam::xxxxxxxxxxxxxxx:mfa/rick.cobb: Failed to execute query.
Traceback (most recent call last):
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/athenacli/sqlexecute.py", line 78, in run
    for result in special.execute(cur, sql):
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/athenacli/packages/special/main.py", line 58, in execute
    raise CommandNotFound
athenacli.packages.special.main.CommandNotFound

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pyathena/common.py", line 166, in _execute
    **request)
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pyathena/util.py", line 44, in retry_api_call
    return retry(func, *args, **kwargs)
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/tenacity/__init__.py", line 358, in call
    do = self.iter(retry_state=retry_state)
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/tenacity/__init__.py", line 319, in iter
    return fut.result()
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/tenacity/__init__.py", line 361, in call
    result = fn(*args, **kwargs)
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/client.py", line 320, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/client.py", line 610, in _make_api_call
    operation_model, request_dict)
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/signers.py", line 149, in sign
    auth = self.get_auth_instance(**kwargs)
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/signers.py", line 229, in get_auth_instance
    frozen_credentials = self._credentials.get_frozen_credentials()
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 518, in get_frozen_credentials
    self._refresh()
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 413, in _refresh
    self._protected_refresh(is_mandatory=is_mandatory_refresh)
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 429, in _protected_refresh
    metadata = self._refresh_using()
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 566, in fetch_credentials
    return self._get_cached_credentials()
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 576, in _get_cached_credentials
    response = self._get_credentials()
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 697, in _get_credentials
    client = self._create_client()
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 716, in _create_client
    frozen_credentials = self._source_credentials.get_frozen_credentials()
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 518, in get_frozen_credentials
    self._refresh()
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 413, in _refresh
    self._protected_refresh(is_mandatory=is_mandatory_refresh)
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 429, in _protected_refresh
    metadata = self._refresh_using()
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 177, in __call__
    return self._refresh()
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 566, in fetch_credentials
    return self._get_cached_credentials()
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 576, in _get_cached_credentials
    response = self._get_credentials()
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 696, in _get_credentials
    kwargs = self._assume_role_kwargs()
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 707, in _assume_role_kwargs
    token_code = self._mfa_prompter(prompt)
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/getpass.py", line 77, in unix_getpass
    passwd = _raw_input(prompt, stream, input=input)
  File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/getpass.py", line 148, in _raw_input
    raise EOFError

I'm currently working around this using a new feature of our development tools where we set the AWS_SESSION... env vars, which is fine, but it'd be nice to support the newer profile stuff.

Test athenacli on Windows

Test athenacli on Windows and make it support Windows.

json output format

This output list is exhaustive, and json is not supported today, right?

When I try --table-format "json", I get unrecognized format_name "json". It seems weird to me that html is supported but not just json.

support ctas command

https://aws.amazon.com/about-aws/whats-new/2018/10/athena_ctas_support/

Add option to set the prompt format

All the query input is left aligned to the end of the prompt.

My prompt over 40 characters wide, or about half the width of the terminal.

eu-central-1:athenacurcfn_example_database>

This wastes a lot of space and causes more ugly wrapping for long names.

I don't need to see the region and the database name on every query. A minimal prompt such as > is just fine.

Your cousin pgcli has a prompt option to control the format.

$ pgcli --help

Usage: pgcli [OPTIONS] [DBNAME] [USERNAME]

Options:
...
  --prompt TEXT              Prompt format (Default: "\u@\h:\d> ").

Could you copy this feature from pgcli?

Feature Request: execute query file from inside CLI

I know that there is support for --execute filename from the command line. But there are times it is useful to execute a query file while inside the REPL. Thanks!

Table auto complete fails when a Glue table name has a "." in its name

Hi, I have a glue table named "example.example".
When I'm using the Database and try to select * from , the auto completion fails, and the following error message is displayed:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/opt/conda/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.7/site-packages/athenacli/completion_refresher.py", line 71, in _bg_refresh
    refresher(completer, executor)
  File "/opt/conda/lib/python3.7/site-packages/athenacli/completion_refresher.py", line 113, in refresh_tables
    completer.extend_columns(executor.table_columns(), kind='tables')
  File "/opt/conda/lib/python3.7/site-packages/athenacli/completer.py", line 124, in extend_columns
    metadata[self.dbname][relname].append(column)
KeyError: '"example.example"'

Is this a known issue?

'REPL' commond error!

New release?

Could we make a new release to share the latest commits?

Can't install promt-tooklit >= 3.0 alongside athenacli

I regularly use both litecli and athenacli for local and remote work. Because this tool requires prompt-toolkit < 3.0, anything else that requires prompt toolkit >= 3.0 will cause pip errors (or just won't install at all, if using poetry). litecli is one such tool that requires prompt-toolkit >= 3.0 that I regularly use for quick local work, and I'd prefer to have both that and athenacli in the same virtual environment, if possible.

Any chance the restriction on prompt-toolkit can be opened up to "<4.0"?

Feature request: output cost and size of query

Something like:

    submission_date = stats['QueryExecution']['Status']['SubmissionDateTime']
    completion_date = stats['QueryExecution']['Status']['CompletionDateTime']
    execution_time = stats['QueryExecution']['Statistics']['EngineExecutionTimeInMillis']
    data_scanned = stats['QueryExecution']['Statistics']['DataScannedInBytes']
    query_cost = data_scanned / 1000000000000.0 * 5.0

    print('Time: {}, CPU Time: {}ms total, Data Scanned: {}, Cost: ${:,.2f}\n'.format(
        str(completion_date - submission_date).split('.')[0],
        execution_time,
        human_readable(data_scanned),
        query_cost
    ))

from

https://github.com/guardian/athena-cli/blob/0d5cc98a1cac21094077ed29cf168686ca61f10d/athena_cli.py#L217-L228

(Currently we only print Time.)

Running py.test throws module errors in fresh clone of the project

As per the documentation, after pulling the requirements-dev.txt using pip, running py.test should run the tests.

However, running py.test throws Module errors in the terminal. See the ascii cast below:

Download log.txt

Click to see log here!

============================= test session starts ==============================
platform darwin -- Python 3.6.5, pytest-3.8.2, py-1.6.0, pluggy-0.7.1
rootdir: /Users/jashgala/personal/Github/athenacli, inifile:
collected 478 items / 11 errors

==================================== ERRORS ====================================
___ ERROR collecting lib/python3.6/site-packages/pbr/tests/test_commands.py ____
ImportError while importing test module '/Users/jashgala/personal/Github/athenacli/lib/python3.6/site-packages/pbr/tests/test_commands.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
lib/python3.6/site-packages/pbr/tests/__init__.py:16: in <module>
    import testscenarios
E   ModuleNotFoundError: No module named 'testscenarios'
_____ ERROR collecting lib/python3.6/site-packages/pbr/tests/test_core.py ______
ImportError while importing test module '/Users/jashgala/personal/Github/athenacli/lib/python3.6/site-packages/pbr/tests/test_core.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
lib/python3.6/site-packages/pbr/tests/__init__.py:16: in <module>
    import testscenarios
E   ModuleNotFoundError: No module named 'testscenarios'
_____ ERROR collecting lib/python3.6/site-packages/pbr/tests/test_files.py _____
ImportError while importing test module '/Users/jashgala/personal/Github/athenacli/lib/python3.6/site-packages/pbr/tests/test_files.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
lib/python3.6/site-packages/pbr/tests/__init__.py:16: in <module>
    import testscenarios
E   ModuleNotFoundError: No module named 'testscenarios'
_____ ERROR collecting lib/python3.6/site-packages/pbr/tests/test_hooks.py _____
ImportError while importing test module '/Users/jashgala/personal/Github/athenacli/lib/python3.6/site-packages/pbr/tests/test_hooks.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
lib/python3.6/site-packages/pbr/tests/__init__.py:16: in <module>
    import testscenarios
E   ModuleNotFoundError: No module named 'testscenarios'
__ ERROR collecting lib/python3.6/site-packages/pbr/tests/test_integration.py __
ImportError while importing test module '/Users/jashgala/personal/Github/athenacli/lib/python3.6/site-packages/pbr/tests/test_integration.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
lib/python3.6/site-packages/pbr/tests/__init__.py:16: in <module>
    import testscenarios
E   ModuleNotFoundError: No module named 'testscenarios'
___ ERROR collecting lib/python3.6/site-packages/pbr/tests/test_packaging.py ___
ImportError while importing test module '/Users/jashgala/personal/Github/athenacli/lib/python3.6/site-packages/pbr/tests/test_packaging.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
lib/python3.6/site-packages/pbr/tests/__init__.py:16: in <module>
    import testscenarios
E   ModuleNotFoundError: No module named 'testscenarios'
___ ERROR collecting lib/python3.6/site-packages/pbr/tests/test_pbr_json.py ____
ImportError while importing test module '/Users/jashgala/personal/Github/athenacli/lib/python3.6/site-packages/pbr/tests/test_pbr_json.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
lib/python3.6/site-packages/pbr/tests/__init__.py:16: in <module>
    import testscenarios
E   ModuleNotFoundError: No module named 'testscenarios'
_____ ERROR collecting lib/python3.6/site-packages/pbr/tests/test_setup.py _____
ImportError while importing test module '/Users/jashgala/personal/Github/athenacli/lib/python3.6/site-packages/pbr/tests/test_setup.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
lib/python3.6/site-packages/pbr/tests/__init__.py:16: in <module>
    import testscenarios
E   ModuleNotFoundError: No module named 'testscenarios'
_____ ERROR collecting lib/python3.6/site-packages/pbr/tests/test_util.py ______
ImportError while importing test module '/Users/jashgala/personal/Github/athenacli/lib/python3.6/site-packages/pbr/tests/test_util.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
lib/python3.6/site-packages/pbr/tests/__init__.py:16: in <module>
    import testscenarios
E   ModuleNotFoundError: No module named 'testscenarios'
____ ERROR collecting lib/python3.6/site-packages/pbr/tests/test_version.py ____
ImportError while importing test module '/Users/jashgala/personal/Github/athenacli/lib/python3.6/site-packages/pbr/tests/test_version.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
lib/python3.6/site-packages/pbr/tests/__init__.py:16: in <module>
    import testscenarios
E   ModuleNotFoundError: No module named 'testscenarios'
_____ ERROR collecting lib/python3.6/site-packages/pbr/tests/test_wsgi.py ______
ImportError while importing test module '/Users/jashgala/personal/Github/athenacli/lib/python3.6/site-packages/pbr/tests/test_wsgi.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
lib/python3.6/site-packages/pbr/tests/__init__.py:16: in <module>
    import testscenarios
E   ModuleNotFoundError: No module named 'testscenarios'
!!!!!!!!!!!!!!!!!!! Interrupted: 11 errors during collection !!!!!!!!!!!!!!!!!!!
=========================== 11 error in 8.32 seconds ===========================

Workaround: Running pytest test seems to be working for the time being

Required argument `s3_staging_dir` or `work_group` not found.

Required argument s3_staging_dir or work_group not found.

Even though these have been defined in the ~/.athenacli/athenaclirc. Screenshot for proof:

I'm working around this by sending in the arguments but is sub-optimal. Could this be because I am not defining aws access key and secret in the config? (I get those from the defined profile I use.)

Support for default credentials.

It is common for the official awscli tools and boto to use credentials in ~/.aws/credentials, or in environment variables. We use this heavily to support logging in using short lived STS tokens, and avoid creating long lived secret keys, which are a security no-no.

It would be great if this tool supported using those default credentials. (And STS tokens)

Feature request: Support converting zero bytes and high-bit-set bytes to octal sequences for VARBINARY

PostgreSQL has a function encode(data, 'escape') where data is a bytea.
https://www.postgresql.org/docs/9.4/functions-binarystring.html

Currently when using athencli w/ the Athena JDBC the default output for VARBINARY is to convert to UTF8 and if unable to convert it just outputs the hex representation of the VARBINARY. It would be extremely useful to have the ability to support this output mode directly in athenacli for any VARBINARY column when set.

Here is an example select on a VARBINARY column from Athena when using the Python pyathenajdbc library which is a wrapper for the Athena JAR. Notice how the ASCII converts but other other bytes are left as-is:

b'\x03\x00\x00+&\xe0\x00\x00\x00\x00\x00Cookie: mstshash=hello\r\n\x01\x00\x08\x00\x03\x00\x00\x00'

Invite users

Hey @bryanck, would you like to try this CLI tool?

Feature request: get query from stdin

Most/some unix programms interpret a file named '-' as 'read from stdin'. Would you take a PR which adds this functionality to https://github.com/dbcli/athenacli/blob/master/athenacli/main.py#L651?

# copied from another project which has this functionality:
    if execute == '-':
        import select
        if select.select([sys.stdin, ], [], [], 0.0)[0]:
            query = sys.stdin.read()
        else:
            raise RuntimeError("No query to execute on stdin")
    else:
       # current code which tries to read it in from file and else takes it as is

Latest release (1.4.0) not working if athenaclirc file not in path

athenacli works with version 1.3.3, but I get this message when trying to run athenacli 1.4.0:

Required argument `s3_staging_dir` or `work_group` not found.

There was an error while connecting to AWS Athena. It could be caused due to
missing/incomplete configuration. Please verify the configuration in ~/.athenacli/athenaclirc
and run athenacli again.

Feature request: better auto-completion

Problems

For very large databases, we are experiencing a ~20 sec delay when switching using use
Autocompletion only works for 1 database at a time (e.g. doesn't work for cross-database queries)

Possible solutions

Fetch auto completions on SELECT database_name.<tab>
Get all the autocompletions up-front?
Cache when switching databases?
Make it configurable?

Feature request: Get signed S3 link

Currently I can output results to a local file using tee, but often I want a signed S3 link to the output.

Proposal

A new command download would output a signed S3 link to the previous query's results.

SELECT DISTINCT

and then hit space I get

SELECT DISTINCT Exception in thread Thread-13:
Traceback (most recent call last):
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/athenacli/0.1.3/libexec/lib/python3.7/site-packages/prompt_toolkit/interface.py", line 865, in run
    completions = list(buffer.completer.get_completions(document, complete_event))
  File "/usr/local/Cellar/athenacli/0.1.3/libexec/lib/python3.7/site-packages/athenacli/completer.py", line 217, in get_completions
    matcher = self.suggestion_matchers[suggestion_type]
KeyError: <class 'list'>

If instead I first open a ( I am able to continue typing the query.

Thanks for this great product. I use it every day :)

Throwing KeyError: 'work_group'

Athenacli recently started throwing KeyError: 'work_group' upon usage. I installed using pip using python 3.7.7 installed with pyenv. Here are some details

> pip show athenacli
Name: athenacli
Version: 1.6.1
Summary: CLI for Athena Database. With auto-completion and syntax highlighting.
Home-page: https://github.com/dbcli/athenacli
Author: athenacli Core Team
Author-email: [email protected]
License: UNKNOWN
Location: /Users/raynehernandez/.pyenv/versions/3.7.7/lib/python3.7/site-packages
Requires: prompt-toolkit, configobj, PyAthena, cli-helpers, botocore, sqlparse, boto3, Pygments, click
Required-by:

> athenacli --profile <profile>
Traceback (most recent call last):
  File "/Users/xxxx/.pyenv/versions/3.7.7/bin/athenacli", line 8, in <module>
    sys.exit(cli())
  File "/Users/xxxx/.pyenv/versions/3.7.7/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/xxxx/.pyenv/versions/3.7.7/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/xxxx/.pyenv/versions/3.7.7/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/xxxx/.pyenv/versions/3.7.7/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/xxxx/.pyenv/versions/3.7.7/lib/python3.7/site-packages/athenacli/main.py", line 653, in cli
    database=database
  File "/Users/xxxx/.pyenv/versions/3.7.7/lib/python3.7/site-packages/athenacli/main.py", line 74, in __init__
    aws_access_key_id, aws_secret_access_key, region, s3_staging_dir, work_group, profile, _cfg
  File "/Users/xxxx/.pyenv/versions/3.7.7/lib/python3.7/site-packages/athenacli/config.py", line 36, in __init__
    self.work_group = self.get_val(work_group, _cfg['work_group'])
  File "/Users/xxxx/.pyenv/versions/3.7.7/lib/python3.7/site-packages/configobj.py", line 554, in __getitem__
    val = dict.__getitem__(self, key)
KeyError: 'work_group'

dbcli / athenacli Goto Github PK

athenacli's People

Contributors

Stargazers

Watchers

Forkers

athenacli's Issues

JOIN doesn't appear

Unhandled Exception

Problems

Possible solutions

Proposal

Recommend Projects

Recommend Topics

Recommend Org

`JOIN` doesn't appear