Code Monkey home page Code Monkey logo

red-panda's Introduction

Red Panda

image

docs license

Easily interact with cloud (AWS) in your Data Science workflow.

Features

  • DataFrame/files to and from S3 and Redshift.
  • Run queries on Redshift in Python.
  • Use built-in Redshift admin queries, such as checking running queries and errors.
  • Use Redshift utility functions to easily accomplish common tasks such as creating a table.
  • Manage files on S3.
  • Query data on S3 directly with Athena.
  • Pandas DataFrame utility functions.

Installation

pip install red-panda

Using red-panda

Import red-panda and create an instance of RedPanda. If you create the instance with dryrun=True (i.e. rp = RedPanda(redshift_conf, s3_conf, dryrun=True)), red-panda will print the planned queries instead of executing them.

from red_panda import RedPanda

redshift_conf = {
    "user": "awesome-developer",
    "password": "strong-password",
    "host": "awesome-domain.us-east-1.redshift.amazonaws.com",
    "port": 5432,
    "dbname": "awesome-db",
}

aws_conf = {
    "aws_access_key_id": "your-aws-access-key-id",
    "aws_secret_access_key": "your-aws-secret-access-key",
    # "aws_session_token": "temporary-token-if-you-have-one",
}

rp = RedPanda(redshift_conf, aws_conf)

Load your Pandas DataFrame into Redshift as a new table.

import pandas as pd

df = pd.DataFrame(data={"col1": [1, 2], "col2": [3, 4]})

s3_bucket = "s3-bucket-name"
s3_path = "parent-folder/child-folder" # optional, if you don't have any sub folders
s3_file_name = "test.csv" # optional, randomly generated if not provided
rp.df_to_redshift(df, "test_table", bucket=s3_bucket, path=s3_path, append=False)

It is also possible to:

  • Upload a DataFrame or flat file to S3.
  • Delete files from S3.
  • Load S3 data into Redshift.
  • Unload a Redshift query result to S3.
  • Obtain a Redshift query result as a DataFrame.
  • Run any query on Redshift.
  • Download S3 file to local.
  • Read S3 file in memory as DataFrame.
  • Run built-in Redshift admin queries, such as getting running query information.
  • Use utility functions such as create_table to quickly create tables in Redshift.
  • Run queries against S3 data directly with Athena using AthenaUtils.
  • Use features separately with RedshiftUtils, S3Utils, AthenaUtils.
s3_key = s3_path + "/" + s3_file_name

# DataFrame uploaded to S3
rp.df_to_s3(df, s3_bucket, s3_key)

# Delete a file on S3
rp.delete_from_s3(s3_bucket, s3_key)

# Upload a local file to S3
pd.to_csv(df, "test_data.csv", index=False)
rp.file_to_s3("test_data.csv", s3_bucket, s3_key)

# Populate a Redshift table from S3 files
# Use a dictionary for column definition, here we minimally define only data_type
redshift_column_definition = {
    "col1": {data_type: "int"},
    "col2": {data_type: "int"},
}
rp.s3_to_redshift(
    s3_bucket, s3_key, "test_table", column_definition=redshift_column_definition
)

# Unload Redshift query result to S3
sql = "select * from test_table"
rp.redshift_to_s3(sql, s3_bucket, s3_path+"/unload", prefix="unloadtest_")

# Obtain Redshift query result as a DataFrame
df = rp.redshift_to_df("select * from test_table")

# Run queries on Redshift
rp.run_query("create table test_table_copy as select * from test_table")

# Download S3 file to local
rp.s3_to_file(s3_bucket, s3_key, "local_file_name.csv")

# Read S3 file in memory as DataFrame
df = rp.s3_to_df(s3_bucket, s3_key, delimiter=",") # csv file in this example

# Since we are only going to use Redshift functionalities, we can just use RedshiftUtils
from red_panda.red_panda import RedshiftUtils
ru = RedshiftUtils(redshift_conf)

# Run built-in Redshift admin queries, such as getting running query information
load_errors = ru.get_load_error(as_df=True)

# Use utility functions such as create_table to quickly create tables in Redshift
ru.create_table("test_table", redshift_column_definition, sortkey=["col2"], drop_first=True)

For full API documentation, visit https://red-panda.readthedocs.io/en/latest/.

TODO

In no particular order:

  • Support more data formats for copy. Currently only support delimited files.
  • Support more data formats for s3 to df. Currently only support delimited files.
  • Improve tests and docs.
  • Better ways of inferring data types from dataframe to Redshift.
  • Explore using S3 Transfer Manager's upload_fileobj for df_to_s3 to take advantage of automatic multipart upload.
  • Add COPY from S3 manifest file, in addition to COPY from S3 source path.
  • Support multi-cloud.
  • Take advantage of Redshift slices for parallel processing. Split files for COPY.

red-panda's People

Contributors

dependabot[bot] avatar yaojiach avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar

Forkers

danielf01 mkosater cascade

red-panda's Issues

os.path.join shouldn't be used for concatenating bucket paths

I've be using the redshift_to_s3 method line 707 and it was failing due to the use of os.path.join. os.path.join will drop earlier arguments if a later argument appears as a absolute path i.e as a S3_PATH would.

image

I will work through a make a pull request.

Specify stricter dependencies in Pipfile

Is your feature request related to a problem? Please describe.
This isn't necessary a bug (hence why I chose feature request) but also not much of a feature, just a suggestion. My team currently uses Poetry to manage our library builds, and we've noticed adding red-panda increases the tool's dependency's resolution time exponentially (literally hours, actually have not been able to make it work so far). Looking a bit deeper, I see that it's probably due to the fact your Pipfile dependencies are all wildcards.

Describe the solution you'd like
If possible, and not too big of a lift, could you provide some more meaningful restrictions on dependency versions? Anything but * would likely be helpful.

More generally, even though adding restrictions might seem an "extra", in my experience it's helpful down the line when you might encounter annoying conflicts (you know what you have, and what you can and/or should change).

Describe alternatives you've considered

  • Not including this in our build: Works but it's obviously not what we'd want!
  • Not using Poetry: Have only tried running with classic setuptools but we lose a good deal of sane safeguards without a tool like this. Have not tried Pipenv yet (may consider it and working that possibility separately), but my above point still holds.

Thanks for your time, and for the work put into this, quite helpful!

Installation failed on Macbook M1

Describe the bug
Hi all,

I just got a new macbook with the M1 chip. When i now want to install red-panda I get an conflicting-dependencies issue. See below.

Can you help me out here?

Thanks in advance!

To Reproduce
Steps to reproduce the behavior:

  1. Open new terminal
  2. Optionally activate the venv where you want it installed
  3. Run pip3 install red-panda
  4. See error:
pip3 install red-panda
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Collecting red-panda
  Using cached red_panda-1.0.2-py3-none-any.whl (23 kB)
Collecting pandas>=1.1.0
  Using cached pandas-1.3.4-cp39-cp39-macosx_11_0_arm64.whl
Collecting boto3>=1.14.38
  Downloading boto3-1.20.16-py3-none-any.whl (131 kB)
     |████████████████████████████████| 131 kB 3.8 MB/s            
Collecting psycopg2-binary>=2.8.5
  Using cached psycopg2-binary-2.9.2.tar.gz (380 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/homebrew/opt/[email protected]/bin/python3.9 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_ed9422e609204c2cba4d022ab1a8aaaa/setup.py'"'"'; __file__='"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_ed9422e609204c2cba4d022ab1a8aaaa/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-cmtd18c7
       cwd: /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_ed9422e609204c2cba4d022ab1a8aaaa/
  Complete output (23 lines):
  running egg_info
  creating /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-cmtd18c7/psycopg2_binary.egg-info
  writing /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-cmtd18c7/psycopg2_binary.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-cmtd18c7/psycopg2_binary.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-cmtd18c7/psycopg2_binary.egg-info/top_level.txt
  writing manifest file '/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-cmtd18c7/psycopg2_binary.egg-info/SOURCES.txt'
  
  Error: pg_config executable not found.
  
  pg_config is required to build psycopg2 from source.  Please add the directory
  containing pg_config to the $PATH or specify the full executable path with the
  option:
  
      python setup.py build_ext --pg-config /path/to/pg_config build ...
  
  or with the pg_config option in 'setup.cfg'.
  
  If you prefer to avoid building psycopg2 from source, please install the PyPI
  'psycopg2-binary' package instead.
  
  For further information please check the 'doc/src/install.rst' file (also at
  <https://www.psycopg.org/docs/install.html>).
  
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/d1/6d/bdb09cc1ccc6aa5e611c4d74daf751f8881ae231933f4ef1d4f6d699d021/psycopg2-binary-2.9.2.tar.gz#sha256=234b1f48488b2f86aac04fb00cb04e5e9bcb960f34fa8a8e41b73149d581a93b (from https://pypi.org/simple/psycopg2-binary/) (requires-python:>=3.6). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached psycopg2-binary-2.9.1.tar.gz (380 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/homebrew/opt/[email protected]/bin/python3.9 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_62a628afc9e64ecea36fdea44ba9f7e9/setup.py'"'"'; __file__='"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_62a628afc9e64ecea36fdea44ba9f7e9/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-p9_lw5b5
       cwd: /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_62a628afc9e64ecea36fdea44ba9f7e9/
  Complete output (23 lines):
  running egg_info
  creating /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-p9_lw5b5/psycopg2_binary.egg-info
  writing /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-p9_lw5b5/psycopg2_binary.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-p9_lw5b5/psycopg2_binary.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-p9_lw5b5/psycopg2_binary.egg-info/top_level.txt
  writing manifest file '/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-p9_lw5b5/psycopg2_binary.egg-info/SOURCES.txt'
  
  Error: pg_config executable not found.
  
  pg_config is required to build psycopg2 from source.  Please add the directory
  containing pg_config to the $PATH or specify the full executable path with the
  option:
  
      python setup.py build_ext --pg-config /path/to/pg_config build ...
  
  or with the pg_config option in 'setup.cfg'.
  
  If you prefer to avoid building psycopg2 from source, please install the PyPI
  'psycopg2-binary' package instead.
  
  For further information please check the 'doc/src/install.rst' file (also at
  <https://www.psycopg.org/docs/install.html>).
  
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/14/65/223a5b4146b1d5d5ab66f16ef194916a1ed9720da1f118d7bfb60b8f2bea/psycopg2-binary-2.9.1.tar.gz#sha256=b0221ca5a9837e040ebf61f48899926b5783668b7807419e4adae8175a31f773 (from https://pypi.org/simple/psycopg2-binary/) (requires-python:>=3.6). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached psycopg2-binary-2.9.tar.gz (379 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/homebrew/opt/[email protected]/bin/python3.9 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_b1799c48aadc400386be76ced2e82dfc/setup.py'"'"'; __file__='"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_b1799c48aadc400386be76ced2e82dfc/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-bq_uv5y2
       cwd: /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_b1799c48aadc400386be76ced2e82dfc/
  Complete output (23 lines):
  running egg_info
  creating /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-bq_uv5y2/psycopg2_binary.egg-info
  writing /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-bq_uv5y2/psycopg2_binary.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-bq_uv5y2/psycopg2_binary.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-bq_uv5y2/psycopg2_binary.egg-info/top_level.txt
  writing manifest file '/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-bq_uv5y2/psycopg2_binary.egg-info/SOURCES.txt'
  
  Error: pg_config executable not found.
  
  pg_config is required to build psycopg2 from source.  Please add the directory
  containing pg_config to the $PATH or specify the full executable path with the
  option:
  
      python setup.py build_ext --pg-config /path/to/pg_config build ...
  
  or with the pg_config option in 'setup.cfg'.
  
  If you prefer to avoid building psycopg2 from source, please install the PyPI
  'psycopg2-binary' package instead.
  
  For further information please check the 'doc/src/install.rst' file (also at
  <https://www.psycopg.org/docs/install.html>).
  
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/39/55/54778407e306de1acd3c616dc6a04e3821effa52c21d6c688d4947878658/psycopg2-binary-2.9.tar.gz#sha256=5515c626df6ce6e9058883ace33fece4e32d3409eb70367cc62952f6d6a50dee (from https://pypi.org/simple/psycopg2-binary/) (requires-python:>=3.6). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached psycopg2-binary-2.8.6.tar.gz (384 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/homebrew/opt/[email protected]/bin/python3.9 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_e1c980a64b8e45f8b37c1c130950f321/setup.py'"'"'; __file__='"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_e1c980a64b8e45f8b37c1c130950f321/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-ddvah6dq
       cwd: /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_e1c980a64b8e45f8b37c1c130950f321/
  Complete output (23 lines):
  running egg_info
  creating /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-ddvah6dq/psycopg2_binary.egg-info
  writing /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-ddvah6dq/psycopg2_binary.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-ddvah6dq/psycopg2_binary.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-ddvah6dq/psycopg2_binary.egg-info/top_level.txt
  writing manifest file '/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-ddvah6dq/psycopg2_binary.egg-info/SOURCES.txt'
  
  Error: pg_config executable not found.
  
  pg_config is required to build psycopg2 from source.  Please add the directory
  containing pg_config to the $PATH or specify the full executable path with the
  option:
  
      python setup.py build_ext --pg-config /path/to/pg_config build ...
  
  or with the pg_config option in 'setup.cfg'.
  
  If you prefer to avoid building psycopg2 from source, please install the PyPI
  'psycopg2-binary' package instead.
  
  For further information please check the 'doc/src/install.rst' file (also at
  <https://www.psycopg.org/docs/install.html>).
  
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/fc/51/0f2c6aec5c59e5640f507b59567f63b9d73a9317898810b4db311da32dfc/psycopg2-binary-2.8.6.tar.gz#sha256=11b9c0ebce097180129e422379b824ae21c8f2a6596b159c7659e2e5a00e1aa0 (from https://pypi.org/simple/psycopg2-binary/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached psycopg2-binary-2.8.5.tar.gz (381 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/homebrew/opt/[email protected]/bin/python3.9 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_4408ab3bc32444438e35a101032b7234/setup.py'"'"'; __file__='"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_4408ab3bc32444438e35a101032b7234/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-0lv3adlz
       cwd: /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_4408ab3bc32444438e35a101032b7234/
  Complete output (23 lines):
  running egg_info
  creating /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-0lv3adlz/psycopg2_binary.egg-info
  writing /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-0lv3adlz/psycopg2_binary.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-0lv3adlz/psycopg2_binary.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-0lv3adlz/psycopg2_binary.egg-info/top_level.txt
  writing manifest file '/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-0lv3adlz/psycopg2_binary.egg-info/SOURCES.txt'
  
  Error: pg_config executable not found.
  
  pg_config is required to build psycopg2 from source.  Please add the directory
  containing pg_config to the $PATH or specify the full executable path with the
  option:
  
      python setup.py build_ext --pg-config /path/to/pg_config build ...
  
  or with the pg_config option in 'setup.cfg'.
  
  If you prefer to avoid building psycopg2 from source, please install the PyPI
  'psycopg2-binary' package instead.
  
  For further information please check the 'doc/src/install.rst' file (also at
  <https://www.psycopg.org/docs/install.html>).
  
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/97/00/ed4c82364741031d745867f83820d4f373aa891098a5785841850491c9ba/psycopg2-binary-2.8.5.tar.gz#sha256=ccdc6a87f32b491129ada4b87a43b1895cf2c20fdb7f98ad979647506ffc41b6 (from https://pypi.org/simple/psycopg2-binary/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Collecting red-panda
  Using cached red_panda-1.0.1-py2.py3-none-any.whl (32 kB)
  Using cached red_panda-1.0.0-py2.py3-none-any.whl (32 kB)
  Using cached red_panda-0.1.11-py2.py3-none-any.whl (29 kB)
Collecting fsspec
  Using cached fsspec-2021.11.1-py3-none-any.whl (132 kB)
Collecting PyAthena
  Using cached PyAthena-2.3.1-py3-none-any.whl (37 kB)
Collecting click
  Downloading click-8.0.3-py3-none-any.whl (97 kB)
     |████████████████████████████████| 97 kB 11.8 MB/s            
Collecting dask
  Using cached dask-2021.11.2-py3-none-any.whl (1.0 MB)
Collecting oss2
  Downloading oss2-2.15.0.tar.gz (226 kB)
     |████████████████████████████████| 226 kB 13.2 MB/s            
  Preparing metadata (setup.py) ... done
Collecting psycopg2-binary
  Using cached psycopg2-binary-2.8.4.tar.gz (378 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/homebrew/opt/[email protected]/bin/python3.9 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_7a92d40e7542405b8d528606b6f99757/setup.py'"'"'; __file__='"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_7a92d40e7542405b8d528606b6f99757/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-y6jie6ed
       cwd: /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_7a92d40e7542405b8d528606b6f99757/
  Complete output (23 lines):
  running egg_info
  creating /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-y6jie6ed/psycopg2_binary.egg-info
  writing /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-y6jie6ed/psycopg2_binary.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-y6jie6ed/psycopg2_binary.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-y6jie6ed/psycopg2_binary.egg-info/top_level.txt
  writing manifest file '/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-y6jie6ed/psycopg2_binary.egg-info/SOURCES.txt'
  
  Error: pg_config executable not found.
  
  pg_config is required to build psycopg2 from source.  Please add the directory
  containing pg_config to the $PATH or specify the full executable path with the
  option:
  
      python setup.py build_ext --pg-config /path/to/pg_config build ...
  
  or with the pg_config option in 'setup.cfg'.
  
  If you prefer to avoid building psycopg2 from source, please install the PyPI
  'psycopg2-binary' package instead.
  
  For further information please check the 'doc/src/install.rst' file (also at
  <http://initd.org/psycopg/docs/install.html>).
  
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/00/7b/a623f49b3248957e7eaaac52eba1117209775d54e7a8501c460473a7ba4f/psycopg2-binary-2.8.4.tar.gz#sha256=3a2522b1d9178575acee4adf8fd9f979f9c0449b00b4164bb63c3475ea6528ed (from https://pypi.org/simple/psycopg2-binary/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached psycopg2-binary-2.8.3.tar.gz (378 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/homebrew/opt/[email protected]/bin/python3.9 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_00e77221f5534f53b9b36aa85dec2bba/setup.py'"'"'; __file__='"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_00e77221f5534f53b9b36aa85dec2bba/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-_l1f1x38
       cwd: /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_00e77221f5534f53b9b36aa85dec2bba/
  Complete output (23 lines):
  running egg_info
  creating /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-_l1f1x38/psycopg2_binary.egg-info
  writing /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-_l1f1x38/psycopg2_binary.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-_l1f1x38/psycopg2_binary.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-_l1f1x38/psycopg2_binary.egg-info/top_level.txt
  writing manifest file '/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-_l1f1x38/psycopg2_binary.egg-info/SOURCES.txt'
  
  Error: pg_config executable not found.
  
  pg_config is required to build psycopg2 from source.  Please add the directory
  containing pg_config to the $PATH or specify the full executable path with the
  option:
  
      python setup.py build_ext --pg-config /path/to/pg_config build ...
  
  or with the pg_config option in 'setup.cfg'.
  
  If you prefer to avoid building psycopg2 from source, please install the PyPI
  'psycopg2-binary' package instead.
  
  For further information please check the 'doc/src/install.rst' file (also at
  <http://initd.org/psycopg/docs/install.html>).
  
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/80/91/91911be01869fa877135946f928ed0004e62044bdd876c1e0f12e1b5fb90/psycopg2-binary-2.8.3.tar.gz#sha256=cd37cc170678a4609becb26b53a2bc1edea65177be70c48dd7b39a1149cabd6e (from https://pypi.org/simple/psycopg2-binary/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached psycopg2-binary-2.8.2.tar.gz (369 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/homebrew/opt/[email protected]/bin/python3.9 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_43b075bb431244bc979dd2ea10264edf/setup.py'"'"'; __file__='"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_43b075bb431244bc979dd2ea10264edf/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-dp4o1gg6
       cwd: /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_43b075bb431244bc979dd2ea10264edf/
  Complete output (23 lines):
  running egg_info
  creating /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-dp4o1gg6/psycopg2_binary.egg-info
  writing /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-dp4o1gg6/psycopg2_binary.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-dp4o1gg6/psycopg2_binary.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-dp4o1gg6/psycopg2_binary.egg-info/top_level.txt
  writing manifest file '/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-dp4o1gg6/psycopg2_binary.egg-info/SOURCES.txt'
  
  Error: pg_config executable not found.
  
  pg_config is required to build psycopg2 from source.  Please add the directory
  containing pg_config to the $PATH or specify the full executable path with the
  option:
  
      python setup.py build_ext --pg-config /path/to/pg_config build ...
  
  or with the pg_config option in 'setup.cfg'.
  
  If you prefer to avoid building psycopg2 from source, please install the PyPI
  'psycopg2-binary' package instead.
  
  For further information please check the 'doc/src/install.rst' file (also at
  <http://initd.org/psycopg/docs/install.html>).
  
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/dc/93/bb5655730913b88f9068c6b596177d1df83be0d476671199e17b06ea8436/psycopg2-binary-2.8.2.tar.gz#sha256=4c2d9369ed40b4a44a8ccd6bc3a7db6272b8314812d2d1091f95c4c836d92e06 (from https://pypi.org/simple/psycopg2-binary/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached psycopg2-binary-2.8.1.tar.gz (368 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/homebrew/opt/[email protected]/bin/python3.9 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_6cb67e7e1f6e461b967e5190ab1cba64/setup.py'"'"'; __file__='"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_6cb67e7e1f6e461b967e5190ab1cba64/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-hp49o63h
       cwd: /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_6cb67e7e1f6e461b967e5190ab1cba64/
  Complete output (23 lines):
  running egg_info
  creating /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-hp49o63h/psycopg2_binary.egg-info
  writing /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-hp49o63h/psycopg2_binary.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-hp49o63h/psycopg2_binary.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-hp49o63h/psycopg2_binary.egg-info/top_level.txt
  writing manifest file '/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-hp49o63h/psycopg2_binary.egg-info/SOURCES.txt'
  
  Error: pg_config executable not found.
  
  pg_config is required to build psycopg2 from source.  Please add the directory
  containing pg_config to the $PATH or specify the full executable path with the
  option:
  
      python setup.py build_ext --pg-config /path/to/pg_config build ...
  
  or with the pg_config option in 'setup.cfg'.
  
  If you prefer to avoid building psycopg2 from source, please install the PyPI
  'psycopg2-binary' package instead.
  
  For further information please check the 'doc/src/install.rst' file (also at
  <http://initd.org/psycopg/docs/install.html>).
  
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/a6/fb/158c6f1dfcc0f0ceb4c3687378b965269079e55de5630c173dd163c6ab21/psycopg2-binary-2.8.1.tar.gz#sha256=a20dfdf73f56da674926a3811929cff9fd23b9af90be9a6c36ac246a3486eef3 (from https://pypi.org/simple/psycopg2-binary/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached psycopg2-binary-2.8.tar.gz (368 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/homebrew/opt/[email protected]/bin/python3.9 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_0081a1fdc5f441398590b04797424c79/setup.py'"'"'; __file__='"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_0081a1fdc5f441398590b04797424c79/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-sf1b0l5h
       cwd: /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_0081a1fdc5f441398590b04797424c79/
  Complete output (23 lines):
  running egg_info
  creating /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-sf1b0l5h/psycopg2_binary.egg-info
  writing /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-sf1b0l5h/psycopg2_binary.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-sf1b0l5h/psycopg2_binary.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-sf1b0l5h/psycopg2_binary.egg-info/top_level.txt
  writing manifest file '/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-sf1b0l5h/psycopg2_binary.egg-info/SOURCES.txt'
  
  Error: pg_config executable not found.
  
  pg_config is required to build psycopg2 from source.  Please add the directory
  containing pg_config to the $PATH or specify the full executable path with the
  option:
  
      python setup.py build_ext --pg-config /path/to/pg_config build ...
  
  or with the pg_config option in 'setup.cfg'.
  
  If you prefer to avoid building psycopg2 from source, please install the PyPI
  'psycopg2-binary' package instead.
  
  For further information please check the 'doc/src/install.rst' file (also at
  <http://initd.org/psycopg/docs/install.html>).
  
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/45/81/ce42d2f87b81616c33284ea52e0b914e45dcae6093655d53602d515aaf57/psycopg2-binary-2.8.tar.gz#sha256=50647aa5f7171153a5f7fa667f99f55468b9b663b997927e4d2e83955b21aa9f (from https://pypi.org/simple/psycopg2-binary/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached psycopg2-binary-2.7.7.tar.gz (428 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/homebrew/opt/[email protected]/bin/python3.9 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_c8bd22eeffa746a6adbe8b8f52fa374a/setup.py'"'"'; __file__='"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_c8bd22eeffa746a6adbe8b8f52fa374a/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-xqdb05gx
       cwd: /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_c8bd22eeffa746a6adbe8b8f52fa374a/
  Complete output (26 lines):
  running egg_info
  creating /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-xqdb05gx/psycopg2_binary.egg-info
  writing /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-xqdb05gx/psycopg2_binary.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-xqdb05gx/psycopg2_binary.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-xqdb05gx/psycopg2_binary.egg-info/top_level.txt
  writing manifest file '/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-xqdb05gx/psycopg2_binary.egg-info/SOURCES.txt'
  /opt/homebrew/lib/python3.9/site-packages/setuptools/command/egg_info.py:624: SetuptoolsDeprecationWarning: Custom 'build_py' does not implement 'get_data_files_without_manifest'.
  Please extend command classes from setuptools instead of distutils.
    warnings.warn(
  
  Error: pg_config executable not found.
  
  pg_config is required to build psycopg2 from source.  Please add the directory
  containing pg_config to the $PATH or specify the full executable path with the
  option:
  
      python setup.py build_ext --pg-config /path/to/pg_config build ...
  
  or with the pg_config option in 'setup.cfg'.
  
  If you prefer to avoid building psycopg2 from source, please install the PyPI
  'psycopg2-binary' package instead.
  
  For further information please check the 'doc/src/install.rst' file (also at
  <http://initd.org/psycopg/docs/install.html>).
  
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/dd/56/c22da10f5a725d61c58a185ec0f803aa2d384646ee8eb83d8ce88ed5edb1/psycopg2-binary-2.7.7.tar.gz#sha256=b19e9f1b85c5d6136f5a0549abdc55dcbd63aba18b4f10d0d063eb65ef2c68b4 (from https://pypi.org/simple/psycopg2-binary/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached psycopg2-binary-2.7.6.1.tar.gz (428 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/homebrew/opt/[email protected]/bin/python3.9 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_afc8b91e7c514764a16618d95d02f051/setup.py'"'"'; __file__='"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_afc8b91e7c514764a16618d95d02f051/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-oe5mm8y0
       cwd: /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_afc8b91e7c514764a16618d95d02f051/
  Complete output (26 lines):
  running egg_info
  creating /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-oe5mm8y0/psycopg2_binary.egg-info
  writing /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-oe5mm8y0/psycopg2_binary.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-oe5mm8y0/psycopg2_binary.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-oe5mm8y0/psycopg2_binary.egg-info/top_level.txt
  writing manifest file '/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-oe5mm8y0/psycopg2_binary.egg-info/SOURCES.txt'
  /opt/homebrew/lib/python3.9/site-packages/setuptools/command/egg_info.py:624: SetuptoolsDeprecationWarning: Custom 'build_py' does not implement 'get_data_files_without_manifest'.
  Please extend command classes from setuptools instead of distutils.
    warnings.warn(
  
  Error: pg_config executable not found.
  
  pg_config is required to build psycopg2 from source.  Please add the directory
  containing pg_config to the $PATH or specify the full executable path with the
  option:
  
      python setup.py build_ext --pg-config /path/to/pg_config build ...
  
  or with the pg_config option in 'setup.cfg'.
  
  If you prefer to avoid building psycopg2 from source, please install the PyPI
  'psycopg2-binary' package instead.
  
  For further information please check the 'doc/src/install.rst' file (also at
  <http://initd.org/psycopg/docs/install.html>).
  
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/59/64/7b0fea58c9b7c609717cc951607fe5edb73e86d46a47b0c7d58586335e53/psycopg2-binary-2.7.6.1.tar.gz#sha256=8d517e8fda2efebca27c2018e14c90ed7dc3f04d7098b3da2912e62a1a5585fe (from https://pypi.org/simple/psycopg2-binary/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached psycopg2-binary-2.7.6.tar.gz (428 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/homebrew/opt/[email protected]/bin/python3.9 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_6c40d527701949fcac224b16b0bbda7d/setup.py'"'"'; __file__='"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_6c40d527701949fcac224b16b0bbda7d/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-s2riwb7l
       cwd: /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_6c40d527701949fcac224b16b0bbda7d/
  Complete output (26 lines):
  running egg_info
  creating /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-s2riwb7l/psycopg2_binary.egg-info
  writing /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-s2riwb7l/psycopg2_binary.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-s2riwb7l/psycopg2_binary.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-s2riwb7l/psycopg2_binary.egg-info/top_level.txt
  writing manifest file '/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-s2riwb7l/psycopg2_binary.egg-info/SOURCES.txt'
  /opt/homebrew/lib/python3.9/site-packages/setuptools/command/egg_info.py:624: SetuptoolsDeprecationWarning: Custom 'build_py' does not implement 'get_data_files_without_manifest'.
  Please extend command classes from setuptools instead of distutils.
    warnings.warn(
  
  Error: pg_config executable not found.
  
  pg_config is required to build psycopg2 from source.  Please add the directory
  containing pg_config to the $PATH or specify the full executable path with the
  option:
  
      python setup.py build_ext --pg-config /path/to/pg_config build ...
  
  or with the pg_config option in 'setup.cfg'.
  
  If you prefer to avoid building psycopg2 from source, please install the PyPI
  'psycopg2-binary' package instead.
  
  For further information please check the 'doc/src/install.rst' file (also at
  <http://initd.org/psycopg/docs/install.html>).
  
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/c0/40/4a48479ac9fdc3e2ffbaf0cebba5c9414a9591588fe550679762b21751dd/psycopg2-binary-2.7.6.tar.gz#sha256=80bf63fa9493c82d72788376c7342a27b8c4f07971be405cbb886c6b025bebc8 (from https://pypi.org/simple/psycopg2-binary/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached psycopg2-binary-2.7.5.tar.gz (427 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/homebrew/opt/[email protected]/bin/python3.9 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_24d51c225a7b46e9b1337e1523316933/setup.py'"'"'; __file__='"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_24d51c225a7b46e9b1337e1523316933/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-f3fkutb6
       cwd: /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_24d51c225a7b46e9b1337e1523316933/
  Complete output (26 lines):
  running egg_info
  creating /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-f3fkutb6/psycopg2_binary.egg-info
  writing /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-f3fkutb6/psycopg2_binary.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-f3fkutb6/psycopg2_binary.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-f3fkutb6/psycopg2_binary.egg-info/top_level.txt
  writing manifest file '/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-f3fkutb6/psycopg2_binary.egg-info/SOURCES.txt'
  /opt/homebrew/lib/python3.9/site-packages/setuptools/command/egg_info.py:624: SetuptoolsDeprecationWarning: Custom 'build_py' does not implement 'get_data_files_without_manifest'.
  Please extend command classes from setuptools instead of distutils.
    warnings.warn(
  
  Error: pg_config executable not found.
  
  pg_config is required to build psycopg2 from source.  Please add the directory
  containing pg_config to the $PATH or specify the full executable path with the
  option:
  
      python setup.py build_ext --pg-config /path/to/pg_config build ...
  
  or with the pg_config option in 'setup.cfg'.
  
  If you prefer to avoid building psycopg2 from source, please install the PyPI
  'psycopg2-binary' package instead.
  
  For further information please check the 'doc/src/install.rst' file (also at
  <http://initd.org/psycopg/docs/install.html>).
  
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/17/dc/3f6bfae668594013cb9ef0cba4f4863451220944650868ad67332ca9069c/psycopg2-binary-2.7.5.tar.gz#sha256=c2ac7aa1a144d4e0e613ac7286dae85671e99fe7a1353954d4905629c36b811c (from https://pypi.org/simple/psycopg2-binary/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached psycopg2-binary-2.7.4.tar.gz (426 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/homebrew/opt/[email protected]/bin/python3.9 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_44a6477f0afe4f69918c226b4168a5e2/setup.py'"'"'; __file__='"'"'/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_44a6477f0afe4f69918c226b4168a5e2/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-v7tkz5na
       cwd: /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-install-lefmapij/psycopg2-binary_44a6477f0afe4f69918c226b4168a5e2/
  Complete output (26 lines):
  running egg_info
  creating /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-v7tkz5na/psycopg2_binary.egg-info
  writing /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-v7tkz5na/psycopg2_binary.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-v7tkz5na/psycopg2_binary.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-v7tkz5na/psycopg2_binary.egg-info/top_level.txt
  writing manifest file '/private/var/folders/0c/6qktn2t907lc0bf85y8h775h0000gn/T/pip-pip-egg-info-v7tkz5na/psycopg2_binary.egg-info/SOURCES.txt'
  /opt/homebrew/lib/python3.9/site-packages/setuptools/command/egg_info.py:624: SetuptoolsDeprecationWarning: Custom 'build_py' does not implement 'get_data_files_without_manifest'.
  Please extend command classes from setuptools instead of distutils.
    warnings.warn(
  
  Error: pg_config executable not found.
  
  pg_config is required to build psycopg2 from source.  Please add the directory
  containing pg_config to the $PATH or specify the full executable path with the
  option:
  
      python setup.py build_ext --pg-config /path/to/pg_config build ...
  
  or with the pg_config option in 'setup.cfg'.
  
  If you prefer to avoid building psycopg2 from source, please install the PyPI
  'psycopg2-binary' package instead.
  
  For further information please check the 'doc/src/install.rst' file (also at
  <http://initd.org/psycopg/docs/install.html>).
  
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/77/09/4991fcd9a8f4bea1ee3948e1729fa17c184d25bd10809bacc143626361b9/psycopg2-binary-2.7.4.tar.gz#sha256=de4f88f823037a71ea5ef3c1041d96b8a68d73343133edda684fd42f575bd9d7 (from https://pypi.org/simple/psycopg2-binary/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Collecting red-panda
  Using cached red_panda-0.1.10-py2.py3-none-any.whl (28 kB)
  Using cached red_panda-0.1.9-py2.py3-none-any.whl (24 kB)
  Using cached red_panda-0.1.8-py2.py3-none-any.whl (24 kB)
  Using cached red_panda-0.1.7-py2.py3-none-any.whl (14 kB)
  Using cached red_panda-0.1.6-py2.py3-none-any.whl (9.5 kB)
  Using cached red_panda-0.1.4-py2.py3-none-any.whl (7.5 kB)
  Using cached red_panda-0.1.3-py2.py3-none-any.whl (7.5 kB)
  Using cached red_panda-0.1.2-py2.py3-none-any.whl (7.5 kB)
  Using cached red_panda-0.1.1-py2.py3-none-any.whl (6.9 kB)
  Using cached red_panda-0.1.0-py2.py3-none-any.whl (6.3 kB)
ERROR: Cannot install red-panda==0.1.0, red-panda==0.1.1, red-panda==0.1.10, red-panda==0.1.11, red-panda==0.1.2, red-panda==0.1.3, red-panda==0.1.4, red-panda==0.1.6, red-panda==0.1.7, red-panda==0.1.8, red-panda==0.1.9, red-panda==1.0.0, red-panda==1.0.1 and red-panda==1.0.2 because these package versions have conflicting dependencies.

The conflict is caused by:
    red-panda 1.0.2 depends on psycopg2-binary>=2.8.5
    red-panda 1.0.1 depends on psycopg2-binary>=2.8.5
    red-panda 1.0.0 depends on psycopg2-binary>=2.8.5
    red-panda 0.1.11 depends on psycopg2-binary
    red-panda 0.1.10 depends on psycopg2-binary
    red-panda 0.1.9 depends on psycopg2-binary
    red-panda 0.1.8 depends on psycopg2-binary
    red-panda 0.1.7 depends on psycopg2-binary
    red-panda 0.1.6 depends on psycopg2-binary
    red-panda 0.1.4 depends on psycopg2-binary
    red-panda 0.1.3 depends on psycopg2-binary
    red-panda 0.1.2 depends on psycopg2-binary
    red-panda 0.1.1 depends on psycopg2-binary
    red-panda 0.1.0 depends on psycopg2-binary

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

Expected behavior
Installation of package without errors :)

Desktop (please complete the following information):

  • OS: version: 12.0.1.

Additional context

  • python version: Python 3.9.9 (but also tried in new venv with python 3.8)

S3 kwargs are ignored when using df_to_redshift

Using red-panda release 1.0.2:

To Reproduce

Steps to reproduce the behavior:

  1. Call df_to_redshift, passing in an S3-specific kwarg like ServerSideEncryption="AES256", and also cleanup=False so you can check the file in S3
  2. Run the above, verifying that it succeeds
  3. Check the file that was written to S3; it will not have encryption enabled

Expected behavior

Because the underlying df_to_s3 function is creating s3_put_kwargs based on the kwargs passed to it, I think the expectation is that S3-specific args will be honored when calling df_to_redshift. That doesn't happen because the only kwargs passed to df_to_s3 are ones listed in PANDAS_TOCSV_KWARGS . I think df_to_s3 needs to receive both "to csv" kwargs and "to s3" kwargs.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.