Code Monkey home page Code Monkey logo

s3transfer's Introduction

Deprecation notice

This package is no longer maintained and has been replaced by Boto3. Issues and pull requests are not reviewed. If you are having an issue with the Boto3 package or the AWS CLI, please open an issue on their respective repositories.

boto

boto 2.49.0

Released: 11-July-2018

Introduction

Boto is a Python package that provides interfaces to Amazon Web Services. Currently, all features work with Python 2.6 and 2.7. Work is under way to support Python 3.3+ in the same codebase. Modules are being ported one at a time with the help of the open source community, so please check below for compatibility with Python 3.3+.

To port a module to Python 3.3+, please view our Contributing Guidelines and the Porting Guide. If you would like, you can open an issue to let others know about your work in progress. Tests must pass on Python 2.6, 2.7, 3.3, and 3.4 for pull requests to be accepted.

Services

At the moment, boto supports:

  • Compute
    • Amazon Elastic Compute Cloud (EC2) (Python 3)
    • Amazon Elastic Map Reduce (EMR) (Python 3)
    • AutoScaling (Python 3)
    • Amazon Kinesis (Python 3)
    • AWS Lambda (Python 3)
    • Amazon EC2 Container Service (Python 3)
  • Content Delivery
    • Amazon CloudFront (Python 3)
  • Database
    • Amazon Relational Data Service (RDS)
    • Amazon DynamoDB (Python 3)
    • Amazon SimpleDB (Python 3)
    • Amazon ElastiCache (Python 3)
    • Amazon Redshift (Python 3)
  • Deployment and Management
    • AWS Elastic Beanstalk (Python 3)
    • AWS CloudFormation (Python 3)
    • AWS Data Pipeline (Python 3)
    • AWS Opsworks (Python 3)
    • AWS CloudTrail (Python 3)
    • AWS CodeDeploy (Python 3)
  • Administration & Security
    • AWS Identity and Access Management (IAM) (Python 3)
    • AWS Key Management Service (KMS) (Python 3)
    • AWS Config (Python 3)
    • AWS CloudHSM (Python 3)
  • Application Services
    • Amazon CloudSearch (Python 3)
    • Amazon CloudSearch Domain (Python 3)
    • Amazon Elastic Transcoder (Python 3)
    • Amazon Simple Workflow Service (SWF) (Python 3)
    • Amazon Simple Queue Service (SQS) (Python 3)
    • Amazon Simple Notification Server (SNS) (Python 3)
    • Amazon Simple Email Service (SES) (Python 3)
    • Amazon Cognito Identity (Python 3)
    • Amazon Cognito Sync (Python 3)
    • Amazon Machine Learning (Python 3)
  • Monitoring
    • Amazon CloudWatch (EC2 Only) (Python 3)
    • Amazon CloudWatch Logs (Python 3)
  • Networking
    • Amazon Route53 (Python 3)
    • Amazon Route 53 Domains (Python 3)
    • Amazon Virtual Private Cloud (VPC) (Python 3)
    • Elastic Load Balancing (ELB) (Python 3)
    • AWS Direct Connect (Python 3)
  • Payments and Billing
    • Amazon Flexible Payment Service (FPS)
  • Storage
    • Amazon Simple Storage Service (S3) (Python 3)
    • Amazon Glacier (Python 3)
    • Amazon Elastic Block Store (EBS)
    • Google Cloud Storage
  • Workforce
    • Amazon Mechanical Turk
  • Other
    • Marketplace Web Services (Python 3)
    • AWS Support (Python 3)

The goal of boto is to support the full breadth and depth of Amazon Web Services. In addition, boto provides support for other public services such as Google Storage in addition to private cloud systems like Eucalyptus, OpenStack and Open Nebula.

Boto is developed mainly using Python 2.6.6 and Python 2.7.3 on Mac OSX and Ubuntu Maverick. It is known to work on other Linux distributions and on Windows. Most of Boto requires no additional libraries or packages other than those that are distributed with Python. Efforts are made to keep boto compatible with Python 2.5.x but no guarantees are made.

Installation

Install via pip:

$ pip install boto

Install from source:

$ git clone git://github.com/boto/boto.git
$ cd boto
$ python setup.py install

ChangeLogs

To see what has changed over time in boto, you can check out the release notes at http://docs.pythonboto.org/en/latest/#release-notes

Finding Out More About Boto

The main source code repository for boto can be found on github.com. The boto project uses the gitflow model for branching.

Online documentation is also available. The online documentation includes full API documentation as well as Getting Started Guides for many of the boto modules.

Boto releases can be found on the Python Cheese Shop.

Join our IRC channel #boto on FreeNode. Webchat IRC channel: http://webchat.freenode.net/?channels=boto

Join the boto-users Google Group.

Getting Started with Boto

Your credentials can be passed into the methods that create connections. Alternatively, boto will check for the existence of the following environment variables to ascertain your credentials:

AWS_ACCESS_KEY_ID - Your AWS Access Key ID

AWS_SECRET_ACCESS_KEY - Your AWS Secret Access Key

Credentials and other boto-related settings can also be stored in a boto config file. See this for details.

s3transfer's People

Contributors

adamwirth avatar annguojia avatar aws-sdk-python-automation avatar awstools avatar cleberzavadniak avatar dbnicholson avatar dependabot[bot] avatar dlm6693 avatar dstufft avatar graebm avatar homebysix avatar hugovk avatar jamesls avatar jdufresne avatar jogusd avatar jonathan343 avatar jordonphillips avatar juice500ml avatar kamalmostafa avatar kdaily avatar kianmeng avatar kyleknap avatar nateprewitt avatar pgrimaud avatar potapy4 avatar rvrangel avatar stealthycoin avatar stobrien89 avatar timgates42 avatar tingdaok avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

kyleknap defconcepts mayuqianjg jamesls jordonphillips pisymbol stealthycoin antmd jagill svleeuwen meiqia kamalmostafa dstufft harrisonfeng amit2014 m-vdb dennistt aweidner mahesh223 jogusd dbnicholson sciclon danielkza lundalogik juice500ml pokainc mtbirdsall edhodapp hugovk thilo-maurer nathan-muir hhamalai quiltdata zehauser dalavancloud homebysix grdguez talmachani ktp-forked-repos bkyryliuk ericzinnikas rvrangel ashokdhoni jasondearte godekgansss adamwirth komin92 cbaines cclauss nateprewitt simonpjakobsen devotionzhu ut-382v-ashkan-and-david aachenmax erupare tingdaok annguojia vz10 kdaily lissom tschoonj liquidpele muralich2002 harsh200 assimilationstheorie kadantte saiprasad16 id-jordan-stefanelli sumank792 jpkrohling justindho vikasgoel ayoub-g kislyuk kianmeng stobrien89 ekmixon mathieuseguin mark25913 yangkghjh balaprvi frankfanslc pierre-muller alvistack rives-de-paris maatticlabs cloud-devops-factory dlm6693 rapkyt iamsaker adamsc64 neithan hssyoo chhafmeh isabella232 hoshimura elmkarami jesuis33 cfxegbert ramil4icr

s3transfer's Issues

test_compat.py fails on MacOS and python 3.8+

This is something that we started see in Nix when we enabled tests as part of the build: NixOS/nixpkgs#154280

____ TestBaseManager.test_can_provide_signal_handler_initializers_to_start _____

self = <tests.unit.test_compat.TestBaseManager testMethod=test_can_provide_signal_handler_initializers_to_start>

    @skip_if_windows('os.kill() with SIGINT not supported on Windows')
    def test_can_provide_signal_handler_initializers_to_start(self):
        manager = self.create_pid_manager()
>       manager.start(signal.signal, (signal.SIGINT, signal.SIG_IGN))

tests/unit/test_compat.py:100:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/managers.py:554: in start
    self._process.start()
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/context.py:284: in _Popen
    return Popen(process_obj)
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <SpawnProcess name='PIDManager-43' parent=65299 initial>
file = <_io.BytesIO object at 0x1131d6630>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'TestBaseManager.create_pid_manager.<locals>.PIDManager'

/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/reduction.py:60: AttributeError

We think it might be due to spawn/fork changes in python 3.8, since it does succeed in python 3.7. NixOS/nixpkgs#154280 (comment) (ht to @thefloweringash for finding that)

Shutdown exception

I've been trying to determine a way to cancel any boto3 upload/download calls. When I was checking the underlying "manager" object in the S3Transfer, i found the "shutdown" method.

But calling this method results in an exception being thrown.

client = boto3.client( 's3' )
transfer = S3Transfer( client )
transfer.download_file( bucket, key, to_file )

# Call just after the download starts
transfer._manager.shutdown( True )

The thrown exception is

python-BaseException
Traceback (most recent call last):
  File "/opt/.../lib/python3.8/site-packages/s3transfer/futures.py", line 278, in cancel
    self._exception = exc_type(msg)
TypeError: 'str' object is not callable

From what I found, the shutdown command is calling a protected member. But the called method signature is different.

def shutdown(self, cancel=False, cancel_msg=''):
    self._shutdown(cancel, cancel, cancel_msg)

def _shutdown(self, cancel, cancel_msg, exc_type=CancelledError):

The location the shutdown call is here:

s3transfer/s3transfer/manager.py

Line 588 in 960fc16

self._shutdown(cancel, cancel, cancel_msg)

readable() is inconsistent across python versions

I was looking to copy what readable() does in order to add a similar writable() function, but it has inconsistent behavior across python versions. readable() will tell you if the file was opened readable (or rather if read() will actually succeed), whereas the code path in python2 will simply tell you if the fileobj has a read method, without any comment on whether or not read will fail. Consider:

Python 2.7.11 (default, Jan 22 2016, 08:29:18)
>>> from s3transfer.compat import readable
>>> readable(open('/tmp/asdfafgafdsfd', 'w'))
True

Python 3.5.1 (default, Jan 22 2016, 08:54:32)
>>> from s3transfer.compat import readable
>>> readable(open('/tmp/adfasfasdfd', 'w'))
False

I would expect us to try, within reason, to have consistent behavior across python versions.

0.4.2: pytest is failing in many tests/integration units 

+ /usr/bin/python3 -Bm pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.9, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2
plugins: forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, cov-2.11.1, httpbin-1.0.0, xdist-2.2.1, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, pyfakefs-4.4.0, freezegun-0.4.2, cases-3.4.6, case-1.5.3, isort-1.3.0, aspectlib-1.5.2, asyncio-0.15.1, toolbox-0.5, xprocess-0.17.1, flaky-3.7.0, requests-mock-1.9.2, aiohttp-0.3.0, checkdocs-2.7.0, mock-3.6.1, hypothesis-6.13.5
collected 677 items

. .                                                                                                                                                                  [  0%]
tests/functional/test_copy.py .................................                                                                                                      [  5%]
tests/functional/test_crt.py sssssss                                                                                                                                 [  6%]
tests/functional/test_delete.py .......                                                                                                                              [  7%]
tests/functional/test_download.py .........................................                                                                                          [ 13%]
tests/functional/test_manager.py ........                                                                                                                            [ 14%]
tests/functional/test_processpool.py ............                                                                                                                    [ 16%]
tests/functional/test_upload.py ............................                                                                                                         [ 20%]
tests/functional/test_utils.py ...                                                                                                                                   [ 20%]
tests/integration/test_copy.py EEE                                                                                                                                   [ 21%]
tests/integration/test_crt.py sssssssssssss                                                                                                                          [ 23%]
tests/integration/test_delete.py E                                                                                                                                   [ 23%]
tests/integration/test_download.py EEEEEEEEEE                                                                                                                        [ 24%]
tests/integration/test_processpool.py EEEE                                                                                                                           [ 25%]
tests/integration/test_s3transfer.py EEEEEEEEEEEE                                                                                                                    [ 27%]
tests/integration/test_upload.py EEEEEEEEEEEEEEE                                                                                                                     [ 29%]
tests/unit/test_bandwidth.py ....................................                                                                                                    [ 34%]
tests/unit/test_compat.py ........                                                                                                                                   [ 35%]
tests/unit/test_copies.py ......                                                                                                                                     [ 36%]
tests/unit/test_crt.py ssssssss                                                                                                                                      [ 37%]
tests/unit/test_delete.py ..                                                                                                                                         [ 38%]
tests/unit/test_download.py ..................................................................                                                                       [ 47%]
tests/unit/test_futures.py .........................................................                                                                                 [ 56%]
tests/unit/test_manager.py .........                                                                                                                                 [ 57%]
tests/unit/test_processpool.py ...................................................                                                                                   [ 65%]
tests/unit/test_s3transfer.py .........................................................                                                                              [ 73%]
tests/unit/test_subscribers.py ........                                                                                                                              [ 74%]
tests/unit/test_tasks.py ...............................                                                                                                             [ 79%]
tests/unit/test_upload.py ...............................................                                                                                            [ 86%]
tests/unit/test_utils.py ............................................................................................                                                [100%]

================================================================================== ERRORS ==================================================================================
___________________________________________________________ ERROR at setup of TestCopy.test_copy_above_threshold ___________________________________________________________

cls = <class 'tests.integration.test_copy.TestCopy'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2e77d700>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2e77d250>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
___________________________________________________________ ERROR at setup of TestCopy.test_copy_below_threshold ___________________________________________________________

cls = <class 'tests.integration.test_copy.TestCopy'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2e77d700>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2e77d250>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_______________________________________________________ ERROR at setup of TestCopy.test_progress_subscribers_on_copy _______________________________________________________

cls = <class 'tests.integration.test_copy.TestCopy'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2e77d700>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2e77d250>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
________________________________________________________ ERROR at setup of TestDeleteObject.test_can_delete_object _________________________________________________________

cls = <class 'tests.integration.test_delete.TestDeleteObject'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2df0e6a0>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2df0ee20>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
___________________________________________________________ ERROR at setup of TestDownload.test_above_threshold ____________________________________________________________

cls = <class 'tests.integration.test_download.TestDownload'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d101310>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d1012e0>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_____________________________________________________ ERROR at setup of TestDownload.test_above_threshold_for_fileobj ______________________________________________________

cls = <class 'tests.integration.test_download.TestDownload'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d101310>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d1012e0>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_______________________________________________ ERROR at setup of TestDownload.test_above_threshold_for_nonseekable_fileobj ________________________________________________

cls = <class 'tests.integration.test_download.TestDownload'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d101310>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d1012e0>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
___________________________________________________________ ERROR at setup of TestDownload.test_below_threshold ____________________________________________________________

cls = <class 'tests.integration.test_download.TestDownload'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d101310>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d1012e0>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_____________________________________________________ ERROR at setup of TestDownload.test_below_threshold_for_fileobj ______________________________________________________

cls = <class 'tests.integration.test_download.TestDownload'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d101310>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d1012e0>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_______________________________________________ ERROR at setup of TestDownload.test_below_threshold_for_nonseekable_fileobj ________________________________________________

cls = <class 'tests.integration.test_download.TestDownload'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d101310>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d1012e0>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_______________________________________________________ ERROR at setup of TestDownload.test_download_to_special_file _______________________________________________________

cls = <class 'tests.integration.test_download.TestDownload'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d101310>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d1012e0>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_______________________________________________ ERROR at setup of TestDownload.test_large_download_exits_quicky_on_exception _______________________________________________

cls = <class 'tests.integration.test_download.TestDownload'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d101310>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d1012e0>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_________________________________________________ ERROR at setup of TestDownload.test_many_files_exits_quicky_on_exception _________________________________________________

cls = <class 'tests.integration.test_download.TestDownload'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d101310>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d1012e0>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
___________________________________________________ ERROR at setup of TestDownload.test_progress_subscribers_on_download ___________________________________________________

cls = <class 'tests.integration.test_download.TestDownload'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d101310>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d1012e0>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_____________________________________________________ ERROR at setup of TestProcessPoolDownloader.test_above_threshold _____________________________________________________

cls = <class 'tests.integration.test_processpool.TestProcessPoolDownloader'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2dff1d60>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2dff17f0>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_____________________________________________________ ERROR at setup of TestProcessPoolDownloader.test_below_threshold _____________________________________________________

cls = <class 'tests.integration.test_processpool.TestProcessPoolDownloader'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2dff1d60>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2dff17f0>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
________________________________________ ERROR at setup of TestProcessPoolDownloader.test_large_download_exits_quickly_on_exception ________________________________________

cls = <class 'tests.integration.test_processpool.TestProcessPoolDownloader'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2dff1d60>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2dff17f0>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
__________________________________________ ERROR at setup of TestProcessPoolDownloader.test_many_files_exits_quickly_on_exception __________________________________________

cls = <class 'tests.integration.test_processpool.TestProcessPoolDownloader'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2dff1d60>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2dff17f0>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
__________________________________________________ ERROR at setup of TestS3Transfers.test_callback_called_once_with_sigv4 __________________________________________________

cls = <class 'tests.integration.test_s3transfer.TestS3Transfers'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d8b5be0>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d8b5790>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
______________________________________________________ ERROR at setup of TestS3Transfers.test_can_configure_threshold ______________________________________________________

cls = <class 'tests.integration.test_s3transfer.TestS3Transfers'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d8b5be0>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d8b5790>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_________________________________________________ ERROR at setup of TestS3Transfers.test_can_send_extra_params_on_download _________________________________________________

cls = <class 'tests.integration.test_s3transfer.TestS3Transfers'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d8b5be0>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d8b5790>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
__________________________________________________ ERROR at setup of TestS3Transfers.test_can_send_extra_params_on_upload __________________________________________________

cls = <class 'tests.integration.test_s3transfer.TestS3Transfers'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d8b5be0>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d8b5790>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_____________________________________________________ ERROR at setup of TestS3Transfers.test_download_above_threshold ______________________________________________________

cls = <class 'tests.integration.test_s3transfer.TestS3Transfers'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d8b5be0>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d8b5790>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_____________________________________________________ ERROR at setup of TestS3Transfers.test_download_below_threshold ______________________________________________________

cls = <class 'tests.integration.test_s3transfer.TestS3Transfers'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d8b5be0>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d8b5790>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
___________________________________________________ ERROR at setup of TestS3Transfers.test_progress_callback_on_download ___________________________________________________

cls = <class 'tests.integration.test_s3transfer.TestS3Transfers'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d8b5be0>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d8b5790>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
____________________________________________________ ERROR at setup of TestS3Transfers.test_progress_callback_on_upload ____________________________________________________

cls = <class 'tests.integration.test_s3transfer.TestS3Transfers'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d8b5be0>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d8b5790>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
______________________________________________________ ERROR at setup of TestS3Transfers.test_upload_above_threshold _______________________________________________________

cls = <class 'tests.integration.test_s3transfer.TestS3Transfers'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d8b5be0>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d8b5790>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
______________________________________________________ ERROR at setup of TestS3Transfers.test_upload_below_threshold _______________________________________________________

cls = <class 'tests.integration.test_s3transfer.TestS3Transfers'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d8b5be0>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d8b5790>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_______________________________________________ ERROR at setup of TestS3Transfers.test_upload_file_above_threshold_with_acl ________________________________________________

cls = <class 'tests.integration.test_s3transfer.TestS3Transfers'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d8b5be0>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d8b5790>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_______________________________________________ ERROR at setup of TestS3Transfers.test_upload_file_above_threshold_with_ssec _______________________________________________

cls = <class 'tests.integration.test_s3transfer.TestS3Transfers'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2d8b5be0>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2d8b5790>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_________________________________________________ ERROR at setup of TestUpload.test_large_upload_exits_quicky_on_exception _________________________________________________

cls = <class 'tests.integration.test_upload.TestUpload'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e0c681e50>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e0c681a90>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
__________________________________________________ ERROR at setup of TestUpload.test_many_files_exits_quicky_on_exception __________________________________________________

cls = <class 'tests.integration.test_upload.TestUpload'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e0c681e50>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e0c681a90>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_____________________________________________________ ERROR at setup of TestUpload.test_progress_subscribers_on_upload _____________________________________________________

cls = <class 'tests.integration.test_upload.TestUpload'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e0c681e50>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e0c681a90>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_________________________________________________________ ERROR at setup of TestUpload.test_upload_above_threshold _________________________________________________________

cls = <class 'tests.integration.test_upload.TestUpload'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e0c681e50>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e0c681a90>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_________________________________________________________ ERROR at setup of TestUpload.test_upload_below_threshold _________________________________________________________

cls = <class 'tests.integration.test_upload.TestUpload'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e0c681e50>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e0c681a90>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
__________________________________________ ERROR at setup of TestUploadSeekableStream.test_large_upload_exits_quicky_on_exception __________________________________________

cls = <class 'tests.integration.test_upload.TestUploadSeekableStream'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2df089a0>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2df08f70>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
___________________________________________ ERROR at setup of TestUploadSeekableStream.test_many_files_exits_quicky_on_exception ___________________________________________

cls = <class 'tests.integration.test_upload.TestUploadSeekableStream'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2df089a0>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2df08f70>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
______________________________________________ ERROR at setup of TestUploadSeekableStream.test_progress_subscribers_on_upload ______________________________________________

cls = <class 'tests.integration.test_upload.TestUploadSeekableStream'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2df089a0>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2df08f70>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
__________________________________________________ ERROR at setup of TestUploadSeekableStream.test_upload_above_threshold __________________________________________________

cls = <class 'tests.integration.test_upload.TestUploadSeekableStream'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2df089a0>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2df08f70>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
__________________________________________________ ERROR at setup of TestUploadSeekableStream.test_upload_below_threshold __________________________________________________

cls = <class 'tests.integration.test_upload.TestUploadSeekableStream'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2df089a0>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2df08f70>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
________________________________________ ERROR at setup of TestUploadNonSeekableStream.test_large_upload_exits_quicky_on_exception _________________________________________

cls = <class 'tests.integration.test_upload.TestUploadNonSeekableStream'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2e12ea90>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2e12e5b0>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
_________________________________________ ERROR at setup of TestUploadNonSeekableStream.test_many_files_exits_quicky_on_exception __________________________________________

cls = <class 'tests.integration.test_upload.TestUploadNonSeekableStream'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2e12ea90>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2e12e5b0>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
____________________________________________ ERROR at setup of TestUploadNonSeekableStream.test_progress_subscribers_on_upload _____________________________________________

cls = <class 'tests.integration.test_upload.TestUploadNonSeekableStream'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2e12ea90>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2e12e5b0>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
________________________________________________ ERROR at setup of TestUploadNonSeekableStream.test_upload_above_threshold _________________________________________________

cls = <class 'tests.integration.test_upload.TestUploadNonSeekableStream'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2e12ea90>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2e12e5b0>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
________________________________________________ ERROR at setup of TestUploadNonSeekableStream.test_upload_below_threshold _________________________________________________

cls = <class 'tests.integration.test_upload.TestUploadNonSeekableStream'>

    @classmethod
    def setUpClass(cls):
        cls.region = 'us-west-2'
        cls.session = botocore.session.get_session()
        cls.client = cls.session.create_client('s3', cls.region)
        cls.bucket_name = random_bucket_name()
>       cls.client.create_bucket(
            Bucket=cls.bucket_name,
            CreateBucketConfiguration={'LocationConstraint': cls.region})

tests/integration/__init__.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/botocore/client.py:386: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    http, parsed_response = self._make_request(
/usr/lib/python3.8/site-packages/botocore/client.py:711: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:132: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/lib/python3.8/site-packages/botocore/endpoint.py:115: in create_request
    self._event_emitter.emit(event_name, request=request,
/usr/lib/python3.8/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
/usr/lib/python3.8/site-packages/botocore/signers.py:90: in handler
    return self.sign(operation_name, request)
/usr/lib/python3.8/site-packages/botocore/signers.py:162: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.auth.S3SigV4Auth object at 0x7f9e2e12ea90>, request = <botocore.awsrequest.AWSRequest object at 0x7f9e2e12e5b0>

    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

/usr/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError
============================================================================= warnings summary =============================================================================
tests/unit/test_utils.py:303
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2/tests/unit/test_utils.py:303: DeprecationWarning: invalid escape sequence \.
    '%s\.[0-9A-Fa-f]{8}$' % filename,

tests/functional/test_copy.py::TestNonMultipartCopy::test_invalid_extra_args
tests/functional/test_copy.py::TestMultipartCopy::test_invalid_extra_args
tests/functional/test_delete.py::TestDeleteObject::test_invalid_extra_args
tests/functional/test_download.py::TestNonRangedDownload::test_invalid_extra_args
tests/functional/test_download.py::TestRangedDownload::test_invalid_extra_args
tests/functional/test_upload.py::TestNonMultipartUpload::test_invalid_extra_args
tests/functional/test_upload.py::TestMultipartUpload::test_invalid_extra_args
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2/tests/__init__.py:454: DeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(ValueError, 'Invalid extra_args'):

tests/functional/test_copy.py::TestNonMultipartCopy::test_raise_exception_on_s3_object_lambda_resource
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2/tests/functional/test_copy.py:284: DeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(ValueError, 'methods do not support'):

tests/functional/test_copy.py::TestNonMultipartCopy::test_raise_exception_on_s3_object_lambda_resource_as_source
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2/tests/functional/test_copy.py:290: DeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(ValueError, 'methods do not support'):

tests/functional/test_copy.py::TestMultipartCopy::test_abort_on_failure
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2/tests/functional/test_copy.py:543: DeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(ClientError, 'ArbitraryFailure'):

tests/functional/test_delete.py::TestDeleteObject::test_raise_exception_on_s3_object_lambda_resource
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2/tests/functional/test_delete.py:72: DeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(ValueError, 'methods do not support'):

tests/functional/test_download.py::TestNonRangedDownload::test_raise_exception_on_s3_object_lambda_resource
tests/functional/test_download.py::TestRangedDownload::test_raise_exception_on_s3_object_lambda_resource
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2/tests/functional/test_download.py:346: DeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(ValueError, 'methods do not support'):

tests/functional/test_manager.py::TestTransferManager::test_cntrl_c_in_context_manager_cancels_incomplete_transfers
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2/tests/functional/test_manager.py:125: DeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(

tests/functional/test_manager.py::TestTransferManager::test_error_in_context_manager_cancels_incomplete_transfers
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2/tests/functional/test_manager.py:88: DeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(FatalError, ref_exception_msg):

tests/functional/test_upload.py::TestNonMultipartUpload::test_raise_exception_on_s3_object_lambda_resource
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2/tests/functional/test_upload.py:274: DeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(ValueError, 'methods do not support'):

tests/unit/test_futures.py::TestTransferCoordinator::test_cancel_with_message
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2/tests/unit/test_futures.py:263: DeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(CancelledError, message):

tests/unit/test_futures.py::TestTransferCoordinator::test_cancel_with_provided_exception
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2/tests/unit/test_futures.py:270: DeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(FatalError, message):

tests/unit/test_futures.py::TestNonThreadedExecutorFuture::test_exception_result
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2/tests/unit/test_futures.py:636: DeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(ValueError, 'message'):

tests/unit/test_manager.py::TestTransferCoordinatorController::test_cancel_with_message
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2/tests/unit/test_manager.py:87: DeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(CancelledError, message):

tests/unit/test_manager.py::TestTransferCoordinatorController::test_cancel_with_provided_exception
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2/tests/unit/test_manager.py:97: DeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(FatalError, message):

tests/unit/test_s3transfer.py::TestMultipartDownloader::test_download_futures_fail_triggers_shutdown
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2/tests/unit/test_s3transfer.py:483: DeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(Exception, "fake download parts error"):

tests/unit/test_s3transfer.py::TestMultipartDownloader::test_io_thread_failure_triggers_shutdown
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2/tests/unit/test_s3transfer.py:458: DeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(Exception, "fake IO error"):

tests/unit/test_subscribers.py::TestSubscribers::test_no_kwargs_in_subclass_subscriber_method
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2/tests/unit/test_subscribers.py:86: DeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(

tests/unit/test_subscribers.py::TestSubscribers::test_not_callable_in_subclass_subscriber_method
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.4.2/tests/unit/test_subscribers.py:81: DeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================= short test summary info ==========================================================================
SKIPPED [1] tests/functional/test_crt.py:170: Test requires awscrt to be installed.
SKIPPED [1] tests/functional/test_crt.py:200: Test requires awscrt to be installed.
SKIPPED [1] tests/functional/test_crt.py:235: Test requires awscrt to be installed.
SKIPPED [1] tests/functional/test_crt.py:152: Test requires awscrt to be installed.
SKIPPED [1] tests/functional/test_crt.py:128: Test requires awscrt to be installed.
SKIPPED [1] tests/functional/test_crt.py:216: Test requires awscrt to be installed.
SKIPPED [1] tests/functional/test_crt.py:110: Test requires awscrt to be installed.
SKIPPED [13] ../../../../../usr/lib/python3.8/site-packages/_pytest/unittest.py:153: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:81: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:66: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:51: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:115: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:121: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:145: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:155: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:150: Test requires awscrt to be installed.
ERROR tests/integration/test_copy.py::TestCopy::test_copy_above_threshold - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_copy.py::TestCopy::test_copy_below_threshold - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_copy.py::TestCopy::test_progress_subscribers_on_copy - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_delete.py::TestDeleteObject::test_can_delete_object - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_download.py::TestDownload::test_above_threshold - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_download.py::TestDownload::test_above_threshold_for_fileobj - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_download.py::TestDownload::test_above_threshold_for_nonseekable_fileobj - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_download.py::TestDownload::test_below_threshold - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_download.py::TestDownload::test_below_threshold_for_fileobj - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_download.py::TestDownload::test_below_threshold_for_nonseekable_fileobj - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_download.py::TestDownload::test_download_to_special_file - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_download.py::TestDownload::test_large_download_exits_quicky_on_exception - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_download.py::TestDownload::test_many_files_exits_quicky_on_exception - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_download.py::TestDownload::test_progress_subscribers_on_download - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_processpool.py::TestProcessPoolDownloader::test_above_threshold - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_processpool.py::TestProcessPoolDownloader::test_below_threshold - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_processpool.py::TestProcessPoolDownloader::test_large_download_exits_quickly_on_exception - botocore.exceptions.NoCredentialsError: Unable t...
ERROR tests/integration/test_processpool.py::TestProcessPoolDownloader::test_many_files_exits_quickly_on_exception - botocore.exceptions.NoCredentialsError: Unable to lo...
ERROR tests/integration/test_s3transfer.py::TestS3Transfers::test_callback_called_once_with_sigv4 - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_s3transfer.py::TestS3Transfers::test_can_configure_threshold - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_s3transfer.py::TestS3Transfers::test_can_send_extra_params_on_download - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_s3transfer.py::TestS3Transfers::test_can_send_extra_params_on_upload - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_s3transfer.py::TestS3Transfers::test_download_above_threshold - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_s3transfer.py::TestS3Transfers::test_download_below_threshold - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_s3transfer.py::TestS3Transfers::test_progress_callback_on_download - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_s3transfer.py::TestS3Transfers::test_progress_callback_on_upload - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_s3transfer.py::TestS3Transfers::test_upload_above_threshold - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_s3transfer.py::TestS3Transfers::test_upload_below_threshold - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_s3transfer.py::TestS3Transfers::test_upload_file_above_threshold_with_acl - botocore.exceptions.NoCredentialsError: Unable to locate credent...
ERROR tests/integration/test_s3transfer.py::TestS3Transfers::test_upload_file_above_threshold_with_ssec - botocore.exceptions.NoCredentialsError: Unable to locate creden...
ERROR tests/integration/test_upload.py::TestUpload::test_large_upload_exits_quicky_on_exception - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_upload.py::TestUpload::test_many_files_exits_quicky_on_exception - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_upload.py::TestUpload::test_progress_subscribers_on_upload - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_upload.py::TestUpload::test_upload_above_threshold - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_upload.py::TestUpload::test_upload_below_threshold - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_upload.py::TestUploadSeekableStream::test_large_upload_exits_quicky_on_exception - botocore.exceptions.NoCredentialsError: Unable to locate ...
ERROR tests/integration/test_upload.py::TestUploadSeekableStream::test_many_files_exits_quicky_on_exception - botocore.exceptions.NoCredentialsError: Unable to locate cr...
ERROR tests/integration/test_upload.py::TestUploadSeekableStream::test_progress_subscribers_on_upload - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_upload.py::TestUploadSeekableStream::test_upload_above_threshold - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_upload.py::TestUploadSeekableStream::test_upload_below_threshold - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_upload.py::TestUploadNonSeekableStream::test_large_upload_exits_quicky_on_exception - botocore.exceptions.NoCredentialsError: Unable to loca...
ERROR tests/integration/test_upload.py::TestUploadNonSeekableStream::test_many_files_exits_quicky_on_exception - botocore.exceptions.NoCredentialsError: Unable to locate...
ERROR tests/integration/test_upload.py::TestUploadNonSeekableStream::test_progress_subscribers_on_upload - botocore.exceptions.NoCredentialsError: Unable to locate crede...
ERROR tests/integration/test_upload.py::TestUploadNonSeekableStream::test_upload_above_threshold - botocore.exceptions.NoCredentialsError: Unable to locate credentials
ERROR tests/integration/test_upload.py::TestUploadNonSeekableStream::test_upload_below_threshold - botocore.exceptions.NoCredentialsError: Unable to locate credentials
==================================================== 603 passed, 28 skipped, 26 warnings, 45 errors in 65.17s (0:01:05) ====================================================

Test error - ERROR: setUpClass (tests.integration.test_s3transfer.TestS3Transfers)

Running the tests on F24 (next Fedora release), this is what I get:


ERROR: setUpClass (tests.integration.test_s3transfer.TestS3Transfers)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/s3transfer-0.0.1/tests/integration/test_s3transfer.py", line 118, in setUpClass
    CreateBucketConfiguration={'LocationConstraint': cls.region})
  File "/usr/lib/python3.5/site-packages/botocore/client.py", line 301, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python3.5/site-packages/botocore/client.py", line 387, in _make_api_call
    operation_model, request_dict)
  File "/usr/lib/python3.5/site-packages/botocore/endpoint.py", line 111, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/lib/python3.5/site-packages/botocore/endpoint.py", line 136, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/usr/lib/python3.5/site-packages/botocore/endpoint.py", line 120, in create_request
    operation_name=operation_model.name)
  File "/usr/lib/python3.5/site-packages/botocore/hooks.py", line 226, in emit
    return self._emit(event_name, kwargs)
  File "/usr/lib/python3.5/site-packages/botocore/hooks.py", line 209, in _emit
    response = handler(**kwargs)
  File "/usr/lib/python3.5/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/usr/lib/python3.5/site-packages/botocore/signers.py", line 123, in sign
    signature_version)
  File "/usr/lib/python3.5/site-packages/botocore/signers.py", line 153, in get_auth_instance
    kwargs['credentials'] = self._credentials.get_frozen_credentials()
AttributeError: 'NoneType' object has no attribute 'get_frozen_credentials'

Full build log
Environment creation

Support Python 3.6

Python 3.6 got released, and some distro (like Fedora) are swithcing to it.

Reading entire file into memory vs using lock to share file among threads

I recently upgraded to boto3 v1.4.0 from v1.2.3. The new version now uses s3transfer to support uploads, and boto3 Bucket objects now provide a method upload_fileobj for uploading files, which is very nice. The upload_fileobj method delegates to s3transfer. However, it appears that s3transfer reads the entire file into memory when chunking very large files, which has the potential to exhaust RAM. Would it make sense to use a lock to allow multiple threads to access the fileobj instead?

Feel free to reuse the code from that final link. I wrote it and would be happy to have it used here.

Submission thread seems to block on unrelated tags

I'm working in some of the future_tag code, and I'm seeing (https://github.com/boto/s3transfer/blob/develop/s3transfer/futures.py#L328)

        with self._lock:
            # Wait if there is a maximum reached
            self._wait_for_futures_to_complete_if_needed(tag)
            # Submit the task to the executor
            future = self._executor.submit(fn, *args, **kwargs)
            # Add the recently added future to the currently running futures
            self._add_future_to_currently_running(future, tag)
        return future

So suppose I have a max tags of {"A": 1, "B": 1} and sequence such as:

T1: submit tag A --> noblock
T1: submit tag A --> block on waiting futures
T2: submit tag B --> block on lock acquire

Where T1/T2 are threads and A/B are tags. This means that even though T2 is submitting tag B which has no futures, the submission on tag B will still block on the with self._lock until a future for tag A completes.

AWS_S3_SIGNATURE_VERSION Environment Variable

I have noticed that when I use the AWSCLI, the AWS_S3_SIGNATURE_VERSION environment variable is not respected. In my situation, I must use AWS_S3_SIGNATURE_VERSION=s3. However, if I set this on the CLI, this is not propagated to the botocore library.

As a "patch", I have updated the botocore/client.py to handle this by having it read the environment variable directly. However, I suspect there is a cleaner way to do this.

Include ExpectedBucketOwner to the Allowed List of upload arguments for multipart upload

Hello,

Currently we are using s3transfer to do s3 upload and would like to make sure we pass "ExpectedBucketOwner" as an argument to protect against bucket sniping when uploading files. However, one bug we found is that s3transfer manager.py is missing "ExpectedBucketOwner" in its ALLOWED_UPLOAD_ARGS list and thus the validation method is throwing the below error, though base api allows "ExpectedBucketOwner" as one of the ExtraArgs

#182

Error triage below

Traceback (most recent call last):
File "/opt/amazon/lib/python3.6/site-packages/deepinsight_document_classification/asynchronous/main.py", line 198, in upload_model
extra_args=extra_args)
File "/opt/amazon/lib/python3.6/site-packages/deepinsight_document_classification/s3.py", line 59, in upload_file
self._s3transfer.upload_file(filename, bucket, key, extra_args=extra_args)
File "/opt/amazon/lib/python3.6/site-packages/boto3/s3/transfer.py", line 277, in upload_file
filename, bucket, key, extra_args, subscribers)
File "/opt/amazon/lib/python3.6/site-packages/s3transfer/manager.py", line 304, in upload
self._validate_all_known_args(extra_args, self.ALLOWED_UPLOAD_ARGS)
File "/opt/amazon/lib/python3.6/site-packages/s3transfer/manager.py", line 446, in _validate_all_known_args
kwarg, ', '.join(allowed)))
"ValueError: Invalid extra_args key 'ExpectedBucketOwner', must be one of: ACL, CacheControl, ContentDisposition, ContentEncoding, ContentLanguage, ContentType, Expires, GrantFullControl, GrantRead, GrantReadACP, GrantWriteACP, Metadata, RequestPayer, ServerSideEncryption, StorageClass, SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5, SSEKMSKeyId, Tagging, WebsiteRedirectLocation"

Stubbed test fail

I think I'm missing something, because there are a lot of problems with stubbed tests. Is there something that needs to be done before running the tests? I've not found anything in the doc:

ERROR: test_for_callback_kwargs_correctness (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 404, in test_for_callback_kwargs_correctness
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_has_transfer_id_associated_to_future (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 375, in test_has_transfer_id_associated_to_future
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_invalid_extra_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 397, in test_invalid_extra_args
self.method(
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 302, in method
raise NotImplementedError('method is not implemented')
NotImplementedError: method is not implemented

ERROR: test_returns_correct_call_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 360, in test_returns_correct_call_args
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_returns_future_with_meta (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 349, in test_returns_future_with_meta
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_for_callback_kwargs_correctness (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 404, in test_for_callback_kwargs_correctness
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_has_transfer_id_associated_to_future (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 375, in test_has_transfer_id_associated_to_future
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_invalid_extra_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 397, in test_invalid_extra_args
self.method(
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 302, in method
raise NotImplementedError('method is not implemented')
NotImplementedError: method is not implemented

ERROR: test_returns_correct_call_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 360, in test_returns_correct_call_args
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_returns_future_with_meta (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 349, in test_returns_future_with_meta
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_for_callback_kwargs_correctness (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 404, in test_for_callback_kwargs_correctness
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_has_transfer_id_associated_to_future (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 375, in test_has_transfer_id_associated_to_future
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_invalid_extra_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 397, in test_invalid_extra_args
self.method(
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 302, in method
raise NotImplementedError('method is not implemented')
NotImplementedError: method is not implemented

ERROR: test_returns_correct_call_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 360, in test_returns_correct_call_args
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_returns_future_with_meta (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 349, in test_returns_future_with_meta
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_for_callback_kwargs_correctness (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 404, in test_for_callback_kwargs_correctness
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_has_transfer_id_associated_to_future (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 375, in test_has_transfer_id_associated_to_future
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_invalid_extra_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 397, in test_invalid_extra_args
self.method(
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 302, in method
raise NotImplementedError('method is not implemented')
NotImplementedError: method is not implemented

ERROR: test_returns_correct_call_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 360, in test_returns_correct_call_args
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_returns_future_with_meta (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 349, in test_returns_future_with_meta
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_for_callback_kwargs_correctness (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 404, in test_for_callback_kwargs_correctness
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_has_transfer_id_associated_to_future (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 375, in test_has_transfer_id_associated_to_future
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_invalid_extra_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 397, in test_invalid_extra_args
self.method(
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 302, in method
raise NotImplementedError('method is not implemented')
NotImplementedError: method is not implemented

ERROR: test_returns_correct_call_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 360, in test_returns_correct_call_args
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_returns_future_with_meta (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 349, in test_returns_future_with_meta
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_for_callback_kwargs_correctness (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 404, in test_for_callback_kwargs_correctness
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_has_transfer_id_associated_to_future (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 375, in test_has_transfer_id_associated_to_future
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_invalid_extra_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 397, in test_invalid_extra_args
self.method(
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 302, in method
raise NotImplementedError('method is not implemented')
NotImplementedError: method is not implemented

ERROR: test_returns_correct_call_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 360, in test_returns_correct_call_args
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_returns_future_with_meta (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 349, in test_returns_future_with_meta
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_for_callback_kwargs_correctness (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 404, in test_for_callback_kwargs_correctness
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_has_transfer_id_associated_to_future (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 375, in test_has_transfer_id_associated_to_future
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_invalid_extra_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 397, in test_invalid_extra_args
self.method(
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 302, in method
raise NotImplementedError('method is not implemented')
NotImplementedError: method is not implemented

ERROR: test_returns_correct_call_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 360, in test_returns_correct_call_args
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_returns_future_with_meta (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 349, in test_returns_future_with_meta
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_for_callback_kwargs_correctness (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 404, in test_for_callback_kwargs_correctness
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_has_transfer_id_associated_to_future (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 375, in test_has_transfer_id_associated_to_future
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_invalid_extra_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 397, in test_invalid_extra_args
self.method(
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 302, in method
raise NotImplementedError('method is not implemented')
NotImplementedError: method is not implemented

ERROR: test_returns_correct_call_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 360, in test_returns_correct_call_args
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_returns_future_with_meta (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 349, in test_returns_future_with_meta
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_for_callback_kwargs_correctness (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 404, in test_for_callback_kwargs_correctness
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_has_transfer_id_associated_to_future (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 375, in test_has_transfer_id_associated_to_future
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_invalid_extra_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 397, in test_invalid_extra_args
self.method(
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 302, in method
raise NotImplementedError('method is not implemented')
NotImplementedError: method is not implemented

ERROR: test_returns_correct_call_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 360, in test_returns_correct_call_args
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_returns_future_with_meta (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 349, in test_returns_future_with_meta
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_for_callback_kwargs_correctness (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 404, in test_for_callback_kwargs_correctness
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_has_transfer_id_associated_to_future (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 375, in test_has_transfer_id_associated_to_future
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_invalid_extra_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 397, in test_invalid_extra_args
self.method(
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 302, in method
raise NotImplementedError('method is not implemented')
NotImplementedError: method is not implemented

ERROR: test_returns_correct_call_args (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 360, in test_returns_correct_call_args
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

ERROR: test_returns_future_with_meta (tests.BaseGeneralInterfaceTest)

Traceback (most recent call last):
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 349, in test_returns_future_with_meta
self._setup_default_stubbed_responses()
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 345, in _setup_default_stubbed_responses
for stubbed_response in self.create_stubbed_responses():
File "/builddir/build/BUILD/s3transfer-0.1.0/tests/init.py", line 322, in create_stubbed_responses
'create_stubbed_responses is not implemented')
NotImplementedError: create_stubbed_responses is not implemented

Ran 1241 tests in 18.075s

FAILED (errors=50)

`S3.Object.copy()` fails with multipart and `ChecksumAlgorithm`

Describe the bug

If you try to copy an object with multipart and create checksums for the destination object it will fail.

Note: using copy with small objects doesn't fail, it does fail in objects whose size is above multipart_threshold.
Note 2: using copy from s3 client has the same effect.

Expected Behavior

copy method should work for multipart object with checksums.

Current Behavior

Running copy with multipart and ChecksumAlgorithm set to SHA256 throws the message:

botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the CompleteMultipartUpload operation: The upload was created using a sha256 checksum. The complete request must include the checksum for each part. It was missing for part 1 in the request.

Reproduction Steps

Run the following code replacing bucket and key accordingly

import boto3
s3= boto3.resource("s3")

dest_bucket = "bucket"
dest_key = "key"
copy_source = {"Bucket": "bucket", "Key": "key"}

s3.Object(dest_bucket, dest_key).copy(
    CopySource=copy_source,
    ExtraArgs={"ChecksumAlgorithm": "SHA256"}
)

and you'll get the following error
Full traceback:

 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File ".venv/lib/python3.7/site-packages/boto3/s3/inject.py", line 572, in object_copy
     Config=Config,
   File ".venv/lib/python3.7/site-packages/boto3/s3/inject.py", line 444, in copy
     return future.result()
   File ".venv/lib/python3.7/site-packages/s3transfer/futures.py", line 103, in result
     return self._coordinator.result()
   File ".venv/lib/python3.7/site-packages/s3transfer/futures.py", line 266, in result
     raise self._exception
   File ".venv/lib/python3.7/site-packages/s3transfer/tasks.py", line 139, in __call__
     return self._execute_main(kwargs)
   File ".venv/lib/python3.7/site-packages/s3transfer/tasks.py", line 162, in _execute_main
     return_value = self._main(**kwargs)
   File ".venv/lib/python3.7/site-packages/s3transfer/tasks.py", line 387, in _main
     **extra_args,
   File ".venv/lib/python3.7/site-packages/botocore/client.py", line 508, in _api_call
     return self._make_api_call(operation_name, kwargs)
   File ".venv/lib/python3.7/site-packages/botocore/client.py", line 911, in _make_api_call
     raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the CompleteMultipartUpload operation: The upload was created using a sha256 checksum. The complete request must include the checksum for each part. It was missing for part 1 in the request.

Possible Solution

The problem resides inside CopyPartTask (s3transfer/copies.py) which doesn't return the checksum if it's in the response, grabbing the checksum from the request and adding it to the return statement fixes this issue

Additional Information/Context

Looking at the CompleteMultipartUpload request looks like it sends ETAG for each part, but not checksum.

Lib version:

  • botocore==1.27.1
  • boto3==1.24.1
  • s3transfer==0.6.0

SDK version used

1.24.1

Environment details (OS name and version, etc.)

MacOS 12.4 Python 3.7

TransferManager.shutdown() is not functional

In TransferManager.shutdown's call to TransferManager._shutdown(), it passes the same boolean parameter twice, thus setting the string param to a bool and exception_type to a string instead of a class

https://github.com/boto/s3transfer/blob/develop/s3transfer/manager.py#L546-L548

    def shutdown(self, cancel=False, cancel_msg=''):
        ...
        self._shutdown(cancel, cancel, cancel_msg)

    def _shutdown(self, cancel, cancel_msg, exc_type=CancelledError):

After fixing the params locally, attempting to cancel an in-progress s3 file transfer using shutdown(cancel=True) will result in deadlock

sample client code

s3client = boto3.client('s3')
transfer_config = boto3.s3.transfer.TransferConfig(max_concurrency=s3concurrency, multipart_threshold=multipart_threshold, use_threads=use_threads)
manager = boto3.s3.transfer.create_transfer_manager(client=s3, config=transfer_config)
transfer = boto3.s3.transfer.S3Transfer(manager=manager)

class AbortableProgress():
  def __init__(self, s3manager, progress_cb):
    self.manager = s3manager
    self.cb      = progress_cb
    self.start   = datetime.datetime.now()

  def __call__(self, *args, **kwargs):
     if self.cb:
       self.cb(*args, **kwargs)
     if self.manager:
       current = datetime.datetime.now()
       elapsed = (current-self.start).seconds
       if elapsed > 3:
         print("ABORT")
         self.manager.shutdown(cancel=True, cancel_msg="time to abort!")

transfer.download_file(sts['bucket'], sts['key'], local_filename, callback=AbortableProgress(manager, progress_callback))

0.5.2: pytest is failing

I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

  • python3 -sBm build -w --no-isolation
  • because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
  • install .whl file in </install/prefix>
  • run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-s3transfer-0.5.2-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-s3transfer-0.5.2-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra --ignore tests/integration
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.2, configfile: pyproject.toml
collected 621 items

tests/functional/test_copy.py F................................                                                                                                      [  5%]
tests/functional/test_crt.py sssssss                                                                                                                                 [  6%]
tests/functional/test_delete.py .......                                                                                                                              [  7%]
tests/functional/test_download.py F.........F....................F...........                                                                                        [ 14%]
tests/functional/test_manager.py ........                                                                                                                            [ 15%]
tests/functional/test_processpool.py ............                                                                                                                    [ 17%]
tests/functional/test_upload.py F.............F.....F.........                                                                                                       [ 22%]
tests/functional/test_utils.py ...                                                                                                                                   [ 23%]
tests/unit/test_bandwidth.py ....................................                                                                                                    [ 28%]
tests/unit/test_compat.py ........                                                                                                                                   [ 30%]
tests/unit/test_copies.py ......                                                                                                                                     [ 31%]
tests/unit/test_crt.py ssssssss                                                                                                                                      [ 32%]
tests/unit/test_delete.py ..                                                                                                                                         [ 32%]
tests/unit/test_download.py ..................................................................                                                                       [ 43%]
tests/unit/test_futures.py .........................................................                                                                                 [ 52%]
tests/unit/test_manager.py .........                                                                                                                                 [ 53%]
tests/unit/test_processpool.py ...................................................                                                                                   [ 62%]
tests/unit/test_s3transfer.py .........................................................                                                                              [ 71%]
tests/unit/test_subscribers.py ........                                                                                                                              [ 72%]
tests/unit/test_tasks.py ...............................                                                                                                             [ 77%]
tests/unit/test_upload.py ...............................................                                                                                            [ 85%]
tests/unit/test_utils.py ............................................................................................                                                [100%]

================================================================================= FAILURES =================================================================================
_________________________________________________________ TestNonMultipartCopy.test_allowed_copy_params_are_valid __________________________________________________________

self = <tests.functional.test_copy.TestNonMultipartCopy testMethod=test_allowed_copy_params_are_valid>

    def test_allowed_copy_params_are_valid(self):
        op_model = self.client.meta.service_model.operation_model('CopyObject')
        for allowed_upload_arg in self._manager.ALLOWED_COPY_ARGS:
>           self.assertIn(allowed_upload_arg, op_model.input_shape.members)
E           AssertionError: 'ChecksumAlgorithm' not found in OrderedDict([('ACL', <StringShape(ObjectCannedACL)>), ('Bucket', <StringShape(BucketName)>), ('CacheControl', <StringShape(CacheControl)>), ('ContentDisposition', <StringShape(ContentDisposition)>), ('ContentEncoding', <StringShape(ContentEncoding)>), ('ContentLanguage', <StringShape(ContentLanguage)>), ('ContentType', <StringShape(ContentType)>), ('CopySource', <StringShape(CopySource)>), ('CopySourceIfMatch', <StringShape(CopySourceIfMatch)>), ('CopySourceIfModifiedSince', <Shape(CopySourceIfModifiedSince)>), ('CopySourceIfNoneMatch', <StringShape(CopySourceIfNoneMatch)>), ('CopySourceIfUnmodifiedSince', <Shape(CopySourceIfUnmodifiedSince)>), ('Expires', <Shape(Expires)>), ('GrantFullControl', <StringShape(GrantFullControl)>), ('GrantRead', <StringShape(GrantRead)>), ('GrantReadACP', <StringShape(GrantReadACP)>), ('GrantWriteACP', <StringShape(GrantWriteACP)>), ('Key', <StringShape(ObjectKey)>), ('Metadata', <MapShape(Metadata)>), ('MetadataDirective', <StringShape(MetadataDirective)>), ('TaggingDirective', <StringShape(TaggingDirective)>), ('ServerSideEncryption', <StringShape(ServerSideEncryption)>), ('StorageClass', <StringShape(StorageClass)>), ('WebsiteRedirectLocation', <StringShape(WebsiteRedirectLocation)>), ('SSECustomerAlgorithm', <StringShape(SSECustomerAlgorithm)>), ('SSECustomerKey', <StringShape(SSECustomerKey)>), ('SSECustomerKeyMD5', <StringShape(SSECustomerKeyMD5)>), ('SSEKMSKeyId', <StringShape(SSEKMSKeyId)>), ('SSEKMSEncryptionContext', <StringShape(SSEKMSEncryptionContext)>), ('BucketKeyEnabled', <Shape(BucketKeyEnabled)>), ('CopySourceSSECustomerAlgorithm', <StringShape(CopySourceSSECustomerAlgorithm)>), ('CopySourceSSECustomerKey', <StringShape(CopySourceSSECustomerKey)>), ('CopySourceSSECustomerKeyMD5', <StringShape(CopySourceSSECustomerKeyMD5)>), ('RequestPayer', <StringShape(RequestPayer)>), ('Tagging', <StringShape(TaggingHeader)>), ('ObjectLockMode', <StringShape(ObjectLockMode)>), ('ObjectLockRetainUntilDate', <Shape(ObjectLockRetainUntilDate)>), ('ObjectLockLegalHoldStatus', <StringShape(ObjectLockLegalHoldStatus)>), ('ExpectedBucketOwner', <StringShape(AccountId)>), ('ExpectedSourceBucketOwner', <StringShape(AccountId)>)])

tests/functional/test_copy.py:257: AssertionError
_________________________________________________________ TestNonRangedDownload.test_allowed_copy_params_are_valid _________________________________________________________

self = <tests.functional.test_download.TestNonRangedDownload testMethod=test_allowed_copy_params_are_valid>

    def test_allowed_copy_params_are_valid(self):
        op_model = self.client.meta.service_model.operation_model('GetObject')
        for allowed_upload_arg in self._manager.ALLOWED_DOWNLOAD_ARGS:
>           self.assertIn(allowed_upload_arg, op_model.input_shape.members)
E           AssertionError: 'ChecksumMode' not found in OrderedDict([('Bucket', <StringShape(BucketName)>), ('IfMatch', <StringShape(IfMatch)>), ('IfModifiedSince', <Shape(IfModifiedSince)>), ('IfNoneMatch', <StringShape(IfNoneMatch)>), ('IfUnmodifiedSince', <Shape(IfUnmodifiedSince)>), ('Key', <StringShape(ObjectKey)>), ('Range', <StringShape(Range)>), ('ResponseCacheControl', <StringShape(ResponseCacheControl)>), ('ResponseContentDisposition', <StringShape(ResponseContentDisposition)>), ('ResponseContentEncoding', <StringShape(ResponseContentEncoding)>), ('ResponseContentLanguage', <StringShape(ResponseContentLanguage)>), ('ResponseContentType', <StringShape(ResponseContentType)>), ('ResponseExpires', <Shape(ResponseExpires)>), ('VersionId', <StringShape(ObjectVersionId)>), ('SSECustomerAlgorithm', <StringShape(SSECustomerAlgorithm)>), ('SSECustomerKey', <StringShape(SSECustomerKey)>), ('SSECustomerKeyMD5', <StringShape(SSECustomerKeyMD5)>), ('RequestPayer', <StringShape(RequestPayer)>), ('PartNumber', <Shape(PartNumber)>), ('ExpectedBucketOwner', <StringShape(AccountId)>)])

tests/functional/test_download.py:404: AssertionError
________________________________________________________ TestNonRangedDownload.test_download_with_checksum_enabled _________________________________________________________

self = <tests.functional.test_download.TestNonRangedDownload testMethod=test_download_with_checksum_enabled>

    def test_download_with_checksum_enabled(self):
        self.extra_args['ChecksumMode'] = 'ENABLED'
        expected_params = {
            'Bucket': self.bucket,
            'Key': self.key,
            'ChecksumMode': 'ENABLED',
        }
        self.add_head_object_response(expected_params)
        self.add_successful_get_object_responses(expected_params)
        future = self.manager.download(
            self.bucket, self.key, self.filename, self.extra_args
        )
>       future.result()

tests/functional/test_download.py:395:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
s3transfer/futures.py:103: in result
    return self._coordinator.result()
s3transfer/futures.py:266: in result
    raise self._exception
s3transfer/tasks.py:269: in _main
    self._submit(transfer_future=transfer_future, **kwargs)
s3transfer/download.py:354: in _submit
    response = client.head_object(
/usr/lib/python3.8/site-packages/botocore/client.py:391: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    request_dict = self._convert_to_request_dict(
/usr/lib/python3.8/site-packages/botocore/client.py:739: in _convert_to_request_dict
    request_dict = self._serializer.serialize_to_request(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.validate.ParamValidationDecorator object at 0x7fb1c5a95670>, parameters = {'Bucket': 'mybucket', 'ChecksumMode': 'ENABLED', 'Key': 'mykey'}
operation_model = OperationModel(name=HeadObject)

    def serialize_to_request(self, parameters, operation_model):
        input_shape = operation_model.input_shape
        if input_shape is not None:
            report = self._param_validator.validate(parameters,
                                                    operation_model.input_shape)
            if report.has_errors():
>               raise ParamValidationError(report=report.generate_report())
E               botocore.exceptions.ParamValidationError: Parameter validation failed:
E               Unknown parameter in input: "ChecksumMode", must be one of: Bucket, IfMatch, IfModifiedSince, IfNoneMatch, IfUnmodifiedSince, Key, Range, VersionId, SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5, RequestPayer, PartNumber, ExpectedBucketOwner

/usr/lib/python3.8/site-packages/botocore/validate.py:360: ParamValidationError
__________________________________________________________ TestRangedDownload.test_download_with_checksum_enabled __________________________________________________________

self = <tests.functional.test_download.TestRangedDownload testMethod=test_download_with_checksum_enabled>

    def test_download_with_checksum_enabled(self):
        self.extra_args['ChecksumMode'] = 'ENABLED'
        expected_params = {
            'Bucket': self.bucket,
            'Key': self.key,
            'ChecksumMode': 'ENABLED',
        }
        expected_ranges = ['bytes=0-3', 'bytes=4-7', 'bytes=8-']
        self.add_head_object_response(expected_params)
        self.add_successful_get_object_responses(
            expected_params, expected_ranges
        )

        future = self.manager.download(
            self.bucket, self.key, self.filename, self.extra_args
        )
>       future.result()

tests/functional/test_download.py:533:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
s3transfer/futures.py:103: in result
    return self._coordinator.result()
s3transfer/futures.py:266: in result
    raise self._exception
s3transfer/tasks.py:269: in _main
    self._submit(transfer_future=transfer_future, **kwargs)
s3transfer/download.py:354: in _submit
    response = client.head_object(
/usr/lib/python3.8/site-packages/botocore/client.py:391: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    request_dict = self._convert_to_request_dict(
/usr/lib/python3.8/site-packages/botocore/client.py:739: in _convert_to_request_dict
    request_dict = self._serializer.serialize_to_request(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.validate.ParamValidationDecorator object at 0x7fb1c5b6da90>, parameters = {'Bucket': 'mybucket', 'ChecksumMode': 'ENABLED', 'Key': 'mykey'}
operation_model = OperationModel(name=HeadObject)

    def serialize_to_request(self, parameters, operation_model):
        input_shape = operation_model.input_shape
        if input_shape is not None:
            report = self._param_validator.validate(parameters,
                                                    operation_model.input_shape)
            if report.has_errors():
>               raise ParamValidationError(report=report.generate_report())
E               botocore.exceptions.ParamValidationError: Parameter validation failed:
E               Unknown parameter in input: "ChecksumMode", must be one of: Bucket, IfMatch, IfModifiedSince, IfNoneMatch, IfUnmodifiedSince, Key, Range, VersionId, SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5, RequestPayer, PartNumber, ExpectedBucketOwner

/usr/lib/python3.8/site-packages/botocore/validate.py:360: ParamValidationError
_______________________________________________________ TestNonMultipartUpload.test_allowed_upload_params_are_valid ________________________________________________________

self = <tests.functional.test_upload.TestNonMultipartUpload testMethod=test_allowed_upload_params_are_valid>

    def test_allowed_upload_params_are_valid(self):
        op_model = self.client.meta.service_model.operation_model('PutObject')
        for allowed_upload_arg in self._manager.ALLOWED_UPLOAD_ARGS:
>           self.assertIn(allowed_upload_arg, op_model.input_shape.members)
E           AssertionError: 'ChecksumAlgorithm' not found in OrderedDict([('ACL', <StringShape(ObjectCannedACL)>), ('Body', <Shape(Body)>), ('Bucket', <StringShape(BucketName)>), ('CacheControl', <StringShape(CacheControl)>), ('ContentDisposition', <StringShape(ContentDisposition)>), ('ContentEncoding', <StringShape(ContentEncoding)>), ('ContentLanguage', <StringShape(ContentLanguage)>), ('ContentLength', <Shape(ContentLength)>), ('ContentMD5', <StringShape(ContentMD5)>), ('ContentType', <StringShape(ContentType)>), ('Expires', <Shape(Expires)>), ('GrantFullControl', <StringShape(GrantFullControl)>), ('GrantRead', <StringShape(GrantRead)>), ('GrantReadACP', <StringShape(GrantReadACP)>), ('GrantWriteACP', <StringShape(GrantWriteACP)>), ('Key', <StringShape(ObjectKey)>), ('Metadata', <MapShape(Metadata)>), ('ServerSideEncryption', <StringShape(ServerSideEncryption)>), ('StorageClass', <StringShape(StorageClass)>), ('WebsiteRedirectLocation', <StringShape(WebsiteRedirectLocation)>), ('SSECustomerAlgorithm', <StringShape(SSECustomerAlgorithm)>), ('SSECustomerKey', <StringShape(SSECustomerKey)>), ('SSECustomerKeyMD5', <StringShape(SSECustomerKeyMD5)>), ('SSEKMSKeyId', <StringShape(SSEKMSKeyId)>), ('SSEKMSEncryptionContext', <StringShape(SSEKMSEncryptionContext)>), ('BucketKeyEnabled', <Shape(BucketKeyEnabled)>), ('RequestPayer', <StringShape(RequestPayer)>), ('Tagging', <StringShape(TaggingHeader)>), ('ObjectLockMode', <StringShape(ObjectLockMode)>), ('ObjectLockRetainUntilDate', <Shape(ObjectLockRetainUntilDate)>), ('ObjectLockLegalHoldStatus', <StringShape(ObjectLockLegalHoldStatus)>), ('ExpectedBucketOwner', <StringShape(AccountId)>)])

tests/functional/test_upload.py:264: AssertionError
_____________________________________________________________ TestNonMultipartUpload.test_upload_with_checksum _____________________________________________________________

self = <tests.functional.test_upload.TestNonMultipartUpload testMethod=test_upload_with_checksum>

    def test_upload_with_checksum(self):
        self.extra_args['ChecksumAlgorithm'] = 'crc32'
        self.add_put_object_response_with_default_expected_params(
            extra_expected_params={'ChecksumAlgorithm': 'crc32'}
        )
        future = self.manager.upload(
            self.filename, self.bucket, self.key, self.extra_args
        )
>       future.result()

tests/functional/test_upload.py:177:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
s3transfer/futures.py:103: in result
    return self._coordinator.result()
s3transfer/futures.py:266: in result
    raise self._exception
s3transfer/tasks.py:139: in __call__
    return self._execute_main(kwargs)
s3transfer/tasks.py:162: in _execute_main
    return_value = self._main(**kwargs)
s3transfer/upload.py:758: in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
/usr/lib/python3.8/site-packages/botocore/client.py:391: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/lib/python3.8/site-packages/botocore/client.py:691: in _make_api_call
    request_dict = self._convert_to_request_dict(
/usr/lib/python3.8/site-packages/botocore/client.py:739: in _convert_to_request_dict
    request_dict = self._serializer.serialize_to_request(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <botocore.validate.ParamValidationDecorator object at 0x7fb1c5dcf4c0>
parameters = {'Body': <s3transfer.utils.ReadFileChunk object at 0x7fb1c6d23610>, 'Bucket': 'mybucket', 'ChecksumAlgorithm': 'crc32', 'Key': 'mykey'}
operation_model = OperationModel(name=PutObject)

    def serialize_to_request(self, parameters, operation_model):
        input_shape = operation_model.input_shape
        if input_shape is not None:
            report = self._param_validator.validate(parameters,
                                                    operation_model.input_shape)
            if report.has_errors():
>               raise ParamValidationError(report=report.generate_report())
E               botocore.exceptions.ParamValidationError: Parameter validation failed:
E               Unknown parameter in input: "ChecksumAlgorithm", must be one of: ACL, Body, Bucket, CacheControl, ContentDisposition, ContentEncoding, ContentLanguage, ContentLength, ContentMD5, ContentType, Expires, GrantFullControl, GrantRead, GrantReadACP, GrantWriteACP, Key, Metadata, ServerSideEncryption, StorageClass, WebsiteRedirectLocation, SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5, SSEKMSKeyId, SSEKMSEncryptionContext, BucketKeyEnabled, RequestPayer, Tagging, ObjectLockMode, ObjectLockRetainUntilDate, ObjectLockLegalHoldStatus, ExpectedBucketOwner

/usr/lib/python3.8/site-packages/botocore/validate.py:360: ParamValidationError
________________________________________________________ TestMultipartUpload.test_multipart_upload_passes_checksums ________________________________________________________

self = <tests.functional.test_upload.TestMultipartUpload testMethod=test_multipart_upload_passes_checksums>

    def test_multipart_upload_passes_checksums(self):
        self.extra_args['ChecksumAlgorithm'] = 'sha1'

        # ChecksumAlgorithm should be passed on the create_multipart call
        self.add_create_multipart_response_with_default_expected_params(
            extra_expected_params={'ChecksumAlgorithm': 'sha1'},
        )

        # ChecksumAlgorithm should be forwarded and a SHA1 will come back
>       self.add_upload_part_responses_with_default_expected_params(
            extra_expected_params={'ChecksumAlgorithm': 'sha1'},
        )

tests/functional/test_upload.py:568:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/functional/test_upload.py:378: in add_upload_part_responses_with_default_expected_params
    self.stubber.add_response(**upload_part_response)
/usr/lib/python3.8/site-packages/botocore/stub.py:228: in add_response
    self._add_response(method, service_response, expected_params)
/usr/lib/python3.8/site-packages/botocore/stub.py:240: in _add_response
    self._validate_response(operation_name, service_response)
/usr/lib/python3.8/site-packages/botocore/stub.py:390: in _validate_response
    validate_parameters(response, output_shape)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

params = {'ChecksumSHA1': 'sum1==', 'ETag': 'etag-1'}, shape = <StructureShape(UploadPartOutput)>

    def validate_parameters(params, shape):
        """Validates input parameters against a schema.

        This is a convenience function that validates parameters against a schema.
        You can also instantiate and use the ParamValidator class directly if you
        want more control.

        If there are any validation errors then a ParamValidationError
        will be raised.  If there are no validation errors than no exception
        is raised and a value of None is returned.

        :param params: The user provided input parameters.

        :type shape: botocore.model.Shape
        :param shape: The schema which the input parameters should
            adhere to.

        :raise: ParamValidationError

        """
        validator = ParamValidator()
        report = validator.validate(params, shape)
        if report.has_errors():
>           raise ParamValidationError(report=report.generate_report())
E           botocore.exceptions.ParamValidationError: Parameter validation failed:
E           Unknown parameter in input: "ChecksumSHA1", must be one of: ServerSideEncryption, ETag, SSECustomerAlgorithm, SSECustomerKeyMD5, SSEKMSKeyId, BucketKeyEnabled, RequestCharged

/usr/lib/python3.8/site-packages/botocore/validate.py:49: ParamValidationError
========================================================================= short test summary info ==========================================================================
SKIPPED [1] tests/functional/test_crt.py:186: Test requires awscrt to be installed.
SKIPPED [1] tests/functional/test_crt.py:220: Test requires awscrt to be installed.
SKIPPED [1] tests/functional/test_crt.py:259: Test requires awscrt to be installed.
SKIPPED [1] tests/functional/test_crt.py:164: Test requires awscrt to be installed.
SKIPPED [1] tests/functional/test_crt.py:132: Test requires awscrt to be installed.
SKIPPED [1] tests/functional/test_crt.py:237: Test requires awscrt to be installed.
SKIPPED [1] tests/functional/test_crt.py:110: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:89: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:69: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:49: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:123: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:131: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:159: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:169: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:164: Test requires awscrt to be installed.
FAILED tests/functional/test_copy.py::TestNonMultipartCopy::test_allowed_copy_params_are_valid - AssertionError: 'ChecksumAlgorithm' not found in OrderedDict([('ACL', <S...
FAILED tests/functional/test_download.py::TestNonRangedDownload::test_allowed_copy_params_are_valid - AssertionError: 'ChecksumMode' not found in OrderedDict([('Bucket',...
FAILED tests/functional/test_download.py::TestNonRangedDownload::test_download_with_checksum_enabled - botocore.exceptions.ParamValidationError: Parameter validation fai...
FAILED tests/functional/test_download.py::TestRangedDownload::test_download_with_checksum_enabled - botocore.exceptions.ParamValidationError: Parameter validation failed:
FAILED tests/functional/test_upload.py::TestNonMultipartUpload::test_allowed_upload_params_are_valid - AssertionError: 'ChecksumAlgorithm' not found in OrderedDict([('AC...
FAILED tests/functional/test_upload.py::TestNonMultipartUpload::test_upload_with_checksum - botocore.exceptions.ParamValidationError: Parameter validation failed:
FAILED tests/functional/test_upload.py::TestMultipartUpload::test_multipart_upload_passes_checksums - botocore.exceptions.ParamValidationError: Parameter validation failed:
================================================================ 7 failed, 599 passed, 15 skipped in 34.65s ================================================================

README warning

Is the warning in the README still relevent (re: work in progress). s3transfer is required by boto3, do we therefore not consider boto3 1.4.x to be production ready?

We audit all OSS in our company and s3transfer is being flagged due to the warning in the README

Allow X-Robots-Tag in the Upload Headers

The Problem

The company I work for has a lot of user generated content, and we generate sitemaps for these and upload them to S3. When users search for our content on google, they often stumble across our sitemaps. We would like for this to not happen.

I found this SO thread that explains how this can be done, which is by specifying the X-Robots-Tag as a header of the S3 object. We tried supplying this data, but we soon realized only these metadata options are allowed.

The Request

We would like to have the X-Robots-Tag extra param available for the upload. This way google won't serve our sitemaps on search.

We use boto3==1.7.48, and it looks like it uses the latest of this library. See here.

TransferManager.download fails when file has zero bytes

It seems that if I attempt to download a file using a target filename, if the downloaded file has a size of zero bytes, an exception is raised.

import boto3, s3transfer.manager
c = boto3.client('s3')
mgr = s3transfer.manager.TransferManager(c)
f = mgr.download('test-bucket', 'zero-byte-file', '/tmp/zero-byte-file.download')
f.result()

raises:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "../venv/lib/python2.7/site-packages/s3transfer/futures.py", line 71, in result
    return self._coordinator.result()
  File "../venv/lib/python2.7/site-packages/s3transfer/futures.py", line 220, in result
    raise self._exception
OSError: [Errno 2] No such file or directory

I think this is caused by DeferredOpenFile. _open_if_needed is never being being called since the file is zero-bytes, and there is no attempt to write to the file. And since the file is never opened, it is never created. Then when the file is renamed, it cannot be found.

This does not appear to happen in version 0.1.4. Version 0.1.4 does not appear to have been using the DeferredOpenFile class for DownloadFilenameOutputManager

TransferConfig documentation about '0' value is wrong

From the docstring:

        :param max_request_queue_size: The maximum amount of S3 API requests
            that can be queued at a time. A value of zero means that there
            is no maximum.

But:

>>> TransferConfig(max_request_queue_size=0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "s3transfer/s3transfer/manager.py", line 147, in __init__
    self._validate_attrs_are_nonzero()
  File "s3transfer/s3transfer/manager.py", line 154, in _validate_attrs_are_nonzero
    '0.' % (attr, attr_val))
ValueError: Provided parameter max_request_queue_size of value 0 must be greater than 0.

I believe this applies to max_submission_queue_size, and max_io_queue_size as well.

TransferManager.copy uses ExpectedBucketOwner in head_object request on source object

The copy operations executes a head_object request on the source object, and uses the ExpectedBucketOwner argument in this request. This results in Forbidden errors when source and destination buckets are owned by different owners.

>>> client.get_bucket_acl(Bucket='bucket-from')['Owner']['ID']
'50cea555-eac6-4373-bc27-7333bfe6abe0:50cea555-eac6-4373-bc27-7333bfe6abe0'
>>> client_to.get_bucket_acl(Bucket='bucket-to')['Owner']['ID']
'88e5206f-48b0-4cb6-b2db-fc26be02031d:88e5206f-48b0-4cb6-b2db-fc26be02031d'
>> source = {'Bucket': 'bucket-from', 'Key': 'foo'}
>>> client.head_object(**source, ExpectedBucketOwner= '50cea555-eac6-4373-bc27-7333bfe6abe0')['ContentLength']
3200000000
>>> client.copy(source, 'bucket-to', 'bar', ExtraArgs={'ExpectedBucketOwner': '88e5206f-48b0-4cb6-b2db-fc26be02031d'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/boto3/s3/inject.py", line 444, in copy
    return future.result()
  File "/usr/local/lib/python3.9/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/usr/local/lib/python3.9/site-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/s3transfer/tasks.py", line 269, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/s3transfer/copies.py", line 118, in _submit
    response = call_args.source_client.head_object(
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
>>> client.put_object(Bucket='bucket-to', Key='bar', ExpectedBucketOwner='88e5206f-48b0-4cb6-b2db-fc26be02031d')['ETag']
'"d41d8cd98f00b204e9800998ecf8427e"'

Latest Release Causes `futures` package to be included in Python3 installs

Hey,

You might see some more traffic here soon. It looks like the latest release, 0.3.5 fixed a type in setup.cfg that caused the futures package to be installed for all installs, not just Python 2.7. aws-cli pins this library to >=0.3.0, <0.4.0 so new installs will automatically grab this version and error out with something similar to the following.

pip, install, awscli) exited with code 1.     ERROR: Command errored out with exit status 1:
     command: /python3/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-dv9sw2br/futures/setup.py'"'"'; __file__='"'"'/tmp/pip-install-dv9sw2br/futures/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-dv9sw2br/futures/pip-egg-info
         cwd: /tmp/pip-install-dv9sw2br/futures/
    Complete output (4 lines):
    This backport is meant only for Python 2.
    It does not work on Python 3, and Python 3 users do not need it as the concurrent.futures package is available in the standard library.
    For projects that work on both Python 2 and 3, the dependency needs to be conditional on the Python version, like so:
    extras_require={':python_version == "2.7"': ['futures']}
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

To fix this issue we had to manually pin s3transfer==0.3.4 for now. Would be awesome if a fix could be released so installing s3transfer on python3 envs works again.

FileExistError when running script on windows [WinError 183]

The application calls run_sql from a SQLquery instance. While running run_sql the error occurs where it tries to rename a file in the local temp directory, generating the FileExistsError. Error happens at least 2 minutes that the run_sql routine is working. Tested on Windows 7 and Windows 10.

It looks like the temporary file to be removed is still opened when it tries to remove it. Problem might be in python implementation of tempfile.NamedTemporaryFile() on Windows. Python implementation of NamedTemporaryFile uses the O_TEMPORARY flag under Windows, which prevents the file from being reopened if the same flag is not provided which might cause process to hang on close, therefore it is not unlinked correctly in our case. 

`\futures.py in result(self)
104 # however if a KeyboardInterrupt is raised we want want to exit
105 # out of this and propogate the exception.
--> 106 return self._coordinator.result()
107 except KeyboardInterrupt as e:
108 self.cancel()

\futures.py in result(self)
263 # final result.
264 if self._exception:
--> 265 raise self._exception
266 return self._result
267

tasks.py in call(self)
124 # main() method.
125 if not self._transfer_coordinator.done():
--> 126 return self._execute_main(kwargs)
127 except Exception as e:
128 self._log_and_set_exception(e)

\tasks.py in _execute_main(self, kwargs)
148 )
149
--> 150 return_value = self._main(**kwargs)
151 # If the task is the final task, then set the TransferFuture's
152 # value to the return value from main().

download.py in _main(self, fileobj, final_filename, osutil)
601 def _main(self, fileobj, final_filename, osutil):
602 fileobj.close()
--> 603 osutil.rename_file(fileobj.name, final_filename)
604
605

utils.py in rename_file(self, current_filename, new_filename)
269
270 def rename_file(self, current_filename, new_filename):
--> 271 rename_file(current_filename, new_filename)
272
273 def is_special_file(cls, filename):

compat.py in rename_file(current_filename, new_filename)
31 # that exception.
32 raise
---> 33 os.rename(current_filename, new_filename)
34 else:
35 rename_file = os.rename

FileExistsError: [WinError 183] Cannot create a file when that file already exists`

Related:

Invalid extra_args key 'GrantWriteACP', must be one of ... 'GrantWriteACL'. Typo?

I'm getting this error when specifying GrantWriteACP in s3transfer's upload_file(extra_args). Is it a typo in this line? In the init.py, and the error message, both say allowed extra_args would be 'GrantWriteACL'. However, in all other boto's documentation, it is all referred to as GrantWriteACP.

And if I follow the same hack from this issue, the writeACP is actually properly applied.

transfer.ALLOWED_UPLOAD_ARGS.append('GrantWriteACP')
transfer.upload_file(filepath, s3.bucket_name, destination_key,
    callback=S3Progress(filepath), extra_args=extra_args.extra_args)

RuntimeError: cannot schedule new futures after interpreter shutdown

Reverting to boto3 1.17.53, which is the last version using s3transfer<0.4 solves the issue.

The calls to upload_fileobj and download_fileobj are performed within a thread via asyncio's run_in_executor.

Traceback (most recent call last):
...
    s3.download_fileobj(self.bucket_name, key, fp)
  File "/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 678, in download_fileobj
    return future.result()
  File "/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 255, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/venv/lib/python3.8/site-packages/s3transfer/download.py", line 355, in _submit
    self._submit_download_request(
  File "/venv/lib/python3.8/site-packages/s3transfer/download.py", line 384, in _submit_download_request
    self._transfer_coordinator.submit(
  File "/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 320, in submit
    future = executor.submit(task, tag=tag)
  File "/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 467, in submit
    future = ExecutorFuture(self._executor.submit(task))
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 181, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown

Traceback (most recent call last):
...
    s3.upload_fileobj(io.BytesIO(data), self.bucket_name, key)
  File "/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 536, in upload_fileobj
    future = manager.upload(
  File "/venv/lib/python3.8/site-packages/s3transfer/manager.py", line 326, in upload
    return self._submit_transfer(
  File "/venv/lib/python3.8/site-packages/s3transfer/manager.py", line 500, in _submit_transfer
    self._submission_executor.submit(
  File "/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 467, in submit
    future = ExecutorFuture(self._executor.submit(task))
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 181, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown

implement presigned urls in botocore

we should be able to generate presigned urls for the upload_file and upload_fileobj methods. Right now this returns:
UnknownClientMethodError: Client does not have method: upload_fileobj

Ensure all pieces come from the same object while do multi-part downloading.

I have done a quick search against the latest code on develop branch, the IFMatch prediction wasn't used in multi-part downloading task.
if someone patches or replaces the file during downloading(large file or slow network), the result may be end up as a mix-in of multiple revisions, especially when chunk size is small.

download.py::GetObjectTask::_main : set current_index earlier in the function

I'm getting an error saying

...
File "/usr/local/lib/python3.6/dist-packages/s3transfer/download.py", line 554, in _main
callbacks, start_index - current_index)
UnboundLocalError: local variable 'current_index' referenced before assignment

meaning there's an exception in the except clause of the try-except of the function.

Line 554 being

s3transfer/s3transfer/download.py

Line 554 in f506f39

callbacks, start_index - current_index)

And indeed, if an error is raised before current_index = start_index in line 530
https://github.com/boto/s3transfer/blob/develop/s3transfer/download.py#L530

then this will happen. Perhaps line 530 can be moved upwards so current_index gets a value? Or initialize it with None so that I can tell whether the exception was raised before/after line 530?

Thanks!

Supporting SSEKMSEncryptionContext when s3 upload extra_arg allow list, Invalid extra_args key 'SSEKMSEncryptionContext', must be one of:

Hello,

Currently we are using s3transfer to do s3 upload via s3-SSE. However, one issue we found is that s3transfer manager.py not supporting

Traceback (most recent call last):
  File "/opt/amazon/lib/python3.6/site-packages/deepinsight_document_classification/asynchronous/main.py", line 198, in upload_model
    extra_args=extra_args)
  File "/opt/amazon/lib/python3.6/site-packages/deepinsight_document_classification/s3.py", line 59, in upload_file
    self._s3transfer.upload_file(filename, bucket, key, extra_args=extra_args)
  File "/opt/amazon/lib/python3.6/site-packages/boto3/s3/transfer.py", line 277, in upload_file
    filename, bucket, key, extra_args, subscribers)
  File "/opt/amazon/lib/python3.6/site-packages/s3transfer/manager.py", line 304, in upload
    self._validate_all_known_args(extra_args, self.ALLOWED_UPLOAD_ARGS)
  File "/opt/amazon/lib/python3.6/site-packages/s3transfer/manager.py", line 446, in _validate_all_known_args
    kwarg, ', '.join(allowed)))
 "ValueError: Invalid extra_args key 'SSEKMSEncryptionContext', must be one of: ACL, CacheControl, ContentDisposition, ContentEncoding, ContentLanguage, ContentType, Expires, GrantFullControl, GrantRead, GrantReadACP, GrantWriteACP, Metadata, RequestPayer, ServerSideEncryption, StorageClass, SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5, SSEKMSKeyId, Tagging, WebsiteRedirectLocation"

Can you please help

0.5.0: pytest warnings

Just normal build, install and test cycle used on building package from non-root account:

  • "setup.py build"
  • "setup.py install --root </install/prefix>"
  • "pytest with PYTHONPATH pointing to setearch and sitelib inside </install/prefix>
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-s3transfer-0.5.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-s3transfer-0.5.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra --ignore tests/integration
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.11, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0
plugins: forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, httpbin-1.0.0, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, freezegun-0.4.2, case-1.5.3, isort-1.3.0, aspectlib-1.5.2, asyncio-0.15.1, toolbox-0.5, xprocess-0.17.1, aiohttp-0.3.0, checkdocs-2.7.0, mock-3.6.1, rerunfailures-9.1.1, requests-mock-1.9.3, cov-2.12.1, pyfakefs-4.5.0, flaky-3.7.0, hypothesis-6.14.0, benchmark-3.4.1, xdist-2.3.0, pylama-7.7.1, datadir-1.3.1, regressions-2.2.0, cases-3.6.3, Faker-8.10.0
collected 619 items

. .                                                                                                                                                                  [  0%]
tests/functional/test_copy.py .................................                                                                                                      [  5%]
tests/functional/test_crt.py sssssss                                                                                                                                 [  6%]
tests/functional/test_delete.py .......                                                                                                                              [  7%]
tests/functional/test_download.py .........................................                                                                                          [ 14%]
tests/functional/test_manager.py ........                                                                                                                            [ 15%]
tests/functional/test_processpool.py ............                                                                                                                    [ 17%]
tests/functional/test_upload.py ............................                                                                                                         [ 22%]
tests/functional/test_utils.py ...                                                                                                                                   [ 22%]
tests/unit/test_bandwidth.py ....................................                                                                                                    [ 28%]
tests/unit/test_compat.py ........                                                                                                                                   [ 29%]
tests/unit/test_copies.py ......                                                                                                                                     [ 30%]
tests/unit/test_crt.py ssssssss                                                                                                                                      [ 32%]
tests/unit/test_delete.py ..                                                                                                                                         [ 32%]
tests/unit/test_download.py ..................................................................                                                                       [ 43%]
tests/unit/test_futures.py .........................................................                                                                                 [ 52%]
tests/unit/test_manager.py .........                                                                                                                                 [ 53%]
tests/unit/test_processpool.py ...................................................                                                                                   [ 61%]
tests/unit/test_s3transfer.py .........................................................                                                                              [ 71%]
tests/unit/test_subscribers.py ........                                                                                                                              [ 72%]
tests/unit/test_tasks.py ...............................                                                                                                             [ 77%]
tests/unit/test_upload.py ...............................................                                                                                            [ 85%]
tests/unit/test_utils.py ............................................................................................                                                [100%]

============================================================================= warnings summary =============================================================================
tests/unit/test_utils.py:303
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0/tests/unit/test_utils.py:303: DeprecationWarning: invalid escape sequence \.
    '%s\.[0-9A-Fa-f]{8}$' % filename,

tests/functional/test_copy.py::TestNonMultipartCopy::test_invalid_extra_args
tests/functional/test_copy.py::TestMultipartCopy::test_invalid_extra_args
tests/functional/test_delete.py::TestDeleteObject::test_invalid_extra_args
tests/functional/test_download.py::TestNonRangedDownload::test_invalid_extra_args
tests/functional/test_download.py::TestRangedDownload::test_invalid_extra_args
tests/functional/test_upload.py::TestNonMultipartUpload::test_invalid_extra_args
tests/functional/test_upload.py::TestMultipartUpload::test_invalid_extra_args
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0/tests/__init__.py:454: PendingDeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(ValueError, 'Invalid extra_args'):

tests/functional/test_copy.py::TestNonMultipartCopy::test_raise_exception_on_s3_object_lambda_resource
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0/tests/functional/test_copy.py:284: PendingDeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(ValueError, 'methods do not support'):

tests/functional/test_copy.py::TestNonMultipartCopy::test_raise_exception_on_s3_object_lambda_resource_as_source
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0/tests/functional/test_copy.py:290: PendingDeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(ValueError, 'methods do not support'):

tests/functional/test_copy.py::TestMultipartCopy::test_abort_on_failure
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0/tests/functional/test_copy.py:543: PendingDeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(ClientError, 'ArbitraryFailure'):

tests/functional/test_delete.py::TestDeleteObject::test_raise_exception_on_s3_object_lambda_resource
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0/tests/functional/test_delete.py:72: PendingDeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(ValueError, 'methods do not support'):

tests/functional/test_download.py::TestNonRangedDownload::test_raise_exception_on_s3_object_lambda_resource
tests/functional/test_download.py::TestRangedDownload::test_raise_exception_on_s3_object_lambda_resource
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0/tests/functional/test_download.py:346: PendingDeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(ValueError, 'methods do not support'):

tests/functional/test_manager.py::TestTransferManager::test_cntrl_c_in_context_manager_cancels_incomplete_transfers
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0/tests/functional/test_manager.py:125: PendingDeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(

tests/functional/test_manager.py::TestTransferManager::test_error_in_context_manager_cancels_incomplete_transfers
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0/tests/functional/test_manager.py:88: PendingDeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(FatalError, ref_exception_msg):

tests/functional/test_upload.py::TestNonMultipartUpload::test_raise_exception_on_s3_object_lambda_resource
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0/tests/functional/test_upload.py:274: PendingDeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(ValueError, 'methods do not support'):

tests/unit/test_futures.py::TestTransferCoordinator::test_cancel_with_message
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0/tests/unit/test_futures.py:263: PendingDeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(CancelledError, message):

tests/unit/test_futures.py::TestTransferCoordinator::test_cancel_with_provided_exception
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0/tests/unit/test_futures.py:270: PendingDeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(FatalError, message):

tests/unit/test_futures.py::TestNonThreadedExecutorFuture::test_exception_result
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0/tests/unit/test_futures.py:636: PendingDeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(ValueError, 'message'):

tests/unit/test_manager.py::TestTransferCoordinatorController::test_cancel_with_message
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0/tests/unit/test_manager.py:87: PendingDeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(CancelledError, message):

tests/unit/test_manager.py::TestTransferCoordinatorController::test_cancel_with_provided_exception
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0/tests/unit/test_manager.py:97: PendingDeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(FatalError, message):

tests/unit/test_s3transfer.py::TestMultipartDownloader::test_download_futures_fail_triggers_shutdown
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0/tests/unit/test_s3transfer.py:483: PendingDeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(Exception, "fake download parts error"):

tests/unit/test_s3transfer.py::TestMultipartDownloader::test_io_thread_failure_triggers_shutdown
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0/tests/unit/test_s3transfer.py:458: PendingDeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(Exception, "fake IO error"):

tests/unit/test_subscribers.py::TestSubscribers::test_no_kwargs_in_subclass_subscriber_method
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0/tests/unit/test_subscribers.py:86: PendingDeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(

tests/unit/test_subscribers.py::TestSubscribers::test_not_callable_in_subclass_subscriber_method
  /home/tkloczko/rpmbuild/BUILD/s3transfer-0.5.0/tests/unit/test_subscribers.py:81: PendingDeprecationWarning: Please use assertRaisesRegex instead.
    with self.assertRaisesRegexp(

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================= short test summary info ==========================================================================
SKIPPED [1] tests/functional/test_crt.py:170: Test requires awscrt to be installed.
SKIPPED [1] tests/functional/test_crt.py:200: Test requires awscrt to be installed.
SKIPPED [1] tests/functional/test_crt.py:235: Test requires awscrt to be installed.
SKIPPED [1] tests/functional/test_crt.py:152: Test requires awscrt to be installed.
SKIPPED [1] tests/functional/test_crt.py:128: Test requires awscrt to be installed.
SKIPPED [1] tests/functional/test_crt.py:216: Test requires awscrt to be installed.
SKIPPED [1] tests/functional/test_crt.py:110: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:81: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:66: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:51: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:115: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:121: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:145: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:155: Test requires awscrt to be installed.
SKIPPED [1] tests/unit/test_crt.py:150: Test requires awscrt to be installed.
============================================================== 603 passed, 15 skipped, 26 warnings in 44.97s ===============================================================

Allow to capture head object (metadata) in download file

Want to download an object along with its associated metadata, and I imagine that's a common use-case. The download_file method already issues a head request in order to get content length and decide whether to use multipart or not. Could you provide some kind of public API to capture this head_object so we don't need to make an extra request (slow, wasteful, and adds more race conditions)? I'm not sure the best interface, maybe just use the return value of download_file (currently just returns None) or add container/callback as optional argument.

TestNonMultipartUpload is probably flaky on x86

Hi,
It has occured two times in last few months that s3transfer fails a particular test, TestNonMultipartUpload, on 32-bit x86 machine. The machine has 3 GB RAM, 1 GB was free at the time of the test. I guess it could be a legitimate resource shortage. Also I guess you don't have much interest in 32-bit x86 and in testing on it in particular.

May I ask you the developers to review this test, confirming that the test has reasons to be such a memory hog. Once confirmed, we can just exclude this one from being run on 32 bits. Maybe you could even add such an exclusion into your test suite.

Thanks in advance.


======================================================================
ERROR: test_uses_provided_osutil (tests.functional.test_upload.TestNonMultipartUpload)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/tmp/portage/dev-python/s3transfer-0.1.13/work/s3transfer-0.1.13/tests/functional/test_upload.py", line 231,
 in test_uses_provided_osutil
    future = self.manager.upload(self.filename, self.bucket, self.key)
  File "/var/tmp/portage/dev-python/s3transfer-0.1.13/work/s3transfer-0.1.13-python2_7/lib/s3transfer/manager.py", line
308, in upload
    call_args, UploadSubmissionTask, extra_main_kwargs)
  File "/var/tmp/portage/dev-python/s3transfer-0.1.13/work/s3transfer-0.1.13-python2_7/lib/s3transfer/manager.py", line
464, in _submit_transfer
    main_kwargs=main_kwargs
  File "/var/tmp/portage/dev-python/s3transfer-0.1.13/work/s3transfer-0.1.13-python2_7/lib/s3transfer/futures.py", line
435, in submit
    future = ExecutorFuture(self._executor.submit(task))
  File "/usr/lib/python2.7/site-packages/concurrent/futures/thread.py", line 122, in submit
    self._adjust_thread_count()
  File "/usr/lib/python2.7/site-packages/concurrent/futures/thread.py", line 138, in _adjust_thread_count
    t.start()
  File "/usr/lib/python2.7/threading.py", line 736, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread

See https://bugs.gentoo.org/660474 , the original report.
Then, it happened to pass on a couple of other occasions, but then it reoccured, see here: https://paste.pound-python.org/show/cJwK8BMbkAHbFfLInDeq/ .

Missing dependency when pip installing

We use s3transfer as a dependency of the awscli package. Recently aws cli stopped working and upon investigation we found that awscli released version 1.20.0 which used s3transfer 0.5.0. Previously it was using the 0.4.x version and s3transfer would cause futures to be installed. With the 0.5.0 version we are no longer seeing the futures package installed, and then upon running awscli we see:

  File "/usr/local/bin/aws", line 27, in <module>
    sys.exit(main())
  File "/usr/local/bin/aws", line 23, in main
    return awscli.clidriver.main()
  File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 69, in main
    driver = create_clidriver()
  File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 79, in create_clidriver
    event_hooks=session.get_component('event_emitter'))
  File "/usr/local/lib/python2.7/dist-packages/awscli/plugin.py", line 44, in load_plugins
    modules = _import_plugins(plugin_mapping)
  File "/usr/local/lib/python2.7/dist-packages/awscli/plugin.py", line 61, in _import_plugins
    module = __import__(path, fromlist=[module])
  File "/usr/local/lib/python2.7/dist-packages/awscli/handlers.py", line 27, in <module>
    from awscli.customizations.cloudformation import initialize as cloudformation_init
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/cloudformation/__init__.py", line 13, in <module>
    from awscli.customizations.cloudformation.package import PackageCommand
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/cloudformation/package.py", line 26, in <module>
    from awscli.customizations.s3uploader import S3Uploader
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3uploader.py", line 22, in <module>
    from s3transfer.manager import TransferManager
  File "/usr/local/lib/python2.7/dist-packages/s3transfer/__init__.py", line 134, in <module>
    import concurrent.futures
ImportError: No module named concurrent.futures

We can work around this by locking to the older awscli or installing futures ourselves, but it seems the s3transfer package really should be identifying it requires futures or you'll have a lot of people hitting this issue. Especially when this package is being installed in build pipelines it can cause major headaches.

Tornado Support?

I started looking at this package when I was trying to determine how easy/possible it would be to get boto to use Tornado's IOLoop/AsyncHTTPClient for pushing data to S3. I see there are many similar abstractions (Futures, queues, task management, etc), but it wasn't clear to me what would need to be adapted to work in the context of Tornado. It seems like it should be possible with an Executor subclass that has shims to convert between Tornado's coroutines and this library. If you're interested in supporting Tornado, I would appreciate some guidance. Thanks!

Typos discovered by codespell

$ codespell

./s3transfer/CHANGELOG.rst:75: enhancment ==> enhancement
./s3transfer/scripts/performance/benchmark-upload:132: performace ==> performance
./s3transfer/scripts/performance/benchmark-download:155: performace ==> performance
./s3transfer/tests/unit/test_tasks.py:162: propogates ==> propagates
./s3transfer/tests/unit/test_tasks.py:219: anounce ==> announce
./s3transfer/tests/unit/test_tasks.py:250: anounce ==> an ounce
./s3transfer/tests/unit/test_tasks.py:271: onece ==> once
./s3transfer/tests/unit/test_subscribers.py:65: overriden ==> over ridden
./s3transfer/tests/unit/test_download.py:81: sigal ==> signal, sigil
./s3transfer/tests/unit/test_download.py:687: intially ==> initially
./s3transfer/tests/unit/test_compat.py:31: execption ==> exception
./s3transfer/tests/unit/test_compat.py:57: becase ==> because
./s3transfer/tests/unit/test_s3transfer.py:457: propogates ==> propagates
./s3transfer/tests/unit/test_s3transfer.py:567: temorary ==> temporary
./s3transfer/tests/unit/test_s3transfer.py:637: propogate ==> propagate
./s3transfer/tests/unit/test_futures.py:214: rasied ==> raised
./s3transfer/tests/unit/test_futures.py:277: succes ==> success
./s3transfer/tests/unit/test_manager.py:108: seperate ==> separately
./s3transfer/tests/unit/test_manager.py:119: seperate ==> separate
./s3transfer/tests/functional/test_download.py:84: incase ==> in case
./s3transfer/tests/functional/test_download.py:282: becasue ==> because
./s3transfer/tests/functional/test_download.py:323: seperate ==> separate
./s3transfer/tests/functional/test_upload.py:445: propogate ==> propagate
./s3transfer/tests/functional/test_upload.py:497: propogated ==> propagates
./s3transfer/tests/functional/test_crt.py:102: resopnse ==> response
./s3transfer/tests/integration/test_download.py:93: preceeding ==> preceding, proceeding
./s3transfer/tests/integration/test_download.py:150: preceeding ==> preceding, proceeding
./s3transfer/tests/integration/test_upload.py:81: preceeding ==> preceding, proceeding
./s3transfer/tests/integration/test_upload.py:138: preceeding ==> preceding, proceeding
./s3transfer/.github/workflows/fail-master-prs.yml:13: aginst ==> against
./s3transfer/s3transfer/utils.py:595: aquired ==> acquired
./s3transfer/s3transfer/upload.py:170: configuartion ==> configuration
./s3transfer/s3transfer/processpool.py:479: proccess ==> process
./s3transfer/s3transfer/bandwidth.py:159: amonut ==> amount
./s3transfer/s3transfer/download.py:495: arguements ==> arguments
./s3transfer/s3transfer/download.py:656: existance ==> existence
./s3transfer/s3transfer/manager.py:62: mangager ==> manager
./s3transfer/s3transfer/manager.py:74: transfering ==> transferring
./s3transfer/s3transfer/manager.py:100: streamming ==> streaming
./s3transfer/s3transfer/manager.py:102: recieving ==> receiving
./s3transfer/s3transfer/manager.py:126: dowload ==> download
./s3transfer/s3transfer/manager.py:645: cancelation ==> cancellation
./s3transfer/s3transfer/manager.py:674: propogate ==> propagate
./s3transfer/s3transfer/futures.py:105: propogate ==> propagate
./s3transfer/s3transfer/futures.py:396: excpetion ==> exception
./s3transfer/s3transfer/futures.py:406: implentation ==> implementation
./s3transfer/s3transfer/futures.py:452: assocaited ==> associated
./s3transfer/s3transfer/futures.py:497: parm ==> param, pram, parma
./s3transfer/s3transfer/tasks.py:120: depenent ==> dependent
./s3transfer/s3transfer/tasks.py:178: keword ==> keyword
./s3transfer/s3transfer/tasks.py:189: occassionally ==> occasionally
./s3transfer/s3transfer/tasks.py:201: deffered ==> differed, deferred
./s3transfer/s3transfer/tasks.py:263: implmentations ==> implementations
./s3transfer/s3transfer/tasks.py:272: anounce ==> announce
./s3transfer/s3transfer/tasks.py:281: submition ==> submission
./s3transfer/s3transfer/tasks.py:327: intialization ==> initialization

Officially support Python 3.9, 3.10 on PyPI

PyPi only lists support of s3transfer to Python <= 3.8. I would like to see s3transfer officially supporting newer versions - Python 3.9, 3.10. And also, development status on PyPI is 3 - Alpha - can you increase it to 5 - Production/Stable maybe?

From my testing, s3transfer==0.5.0 works with Python 3.9, 3.10.

ImportError: cannot import name 'ReadTimeoutError'

Pins:

botocore>=1.12.89,<1.13.0
s3transfer>=0.2.0,<0.3.

Just started happening today:

from boto3.s3.transfer import create_transfer_manager
File "/var/runtime/boto3/s3/transfer.py", line 129, in <module>
from s3transfer.manager import TransferConfig as S3TransferConfig
File "/var/task/s3transfer/manager.py", line 21, in <module>
from s3transfer.utils import get_callbacks
File "/var/task/s3transfer/utils.py", line 27, in <module>
from botocore.exceptions import ReadTimeoutError
ImportError: cannot import name 'ReadTimeoutError'

Timeline for GA?

Is this code recommended for use in production systems? Is there a timeline for integrating this with the main boto libs?

s3_client.copy() drops metadata for small files

Describe the bug

s3_client.copy() stores metadata for files large enough to trigger multipart handling, but smaller files trigger a different code path, and the metadata is lost.

Expected Behavior

metadata should be copied regardless of the file size

Current Behavior

metadata is missing

Reproduction Steps

    import boto3                                                                                                                                                                         
    import time                                                                                                                                                                          
    s3Res = boto3.resource("s3")                                                                                                                                                         
    s3Client = boto3.client("s3")                                                                                                                                                        
                                                                                                                                                                                         
    srcBucket = 'yellow7-nf-runs'                                                                                                                                                        
    srcKey = 'results/theoriginal'                                                                                                                                                       
                                                                                                                                                                                         
    destBucket = srcBucket                                                                                                                                                               
    destKey = 'results/thecopy'                                                                                                                                                          
                                                                                                                                                                                         
    metaKey = 'somekeyname'                                                                                                                                                              
    metaVal = str(time.time())                                                                                                                                                           
                                                                                                                                                                                         
    # config.multipart_threshold from 
    # https://github.com/boto/s3transfer/blob/6bae40b6306b2071c06c854608a14ba60e478d74/s3transfer/__init__.py#L117
    multipart_threshold_size = 8388608                                                                                                                                                   
                                                                                                                                                                                         
    for size in [                                                                                                                                                                        
            multipart_threshold_size + 1, # slightly bigger, works                                                                                                                       
            multipart_threshold_size,     # slightly bigger, works                                                                                                                       
            multipart_threshold_size - 1, # slightly smaller, fails                                                                                                                      
            multipart_threshold_size - 2, # slightly smaller, fails                                                                                                                      
    ]:                                                                                                                                                                                   
        print(f"populating {size} bytes into s3://{srcBucket}/{srcKey}")                                                                                                                 
        s3Res.Object(srcBucket, srcKey).put(Body=FakeFile(size))                                                                                                                         
                                                                                                                                                                                         
        s3Client.copy(                                                                                                                                                                   
            CopySource={"Bucket": srcBucket, "Key": srcKey},                                                                                                                             
            Bucket=destBucket,                                                                                                                                                           
            Key=destKey,                                                                                                                                                                 
            ExtraArgs={"Metadata": {metaKey: metaVal}},                                                                                                                                  
        )                                                                                                                                                                                
                                                                                                                                                                                         
        for i in range(5):                                                                                                                                                               
            gotval = s3Res.Object(destBucket, destKey).metadata.get(metaKey)                                                                                                             
            if gotval is None:                                                                                                                                                           
                print("metadata is still empty")                                                                                                                                         
                time.sleep(1)                                                                                                                                                            
            else:                                                                                                                                                                        
                print("metadata was found", gotval)                                                                                                                                      
                break

I'm using this to emulate variable sized files, but you can replace it with a static file.

class FakeFile():                                                                                                                                                                        
    '''A file-like object that generates a fixed byte stream, so that uploads                                                                                                            
    can be verified without having a large file on disk.''' 
    # https://waymoot.org/home/python_string/
    # https://stackoverflow.com/questions/43463180/efficient-way-to-generate-content-for-a-fake-file-like-object-in-python                                                                                                                             
    def __init__(self, length = 10 * 1024 * 1024): # 10 megabytes                                                                                                                        
        self.idx = 0                                                                                                                                                                     
        self.length = length                                                                                                                                                             
    def read(self, size=None):                                                                                                                                                           
        r = b''                                                                                                                                                                          
        max = self.length - self.idx                                                                                                                                                     
        if size == None or size > max:                                                                                                                                                   
            size = max                                                                                                                                                                   
        r = b"".join([b'%x' % (c%16) for c in range(self.idx, self.idx+size)])                                                                                                           
        self.idx += size                                                                                                                                                                 
        return r                                                                                                                                                                         
    #def readLine(self, size=None): # - read one entire line from the file.                                                                                                              
    #    return self.read()                                                                                                                                                              
    #def close(): # - close the file.                                                                                                                                                    
    #    pass                                                                                                                                                                            
    def seek(self, i, dir=None):                                                                                                                                                         
        import os                                                                                                                                                                        
        if dir is None:                                                                                                                                                                  
            dir=os.SEEK_SET                                                                                                                                                              
        if dir==os.SEEK_CUR:                                                                                                                                                             
            self.idx += i                                                                                                                                                                
        elif dir==os.SEEK_SET:                                                                                                                                                           
            self.idx = i                                                                                                                                                                 
        elif dir==os.SEEK_END:                                                                                                                                                           
            self.idx = self.length - i                                                                                                                                                   
        else:                                                                                                                                                                            
            raise ValueError()                                                                                                                                                           
                                                                                                                                                                                         
    def tell(self):                                                                                                                                                                      
        return self.idx

Possible Solution

the size determination is made, at

s3transfer/s3transfer/copies.py

Line 74 in 7a0d980

def _submit(

and results in either calling _submit_copy_request or _submit_multipart_request

        if transfer_future.meta.size < config.multipart_threshold:                                                                                                                                                            
            self._submit_copy_request(                                                                                                                                                                                        
                client, config, osutil, request_executor, transfer_future                                                                                                                                                     
            )                                                                                                                                                                                                                 
        else:                                                                                                                                                                                                                 
            self._submit_multipart_request(                                                                                                                                                                                   
                client, config, osutil, request_executor, transfer_future                                                                                                                                                     
            )

Additional Information/Context

No response

SDK version used

boto3 1.24.37

Environment details (OS name and version, etc.)

Ubuntu 22.04 LTS

Avoid unnecessary head_object requests

I am using boto to download "folders" containing lots of small files from S3. I have noticed that an extra HEAD request is being performed before each GET, which results in slowdowns, throttling and $$$.

I understand that knowing the size beforehand is required to to decide whether to do a multipart download or not, but I consider the current behavior to be wasteful, because:

  • I might disable multipart but the request is still performed
  • I might already know the size of the object (from the list_objects that I am iterating on)

I propose supporting a "size hint" parameter to bypass the head_object call.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.