iterative / pytest-servers Goto Github PK

View Code? Open in Web Editor NEW

13.0 15.0 4.0 182 KB

Create temporary directories on the various filesystems for testing

License: Apache License 2.0

Python 100.00%

dvc python storage testing test

pytest-servers's Introduction

pytest servers

Features

Create temporary directories on the following filesystems:

Local fs
In-memory fs
S3, both using mock S3 remotes (https://github.com/spulec/moto) and real S3 remotes
Azure, both using mock Azure remotes (https://github.com/Azure/Azurite) via docker and using real Azure Storage remotes
Google Cloud Storage, both using mock GCS remote (https://github.com/fsouza/fake-gcs-server) via docker and using real Google Storage Remotes

Installation

You can install pytest servers via pip from PyPI:

$ pip install pytest-servers

To use temporary S3 paths:

$ pip install pytest-servers[s3]

To use temporary Azure paths

$ pip install pytest-servers[azure]

To use temporary Google Cloud Storage paths

$ pip install pytest-servers[gcs]

To install all extras:

$ pip install pytest-servers[all]

Usage

The main fixture provided by pytest-servers provides is tmp_upath_factory, which can be used to generate temporary paths on different (mocked) filesystems:

def test_something_on_s3(tmp_upath_factory):
    path = tmp_upath_factory.mktemp("s3")
    foo = path / "foo"
    foo.write_text("foo")
    ...

mktemp supports the following filesystem types:

local (local filesystem)
memory (in-memory filesystem)
s3 (Amazon S3)
gcs (Google Cloud Storage)
azure (Azure Storage)

Some convenience fixtures that wrap tmp_upath_factory.mktemp and return a paths on these filesystems are also available:

tmp_local_path
tmp_memory_path
tmp_s3_path
tmp_gcs_path
tmp_azure_path

The tmp_upath fixture can be used for parametrizing paths with pytest's indirect parametrization:

@pytest.mark.parametrize("tmp_upath", ["local", "s3", "gcs"], indirect=True)
def test_something(tmp_upath):
    pass

In order to use real remotes instead of mocked ones, use tmp_upath_factory with the following methods

tmp_upath_factory.s3(region_name, client_kwargs) where client_kwargs are passed to the underlying S3FileSystem/boto client
tmp_upath_factory.gcs(endpoint_url)
tmp_upath_factory.azure(connection_string)

Versioning support can be used by using the versioning fixture. This is currently supported for s3 and gcs remotes

# using mktemp
def test_something_on_s3_versioned(tmp_upath_factory):
    path = tmp_upath_factory.mktemp("s3", version_aware=True)
    assert path.fs.version_aware # bucket has versioning enabled

# or, using remote-specific fixtures
def test_something_on_s3_versioned(tmp_s3_path, versioning):
    assert tmp_s3_path.fs.version_aware # bucket has versioning enabled

Contributing

Contributions are very welcome. To learn more, see the Contributor Guide.

License

Distributed under the terms of the Apache 2.0 license, pytest servers is free and open source software.

Issues

If you encounter any problems, please file an issue along with a detailed description.

pytest-servers's People

Contributors

Stargazers

Watchers

Forkers

dtrifiro rlamy efiop gbrennon sablokgaurav

pytest-servers's Issues

add tmp_upath_factory

`tmp_upath_factory.mktemp("local")` breaks `pytest-xdist`

The tmp_upath_factory fixture introduced on #8 relies on the private pytest.TempPathFactory.from_config class method to create a new tmp_path_factory from an existing request fixture.

This is problematic, not only because it's using a private method that shouldn't be used, but also because it breaks pytest-xdist due to a complex corner case.

Error description

Running e.g. os.chdir() with a given directory and then deleting that directory will cause any further calls to e.g. os.getcwd() to raise a FileNotFoundError exception; this will also happen if the directory was deleted & re-created with the same path.

Accessing the TempPathFactory.getbasetemp property causes the basetemp directory to be deleted & re-created¹ if it already existed and TempPathFactory.__init__() was called with the given_basetemp argument.

It turns out that pytest-xdist creates individual basetemp directories for each distributed process, causing given_basetemp to be set as well, and triggering a second deletion & re-creation of the directory.

Note

Debugging this issue from inside Python was really challenging, but pwdx(1) was helpful to determine the current working directory for the running process, and even hint that it was being deleted.

Minimal example

Note

Use pytest test.py --numprocesses=1 to enable pytext-xdist

With `pytest-servers`

import os

def test(tmp_path_factory, tmp_upath_factory):
    os.chdir(tmp_path_factory.mktemp("example"))
    tmp_upath_factory.mktemp("local")
    os.getcwd()

>       os.getcwd()
E       FileNotFoundError: [Errno 2] No such file or directory

Without `pytest-servers` (simplified)

import os
import pytest

def test(request, tmp_path_factory):
    os.chdir(tmp_path_factory.mktemp("example"))
    pytest.TempPathFactory.from_config(request.config).mktemp("pytest-servers")
    os.getcwd()

>       os.getcwd()
E       FileNotFoundError: [Errno 2] No such file or directory

Relevant code

`pytest-servers`

pytest-servers/src/pytest_servers/fixtures.py

Lines 25 to 28 in b429202

    
           @pytest.fixture(scope="session") 
        
           def tmp_upath_factory(request: pytest.FixtureRequest) -> TempUPathFactory: 
        
               """Return a TempUPathFactory instance for the test session.""" 
        
               return TempUPathFactory.from_request(request)

pytest-servers/src/pytest_servers/factory.py

Lines 52 to 53 in b429202

    
           @classmethod 
        
           def from_request(

pytest-servers/src/pytest_servers/factory.py

Lines 61 to 66 in b429202

    
           tmp_upath_factory._local_path_factory = (  # noqa: SLF001 
        
               pytest.TempPathFactory.from_config( 
        
                   request.config, 
        
                   _ispytest=True, 
        
               ) 
        
           )

pytest-servers/src/pytest_servers/factory.py

Lines 162 to 165 in b429202

    
           def local(self) -> LocalPath: 
        
               """Create a local temporary path.""" 
        
               mktemp = ( 
        
                   self._local_path_factory.mktemp

pytest-servers/src/pytest_servers/factory.py

Line 169 in b429202

return LocalPath(mktemp("pytest-servers")) # type: ignore[operator]

`pytest-xdist`

https://github.com/pytest-dev/pytest-xdist/blob/5b83de49c5a1e2a681be2ded651e0e7769b25ea0/src/xdist/workermanage.py#L271-L273

            if hasattr(self.config, "_tmp_path_factory"):
                basetemp = self.config._tmp_path_factory.getbasetemp()
                option_dict["basetemp"] = str(basetemp / name)

https://github.com/pytest-dev/pytest-xdist/blob/5b83de49c5a1e2a681be2ded651e0e7769b25ea0/src/xdist/remote.py#L350

    setup_config(config, option_dict.get("basetemp"))

https://github.com/pytest-dev/pytest-xdist/blob/5b83de49c5a1e2a681be2ded651e0e7769b25ea0/src/xdist/remote.py#L316

def setup_config(config, basetemp):

https://github.com/pytest-dev/pytest-xdist/blob/5b83de49c5a1e2a681be2ded651e0e7769b25ea0/src/xdist/remote.py#L324

    config.option.basetemp = basetemp

`pytest`

https://github.com/pytest-dev/pytest/blob/532782a22807c9dd0b0f8c7b1d114bda67f03593/src/_pytest/tmpdir.py#L79-L80

    @classmethod
    def from_config(

https://github.com/pytest-dev/pytest/blob/532782a22807c9dd0b0f8c7b1d114bda67f03593/src/_pytest/tmpdir.py#L103-L104

        return cls(
            given_basetemp=config.option.basetemp,

https://github.com/pytest-dev/pytest/blob/532782a22807c9dd0b0f8c7b1d114bda67f03593/src/_pytest/tmpdir.py#L56-L58

    def __init__(
        self,
        given_basetemp: Optional[Path],

https://github.com/pytest-dev/pytest/blob/532782a22807c9dd0b0f8c7b1d114bda67f03593/src/_pytest/tmpdir.py#L73

            self._given_basetemp = Path(os.path.abspath(str(given_basetemp)))

https://github.com/pytest-dev/pytest/blob/532782a22807c9dd0b0f8c7b1d114bda67f03593/src/_pytest/tmpdir.py#L150-L154

        if self._given_basetemp is not None:
            basetemp = self._given_basetemp
            if basetemp.exists():
                rm_rf(basetemp)
            basetemp.mkdir(mode=0o700)

Historical note: the logic to delete & recreate the basetemp directory when accessing the getbasetemp property, if the path already existed, was introduced with https://github.com/pytest-dev/pytest/commit/f456e376b99722f8e094ac52698e0fa9252ae686 ↩

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

This repository currently has no open or pending branches.

Detected dependencies

regex

src/pytest_servers/azure.py

mcr.microsoft.com/azure-storage/azurite 3.29.0

src/pytest_servers/gcs.py

fsouza/fake-gcs-server 1.47.8

Check this box to trigger a request for Renovate to run again on this repository

add real_{s3,gs,azure} fixtures

This is useful for cases when you want to use real clouds (e.g. for benchmarks).

s3: use moto's server in a separate thread

See #86

Failure setting up session-scoped fixtures with pytest-xdist

When using pytest-xdist with multiple processes session-scoped fixtures, such as fake_gcs_server and azurite, are being invoked multiple times, failing.

xdist provides a recipe for executing session scoped fixtures only once:
https://pypi.org/project/pytest-xdist/#making-session-scoped-fixtures-execute-only-once

provide `fs` and `tmp_fs_path` fixture

Common usecases for pytest-servers include accessing the underlying filesystem (fs fixture) and accessing the actual path on the filesystem tmp_fs_path

docker tags not pinned

docker tags for azurite and fake-gcs-server are currently not pinned

s3 server very slow on Windows

Using the s3 server results in very long test times. I've tracked it down to aiohttp sometimes spending 2 seconds trying and failing to establish an IPv6 connection to localhost. I don't know how to solve the underlying issue, but we could workaround it by forcing IPv4 resoltion on Windows.

Reproducer:

def test_foo(tmp_upath_factory):
    src_path = tmp_upath_factory.mktemp("s3", version_aware=True)
    (src_path / "description").write_text("Cats and Dogs")
    (src_path / "cats").mkdir()
    (src_path / "cats" / "cat1").write_text("meow")
    (src_path / "cats" / "cat2").write_text("mrow")
    (src_path / "dogs").mkdir()
    (src_path / "dogs" / "dog1").write_text("woof")
    (src_path / "dogs" / "dog2").write_text("arf")
    (src_path / "dogs" / "dog3").write_text("bark")
    (src_path / "dogs" / "others").mkdir()
    (src_path / "dogs" / "others" / "dog4").write_text("ruff")

Running this test on Windows with s3fs debug logging enabled results in:

(win) PS Z:\devel\iterative\examples\windebug> pytest .\test_foo.py -s
Test session starts (platform: win32, Python 3.11.2, pytest 7.2.0, pytest-sugar 0.9.6)
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: Z:\devel\iterative\examples\windebug
plugins: asyncio-0.20.3, shutil-1.7.0, sugar-0.9.6, virtualenv-1.7.0, mock-3.8.2, hypothesis-6.68.1, servers-0.2.1.dev0+gc103d6a.d20230223, benchmark-4.0.0, cov-3.0.0
asyncio: mode=Mode.STRICT
collecting ... 
Starting a new Thread with MotoServer running on localhost:0...
2023-02-23 16:02:39,187 - s3fs - DEBUG - _upload_chunk -- Upload for <File-like object S3FileSystem, pytest-servers-bqqsgx/description>, final=True, loc=13, buffer loc=13
2023-02-23 16:02:39,187 - s3fs - DEBUG - commit -- Commit <File-like object S3FileSystem, pytest-servers-bqqsgx/description>
2023-02-23 16:02:39,187 - s3fs - DEBUG - commit -- One-shot upload of <File-like object S3FileSystem, pytest-servers-bqqsgx/description>
2023-02-23 16:02:39,187 - s3fs - DEBUG - set_session -- Setting up s3fs instance
2023-02-23 16:02:39,204 - s3fs - DEBUG - set_session -- RC: caching enabled? False (explicit option is False)
2023-02-23 16:02:39,369 - s3fs - DEBUG - _call_s3 -- CALL: put_object - ({},) - {'Key': 'description', 'Bucket': 'pytest-servers-bqqsgx', 'ACL': ''}
2023-02-23 16:02:41,444 - s3fs - DEBUG - _call_s3 -- CALL: head_object - ({},) - {'Bucket': 'pytest-servers-bqqsgx', 'Key': 'cats'}
2023-02-23 16:02:41,460 - s3fs - DEBUG - _error_wrapper -- Client error (maybe retryable): An error occurred (404) when calling the HeadObject operation: Not Found
2023-02-23 16:02:41,460 - s3fs - DEBUG - _call_s3 -- CALL: list_objects_v2 - ({},) - {'Bucket': 'pytest-servers-bqqsgx', 'Prefix': 'cats/', 'Delimiter': '/', 'MaxKeys': 1}
2023-02-23 16:02:43,518 - s3fs - DEBUG - _call_s3 -- CALL: list_objects_v2 - () - {'MaxKeys': 1, 'Bucket': 'pytest-servers-bqqsgx'}
2023-02-23 16:02:43,533 - s3fs - DEBUG - _upload_chunk -- Upload for <File-like object S3FileSystem, pytest-servers-bqqsgx/cats/cat1>, final=True, loc=4, buffer loc=4
2023-02-23 16:02:43,533 - s3fs - DEBUG - commit -- Commit <File-like object S3FileSystem, pytest-servers-bqqsgx/cats/cat1>
2023-02-23 16:02:43,533 - s3fs - DEBUG - commit -- One-shot upload of <File-like object S3FileSystem, pytest-servers-bqqsgx/cats/cat1>
2023-02-23 16:02:43,533 - s3fs - DEBUG - _call_s3 -- CALL: put_object - ({},) - {'Key': 'cats/cat1', 'Bucket': 'pytest-servers-bqqsgx', 'ACL': ''}
2023-02-23 16:02:45,590 - s3fs - DEBUG - _upload_chunk -- Upload for <File-like object S3FileSystem, pytest-servers-bqqsgx/cats/cat2>, final=True, loc=4, buffer loc=4
2023-02-23 16:02:45,590 - s3fs - DEBUG - commit -- Commit <File-like object S3FileSystem, pytest-servers-bqqsgx/cats/cat2>
2023-02-23 16:02:45,590 - s3fs - DEBUG - commit -- One-shot upload of <File-like object S3FileSystem, pytest-servers-bqqsgx/cats/cat2>
2023-02-23 16:02:45,590 - s3fs - DEBUG - _call_s3 -- CALL: put_object - ({},) - {'Key': 'cats/cat2', 'Bucket': 'pytest-servers-bqqsgx', 'ACL': ''}
2023-02-23 16:02:45,602 - s3fs - DEBUG - _call_s3 -- CALL: head_object - ({},) - {'Bucket': 'pytest-servers-bqqsgx', 'Key': 'dogs'}
2023-02-23 16:02:47,640 - s3fs - DEBUG - _error_wrapper -- Client error (maybe retryable): An error occurred (404) when calling the HeadObject operation: Not Found
2023-02-23 16:02:47,640 - s3fs - DEBUG - _call_s3 -- CALL: list_objects_v2 - ({},) - {'Bucket': 'pytest-servers-bqqsgx', 'Prefix': 'dogs/', 'Delimiter': '/', 'MaxKeys': 1}
2023-02-23 16:02:47,656 - s3fs - DEBUG - _call_s3 -- CALL: list_objects_v2 - () - {'MaxKeys': 1, 'Bucket': 'pytest-servers-bqqsgx'}
2023-02-23 16:02:49,705 - s3fs - DEBUG - _upload_chunk -- Upload for <File-like object S3FileSystem, pytest-servers-bqqsgx/dogs/dog1>, final=True, loc=4, buffer loc=4
2023-02-23 16:02:49,705 - s3fs - DEBUG - commit -- Commit <File-like object S3FileSystem, pytest-servers-bqqsgx/dogs/dog1>
2023-02-23 16:02:49,705 - s3fs - DEBUG - commit -- One-shot upload of <File-like object S3FileSystem, pytest-servers-bqqsgx/dogs/dog1>
2023-02-23 16:02:49,720 - s3fs - DEBUG - _call_s3 -- CALL: put_object - ({},) - {'Key': 'dogs/dog1', 'Bucket': 'pytest-servers-bqqsgx', 'ACL': ''}
2023-02-23 16:02:51,742 - s3fs - DEBUG - _upload_chunk -- Upload for <File-like object S3FileSystem, pytest-servers-bqqsgx/dogs/dog2>, final=True, loc=3, buffer loc=3
2023-02-23 16:02:51,742 - s3fs - DEBUG - commit -- Commit <File-like object S3FileSystem, pytest-servers-bqqsgx/dogs/dog2>
2023-02-23 16:02:51,742 - s3fs - DEBUG - commit -- One-shot upload of <File-like object S3FileSystem, pytest-servers-bqqsgx/dogs/dog2>
2023-02-23 16:02:51,742 - s3fs - DEBUG - _call_s3 -- CALL: put_object - ({},) - {'Key': 'dogs/dog2', 'Bucket': 'pytest-servers-bqqsgx', 'ACL': ''}
2023-02-23 16:02:51,760 - s3fs - DEBUG - _upload_chunk -- Upload for <File-like object S3FileSystem, pytest-servers-bqqsgx/dogs/dog3>, final=True, loc=4, buffer loc=4
2023-02-23 16:02:51,760 - s3fs - DEBUG - commit -- Commit <File-like object S3FileSystem, pytest-servers-bqqsgx/dogs/dog3>
2023-02-23 16:02:51,760 - s3fs - DEBUG - commit -- One-shot upload of <File-like object S3FileSystem, pytest-servers-bqqsgx/dogs/dog3>
2023-02-23 16:02:51,760 - s3fs - DEBUG - _call_s3 -- CALL: put_object - ({},) - {'Key': 'dogs/dog3', 'Bucket': 'pytest-servers-bqqsgx', 'ACL': ''}
2023-02-23 16:02:53,823 - s3fs - DEBUG - _call_s3 -- CALL: head_object - ({},) - {'Bucket': 'pytest-servers-bqqsgx', 'Key': 'dogs/others'}
2023-02-23 16:02:53,823 - s3fs - DEBUG - _error_wrapper -- Client error (maybe retryable): An error occurred (404) when calling the HeadObject operation: Not Found
2023-02-23 16:02:53,823 - s3fs - DEBUG - _call_s3 -- CALL: list_objects_v2 - ({},) - {'Bucket': 'pytest-servers-bqqsgx', 'Prefix': 'dogs/others/', 'Delimiter': '/', 'MaxKeys': 1}
2023-02-23 16:02:55,880 - s3fs - DEBUG - _call_s3 -- CALL: list_objects_v2 - () - {'MaxKeys': 1, 'Bucket': 'pytest-servers-bqqsgx'}
2023-02-23 16:02:55,880 - s3fs - DEBUG - _upload_chunk -- Upload for <File-like object S3FileSystem, pytest-servers-bqqsgx/dogs/others/dog4>, final=True, loc=4, buffer loc=4
2023-02-23 16:02:55,896 - s3fs - DEBUG - commit -- Commit <File-like object S3FileSystem, pytest-servers-bqqsgx/dogs/others/dog4>
2023-02-23 16:02:55,896 - s3fs - DEBUG - commit -- One-shot upload of <File-like object S3FileSystem, pytest-servers-bqqsgx/dogs/others/dog4>
2023-02-23 16:02:55,896 - s3fs - DEBUG - _call_s3 -- CALL: put_object - ({},) - {'Key': 'dogs/others/dog4', 'Bucket': 'pytest-servers-bqqsgx', 'ACL': ''}

 test_foo.py ✓                                                                                                                                                                                                                                                                                       100% ██████████
================================================================================================================================================ warnings summary =================================================================================================================================================
test_foo.py::test_foo
  Z:\devel\iterative\dql\venv\win\Lib\site-packages\botocore\utils.py:15: DeprecationWarning: 'cgi' is deprecated and slated for removal in Python 3.13
    import cgi

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

Results (24.66s):
       1 passed

make all fixtures to be function scoped

We should provide function scoped by default for correctness, and we can offer fixtures with a different scope if needed.

Ideally, it'd look something like this:

def s3_server_impl():
   yield MockedS3Server()

@pytest.fixture():
def s3_server():
   yield s3_server_impl()

@pytest.fixture(scope="session"):
def session_s3_server():
   yield s3_server_impl()

...

gs: add environment credentials override fixture

The cloud credentials messages in the logs are just debug messages, and are related to GCSFileSystem trying to get credentials from various sources (including environment) before giving up, see
https://github.com/fsspec/gcsfs/blob/fef6c0b098342a95990fd5aad8c3353d257f8d1b/gcsfs/credentials.py#L225-L242.

Note that these credentials are not used to access storage.googleapis.com because endpoint_url is overriden.

We could get rid of those warnings by setting GOOGLE_APPLICATION_CREDENTIALS to the path of a service account private key json, something similar to what we do with S3 with fake_creds_file. Trying to decide whether it's better to have a fixture for that here or in pytest-servers, although I'm opting for the former.

Originally posted by @dtrifiro in iterative/dvc-gs#8 (comment)

s3: subprocess stdout/stderr not closed on teardown

You can try reproducing by using pytest -Werror.

$ pytest -Werror

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ERROR at teardown of test_merge[ancestor_dict5-our_dict5-their_dict5-merged_dict5] ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

cls = <class '_pytest.runner.CallInfo'>, func = <function call_runtest_hook.<locals>.<lambda> at 0x7f3d3d844ee0>, when = 'teardown', reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    @classmethod
    def from_call(
        cls,
        func: "Callable[[], TResult]",
        when: "Literal['collect', 'setup', 'call', 'teardown']",
        reraise: Optional[
            Union[Type[BaseException], Tuple[Type[BaseException], ...]]
        ] = None,
    ) -> "CallInfo[TResult]":
        """Call func, wrapping the result in a CallInfo.
    
        :param func:
            The function to call. Called without arguments.
        :param when:
            The phase in which the function is called.
        :param reraise:
            Exception or exceptions that shall propagate if raised by the
            function, instead of being wrapped in the CallInfo.
        """
        excinfo = None
        start = timing.time()
        precise_start = timing.perf_counter()
        try:
>           result: Optional[TResult] = func()

.nox/tests-3-10/lib/python3.10/site-packages/_pytest/runner.py:338: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.nox/tests-3-10/lib/python3.10/site-packages/_pytest/runner.py:259: in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
.nox/tests-3-10/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
.nox/tests-3-10/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
.nox/tests-3-10/lib/python3.10/site-packages/_pytest/unraisableexception.py:93: in pytest_runtest_teardown
    yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
        with catch_unraisable_exception() as cm:
            yield
            if cm.unraisable:
                if cm.unraisable.err_msg is not None:
                    err_msg = cm.unraisable.err_msg
                else:
                    err_msg = "Exception ignored in"
                msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
                msg += "".join(
                    traceback.format_exception(
                        cm.unraisable.exc_type,
                        cm.unraisable.exc_value,
                        cm.unraisable.exc_traceback,
                    )
                )
>               warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E               pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
E               
E               Traceback (most recent call last):
E                 File "/home/saugat/projects/iterative/dvc-data/.nox/tests-3-10/lib/python3.10/site-packages/pytest_servers/s3.py", line 58, in close
E                   self.proc = None
E               ResourceWarning: unclosed file <_io.BufferedReader name=13>

.nox/tests-3-10/lib/python3.10/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning
           
pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>

3.12 compatibility

Related: #149

Flaky CI on MacOS

CI runs on MacOS are failing intermittently, due to timeouts in setting up fake_gcs_server or azurite. This is annoying in downstream projects testing on MacOS, as it causes spurious failures as well

tmp_upath_factory: could it be more simpler?

We have factories for other filesystems based upath as:

tmp_upath_factory.local_temp_path(...)
tmp_upath_factory.s3_temp_path(..)
...

Could it be more simpler without redundant temp_path prefix? eg:

tmp_upath_factory.local(...)
tmp_upath_factory.s3(...)
...

I guess the arguments against my suggestion could be:

that this will be confusing with tmp_upath_factory.mktemp("s3"), etc.
or, that it breaks symmetry with individual fixtures: local_path, s3_path, etc.
(how about tmp_upath_factory.local_path(...) then? I'd still cut path though).

	@pytest.fixture(scope="session")
	def tmp_upath_factory(request: pytest.FixtureRequest) -> TempUPathFactory:
	"""Return a TempUPathFactory instance for the test session."""
	return TempUPathFactory.from_request(request)

	tmp_upath_factory._local_path_factory = ( # noqa: SLF001
	pytest.TempPathFactory.from_config(
	request.config,
	_ispytest=True,
	)
	)

	def local(self) -> LocalPath:
	"""Create a local temporary path."""
	mktemp = (
	self._local_path_factory.mktemp

iterative / pytest-servers Goto Github PK

pytest-servers's Introduction

pytest servers

Features

Installation

Usage

Contributing

License

Issues

pytest-servers's People

Contributors

Stargazers

Watchers

Forkers

pytest-servers's Issues

Error description

Minimal example

With pytest-servers

Without pytest-servers (simplified)

Relevant code

pytest-servers

pytest-xdist

pytest

Footnotes

Detected dependencies

Recommend Projects

Recommend Topics

Recommend Org

With `pytest-servers`

Without `pytest-servers` (simplified)

`pytest-servers`

`pytest-xdist`

`pytest`