Code Monkey home page Code Monkey logo

ubelt's Introduction

GithubActions ReadTheDocs Pypi Downloads Codecov CircleCI Appveyor

image

Ubelt is a utility library for Python with a stdlib like feel.

Elevator Pitch:

Is the Python standard library good? Yes. Could it's conciseness be improved? Yes. Ubelt aims to provide a quicker way to express things you can do in the standard library. Progress? ub.ProgIter. Hashing? ub.hash_data / ub.hash_file. Caching? ub.Cacher / ub.CacheStamp. Shell commands? ub.cmd. There are similar functions for downloading data, futures-based parallel (or serial) job execution, pretty reprs, path management, iteration, and one of my favorites: set operation enriched dictionaries: ub.udict.

There are 120ish functions and classes to help make your code shorter and easier to express concisely. The library is fast to install and import, all dependencies are optional. As of 2023 it is 6 years old, regularly maintained, and mature. It is well tested and has moderate usage.

To learn more, the function usefulness chart is a good place to start. This shows how often I use particular functions, and while some of the less used ones are candidates for removal, some of them still worth checking out. For a slightly slower start, read the introduction:

Introduction:

Ubelt is a lightweight library of robust, tested, documented, and simple functions that extend the Python standard library. It has a flat API that all behaves similarly on Windows, Mac, and Linux (up to some small unavoidable differences). Almost every function in ubelt was written with a doctest. This provides helpful documentation and example usage as well as helping achieve 100% test coverage (with minor exceptions on Windows).

  • Goal: provide simple functions that accomplish common tasks not yet addressed by the python standard library.
  • Constraints: Must be low-impact pure python; it should be easy to install and use.
  • Method: All functions are written with docstrings and doctests to ensure that a baseline level of documentation and testing always exists (even if functions are copy/pasted into other libraries)
  • Motto: Good utilities lift all codes.

Read the docs here: http://ubelt.readthedocs.io/en/latest/auto/

These are some of the tasks that ubelt's API enables:

  • extended pathlib with expand, ensuredir, endswith, augment, delete (ub.Path)
  • get paths to cross platform data/cache/config directories (ub.Path.appdir, ...)
  • perform set operations on dictionaries (SetDict)
  • a dictionary with extended helper methods like subdict, take, peek_value, invert, sorted_keys, sorted_vals (UDict)
  • hash common data structures like list, dict, int, str, etc. (hash_data)
  • hash files (hash_file)
  • cache a block of code (Cacher, CacheStamp)
  • time a block of code (Timer)
  • show loop progress with less overhead than tqdm (ProgIter)
  • download a file with optional caching and hash verification (download, grabdata)
  • run shell commands (cmd)
  • find a file or directory in candidate locations (find_path, find_exe)
  • string-repr for nested data structures (urepr)
  • color text with ANSI tags (color_text)
  • horizontally concatenate multiline strings (hzcat)
  • create cross platform symlinks (symlink)
  • import a module using the path to that module (import_module_from_path)
  • check if a particular flag or value is on the command line (argflag, argval)
  • memoize functions (memoize, memoize_method, memoize_property)
  • build ordered sets (oset)
  • argmax/min/sort on lists and dictionaries (argmin, argsort,)
  • get a histogram of items or find duplicates in a list (dict_hist, find_duplicates)
  • group a sequence of items by some criterion (group_items)

Ubelt is small. Its top-level API is defined using roughly 40 lines:

from ubelt.util_arg import (argflag, argval,)
from ubelt.util_cache import (CacheStamp, Cacher,)
from ubelt.util_colors import (NO_COLOR, color_text, highlight_code,)
from ubelt.util_const import (NoParam,)
from ubelt.util_cmd import (cmd,)
from ubelt.util_dict import (AutoDict, AutoOrderedDict, SetDict, UDict, ddict,
                             dict_diff, dict_hist, dict_isect, dict_subset,
                             dict_union, dzip, find_duplicates, group_items,
                             invert_dict, map_keys, map_vals, map_values,
                             named_product, odict, sdict, sorted_keys,
                             sorted_vals, sorted_values, udict, varied_values,)
from ubelt.util_deprecate import (schedule_deprecation,)
from ubelt.util_download import (download, grabdata,)
from ubelt.util_download_manager import (DownloadManager,)
from ubelt.util_func import (compatible, identity, inject_method,)
from ubelt.util_repr import (ReprExtensions, urepr,)
from ubelt.util_futures import (Executor, JobPool,)
from ubelt.util_io import (delete, touch,)
from ubelt.util_links import (symlink,)
from ubelt.util_list import (allsame, argmax, argmin, argsort, argunique,
                             boolmask, chunks, compress, flatten, iter_window,
                             iterable, peek, take, unique, unique_flags,)
from ubelt.util_hash import (hash_data, hash_file,)
from ubelt.util_import import (import_module_from_name,
                               import_module_from_path, modname_to_modpath,
                               modpath_to_modname, split_modpath,)
from ubelt.util_indexable import (IndexableWalker, indexable_allclose,)
from ubelt.util_memoize import (memoize, memoize_method, memoize_property,)
from ubelt.util_mixins import (NiceRepr,)
from ubelt.util_path import (ChDir, Path, TempDir, augpath, ensuredir,
                             expandpath, shrinkuser, userhome,)
from ubelt.util_platform import (DARWIN, LINUX, POSIX, WIN32, find_exe,
                                 find_path, platform_cache_dir,
                                 platform_config_dir, platform_data_dir,)
from ubelt.util_str import (codeblock, hzcat, indent, paragraph,)
from ubelt.util_stream import (CaptureStdout, CaptureStream, TeeStringIO,)
from ubelt.util_time import (Timer, timeparse, timestamp,)
from ubelt.util_zip import (split_archive, zopen,)
from ubelt.orderedset import (OrderedSet, oset,)
from ubelt.progiter import (ProgIter,)

Installation:

Ubelt is distributed on pypi as a universal wheel and can be pip installed on Python 3.6+. Installations are tested on CPython and PyPy implementations. For Python 2.7 and 3.5, the last supported version was 0.11.1.

pip install ubelt

Note that our distributions on pypi are signed with GPG. The signing public key is D297D757; this should agree with the value in dev/public_gpg_key.

Function Usefulness

When I had to hand pick a set of functions that I thought were the most useful I chose these and provided some comment on why:

But a better way might to objectively measure the frequency of usage and built a histogram of usefulness. I generated this histogram using python dev/maintain/gen_api_for_docs.py, which roughly counts the number of times I've used a ubelt function in another project. Note: this measure is biased towards older functions.

Function name Usefulness
ubelt.urepr

4327

ubelt.Path

2125

ubelt.paragraph

1349

ubelt.ProgIter

747

ubelt.cmd

657

ubelt.codeblock

611

ubelt.udict

603

ubelt.expandpath

508

ubelt.take

462

ubelt.oset

342

ubelt.ddict

341

ubelt.iterable

313

ubelt.flatten

303

ubelt.group_items

287

ubelt.NiceRepr

270

ubelt.ensuredir

267

ubelt.map_vals

265

ubelt.peek

262

ubelt.NoParam

248

ubelt.dzip

239

ubelt.odict

236

ubelt.hash_data

200

ubelt.argflag

184

ubelt.grabdata

161

ubelt.dict_hist

156

ubelt.identity

156

ubelt.dict_isect

152

ubelt.Timer

145

ubelt.memoize

142

ubelt.argval

134

ubelt.allsame

133

ubelt.color_text

129

ubelt.schedule_deprecation

123

ubelt.augpath

120

ubelt.dict_diff

117

ubelt.IndexableWalker

116

ubelt.compress

116

ubelt.JobPool

107

ubelt.named_product

104

ubelt.hzcat

90

ubelt.delete

88

ubelt.unique

84

ubelt.WIN32

78

ubelt.dict_union

76

ubelt.symlink

76

ubelt.indent

69

ubelt.ensure_app_cache_dir

67

ubelt.iter_window

62

ubelt.invert_dict

58

ubelt.memoize_property

57

ubelt.import_module_from_name

56

ubelt.argsort

55

ubelt.timestamp

54

ubelt.modname_to_modpath

53

ubelt.find_duplicates

53

ubelt.hash_file

51

ubelt.find_exe

50

ubelt.map_keys

50

ubelt.dict_subset

50

ubelt.Cacher

49

ubelt.chunks

47

ubelt.sorted_vals

40

ubelt.CacheStamp

38

ubelt.highlight_code

37

ubelt.argmax

36

ubelt.writeto

36

ubelt.ensure_unicode

32

ubelt.sorted_keys

30

ubelt.memoize_method

29

ubelt.compatible

24

ubelt.import_module_from_path

24

ubelt.Executor

23

ubelt.readfrom

23

ubelt.modpath_to_modname

17

ubelt.AutoDict

17

ubelt.touch

17

ubelt.inject_method

14

ubelt.timeparse

13

ubelt.ChDir

11

ubelt.shrinkuser

11

ubelt.argmin

10

ubelt.varied_values

9

ubelt.split_modpath

8

ubelt.LINUX

8

ubelt.download

7

ubelt.NO_COLOR

7

ubelt.OrderedSet

6

ubelt.zopen

6

ubelt.CaptureStdout

6

ubelt.DARWIN

5

ubelt.boolmask

4

ubelt.find_path

4

ubelt.get_app_cache_dir

4

ubelt.indexable_allclose

3

ubelt.UDict

3

ubelt.SetDict

2

ubelt.AutoOrderedDict

2

ubelt.argunique

2

ubelt.map_values

1

ubelt.unique_flags

1

ubelt.userhome

0

ubelt.split_archive

0

ubelt.sorted_values

0

ubelt.sdict

0

ubelt.platform_data_dir

0

ubelt.platform_config_dir

0

ubelt.platform_cache_dir

0

ubelt.get_app_data_dir

0

ubelt.get_app_config_dir

0

ubelt.ensure_app_data_dir

0

ubelt.ensure_app_config_dir

0

ubelt.TempDir

0

ubelt.TeeStringIO

0

ubelt.ReprExtensions

0

ubelt.POSIX

0

ubelt.DownloadManager

0

ubelt.CaptureStream

0

Examples

The most up to date examples are the doctests. We also have a Jupyter notebook: https://github.com/Erotemic/ubelt/blob/main/docs/notebooks/Ubelt%20Demo.ipynb

Here are some examples of some features inside ubelt

Paths

Ubelt extends pathlib.Path by adding several new (often chainable) methods. Namely, augment, delete, expand, ensuredir, shrinkuser. It also modifies behavior of touch to be chainable. (New in 1.0.0)

Hashing

The ub.hash_data constructs a hash for common Python nested data structures. Extensions to allow it to hash custom types can be registered. By default it handles lists, dicts, sets, slices, uuids, and numpy arrays.

Support for torch tensors and pandas data frames are also included, but needs to be explicitly enabled. There also exists an non-public plugin architecture to extend this function to arbitrary types. While not officially supported, it is usable and will become better integrated in the future. See ubelt/util_hash.py for details.

Caching

Cache intermediate results from blocks of code inside a script with minimal boilerplate or modification to the original code.

For direct caching of data, use the Cacher class. By default results will be written to the ubelt's appdir cache, but the exact location can be specified via dpath or the appname arguments. Additionally, process dependencies can be specified via the depends argument, which allows for implicit cache invalidation. As far as I can tell, this is the most concise way (4 lines of boilerplate) to cache a block of code with existing Python syntax (as of 2022-06-03).

For indirect caching, use the CacheStamp class. This simply writes a "stamp" file that marks that a process has completed. Additionally you can specify criteria for when the stamp should expire. If you let CacheStamp know about the expected "product", it will expire the stamp if that file has changed, which can be useful in situations where caches might becomes corrupt or need invalidation.

See https://ubelt.readthedocs.io/en/latest/auto/ubelt.util_cache.html for more details about Cacher and CacheStamp.

Loop Progress

ProgIter is a no-threads attached Progress meter that writes to stdout. It is a mostly drop-in alternative to tqdm. The advantage of ``ProgIter`` is that it does not use any python threading, and therefore can be safer with code that makes heavy use of multiprocessing.

Note: ProgIter is also defined in a standalone module: pip install progiter)

Command Line Interaction

The builtin Python subprocess.Popen module is great, but it can be a bit clunky at times. The os.system command is easy to use, but it doesn't have much flexibility. The ub.cmd function aims to fix this. It is as simple to run as os.system, but it returns a dictionary containing the return code, standard out, standard error, and the Popen object used under the hood.

This utility is designed to provide as consistent as possible behavior across different platforms. We aim to support Windows, Linux, and OSX.

Also note the use of ub.urepr (previously ub.repr2) to nicely format the output dictionary.

Additionally, if you specify verbose=True, ub.cmd will simultaneously capture the standard output and display it in real time (i.e. it will "tee" the output).

A common use case for ub.cmd is parsing version numbers of programs

This allows you to easily run a command line executable as part of a python process, see what it is doing, and then do something based on its output, just as you would if you were interacting with the command line itself.

The idea is that ub.cmd removes the need to think about if you need to pass a list of args, or a string. Both will work.

New in 1.0.0, a third variant with different consequences for executing shell commands. Using the system=True kwarg will directly use os.system instead of Popen entirely. In this mode it is not possible to tee the output because the program is executing directly in the foreground. This is useful for doing things like spawning a vim session and returning if the user manages to quit vim.

Downloading Files

The function ub.download provides a simple interface to download a URL and save its data to a file.

The function ub.grabdata works similarly to ub.download, but whereas ub.download will always re-download the file, ub.grabdata will check if the file exists and only re-download it if it needs to.

New in version 0.4.0: both functions now accepts the hash_prefix keyword argument, which if specified will check that the hash of the file matches the provided value. The hasher keyword argument can be used to change which hashing algorithm is used (it defaults to "sha512").

Dictionary Set Operations

Dictionary operations that are analogous to set operations. See each funtions documentation for more details on the behavior of the values. Typically the last seen value is given priority.

I hope Python decides to add these to the stdlib someday.

  • ubelt.dict_union corresponds to set.union.
  • ubelt.dict_isect corresponds to set.intersection.
  • ubelt.dict_diff corresponds to set.difference.

New in Version 1.2.0: Ubelt now contains a dictionary subclass with set operations that can be invoked as ubelt.SetDict or ub.sdict. Note that n-ary operations are supported.

Note this functionality and more is available in ubelt.UDict or ub.udict.

Grouping Items

Given a list of items and corresponding ids, create a dictionary mapping each id to a list of its corresponding items. In other words, group a sequence of items of type VT and corresponding keys of type KT given by a function or corresponding list, group them into a Dict[KT, List[VT] such that each key maps to a list of the values associated with the key. This is similar to pandas.DataFrame.groupby.

Group ids can be specified by a second list containing the id for each corresponding item.

They can also be given by a function that is executed on each item in the list

Dictionary Histogram

Find the frequency of items in a sequence. Given a list or sequence of items, this returns a dictionary mapping each unique value in the sequence to the number of times it appeared. This is similar to pandas.DataFrame.value_counts.

Each item can also be given a weight

Dictionary Manipulation

Map functions across dictionarys to transform the keys or values in a dictionary. The ubelt.map_keys function applies a function to each key in a dictionary and returns this transformed copy of the dictionary. Key conflict behavior currently raises and error, but may be configurable in the future. The ubelt.map_vals function is the same except the function is applied to each value instead. I these functions are useful enough to be ported to Python itself.

Take a subset of a dictionary. Note this is similar to ub.dict_isect, except this will raise an error if the given keys are not in the dictionary.

The ubelt.take function works on dictionarys (and lists). It is similar to ubelt.dict_subset, except that it returns just a list of the values, and discards information about the keys. It is also possible to specify a default value.

Invert the mapping defined by a dictionary. By default invert_dict assumes that all dictionary values are distinct (i.e. the mapping is one-to-one / injective).

However, by specifying unique_vals=False the inverted dictionary builds a set of keys that were associated with each value.

New in Version 1.2.0: Ubelt now contains a dictionary subclass ubelt.UDict with these quality of life operations (and also inherits from ubelt.SetDict). The alias ubelt.udict can be used for quicker access.

Next time you have a default configuration dictionary like and you allow the developer to pass keyword arguments to modify these behaviors, consider using dictionary intersection (&) to separate out only the relevant parts and dictionary union (|) to update those relevant parts. You can also use dictionary differences (-) if you need to check for unused arguments.

Produces:

-- Run with some specified --
algo1_specified = {
    'opt2': 'fox',
}
algo2_specified = {
    'src': 'box',
}
algo1_config={'opt1': 10, 'opt2': 'fox'}
algo2_config={'src': 'box', 'dst': './there'}
The following kwargs were unused {}


-- Run with extra unspecified --
algo1_specified = {}
algo2_specified = {}
algo1_config={'opt1': 10, 'opt2': 11}
algo2_config={'src': './here/', 'dst': './there'}
The following kwargs were unused {'a': 1, 'b': 2}

Find Duplicates

Find all duplicate items in a list. More specifically, ub.find_duplicates searches for items that appear more than k times, and returns a mapping from each duplicate item to the positions it appeared in.

Cross-Platform Config and Cache Directories

If you have an application which writes configuration or cache files, the standard place to dump those files differs depending if you are on Windows, Linux, or Mac. Ubelt offers a unified functions for determining what these paths are.

New in version 1.0.0: the ub.Path.appdir classmethod provides a way to achieve the above with a chainable object oriented interface.

The ub.Path.appdir(..., type='cache'), ub.Path.appdir(..., type='config'), and ub.Path.appdir(..., type='data') functions find the correct platform-specific location for these files and calling ensuredir ensures that the directories exist.

The config root directory is ~/AppData/Roaming on Windows, ~/.config on Linux and ~/Library/Application Support on Mac. The cache root directory is ~/AppData/Local on Windows, ~/.config on Linux and ~/Library/Caches on Mac.

Example usage on Linux might look like this:

The ub.symlink function will create a symlink similar to os.symlink. The main differences are that 1) it will not error if the symlink exists and already points to the correct location. 2) it works* on Windows (*hard links and junctions are used if real symlinks are not available)

AutoDict - Autovivification

While the collections.defaultdict is nice, it is sometimes more convenient to have an infinitely nested dictionary of dictionaries.

String-based imports

Ubelt contains functions to import modules dynamically without using the python import statement. While importlib exists, the ubelt implementation is simpler to user and does not have the disadvantage of breaking pytest.

Note ubelt simply provides an interface to this functionality, the core implementation is in xdoctest (over as of version 0.7.0, the code is statically copied into an autogenerated file such that ubelt does not actually depend on xdoctest during runtime).

Related to this functionality are the functions ub.modpath_to_modname and ub.modname_to_modpath, which statically transform (i.e. no code in the target modules is imported or executed) between module names (e.g. ubelt.util_import) and module paths (e.g. ~/.local/conda/envs/cenv3/lib/python3.5/site-packages/ubelt/util_import.py).

Horizontal String Concatenation

Sometimes its just prettier to horizontally concatenate two blocks of text.

Timing

Quickly time a single line.

External tools

Some of the tools in ubelt also exist as standalone modules. I haven't decided if its best to statically copy them into ubelt or require on pypi to satisfy the dependency. There are some tools that are not used by default unless you explicitly allow for them.

Code that is currently statically included (vendored):

Code that is completely optional, and only used in specific cases:

  • Numpy - ub.urepr will format a numpy array nicely by default
  • xxhash - this can be specified as a hasher to ub.hash_data
  • Pygments - used by the util_color module.
  • dateutil - used by the util_time module.

Similar Tools

UBelt is one of many Python utility libraries. A selection of similar libraries are listed here.

Libraries that contain a broad scope of utilities:

Libraries that contain a specific scope of utilities:

Libraries that contain one specific data structure or utility:

Jaraco (i.e. Jason R. Coombs) has an extensive library of utilities:

Ubelt is included in the the [bestof-python list](https://github.com/ml-tooling/best-of-python), which contains many other tools that you should check out.

History:

Ubelt is a migration of the most useful parts of utool(https://github.com/Erotemic/utool) into a standalone module with minimal dependencies.

The utool library contains a number of useful utility functions, but it also contained non-useful functions, as well as the kitchen sink. A number of the functions were too specific or not well documented. The ubelt is a port of the simplest and most useful parts of utool.

Note that there are other cool things in utool that are not in ubelt. Notably, the doctest harness ultimately became xdoctest. Code introspection and dynamic analysis tools were ported to xinspect. The more IPython-y tools were ported to xdev. Parts of it made their way into scriptconfig. The init-file generation was moved to mkinit. Some vim and system-y things can be found in vimtk.

Development on ubelt started 2017-01-30 and development of utool mostly stopped on utool was stopped later that year, but received patches until about 2020. Ubelt achieved 1.0.0 and removed support for Python 2.7 and 3.5 on 2022-01-07.

Notes.

PRs are welcome.

Also check out my other projects which are powered by ubelt:

And my projects related to ubelt:

ubelt's People

Contributors

dependabot[bot] avatar erezsh avatar erotemic avatar mgorny avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ubelt's Issues

`pip install ubelt` incorrectly installs jaraco on linux

The jaraco module is only needed on windows for the symlink functionality. As such it should not be a requirement when installing ubelt on Linux. However, when you pip install ubelt (current pypi version is at 0.2.0), it will install this package. This is undesirable because it makes it appear that ubelt is more heavyweight than it really is.

I'm not sure why this is happening the requirement is specfied in requirements-win32.txt and setup.py should only see if sys.platform.startswith('win32'). Perhaps pypi is parsing the that file? Not exactly sure yet.

1.2.3: The test_deprecated_grabdata_args test fails

Describe the bug

========================================================================================== FAILURES ==========================================================================================
_______________________________________________________________________________ test_deprecated_grabdata_args ________________________________________________________________________________

    def test_deprecated_grabdata_args():
        with pytest.warns(DeprecationWarning):
            import hashlib
            url = _demo_url()
            # dpath = ub.Path.appdir('ubelt/tests/test_download').ensuredir()
            # fname = basename(url)
            # fpath = join(dpath, fname)
>           got_fpath = ub.grabdata(
                url, hash_prefix='e09c80c42fda55f9d992e59ca6b3307d',
                hasher=hashlib.md5())

tests/test_download.py:383: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
ubelt/util_download.py:444: in grabdata
    stamp.renew()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <ubelt.util_cache.CacheStamp object at 0x904747d00>, cfgstr = None, product = None

    def renew(self, cfgstr=None, product=None):
        """
        Recertify that the product has been recomputed by writing a new
        certificate to disk.
    
        Returns:
            None | dict: certificate information if enabled otherwise None.
    
        Example:
            >>> # Test that renew does nothing when the cacher is disabled
            >>> import ubelt as ub
            >>> dpath = ub.Path.appdir('ubelt/tests/cache-stamp-renew').ensuredir()
            >>> self = ub.CacheStamp('foo', dpath=dpath, enabled=False)
            >>> assert self.renew() is None
        """
        if not self.cacher.enabled:
            return None
        if cfgstr is not None:  # nocover
            from ubelt import schedule_deprecation
            schedule_deprecation(
                modname='ubelt',
                migration='Do not pass cfgstr to renew. Use the class depends arg',
                name='cfgstr', type='CacheStamp.renew arg',
                deprecate='1.1.0', error='1.3.0', remove='1.4.0',
            )
        if product is not None:  # nocover
            from ubelt import schedule_deprecation
            schedule_deprecation(
                modname='ubelt',
                migration='Do not pass product to renew. Use the class product arg',
                name='product', type='CacheStamp.renew arg',
                deprecate='1.1.0', error='1.3.0', remove='1.4.0',
            )
        certificate = self._new_certificate(cfgstr, product)
        err = self._check_certificate_hashes(certificate)
        if err:
>           raise RuntimeError(err)
E           RuntimeError: hash_prefix_mismatch

ubelt/util_cache.py:1211: RuntimeError
------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------
[cacher] ... file_10_0.txt.stamp cache miss
[cacher] stamp expired no_cert
Downloading url='http://localhost:17766/file_10_0.txt' to fpath='/disk-samsung/freebsd-ports/devel/py-ubelt/work-py39/.cache/ubelt/file_10_0.txt'
 10/10... rate=129294.24 Hz, eta=0:00:00, total=0:00:00
invalid hash prefix value (expected "e09c80c42fda55f9d992e59ca6b3307d", got "22d42eb002cefa81e9ad604ea57bc01d")
====================================================================================== warnings summary ======================================================================================
../../../../../../usr/local/lib/python3.9/site-packages/pytest_freezegun.py:17
  /usr/local/lib/python3.9/site-packages/pytest_freezegun.py:17: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    if LooseVersion(pytest.__version__) < LooseVersion('3.6.0'):

tests/test_pathlib.py::test_move_meta
tests/test_pathlib.py::test_move_basic
tests/test_pathlib.py::test_move_dir_to_existing_dir_noconflict
tests/test_pathlib.py::test_move_dir_to_existing_dir_withconflict
tests/test_pathlib.py::test_move_dir_to_non_existing
tests/test_pathlib.py::test_move_to_nested_non_existing
  /disk-samsung/freebsd-ports/devel/py-ubelt/work-py39/ubelt-1.2.3/ubelt/util_path.py:1384: UserWarning: The ub.Path.move function is experimental and may change! Do not rely on this behavior yet!
    warnings.warn('The ub.Path.move function is experimental and may change! '

tests/test_pathlib.py::test_copy_dir_to_existing_dir_withconflict
tests/test_pathlib.py::test_copy_meta
tests/test_pathlib.py::test_copy_basic
tests/test_pathlib.py::test_copy_to_nested_non_existing_with_different_symlink_flags
tests/test_pathlib.py::test_copy_dir_to_non_existing
tests/test_pathlib.py::test_copy_dir_to_existing_dir_noconflict
  /disk-samsung/freebsd-ports/devel/py-ubelt/work-py39/ubelt-1.2.3/ubelt/util_path.py:1291: UserWarning: The ub.Path.copy function is experimental and may change, in corner cases. Primary cases seem stable.
    warnings.warn('The ub.Path.copy function is experimental and may change, '

tests/test_indexable.py::test_indexable_walker_map_patterns
  /disk-samsung/freebsd-ports/devel/py-ubelt/work-py39/ubelt-1.2.3/ubelt/util_indexable.py:506: DeprecationWarning: The "indexable_allclose" function was deprecated, will cause an error and will be removed. The current version is 1.2.3. Use `ub.IndexableWalker(items1).allclose(ub.IndexableWalker(items2))` instead
    ub.schedule_deprecation(

tests/test_path.py::test_tempdir
tests/test_import.py::test_import_modpath_basic
tests/test_import.py::test_package_submodules
tests/test_import.py::test_import_modpath_package
tests/test_import.py::test_modpath_to_modname
tests/test_import.py::test_modname_to_modpath_package
tests/test_import.py::test_modname_to_modpath_single
tests/test_import.py::test_modname_to_modpath_namespace
  /disk-samsung/freebsd-ports/devel/py-ubelt/work-py39/ubelt-1.2.3/ubelt/util_path.py:368: DeprecationWarning: The "TempDir" class was deprecated in 1.2.0, will cause an error in 1.4.0 and will be removed in 1.5.0. The current version is 1.2.3. Use tempfile instead
    ub.schedule_deprecation(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================== short test summary info ===================================================================================
SKIPPED [1] tests/test_futures.py:42: long test, demos that timeout does not work with SerialExecutor
SKIPPED [1] tests/test_editable_modules.py:415: UBELT_DO_EDITABLE_TESTS is not enabled
SKIPPED [1] tests/test_hash.py:435: blake3 is not available
SKIPPED [1] tests/test_download.py:256: This takes a long time to timeout and I dont understand why
=================================================================== 1 failed, 202 passed, 4 skipped, 22 warnings in 5.90s ====================================================================

Desktop (please complete the following information):

  • OS: FreeBSD 13.1
  • Ubelt version 1.2.3
  • Python version 3.9

Investigate code scanning alert - is this a real vulnerability?

A code scanner popped up with this as a potential vulnerability. I'm not sure if logging the hash of a file to stdout is leaking anything of relevance. I don't see how it is sensitive information. But if someone can point out if this is a vulnerability, and if so, why? Then, we can remove the log message. But I've found this very useful when establishing the initial hash of expected data (which itself mitigates a security issue!). But this is still UX, so if this somehow is sensitive, then It would be helpful if someone could explain or ellaborate.

Tracking issue for:

ub.argval strips equals signs ("=") from the value

Simple bug. Making an issue so the fix can point to it.

Bugged behavior:

        >>> import ubelt as ub
        >>> argv = ['--path=/path/with/k=3']
        >>> ub.argval('--path', argv=argv) == '/path/with/k=3'

parsing this path should result in /path/with/k=3, but current version results in /path/with/k3 instead.

23 errors in tests

Describe the bug

=========================================================================================== ERRORS ===========================================================================================
__________________________________________________________________________ ERROR at setup of test_download_no_fpath __________________________________________________________________________

module = <module 'test_download' from '/disk-samsung/freebsd-ports/devel/py-ubelt/work-py39/ubelt-1.2.1/tests/test_download.py'>

    def setup_module(module):
        """ setup any state specific to the execution of the given module."""
>       SingletonTestServer.instance()

tests/test_download.py:664: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_download.py:419: in instance
    self = cls()
tests/test_download.py:463: in __init__
    info = ub.cmd(server_cmd, detach=True, cwd=dpath)
ubelt/util_cmd.py:290: in cmd
    info = {'proc': make_proc(), 'command': command_text}
ubelt/util_cmd.py:276: in make_proc
    proc = subprocess.Popen(args, stdout=subprocess.PIPE,
/usr/local/lib/python3.9/subprocess.py:951: in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Popen: returncode: 255 args: ['python', '-m', 'http.server', '43123']>, args = ['python', '-m', 'http.server', '43123'], executable = b'python', preexec_fn = None, close_fds = True
pass_fds = (), cwd = '/disk-samsung/freebsd-ports/devel/py-ubelt/work-py39/.cache/ubelt/tests/test_download/simple_server', env = None, startupinfo = None, creationflags = 0, shell = False
p2cread = -1, p2cwrite = -1, c2pread = 11, c2pwrite = 12, errread = 13, errwrite = 14, restore_signals = True, gid = None, gids = None, uid = None, umask = -1, start_new_session = False

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       restore_signals,
                       gid, gids, uid, umask,
                       start_new_session):
        """Execute program (POSIX version)"""
    
        if isinstance(args, (str, bytes)):
            args = [args]
        elif isinstance(args, os.PathLike):
            if shell:
                raise TypeError('path-like args is not allowed when '
                                'shell is true')
            args = [args]
        else:
            args = list(args)
    
        if shell:
            # On Android the default shell is at '/system/bin/sh'.
            unix_shell = ('/system/bin/sh' if
                      hasattr(sys, 'getandroidapilevel') else '/bin/sh')
            args = [unix_shell, "-c"] + args
            if executable:
                args[0] = executable
    
        if executable is None:
            executable = args[0]
    
        sys.audit("subprocess.Popen", executable, args, cwd, env)
    
        if (_USE_POSIX_SPAWN
                and os.path.dirname(executable)
                and preexec_fn is None
                and not close_fds
                and not pass_fds
                and cwd is None
                and (p2cread == -1 or p2cread > 2)
                and (c2pwrite == -1 or c2pwrite > 2)
                and (errwrite == -1 or errwrite > 2)
                and not start_new_session
                and gid is None
                and gids is None
                and uid is None
                and umask < 0):
            self._posix_spawn(args, executable, env, restore_signals,
                              p2cread, p2cwrite,
                              c2pread, c2pwrite,
                              errread, errwrite)
            return
    
        orig_executable = executable
    
        # For transferring possible exec failure from child to parent.
        # Data format: "exception name:hex errno:description"
        # Pickle is not used; it is complex and involves memory allocation.
        errpipe_read, errpipe_write = os.pipe()
        # errpipe_write must not be in the standard io 0, 1, or 2 fd range.
        low_fds_to_close = []
        while errpipe_write < 3:
            low_fds_to_close.append(errpipe_write)
            errpipe_write = os.dup(errpipe_write)
        for low_fd in low_fds_to_close:
            os.close(low_fd)
        try:
            try:
                # We must avoid complex work that could involve
                # malloc or free in the child process to avoid
                # potential deadlocks, thus we do all this here.
                # and pass it to fork_exec()
    
                if env is not None:
                    env_list = []
                    for k, v in env.items():
                        k = os.fsencode(k)
                        if b'=' in k:
                            raise ValueError("illegal environment variable name")
                        env_list.append(k + b'=' + os.fsencode(v))
                else:
                    env_list = None  # Use execv instead of execve.
                executable = os.fsencode(executable)
                if os.path.dirname(executable):
                    executable_list = (executable,)
                else:
                    # This matches the behavior of os._execvpe().
                    executable_list = tuple(
                        os.path.join(os.fsencode(dir), executable)
                        for dir in os.get_exec_path(env))
                fds_to_keep = set(pass_fds)
                fds_to_keep.add(errpipe_write)
                self.pid = _posixsubprocess.fork_exec(
                        args, executable_list,
                        close_fds, tuple(sorted(map(int, fds_to_keep))),
                        cwd, env_list,
                        p2cread, p2cwrite, c2pread, c2pwrite,
                        errread, errwrite,
                        errpipe_read, errpipe_write,
                        restore_signals, start_new_session,
                        gid, gids, uid, umask,
                        preexec_fn)
                self._child_created = True
            finally:
                # be sure the FD is closed no matter what
                os.close(errpipe_write)
    
            self._close_pipe_fds(p2cread, p2cwrite,
                                 c2pread, c2pwrite,
                                 errread, errwrite)
    
            # Wait for exec to fail or succeed; possibly raising an
            # exception (limited in size)
            errpipe_data = bytearray()
            while True:
                part = os.read(errpipe_read, 50000)
                errpipe_data += part
                if not part or len(errpipe_data) > 50000:
                    break
        finally:
            # be sure the FD is closed no matter what
            os.close(errpipe_read)
    
        if errpipe_data:
            try:
                pid, sts = os.waitpid(self.pid, 0)
                if pid == self.pid:
                    self._handle_exitstatus(sts)
                else:
                    self.returncode = sys.maxsize
            except ChildProcessError:
                pass
    
            try:
                exception_name, hex_errno, err_msg = (
                        errpipe_data.split(b':', 2))
                # The encoding here should match the encoding
                # written in by the subprocess implementations
                # like _posixsubprocess
                err_msg = err_msg.decode()
            except ValueError:
                exception_name = b'SubprocessError'
                hex_errno = b'0'
                err_msg = 'Bad exception data from child: {!r}'.format(
                              bytes(errpipe_data))
            child_exception_type = getattr(
                    builtins, exception_name.decode('ascii'),
                    SubprocessError)
            if issubclass(child_exception_type, OSError) and hex_errno:
                errno_num = int(hex_errno, 16)
                child_exec_never_called = (err_msg == "noexec")
                if child_exec_never_called:
                    err_msg = ""
                    # The error must be from chdir(cwd).
                    err_filename = cwd
                else:
                    err_filename = orig_executable
                if errno_num != 0:
                    err_msg = os.strerror(errno_num)
>               raise child_exception_type(errno_num, err_msg, err_filename)
E               FileNotFoundError: [Errno 2] No such file or directory: 'python'

/usr/local/lib/python3.9/subprocess.py:1821: FileNotFoundError
----------------------------------------------------------------------------------- Captured stdout setup ------------------------------------------------------------------------------------
port = 43123
_________________________________________________________________________ ERROR at setup of test_download_with_fpath _________________________________________________________________________

module = <module 'test_download' from '/disk-samsung/freebsd-ports/devel/py-ubelt/work-py39/ubelt-1.2.1/tests/test_download.py'>

    def setup_module(module):
        """ setup any state specific to the execution of the given module."""
>       SingletonTestServer.instance()

tests/test_download.py:664: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_download.py:419: in instance
    self = cls()
tests/test_download.py:463: in __init__
    info = ub.cmd(server_cmd, detach=True, cwd=dpath)
ubelt/util_cmd.py:290: in cmd
    info = {'proc': make_proc(), 'command': command_text}
ubelt/util_cmd.py:276: in make_proc
    proc = subprocess.Popen(args, stdout=subprocess.PIPE,
/usr/local/lib/python3.9/subprocess.py:951: in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Popen: returncode: 255 args: ['python', '-m', 'http.server', '43123']>, args = ['python', '-m', 'http.server', '43123'], executable = b'python', preexec_fn = None, close_fds = True
pass_fds = (), cwd = '/disk-samsung/freebsd-ports/devel/py-ubelt/work-py39/.cache/ubelt/tests/test_download/simple_server', env = None, startupinfo = None, creationflags = 0, shell = False
p2cread = -1, p2cwrite = -1, c2pread = 11, c2pwrite = 12, errread = 13, errwrite = 14, restore_signals = True, gid = None, gids = None, uid = None, umask = -1, start_new_session = False

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       restore_signals,
                       gid, gids, uid, umask,
                       start_new_session):
        """Execute program (POSIX version)"""
    
        if isinstance(args, (str, bytes)):
            args = [args]
        elif isinstance(args, os.PathLike):
            if shell:
                raise TypeError('path-like args is not allowed when '
                                'shell is true')
            args = [args]
        else:
            args = list(args)
    
        if shell:
            # On Android the default shell is at '/system/bin/sh'.
            unix_shell = ('/system/bin/sh' if
                      hasattr(sys, 'getandroidapilevel') else '/bin/sh')
            args = [unix_shell, "-c"] + args
            if executable:
                args[0] = executable
    
        if executable is None:
            executable = args[0]
    
        sys.audit("subprocess.Popen", executable, args, cwd, env)
    
        if (_USE_POSIX_SPAWN
                and os.path.dirname(executable)
                and preexec_fn is None
                and not close_fds
                and not pass_fds
                and cwd is None
                and (p2cread == -1 or p2cread > 2)
                and (c2pwrite == -1 or c2pwrite > 2)
                and (errwrite == -1 or errwrite > 2)
                and not start_new_session
                and gid is None
                and gids is None
                and uid is None
                and umask < 0):
            self._posix_spawn(args, executable, env, restore_signals,
                              p2cread, p2cwrite,
                              c2pread, c2pwrite,
                              errread, errwrite)
            return
    
        orig_executable = executable
    
        # For transferring possible exec failure from child to parent.
        # Data format: "exception name:hex errno:description"
        # Pickle is not used; it is complex and involves memory allocation.
        errpipe_read, errpipe_write = os.pipe()
        # errpipe_write must not be in the standard io 0, 1, or 2 fd range.
        low_fds_to_close = []
        while errpipe_write < 3:
            low_fds_to_close.append(errpipe_write)
            errpipe_write = os.dup(errpipe_write)
        for low_fd in low_fds_to_close:
            os.close(low_fd)
        try:
            try:
                # We must avoid complex work that could involve
                # malloc or free in the child process to avoid
                # potential deadlocks, thus we do all this here.
                # and pass it to fork_exec()
    
                if env is not None:
                    env_list = []
                    for k, v in env.items():
                        k = os.fsencode(k)
                        if b'=' in k:
                            raise ValueError("illegal environment variable name")
                        env_list.append(k + b'=' + os.fsencode(v))
                else:
                    env_list = None  # Use execv instead of execve.
                executable = os.fsencode(executable)
                if os.path.dirname(executable):
                    executable_list = (executable,)
                else:
                    # This matches the behavior of os._execvpe().
                    executable_list = tuple(
                        os.path.join(os.fsencode(dir), executable)
                        for dir in os.get_exec_path(env))
                fds_to_keep = set(pass_fds)
                fds_to_keep.add(errpipe_write)
                self.pid = _posixsubprocess.fork_exec(
                        args, executable_list,
                        close_fds, tuple(sorted(map(int, fds_to_keep))),
                        cwd, env_list,
                        p2cread, p2cwrite, c2pread, c2pwrite,
                        errread, errwrite,
                        errpipe_read, errpipe_write,
                        restore_signals, start_new_session,
                        gid, gids, uid, umask,
                        preexec_fn)
                self._child_created = True
            finally:
                # be sure the FD is closed no matter what
                os.close(errpipe_write)
    
            self._close_pipe_fds(p2cread, p2cwrite,
                                 c2pread, c2pwrite,
                                 errread, errwrite)
    
            # Wait for exec to fail or succeed; possibly raising an
            # exception (limited in size)
            errpipe_data = bytearray()
            while True:
                part = os.read(errpipe_read, 50000)
                errpipe_data += part
                if not part or len(errpipe_data) > 50000:
                    break
        finally:
            # be sure the FD is closed no matter what
            os.close(errpipe_read)
    
        if errpipe_data:
            try:
                pid, sts = os.waitpid(self.pid, 0)
                if pid == self.pid:
                    self._handle_exitstatus(sts)
                else:
                    self.returncode = sys.maxsize
            except ChildProcessError:
                pass
    
            try:
                exception_name, hex_errno, err_msg = (
                        errpipe_data.split(b':', 2))
                # The encoding here should match the encoding
                # written in by the subprocess implementations
                # like _posixsubprocess
                err_msg = err_msg.decode()
            except ValueError:
                exception_name = b'SubprocessError'
                hex_errno = b'0'
                err_msg = 'Bad exception data from child: {!r}'.format(
                              bytes(errpipe_data))
            child_exception_type = getattr(
                    builtins, exception_name.decode('ascii'),
                    SubprocessError)
            if issubclass(child_exception_type, OSError) and hex_errno:
                errno_num = int(hex_errno, 16)
                child_exec_never_called = (err_msg == "noexec")
                if child_exec_never_called:
                    err_msg = ""
                    # The error must be from chdir(cwd).
                    err_filename = cwd
                else:
                    err_filename = orig_executable
                if errno_num != 0:
                    err_msg = os.strerror(errno_num)
>               raise child_exception_type(errno_num, err_msg, err_filename)
E               FileNotFoundError: [Errno 2] No such file or directory: 'python'

/usr/local/lib/python3.9/subprocess.py:1821: FileNotFoundError
_________________________________________________________________________ ERROR at setup of test_download_chunksize __________________________________________________________________________

module = <module 'test_download' from '/disk-samsung/freebsd-ports/devel/py-ubelt/work-py39/ubelt-1.2.1/tests/test_download.py'>

    def setup_module(module):
        """ setup any state specific to the execution of the given module."""
>       SingletonTestServer.instance()

tests/test_download.py:664: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_download.py:419: in instance
    self = cls()
tests/test_download.py:463: in __init__
    info = ub.cmd(server_cmd, detach=True, cwd=dpath)
ubelt/util_cmd.py:290: in cmd
    info = {'proc': make_proc(), 'command': command_text}
ubelt/util_cmd.py:276: in make_proc
    proc = subprocess.Popen(args, stdout=subprocess.PIPE,
/usr/local/lib/python3.9/subprocess.py:951: in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Popen: returncode: 255 args: ['python', '-m', 'http.server', '43123']>, args = ['python', '-m', 'http.server', '43123'], executable = b'python', preexec_fn = None, close_fds = True
pass_fds = (), cwd = '/disk-samsung/freebsd-ports/devel/py-ubelt/work-py39/.cache/ubelt/tests/test_download/simple_server', env = None, startupinfo = None, creationflags = 0, shell = False
p2cread = -1, p2cwrite = -1, c2pread = 11, c2pwrite = 12, errread = 13, errwrite = 14, restore_signals = True, gid = None, gids = None, uid = None, umask = -1, start_new_session = False

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       restore_signals,
                       gid, gids, uid, umask,
                       start_new_session):
        """Execute program (POSIX version)"""
    
        if isinstance(args, (str, bytes)):
            args = [args]
        elif isinstance(args, os.PathLike):
            if shell:
                raise TypeError('path-like args is not allowed when '
                                'shell is true')
            args = [args]
        else:
            args = list(args)
    
        if shell:
            # On Android the default shell is at '/system/bin/sh'.
            unix_shell = ('/system/bin/sh' if
                      hasattr(sys, 'getandroidapilevel') else '/bin/sh')
            args = [unix_shell, "-c"] + args
            if executable:
                args[0] = executable
    
        if executable is None:
            executable = args[0]
    
        sys.audit("subprocess.Popen", executable, args, cwd, env)
    
        if (_USE_POSIX_SPAWN
                and os.path.dirname(executable)
                and preexec_fn is None
                and not close_fds
                and not pass_fds
                and cwd is None
                and (p2cread == -1 or p2cread > 2)
                and (c2pwrite == -1 or c2pwrite > 2)
                and (errwrite == -1 or errwrite > 2)
                and not start_new_session
                and gid is None
                and gids is None
                and uid is None
                and umask < 0):
            self._posix_spawn(args, executable, env, restore_signals,
                              p2cread, p2cwrite,
                              c2pread, c2pwrite,
                              errread, errwrite)
            return
    
        orig_executable = executable
    
        # For transferring possible exec failure from child to parent.
        # Data format: "exception name:hex errno:description"
        # Pickle is not used; it is complex and involves memory allocation.
        errpipe_read, errpipe_write = os.pipe()
        # errpipe_write must not be in the standard io 0, 1, or 2 fd range.
        low_fds_to_close = []
        while errpipe_write < 3:
            low_fds_to_close.append(errpipe_write)
            errpipe_write = os.dup(errpipe_write)
        for low_fd in low_fds_to_close:
            os.close(low_fd)
        try:
            try:
                # We must avoid complex work that could involve
                # malloc or free in the child process to avoid
                # potential deadlocks, thus we do all this here.
                # and pass it to fork_exec()
    
                if env is not None:
                    env_list = []
                    for k, v in env.items():
                        k = os.fsencode(k)
                        if b'=' in k:
                            raise ValueError("illegal environment variable name")
                        env_list.append(k + b'=' + os.fsencode(v))
                else:
                    env_list = None  # Use execv instead of execve.
                executable = os.fsencode(executable)
                if os.path.dirname(executable):
                    executable_list = (executable,)
                else:
                    # This matches the behavior of os._execvpe().
                    executable_list = tuple(
                        os.path.join(os.fsencode(dir), executable)
                        for dir in os.get_exec_path(env))
                fds_to_keep = set(pass_fds)
                fds_to_keep.add(errpipe_write)
                self.pid = _posixsubprocess.fork_exec(
                        args, executable_list,
                        close_fds, tuple(sorted(map(int, fds_to_keep))),
                        cwd, env_list,
                        p2cread, p2cwrite, c2pread, c2pwrite,
                        errread, errwrite,
                        errpipe_read, errpipe_write,
                        restore_signals, start_new_session,
                        gid, gids, uid, umask,
                        preexec_fn)
                self._child_created = True
            finally:
                # be sure the FD is closed no matter what
                os.close(errpipe_write)
    
            self._close_pipe_fds(p2cread, p2cwrite,
                                 c2pread, c2pwrite,
                                 errread, errwrite)
    
            # Wait for exec to fail or succeed; possibly raising an
            # exception (limited in size)
            errpipe_data = bytearray()
            while True:
                part = os.read(errpipe_read, 50000)
                errpipe_data += part
                if not part or len(errpipe_data) > 50000:
                    break
        finally:
            # be sure the FD is closed no matter what
            os.close(errpipe_read)
    
        if errpipe_data:
            try:
                pid, sts = os.waitpid(self.pid, 0)
                if pid == self.pid:
                    self._handle_exitstatus(sts)
                else:
                    self.returncode = sys.maxsize
            except ChildProcessError:
                pass
    
            try:
                exception_name, hex_errno, err_msg = (
                        errpipe_data.split(b':', 2))
                # The encoding here should match the encoding
                # written in by the subprocess implementations
                # like _posixsubprocess
                err_msg = err_msg.decode()
            except ValueError:
                exception_name = b'SubprocessError'
                hex_errno = b'0'
                err_msg = 'Bad exception data from child: {!r}'.format(
                              bytes(errpipe_data))
            child_exception_type = getattr(
                    builtins, exception_name.decode('ascii'),
                    SubprocessError)
            if issubclass(child_exception_type, OSError) and hex_errno:
                errno_num = int(hex_errno, 16)
                child_exec_never_called = (err_msg == "noexec")
                if child_exec_never_called:
                    err_msg = ""
                    # The error must be from chdir(cwd).
                    err_filename = cwd
                else:
                    err_filename = orig_executable
                if errno_num != 0:
                    err_msg = os.strerror(errno_num)
>               raise child_exception_type(errno_num, err_msg, err_filename)
E               FileNotFoundError: [Errno 2] No such file or directory: 'python'

/usr/local/lib/python3.9/subprocess.py:1821: FileNotFoundError
_______________________________________________________________________ ERROR at setup of test_download_cover_hashers ________________________________________________________________________

module = <module 'test_download' from '/disk-samsung/freebsd-ports/devel/py-ubelt/work-py39/ubelt-1.2.1/tests/test_download.py'>

    def setup_module(module):
        """ setup any state specific to the execution of the given module."""
>       SingletonTestServer.instance()

tests/test_download.py:664: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_download.py:419: in instance
    self = cls()
tests/test_download.py:463: in __init__
    info = ub.cmd(server_cmd, detach=True, cwd=dpath)
ubelt/util_cmd.py:290: in cmd
    info = {'proc': make_proc(), 'command': command_text}
ubelt/util_cmd.py:276: in make_proc
    proc = subprocess.Popen(args, stdout=subprocess.PIPE,
/usr/local/lib/python3.9/subprocess.py:951: in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Popen: returncode: 255 args: ['python', '-m', 'http.server', '43123']>, args = ['python', '-m', 'http.server', '43123'], executable = b'python', preexec_fn = None, close_fds = True
pass_fds = (), cwd = '/disk-samsung/freebsd-ports/devel/py-ubelt/work-py39/.cache/ubelt/tests/test_download/simple_server', env = None, startupinfo = None, creationflags = 0, shell = False
p2cread = -1, p2cwrite = -1, c2pread = 11, c2pwrite = 12, errread = 13, errwrite = 14, restore_signals = True, gid = None, gids = None, uid = None, umask = -1, start_new_session = False

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       restore_signals,
                       gid, gids, uid, umask,
                       start_new_session):
        """Execute program (POSIX version)"""
    
        if isinstance(args, (str, bytes)):
            args = [args]
        elif isinstance(args, os.PathLike):
            if shell:
                raise TypeError('path-like args is not allowed when '
                                'shell is true')
            args = [args]
        else:
            args = list(args)
    
        if shell:
            # On Android the default shell is at '/system/bin/sh'.
            unix_shell = ('/system/bin/sh' if
                      hasattr(sys, 'getandroidapilevel') else '/bin/sh')
            args = [unix_shell, "-c"] + args
            if executable:
                args[0] = executable
    
        if executable is None:
            executable = args[0]
    
        sys.audit("subprocess.Popen", executable, args, cwd, env)
    
        if (_USE_POSIX_SPAWN
                and os.path.dirname(executable)
                and preexec_fn is None
                and not close_fds
                and not pass_fds
                and cwd is None
                and (p2cread == -1 or p2cread > 2)
                and (c2pwrite == -1 or c2pwrite > 2)
                and (errwrite == -1 or errwrite > 2)
                and not start_new_session
                and gid is None
                and gids is None
                and uid is None
                and umask < 0):
            self._posix_spawn(args, executable, env, restore_signals,
                              p2cread, p2cwrite,
                              c2pread, c2pwrite,
                              errread, errwrite)
            return
    
        orig_executable = executable
    
        # For transferring possible exec failure from child to parent.
        # Data format: "exception name:hex errno:description"
        # Pickle is not used; it is complex and involves memory allocation.
        errpipe_read, errpipe_write = os.pipe()
        # errpipe_write must not be in the standard io 0, 1, or 2 fd range.
        low_fds_to_close = []
        while errpipe_write < 3:

Expected behavior
A clear and concise description of what you expected to happen.

Desktop (please complete the following information):

  • OS: FreeBSD 13.1
  • Ubelt version 1.2.1
  • Python version 3.9

Additional context
Add any other context about the problem here.

Provide a subprocess-compatible interface in cmd

I was exploring using subprocess_tee when I found it doesn't support Windows and has issues with Python 3.12.

When I tried dropping in ubelt.cmd as a replacement, I found it has a very different interface than subprocess, returning a dict instead of an object and having different keys.

It would be nice if the result from command was an object similar to the one returned by subprocess.Popen (or even identical), so that code like this can readily switch between the implementations.

test_numpy_object_array fails: TypeError: directly hashing ndarrays with dtype=object is unstable

Describe the bug


――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― test_numpy_object_array ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

    def test_numpy_object_array():
        """
        _HASHABLE_EXTENSIONS = ub.util_hash._HASHABLE_EXTENSIONS
        """
        if np is None:
            pytest.skip('requires numpy')
        # An object array should have the same repr as a list of a tuple of data
        data = np.array([1, 2, 3], dtype=object)
>       objhash = ub.hash_data(data)

tests/test_hash.py:245: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
ubelt/util_hash.py:1107: in hash_data
    _update_hasher(hasher, data, types=types, extensions=extensions)
ubelt/util_hash.py:953: in _update_hasher
    prefix, hashable = _convert_to_hashable(data, types,
ubelt/util_hash.py:875: in _convert_to_hashable
    prefix, hashable = hash_func(data)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

data = array([1, 2, 3], dtype=object)

    @self.register(np.ndarray)
    def _convert_numpy_array(data):
        """
        Example:
            >>> import ubelt as ub
            >>> if not ub.modname_to_modpath('numpy'):
            ...     raise pytest.skip()
            >>> import numpy as np
            >>> data_f32 = np.zeros((3, 3, 3), dtype=np.float64)
            >>> data_i64 = np.zeros((3, 3, 3), dtype=np.int64)
            >>> data_i32 = np.zeros((3, 3, 3), dtype=np.int32)
            >>> hash_f64 = _hashable_sequence(data_f32, types=True)
            >>> hash_i64 = _hashable_sequence(data_i64, types=True)
            >>> hash_i32 = _hashable_sequence(data_i64, types=True)
            >>> assert hash_i64 != hash_f64
            >>> assert hash_i64 != hash_i32
        """
        if data.dtype.kind == 'O':
            msg = 'directly hashing ndarrays with dtype=object is unstable'
>           raise TypeError(msg)
E           TypeError: directly hashing ndarrays with dtype=object is unstable

ubelt/util_hash.py:546: TypeError

Version: 1.3.0
Python-3.9
FreeBSD 13.2

help to confirm the depends of the module

Describe the bug
This is not bug, just one confirmation.

I am packaging the ubelt into Debian, everything is well but when run test cases I got:

removing build/bdist.linux-x86_64/wheel
* Building wheel...
Successfully built ubelt-1.2.3-py3-none-any.whl
I: pybuild plugin_pyproject:118: Unpacking wheel built for python3.10 with "installer" module
   dh_auto_test -O--buildsystem=pybuild
I: pybuild base:240: cd /<<PKGBUILDDIR>>/.pybuild/cpython3_3.11/build; python3.11 -m pytest tests
ERROR: usage: __main__.py [options] [file_or_dir] [file_or_dir] [...]
__main__.py: error: unrecognized arguments: --xdoctest --xdoctest-style=google
  inifile: /<<PKGBUILDDIR>>/pyproject.toml
  rootdir: /<<PKGBUILDDIR>>

E: pybuild pybuild:379: test: plugin pyproject failed with: exit code=4: cd /<<PKGBUILDDIR>>/.pybuild/cpython3_3.11/build; python3.11 -m pytest tests
I: pybuild base:240: cd /<<PKGBUILDDIR>>/.pybuild/cpython3_3.10/build; python3.10 -m pytest tests
ERROR: usage: __main__.py [options] [file_or_dir] [file_or_dir] [...]
__main__.py: error: unrecognized arguments: --xdoctest --xdoctest-style=google
  inifile: /<<PKGBUILDDIR>>/pyproject.toml
  rootdir: /<<PKGBUILDDIR>>

E: pybuild pybuild:379: test: plugin pyproject failed with: exit code=4: cd /<<PKGBUILDDIR>>/.pybuild/cpython3_3.10/build; python3.10 -m pytest tests
dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p "3.11 3.10" returned exit code 13
make: *** [debian/rules:4: binary] Error 25
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2

Please help me to confirm the xdoctest module is the repo?
If so, I have to package it first to meet the requierment.
thanks.

Test ubelt/util_platform.py::find_exe:0 fails if which(1) is not installed

Describe the bug
Contrary to the popular belief, which(1) is not part of POSIX base system and it is entirely possible not to have it installed at all. However, ubelt's tests assume that it is always present and fail when it isn't:

________________________________________________________ [xdoctest] find_exe:0 ________________________________________________________
* REASON: TypeError
DOCTEST DEBUG INFO
  XDoc "/tmp/portage/dev-python/ubelt-1.1.0/work/ubelt-1.1.0/ubelt/util_platform.py::find_exe:0", line 3 <- wrt doctest
  File "/tmp/portage/dev-python/ubelt-1.1.0/work/ubelt-1.1.0/ubelt/util_platform.py", line 288, <- wrt source file
DOCTEST PART BREAKDOWN
Failed Part:
    1 >>> find_exe('ls')
    2 >>> find_exe('ping')
    3 >>> assert find_exe('which') == find_exe(find_exe('which'))
    4 >>> find_exe('which', multi=True)
    5 >>> find_exe('ping', multi=True)
    6 >>> find_exe('cmake', multi=True)
    7 >>> find_exe('nvcc', multi=True)
    8 >>> find_exe('noexist', multi=True)
DOCTEST TRACEBACK
Traceback (most recent call last):

  File "/usr/lib/python3.8/site-packages/xdoctest/doctest_example.py", line 653, in run
    exec(code, test_globals)

  File "<doctest:/tmp/portage/dev-python/ubelt-1.1.0/work/ubelt-1.1.0/ubelt/util_platform.py::find_exe:0>", line rel: 3, abs: 288, in <module>
    >>> assert find_exe('which') == find_exe(find_exe('which'))

  File "/tmp/portage/dev-python/ubelt-1.1.0/work/ubelt-1.1.0/ubelt/util_platform.py", line 318, in find_exe
    for fpath in results:

  File "/tmp/portage/dev-python/ubelt-1.1.0/work/ubelt-1.1.0/ubelt/util_platform.py", line 315, in <genexpr>
    results = (fpath for fpath in candidates

  File "/tmp/portage/dev-python/ubelt-1.1.0/work/ubelt-1.1.0/ubelt/util_platform.py", line 386, in find_path
    for candidate in candidates:

  File "/tmp/portage/dev-python/ubelt-1.1.0/work/ubelt-1.1.0/ubelt/util_platform.py", line 373, in <genexpr>
    candidates = (join(dpath, name) for dpath in dpaths)

  File "/usr/lib/python3.8/posixpath.py", line 90, in join
    genericpath._check_arg_types('join', a, *p)

  File "/usr/lib/python3.8/genericpath.py", line 152, in _check_arg_types
    raise TypeError(f'{funcname}() argument must be str, bytes, or '

TypeError: join() argument must be str, bytes, or os.PathLike object, not 'NoneType'

DOCTEST REPRODUCTION
CommandLine:
    pytest /tmp/portage/dev-python/ubelt-1.1.0/work/ubelt-1.1.0/ubelt/util_platform.py::find_exe:0
/tmp/portage/dev-python/ubelt-1.1.0/work/ubelt-1.1.0/ubelt/util_platform.py:288: TypeError
-------------------------------------------------------- Captured stdout call ---------------------------------------------------------
====== <exec> ======
* DOCTEST : /tmp/portage/dev-python/ubelt-1.1.0/work/ubelt-1.1.0/ubelt/util_platform.py::find_exe:0, line 286 <- wrt source file

To Reproduce

  1. Uninstall which(1).
  2. pytest ubelt/util_platform.py::find_exe

Expected behavior
Tests passing.

Desktop (please complete the following information):

  • OS: Gentoo Linux
  • Ubelt version 1.1.0
  • Python version 3.8.13 (not really relevant)

FileNotFoundError attempting to launch sys.executable on Windows

In jaraco/safety-tox, I'm attempting to write a script that will wrap tox and tee its output so the output can be inspected after the run and conditionally alter the return code (bypassing failures for missing dependencies).

When I try to use ubelt.cmd to launch the subprocess, however, it fails with this traceback:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\hostedtoolcache\windows\Python\3.11.0\x64\Lib\site-packages\safety-tox.py", line 72, in <module>
    __name__ == '__main__' and run(sys.argv[1:])
                               ^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.0\x64\Lib\site-packages\safety-tox.py", line 69, in run
    raise SystemExit(Handler().run(args))
                     ^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.0\x64\Lib\site-packages\safety-tox.py", line 51, in run
    proc = self.runner(cmd)
           ^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.0\x64\Lib\site-packages\jaraco\functools.py", line 35, in <lambda>
    return lambda *args, **kwargs: f1(f2(*args, **kwargs))
                                      ^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.0\x64\Lib\site-packages\ubelt\util_cmd.py", line 316, in cmd
    proc = make_proc()
           ^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.0\x64\Lib\site-packages\ubelt\util_cmd.py", line 293, in make_proc
    proc = subprocess.Popen(args, stdout=subprocess.PIPE,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.0\x64\Lib\subprocess.py", line 1022, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\hostedtoolcache\windows\Python\3.11.0\x64\Lib\subprocess.py", line 1491, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] The system cannot find the file specified

It seems that the way ubelt is manipulating the arguments, it doesn't allow the command to execute the way it would naturally.

I try to avoid shell=True and always pass a sequence of args for the command as I find that to be the most portable. Forcing a string for the cmd leads to issues like seen above.

Is there any way to get the tee functionality from ubelt without any other manipulation?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.