Code Monkey home page Code Monkey logo

pytoolbox's Introduction

Pytoolbox

https://badge.fury.io/py/pytoolbox.png https://coveralls.io/repos/davidfischer-ch/pytoolbox/badge.png

Afraid of red status ? Please click on the link, sometimes this is not my fault ;-)

This module is a Toolbox for Python scripts.

Documentation: https://pytoolbox.readthedocs.org

Repository: https://github.com/davidfischer-ch/pytoolbox

This library supports Python 3.11 and more recent.

What the release number stands for ?

I do my best to follow this interesting recommendation : Semantic Versioning 2.0.0

How to install it ?

Install some packages that are not handled by pip:

$ sudo apt install liblzma-dev libxml2-dev libxslt-dev libyaml-dev libz-dev
$ sudo apt install ffmpeg git-core python3-dev python3-gi python3-pip screen

If planning to use the imaging extra, especially the exif classes, then you'll have to install:

$ sudo apt install libcairo2 libcairo2-dev libexiv2-dev libgexiv2-dev libgirepository1.0-dev

The gir1.2-gexiv2-0.10 should also be installed, maybe its already the case.

You may find useful to read PyGObject's documentation.

If planning to use the vision feature, then you have to install some requirements for dlib:

$ sudo apt install build-essential cmake pkg-config

See https://learnopencv.com/install-dlib-on-ubuntu/ for an up-to-date procedure.

Make sure that pip is up-to-date (PIPception):

$ source /some/python3/venv/bin/active
$ pip install --upgrade pip setuptools wheel

Then, you only need to run setup.py:

$ source /some/python3/venv/bin/activate
$ pip install .

How to enable features/modules ?

Example:

$ python setup.py install --help

Common commands: (see '--help-commands' for more)

  setup.py build      will build the package underneath 'build/'
  setup.py install    will install the package

Global options:
  --verbose (-v)  run verbosely (default)
  --quiet (-q)    run quietly (turns verbosity off)
  --dry-run (-n)  don't actually do anything
  --help (-h)     show detailed help message
  --no-user-cfg   ignore pydistutils.cfg in your home directory

Options for 'WithExtra' command:
  ...
  --extra-all                          Install dependencies for All Modules.
  --extra-atlassian                    Install dependencies for Atlassian.
  --extra-aws                          Install dependencies for AWS.
  --extra-django                       Install dependencies for Django.
  --extra-django-filter                Install dependencies for Django Filter.
  --extra-django-formtools             Install dependencies for Django Form Tools.
  --extra-flask                        Install dependencies for Flask.
  --extra-imaging                      Install dependencies for Imaging.
  --extra-jinja2                       Install dependencies for Jinja2.
  --extra-mongodb                      Install dependencies for MongoDB.
  --extra-network                      Install dependencies for Networking.
  --extra-pandas                       Install dependencies for Pandas.
  --extra-rest-framework               Install dependencies for Django REST Framework.
  --extra-selenium                     Install dependencies for Selenium.
  --extra-smpte2022                    Install dependencies for SMPTE-2022.
  --extra-unittest                     Install dependencies for Unit Test.
  --extra-vision                       Install dependencies for Vision.
  --extra-voluptuous                   Install dependencies for Voluptuous.
  --extra-doc                          Install dependencies for Pytoolbox Docs.
  --extra-test                         Install dependencies for Pytoolbox Tests.

usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help


$ python setup.py install --extra-smpte2022

Another way to do this, with pip:

$ pip install -e .[django,flask,mongo,smpte2022]

How to check coverage ?

$ source /some/python3/venv/bin/activate
$ pip install -e .[all,test]
$ flake8 pytoolbox
$ pytest
$ xdg-open htmlcov/index.html

Remarks:

  • All Django related modules are excluded from tests!
  • However I am using them with Django 1.8 up to 3.1.0.

How to use it ?

Here is an example hello.py using the cmd function provided by pytoolbox:

$ from pytoolbox.subprocess import cmd
$ print(cmd('echo Hello World!')['stdout'])

How to generate documentation ?

The documentation is generated by Sphinx. In fact most of this documentation is extracted from the docstrings of the code.

Here is the procedure:

$ source /some/python3/venv/bin/activate
$ pip install -e .[docs]
$ xdg-open docs/build/html/index.html

How to add it to dependencies of my own project ?

Here is an example setup.py for a project called my-cool-project:

  from setuptools import setup

  setup(
name='my-cool-project',
            version='0.8',
            author='Firstname Lastname',
            author_email='[email protected]',
            install_requires=['...', 'pytoolbox>=14<15', '...'],
            tests_require=['...', 'pytest', '...'],
            license='GPLv3',
            url='https://github.com/nickname/my-cool-project',
            packages=['my_cool_project'])

See pip vcs support to get further details about this.

You also need to install git-core, but it is probably already the case, at least on your development computer ;-)

2012 - 2024 David Fischer

pytoolbox's People

Contributors

aa6moham avatar davidfischer-ch avatar dependabot[bot] avatar kyouko-taiga avatar smarter avatar tobbee avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

pytoolbox's Issues

Cleanup setup.py

Implement develop install test commands for handling the --extra-* arguments

Cover Django's related modules with tests

This library is heavily used and tests (integration tests) by the Cloud Transcoding Platform CloudNcode.com and other projects, but this is not an excuse and I must kick my ass to add unit-tests here!

Bug: FEC reset at sequence wraparound

Thanks for nice software. I use it for some testing and found that the FecGenerator has a bug at sequence number wrap-around.

on_reset() reports

       Media seq=0 is out of sequence (expected 0) : FEC algorithm resetted !

This is due to the line generator.py:208 which reads

        if self._media_sequence and media.sequence == self._media_sequence:

but should read

       if self._media_sequence is not None and media.sequence == self._media_sequence:

Drop Python 2 support

This will allow to cleanup to code and make it more robust (e.g. using forced kwargs).

Support Python 3

Two options:

  • Convert with 2to3
  • Use module six + various in-code refinements

Test compatibility with pathlib.Path

Add tests proving the library behave properly when using pathlib.Path instead of str.
Even use this as a criteria to make it smarter (hint: is_file or is_path automagic default).

Tests : Create a better filter

At the moment pytoolbox_runtests.py is configured as such:

  • django module is ignored : Django complains about not founding settings ...
  • ming module is ignored if python3 : ming is not compatible ...

Actually, the ignore filter is : django.py|schema.py|session.py for python3 and only django.py for python2. This must be improved as schema.py and session.py is not explicitly linked to ming/. However I do not know and don't have time to investigate why a regex does not work.

Publish some code that was private

Integrate code I developed to various modules :

  • Add Ansible module
  • Add Git module
  • Add GitLab module
  • Add MkDocs module
  • Add Security module
  • Add Terraform module
  • Other to various modules
  • GitLab CLI (processor & co)

chown: handle broken symlinks

ebu21516:2013-12-18 15:26:34 INFO juju juju-log.go:66 oscied-orchestra/0: [HOOK] Exiting OrchestraHooks hook config-changed
ebu21516:2013-12-18 15:26:34 INFO juju.worker.uniter context.go:255 HOOK Traceback (most recent call last):
ebu21516:2013-12-18 15:26:34 INFO juju.worker.uniter context.go:255 HOOK File "/usr/lib/python2.7/runpy.py", line 162, in run_module_as_main
ebu21516:2013-12-18 15:26:34 INFO juju.worker.uniter context.go:255 HOOK "main", fname, loader, pkg_name)
ebu21516:2013-12-18 15:26:34 INFO juju.worker.uniter context.go:255 HOOK File "/usr/lib/python2.7/runpy.py", line 72, in run_code
ebu21516:2013-12-18 15:26:34 INFO juju.worker.uniter context.go:255 HOOK exec code in run_globals
ebu21516:2013-12-18 15:26:34 INFO juju.worker.uniter context.go:255 HOOK File "/var/lib/juju/agents/unit-oscied-orchestra-0/charm/library/oscied_lib/OrchestraHooks.py", line 306, in
ebu21516:2013-12-18 15:26:34 INFO juju.worker.uniter context.go:255 HOOK DEFAULT_OS_ENV).trigger()
ebu21516:2013-12-18 15:26:34 INFO juju.worker.uniter context.go:255 HOOK File "/usr/local/lib/python2.7/dist-packages/pytoolbox-6.0.4_beta-py2.7.egg/pytoolbox/juju.py", line 540, in trigger
ebu21516:2013-12-18 15:26:34 INFO juju.worker.uniter context.go:255 HOOK getattr(self, u'hook
{0}'.format(hook_name.replace(u'-', u'
')))()
ebu21516:2013-12-18 15:26:34 INFO juju.worker.uniter context.go:255 HOOK File "/var/lib/juju/agents/unit-oscied-orchestra-0/charm/library/oscied_lib/OrchestraHooks.py", line 210, in hook_config_changed
ebu21516:2013-12-18 15:26:34 INFO juju.worker.uniter context.go:255 HOOK chown(local_cfg.sites_directory, DAEMON_USER, DAEMON_GROUP, recursive=True)
ebu21516:2013-12-18 15:26:34 INFO juju.worker.uniter context.go:255 HOOK File "/usr/local/lib/python2.7/dist-packages/pytoolbox-6.0.4_beta-py2.7.egg/pytoolbox/filesystem.py", line 263, in chown
ebu21516:2013-12-18 15:26:34 INFO juju.worker.uniter context.go:255 HOOK os.chown(os.path.join(root, filename), uid, gid)
ebu21516:2013-12-18 15:26:34 INFO juju.worker.uniter context.go:255 HOOK OSError: [Errno 2] No such file or directory: '/var/www/webui/medias'

pytoolbox.network.smpte2022.base : 496

shouldn't the check in base.py for packet.sequence (line 496) be:
if packet.sequence != (fec.snbase + ifec.offset) % (RtpPacket.S_MASK + 1)
instead of
if packet.sequence != (fec.snbase + i
fec.offset) % RtpPacket.S_MASK ?

the latter check fails when the packet sequence number is 65535

Handle ffmpeg bad time values

[2014-03-17 10:17:29,584: ERROR/MainProcess] Task encodebox.tasks.transcode[d909578c-6014-46a7-8d5a-e7449b095153] raised unexpected: ValueError('invalid literal for float(): -577014:-32:-22.-77',)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/celery-3.1.9-py2.7.egg/celery/app/trace.py", line 238, in trace_task
R = retval = fun(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/celery-3.1.9-py2.7.egg/celery/app/trace.py", line 416, in protected_call
return self.run(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/encodebox-0.5.0_beta-py2.7.egg/encodebox/tasks.py", line 89, in transcode
for statistics in encoder_module.encode(transcode_pass[1], transcode_pass[2], transcode_pass[3]):
File "/usr/local/lib/python2.7/dist-packages/pytoolbox/ffmpeg.py", line 208, in encode
ratio = time_ratio(out_duration, in_duration)
File "/usr/local/lib/python2.7/dist-packages/pytoolbox/datetime.py", line 133, in time_ratio
ratio = total_seconds(numerator) / total_seconds(denominator)
File "/usr/local/lib/python2.7/dist-packages/pytoolbox/datetime.py", line 169, in total_seconds
return float(time)
ValueError: invalid literal for float(): -577014:-32:-22.-77

Fix basic Django deprecation warning in django.signals.dispatch

../../../../../../../tmp/.local/lib/python3.8/site-packages/pytoolbox/django/signals/dispatch.py:14
/tmp/.local/lib/python3.8/site-packages/pytoolbox/django/signals/dispatch.py:14: RemovedInDjango40Warning: The providing_args argument is deprecated. As it is purely documentational, it has no replacement. If you rely on this argument as documentation, you can move the text to a code comment or docstring.
super().init(providing_args, use_caching)
-- Docs: https://docs.pytest.org/en/stable/warnings.html

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.