Code Monkey home page Code Monkey logo

failpool's Introduction

FailPool

A mod to the cpython multiprocessing pool class that terminates the pool when an error occurs.

A progress bar with ETA will be written to stdout on pool join, or a job total to the logger. Pool totals will not be completely accurate because the job totals aren't tracked internally by Pool.

Keeping up to date

Since this is a mod against cpython internals, it can be risky to run on unchecked versions.

The most important part is that the _timeout_join functions does the same as join, but with timeouts.

TODO

  1. Reevaluate progress bars, imports, etc
  2. Enable user-provided error_callback and just call mine after
  3. Make better
  4. Make pool kill optional while keeping timeout join (because pool can lock up otherwise!)[https://bugs.python.org/issue31782]
  5. Check on FailThreadPoolExecutor becasue it probably works, buuuuut...

FailPool

When an exception is raised, the pool will be terminated and the exception re-raised.

Note that specific submit functions will need to be overridden before use.

FailThreadPoolExecutor

When an exception is raised, the work queue will be purged and the pool shutdown. The exception will then be re-raised.

failpool's People

Contributors

nivintw avatar vilhelmen avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.