Code Monkey home page Code Monkey logo

file_hash's Introduction

file_hash

File hasing plugin for Flexget

Notes

  • While it may work on other versions, I only will support Python 3.6+
  • I have decided that as the depreciation of Python 2.7 is a little over a year away, I will not be bothering to support it.
  • I have plans to add support for asyncio
  • HEY, LISTEN: Even though stop and time are in the schema, they are not implemented yet.

Usage

  • The default hashing method depends on your python installation:
  • If you have Python 3.6+, it will use BLAKE2b
  • Otherwise, it will use MD5
  • The list of usable methods is determined by hashlib.algorithms_available
  • To use the default hashing method:
  • file_hash: yes
  • To use a custom hashing method (for example sha1):
  • file_hash: sha1
  • You may choose MAX 2 of the following options: size, start, stop.
  • Using all three of these together sets up a chance that the difference between start and stop could be either smaller or larger than size.
  • If the start position is larger than size of the file, one of two things will happen:
  1. If the size of the file is smaller than size, start is set to 0, OR
  2. If the size of the file is larger than size (but still smaller than start), start will be set to the size of the file minus size
  • To use advanced options, see below

Example

templates:
  
  file_hash_basic:
    <<: *any-file-input-plugin
    file_hash: yes
    
  file_hash_basic_extended: # These are the settings when you use "file_hash: yes"
    <<: *any-file-input-plugin
    file_hash:
      algorithm: blake2b # Or if your system doesn't have blake2b, this will be md5
      size: 25
      start: 50
    
  file_hash_algorithm:
    <<: *any-file-input-plugin
    file_hash: sha1
    
  file_hash_advanced:
    <<: *any-file-input-plugin
    file_hash:
      algorithm: sha256 # Optional, default will be chosen if this is not set
      size: 1 # Will hash 1MiB of the given file
      start: 25 # Will start at 25MiB into the file, see usage for how this applies to files smaller than this value

file_hash's People

Watchers

avatar avatar

file_hash's Issues

Memory Error

Need to split files into digestible chunks. 100MB maybe? Configurable, that's it.

2018-10-27 03:59 VERBOSE  file_hash     collect_complete_files Hasing /media/downloads/seeding/[HorribleSubs] Akkun to Kanojo - 20 [720p].mkv
2018-10-27 03:59 VERBOSE  file_hash     collect_complete_files Hasing with algorithm: blake2b
2018-10-27 03:59 DEBUG    file_hash     collect_complete_files [HorribleSubs] Akkun to Kanojo - 20 [720p]:\nhash: f6cd5737dae643cdbded8cec55c79c632019b2d75f164b2fae5fe33f9814ccd07f31e84c7adf7f96debe356b4ce6c048d9439749032ba996ce990a493f8d4ba9\nmtime: 1534519974.389479\nbyes: 48651507
2018-10-27 03:59 VERBOSE  file_hash     collect_complete_files Hasing /media/downloads/seeding/[Erai-raws] Shoujo Kageki Revue Starlight - 11 [720p][Multiple Subtitle].mkv
2018-10-27 03:59 VERBOSE  file_hash     collect_complete_files Hasing with algorithm: blake2b
2018-10-27 03:59 DEBUG    file_hash     collect_complete_files [Erai-raws] Shoujo Kageki Revue Starlight - 11 [720p][Multiple Subtitle]:\nhash: 5c383ed70ebb98894b831008ece74703c6d42ac0e206258af20b76387c9d7e8db1535d5146c61cfd132f092a9ab07899e043e460ab3ac92a3e445340e12665b8\nmtime: 1538625677.7178843\nbyes: 147434571
2018-10-27 03:59 VERBOSE  file_hash     collect_complete_files Hasing /media/downloads/seeding/[HA／ss／] Sunohara-sou no Kanrinin-san - 10 [C0BBA196].mkv
2018-10-27 03:59 VERBOSE  file_hash     collect_complete_files Hasing with algorithm: blake2b
2018-10-27 03:59 DEBUG    file_hash     collect_complete_files [HA／ss／] Sunohara-sou no Kanrinin-san - 10 [C0BBA196]:\nhash: 2b411e4737da6438d384e3316b9bfb2bf9427b73c39f9046d85447ada3455724714d16d89721fd75e7b1978f4cbd163f34976e842fbea6a4b8ff93fbc901af73\nmtime: 1536455754.0\nbyes: 222315361
2018-10-27 03:59 VERBOSE  file_hash     collect_complete_files Hasing /media/downloads/seeding/Its.Always.Sunny.in.Philadelphia.S13E05.The.Gang.Gets.New.Wheels.1080p.AMZN.WEBRip.DDP5.1.x264-NTb[rarbg]/RARBG.txt
2018-10-27 03:59 VERBOSE  file_hash     collect_complete_files Hasing with algorithm: blake2b
2018-10-27 03:59 DEBUG    file_hash     collect_complete_files RARBG:\nhash: c3103d005bf5ebe9222e9a7ac9136cb97e2075f373d56ddaee7f29137b6e843afe1dc883888fd7485a89116928e156f36f4256633f94559e1b28b9d2581b6c96\nmtime: 1534391981.3110278\nbyes: 30
2018-10-27 03:59 VERBOSE  file_hash     collect_complete_files Hasing /media/downloads/seeding/Its.Always.Sunny.in.Philadelphia.S13E05.The.Gang.Gets.New.Wheels.1080p.AMZN.WEBRip.DDP5.1.x264-NTb[rarbg]/Its.Always.Sunny.in.Philadelphia.S13E05.The.Gang.Gets.New.Wheels.1080p.AMZN.WEB-DL.DDP5.1.H.264-NTb.mkv
2018-10-27 03:59 VERBOSE  file_hash     collect_complete_files Hasing with algorithm: blake2b
2018-10-27 03:59 WARNING  apscheduler.scheduler                 Execution of job "download_anime (trigger: interval[0:20:00], next run at: 2018-10-27 03:59:28 EDT)" skipped: maximum number of running instances reached (1)
2018-10-27 03:59 DEBUG    file_hash     collect_complete_files Its.Always.Sunny.in.Philadelphia.S13E05.The.Gang.Gets.New.Wheels.1080p.AMZN.WEB-DL.DDP5.1.H.264-NTb:\nhash: 98729201d036c64b9b1ab8b7bfee6209be357c4c2612a52214bef50719e67f800482243d49cce01012e6a5ecc4eae9443b8ca66c3e8694f2257233e3b4128c22\nmtime: 1538643698.1400626\nbyes: 1895645086
2018-10-27 03:59 VERBOSE  file_hash     collect_complete_files Hasing /media/downloads/seeding/The.Hateful.Eight.2015.1080p.BluRay.REMUX.AVC.DTS-HD.MA.5.1-RARBG/The.Hateful.Eight.2015.1080p.BluRay.REMUX.AVC.DTS-HD.MA.5.1-RARBG.nfo
2018-10-27 03:59 VERBOSE  file_hash     collect_complete_files Hasing with algorithm: blake2b
2018-10-27 03:59 DEBUG    file_hash     collect_complete_files The.Hateful.Eight.2015.1080p.BluRay.REMUX.AVC.DTS-HD.MA.5.1-RARBG:\nhash: 95b6090cde79ffccefa60f39a268d2342230c761cc0aa35c6a8926480344b509caf48830e8cd164470a56f9a3799b2eaaca5dda967c3ec87e9ea0399e0843263\nmtime: 1536632804.9134138\nbyes: 7942
2018-10-27 03:59 VERBOSE  file_hash     collect_complete_files Hasing /media/downloads/seeding/The.Hateful.Eight.2015.1080p.BluRay.REMUX.AVC.DTS-HD.MA.5.1-RARBG/RARBG.com.txt
2018-10-27 03:59 VERBOSE  file_hash     collect_complete_files Hasing with algorithm: blake2b
2018-10-27 03:59 DEBUG    file_hash     collect_complete_files RARBG.com:\nhash: af3e6852dc29c6326f5e6b44b1ae39707c426f586f74d4bfc4f4567ca2cb758dc3f7427a74f18866572d9f18f364193dd8c88d8715d6de248a08daace2ba1dea\nmtime: 1538431057.9458773\nbyes: 34
2018-10-27 03:59 VERBOSE  file_hash     collect_complete_files Hasing /media/downloads/seeding/The.Hateful.Eight.2015.1080p.BluRay.REMUX.AVC.DTS-HD.MA.5.1-RARBG/The.Hateful.Eight.2015.1080p.BluRay.REMUX.AVC.DTS-HD.MA.5.1-RARBG.mkv
2018-10-27 03:59 VERBOSE  file_hash     collect_complete_files Hasing with algorithm: blake2b
2018-10-27 03:59 CRITICAL task          collect_complete_files BUG: Unhandled error in plugin file_hash: 
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/flexget/task.py", line 486, in __run_plugin
    return method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/flexget/event.py", line 23, in __call__
    return self.func(*args, **kwargs)
  File "/config/plugins/file_hash/file_hash.py", line 56, in on_task_metainfo
    current_hasher.update(to_hash.read())
MemoryError

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.