Code Monkey home page Code Monkey logo

animethemes-dl's Introduction

animethemes-dl

GitHub GitHub code size in bytes GitHub last commit GitHub Release

what's this project

This project allows you to automaticaly download opening and ending songs from all of your favorite anime without the need of downloading everything yourself. Since almost every weeb uses MAL to track the anime he's watching, this tool is really useful, as every information you need to give it has been written down already. All you need to do is to enter your MAL or AniList username.

reminder

All videos are downloaded from animethemes.moe. If you plan on using this program just for looking at openings, I recommend using themes.moe or their own site instead. This program is made for creating your own playlist and such.

what's this project for

This project was made for batch downloading themes from anime you have watched, but is programmed so it's easily improved, making it possible to add very easily. It's made with both command line usage and with python as a module.

how to install

  • clone this repository from github.com or download it from pip with pip install animethemes-dl
  • if you cloned, do pip install -r requirements.txt to install all required modules
  • install ffmpeg into the same folder or in PATH

usage in command line

make sure you have ffmpeg and python installed To run in console use animethemes-dl if installed with pip. python -m animethemes-dl if you have cloned the repository.

These commands will be reffered to as animethemes-dl in the documentation.

command line documentation

The script should raise errors in case you pass in an improper arg, but sometimes an error won't be raised if the error is not obvious, therefore make sure you read the documentation before running it.

You must set a username and a save folder.

animelist

You must set a username. By default usernames are assumed to be a MAL user, you can use a different site with --site.

--animelist-args can be:

  • url args for MAL ``
  • query and variables for POST request for AniList

--animelist-args are passed as a <key>=<value> pairs, for example: sort1=1,sort2=14

animelist filters

There are filters for minimum score and priority.

--minscore is the minimum score between 0 and 10. --minpriority is the minimum priority. For mal, use Low=0,Medium=1,High=2 --range <start> <end> only gets a slice of the animelist.

tag filters

To download only openings or only endings, use --OP or --ED

By default, you can just use a --smart filter, that takes out all the dialogue. This works by removing all themes that contains a part of the episode and spoilers at the same time. This works 95% of the time.

Since animethemes can have a single song bound to multiple anime, --no-copy filters them.

You can set --banned-tags or --required-tags. These will take multiple tags, possible tags are:

Tag Meaning
spoiler Video contains spoilers.
nsfw Video is NSFW.
nc No captions/no credits.
subbed Video includes English subtitles of dialogue.
lyrics Video includes English lyrics as subtitles.
uncen Video does not have censorship.

You can set a --min-resolution, they show up in 420,720,1080.

You can set the required --source, possible sources are:

Source Meaning
BD Video is sourced from a Blu-ray disc.
DVD Video is sourced from a DVD.
Video is sourced from a TV release.

Some themes contain a part of the episode. You can set a --overlap to show only some overlaps.

Overlap Meaning
Over Part of episode is over the video.
Transition Part of episode transitions into the video.
None No dialogue in video.

If you're only looking to remove dialogue, transitions are fairly fine. They don't even have dialogue most of the time, I recommend just banning Over

download

Downloads are by default disabled for both video and audio. You can enable it by setting a save folder. Save folders are set with -a (audio) and -v (video).

The filename format can be changed with --filename.

The possible formats are defined in this table:

Format Meaning
anime_id Animethemes' id of anime.
anime_name Name of Anime.
anime_slug Animethemes' slug of anime.
anime_year Year the anime came out.
anime_season Season the anime came out.
theme_id Animethemes' id of theme.
theme_type Type of theme (OP/ED).
theme_sequence Sequence of theme.
theme_group Group of theme (e.g. language).
theme_slug Animethemes' slug of theme (type+sequence).
entry_id Animethemes' id of entry.
entry_version Version of entry ("" or 1+).
entry_notes Notes of entry (e.g. SFX version).
video_id Animethemes' id of video.
video_basename Animethemes' basename of video.
video_filename Basename without the filetype.
video_size Size of file in bytes.
video_resolution Resolution of video.
video_source Where the video was sourced from.
video_overlap Episode overlap over video.
song_id Animethemes' id of song.
song_title Title of song.
video_filetype Filetype of video.
anime_filename Name of anime used in filenames.

formats should be used as a python format string, meaning that it will be put as %(format)s. For example %(anime_filename)s-%(theme_slug)s.%(video_filetype)s.

Windows and Linux banned characters will be removed by default, to remove those and also unicode characters use --ascii

You can disable redownloading with -r. This is highly recommended. If you have downloaded video you can --update theme, this will check file validity by looking at the filesize. It will also update audio files if the video is downloaded.

You can add a coverart to audio files with --coverart, --coverart takes in a resolution, if set, image will be fetched from anilist.co, with high resolutions it's recommended to save them in --coverart-folder.

Downloader timeout can be changed with --timeout and max amount of retries with --retries.

Sometimes when using filters a video that you wanted gets filtered out. you can --force-videos and keep them this way.

re:zero for example has lots of unique EDs, but they often have an overlay, meaning smart filter will remove them.

Data from animethemes is sending a lot of requests at the same time, so to reduce stress on the servers, the data is saved in a temp folder. You can change it's max age with --max-cache-age.

statuses

You can download anime that you have --on-hold,--dropped or --planned.

compression

Downloaded files can be compressed in case you want to save them.

It will be enabled by setting a directory you want to compress with --compress-dir, this should be the same directory as you chosen one. The destination file is set with --compress-name, set it without the extension. You can choose the --compress-format, this must be a format allowed by shutils.make_archive.

Additionally you can set the --compress-base.

printing

You can set the loglevel with --loglevel. This will set the logger.setLevel(...). There are quick commands --quiet (print none) and --verbose (print all). To restrict download and ffmpeg messages, you MUST use --quiet.

You can disable color with --no-color.

utilities

In case you haven't added ffmpeg to path, you can set the path with --ffmpeg.

In case the mp3 tags are not showing, you can specify to --use-id3v23, that will allow support for older systems.

You can --repair in case the script made some errors or you picked wrong options. This will delete unexpected files and readd metadata.

settings

You can load options from a file with --options, the file is in json format.

The default options are:

{
    "animelist": {
        "username": "",
        "site": "MyAnimeList",
        "animelist_args": {},
        "minpriority": 0,
        "minscore": 0,
        "range": [0,0]
    },
    "filter": {
        "smart": false,
        "no_copy": false,
        "type": null,
        "spoiler": null,
        "nsfw": null,
        "resolution": 0,
        "nc": null,
        "subbed": null,
        "lyrics": null,
        "uncen": null,
        "source": null,
        "overlap": null
    },
    "download": {
        "filename": "%(anime_filename)s-%(theme_slug)s.%(video_filetype)s",
        "audio_folder": null,
        "video_folder": null,
        "no_redownload": false,
        "update": false,
        "ascii": false,
        "timeout": 5,
        "retries": 3,
        "max_cache_age": 10368000,
        "force_videos": []
    },
    "coverart": {
        "resolution": 0,
        "folder": null
    },
    "compression": {
        "root_dir": null,
        "base_name": "animethemes",
        "format": "tar",
        "base_dir": null
    },
    "statuses": [1,2],
    "quiet": false,
    "no_colors": false,
    "ffmpeg": "ffmpeg",
    "id3v2_version": 4,
    "ignore_prompts": false
}

You can generate the options with python -m animethemes_dl.options.

code documentation

The code uses the module models that contains models of typing.TypedDict. Meaning python 3.8 is required. Module parsers contains all parsers for MAL, Anilist and themes.moe. Module tools contains extra tools for animethemes-dl.

examples:

# parsers module uses API's to get data
import animethemes_dl.parsers as parsers
parsers.fetch_animethemes(username) # fetchess raw data
parsers.get_download_data(username) # gets download data

# models module uses typedDict to help linters
import animethemes_dl.models as models
animelist: AnimeThemeAnime = _myanimefunc()
metadata: Metadata = _mymetadatafunc()

# tools have multiple tools used for several stuff
import animethemes_dl.tools as tools
tools.ffmpeg_convert(webm_file,mp3_file) # converts a webm file
tools.COLORS['progress'] = Fore.CYAN # changes colors
tools.compress_files(base,'zip',root) # compresses a direcotory
tools.update_metadata(
  parsers.get_download_data(username),False
) # updates metadata of all audio files

# you can implement your own batch dl
import animethemes_dl
data = parsers.get_download_data(username)
for theme in data:
  animethemes_dl.download_theme(theme,True)

# you can directly change options
animethemes_dl.setOptions(options)

# you can make special catchers
import animethemes_dl.errors as errors
try:
  animethemes_dl.batch_download(data)
except FfmpegException:
  print('oh no')

how does it work?

  • parser
    • get data from MAL/AniList
    • get data from themes.moe
    • combine data
    • filter out unwanted themes
    • create download data
  • download
    • download video file
    • convert video to audio
      • convert with ffmpeg
      • add mp3 metadata
  • optional
    • compress files

TODO

  • code optimizations
  • improve code documentation
  • make a better README (too complicated rn)
  • concurrent downloads, since animethemes disabled multithreaded dl.
  • support for aria2c

animethemes-dl's People

Contributors

thesadru avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

animethemes-dl's Issues

Can't download anything

  • Windows 10
  • Python 3.8.0

I'm not very good at coding and I didn't really understand the documentation so I apologize in advance if I'm making some really obvious mistake.

I'm trying to download some openings from my anilist but when I run the command
animethemes-dl --al nya -v C:\Users\mateo\Documents\themes --smart --min-resolution 1080
it results with

Traceback (most recent call last):
  File "C:\Users\mateo\AppData\Local\Programs\Python\Python38-32\Scripts\animethemes-dl-script.py", line 11, in <module>
    load_entry_point('animethemes-dl==2.2.2.1', 'console_scripts', 'animethemes-dl')()
  File "C:\Users\mateo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pkg_resources\__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "C:\Users\mateo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pkg_resources\__init__.py", line 2852, in load_entry_point
    return ep.load()
  File "C:\Users\mateo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pkg_resources\__init__.py", line 2443, in load
    return self.resolve()
  File "C:\Users\mateo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pkg_resources\__init__.py", line 2449, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "C:\Users\mateo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\animethemes_dl\__init__.py", line 4, in <module>
    from .downloader import *
  File "C:\Users\mateo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\animethemes_dl\downloader.py", line 14, in <module>
    from .parsers import get_download_data
  File "C:\Users\mateo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\animethemes_dl\parsers\__init__.py", line 4, in <module>
    from .animelist import ANIMELISTSITES ,AniList, MyAnimeList
  File "C:\Users\mateo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\animethemes_dl\parsers\animelist.py", line 18, in <module>
    from .utils import Measure, add_url_kwargs
  File "C:\Users\mateo\AppData\Local\Programs\Python\Python38-32\lib\site-packages\animethemes_dl\parsers\utils.py", line 9, in <module>
    def add_url_kwargs(url: str, kwargs: dict[str,Any]) -> str:
TypeError: 'type' object is not subscriptable

I'm not sure if my syntax is wrong, I've also tried to downloading openings using different parameters and even tried fixing it by installing it using git clone but they all result in the same Traceback (most recent call last): thing

Any help would be appreciated.

Cannot reach animethemes.dev

Para has announced in the animethemes' discord server that they will be using staging.animethemes.moe from now on.

I am aware of this issue and will fix it in a couple hours.

cannot download, animethemes is down

  • Platform: Windows 10 (also tested on Manjaro Linux and Repl.it)
  • Python Version: 3.9.0
  • Extra Workaraounds: Remove the #!/usr/bin/env python line from main.py as this caused it to fail to import pySmartDl

EDIT: After some debugging in PyCharm it appears that the issue is when SmartDL() is called on line 227 in downloader.py


I'm attempting to download all themes from my Anilist, using the following command:
py .\main.py --al Yellowsink -v C:\Users\caina\Documents\MEGA\Vids\Anime_OPs
however it throws an exception and doesn't work:

[progress] initializing program
[get] getting data from anilist.co | 279ms
[get] getting data from themes.moe | 966ms
[parse] getting download data
[progress] started downloading
[download] Steins;Gate OP (Hacking to the Gate).webm
Traceback (most recent call last):
  File "C:\Users\caina\source\repos\animethemes-dl\main.py", line 163, in <module>
    batch_download(
  File "C:\Users\caina\source\repos\animethemes-dl\downloader.py", line 422, in batch_download
    download_multi_theme(download_data,webm_folder,mp3_folder)
  File "C:\Users\caina\source\repos\animethemes-dl\downloader.py", line 402, in download_multi_theme
    filename = download_chooser(mthd)
  File "C:\Users\caina\source\repos\animethemes-dl\downloader.py", line 381, in <lambda>
    download_theme(
  File "C:\Users\caina\source\repos\animethemes-dl\downloader.py", line 227, in download_theme
    obj = SmartDL(theme_data["mirrors"],filename,progress_bar=not Opts.Print.quiet)
  File "C:\Users\caina\AppData\Local\Programs\Python\Python39\lib\site-packages\pySmartDL\pySmartDL.py", line 136, in __init__
    if not utils.is_HTTPRange_supported(self.url, timeout=self.timeout):
  File "C:\Users\caina\AppData\Local\Programs\Python\Python39\lib\site-packages\pySmartDL\utils.py", line 100, in is_HTTPRange_supported
    fullsize = get_filesize(url, timeout=timeout)
  File "C:\Users\caina\AppData\Local\Programs\Python\Python39\lib\site-packages\pySmartDL\utils.py", line 127, in get_filesize
    urlObj = urllib.request.urlopen(url, timeout=timeout)
  File "C:\Users\caina\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\caina\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 517, in open
    response = self._open(req, data)
  File "C:\Users\caina\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "C:\Users\caina\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 494, in _call_chain
    result = func(*args)
  File "C:\Users\caina\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1385, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "C:\Users\caina\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1346, in do_open
    r = h.getresponse()
  File "C:\Users\caina\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1347, in getresponse
    response.begin()
  File "C:\Users\caina\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "C:\Users\caina\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

Can't download: KeyError: Resource

Hello, I don't know Python well so I've been trying to figure it out and fix the problem myself but am super stuck.

Using: Windows 10, Python 3.9.5

animethemes-dl alpha_cyberus -a C:\Users\petad\Music\Anime --range 1 2 --coverart
(Put in the range because I want to test that it works with a single show before pulling the music out of a whole block of my list. The anime its looking at is Cestvs: The Roman Fighter)

Traceback (most recent call last):
File "C:\Python39\Scripts\animethemes-dl-script.py", line 35, in
sys.exit(load_entry_point('animethemes-dl==2.2.2.4', 'console_scripts', 'animethemes-dl')())
File "c:\python39\lib\site-packages\animethemes_dl\main.py", line 444, in main
batch_download(options)
File "c:\python39\lib\site-packages\animethemes_dl\downloader.py", line 158, in batch_download
data = get_download_data(
File "c:\python39\lib\site-packages\animethemes_dl\parsers\dldata.py", line 258, in get_download_data
raw = get_animethemes(username, site, **animelist_args)
File "c:\python39\lib\site-packages\animethemes_dl\parsers\parser.py", line 22, in get_animethemes
return fetch_animethemes(titles)
File "c:\python39\lib\site-packages\animethemes_dl\parsers\animethemes.py", line 145, in fetch_animethemes
anime = ft_anime.result()
File "c:\python39\lib\concurrent\futures_base.py", line 445, in result
return self.__get_result()
File "c:\python39\lib\concurrent\futures_base.py", line 390, in __get_result
raise self._exception
File "c:\python39\lib\concurrent\futures\thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "c:\python39\lib\site-packages\animethemes_dl\parsers\animethemes.py", line 83, in fetch_anime
anime = verify_anime(data['search']['anime'],alid,alsite)
File "c:\python39\lib\site-packages\animethemes_dl\parsers\animethemes.py", line 67, in verify_anime
for resource in anime['resources']:
KeyError: 'resources'

If I understand the code right, it's supposed to connect to the Resource API endpoint of animethemes.moe but when I look at the returned value in animethemes.py line 51 r = session.get(URL.format(title)) using print(r.text), it returns the following dictionaries: search{ anime, artists, entries, series, songs, synonyms, themes, videos}. I don't know if Resources is meant to have been returned as part of this dictionary or as another one?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.