lidarr / lidarrapi.metadata Goto Github PK

This hosts the custom metadata API and resources Lidarr relies on.

Python 88.47% HTML 0.08% Dockerfile 0.40% Shell 8.69% JavaScript 1.94% Nix 0.40% Rust 0.03%

lidarrapi.metadata's Introduction

Lidarr

Lidarr is a music collection manager for Usenet and BitTorrent users. It can monitor multiple RSS feeds for new tracks from your favorite artists and will grab, sort and rename them. It can also be configured to automatically upgrade the quality of files already downloaded when a better quality format becomes available.

Major Features Include:

Support for major platforms: Windows, Linux, macOS, Raspberry Pi, etc.
Automatically detects new tracks.
Can scan your existing library and download any missing tracks.
Can watch for better quality of the tracks you already have and do an automatic upgrade.
Automatic failed download handling will try another release if one fails
Manual search so you can pick any release or to see why a release was not downloaded automatically
Fully configurable track renaming
Full integration with SABnzbd and NZBGet
Full integration with Kodi, Plex (notification, library update, metadata)
Full support for specials and multi-album releases
And a beautiful UI

Support

Note: GitHub Issues are for Bugs and Feature Requests Only

Contributors

This project exists thanks to all the people who contribute. Contribute.

Backers

Thank you to all our backers! 🙏 Become a backer

Mega Sponsors

JetBrains

Thank you to JetBrains for providing us with free licenses to their great tools.

DigitalOcean

This project is also supported by DigitalOcean

License

GNU GPL v3
Copyright 2010-2021

lidarrapi.metadata's People

Stargazers

Watchers

Forkers

ta264 wlampe webmosher ombre8 ganonmaster pearsonflyer 6cubi57z texke

lidarrapi.metadata's Issues

Sentry updates

Upgrade sentry from raven to unified sdk
Rate limit queries
Group same error
Remove DSN from default config

ProgrammingError: syntax error at or near "AND"

https://sentry.io/lidarr/metadata-api/issues/514008365/

ProgrammingError: syntax error at or near "AND"
LINE 36: ...ating_count * release_group_meta.rating) END DESC AND UPPER(...
                                                              ^

(5 additional frame(s) were not displayed)
...
  File "lidarrmetadata/api.py", line 211, in chart_route
    return jsonify(charts[key](count, **chart_kwargs))

ProgrammingError: syntax error at or near "AND"
LINE 36: ...ating_count * release_group_meta.rating) END DESC AND UPPER(...
                                                              ^

Caused by changes in #51.

Add country code as parameter or layer for charts

Allow passing country code so that itunes/apple charts can be pulled based on country preference.

API stops working and throws 502 status

ValueError: No JSON object could be decoded

https://sentry.io/lidarr/metadata-api/issues/399475744/

ValueError: No JSON object could be decoded
(10 additional frame(s) were not displayed)
...
  File "lidarrmetadata/api.py", line 358, in search_route
    return search_artist()
  File "lidarrmetadata/api.py", line 346, in search_artist
    artist['Images'] = artist_art_providers[0].get_artist_images(artist['Id'])

ValueError: No JSON object could be decoded

Use normal MediaWikiAPI package

There is still a dependency link to my fork of MediaWikiAPI. Those changed have been merged upstream so we can use that package.

https://github.com/lidarr/LidarrAPI.Metadata/blob/e8abebcfb3fdbec8fa2bc53d4ad5bd3a5bef8366/setup.py#L52

Use release titles instead of release group titles

In some cases, releases can have different names than the release group, such as https://musicbrainz.org/release-group/20292abb-7023-495b-8081-4b7be106a3b1

Prioritize artist in search if query matches Sort Name

Add CASE WHEN to_tsvector('mb_simple', artist.sort_name) = to_tsvector('mb_simple', %s) THEN 0 ELSE 1 END, right after artist.name.

ValueError: No JSON object could be decoded

https://sentry.io/lidarr/metadata-api/issues/477218613/

ValueError: No JSON object could be decoded
(9 additional frame(s) were not displayed)
...
  File "lidarrmetadata/api.py", line 114, in get_artist_info
    artist['Images'] = artist_art_providers[0].get_artist_images(mbid)

ValueError: No JSON object could be decoded

MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk

Hi,

I'm currently running lidarr in a synology NAS using a docker image from linuxserver/lidarr. This has been working great until lately I've been hitting the following error when adding an artist or refreshing metadata

[v0.4.0.524] NzbDrone.Common.Http.HttpException: HTTP request failed: [500:InternalServerError] [GET] at [https://api.lidarr.audio/api/v0.3/album/28f24dc6-8fd8-3635-a1cb-5a394fcf542d?release=]
  at NzbDrone.Core.MetadataSource.SkyHook.SkyHookProxy.GetAlbumInfo (System.String foreignAlbumId, System.String releaseId) [0x000ce] in C:\projects\lidarr\src\NzbDrone.Core\MetadataSource\SkyHook\SkyHookProxy.cs:132 
  at NzbDrone.Core.Music.AddAlbumService.AddSkyhookData (NzbDrone.Core.Music.Album newAlbum) [0x00000] in C:\projects\lidarr\src\NzbDrone.Core\Music\AddAlbumService.cs:77 
  at NzbDrone.Core.Music.AddAlbumService.AddAlbums (System.Collections.Generic.List`1[T] newAlbums) [0x0001d] in C:\projects\lidarr\src\NzbDrone.Core\Music\AddAlbumService.cs:58 
  at NzbDrone.Core.Music.RefreshArtistService.RefreshArtistInfo (NzbDrone.Core.Music.Artist artist, System.Boolean forceAlbumRefresh) [0x0026e] in C:\projects\lidarr\src\NzbDrone.Core\Music\RefreshArtistService.cs:132 
  at NzbDrone.Core.Music.RefreshArtistService.Execute (NzbDrone.Core.Music.Commands.RefreshArtistCommand message) [0x00043] in C:\projects\lidarr\src\NzbDrone.Core\Music\RefreshArtistService.cs:178 
{"error":"MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error."}

Any idea on how to fix this?

Thanks

IndexError: list index out of range, metadata doesn’t handle mbid that are not valid

https://sentry.io/lidarr/metadata-api/issues/355089233/

IndexError: list index out of range
(3 additional frame(s) were not displayed)
...
  File "lidarrmetadata/api.py", line 70, in get_artist_info
    artist = artist_providers[0].get_artist_by_id(mbid)

IndexError: list index out of range

DataError: invalid input syntax for uuid: "e132d370-2a59-4437-8610-756df28a5a0"

Should validate UUID and return 4xx error if invalid.

https://sentry.io/lidarr/metadata-api/issues/355085931/

DataError: invalid input syntax for uuid: "e132d370-2a59-4437-8610-756df28a5a0"
LINE 9: WHERE artist.gid = 'e132d370-2a59-4437-8610-756df28a5a0'
                           ^

(5 additional frame(s) were not displayed)
...
  File "lidarrmetadata/api.py", line 67, in get_artist_info
    artist = artist_providers[0].get_artist_by_id(mbid)

DataError: invalid input syntax for uuid: "e132d370-2a59-4437-8610-756df28a5a0"
LINE 9: WHERE artist.gid = 'e132d370-2a59-4437-8610-756df28a5a0'
                           ^

Add genres

For Lidarr/Lidarr#426.

ValueError: A string literal cannot contain NUL (0x00) characters.

Clean query string of illegal characters and add tests.

https://sentry.io/lidarr/metadata-api/issues/650086160/

ValueError: A string literal cannot contain NUL (0x00) characters.
(5 additional frame(s) were not displayed)
...
  File "lidarrmetadata/api.py", line 388, in search_route
    return search_artist()
  File "lidarrmetadata/api.py", line 326, in search_artist
    artists = search_providers[0].search_artist_name(query, limit=limit)

ValueError: A string literal cannot contain NUL (0x00) characters.

AttributeError: 'NoneType' object has no attribute 'group'

https://sentry.io/lidarr/metadata-api/issues/419753825/

AttributeError: 'NoneType' object has no attribute 'group'
(6 additional frame(s) were not displayed)
...
  File "lidarrmetadata/api.py", line 259, in search_route
    return search_artist()
  File "lidarrmetadata/api.py", line 239, in search_artist
    wikipedia_links[0]['target'])

AttributeError: 'NoneType' object has no attribute 'group'

OperationalError: could not connect to server: Connection refused

https://sentry.io/lidarr/lidarr-api-metadata/issues/469515531/?referrer=github_plugin

OperationalError: could not connect to server: Connection refused
	Is the server running on host "musicbrainz" (172.255.0.20) and accepting
	TCP/IP connections on port 5432?

(9 additional frame(s) were not displayed)
...
  File "lidarrmetadata/api.py", line 101, in get_artist_info
    artist = artist_providers[0].get_artist_by_id(mbid)

OperationalError: could not connect to server: Connection refused
	Is the server running on host "musicbrainz" (172.255.0.20) and accepting
	TCP/IP connections on port 5432?

Cache expiration headers

Add headers to control cloudflare cache expiration.

https://support.cloudflare.com/hc/en-us/articles/115003206852-Origin-Cache-Control

API caching doesn't care about version

API caching will cache the same key for different versions, so updates aren't reflected. In particular, testing responses can be given on production servers. Should be fixed once application roots are appropriately set.

Encoding issues

Issues with encodings with artists that have non-ascii characters in their or their albums' names, such as https://musicbrainz.org/artist/93bd1037-d736-42ca-93c3-c397f2cb1165

List Matching: BTS

Lidarr never seems to correctly match BTS from lists resulting in random artists being added. I assume this is due to their non-English name as default in Musicbrainz.

https://beta.musicbrainz.org/artist/0d79fe8e-ba27-4859-bb8c-2f255f346853

Add Cloudflare free tier in front of API

Add cloudflare with 15min cache in front of api to reduce hits

Process wikidata links

Musicbrainz is moving away from wikipedia links to wikidata. See https://blog.musicbrainz.org/2015/05/08/downstream-wikipedia-link-usage-and-migration-to-wikidata/.

SSLError: HTTPSConnectionPool(host='en.wikipedia.org', port=443): Max retries exceeded with url: /w/api.php...

https://sentry.io/lidarr/metadata-api/issues/436386164/

SSLError: HTTPSConnectionPool(host='en.wikipedia.org', port=443): Max retries exceeded with url: /w/api.php?inprop=url&redirects=&format=json&ppprop=disambiguation&prop=info%7Cpageprops&titles=Alesha_Dixon&action=query (Caused by SSLError(SSLEOFError(8, u'EOF occurred in violation of protocol (_ssl.c:590)'),))
(19 additional frame(s) were not displayed)
...
  File "lidarrmetadata/api.py", line 105, in get_artist_info
    wikipedia_links[0]['target'])

SSLError: HTTPSConnectionPool(host='en.wikipedia.org', port=443): Max retries exceeded with url: /w/api.php?inprop=url&redirects=&format=json&ppprop=disambiguation&prop=info%7Cpageprops&titles=Alesha_Dixon&action=query (Caused by SSLError(SSLEOFError(8, u'EOF occurred in violation of protocol (_ssl.c:590)'),))

Add Album search route and Album info route

IndexError: list index out of range

https://sentry.io/lidarr/metadata-api/issues/520657192/

IndexError: list index out of range
(11 additional frame(s) were not displayed)
...
  File "lidarrmetadata/api.py", line 108, in get_artist_info
    wikipedia_links[0]['target'])

IndexError: list index out of range

Old caching used for fanart requests

util.Cache is still used for fanart request caching. This could cause images to be cached until the server is restarted along with continually increasing memory usage. Check for usages of the class linked below.

https://github.com/lidarr/LidarrAPI.Metadata/blob/e8abebcfb3fdbec8fa2bc53d4ad5bd3a5bef8366/lidarrmetadata/util.py#L128

AssertionError: This shouldn't happen. Please report on GitHub: github.com/goldsmith/Wikipedia

https://sentry.io/lidarr/metadata-api/issues/429861202/

AssertionError: This shouldn't happen. Please report on GitHub: github.com/goldsmith/Wikipedia
(9 additional frame(s) were not displayed)
...
  File "lidarrmetadata/api.py", line 115, in get_artist_info
    wikipedia_links[0]['target'])

AssertionError: This shouldn't happen. Please report on GitHub: github.com/goldsmith/Wikipedia

Handle translation for special mb characters

Should handle and cross over characters that are special to musicbrainz, ex. "-" example

This bit from headphones may be helpful here:

_XLATE_SPECIAL = {
    # Translation table.
    # Cover additional special characters processing normalization.
    u"'": '',         # replace apostrophe with nothing
    u"’": '',         # replace musicbrainz style apostrophe with nothing
    u'&': ' and ',     # expand & to ' and '
}

_XLATE_MUSICBRAINZ = {
    # Translation table for Musicbrainz.
    u"…": '...',     # HORIZONTAL ELLIPSIS (U+2026)
    u"’": "'",       # APOSTROPHE (U+0027)
    u"‐": "-",       # EN DASH (U+2013)
    u'\u8208','-',   # HYPHEN (U+8208)
    u'\u8243','”',   # DOUBLE PRIME (U+8243)
}

AttributeError: 'module' object has no attribute 'extract_items'

https://sentry.io/lidarr/metadata-api/issues/442440142/

AttributeError: 'module' object has no attribute 'extract_items'
(3 additional frame(s) were not displayed)
...
  File "lidarrmetadata/api.py", line 208, in chart_route
    return jsonify(charts[key](count, **chart_kwargs))

AttributeError: 'module' object has no attribute 'extract_items'

Due to change in pylast at some point. Should specify dependency versions.

ConnectionError: HTTPSConnectionPool(host='en.wikipedia.org', port=443): Max retries exceeded with url: /w/api.php...

https://sentry.io/lidarr/lidarr-api-metadata/issues/786046421/?referrer=github_plugin

ConnectionError: HTTPSConnectionPool(host='en.wikipedia.org', port=443): Max retries exceeded with url: /w/api.php?inprop=url&redirects=&format=json&prop=info%7Cpageprops&titles=Zatox&action=query (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f398b48dad0>: Failed to establish a new connection: [Errno 101] Network unreachable',))
(17 additional frame(s) were not displayed)
...
  File "lidarrmetadata/api.py", line 125, in get_artist_info
    wikipedia_links[0]['target'])

ConnectionError: HTTPSConnectionPool(host='en.wikipedia.org', port=443): Max retries exceeded with url: /w/api.php?inprop=url&redirects=&format=json&prop=info%7Cpageprops&titles=Zatox&action=query (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f398b48dad0>: Failed to establish a new connection: [Errno 101] Network unreachable',))

Cache expiration for new releases

Cached responses may be returned after an artist has had a new release. Purge cache of artist responses when there are updates. Maybe scheduled task?

Missing overviews

Missing overviews for artists with valid wikipedia links, such as 93bd1037-d736-42ca-93c3-c397f2cb1165 and 4f93401e-f3dd-4961-8b4e-708be531c3ce. Seems that MediaWikiAPI either doesn't like encoded names or doesn't like non en.wikipedia sites.

psycopg2 package name changing

UserWarning: The psycopg2 wheel package will be renamed from release 2.8;
in order to keep installing from binary please use "pip install psycopg2-binary" instead.
For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.

ConnectionError: HTTPSConnectionPool(host='webservice.fanart.tv', port=443): Max retries exceeded with url: /v3/mu...

https://sentry.io/lidarr/metadata-api/issues/565184550/

ConnectionError: HTTPSConnectionPool(host='webservice.fanart.tv', port=443): Max retries exceeded with url: /v3/music/9fda77e4-8235-4f5d-a05c-3657bb550971/?api_key=651421d4a26a950283600e66a2a727fe (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f2065728090>: Failed to establish a new connection: [Errno 110] Operation timed out',))
(12 additional frame(s) were not displayed)
...
  File "lidarrmetadata/api.py", line 390, in search_route
    return search_artist()
  File "lidarrmetadata/api.py", line 356, in search_artist
    artist['Images'] = artist_art_providers[0].get_artist_images(artist['Id'])

ConnectionError: HTTPSConnectionPool(host='webservice.fanart.tv', port=443): Max retries exceeded with url: /v3/music/9fda77e4-8235-4f5d-a05c-3657bb550971/?api_key=651421d4a26a950283600e66a2a727fe (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f2065728090>: Failed to establish a new connection: [Errno 110] Operation timed out',))

Config does not parse types correctly

Config setting for HTTP_PORT results in string, suggesting that config doesn't parse original type.

The config entry 'server.socket_port' in section 'server.socket_port' is of type 'str', which does not match the expected type 'int'.

[28/Feb/2018:23:10:36] ENGINE Started monitor thread 'Autoreloader'.
[28/Feb/2018:23:10:36] ENGINE Error in HTTP server: shutting down
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/cherrypy/process/servers.py", line 225, in _start_http_thread
    self.httpserver.start()
  File "/usr/local/lib/python2.7/site-packages/cheroot/server.py", line 1516, in start
    self.bind(af, socktype, proto)
  File "/usr/local/lib/python2.7/site-packages/cheroot/server.py", line 1602, in bind
    self.socket.bind(self.bind_addr)
  File "/usr/local/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
TypeError: an integer is required

Don't cache api info

API info at the root route should not be included in cloudflare caching.

Searching for some bands don't seem to work.

Musicbrainz url https://musicbrainz.org/artist/7f41ac4b-9359-4f77-93ec-7c404f727f40.
https://musicbrainz.org/artist/93e747b8-2896-439d-8c46-29158e8cff90
https://musicbrainz.org/recording/3cae81cd-8863-4eea-b7f8-9260bca34382
https://musicbrainz.org/artist/1f9df192-a621-4f54-8850-2c5373b7eac9

Tracks in album response reference artists that are not in the artists list

See Lidarr/Lidarr#572

Examples:

https://api.lidarr.audio/api/v0.4/album/1ac6af64-25af-3ccb-a913-a7890ae8035c track 82c3422e-204e-39ce-8ba6-0482a5196374
https://api.lidarr.audio/api/v0.4/album/f3588439-a12d-37a1-a5d1-638538e7b2bd track d5007ed8-5584-4049-b733-60cd43ce4a95
https://api.lidarr.audio/api/v0.4/album/549fc6c2-fbe7-3e9e-8e99-3755d55ca9f9 track ca217467-179f-3f7a-80db-e43b4d941623

Request hangs on db connection error

Should time out if behind nginx. Add timeout to db interactions.

IndexError: list index out of range

https://sentry.io/lidarr/metadata-api/issues/525756003/

IndexError: list index out of range
(12 additional frame(s) were not displayed)
...
  File "lidarrmetadata/api.py", line 355, in search_route
    return search_artist()
  File "lidarrmetadata/api.py", line 335, in search_artist
    wikipedia_links[0]['target'])

IndexError: list index out of range

Release sorting is incorrect

Release sorting by date is incorrect due to storing in a dict instead of OrderedDict.

IOError: [Errno 2] No such file or directory: '../sql/artist_search_name.sql'

https://sentry.io/lidarr/metadata-api/issues/347214986/

IOError: [Errno 2] No such file or directory: '../sql/artist_search_name.sql'
(5 additional frame(s) were not displayed)
...
  File "lidarrmetadata/api.py", line 208, in search_route
    return search_artist()
  File "lidarrmetadata/api.py", line 182, in search_artist
    artists = search_providers[0].search_artist_name(query)

IOError: [Errno 2] No such file or directory: '../sql/artist_search_name.sql'

Overview field not in responses without overview

Should return overview field with either null or empty string.

Should trim leading or trailing spaces from search route

See title

Return Rating for Albums and Artists

Return a rating and number votes for album call and artist call. Possibly use metacritic or maybe store our own ratings from user ratings? Not sure whats best here.

Punctuation not properly de-internationalized

Test Case:

https://musicbrainz.org/artist/cfd45526-701e-457a-b3bd-013506a890a6
lidarr:lidarr:cfd45526-701e-457a-b3bd-013506a890a6

Lidarr fails with ', MusicBrainz succeeds with the same. Lidarr only succeeds with explicit ’ (UTF8).

Possibly need https://github.com/metabrainz/postgresql-musicbrainz-unaccent on the Pg-side, which does a pretty good job of handling cases like this.

ConnectionError: ('Connection aborted.', BadStatusLine("''",))
(17 additional frame(s) were not displayed)
...
  File "lidarrmetadata/api.py", line 111, in get_artist_info
    wikipedia_links[0]['target'])

ConnectionError: ('Connection aborted.', BadStatusLine("''",))