crews-be is an Express application that finds Spotify album credits in Discogs. It exposes one endpoint that, given the album id, requests the album data from Spotify API and immediately after searches for it in Discogs. As it finds credits for it, it updates the state (in app.locals) with such information. It's the backend that supports the crews app also available in my Github profile.

Application keys are needed for both Spotify and Discogs.

Application is deployed at Heroku

Try it out

The app accepts the following environment configuration variables:

1. Clone repo and npm install
2. Create your Spotify and Discogs applications and have their keys handy
3. Drop your env vars in a .env file.
4. Run npm start.
5. Request an album at localhost:<PORT>/data/album/:spotifyAlbumId

You'll notice the app responds very quickly to the client with an empty bestMatch with 0 progress. That just means the search started. You can keep requesting the album and check how the progress goes, but the console will also inform you about it. Ideally, a web client should poll the endpoint to find out about new data and stop when progress is 100. Since the Discogs API is the main limitant here and it's restricted to one request per second, it's not recommended to request the album more than once per second, since there won't be anything new before that.

Once the search finishes the data remains in the state for as long as the app is in memory, so subsequent requests for the album should show all the data found about the album.

The Discogs API exposes a search endpoint. The details of the results then have to be fetched one by one using the release endpoint.

Discogs API requests are limited. Some albums can throw tens and even hundreds of results. This can make some searches very long to complete. It is possible though, and considerably likely, that the first results are the most relevant. With this in mind the app exhibits the following behaviors:

• All the requests made to Discogs are throttled to avoid reaching the limit. I made a specific NPM package (throxy) solely for this.
• The 429 Too Many Requests has still been observed rarely (known issue) so it's handled.
• The album endpoint response has a progress value. This does not reflect found data, just how many operations have been performed to complete the search. The total amount of operations is given by the sum of the times the Discogs search endpoint has to be called and the amount of releases that need to be individually requested.
• The album endpoint response has a bestMatch object that contains all the found data for the album. Subsequent requests to the album endpoint will have it updated or not as the progress increases.
• Search operations are performed in strict sequential behavior. Otherwise a single search would hog the operations queue (the one that throxy handles).

Clients are suppossed to poll the album endpoint until progress reaches 100. All this also allows to start many searches at the same time and use the Discogs API resources efficiently for all the clients.

The app outputs info about the searches both to console and to disk (/log folder). In the console, a tag with the name of the album and artist being searched lets you know what every message is about. On disk, the logs folder contains a file for every album. The name of such file is the Spotify album ID. Logging is done using winston

Test are contained in *.spec.js files along the module they're testing. Frameworks used are mocha, sinon and Node's assert.