elvis-project / cbsmr-patternfinder Goto Github PK

Gabriel was testing the app and the flask-postgres connection timed out

fix by making a new connection each time

• docker deployment bug with protobuf messages not working on POST
  (works without containers, when run locally on separate services)

• capture piece name

Currently, the query is being displayed in the browser title tag.

We should add an HTML title tag: "Patternfinder" or "SMR"

to make the codebase more consistent and pass it off to susbequent developers

Hi,

Is there a way to use patternfinder with Windows?

Thanks.

http://132.206.14.238/search?query=**kern%0D%0A*clefG2%0D%0A*k%5B%5D%0D%0A%3D-%0D%0A4c+4e+4a+4cc%0D%0A4B-+f+b-+dd&rpp=5&page=0&tnps=0%2C12&intervening=0%2C12&inexact=0%2C1&collection=2

puts measure 26 and 27 from pid 5683 even though the excerpt URL asks for measures 1 and 2

• Highlighted excerpts have a high frequency of voice crossing. Is this because of music21 output, or do these pieces really have voice crossing?

• actually check that these notes occur in the score / original encodings

Check for this (and other) music snippets:

**kern
*clefG2
*k[]
=-
4g 4e
4d 4a
4c# 4f#
4d 4g
4e 4g

and update :ref tags in the code with the correct version e.g. L80 and L85 of flask/app.py

Increase music context by a measure before and after.

There should be an end-to-end integration test that stands up all services, inserts pieces, and executes a query

Without that, any changes against the code base can't be checked for correctness

Some MIDI files are not properly highlighted. Since this does not happen for all MIDI files, it is still not clear if this happens at indexing or searching time.

@DavidGarfinkle could you please add the filenames of a few of the failing MIDI files?

The Docker container should start the web application after a system boot or app crash.

The default number of intervening should be 0, with no transposition

The prefix of all results are also returned. in another sense, results that are too similar to higher-ranking results are taking priority over novel results

decide on which ones we will omit, and provide options for including them

a quote from how google solves this problem
"In order to show you the most relevant results, we have omitted some entries very similar to the 80 already displayed.
If you like, you can repeat the search with the omitted results included.
"

users should be able to toggle for which collection they are searching over

for example, check 20th result:

http://localhost/search?query=%2A%2Akern%0D%0A%2AclefG2%0D%0A%2Ak%5B%5D%0D%0A%3D-%0D%0A4g+dd%0D%0A4d%0D%0A4a+f%0D%0A4c+4b%0D%0A&page=1&rpp=20

(with /testdata/palestrina_masses/mid/* as database)

Melodic queries (such as http://132.206.14.238/search?query=**kern%0D%0A*clefG2%0D%0A*k%5B%5D%0D%0A%3D-%0D%0A4c+%0D%0A4d+%0D%0A4e+%0D%0A4f+%0D%0A4g%0D%0A&rpp=5&page=0&tnps=-12%2C12&intervening=0%2C0&inexact=0%2C0) return messages that are larger than the GRPC default of 4MB.

failed to search: <_Rendezvous of RPC that terminated with: status = StatusCode.RESOURCE_EXHAUSTED details = "Received message larger than max (5418422 vs. 4194304)" debug_error_string = "{"created":"@1567112503.150641370","description":"Received message larger than max (5418422 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":190,"grpc_status":8}" >

We should increase the maximum:

https://godoc.org/google.golang.org/grpc#MaxRecvMsgSize

Integration test wth flask app on a two-chord query returns nonsensical results, but unit tests with smaller queries & scores are passing

Possible points of failure:

• indexing layer (extracting notes from scores and lexicographically sorting them)
• calculating vectors between notes
• serializing and unserializing those vectors to disk
• the search algorithm
• highlighted excerpts may be colouring the wrong notes

Attempts at isolating the problem:

• insert unit-test data into the flask app and run integration tests on them. This seems to work fine
• use excerpts of larger pieces to create better unit tests. this is showing possible bugs in the indexing layer (should investigate further)

maybe minification will help?

Write Wiki with one page per container, as well as a page for the general overview of the app. Write thorough information about the API endpoints in order to facilitate communication with the app.

Also, add a page for end users, with screenshots and direct URLs to easily see the results of a given set of queries.

Pagination is already implemented in the URL, but we need to add buttons in the UI to allow users to navigate through the results.

With the current behaviour, new data is always added to the database and there is no checking for duplication.

One long term feature would be to warn if duplicated data is being uploaded. For now, you should make sure you FLUSH data before uploading new one.

The C code was never formally checked for memory leaks
This could lead to unstable behaviour in the future
I should run a memory management tool like http://www.valgrind.org/ to defensively erase those errors before they crash the component in production

indexers.notes does not return the same data running on a midi file versus the musicxml output of that midi file as output by music21.converter.parse(midi).write('xml')

GET http://132.206.14.238/excerpt?pid=9336&nid=406%2C407%2C408%2C416%2C417%2C418%2C419 500 (INTERNAL SERVER ERROR)

We should provide some visual feedback to let the user know the query is being processed on the back end./

The footer is not rendered properly, there is no About page or link to github

When doing docker-compose up I am getting the following error:

Step 3/7 : RUN make -B
 ---> Running in 86c6e4e51b52
gcc w2/w2.c w2/c_pqueue/pqueue.c -g -o w2/w2
w2/w2.c:28:1: warning: useless storage class specifier in empty declaration
 };
 ^
/usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../lib/Scrt1.o: in function `_start_c':
/home/buildozer/aports/main/musl/src/musl-1.1.20/crt/crt1.c:17: undefined reference to `main'
collect2: error: ld returned 1 exit status
make: *** [Makefile:2: w] Error 1

It seems there are some dependencies in your machine that are required for the container.

api endpoint generation should come directly from the source code
check out https://flask-apispec.readthedocs.io/en/latest/

currently, just a piece ID and the symbolic data is stored, but no metadata

Currently, a three-note melodic search with: maximum number of intervening notes and any transposition returns zero ocurrences results.

http://132.206.14.238/search?query=**kern%0D%0A*clefG2%0D%0A*k%5B%5D%0D%0A%3D-%0D%0A4c%0D%0A4d%0D%0A4e&rpp=20&page=0&tnps=-12%2C12&intervening=0%2C15&inexact=0%2C0&collection=0

Since the filters are quite tolerant, we expect the searching engine to return a number of pieces.

We need a toggle or filter in the UI to select among the different collections.

The filter should also create the &collection=xx that will be added to the HTTP request.

Currently the search component only operates over the entire database for every request

It should be possible to pass in a list of piece Ids to search over

load classic query
0-0
0-0

flask_1  | [pid: 8|app: 0|req: 52/176] 132.206.14.126 () {42 vars in 889 bytes} [Thu Aug 29 18:28:53 2019] GET /excerpt?pid=530&nid=156%2C157%2C158%2C159%2C160%2C161%2C162%2C163 => generated 0 byte
s in 1832 msecs (HTTP/1.1 200) 2 headers in 0 bytes (0 switches on core 0)
flask_1  | connecting to host=db port=5432 dbname=postgres user=postgres password=postgres
flask_1  | [2019-08-29 18:28:54,954] ERROR in app: Exception on /search [GET]
flask_1  | Traceback (most recent call last):
flask_1  |   File "/opt/venv/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
flask_1  |     response = self.full_dispatch_request()
flask_1  |   File "/opt/venv/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
flask_1  |     rv = self.handle_user_exception(e)
flask_1  |   File "/opt/venv/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
flask_1  |     reraise(exc_type, exc_value, tb)
flask_1  |   File "/opt/venv/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
flask_1  |     raise value
flask_1  |   File "/opt/venv/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
flask_1  |     rv = self.dispatch_request()
flask_1  |   File "/opt/venv/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
flask_1  |     return self.view_functions[rule.endpoint](**req.view_args)
flask_1  |   File "./app.py", line 151, in search
flask_1  |     tnps_ints[1] += 1 # increase range to include end
flask_1  | IndexError: list index out of range

CG DA EB FC
page 1, then page 100, then rpp 50 with page 100

filter the algorithm results in the golang layer because that's where we're also sorting the results

rpp=50 times out trying to fetch so many excerpts

The containers have to be finished, tested, and deployed to our development environment.

• Implement Nginx service
• Populate the index database
• Deploy all containers and services to dev environment

Search parameters (i.e., number of intervening notes and type of search) is currently fix. The user should be able to choose these values.

The baseline is what we currently have in the Patternfinder app.