derrickoswald / things Goto Github PK

There are a number of issues with the search system as it now stands, which is:

• couchdb-lucene
• executing as a external process
• running lucene

The issues are:

• The external process is executed by CouchDB (i.e. it has the ppid of CouchDB) but is not shut down when CouchDB is restarted. Instead it reverts to ppid 1 and another process is started by CouchDB, leading to multiple processes in memory.
• The credentials to access CouchDB, i.e. the admin username and password, need to be placed in the couchdb-lucene configuration file - in plain text - rather than using proxy authentication and reading the secret key from the CouchDB configuration (see the user_manager code for how this is handled).
• The logging from couchdb-lucene to the console (as configured in conf/log4j.xml) does not conform to the logging format expected by the stdin reader of the CouchDB externals code, leading to messages in the log like Ignoring OS daemon request: {{error,garbage_after_value}.
• Each database is indexed separately with the result that there are multiple queries required to do a search.
• The installation of couchdb-lucene requires compilation from scratch, with the inherent overhead of installing maven and all the crap it downloads.
• The Lucene install is nearly 60MB for some simple functionality.
• It is unclear how search can be extended to other systems. So far all the documents need to be local, and querying other things systems would require either CORS * or be done server side with another nodejs process

This is probably because couchdb-lucene was written before the externals API, and was probably not intended to run as a external process. There isn't anything wrong with Lucene per se, but the bolting on isn't working very well and could be improved.

This enhancement would probably involve either improving the couchdb-lucene integration or finding a better alternative to using the Lucene search engine.

In the current version, replication uses CouchDB to copy entire databases. This requires a server with an IP address and possibly a DNS name. Since the metadata of public things is published via BitTorrent seeding, this should not be required.
What seems to be needed is a .torrent file fetcher that would download the .torrent file and add the meta data from the .torrent file to a local copy of the remote things public database - as if it were replicated via CouchDB.
This would probably be written as a external server (node.js?) spawned by CouchDB that would monitor the thing_tracker database. For things (SHA1 hash codes) that had no local metadata, it would request Deluge to download the .torrent file and possibly the attachments (data files) as well. When the torrent file is downloaded, update CouchDB and delete the torrent from Deluge.