patrickfrey / struswebservice Goto Github PK

Issue by andreasbaumann
Saturday Apr 16, 2016 at 18:16 GMT
Originally opened as https://github.com/Eurospider/strusWebService/issues/5

when inserting,
when querying.

when inserting,
when querying.

Currently we use too many mutexes for things which are already synchronized in strus itself,
we don't share certain sharable objects (as the QueryProcessor) and thus have to recreate
the module functions on every query call. As the storage interface is bound to change anyway,
a major overhaul of the core architecture is needed.

We cannot use base request/answers and implement the common (de)serialization
in a base serialization template (at least, I don't know how).

Issue by andreasbaumann
Saturday Apr 23, 2016 at 08:30 GMT
Originally opened as https://github.com/Eurospider/strusWebService/issues/12

Currently we get messages like:

shell> strusWebService -v
terminate called after throwing an instance of 'cppcms::cppcms_error'
  what():  No configuration defined
Abort (core dumped)

This is not very user-friendly.

The web service configuration contains:

"metadata" : [
                                        { "name" : "doclen", "type" : "UInt16" },
                                        { "name" : "docweight", "type" : "Float32" },
                                        { "name" : "date", "type" : "UInt16x" }
                        ]

an illegal type Uint16x.

Creating an index yields:

curl -XPOST -H 'Content-Type: application/json' -d '{ "params" : {} }' http://localhost:8080/strus/index/create/A
{"result":"ok"}

No error message!

Also in the log I see:

2016-05-20 13:48:59; strusWebService, debug: Storage config string: database=leveldb; path=./storage/B; compression=yes; cache=8388608; max_open_files=1000; write_buffer_size=4194304; block_size=4096; metadata=doclen UInt16, docweight Float32, date UInt16x (strusWebService.cpp:381)
2016-05-20 13:48:59; strusWebService, debug: Simple storage config string: path=./storage/B (strusWebService.cpp:394)

No indication of errors.

The indexes have been created:

shell> curl http://localhost:8080/strus/index/list
{"indexes":["A","B"],"result":"ok"}

Getting the configuration of the index yields:

curl http://localhost:8080/strus/index/config/A
{"err":{"code":2404,"msg":"error creating storage client: error creating storage client: corrupt storage, not all mandatory variables defined"},"result":"error"}

The web service configuration contains:

"metadata" : [
                                        { "name" : "doclen", "type" : "UInt16" },
                                        { "name" : "docweight", "type" : "Float32" },
                                        { "name" : "date", "type" : "UInt16x" }
                        ]

an illegal type Uint16x.

Creating an index yields:

curl -XPOST -H 'Content-Type: application/json' -d '{ "params" : {} }' http://localhost:8080/strus/index/create/A
{"result":"ok"}

No error message!

Also in the log I see:

2016-05-20 13:48:59; strusWebService, debug: Storage config string: database=leveldb; path=./storage/B; compression=yes; cache=8388608; max_open_files=1000; write_buffer_size=4194304; block_size=4096; metadata=doclen UInt16, docweight Float32, date UInt16x (strusWebService.cpp:381)
2016-05-20 13:48:59; strusWebService, debug: Simple storage config string: path=./storage/B (strusWebService.cpp:394)

No indication of errors.

The indexes have been created:

shell> curl http://localhost:8080/strus/index/list
{"indexes":["A","B"],"result":"ok"}

Getting the configuration of the index yields:

curl http://localhost:8080/strus/index/config/A
{"err":{"code":2404,"msg":"error creating storage client: error creating storage client: corrupt storage, not all mandatory variables defined"},"result":"error"}

cppcms json gives us only a bad_cast expection, no position, no clear indication what's
wrong. parameters which are wrong get set do a default. This is all not very user-friendly.
cppcms::json must be improved (local fork) or we use another or our own template
serialization code.

The following tests FAILED:
         28 - query_in_url (Failed)
         29 - query_A (Failed)

This one seems trivial:

1c1
< {"err":{"code":4011,"msg":"cannot evaluate query, no selection features defined"},"result":"error"}

---
> {"ranklist":{"ranks":[{"attributes":[{"key":"docid","value":"doc3"},{"key":"title","value":"This is a Hello World Document"}],"docno":4,"weight":0},{"attributes":[{"key":"docid","value":"doc2"},{"key":"title","value":"This is a Hello World Document"}],"docno":2,"weight":0},{"attributes":[{"key":"docid","value":"doc1"},{"key":"title","value":"This is a Hello World Document"}],"docno":1,"weight":0}]},"result":"ok"}

Scores and weights are floats which cannot be handled by just diffing the results:

< {"ranklist":{"documents_ranked":3,"documents_visited":3,"passes_evaluated":0,"ranks":[{"attributes":[{"key":"docid","value":"doc3"}],"docno":4,"weight":5.205622073845007e-06},{"attributes":[{"key":"docid","value":"doc2"}],"docno":2,"weight":2.307670229129144e-06},{"attributes":[{"key":"docid","value":"doc1"}],"docno":1,"weight":1.110571247409098e-06}]},"result":"ok"}

---
> {"ranklist":{"ranks":[{"attributes":[{"key":"attribute","value":"doc3"}],"docno":3,"weight":1.850274916547851e-16},{"attributes":[{"key":"attribute","value":"doc2"}],"docno":2,"weight":1.850274916547851e-16},{"attributes":[{"key":"attribute","value":"doc1"}],"docno":1,"weight":1.850274916547851e-16}]},"result":"ok"}

We must demarshall the .res and the .must file and make a special comparition
operation for floats.

makes a bad democase currently :-(

cppcms json gives us only a bad_cast expection, no position, no clear indication what's
wrong. parameters which are wrong get set do a default. This is all not very user-friendly.
cppcms::json must be improved (local fork) or we use another or our own template
serialization code.

especially handy for attribute tagging.

transactions across storages could be handy.

Currently the old weighting before 0.9 is simply mapped to one fixed
scalar function. Issues to be solved are especially how does the JSON
protocol change, how do we refernce scalar functions andor define them.

The parameter for storage creation and storage usage should be strictly
separated. Also not quite sure what happens if storages are opened with
different parameters than they were created (e.g. compression, block_size).

Issue by andreasbaumann
Saturday Apr 16, 2016 at 18:17 GMT
Originally opened as https://github.com/Eurospider/strusWebService/issues/6

transactions across storages could be handy.

no big harm currently, we only have a default LevelDB implementation.

Issue by andreasbaumann
Saturday Apr 16, 2016 at 18:14 GMT
Originally opened as https://github.com/Eurospider/strusWebService/issues/3

Only the part using the default parameters is currently implemented.
They cannot be ommited in the request. They are not persistent across sessions.

no big harm currently, we only have a default LevelDB implementation.

Issue by andreasbaumann
Saturday Apr 16, 2016 at 18:15 GMT
Originally opened as https://github.com/Eurospider/strusWebService/issues/4

no big harm currently, we only have a default LevelDB implementation.

Issue by andreasbaumann
Thursday Apr 28, 2016 at 07:55 GMT
Originally opened as https://github.com/Eurospider/strusWebService/issues/13

Currently we use too many mutexes for things which are already synchronized in strus itself,
we don't share certain sharable objects (as the QueryProcessor) and thus have to recreate
the module functions on every query call. As the storage interface is bound to change anyway,
a major overhaul of the core architecture is needed.

Issue by andreasbaumann
Saturday Apr 16, 2016 at 18:25 GMT
Originally opened as https://github.com/Eurospider/strusWebService/issues/10

We cannot use base request/answers and implement the common (de)serialization
in a base serialization template (at least, I don't know how).

makes a bad democase currently :-(

Issue by andreasbaumann
Thursday Mar 31, 2016 at 17:39 GMT
Originally opened as https://github.com/Eurospider/strusWebService/issues/2

transactions across storages could be handy.

We cannot use base request/answers and implement the common (de)serialization
in a base serialization template (at least, I don't know how).

Currently we have a subtle logic how to find out whether a parameter has to be
passed as string/numeric parameter or as feature to a weighting or summarizer
function:

• the parameter name exists in a location which is clearly expecting a feature
  (e. g. 'selection' is only accepting features)
• we take the desciption of the function as reference

Currently the old weighting before 0.9 is simply mapped to one fixed
scalar function. Issues to be solved are especially how does the JSON
protocol change, how do we refernce scalar functions andor define them.

On the fly changes to the metadata configuration via HTTP protocol. Very handy when
an installation gets bigger and metadata types have to be adapted. Attributes have no
types (i.e. string only) for now and feature types are strings too, so dynamic changes
are possible implicitly.

when inserting,
when querying.

Currently we have a subtle logic how to find out whether a parameter has to be
passed as string/numeric parameter or as feature to a weighting or summarizer
function:

• the parameter name exists in a location which is clearly expecting a feature
  (e. g. 'selection' is only accepting features)
• we take the desciption of the function as reference

Issue by andreasbaumann
Saturday Apr 16, 2016 at 18:26 GMT
Originally opened as https://github.com/Eurospider/strusWebService/issues/11

As in a database. This helps users of the service to react properly.

The parameter for storage creation and storage usage should be strictly
separated. Also not quite sure what happens if storages are opened with
different parameters than they were created (e.g. compression, block_size).

at least we should be able to provide protection for

• queries
• modification commands (insert, update, delete)
• introspection
• DDL: create, destroy indexes

2016-06-07 08:36:10; cppcms_http, info: POST /strus/index/delete/rrb (http_api.cpp:251)
strusWebService: db/version_set.cc:789: leveldb::VersionSet::~VersionSet(): Assertion `dummy_versions_.next_ == &dummy_versions_' failed.

Restarting the webservice solves the problem, after restart the index can be deleted
without any problems.

at least we should be able to provide protection for

• queries
• modification commands (insert, update, delete)
• introspection
• DDL: create, destroy indexes

especially handy for attribute tagging.

Issue by andreasbaumann
Saturday Apr 16, 2016 at 18:19 GMT
Originally opened as https://github.com/Eurospider/strusWebService/issues/7

cppcms json gives us only a bad_cast expection, no position, no clear indication what's
wrong. parameters which are wrong get set do a default. This is all not very user-friendly.
cppcms::json must be improved (local fork) or we use another or our own template
serialization code.

Issue by andreasbaumann
Thursday Mar 31, 2016 at 17:37 GMT
Originally opened as https://github.com/Eurospider/strusWebService/issues/1

implemented, but currently not enough tested.

If I set a summarizer name to "snippet" and I'm getting elements from the forward index,
I get "forward" as name of the attribute I have to fetch. This is not very logical. :-)

On the fly changes to the metadata configuration via HTTP protocol. Very handy when
an installation gets bigger and metadata types have to be adapted. Attributes have no
types (i.e. string only) for now and feature types are strings too, so dynamic changes
are possible implicitly.

Issue by andreasbaumann
Saturday Apr 16, 2016 at 18:23 GMT
Originally opened as https://github.com/Eurospider/strusWebService/issues/9

Currently we have a subtle logic how to find out whether a parameter has to be
passed as string/numeric parameter or as feature to a weighting or summarizer
function:

• the parameter name exists in a location which is clearly expecting a feature
  (e. g. 'selection' is only accepting features)
• we take the desciption of the function as reference

Issue by andreasbaumann
Saturday Apr 16, 2016 at 18:20 GMT
Originally opened as https://github.com/Eurospider/strusWebService/issues/8

we write messages in a temporary file which we rewind on error output.
we should use string buffers only. can we pass a NULL as file pointer to
the ErrorHandler of strus?

Currently we use too many mutexes for things which are already synchronized in strus itself,
we don't share certain sharable objects (as the QueryProcessor) and thus have to recreate
the module functions on every query call. As the storage interface is bound to change anyway,
a major overhaul of the core architecture is needed.

The following tests FAILED:
         28 - query_in_url (Failed)
         29 - query_A (Failed)

This one seems trivial:

1c1
< {"err":{"code":4011,"msg":"cannot evaluate query, no selection features defined"},"result":"error"}

---
> {"ranklist":{"ranks":[{"attributes":[{"key":"docid","value":"doc3"},{"key":"title","value":"This is a Hello World Document"}],"docno":4,"weight":0},{"attributes":[{"key":"docid","value":"doc2"},{"key":"title","value":"This is a Hello World Document"}],"docno":2,"weight":0},{"attributes":[{"key":"docid","value":"doc1"},{"key":"title","value":"This is a Hello World Document"}],"docno":1,"weight":0}]},"result":"ok"}

Scores and weights are floats which cannot be handled by just diffing the results:

< {"ranklist":{"documents_ranked":3,"documents_visited":3,"passes_evaluated":0,"ranks":[{"attributes":[{"key":"docid","value":"doc3"}],"docno":4,"weight":5.205622073845007e-06},{"attributes":[{"key":"docid","value":"doc2"}],"docno":2,"weight":2.307670229129144e-06},{"attributes":[{"key":"docid","value":"doc1"}],"docno":1,"weight":1.110571247409098e-06}]},"result":"ok"}

---
> {"ranklist":{"ranks":[{"attributes":[{"key":"attribute","value":"doc3"}],"docno":3,"weight":1.850274916547851e-16},{"attributes":[{"key":"attribute","value":"doc2"}],"docno":2,"weight":1.850274916547851e-16},{"attributes":[{"key":"attribute","value":"doc1"}],"docno":1,"weight":1.850274916547851e-16}]},"result":"ok"}

We must demarshall the .res and the .must file and make a special comparition
operation for floats.

no big harm currently, we only have a default LevelDB implementation.