It is useful a GET request, ex: /version
and the response could be:
{
"version": "x.y.z",
"git_hash": "hash",
"compile_date": "date_time"
.... etc.
}

When STH service is started without a tty sudo command gives an error. So command sudo is replaced by su command.

It would be nice to have that the STH component subscribes (or renews its subscription) to the interested NGSI entities once it starts up. The subscription parameters could be provided in a JSON file (mimicing the NGSI subscription payload) or via the command line.

License header used in Cygnus:

/**
 * Copyright 2015 Telefonica Investigación y Desarrollo, S.A.U
 *
 * This file is part of fiware-cygnus (FI-WARE project).
 *
 * fiware-cygnus is free software: you can redistribute it and/or modify it under the terms of the GNU Affero
 * General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your
 * option) any later version.
 * fiware-cygnus is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the
 * implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License
 * for more details.
 *
 * You should have received a copy of the GNU Affero General Public License along with fiware-cygnus. If not, see
 * http://www.gnu.org/licenses/.
 *
 * For those usages not covered by the GNU Affero General Public License please contact with iot_support at tid dot es
 */

LICENSE file used at Cygnus: https://github.com/telefonicaid/fiware-cygnus/blob/master/LICENSE

Currently there is an issue (Automattic/mongoose#2073) about a possible memory leak when using #connection.useDB() which makes the following warning to be shown during the tests: "(node) warning: possible EventEmitter memory leak detected. 11 error listeners added. Use emitter.setMaxListeners() to increase limit." Another issue in Node.js related to this issue can be located at nodejs/node-v0.x-archive#5108

There are also some StackOverflow questions about this issue at:

• http://stackoverflow.com/questions/26018752/warning-when-connecting-to-mongodb-warning-possible-eventemitter-memory-leak
• http://stackoverflow.com/questions/15581978/nodejs-how-to-debug-eventemitter-memory-leak-detected-11-listeners-added

In a machine of the FIWARe Lab Ubuntu 12 I'm getting the following error, when I try to start the STH with a non-root user.

npm ERR! npm v2.5.1
npm ERR! code ELIFECYCLE
npm ERR! [email protected] start: node ./src/sth_app
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script 'node ./src/sth_app'.
npm ERR! This is most likely a problem with the fiware-comet package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node ./src/sth_app
npm ERR! You can get their info via:
npm ERR! npm owner ls fiware-comet
npm ERR! There is likely additional logging output above.
npm ERR! Linux 3.2.0-29-generic
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "start"
npm ERR! node v0.12.0
npm ERR! npm v2.5.1

npm ERR! Uncaught, unspecified "error" event.
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR! http://github.com/npm/npm/issues

npm ERR! Please include the following file with any support request:
npm ERR! /home/gtoro/git/telefonicaid/IoT-STH/npm-debug.log

Currently, MongoDB does not support acting on the same property and update operation using the operators $setOnInsert and $set and consequently 2 update operations are needed. We should reduce it to 1 as soon as MongoDB supports it.

For more information see http://stackoverflow.com/questions/23992723/findandmodify-fails-with-error-cannot-update-field1-and-field1-at-the-same

• Assigned to @vgarciag :)

The maximum size of the log files should be configurable. If the maximum size is reached, a new log file should be created with a new name (typically using suffixes to distinguish amongst them).

1. The references should be removed from the main README.md file, the src directory and the test/node directory. @gtorodelvalle will take care of this.
2. The references should also be removed from the package directory. Could you, @vgarciag , take care of this please? :)

Apart from requesting aggregated data from the database, the raw events (aka. raw data) should also be made available via GET requests.

Ex:
Fiware-Service: my_service

In mongo:

sth_orion

sth prefix is correct, but orion is service by default

The LICENSE file will depend on the chosen license type.

The ContributionPolicy.txt may be directly copied from https://github.com/telefonicaid/fiware-connectors

Tests to cover the functioning of the sth_server.js should be included.

Request:

url    : http://qa-bigdata-sth-01:8666/notify
headers: {'Fiware-Service': 'my_service', 'Content-Type': 'application/json', 'Fiware-ServicePath': '/my_service_path', 'Accept': 'application/json', 'User-Agent': 'orion/0.10.0'}
payload: {"originator": "localhost", "subscriptionId": "51c0ac9ed714fb3b37d7d5a8", "contextResponses": [{"contextElement": {"attributes": [{"type": "celcius", "name": "name_EVwy_0", "value": "gdfgfdgf"}], "type": "house", "id": "room2", "isPattern": "false"}, "statusCode": {"code": "200", "reasonPhrase": "OK"}}]}

Initially write in log:

time=2015-04-28T13:07:22.664Z | lvl=FATAL | corr=NA | trans=NA | op=OPER_STH_SERVER_LOG | msg=Attribute value not aggregatable, alarm_status=ALARM

But after occurs a timeout, responds Connection aborted and write in log:

time=2015-04-28T13:09:22.693Z | lvl=FATAL | corr=NA | trans=NA | op=OPER_STH_SERVER_LOG | msg=POST /notify, event={"request":"1430226442655:qa-bigdata-sth-01:22935:i91bidyo:10006","timestamp":1430226562693,"tags":["request","closed","error"],"internal":true}, alarm_status=ALARM

ex:

timestamp: 1430132288
GMT: Mon, 27 Apr 2015 10:58:08 GMT
Your time zone: lunes, 27 de abril de 2015 12:58:08 GMT+2:00 DST

This failure affect:

• offset in hour resolution

          date: 1430146848.0
          date: 2015-04-27 15:00:48 GMT+2:00
          origin: 2015-04-27 00:00:00 
         { "offset" : 13, "samples" : 1, "sum" : 29.5, "sum2" : 870.25, "min" : 29.5, "max" : 29.5 }, 

• origin in minute and second resolutions

minute:

        date: 1430146872.0
        human date: 2015-04-27 15:01:12
        origin: 2015-04-27 13:00:00

second:

        date: 1430146896.0
        human date: 2015-04-27 15:01:36
        origin: 2015-04-27 13:01:00

At the moment in docs is stored the recvTime field and is useful append the recvTimeTs field too.
Ex:

{
        "_id" : ObjectId("5541dfd60a673a1e0476ef90"),
        "recvTime" : ISODate("2015-04-30T07:55:02.383Z"),
        "entityId" : "room2",
        "entityType" : "room",
        "attrName" : "name_KB6o_0",
        "attrType" : "celcius",
        "attrValue" : "64.5"
}

Related to: telefonicaid/fiware-cygnus#372

in Doc:

lastN: Only the requested last entries should be returned.

Example:
request with hLimit and hOffset:

.../STH/v1/contextEntities/type/room/id/room2/attributes/tempe_0?hLimit=20&hOffset=0&dateFrom=2014-02-14T00:00:00.000Z&dateTo=2016-02-14T23:59:59.99

Response:

{
    "contextResponses":[
        {
            "contextElement":{
                "attributes":[
                    {
                        "name":"tempe_0",
                        "values":[
                            {
                                "recvTime":"2015-05-06T08:23:00.673Z",
                                "attrValue":"28.1"
                            },
                            {
                                "recvTime":"2015-05-06T08:23:44.386Z",
                                "attrValue":"28.2"
                            },
                            {
                                "recvTime":"2015-05-06T08:25:45.924Z",
                                "attrValue":"83.3"
                            },
                            {
                                "recvTime":"2015-05-06T08:57:18.307Z",
                                "attrValue":"16.6"
                            },
                            {
                                "recvTime":"2015-05-06T08:59:08.819Z",
                                "attrValue":"1.9"
                            }
                        ]
                    }
                ],
                "id":"room2",
                "isPattern":false,
                "type":"room"
            },
            "statusCode":{
                "code":"200",
                "reasonPhrase":"OK"
            }
        }
    ]
}

request with lastN:

.../STH/v1/contextEntities/type/room/id/room2/attributes/tempe_0?lastN=1&dateFrom=2014-02-14T00:00:00.000Z&dateTo=2016-02-14T23:59:59.99

response:

{
    "contextResponses":[
        {
            "contextElement":{
                "attributes":[
                    {
                        "name":"tempe_0",
                        "values":[
                            {
                                "recvTime":"2015-05-06T08:23:00.673Z",
                                "attrValue":"28.1"
                            }
                        ]
                    }
                ],
                "id":"room2",
                "isPattern":false,
                "type":"room"
            },
            "statusCode":{
                "code":"200",
                "reasonPhrase":"OK"
            }
        }
    ]
}

Expected value:

  {
          "recvTime":"2015-05-06T08:59:08.819Z",
          "attrValue":"1.9"
  }

We launch a performance test during 4h with:
10 databases and 40 threads by database = 400 notifications at the same time
only one collection in each database, usin per-service-path model

we have saw that the throughput gradually descend to the point that sth, stops receiving the notifications

Although the formal way to populate the time series database managed by the STH component is using the Cygnus component and the sink which will be developed for it, an alternative (so-called 'minimalist') way to populate this database is required.

This alternative mechanism will consist on the direct subscription by the STH component to an Orion Context Broker to receive notifications when the value of the attributes of interest change. As a consequence of these notifications, the STH will update the STH time series database updating the aggregated data for the concrete attributes.

Since both mechanisms (the formal one using Cygnus and the minimalist one using the STH directly) update the same database, it is the responsibility of the people or software in charge of creating the needed subscriptions to avoid updating the time series database twice. This would happen if both mechanisms are enabled for the same attribute of the same entity.

Ex:

time=2015-04-16T15:27:37.874Z | lvl=FATAL | corr=NA | trans=NA | op=OPER_STH_SERVER_LOG | msg=Attribute value not aggregatable, alarm_status=ALARM

It could be WARN instead of FATAL

The response format described at https://github.com/telefonicaid/IoT-STH#-consuming-raw-data doesn't match with the current response of STH. "Values" should be a list of dictionaries.

Documentation raw data response format:

...
"attributes": [
                    {
                        "name": "attrName",
                        "values": [
                            {
                                {
                                    "recvTime": "2014-02-14T13:43:33.306Z",
                                    "attrValue": "21.28"
                                },
                                {
                                   "recvTime": "2014-02-14T13:43:34.636Z",
                                   "attrValue": "23.42"
                                },
                                {
                                    "recvTime": "2014-02-14T13:43:35.424Z",
                                    "attrValue": "22.12"
                                }
                            }
                        ]
                    }
                ],
...

Real response of STH:

...
"attributes": [
                    {
                        "name": "attrName",
                        "values": [
                                {
                                    "recvTime": "2014-02-14T13:43:33.306Z",
                                    "attrValue": "21.28"
                                },
                                {
                                   "recvTime": "2014-02-14T13:43:34.636Z",
                                   "attrValue": "23.42"
                                },
                                {
                                    "recvTime": "2014-02-14T13:43:35.424Z",
                                    "attrValue": "22.12"
                                }
                        ]
                    }
                ],
...

Currently and due to winstonjs/winston#228 we use a setTimeout() construction to let winston flush all the messages to a file before exiting the application. This hack is currently used in the sth_app.js file.
We should avoid using this hack since nothing assures us that all the messages will be flushed to the file when using the included timeout (currently 500ms).

See https://github.com/telefonicaid/IoT-STH/blob/develop/src/sth_configuration.js#L70

Currently the MongoDB $ positional update operator cannot be combined with upserts (see http://docs.mongodb.org/manual/reference/operator/update/positional/#upsert).

This issue is known and currently under study: https://jira.mongodb.org/browse/SERVER-3326
Once the issue is solved, it will be possible to prepopulate collections or update their docs using just one update operation like this:

  collection.update(
    // Returning all the update operators currently returned by getAggregateUpdate4Insert
    //  and getAggregateUpdate4Update in the same object
    getAggregateUpdateCondition(entityId, entityType, attrName, resolution, recvTime),
    getAggregateUpdate(attrValue),
    {
      upsert: true,
      writeConcern: {
        w: !isNaN(sthConfig.WRITE_CONCERN) ? parseInt(sthConfig.WRITE_CONCERN) : sthConfig.WRITE_CONCERN
      }
    },
    function (err) {
      callback(err);
    }
  );

At least for Java developments, Telefónica has a telefonica_checkstyle.xml file that can be configured in your favourite IDE in order to check if the required style is being met.

I don't know if something similar exists for Javascript, I suppose so...

STR (Steps To Reproduce):

1. For a Fiware-ServicePath header such as "/my_serv_path_0"

OBSERVED:
2. The collections created in MongoDB are named : sth_/my_serv_path_0 and sth_/my_serv_path_0.aggr

EXPECTED:
2. The collections created in MongoDB should be named : sth__my_serv_path_0 and sth__my_serv_path_0.aggr This is, subsituting "/" by "_".

The logging format of the STH REST API implementation must adhere to the FIWARE format which is basically the following:

time= | lvl= | corr=<UNICA_CORRELATOR> | trans=<TRANSACTION_ID> | op=<OPERATION_TYPE> | msg=

where:

• TIMESTAMP should be formatted in IEEE RFC 3339 format
• LEVEL is the level of the log (INFO, WARN, ERROR, FATAL, DEBUG or TRACE)
• UNICA_CORRELATOR is the value of the Unica-Correlator HTTP header
• TRANSACTION_ID is an unique id to identify the ongoing transaction
• OPERATION_TYPE is an id referred to the ongoing operation
• MESSAGE typically contains some descriptive information about the event being logged

Request:

{
    "originator":"localhost",
    "subscriptionId":"51c0ac9ed714fb3b37d7d5a8",
    "contextResponses":[
        {
            "contextElement":{
                "attributes":[
                    {
                        "type":"celcius",
                        "name":"name_ni9w_0",
                        "value":"34"
                    },
                    {
                        "type":"celcius",
                        "name":"name_ni9w_1",
                        "value":"34"                       
                    },
                    {
                        "type":"celcius",
                        "name":"name_ni9w_2",
                        "value":"34"
                    }
                ],
                "type":"room",
                "id":"room2",
                "isPattern":"false"
            },
            "statusCode":{
                "code":"200",
                "reasonPhrase":"OK"
            }
        }
    ]

}

In mongo:

{ "_id" : ObjectId("5538ecbc3a4b153e0fba2837"), "timestamp" : ISODate("2015-04-23T12:59:40.272Z"), "entityId" : "room2", "entityType" : "room", "attrName" : "name_ni9w_2", "attrType" : "celcius", "attrValue" : "34" }
{ "_id" : ObjectId("5538ecbc3a4b153e0fba2838"), "timestamp" : ISODate("2015-04-23T12:59:40.272Z"), "entityId" : "room2", "entityType" : "room", "attrName" : "name_ni9w_2", "attrType" : "celcius", "attrValue" : "34" }
{ "_id" : ObjectId("5538ecbc3a4b153e0fba2839"), "timestamp" : ISODate("2015-04-23T12:59:40.272Z"), "entityId" : "room2", "entityType" : "room", "attrName" : "name_ni9w_2", "attrType" : "celcius", "attrValue" : "34" }

STH software should have a way to distribute it bringing facilities to system administrators to install it, stop, start process and log maintenance.

Moreover this package features adds the possibility to admin one or more instance of STH in a simple way using standard init.d scripts

example of raw response:

{"contextResponses":[
             {"contextElement":
                          {"attributes":[
                                      {"name":"tempe_0",
                                       "values":[
                                                 {"recvTime":"2015-05-06T08:23:00.673Z",
                                                  "attrValue":"28.1"},                     
                                                 {"recvTime":"2015-05-06T08:23:44.386Z",
                                                  "attrValue":"28.2"},  
                                                 {"recvTime":"2015-05-06T08:25:45.924Z",
                                                  "attrValue":"83.3"}
                                         ]
                                     }
                               ],
                           "id":"room2",
                           "isPattern":false,
                           "type":"room"},
              "statusCode":{
                      "code":"200",
                      "reasonPhrase":"OK"}
            }
  ]
}

The attribute type could change.

Include information about the installation and use of the STH REST API app.

if we execute the npm install command shows this message:

npm WARN package.json [email protected] No repository field.

Right now there are functions in the sth_helpers.js module for server and database operations which should be included into their respective modules:

• stopServer() -> sth_server.js
• closeDBConnection() -> sth_database.js

As the dates (dateFrom and dateTo) are optional, you may not use dateFrom, in this case what is its value? The beginning of time?

Is neccesary the exception message? I recommend use a message more especific to the error.

Error returned:

net.js:1236
    throw new Error('Not running');
          ^
Error: Not running
    at Server.close (net.js:1236:11)
    at internals.Connection._stop (/home/sysadmin/git/IoT-STH/node_modules/hapi/lib/connection.js:188:19)
    at /home/sysadmin/git/IoT-STH/node_modules/hapi/lib/server.js:210:20
    at iterate (/home/sysadmin/git/IoT-STH/node_modules/hapi/node_modules/items/lib/index.js:35:13)
    at Object.exports.serial (/home/sysadmin/git/IoT-STH/node_modules/hapi/node_modules/items/lib/index.js:38:9)
    at internals.Server.stop (/home/sysadmin/git/IoT-STH/node_modules/hapi/lib/server.js:208:11)
    at Object.stopServer (/home/sysadmin/git/IoT-STH/src/sth_server.js:163:14)
    at exitGracefully (/home/sysadmin/git/IoT-STH/src/sth_app.js:45:15)
    at process.module.exports.startup (/home/sysadmin/git/IoT-STH/src/sth_app.js:114:12)
    at process.emit (events.js:95:17)
npm ERR! weird error 7
npm ERR! not ok code 0

The responses provided by the STH REST API server should adhere to the NGSI format (for further information see http://technical.openmobilealliance.org/Technical/release_program/docs/NGSI/V1_0-20101207-C/OMA-TS-NGSI_Context_Management-V1_0-20100803-C.pdf).

Currently the logging levels are TRACE, INFO, DEBUG, WARN, ERROR, FATAL when they should be, in ascending order of importance and severity: DEBUG, INFO, WARN, ERROR

The INFO level should only include information about the starting and stopping of the server and its components as well as 1 minute interval "proof of life" including some KPIs of interest such as requests attended, memory consumption or any other which may be considered of interest.

We need a proof-of-concept (PoC) implementation of the STH REST API for querying aggregated data.

This PoC will expose an HTTP endpoint attending GET requests accepting the following query parameters:

• 'aggrMethod': Mandatory. The aggregation method. Valid values are: 'max', 'min', 'sum' and 'sum2'.
• 'aggrPeriod': Mandatory. The resolution or the time-frame the aggregated data will be grouped by. Valid values are: 'month', 'day', 'hour', 'minute', 'second'.
• 'dateFrom': Optional. The origin of time from which to return the aggregated data.
• 'dateTo: Optional. The final time to which to return the aggregated data.

To this regard, the expected behaviour based on concretes requests is the following:

• /STH/v1/contextEntities/type/quantity/id/entityId/attributes/attributeId?aggrMethod=min&aggrPeriod=second
  ** It should return the 'min' available aggregated data for the current minute grouped by seconds
• /STH/v1/contextEntities/type/quantity/id/entityId/attributes/attributeId?aggrMethod=min&aggrPeriod=minute
  ** It should return the 'min' available aggregated data for the current hour grouped by minutes
• /STH/v1/contextEntities/type/quantity/id/entityId/attributes/attributeId?aggrMethod=min&aggrPeriod=hour
  ** It should return the 'min' available aggregated data for the current day grouped by hours
• /STH/v1/contextEntities/type/quantity/id/entityId/attributes/attributeId?aggrMethod=min&aggrPeriod=day
  ** It should return the 'min' available aggregated data for the current month grouped by days
  • /STH/v1/contextEntities/type/quantity/id/entityId/attributes/attributeId?aggrMethod=min&aggrPeriod=month
    ** It should return the 'min' available aggregated data for the current year grouped by months
  • /STH/v1/contextEntities/type/quantity/id/entityId/attributes/attributeId?aggrMethod=min&aggrPeriod=hour&dateFrom=2015-02-01T00:00:00&dateTo=2015-02-31T23:59:59
    ** It should return the 'min' available aggregated data from 2015-02-01T00:00:00 (UTC time) to 2015-02-31T23:59:59 (UTC time) grouped by hours

The response for a request for 'min' aggregated data grouped by hours looks like this:

[{"range":"day","resolution":"hour","type":"quantity","origin":"2015-02-27T00:00:00.000Z","values":[{"offset":0,"samples":0,"min":null},{"offset":1,"samples":2,"min":14.46},{"offset":2,"samples":0,"min":null},{"offset":3,"samples":0,"min":null},{"offset":4,"samples":0,"min":null},{"offset":5,"samples":0,"min":null},{"offset":6,"samples":0,"min":null},{"offset":7,"samples":0,"min":null},{"offset":8,"samples":0,"min":null},{"offset":9,"samples":0,"min":null},{"offset":10,"samples":2,"min":61.99},{"offset":11,"samples":0,"min":null},{"offset":12,"samples":0,"min":null},{"offset":13,"samples":0,"min":null},{"offset":14,"samples":0,"min":null},{"offset":15,"samples":0,"min":null},{"offset":16,"samples":0,"min":null},{"offset":17,"samples":0,"min":null},{"offset":18,"samples":0,"min":null},{"offset":19,"samples":0,"min":null},{"offset":20,"samples":0,"min":null},{"offset":21,"samples":0,"min":null},{"offset":22,"samples":0,"min":null},{"offset":23,"samples":0,"min":null}]}]

where:

• range: It is the time range of each of the values returned. In this case, a day.
• resolution: It is the resolution (also known as the aggregation period (aggrPeriod)). In this case, hours.
• type: It is the type of data. In this case a quantity.
• origin: It is the origin of time for each one of the values. It depends on the range and the resolution. For a resolution of hours and range of days, it is a concrete day for which there is aggregated data available grouped by hours.
• offset: It depends on the range and the resolution. Since in this example the resolution is 'hours' and the range is 'days', it refers to the concrete hour in the day pointed by the origin.
• samples: It is the number of samples (or events) available for a concrete offset from the origin.
• min: In this concrete example it is the minimum value, but it could be any of the aggregation methods. In this case, It is the minimum value for the concrete offset from the origin. null if no samples for the offset.

See https://github.com/telefonicaid/IoT-STH/blob/develop/src/sth_configuration.js#L69

The STH component should be able to connect to isolated instances of MongoDB databases as well as to replica sets.

Example:
A notification with date 2015-05-07, in month resolution the offset should be 5 and at the moment is 4.