lsir / gsn Goto Github PK

Hi ,

I notice that h2 jdbc driver gets loaded during startup even though it is not specified in conf/gsn.xml file. I assume h2 is being used in postgres mode. Is there a way to disable this ? This is preventing me from using specific postgres extensions like jsonb.

A workaround will be helpful.

Thanks,

How can we store the geographical location of a sensor into database ? Is GSN using a php connection ? Can someone help me ?

There are too many servelts containing html code. These should be done using jsp pages...

I was just curious if it was possible for wrappers to subscribe to data streams for outbound messaging? For example, if I created a wrapper for my device, is it possible for the wrapper to subscribe to a data stream that has published commands for the device?

headers with multiple lines (e.g. with \n \r characters) should be escaped or encoded in one line, for the csv output

Use of reliable message queuing system to achieve non blocking communication inside GSN

In the current implementation when a push request arrives, it gets registered and the data transmission starts immediately. If the client virtual sensor is not completely loaded the received remote data is going to be lost.
(Reported by mriahi on sourceforge)

Idea: a sensor could be a geojson feature.

Hi，
I've tried many times, but it doesn't work. When I click on the right-bar to add a VSensor (eg. multiFormatSample), it displays the data correctly. However if I press "F5" to reload the home page, the sensor data becomes 'null', and when the "auto-refresh" is set, the page refresh every x seconds normally but the sensor data never change ( always = null) .

Environment:
Win7(64bit); jdk1.6.0_45; apache-ant-1.9.4 chrome+IE
also tried
XP(32bit); jdk1.6.0_45; apache-ant-1.9.4 chrome+IE

Can anyone help me ? Thanks in advance!

Is it possible to modify database and then deploy sensors ? I got several errors occured when I tried to modify database and run the sensors again, even when I had default value of the sensors initialized to NULL.

Let's say I wanted to use a third-party API as a data source, for example, Weather Underground. This API accepts GET requests and produces JSON in return (for the most part). Is there a way I could define a virtual sensor that can connect to an API and accept JSON data? I suppose this might be similar to the CSV Wrapper. Would the HTTP GET wrapper be appropriate for this?

Thanks!

The class ContainerInfoHandler is responsible for returning the last values for each virtual sensors as an xml file. But if several tuples have their timestamp equal to the max(timestamp), they will be merged and there will be duplicated fields with no way to group them.
(Moreover the webinput fields will be duplicated too)

Imported from: https://sourceforge.net/apps/trac/gsn/ticket/92

It should list all sensors in a csv table

I noticed that if I create a CSV virtual sensor and use the field names "latitude" and "longitude," then on the web page the actual <addressing> values for latitude and longitude get shown next to the data values. For example, when the virtual sensor is first loaded

But after the first auto-refresh occurs it shows this

I used 0.0, 0.0 as my latitude/longitude in the <addressing> section of the virtual sensor. All of the data is recorded in the database properly, so it's just an issue in the view. Is there a simple solution for this? Thanks!

The CSV wrapper is currently used primarily to read data acquired by expensive Campbell logger software. An open source version of this software has been created: http://engineering.arm.gov/~choudhury/PbCdlComm/

This should be converted into a GSN wrapper, removing a step in the acquisition chain.
(issue copied from SourceForge)

Use of reliable message queuing system to achieve non blocking communication inside GSN

Hi,

Postgres 9.4 has recently added support for jsonb datatype which stores entire json data as a blob and supports querying (using enhanced SQL)on nested data fields in the JSON object. I was thinking on adding such a SQL query to virtual sensor to filter out required fields. If we have to achieve this, I believe we have to modify major part of storage layer of GSN. Do you think this is feasible. Any thoughts on this will be helpful.

Thanks.

Virtual sensors may generate several output streams, from one single wrapper. (for example when the wrapper serves as endpoint for multiple sensors). But currently only a single stream is defined as output of a VS.

my suggestion is to define the different output streams in the VS xml files by giving a "name" attribute to each element. Then the addressing would typically be virtualsensor_name/outputstream_name.

Some function will also need to be added an extra parameter like containerImpl.publishData or VirtualSensorDataListener.consume for example.

Feel free to comment if you find a nicer way to implement it...

how can I deploy a new virtual sensor into gsn ? How can I get the values ? Can someone please post the commands required for this ?

Currently the graph dependency is built in the gsn.beans.Modifications class, based on hard-coded wrappers and parameters of those. Adding new communication methods (like zeromq) needs then to modify this.
A workaround was implemented in the virtual sensor loader, by loading files in the lexical order if not contradicting the graph from Modifications.
But to make it explicit, one could add field in the vs xml file and then the dependency graph could be built from that. Of course then the ordering is let to the user, who can then make mistakes...
The virtual sensors loader can also be resilient enough to re-try loading vs that failed until all are launched, independently of the order.

Can we measure reliability of temperature values given by a sensor in GSN ? What is the possible way ?

I'm trying to set up the ZeroMQ and I have both memoryDataVS.xml and zeromq.xml in my virtual-sernsors directory; enabled zeroMQ in the conf/gsn.xml and setup the local version of zeromq.xml file according to https://github.com/LSIR/gsn/wiki/ZeroMQWrapper What I get after running ant gsn is:

 [java] ERROR [2014-11-17 22:32:02,983] [VSensorLoader-Thread0] (VSensorLoader.java:172)     - Unable to load VSensor zeromq, retrying later...
 [java] java.lang.NullPointerException
 [java]     at gsn.storage.StorageManager.executeCreateTable(StorageManager.java:457)
 [java]     at gsn.VSensorLoader.createWrapper(VSensorLoader.java:422)
 [java]     at gsn.VSensorLoader.prepareStreamSource(VSensorLoader.java:436)
 [java]     at gsn.VSensorLoader.createInputStreams(VSensorLoader.java:392)
 [java]     at gsn.VSensorLoader.loadPlugin(VSensorLoader.java:203)
 [java]     at gsn.VSensorLoader.loadPlugin(VSensorLoader.java:170)
 [java]     at gsn.VSensorLoader.run(VSensorLoader.java:118)
 [java]     at java.lang.Thread.run(Thread.java:745)
 [java] java.lang.NullPointerException

Any Ideas why? NullPointerException happens when creating the table because the structure is null, The structure is set this way:
structure = kryo.readObjectOrNull(new Input(new ByteArrayInputStream(rec)),DataField[].class);

Is there some additional setting not mentioned in the documentation? Thank you!

It is not possible to log out of the web interface under a secure connection. Users have to switch to a http connection to log out.

See OpenIotOrg/openiot#48

I find now that when stopping GSN (ant stop), there is an error if any
virtual sensor using local wrapper or either of the remote wrappers.

I can load memoryDataVS.xml and freeDiskSpace-NeedsJDK6.xml and the stop
script executes without error.

However if any of:

• diskSpacePlotVS-NeedsJDK6.xml (wrapper="remote-rest">),
• memoryPlotVS.xml (wrapper="local">),
• remote_rest_test.xml (wrapper="remote-rest">) or
• remote_push_test.xml (wrapper="remote">)
  Are loaded when the stop script is executed, then errors are generated.
  There is some difference in the errors as can be seen in the attached log
  file which was recorded while I started and stopped GSN with different VS's
  loaded.

I guess that errors when stopping are not critical but they may be
symptomatic of something more serious.

Imported from here: https://sourceforge.net/apps/trac/gsn/ticket/44
Files are on SourceForge Issue tracker: https://sourceforge.net/apps/trac/gsn/ticket/44

Request to read in the data using the grid wrapper and write both the grid binary object and the rendered image in the same virtual sensor.

when acquiring values from certain sensors it can be meaningful to process them as unsigned and store them this way to save space. Some tricks can be used to map for example unsigned int to long, and use them instead.
Java, before version 8, doesn't support unsigned numbers. But upgrading to java 8 would also need some work.

I installed the GSN server using the gsn_1.1.8_all.deb file. In the /var/log/gsn.log i get some informations about how the server works but no DEBUG. Is there any way to enable debuging ?

Tnx a lot in advanced

I think that there is a bug in the time based sliding mechanism.

With any slide value (count or time based) other than "1" I get the error shown in Error.log when starting the VS deg_days.xml.

Imported from: https://sourceforge.net/apps/trac/gsn/ticket/76

Either multiple scales or arithmetic operations in web
interface query operations to allow parameters to be plotted on the same scale. (nick)

Hello,

I am trying to write a virtual sensor using the CSV Wrapper, and one of the fields of my data is a timestamp in epoch format. I see that in CSVHandler.java you use the forPattern(String pattern) function to parse a date format given in the virtual sensor description XML file. For example:

<predicate key="formats">numeric,timestamp("yyyy-MM-dd HH:mm:ss.SSS"),...</predicate>

However, the problem is that I don't know what pattern to use to allow for the input of an epoch time number such as 1318316400. Is it possible to use the epoch timestamp as a parameter to the DateTime() constructor, as in this answer?

Basically, how can I use epoch time as an input for timestamps in GSN?

Thanks

Adding a script in the installer to start the gsn in the debug mode. In this way the user doesn't need to edit the log4j.properties file to change the logging level to debug.

I have spotted sporadic stream element duplicates in our GSN database. The only value which differs within a duplicate, are the TIMED entries. They differ in a few milliseconds. All other values are exactly the same and are distinctive duplicates. The VSs in which I have spotted these duplicates are using the local wrapper connected to one other VS and the scriptlet processor with a simple time conversion groovy script.

Do you have ever met the same problem or have any clue what could cause this duplicates?

Hello, I am having a little trouble setting up a timestamp field to be correctly parsed from a CSV file. My description file looks like this

<virtual-sensor name="green_button_energy" priority="10" publish-to-lsm="true">
    <processing-class>
        <class-name>gsn.vsensor.BridgeVirtualSensor</class-name>
        <output-structure>
            <field name="period_start" type="string" />
            <field name="period_end" type="string" />
            <field name="usage_KWh" type="double" />
        </output-structure>
    </processing-class>
    <description>SCE Green Button Energy Data</description>
    <life-cycle pool-size="10"/>
    <addressing>
        <predicate key="geographical">Test</predicate>
        <predicate key="LATITUDE">100.0</predicate>
        <predicate key="LONGITUDE"> -100.0</predicate>
    </addressing>
    <storage history-size="24h" />
    <streams>
        <stream name="input1">
            <source alias="source1" sampling-rate="1" storage-size="1">
                <address wrapper="csv">
                    <predicate key="file">data/green_button_energy.csv</predicate>
                    <predicate key="fields">period_start, period_end, usage_KWh, Quality</predicate>
                    <predicate key="formats">timestamp(y-M-d H:m:s),timestamp(y-M-d H:m:s),numeric, string</predicate>
                    <predicate key="bad-values">NaN,6999,-6999,null</predicate>
                    <predicate key="timezone">Etc/GMT-8</predicate>
                    <predicate key="sampling">1000</predicate>
                    <predicate key="check-point-directory">csv-check-points</predicate>
                    <predicate key="skip-first-lines">1</predicate>
                    <predicate key="separator">,</predicate>
                </address>
                <query>select * from wrapper
                </query>
            </source>
            <query>select period_start, period_end, usage_KWh from source1</query>
        </stream>
    </streams>
</virtual-sensor>

But when I start my GSN server I get this error:

[java] java.lang.RuntimeException: The streamElement produced by :green_button_energy Virtual Sensor is not compatible with the defined streamElement.
[java] The expected stream element structure (specified in /my/path/gsn/virtual-sensors/green_button_energy.xml is [period_start (Varchar) , period_end (Varchar) , usage_kwh (Double) , ] but the actual stream element received from the green_button_energy has the [PERIOD_START(BigInt),PERIOD_END(BigInt),USAGE_KWH(Double), ] thus the stream element dropped !!!

I can see that it is complaining about the mismatch of types where I defined the formats of the wrapper and where I defined the output structure. It seems like when I define the wrapper with
<predicate key="formats">timestamp(y-M-d H:m:s),...</predicate> it actually stores this as type bigint in the database.

My question is, how can define the timestamps such that in the output structure I can also see it in the "y-M-d HH:mm:ss" format? What "type" should I use in the line <field name="period_end" type="string" />?

Depending when it fails in the Virtual sensor loader (https://github.com/LSIR/gsn/blob/master/src/main/java/gsn/VSensorLoader.java#L196), the initialize method of wrapper may be called, but never the dispose method. If those are used for allocating and releasing resources, it may lead to memory leaks...
One such case is the monitoring.

Some of the new features can be added there and at the same time correcting this error which was reported today:

" The 'data stream processing and time model' described in pages 9-14 of the Book of GSN states that the listing 2.2 on page 11 is for the 'police station' example and 2.3 for the 'insurance' example.
However from the text itself I get an impression that 2.2 is for insurance and 2.3 for police station examples."

Currently the interface to the different databases backend is through the StorageManager. It was built assuming SQL databases (mySQL, postgresQL,...), offering functions that takes SQL queries as input and JDBC drivers are doing the connection to the databases.
But if we want to introduce NoSQL backends, like HBASE, or even memcached, the process of understanding the SQL query and performing it is done at the JDBC driver level (like Apache Phoenix project). Having such adapters would work out almost of the box, but when looking at the whole streaming process would introduce a huge and useless overhead.
Indeed, a lot of functions are just building SQL queries to be able to interact with the DB, but if we know that the DB is just a K-V store, and that the query will just need to be parsed in the next function call, why still using SQL internally?
SQL is very handy for using in the VS configuration files and should be kept here, but it should be parsed much sooner and replaced by an abstraction that can be efficiently passed to SQL and NoSQL backends.

This is just a suggestion and it is not clear for me yet at which level this should be implemented (JDBC, StorageManager, SlidingTimeWindows, VSConfig) ?

I have the following problem with the Unix version :
bash: ./gsn-start.sh: /bin/bash^M: bad interpreter: No such file or directory
Thank you!

What is the unit of time measured by sensors in GSN ? I get something like this ?

Hello,
What are the functionalities available in Tiny-GSN? Can it push data to XGSN? Can it sense data semantically? Read me file does not give all these details. Thanks.

How can we implement a multiple value sensing sensor in GSN other than the example ones given in the GSN. I would like to deploy a sensor that measures pressure as well as temperature.

make a small change to the GML Handler
(https://gsn.svn.sourceforge.net/svnroot/gsn/trunk/src/gsn/http/GMLHandler.java)
In order to prevent future sensor misconfigurations where people can put spaces in the
content of the latitude and longitude elements,

suggested solution by user :

to the code reading these values. You can do for example:

for (KeyValue df : sensorConfig.getAddressing()) {
if
(StringEscapeUtils.escapeXml(df.getKey().toString().toLowerCase()).contentEquals("latitude"))
lat = (new
String(StringEscapeUtils.escapeXml(df.getValue().toString()))).trim();
if
(StringEscapeUtils.escapeXml(df.getKey().toString().toLowerCase()).contentEquals("longitude"))
lon = (new
String(StringEscapeUtils.escapeXml(df.getValue().toString()))).trim();
}

or whatever you think is more appropriate to remove the leading and
trailing spaces from the values of those coordinates.

And because I can also assume that people might put non-numeric
characters inside the coordinates by mistake (for example the letter
"o" instead of "0"), you can do this also:

for (KeyValue df : sensorConfig.getAddressing()) {
if
(StringEscapeUtils.escapeXml(df.getKey().toString().toLowerCase()).contentEquals("latitude"))
lat = (new
String(StringEscapeUtils.escapeXml(df.getValue().toString()))).trim();
if (lat != null && (!(lat.matches("-?\d+(.\d+)?"))))
lat = null;
if
(StringEscapeUtils.escapeXml(df.getKey().toString().toLowerCase()).contentEquals("longitude"))
lon = (new
String(StringEscapeUtils.escapeXml(df.getValue().toString()))).trim();
if (lon != null && (!(lon.matches("-?\d+(.\d+)?"))))
lon = null;
}

Please check the logic, because I didn't compile nor tested the above
code. I have used the pattern "-?\d+(.\d+)?" because latitude and
longitude can also be negative in other parts of the world outside
Europe, hence the optional "-".

can use this java library: https://github.com/okleine/nCoAP for example

How can I get the location value into the database in GSN ?

I had noticed something a bit strange about the way that ChartVS portrays data that includes null values.

In the screenshot, you will see that the plot for V178 which is pretty much complete starts at a bit after midnight while the other two plots which have many null values (node batteries nearly flat) go back to the previous midnight.

The configuration of the VS is attached (cis_temp_chart.xml).

The source of the data is another VS that saves the temperature values from each of the six nodes in the network. (The gsn.cis_room_temp data is exported in the xls).

I set 216 so that the chart showed 36 hours of data (10 minute intervals). The node batteries are getting low and I am now seeing a lot of null values for some nodes. (I had set in cis_room_temp as well thinking that I needed to retain the data for the chart VS but now realise that probably isn’t necessary.

I haven’t looked at how the charting routine works but it seem it does not count the null values in the data it retains so that the data for nodes V145 and V144 is retained back until it has 216 non null values. Note that the time scale is more compressed before the start of the V178 data.

Would it be possible to make the chart processing class set a time range for history-size instead of a number of records?
--- Imported from https://sourceforge.net/apps/trac/gsn/ticket/86
File: https://sourceforge.net/apps/trac/gsn/attachment/ticket/86/CIS%20Room%20Temperatures.xlsx

Hi All,

I have been trying to build GSN from the source code and every time it fails with the below error. I am facing this issue since 10 days. It used to work fine before that. It will be helpful if someone can help me resolve this issue.

Steps to reproduce the issue:

1. On a fresh box/VM, Install java, ant and other packages required for GSN.
2. git clone https://github.com/LSIR/gsn.git
3. cd gsn
4. ant
5. ant gsn

Build log:

[artifact:dependencies] org.apache.geronimo.specs:geronimo-stax-api_1.0_spec:jar:1.0.1
[artifact:dependencies]
[artifact:dependencies] from the specified remote repositories:
[artifact:dependencies] apache.snapshots (http://repository.apache.org/snapshots),
[artifact:dependencies] central (http://repo1.maven.org/maven2),
[artifact:dependencies] TypesafeRepository (http://repo.typesafe.com/typesafe/releases/),
[artifact:dependencies] LSIR (http://osper.epfl.ch:8081/artifactory/gsn-release/)
[artifact:dependencies]
[artifact:dependencies] Path to dependency:
[artifact:dependencies] 1) gsn:gsn:jar:1.1.5
[artifact:dependencies] 2) org.apache.axis2:axis2-adb:jar:1.5.5
[artifact:dependencies] 3) org.apache.axis2:axis2-kernel:jar:1.5.5
[artifact:dependencies] 4) org.apache.ws.commons.axiom:axiom-api:jar:1.2.11
[artifact:dependencies]
[artifact:dependencies]
[artifact:dependencies] Not a v4.0.0 POM. for project org.apache.geronimo.genesis.config:config at /root/.m2/repository/org/apache/geronimo/genesis/config/config/1.1/config-1.1.pom

BUILD FAILED
/home/ec2-user/gsn/build.xml:480: Unable to resolve artifact: Unable to get dependency information: Unable to read the metadata file for artifact 'org.apache.geronimo.specs:geronimo-stax-api_1.0_spec:jar': Cannot find parent: org.apache.geronimo.genesis.config:config for project: null:project-config:pom:1.1 for project null:project-config:pom:1.1
org.apache.geronimo.specs:geronimo-stax-api_1.0_spec:jar:1.0.1

from the specified remote repositories:
apache.snapshots (http://repository.apache.org/snapshots),
central (http://repo1.maven.org/maven2),
TypesafeRepository (http://repo.typesafe.com/typesafe/releases/),
LSIR (http://osper.epfl.ch:8081/artifactory/gsn-release/)

Path to dependency:
1) gsn:gsn:jar:1.1.5
2) org.apache.axis2:axis2-adb:jar:1.5.5
3) org.apache.axis2:axis2-kernel:jar:1.5.5
4) org.apache.ws.commons.axiom:axiom-api:jar:1.2.11

Total time: 1 second

.

Thanks,
Varun

Current version of the remote wrapper has some difficulties recovering when connection is lost.
A system based on Message Queues should be implemented, using existing and tested solutions, such as RabbitMQ, ZeroMQ,... (cf. http://queues.io/)
This can also be linked to the solution to the internal mesages queues.