Error writing JSON points to InfluxDB about influxdb-java HOT 15 CLOSED

Hi Dennis

I think this is related to #54 , can you please update your application to use the latest head of influxdb-java and retest.

Greetings
Stefan

from influxdb-java.

Hi Stefan,

thanks for the hint. Give me a day to test it and I'll get back to you.

Greetings
Dennis

from influxdb-java.

Hi Stefan,

unfortunately it is not working, I am still getting the same error. I tried it with an sqlite database containing 20 entries and another one containing 5000 entries (strings containing json of BatchPoints).

Greetings
Dennis

from influxdb-java.

And you did drop all existing data in influxdb before rerunning the tests ?
Just to be sure

from influxdb-java.

Yes, I added a new database. But it seems like it was an error in my application, because after a few more optimizations it is working now. I'm gonna test it in detail in the next days and reopen the case if any problems are occuring.

Thanks!

from influxdb-java.

Hu,
thanks dennis

from influxdb-java.

Hi Stefan,

sorry to reopen again, but I experienced some very strange behaviour while further testing. I am collecting sensor data, containing one field (named value, a float number) and a different set of tags. Even data from the same sensor is sometimes working and sometimes not, but without any pattern (or let's say no pattern I could recognize). If you'd like, I could post some examples, but it is a lot of text. Any idea how to narrow down the problem?

Greetings
Dennis

from influxdb-java.

Hi Dennis,

I think the problem is because you store float's, if the float is a small number say 10.2 it will be sent to influxdb as 10.19999983 for example, and influxdb stores this value as a float internally.

If you try to store a float, 10002.23 for example a 10002 is sent to influxdb and it will store it as a int64.
So whatever comes first, the following write will not match in terms of type and i can image that this is the root cause of the problems you see.

From my perspective, using float's to be stored is a bad idea, try to convert them to a BigDecimal or at least a double.

Greetings
Stefan

from influxdb-java.

HI Stefan,

thanks, but that was unfortunately not the solution. I changed my values to BigDecimal, but without any difference. The error is still happening from time to time (same error). The way I converted from Float to BigDecimal is the following:

BigDecimal bigDecimal = new BigDecimal(floatNumber.toString());

Same result for doubles. But it is happening less in the following order:
float > BigDecimal > double

Is it probably a solution to add a possibility to define values type when adding fields to points?

Greetings
Dennis

from influxdb-java.

Hi Dennis,

actually the lineprotocoll does not have any possibility to specify the type of the given value.
Can you please open a ticket at influxdb, referencing this one and ask how to solve this kind of problem.

Greetings
Stefan

from influxdb-java.

Hi Stefan,

the guys from influxdb confirmed (influxdata/influxdb#3305) what I already guessed yesterday: the error is causes by first writing an integer and then a float/BigDecimal/double or the other way round. So I did some further research and think the problem is based on NumberFormat in concatenateFields() of Point.java. When you execute the following code you are getting results like in the comment of each line.

NumberFormat numberFormat = NumberFormat.getInstance(Locale.ENGLISH);
numberFormat.setMaximumFractionDigits(340);
numberFormat.setGroupingUsed(false);

System.out.println(numberFormat.format(10)); // 10
System.out.println(numberFormat.format(10.0)); // 10
System.out.println(numberFormat.format(10.1)); // 10.1

So the problem is that decimals with 0 after the decimal separator are written to line protocol as integers. If the next point now contains decimals after the separator I'm getting the error. Therefore I've rewritten the the concatenateFields() method:

private StringBuilder concatenateFields() {
        final StringBuilder sb = new StringBuilder();
        final int fieldCount = this.fields.size();
        int loops = 0;

        NumberFormat numberFormat = NumberFormat.getInstance(Locale.ENGLISH);
        numberFormat.setMaximumFractionDigits(340);
        numberFormat.setGroupingUsed(false);
        numberFormat.setMinimumFractionDigits(1);

        for (Entry<String, Object> field : this.fields.entrySet()) {
            sb.append(KEY_ESCAPER.escape(field.getKey())).append("=");
            loops++;
            Object value = field.getValue();
            if (value instanceof String) {
                String stringValue = (String) value;
                sb.append("\"").append(FIELD_ESCAPER.escape(stringValue)).append("\"");
            } else if (value instanceof Integer || value instanceof Long) {
                sb.append(numberFormat.format(value));
            } else if (value instanceof Number) {
                numberFormat.setMinimumFractionDigits(1);
                sb.append(numberFormat.format(value));
                numberFormat.setMinimumFractionDigits(0);
            } else {
                sb.append(value);
            }

            if (loops < fieldCount) {
                sb.append(",");
            }
        }
        return sb;
    }

I did some overnight testing with about 1.2 million points and didn't get a single error. So I think this could be the solution.

Greetings
Dennis

from influxdb-java.

@majst01 Somehow current master (526fe08) contains only partial changes from last comment of @DennisBauer. Which means that current master does not handle properly case when value is of int type(eg.: if value is 1, it will get converted to 1.0).
So 526fe08 does not close #55.

from influxdb-java.

Hi,

what kind of problem do you have with 1.0 stored for a int which was 1 ?

from influxdb-java.

My point is that if user of library passes integer, library must guarantee that the type is preserved. The same for float. It looks like code of @DennisBauer in the last comment supports that (to be unit-tested, though).
https://influxdb.com/docs/v0.9/query_language/spec.html

from influxdb-java.

Hi,

current master is b70c1ad and this include the commit you are missing, this was 9 days ago. Why do you think 526fe08 is head of master ?

So please check out the actual head from master and you are done.
Greetings
Stefan

from influxdb-java.