ELK 5.6.0.1 Kibana [illegal_argument_exception] about kts5 HOT 22 CLOSED

@atbohmer - thank you for the testing and feedback - all changes are not merged to master.

from kts5.

I think you have inverted the steps in described in the Readme - you need to do -

find KTS5/dashboards/ -type f -exec sed -i -e 's/.raw/.keyword/g' {} ;

after

./load.sh

from kts5.

I inverted indeed because you need to alter the files before you load them into the elasticsearch index in my opinion. Could you please explain? Thanks!

from kts5.

oh - of course - you are correct, sorry about that.
On which dashboards do you get that warning?
Didi you restart both Kibana and ES ?

from kts5.

No prob, great you responded so quick! 👍
After deleting the Kibana index, Kibana is running well, I'm able to browse the other indices via Discover, until I 'insert' the dashboards via the load.sh script. Then the Status: Red pops up :

ui settings : Elasticsearch plugin is red
plugin:[email protected] : [illegal_argument_exception] mapper [hits] cannot be changed from type [long] to [integer]

This worked well with version 5.5.

from kts5.

Do you get an error during the execution of the load script?

from kts5.

Indeed!
Loading dashboard SN-TLS:

• curl -H 'Content-Type: application/json' -XPUT http://127.0.0.1:9200/.kibana/dashboard/SN-TLS -d @dashboards/dashboard/SN-TLS.json
  {"error":{"root_cause":[{"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [hits] within [dashboard] is not allowed"}],"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [hits] within [dashboard] is not allowed"},"status":400}+ echo

from kts5.

Seemed a coincidence: after removing the kibana index , find .. , load.sh now does not generate an error but still the RED kibana status.

from kts5.

So is there any err msg ? (including in the logs)

from kts5.

This is what elasticsearch is logging:
[2017-09-12T15:11:15,988][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [server] failed to put mappings on indices [[[.kibana/hmuGb-NmTMWXk6Y_P32vcA]]], type [timelion-sheet]
java.lang.IllegalArgumentException: mapper [hits] cannot be changed from type [long] to [integer]
If you like complete logging, please specify which and I will send them per email.

from kts5.

@atbohmer - do you use the provided template - https://github.com/StamusNetworks/KTS5/tree/master/es-template ?

from kts5.

Yes I do. Just upgraded a CentOS 7 test VM which I used to build the initial setup, and same shit happens. Maybe they broke something in Kibana from 5.5 to 5.6? A regression bug maybe?

from kts5.

Some extra elasticsearch logging info from the test VM:
[2017-09-12T16:26:50,340][INFO ][o.e.c.r.a.AllocationService] [server] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[logstash-tls-2017.07.26][0], [logstash-tls-2017.07.26][1]] ...]).
[2017-09-12T16:26:50,357][INFO ][o.e.c.m.MetaDataMappingService] [server] [.kibana/TXwOG9JdRcuJjDKl7oqCRw] create_mapping [default]
[2017-09-12T16:26:50,382][INFO ][o.e.c.m.MetaDataMappingService] [server] [.kibana/TXwOG9JdRcuJjDKl7oqCRw] create_mapping [url]
[2017-09-12T16:26:50,418][INFO ][o.e.c.m.MetaDataMappingService] [server] [.kibana/TXwOG9JdRcuJjDKl7oqCRw] create_mapping [server]
[2017-09-12T16:26:50,450][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [server] failed to put mappings on indices [[[.kibana/TXwOG9JdRcuJjDKl7oqCRw]]], type [timelion-sheet]
java.lang.IllegalArgumentException: mapper [hits] cannot be changed from type [long] to [integer]
at org.elasticsearch.index.mapper.MappedFieldType.checkTypeName(MappedFieldType.java:151) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.index.mapper.MappedFieldType.checkCompatibility(MappedFieldType.java:163) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.index.mapper.FieldTypeLookup.checkCompatibility(FieldTypeLookup.java:128) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.index.mapper.FieldTypeLookup.copyAndAddAll(FieldTypeLookup.java:94) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:400) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:336) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:268) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:311) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.0.jar:5.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]

from kts5.

Found this thread https://discuss.elastic.co/t/kibana-discovery-and-timelion-page-are-blank/89092/13 , if i stop kibana, remove the index, start kibana, create a timelinion and save , all works well, Then I import the dashboards via load.sh en status goes to Red. But this error (for all dashboards) from the load.sh script seems now consistent when following this procedure:

• for file in '$DIR/dashboard/*.json'
  ++ get_name dashboards/dashboard/SN-VLAN.json
  ++ basename dashboards/dashboard/SN-VLAN.json .json
  ++ sed -e 's/ /%20/g'
• name=SN-VLAN
• echo 'Loading dashboard SN-VLAN:'
  Loading dashboard SN-VLAN:
• curl -H 'Content-Type: application/json' -XPUT http://127.0.0.1:9200/.kibana/dashboard/SN-VLAN -d @dashboards/dashboard/SN-VLAN.json
  {"error":{"root_cause":[{"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [hits] within [dashboard] is not allowed"}],"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [hits] within [dashboard] is not allowed"},"status":400}+ echo

from kts5.

yes - it seems a breaking change in Kibana's upgrade :)
I will investigate further and feedback.

from kts5.

@atbohmer - i just pushed a change, could you please retry the import?

from kts5.

Great but I'm afraid to no avail, same situation, status Red after running load.sh:
Loading dashboard SN-ALERTS:
{"error":{"root_cause":[{"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [hits] within [dashboard] is not allowed"}],"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [hits] within [dashboard] is not allowed"},"status":400}

Elasticsearch log:
[2017-09-12T19:56:52,609][INFO ][o.e.c.m.MetaDataDeleteIndexService] [ids server] [.kibana/2f1G10tMQpK6pjp5as_rUQ] deleting index
[2017-09-12T19:56:52,665][INFO ][o.e.c.m.MetaDataCreateIndexService] [ids server] [.kibana] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings []
[2017-09-12T19:56:52,743][INFO ][o.e.c.m.MetaDataMappingService] [ids server] [.kibana/8i25UxF3Qwq8n9GSOOQ47Q] create_mapping [index-pattern]
[2017-09-12T19:56:52,914][INFO ][o.e.c.m.MetaDataMappingService] [ids server] [.kibana/8i25UxF3Qwq8n9GSOOQ47Q] create_mapping [search]
[2017-09-12T19:56:53,033][INFO ][o.e.c.m.MetaDataMappingService] [ids server] [.kibana/8i25UxF3Qwq8n9GSOOQ47Q] create_mapping [visualization]
[2017-09-12T19:56:53,263][INFO ][o.e.c.m.MetaDataMappingService] [ids server] [.kibana/8i25UxF3Qwq8n9GSOOQ47Q] create_mapping [default]
[2017-09-12T19:56:53,288][INFO ][o.e.c.m.MetaDataMappingService] [ids server] [.kibana/8i25UxF3Qwq8n9GSOOQ47Q] create_mapping [config]
[2017-09-12T19:56:53,313][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [ids server] failed to put mappings on indices [[[.kibana/8i25UxF3Qwq8n9GSOOQ47Q]]], type [dashboard]
java.lang.IllegalArgumentException: mapper [hits] cannot be changed from type [long] to [integer]

from kts5.

@atbohmer - can you try the fix in the branch here:

• git clone -b loadscript-fix-v1 https://github.com/StamusNetworks/KTS5.git
• run the load.sh script as usual
• open Kibana's management settings, indexes - select logstash-* as default index
• systemctl restart elasticsearch logstash kibana
• try the dashboards

Feedback is welcome.

from kts5.

Yes! Working on the box already at 5.6.0.1 and also give it a try on the box running 5.5, run the 'new' load.sh, upgraded to 5.6 and it kept working! Seems you now also replace the .kibana mapping which is later on completed by Kibana?

from kts5.

@atbohmer - does it import ok the dashboards on 5.5 too?

from kts5.

from kts5.

check 1) A new Centos 7.4 VM, ELK 5.5.3 rpm's installed, setup config, importing data via logstash eve.jon file, starting Kibana, check working, stopped Kibana, git clone non fixed dashboards, load.sh, start Kibana, all well. Stopped Kibana, git pull fixed dashboards, load.sh, start Kibana, all well!

check 2) Again a new CentOS 7.4 VM, ELK 5.5.3 rpm's installed, setup config, importing data via logstash eve.jon file, starting Kibana, check working, stopped Kibana, git clone fixed dashboards, start Kibana, all well!

from kts5.