Manual templates ignored when multiple indexes specified about logstash-output-elasticsearch HOT 12 CLOSED

I'm confused by the relative sameness of the last two examples. Also, you have some unnecessary options. For example, cluster is unnecessary if you're using protocol => "http", and localhost is the default host, so that line is also superfluous. In order to make this easier to read, I'm trying to simplify these.

Working

output {
  if "state" in [tags] {
    elasticsearch {
      index => "state-%{+YYYY.MM}"
      protocol => "http"
      template => "/state-index-template.json"
      template_overwrite => true
    }
  }
  if "logs" in [tags] {
    elasticsearch {
      index => "logs-%{+YYYY.MM}"
      protocol => "http"
      template => "/logs-index-template.json"
      template_overwrite => true
    }
  }
}

Not Working

output {
  if "logs" in [tags] {
    elasticsearch {
      index => "logs-%{+YYYY.MM}"
      protocol => "http"
      template => "/logs-index-template.json"
      template_overwrite => true
    }
  }
  if "state" in [tags] {
    elasticsearch {
      index => "state-%{+YYYY.MM}"
      protocol => "http"
      template => "/state-index-template.json"
      template_overwrite => true
    }
  }
}

When I simplify these, the only difference is the order. Are you suggesting that the order affects the outcome? That in your case, if state comes before logs, that templates are properly applied, but not the if logs comes before state?

from logstash-output-elasticsearch.

Sorry... I dumped the config from my vagrant build which I use for development. Similar behavior though is seen on my cluster. I kept the config very similar to my cluster so there is a lot of superfluous data.

My cluster outputs to 4 different indexes. I dropped down to two indexes to test.

The order does affect the outcome and I am not totally sure why.
What I have found, if I send an event through logstash with tag "logs". If "logs" is at the bottom of the list of elasticsearch outputs, the template is applied. If it is at the top, the template isn't applied.

from logstash-output-elasticsearch.

Have you considered using an else if in between the if blocks?

output {
  if "logs" in [tags] {
    elasticsearch {
      index => "logs-%{+YYYY.MM}"
      protocol => "http"
      template => "/logs-index-template.json"
      template_overwrite => true
    }
  } else if "state" in [tags] {
    elasticsearch {
      index => "state-%{+YYYY.MM}"
      protocol => "http"
      template => "/state-index-template.json"
      template_overwrite => true
    }
  }
}

This would force it to be either/or.

from logstash-output-elasticsearch.

Yes. I tried that as well. Same failures.

from logstash-output-elasticsearch.

Okay. We'll look at what's going on here. Thanks for the configs. We will see if we can replicate this.

from logstash-output-elasticsearch.

out of curiosity, can you paste in the contents of the templates?

from logstash-output-elasticsearch.

I apologize but I got wrapped up in a different project and didn't see the response till now. I will post the template tomorrow.

from logstash-output-elasticsearch.

Here is the template. All templates are the same except for template name.

{
"template" : "logs-_",
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"default" : {
"date_detection" : false,
"all" : {"enabled" : true},
"dynamic_templates" : [ {
"message_field" : {
"match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "not_analyzed", "omit_norms" : true
}
}
}, {
"string_fields" : {
"match" : "",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fields" : {
"raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
}
}
}
} ],
"properties" : {
"@Version": { "type": "string", "index": "not_analyzed" },
"@timestamp" : { "type" : "date" },
"logtime" : { "type" : "string", "index" : "not_analyzed" },
"logDate2" : { "type" : "string", "index" : "not_analyzed" },
"geoip" : {
"type" : "object",
"dynamic": true,
"path": "full",
"properties" : {
"location" : { "type" : "geo_point" }
}
}
}
}
}
}

from logstash-output-elasticsearch.

Having more or less the same issue.
Using logstash-rc2 & ES 1.5.0-1
Started with default elasticsearch output configuration section:

output {
    elasticsearch { 
      host => localhost
      protocol => http
    }
}

For testing purposes I added an check on a field to output to a specific index:

 output {
  if [ES_dest] == "dfit" {
    elasticsearch { 
      host => localhost
      protocol => http
      index => "syslog-dfit-%{+YYYY.MM.dd}"
    }
  }
}

Removed that check and all syslog-dfit indexes and let all events flow to default index (logstash) template again. That is what I thought would happen.
But when I check: curl -XGET 'http://localhost:9200/_template?pretty' again I see:

 {
  "logstash" : {
 "order" : 0,
  "template" : "syslog-dfit-*",
"settings" : {
  "index.refresh_interval" : "5s"
},
"mappings" : {
  "_default_" : {
    "dynamic_templates" : [ {
      "message_field" : {
        "mapping" : {
          "index" : "analyzed",
          "omit_norms" : true,
          "type" : "string"
        },
        "match_mapping_type" : "string",
        "match" : "message"
      }
    }, {
      "string_fields" : {
        "mapping" : {
          "index" : "analyzed",
          "omit_norms" : true,
          "type" : "string",
          "fields" : {
            "raw" : {
              "index" : "not_analyzed",
              "ignore_above" : 256,
              "type" : "string"
            }
          }
        },
        "match_mapping_type" : "string",
        "match" : "*"
      }
    } ],
    "properties" : {
      "geoip" : {
        "dynamic" : true,
        "path" : "full",
        "properties" : {
          "location" : {
            "type" : "geo_point"
          }
        },
        "type" : "object"
      },
      "@version" : {
        "index" : "not_analyzed",
        "type" : "string"
      }
    },
    "_all" : {
      "enabled" : true
    }
  }
},
"aliases" : { }
  }
}

Seems the template name remains syslog-dfit-* and is not updated to logstash-*
Or no new "logstash" template is added. Hope this can help in solving the problem.

from logstash-output-elasticsearch.

I hit a similar issue (template not being uploaded by logstash) until I set "manage_template => true", then it started working. I don't see "manage_template" mentioned above.

from logstash-output-elasticsearch.

Because I caught the same issue, here is an answer.

I think, you forgot to set template name. By default, the "logstash" name is used. So, one template will overwrite another one, and then again (when rotating) and again...

output {
  if "logs" in [tags] {
    elasticsearch {
      index => "logs-%{+YYYY.MM}"
      protocol => "http"
      template => "/logs-index-template.json"
      template_overwrite => true
      template_name => "logs-index-template"    # this is the trick
    }
  } else if "state" in [tags] {
    elasticsearch {
      index => "state-%{+YYYY.MM}"
      protocol => "http"
      template => "/state-index-template.json"
      template_overwrite => true
      template_name => "state-index-template"    # this is the trick.
    }
  }
}

from logstash-output-elasticsearch.

@KIVagant seems to have the right answer. I'm going to close this unless anyone has anything new here. If so, please open a new issue.

from logstash-output-elasticsearch.