Support for index templates about logstash-output-elasticsearch HOT 6 CLOSED

This seems like an ideal solution for #25.

Instead of checking if a new template needs to be created with each event, create an index template that works for all events at startup.

from logstash-output-elasticsearch.

How can it know what all possible events are at startup time? That's a lot to expect.

from logstash-output-elasticsearch.

It doesn't need to!

There's definitely some setups that wouldn't work with this, depending on how the index/type is chosen for each event. But this would enable the following arrangement, which seems pretty common:

• You have daily indexes called [logstash-]YYYY-MM-DD that are automatically created when logstash stores events into them
• There's a standard schema for all the types of events I add to these indexes
• Logstash creates an index template that matches all indexes named logstash-*, with mappings for all the types that'll be stored (could default to the standard logstash ES mapping)
• You can now safely restart logstash with a new configuration and it'll update the index template to process any new mappings (once the index rolls over).

It's not a perfect solution, but the current system with manage_template only works when you have one giant index, not a series of timestamped ones. I've pretty much been doing this manually with index templates anyways, so it seems like managing it in logstash would be useful.

from logstash-output-elasticsearch.

@untergeek Have I misunderstood this? Is it possible to define a template at startup that'll cover all indices with a common prefix?

I was under the impression that it created a mapping for a specific index at startup if manage_template => true.

from logstash-output-elasticsearch.

The default template already matches any index that starts with logstash by way of logstash*.

manage_template is merely a true/false declaration. You can use multiple elasticsearch output blocks and conditionals to have different templates managed for different index destinations.

The point I was hoping to make was that if you know what your mapping will be, then there is no reason to have dynamic templating. Perhaps this discussion would benefit from a common understanding of terms used. When I say dynamic templating, I mean that individual elements of the Elasticsearch index mapping are handled in a deterministic way by the Elasticsearch output plugin, as in #25. This is not feasible because the template is uploaded before a single event is sent. This makes it completely impossible to do in a deterministic way because there are not, as yet, event strings from which to derive type or date elements. If we hacked the code to do this in a deterministic way, a call to elasticsearch would become necessary for each event (or state would have to be kept and compared against for every event), which would make Logstash-to-Elasticsearch indexing much, much slower.

from logstash-output-elasticsearch.

I see now, thanks for taking the time to explain this. I did misunderstand what logstash is doing here - it already does what I'm describing!

I was thinking that dynamic templating meant any templating in the index name, including the timestamp, and that the "template" in this context was in reality a "mapping" that was uploaded to the index along with the first event. This doesn't really make sense, and I can see now that it's actually uploading an index template.

Regarding dynamic templating as you describe it, I completely agree. The way logstash currently handles this makes sense, and I don't see a good reason to process templates on every event.

from logstash-output-elasticsearch.