mkocikowski / estools Goto Github PK

after upload, record metrics (data size, time) as a single line, possibly appending to a log file, like siege does.

downloads a ton of crap, user should have the option to install pyrax, and, if it is installed, then then rack cloud download functionality should be available

If alias is specified, then when new index is created and the alias is re-pointed to it, close the old index.

out of date.

Following an upload, the sequence was: close the 'old' index, remove the alias, assign the alias to new index. This doesn't work with 2.x: after the index is closed, the alias can't be removed, and so it then points to multiple indexes. So the solution here is to not close the old index (hack, but works; the tools will be rewritten from the ground up any way soon-ish). Version bump to 0.3.3

when running esdump against an elasticseach5 cluster I get the following output:

Traceback (most recent call last):
  File "/usr/local/bin/esdump", line 9, in <module>
    load_entry_point('estools==0.3.4', 'console_scripts', 'esdump')()
  File "/usr/local/lib/python2.7/dist-packages/estools/dump/dump.py", line 54, in main
    for doc in run(params=args, session=requests.Session()):
  File "/usr/local/lib/python2.7/dist-packages/estools/common/api.py", line 164, in scan
    url, response = _scan_query(params=params, query=query)
  File "/usr/local/lib/python2.7/dist-packages/estools/common/api.py", line 30, in wrapper
    raise RuntimeError("status code %i returned by es call" % response.status_code)
RuntimeError: status code 400 returned by es call

When an input file reaches EOF / closes for some other reason, there should be an option to reopen / retry that file; I need that for reading from named pipes, where the producer can terminate, and then another producer can start writing - I want the loader to have the option to continue reading from that file. Specifically, this comes up for me when a kafka consumer is writing to a named pipe, and it dies / quits / another consumer comes up.

Dump data and / or mapping. Make it so that it can be piped straight into 'esload' for easy copying of indexes.

This lets you specify which of document's fields should be used as the doc id. It should be --id-field ('path' is a misnomer).

With --mapping-path flag. This would be mutually exclusive with --id-field. First do #14

when the openstack env variables are not defined, fails to run. fix it.

Default should be 'leave as is' but it should be possible to specify cleanup - delete the index.

Ability to set an alias to the index after successful upload. --alias flag?

Ability to read straight from cloudfiles

https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html#segments-and-merging

https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html#segments-and-merging

Make this a simple, single purpose tool. Doesn't matter where the data is coming from / going to: this is what pipes are for.

Need --batch-size and --optimize-segments flags. For now, --follow should disable batching / optimization, but down the line these should work together.

Be able to connect to elastic hosts over https using basic auth

Stdin reads json objects, not file names. Keep it simple.

Flexible mapping: one input to many indexes, many inputs to one index. Depends on #12