Credo 0.6.0 has been released and so this library should be updated to use it. This requires adding @spec definitions to all the functions, and probably a few other small things.

I do not personally have the time for this, so I'm opening this issue in case someone wonders why this hasn't been done yet.

Not sure how common a use case this is, but this library looks great for my needs other than one small problem, our model doesn't have one index, but per-company indexes... so to use the Document module, we can't define @es_index at the module level, but invocation time of methods... I can work on a PR to add this capability if you're open to it, or if you have time and know how you'd like it done that would be awesome! (could do same for @es_type but that seems less likely to be invocation independent...)

I have an app in JAVA using Elasticsearch queries and when experimenting with Elixir / Phoenix I found some significant performance decrease. The detailed code in Phoenix controller is not significant, it is just simple controller doing one ES call. To compare I used Elastic.HTTP and HTTPotion directly to see difference. Here are results (tested by Apache Benchmark)

a) HTTPotion.get("http://localhost:9200/hon_cz/catalog/_search")
460 req / sec

no failed reuqests

b) Elastic.HTTP.get("http://localhost:9200/hon_cz/catalog/_search")
330 req/sec

Using ElasticHTTP I can even see many failed request:
Complete requests: 1000
Failed requests: 41
(Connect: 0, Receive: 0, Length: 41, Exceptions: 0)

Without any HTTPPotion or Elastic call in controller I can get 1200 req/sec so Phoenix is quite fast ... so just one line of Elastic.HTTPP makes req/sec drop from 1200 to 330, which is significant especially when ES queries say it took 2ms for ES to find data ... ES doesnt seem to be bottleneck here as my former Java code can make 800 req/sec with the same ES call. Any idea whats wrong here?

When I do indexing like this:

ProductDocument.index(product.id, data)

every now and then, I get this error:

** (FunctionClauseError) no function clause matching in Elastic.ResponseHandler.process/1
    (elastic) lib/elastic/response_handler.ex:4: Elastic.ResponseHandler.process(%HTTPotion.ErrorResponse{message: "req_timedout"})
    (elixir) lib/enum.ex:651: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir) lib/enum.ex:651: Enum.each/2
    (elixir) lib/code.ex:363: Code.require_file/2
    (mix) lib/mix/tasks/run.ex:71: Mix.Tasks.Run.run/1
    (mix) lib/mix/task.ex:296: Mix.Task.run_task/3
    (mix) lib/mix/cli.ex:58: Mix.CLI.run_task/2
    (elixir) lib/code.ex:363: Code.require_file/2

The indexing works( ie the document is indexed and can be search) but the script that does multiple indexing operations breaks after the first operation.

It seems that assigning the result to a variable
response = ProductDocument.index(product.id, data)
doesn't break the script after the first indexing operation but I'm not sure if this was the intended behaviour of the package.

What would be a better solution( if any)?

Regarding this line:

Could we handle the error case for the Document.API.search as well? If for example the index doesn't exist it creates a nasty exception.

Is there a good reason why there aren't any? If not, I might go on and add some.

Normally I would expect that:

ProductDocument.delete("/product") # where product is the @es_index

To make a
DELETE /product

And ES to delete all the data in that index and respond with

{
  "acknowledged": true
}

What actually happens is that no record gets deleted and I receive:

{:ok, 201,
 %{"_id" => "1", "_index" => "product",
   "_shards" => %{"failed" => 0, "successful" => 1, "total" => 2},
   "_type" => "product", "_version" => 1, "created" => true,
   "result" => "created"}}

I don't think this is consistent with the ES API and I'm not exactly sure why this happens.
I run
DELETE /product
in Kibana and
curl -XDELETE "http://localhost:9200/product"
with cURL and both work as expected( delete the index).

Only the delete method from the ES package doesn't.

Hey Ryan,

When making HTTP post requests using Elastic.HTTP.post/2 to Bulk or Multi Search APIs, I get: You must add a newline \n character error from Elasticsearch. Looking at the code, I see that just before making an HTTP request you encode the body using encode_body/1 function which uses Jason. Now when I run Jason.encode/2 on a binary, it does not return it as is but puts escaping characters all over the place:

iex(5)> Jason.encode("{x: 1}\n")
{:ok, "\"{x: 1}\\n\""}

As you can see, the newline character is being escaped which won't work with Elasticsearch.

I'll be more than happy to create a PR for this.

Cheers.

Hey @radar,

Working on tests and docs for #2, but wanted to discuss one other concern... Currently, with the API access and secret needed in config, they're required at compile time, which also means they're included in the compiled code. I'm wondering if this is the best way, and if this doesn't present security concerns, as well as the need to recompile the app if the only thing changing is credentials/endpoint. It also makes it a little harder to dockerize my app, as I need these values at container build time instead of runtime, and I can't reuse a staging container in production without recompiling. What do you think? Do you have same issues/dealing with them some other way?

Is elastic capable of connecting to more than a single cluster? Based on the docs it seems like you can only specify connection details for a single cluster.

The code examples at:
https://hexdocs.pm/elastic/Elastic.Document.API.html
for Answer.raw_search, Answer.get and so on need extra indentation to be displayed as code.

Hi,

I've found a problem with the Scroller implementation.

GenServer's default timeout for every call function is 5000 (5 sec), that value is used in the Scroller by default and there's currently no way of changing it. Meaning that, when you ask for the next_page and the Scroll.next() takes longer than 5 seconds the Scroller will crash with a timeout.

This can be fixed by placing the HTTP timeout env in the next_page call, which will then change the timeout value for it.

I can work on a pull request if needed.

I noticed that HTTPotion is deprecated.. The suggestion from the package author there is to use Tesla instead.

Changing this is a pretty big change, so I wanted to ask before going ahead with this, whether you have better plans in store?

I'm trying your library in my new empty Elixir 1.4 mix project referenced as {:elastic, "~> 2.3.5"} . Deps loaded and config defines base_url as http://localhost:9200 but in iex, when making a simple search, I'm getting the error as below. Any idea whats wrong here? Do I have to start some worker??

iex(1)> Elastic.HTTP.get("/netnotes/_search")
** (ArgumentError) argument error
    (stdlib) :ets.lookup(:ibrowse_lb, {'localhost', 9200})
             c:/eprojects/elixir/u2/deps/ibrowse/src/ibrowse.erl:328: :ibrowse.send_req/6
             lib/httpotion.ex:355: HTTPotion.request/3
             lib/elastic/http.ex:100: Elastic.HTTP.request/3

If I have the following:

defmodule Es.Account.User do
@es_type "user"
@es_index "user"

use Elastic.Document.API

defstruct id: nil,
email: :string,
first_name: :string,
last_name: :string,
mobile: :string,
customer_number: :string,
profiles: [
]
end

And a profile is a list with maps how would I configure it so elastic search knows about them? I.e. https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html

When HTTPotion is unable to resolve base_url address an exception is being thrown:
(FunctionClauseError) no function clause matching in Elastic.ResponseHandler.process/1

To replicate it use https://something.com:9200 as base_url.

Make sure these boxes are checked before submitting the issue:

• I have provided steps to reproduce the issue.
  I want to use elastic in my release.
• I have explained clearly what is happening, and what I expect to happen.
  I expect that release will work, and it doesn't

Elixir app release is compiled w/o Mix while it's not clear during compiling process it will lead to a app crash right after Mix.* will be evaluated.

Culprit: https://github.com/radar/elastic/blob/master/lib/elastic/index.ex#L81

Steps to reproduce: use exrm/distillery to build release, start. Execute code related to elastic lookup, get erl_crash.dump

Hi Ryan

I'm using AWS elasticsearch version 6.4. When I try to create an index, I get the following error:

Elastic.Index.create("post")
{:error, 400,
 %{
   "error" => %{
     "reason" => "request [/elastic_dev_post] contains unrecognized parameters: [X-Amz-Algorithm], [X-Amz-Credential], [X-Amz-Date], [X-Amz-Expires], [X-Amz-Signature], [X-Amz-SignedHeaders]",
     "root_cause" => [
       %{
         "reason" => "request [/elastic_dev_post] contains unrecognized parameters: [X-Amz-Algorithm], [X-Amz-Credential], [X-Amz-Date], [X-Amz-Expires], [X-Amz-Signature], [X-Amz-SignedHeaders]",
         "type" => "illegal_argument_exception"
       }
     ],
     "type" => "illegal_argument_exception"
   },
   "status" => 400
 }}

Here are the parameters being sent to AWSAuth:
AWSAuth.sign_url("xxxx", "xxxxx", "put", %URI{authority: "search-blog-dev-sdgsergdsgfdg23423.us-east-1.es.amazonaws.com", fragment: nil, host: "search-blog-dev-sdgsergdsgfdg23423.us-east-1.es.amazonaws.com", path: "/elastic_dev_post", port: 443, query: nil, scheme: "https", userinfo: nil}, "us-east-1", "es", %{"Content-Type" => "application/json"}, ~N[2019-03-01 04:36:31.902975], [])

Which creates a URL like this:
https://search-blog-dev-sdgsergdsgfdg23423.us-east-1.es.amazonaws.com/elastic_dev_post?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=XXXX%2Fus-east-1%2Fes%2Faws4_request&X-Amz-Date=20190301T120111Z&X-Amz-Expires=86400&X-Amz-Signature=xxxxxXXXXXXXxxxxxx&X-Amz-SignedHeaders=Content-Type%3Bhost

Only thing I could find on google was this post of someone getting the same error message but it got no replies. I feel like if I'm the only person getting this error when using the library with AWS I must have something configured wrong?

Here's my config:

config :elastic,
  base_url: "https://search-blog-dev-sdgsergdsgfdg23423.us-east-1.es.amazonaws.com",
  index_prefix: "elastic",
  use_mix_env: true,
  aws: %{
    enabled: true,
    access_key_id: System.get_env("AWS_ACCESS_KEY_ID"),
    secret_access_key: System.get_env("AWS_SECRET_ACCESS_KEY"),
    region: "us-east-1"
  }

Any help is appreciated, thanks

I have no expertise here, but was curious about choices of HTTPoison vs HTTPotion and came across this article: http://thiagoborg.es/blog/2016/03/20/httpotion-or-httpoison/

Curious about choice of HTTPoison vs Potion (if you had any special reason for choosing one over the other) and whether we should consider passing ibrowse options into Potion requests, and/or allowing config or per request to override these, assuming we stick with Potion? Could also switch to Poison, but I have no particular opinion there, just realized most of the other elastic and HTTP related libs I've been looking at have been using Poison, and wonder if that's because it doesn't require this kind of config for more optimal results.