Where can I find a Go client for ES?

I was trying to do a scroll query and hit an issue caused by the formatting of time durations that are passed to Elasticsearch. You can see and example in:

https://play.golang.org/p/fKd7ErkUTJB

I tracked down the line in the generator but I wasn't exactly sure how to regenerate the source. (I think this would be a good topic for the readme or maybe a contributing doc.)

The scroll time was being formatted based on the Go time.Duration format. So for one minute it would use 16666h40m0s. Elasticsearch supports nanoseconds as a time unit so it can pass the Go time.Duration value through without any conversion just by appending "nanos".

diff --git a/internal/cmd/generate/commands/gensource/generator.go b/internal/cmd/generate/commands/gensource/generator.go
index cf05218..50b7936 100755
--- a/internal/cmd/generate/commands/gensource/generator.go
+++ b/internal/cmd/generate/commands/gensource/generator.go
@@ -625,7 +625,7 @@ func (r ` + g.Endpoint.MethodWithNamespace() + `Request) Do(ctx context.Context,
                                fieldValue = `strings.Join(r.` + fieldName + `, ",")`
                        case "time.Duration":
                                fieldCondition = `r.` + fieldName + ` != 0`
-                               fieldValue = `time.Duration(r.` + fieldName + ` * time.Millisecond).String()`
+                               fieldValue = `strconv.FormatInt(int64(r.` + fieldName +`), 10) + "nanos"`
                        default: // interface{}
                                fieldCondition = `r.` + fieldName + ` != nil`
                                // TODO: Use type switching instead?

like this in java

IndexRequest indexRequest = new IndexRequest("index", "type", "1")
.source(jsonBuilder()
.startObject()
.field("name", "Joe Smith")
.field("gender", "male")
.endObject());
UpdateRequest updateRequest = new UpdateRequest("index", "type", "1")
.doc(jsonBuilder()
.startObject()
.field("gender", "male")
.endObject())
.upsert(indexRequest);
client.update(updateRequest).get();

func elast() error { api := snapshot.New(transport.New()) snap, err := api.GetRepository() if err != nil { return err } defer snap.Response.Body.Close() body, err := ioutil.ReadAll(snap.Response.Body) if err != nil { return err } fmt.Println(string(body)) return nil }
always returns
{ "name" : "nV2Q0j1", "cluster_name" : "elasticsearch", "cluster_uuid" : "ng6xB6B5T_e8oZpxhcw6mQ", "version" : { "number" : "5.6.11", "build_hash" : "bc3eef4", "build_date" : "2018-08-16T15:25:17.293Z", "build_snapshot" : false, "lucene_version" : "6.6.1" }, "tagline" : "You Know, for Search" }
Expected result
{ "elastic": { "type": "fs", "settings": { "compress": "true", "location": "/mnt/elastic" } } }

Almost all files seem to be executable, seems like a mistake

I see that there is a method to get Response as string. However there is not to get request as string. Sometines it's useful for debugging.
The only way I can do it is creating a custom transport, like this:

package main

import (
	"context"
	"log"
	"net/http"
	"net/http/httputil"
	"strings"

	"github.com/elastic/go-elasticsearch"
	"github.com/elastic/go-elasticsearch/esapi"
)

type LogRequest string

type LoggerTransport struct {
	transport http.RoundTripper
}

func (t *LoggerTransport) RoundTrip(req *http.Request) (*http.Response, error) {

	lr := req.Context().Value(LogRequest("log"))
	if lr != nil && lr.(bool) == true {
		requestDump, err := httputil.DumpRequest(req, true)
		if err == nil {
			log.Printf("Request: %s", string(requestDump))
		}
	}

	return t.transport.RoundTrip(req)
}

func main() {

	log.SetFlags(0)

	es, err := elasticsearch.NewClient(elasticsearch.Config{Transport: &LoggerTransport{transport: http.DefaultTransport}})
	if err != nil {
		log.Fatalf("Error creating the client: %s", err)
	}
	
	req := esapi.IndexRequest{
		Index:      "test",
		DocumentID: "1",
		Body:       strings.NewReader(`{"title" : "` + "Test title" + `"}`),
		Refresh:    "true",
	}
	
	res, err := req.Do(context.WithValue(context.Background(), LogRequest("log"), true), es)
	if err != nil {
		log.Fatalf("Error indexing document: %s", err)
	}

	log.Printf("Response: %s", res.String())
}

Through a WithParam context selectively I log or not.
It would be possible to have another way to do this?

Thanks.

The repository contains the only tags: v.0.0.0 and v.7.0.0-rc1.

Could you please add tags for v.6.0.0 and v.5.0.0? It'll make possible to use gopkg.in

Hi! Where's the documentation (or it's the code itself)? Thank you!

only v0.0.0 support dep, v0.0.0 is not the lastest.

When I package dependencies with govendor, an error occured: "use of internal package github.com/elastic/go-elasticsearch/internal/version not allowed".How can I fix it?
Package version:elasticsearch/v6
Operating system: Mac os 10.14.1

If I have a elasticsearch cluster but have no Authentication，I can not use it

olivere/elastic.bulk.api better than this.bulk.api

when the number of request growing, goroutine number keeps growing.

b, _ := json.Marshal(order)

resp, err := o.es.Index(
	order.GetCertainIndex(),
	bytes.NewReader(b),
	o.es.Index.WithDocumentID(order.ID),
	o.es.Index.WithDocumentType(order.GetType()),
	o.es.Index.WithRefresh("true"))

file: estransport/estransport.go:116

	start := time.Now().UTC()
	res, err := c.transport.RoundTrip(req) // if err != nil, then res will be nil
	dur := time.Since(start)

	if c.logger != nil { // and only happen when client with logger specific
		var dupRes http.Response
		if res != nil {
			dupRes = *res
		}
		if c.logger.RequestBodyEnabled() {
			if req.Body != nil && req.Body != http.NoBody {
				req.Body = ioutil.NopCloser(dupReqBody)
			}
		}
		if c.logger.ResponseBodyEnabled() {
			if res.Body != nil && res.Body != http.NoBody { // if res == nil, will panic
				b1, b2, _ := duplicateBody(res.Body)
				dupRes.Body = b1
				res.Body = b2
			}
		}
		c.logger.LogRoundTrip(req, &dupRes, err, start, dur) // errcheck exclude
	}

Page: https://godoc.org/github.com/elastic/go-elasticsearch#Client

The link for esapi.API points to https://godoc.org/github.com/elastic/go-elasticsearch/v8/esapi#API which 404's:

There is the example about sdk's mget?

the image is error

Summary

The Scroll API takes in a scroll ID to identify subsequent pages. This scroll ID can be passed either through the URL path (POST /_search/scroll/<scroll_id>) or through the POST body ({ scroll_id: "<scroll_id>" }) per https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html.

In the case of this library, the scroll ID is passed in both the URL path and the POST body. However, this breaks the scroll API in cases where the scroll_id is too long to fit in the URL. When this happens you receive a TCP broken pipe error.

Steps to reproduce

1. Create a cluster with > 10k shards
2. Fire a search request with a small size and enable scrolling
3. Use the scroll API with this extremely long scroll_id
4. Observe the TCP broken pipe error

Remediation

Remove the scroll_id from the URL path. This is redundant and is length constrained compared to the POST body.

I believe the code is here:

Including user:pass in the URL is, as we've learned from time to time on Logstash, not a safe thing to do. The reason is that you can easily make mistakes when logging such that you intend to log the URL, but that URL includes secrets, and now you've written your secrets to the log file.

Proposal: Include a way to provide credentials without embedded them into the URL string.

111/main.go:34:3: unknown field 'Username' in struct literal of type elasticsearch.Config
111/main.go:35:3: unknown field 'Password' in struct literal of type elasticsearch.Config
111/main.go:36:3: unknown field 'CloudID' in struct literal of type elasticsearch.Config
111/main.go:37:3: unknown field 'APIKey' in struct literal of type elasticsearch.Config

func main() {
  config1 := elasticsearch.Config{
  Addresses: []string{
    "http://elasticsearch.aliyuncs.com:9200",
  },
  Username:  "elastic",
  Password:  "elastic1",
  CloudID:   "",
  APIKey:    "",
  Transport: &http.Transport{
    MaxIdleConnsPerHost:   10,
    ResponseHeaderTimeout: time.Second,
    DialContext:           (&net.Dialer{Timeout: time.Second}).DialContext,
    TLSClientConfig: &tls.Config{
      MinVersion: tls.VersionTLS11,
      // ...
    },
  },
    //Logger:    nil,
  }
  
  es, err := elasticsearch.NewClient(config1)
  fmt.Println(es,err)
}

Issue

When adding a document to Elasticsearch (v7.0.1) using go-elasticsearch on master branch, it is not possible to use the '/' as a separator within the document ID field e.g. title/test.txt.

Sample code:

func example() {
	client, _ := es.NewDefaultClient()
	docID := "title/test.txt"
	req := esapi.IndexRequest{
		Index:      "test",
		DocumentID: docID,
		Body:       strings.NewReader(`{"title" : "test.txt"}"`),
		Refresh:    "true",
	}

	res, _ := req.Do(context.Background(), client)
	defer res.Body.Close()

	if res.IsError() {
		log.Printf("[%s] Error indexing document ID=%s", res.String(), docID)
	}
}

Using curl I can insert the document by manually encoding the '/' as below:

curl -X PUT "localhost:9200/test/_doc/title%2Ftest" -H 'Content-Type: application/json' -d' 
{ 
    "title" : "test.txt" 
} 
' 

Expected

200 OK with document added to Elasticsearch.

Actual

Error produced as below:

[[400 Bad Request] {"error":"no handler found for uri [/test/_doc/title/test.txt?refresh=true] and method [PUT]"}] Error indexing document ID=title/test.txt

Root Cause?

It would seem from looking at the code that the URL is not constructed correctly. Within esapi.request.go it constructs the request as below:

func newRequest(method, path string, body io.Reader) (*http.Request, error) {
	r := http.Request{
		Method:     method,
		URL:        &url.URL{Path: path},
		Proto:      "HTTP/1.1",
		ProtoMajor: 1,
		ProtoMinor: 1,
		Header:     make(http.Header),
	}
  ...

The path, specifically the document ID part, does not get encoded and the URL structure is not initialised correctly, meaning the necessary RawPath field is not set. In order to resolve this issue, the document ID would need to be encoded prior to use and the URL should be initialised using the url.Parse(path) correctly setting Path and RawPath fields.

While pairing with @Dr-Syn on using this package, we found that basic auth seems to be not yet documented?

For basic auth:

• Scanning the code gives the feeling that including the user:pass in the Addresses list works for this purpose

For custom headers

• It's also useful sometimes to add custom headers, for example, with es-security-runas-user or for passing other credential headers (JWT tokens, Cookies, etc).

mget error 503 ,but no any words.

I test data by postman . it is ok。

Hi,
I'm new to go lang.
I was trying to use your library to index some events into elasticsearch with this code.
It did create the test index on elasticsearch cluster but did not index any events into it and returned this error: "400 Bad Request"
Please tell me what is wrong? This is my code:

package main

import (
   "context"
   "fmt"
   "github.com/elastic/go-elasticsearch"
   "github.com/elastic/go-elasticsearch/esapi"
   "io/ioutil"
   "net/http"
   "strconv"
   "strings"
   "testing"
)

var defaultResponse = http.Response{
   Status:        "200 OK",
   StatusCode:    200,
   ContentLength: 2,
   Header:        http.Header(map[string][]string{"Content-Type": {"application/json"}}),
   Body:          ioutil.NopCloser(strings.NewReader(`{}`)),
}

type FakeTransport struct {
   FakeResponse *http.Response
}

func newFakeTransport(b *testing.B) *FakeTransport {
   return &FakeTransport{FakeResponse: &defaultResponse}
}

func main() {
   ctx := context.Background()

   client, err := elasticsearch.NewClient(elasticsearch.Config{
      Addresses: []string{"http://127.0.0.1:9200"},
   })
   if err != nil {
      fmt.Println("ERROR: %s", err)
   }
   var body strings.Builder

   for i := 0; i < 100; i++ {
      docID := strconv.FormatInt(int64(i), 10)

      body.Reset()
      body.WriteString(`{"foo" : "bar `)
      body.WriteString(docID)
      body.WriteString(` " }`)

      req := esapi.IndexRequest{
         Index:      "test",
         DocumentID: docID,
         Body:       strings.NewReader(body.String()),
         Refresh:    "true",
         Pretty:     true,
         Timeout:    100,
      }
      if _, err := req.Do(ctx, client); err != nil {
         fmt.Println("Unexpected error when getting a response: %s", err)
      }
   }
}

Thanks!

It'd be nice to see this client cover the watcher API's as well.

Wiki Reference:
https://www.elastic.co/guide/en/elasticsearch/reference/7.0/watcher-api.html

generator/api

• figure out how we end up with the wrong package name in some cases (do.tmpl)
• add typed accessors and/or fields to structured responses (e.g., to api.IndexResponse). Changes should go to generator/api/templates/method.tmpl
• handle multiple HTTP methods in index
• render bulk and dict types in Param.String()

client

• we should find a trick to link to api from client in godoc, so people can see the API methods embedded in client.Client more clearly
• support additional client options (client/option.go)

generator/test

• resolve variables set by set before using them in other actions (right now they stay as $<var>)
• implement catch et al
• implement regex matching in generator/test/action/match.go
• implement generator/test/action/skip.go to skip tests
• keep raw name for test as comment to test (testcase.go/test.tmpl)

Cleanup:

• remove t.Skip() in test.tmpl
• handle variables that are declared and not used in test.tmpl (still an issue after all other TODOs?)

Nits:

• make type code in param.go its own type
• the code in generator/test/util_test.go should be shared with the generator/api package.

Hi,
I am trying to find a sample code to use scan() or scroll() api in the client, that would fetch paginated results from ES.
Can someone please point me to the reference on how to use that ?

When I use BodyString() to create mappings,I meet Error 400 (Bad Request): Failed to parse content to map [type=parse_exception].
I use elastic v6 And I use the example code in "https://github.com/olivere/elastic/wiki/CreateIndex"
How should I resolve it.

Not sure if this is intentional or not. I was surprised to find the v8 module in my application's go.mod when only using the v7 module imports in my application code.

Driver for database/sql interface using the SQL feature introduced in 6.3.

May I set id ? When I create the document ,using the sdk

Do we have the ability to search my ES indices based on date-range/DateMath ?

For example, if I have ES indices of format test-<data_format> and I want to search data across specific indices, for example now-10d to now.

Is this case feasible using this package ?

Where is your document, please? The official website example is wrong

elasticsearch.Client.Get() returns *Response, error

Most responses in Elasticsearch will have at least a top-level predictable response structure. It'd be nice, for example, if Get() returned something like a GetResponse.

The workaround is roughly the following for every kind of response:

var getResponse struct {
  Index string `json:"_index"`
  ID string `json:"_id"`
  // a dozen other lines
}

err = json.NewDecoder(response.Body).Decode(&getResponse)
...
// do something with getResponse

Thoughts? Maybe I'm missing something in the docs, but it'd be very nice not to have to define response type structs all over the place.

Is there any estimated availability date? i.e. officially released for 5.x

I am all for an officially supported Go client, but https://github.com/olivere/elastic has been the library of choice for several years. Have you considered contributing or collaborating on this library? Have you reached out to the maintainer at all? It feels wasteful to create an alternative.

I want to implement the retry of the connection to elasticsearch. But I could not find any proper documentation or sample snippet regarding the issue.
Can someone please guide how to implement retry feature.

I downloaded the code to perform tests and added the code that is in the README.

The following error appears:

undefined: client.WithHosts
cannot use body (type map[string]interface {}) as type api.SearchOption in argument to c.API.Search

I open the code in options.go in the client folder and I see that it is commented.

Can you put an example that works in the README? Or tell me where I can find how to run it.

Thank you.

Regards,
Natalia.

使用超时选项，s.client.Search(s.client.Search.WithIndex("index"), s.client.Search.WithBody(body), s.client.Search.WithTimeout(180000))
生成的请求URL为：
/index/_search?timeout=3m0s
ES 6.4 版本不能解析timeout参数，响应的错误信息为：
"reason": "failed to parse [3m0s]"
请求URL修改为 /index/_search?timeout=3m 则可以正常工作。
希望可以帮助到你们。

unknown import path "github.com/elastic/go-elasticsearch/v6": cannot find package

What do you think if I can create PR to migrate to https://golang.github.io/dep/docs/migrating.html?

When I use this vendor to connect elasticsearch 2.x, It's error, I know it's support for 7.x now, but it's no error when I doing NewClient, then I use client.Info(), it's nil, and the error is dial tcp IP:PORT: i/o timeout.

It'd be great to avoid handling huge strings.Builder for Search queries.
Is a functional client similar to the one in olivere/elastic in the roadmap and, if so, what is the timeline for this feature?

I know the client targets 7.x of elasticsearch, but there aren't even any official docker images for elasticsearch 7.x

Kind of a big miss not to support ES 6 before ES7 is even readily available.

newRequest only sets the body for specific types, as seen below. There should be a fallback for other io.Reader types.

[400 Bad Request] search_phase_execution_exception: all shards failed error when I specify 'WithSize' >=10000

This seems to be WIP for last year. Is there some deadline on when it will be ready for production use?