Add an option to enable logging (and add logging).

Once documentation is complete, update the information on the gemspec metadata.

Update 08/12/2020 - Current documentation for Enterprise Search clients is hosted here. The link will be updated once we update that page with the docs.

• Installation instructions
• Usage instructions
• License

Add license headers to every source code file.

elasticsearch-transport has a DEFAULT_PORT value of 9200. When a port is not provided, the behaviour should be:

• If you're using elasticsearch-ruby, default should be 9200
• If you're using enterprise-search-ruby, default should be 3002

Add tests to make sure this behaviour applies.

The engineering team here at Elastic is looking for developers to participate in research and feedback sessions to learn more about how you use our Ruby client and what improvements we can make to their design and your workflow. If you're interested in sharing your insights into developer experience and language client design, please fill out this short form. Depending on the number of responses we get, we may either contact you for a 1:1 conversation or a focus group with other developers who use the same client.

Thank you in advance - your feedback is crucial to improving the user experience for all Elasticsearch developers!

• Workplace Search
• Enterprise Search
• App-Search

From #11:

• Get the types for the Object typed parameters

Once the Spec is updated with the latest, fix the generation for endpoints documentation following refs in the OpenAPI Spec. Relevant to the search endpoint in Workplace now.

Add OAuth for Workplace Search

Hey! I'm trying to decode elastic URL queries from elastic's search-ui package. I've written a lot of manual code to convert a query string to an app search compatible filter/query combo. I based most of it off of the urlToState in URLManager from search-ui

I was wondering if the team would be open to including an official version of this parsing logic in this library, so that we can parse search-ui query strings as app search queries from ruby

Code is below – if you think it's a good idea, I can make a PR to incorporate this into the codebase, or if one of y'all would like to tackle this from the ground up, that would work too – thanks!

# typed: true
module ElasticService
  class Util
    extend T::Sig

    # TODO: Probably need to support nested filters?
    sig { params(query_string: String).returns(T.untyped) }
    def self.query_string_to_app_search_query(query_string)
      query_object_raw = raw_nested_query_to_elastic_query(
        Rack::Utils.parse_nested_query(query_string.delete_prefix("?"))
      )

      cleaned_filters = query_object_raw['filters'].map do |filter_entry|
        {
          "any" => filter_entry["values"].map do |sub_filter|
            cleaned_sub_filter = sub_filter
            if cleaned_sub_filter.class == Hash
              # Elastic gets angry if we add extra fields (:
              cleaned_sub_filter = cleaned_sub_filter.except("name")
            end
            {filter_entry["field"] => cleaned_sub_filter}
          end
        }
      end

      filters = {
        "all" => cleaned_filters
      }

      {
        "query" => query_object_raw['q'] || '',
        "filters" => filters,
      }
    end


    def self.parse_escaped_string(value_raw)
      value = CGI.unescape(value_raw)
      if /^n_.*_n$/.match(value)
        # Remove first and last 2 characters from string
        return value.delete_prefix("n_").delete_suffix("_n").to_f
      end

      if /^b_.*_b$/.match(value)
        # Remove first and last 2 characters from string
        return value.delete_prefix("b_").delete_suffix("_b").to_bool
      end

      # Need rfc3339 for elastic to be happy
      # but JSON stringifies to iso8601
      return Date.iso8601(value).rfc3339 rescue value
    end

    # Rack's parse_nested_query doesn't convert array-like things to arrays!
    # This handles that, so that we can manipulate the list of filters without
    # introducing "holes" into our filter array
    # Also handles elastic primitive encoding (i.e. "n_123_n") and converts date strings properly
    def self.raw_nested_query_to_elastic_query(nested_query, key_context = nil)
      if nested_query.class == Hash
        nested_query = T.let(nested_query, Hash)
        # If every key is an int, convert to array
        if nested_query.keys.all? {|k| T.let(Integer(k, exception: false), T.nilable(Integer)) && k.to_i >= 0}
          max_index = nested_query.keys.map(&:to_i).max
          backing_array = Array.new(max_index + 1)
          nested_query.each do |k, v|
            backing_array[k.to_i] = v
          end
          # Recurse now that it's been transformed to an array
          return raw_nested_query_to_elastic_query(backing_array)
        else
          return nested_query.map do |k, v| 
            [
              k,
              raw_nested_query_to_elastic_query(v, k)
            ]
          end.to_h
        end
      elsif nested_query.class == Array
        return nested_query.map {|el| raw_nested_query_to_elastic_query(el)}
      elsif nested_query.class == String
          return parse_escaped_string(nested_query)
      else
        return nested_query
      end
    end
  end
end

We should support per-request http_auth. We'll need this for OAuth.

It seems like the engine authentication endpoint, as described here, is currently missing.

I believe adding this endpoint could fix an issue concerning the Logstash App Search output plugin, which I'm currently experiencing.

Hello! I apologize if this is not the right spot for this but I didn't know where to go. My team is very interested in utilizing app search and we are a .NET shop. Are there any plans to get a .NET client for enterprise search? I would be happy to contribute a little bit of my time to one.

Found my way to this repository per the Deprecation warning in https://github.com/elastic/app-search-ruby

Issue

receiving Faraday::ConnectionFailed error whenever I try to run API calls through this Gem.

Steps I took

I tried both standalone and through Enterprise search client setup

# didn't work
client = Elastic::EnterpriseSearch::AppSearch::Client.new(host: "https://<id>.ent-search.us-east-1.aws.cloud.es.io", http_auth: "private-api-key")

# also didn't work
ent_client = Elastic::EnterpriseSearch::Client.new(host: "https://<id>.ent-search.us-east-1.aws.cloud.es.io", http_auth: "private-api-key")

# also didn't work
ent_client = Elastic::EnterpriseSearch::Client.new(host: "https://<id>.ent-search.us-east-1.aws.cloud.es.io")
app_client = ent_client.app_search(http_auth: "private-api-key")

# also didn't work
ent_client = Elastic::EnterpriseSearch::Client.new(host: "https://<id>.ent-search.us-east-1.aws.cloud.es.io")
app_client = ent_client.app_search.http_auth = "private-api-key"

# and also didn't work, when I tried to just search against my existing engine
ent_client = Elastic::EnterpriseSearch::Client.new(host: "https://<id>.ent-search.us-east-1.aws.cloud.es.io")
app_client = ent_client.app_search(http_auth: "search-api-key")

And I tried:

client.list_engines
client.index_documents
client.version

and all received a connection failed error.

I also verified that my app client http_auth returned my API key, app_client.http_auth, which it did.

For now, I'm planning on going back to the other Gem which does appear to work for me. Let me know if there is anything else you'd like me to try!

Add documentation on how to use the App Search APIs to README.

Have Enterprise Search running and the clients making successful requests on Jenkins.

https://www.elastic.co/guide/en/app-search/current/authentication.html#authentication-signed

I've done an investigation recently trying out every Enterprise Search API clients and making sure they all used persistent HTTP connections whenever possible to reduce the impact of TCP and SSL handshakes on API requests to Elastic Cloud instances. While doing so, I have noticed, that there is no way to tell the Enterprise Search client to use a specific adapter and it is not immediately obvious what adapters are supported and how they impact performance.

As a result of this ambiguity, a simple ruby script that performs a 100 requests to an Elastic Cloud cluster takes 3.5 times longer that it has to since the default adapter does not support persistent connections.

I think it would be very useful to directly call out the existence of different adapters in the underlying transport client and explain how not using a proper adapter is going to extremely detrimental to any production use of the API client.

Additionally, allowing users to pass the :adapter argument to the client would make it much easier to control what adapters are used (passing it through to the underlying transport client). Without it, users have to manually construct a transport object and pass it to the API, which is a bit awkward IMO.

We have code examples for each client in the Elasticsearch documentation. For example bulk and for cloud.

We should add client usage examples to the Enterprise Search documentation too., e.g. indexing documents.

Hi Enterprise Search Team,

According to this documentation, App Search seems to only support 4 field types of Text, Number, Date, and Geolocation.

However, I was wondering if there is a way to upload data with JSON field type

The transport layer in charge of the HTTP requests will be replaced with Elasticsearch client's transport layer elasticsearch-transport.

For AppSearch, in some endpoints parameters can be sent as part of the body or the query parameter. Initially I'm working around this, but we need to make it a seamless experience for the user. So will use only body for the request, but the method signature for the API endpoints (e.g. create_engine) have to be just one hash and the required parameters need to be validated. Needs some changes to the generator.

Relates to #26.

Assignment is here:

Set up JUnit output for tests in Jenkins. Depends on #5

Add documentation on how to use the Workplace Search APIs to README.

DateTime values need timezone info on the server ,so when a date time is provided without timezone info, use local timezone.

The client needs some testing for the proxy parameter.

Add support for custom headers on each individual request.

On release, the generated API documentation should point to the specific version's docs, instead of current.

Similar to Elasticsearch's REST YAML tests, we need to build a common integration testing framework.

This will help make sure clients are testing what needs to be tested and would make it easier for new clients to be developed.

Add elasticsearch-transport version to the request user agent.

For the latest v7.x versions, (v7.13~7.16) the Gemspec file is specifying elasticsearch-transport version as '~> 7.13.0'
https://github.com/elastic/enterprise-search-ruby/blob/7.16/elastic-enterprise-search.gemspec#L50

I was wondering if the minor version of the elasticsearch-transport should have been going up alongside the increase in the minor version of enterprise-search-ruby.

Otherwise, this creates an issue where attempt to install both elasticsearch-ruby and enterprise-search-ruby runs into the following dependency conflict for elasticsearch-transport

  In Gemfile:
    elastic-enterprise-search (= 7.14.0) was resolved to 7.14.0, which depends on
      elasticsearch-transport (~> 7.13.0)

    elasticsearch (= 7.14.1) was resolved to 7.14.1, which depends on
      elasticsearch-transport (= 7.14.1)

In Logstash's Enterprise Search plugin, for the Workplace output we use a method to check the connections to the remote:
https://github.com/logstash-plugins/logstash-integration-elastic_enterprise_search/blob/8ff2f8fc982f8036772430c670d1717895333a97/lib/logstash/outputs/elastic_workplace_search.rb#L103-L105

With version 8.0 of Workplace search the underlying http endpoint has been removed, made some test to fail as described in logstash-plugins/logstash-integration-elastic_enterprise_search#14.

The solution could be to use other methods, that implicitly prove the connectivity.

The Workplace client should provide an explicit API to check the connectivity, so that's future proof.
Elasticserach Ruby client provide a similar feature, as reference: https://www.rubydoc.info/gems/elasticsearch-api/Elasticsearch/API/Actions#ping-instance_method

• Run tests in CI.
• Set up Rubocop.