I am trying to run the sample code on Java 8 getting the unsupported class version error. Seems like the libraries are all built on Java 11 and are not able to be reference from Java 8. below is the error I am trying to run the GetVersion.java on jre1.8.0_40.

Exception in thread "main" java.lang.UnsupportedClassVersionError: org/opensearch/client/transport/OpenSearchTransport has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(Unknown Source) at java.security.SecureClassLoader.defineClass(Unknown Source) at java.net.URLClassLoader.defineClass(Unknown Source) at java.net.URLClassLoader.access$100(Unknown Source) at java.net.URLClassLoader$1.run(Unknown Source) at java.net.URLClassLoader$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Unknown Source) at java.lang.Class.privateGetMethodRecursive(Unknown Source) at java.lang.Class.getMethod0(Unknown Source) at java.lang.Class.getMethod(Unknown Source) at sun.launcher.LauncherHelper.validateMainClass(Unknown Source) at sun.launcher.LauncherHelper.checkAndLoadMain(Unknown Source)

Hi team,

Starting from 2.7, Cross cluster replication would support replication of knn indices.
This change will only be available on managed service OS version 2.7 which is not yet released.

Requesting guidance on the process to update the public facing documentation.

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/replication.html#replication-limitations

Cross-cluster replication doesn't work with data streams or the k-NN plugin.

Changed to
Cross-cluster replication doesn't work with data streams or the k-NN plugin.
Cross-cluster replication doesn't work with k-NN plugin for OpenSearch version less than 2.7.

In this file:
https://github.com/awsdocs/amazon-elasticsearch-service-developer-guide/blob/master/doc_source/sizing-domains.md

The first two sections describe how to calculate the Minimum required storage and the Shard Count based on total storage size of the cluster.

The document then proceeds to give a "general guidance" about how to choose instance types and number with no obvious direct linking to the information presented in the two previous sections, especially the information regarding shards.

The examples in the last section are vague and do not leverage the information provided in the document up to the point to make a concise explanation.

As an example of vagueness consider this Item:

If you have a 184 GiB storage requirement and the recommended minimum number of three instances, use the equation 184 / 3 = 61 GiB to find the amount of storage that each instance needs. In this example, you might select three m4.large.elasticsearch instances for your cluster, each using a 90 GiB EBS storage volume so that you have a safety net and some room for growth over time. This configuration provides 6 vCPU cores and 24 GiB of memory, so it's suited to lighter workloads.

For a more substantial example, consider a 14 TiB storage requirement and a heavy workload. In this case, you might choose to begin testing with 2 * 144 = 288 vCPU cores and 8 * 144 = 1152 GiB of memory. These numbers work out to approximately 18 i3.4xlarge.elasticsearch instances. If you don't need the fast, local storage, you could also test 18 r5.4xlarge.elasticsearch instances, each using a 1 TiB EBS storage volume.

Where does the number 144 came from?
Why is the shard information ignored in the estimation examples? Is it there but not clearly obvious?

The document leaves the reader feeling more like they're back at square 1 before they started reading.

Coming from opensearch-project/OpenSearch#1400 customers want to easily be able to find higher level clients that natively (will) support such features as request signing and work with AWS OpenSearch.

aws docs is usually where I look for information about aws services, when I saw the low level code as how we should communicate with opensearch it left me 😮

I am new to web development and used this tutorial to build my first static website with AWS. I successfully completed all modules and have a working webpage with user auth using AWS Cognito.

I added another webpage that uses AWS Elasticsearch to search IMDB movies. I put the search.js, which includes my api gateway endpoint in the js folder.

Now a user can use my s3endpointwebsite.com/js/search.js and see my api gateway endpoint and my cognito ids.

How do I prevent someone from missing my api gateway endpoint?

Coming from opensearch-project/opensearch-java#287

When I clicked on the GitHub button at the top of this page https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/release-notes.html#release-table I was expecting to find https://github.com/awsdocs/amazon-elasticsearch-service-developer-guide/blob/master/doc_source/release-notes.md up-to-date. However those two seem to be very different from each other.

• Perhaps there's a new source for those page(s)?
• Although in AWS-managed ES console the message "A new service software release R20190113 is available." appears, but https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/release-notes.html#release-table doesn't include it?

Thank you.

According to documentation section, to connect local Kibana to AWS ES host, I need to configure it to http endpoint, but it will not work that way, instead, I've come up with working local Kibana only using https endpoint and ES version 7. Version 6 does not work both ways. This section definitely needs an update, since even config example has wrong config section names.

According to the release notes, update R20190221 introduces alerting. The alerting feature even has a page in the documentation.

However, when I start a new ElasticSearch Service 6.42 instance with update R20190221 in N. Virgina and access Kibana I am unable to see a new section called Alerting in the UI. Am I missing something here?

Would it be possible to include a screen shot to show how the Alerting feature can be choosen?

In the section "Using a Proxy to Access Amazon ES from Kibana", it mentioned applying "a resource-based policy that specifies roles and IP addresses" to ES domain. However, an ES domain with VPC endpoint does not support IP-based access policy. When applying this, user will get the error message like below.

UpdateElasticsearchDomainConfig: {"message":"You can’t attach an IP-based policy to a domain that has a VPC endpoint. Instead, use a security group to control IP-based access."}

This paragraph and sample should be corrected.

At the top of this page, it says the page describes how to sign HTTP request, but it doesn't mention when you'd want to. I've worked with aws and know how to implement sigv4, I know the context. Not everyone that finds the page will know. A simple list like
Need Sigv4:

• from one VPC to a different VPC
• from an external service to a VPC protected
  Don't need Sigv4:
• call from within the same VPC

In addition to having the code on the page, it should also have a link to a working example on Github repo. You mentioned using a high level REST client here aws/aws-sdk-java#861 , the page should provide a link to any recommended third party libraries or just point to a repo if AWS formal documentation policies prevent you from making official recommendations.

When I write docs, I always try to include a "troubleshooting" section of the common errors people will see. I've done this on the Apache projects I've contributed to and other OSS projects. One thing I did to troubleshoot issues was to use my Cloud9 environment and manually send curl commands. This is especially useful in the case where the ES domain is in a VPC that is locked down and doesn't allow SSH.

On this page, it covers kibana and logstash. It would be useful to have a short explanation of where one finds it (in the console) and how the format differs from the VPC endpoint. Say a person is debugging some code, but someone else pasted in the wrong URL. Depending on the editing tool the person is using, it could be hard to read the URL. Example would be in a terminal window with vi editor. Even before my eyes started to go bad, reading long URL in a terminal window is hard on the eyes and very easy to miss. I find little tips like that help users when I wrote docs.

On the best practices page https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-bp.html it covers the management and configuration of ES domains. It lacks programming best practices. Off the top of my head:

• should you use SNS to queue up documents for indexing
• what are the common queue design patterns using AWS services? S3 vs RDS vs DynamoDB
• what are the anti-patterns for ES that developers should avoid

The tutorial page https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/search-example.html only covers python. I know python is considered sexy right now, but enterprises building apps are more likely using Java for type-safety and ease of debugging. There really should be a Java example.

I'm very new to OpenSearch in general, so forgive me if this isn't right place for this. From What is Amazon OpenSearch Service? - Amazon OpenSearch Service, the supported versions are 1.0 and 1.1. However, the latest version is 1.3.0 (or maybe 1.3.1?), and I note that 1.2.3 updated Log4j.

With OpenSearch version 2.0 and later, alerting destinations have been replaced with Notifications, a unified notifications system in OpenSearch. The alerting page and any applicable pages should be updated to reflect these changes:

• Introduction to Notifications plugin
• Existing destinations are automatically migrated to Notifications plugin as Channels
• If some destinations failed to migrate, they will continue to work
• Users will manage Channels in Notifications going forward

For reference: https://opensearch.org/docs/latest/monitoring-plugins/alerting/monitors/#questions-about-destinations

As @andywickersham pointed out here,

awsdocs/iam-user-guide#163

Creating a Service Linked Role for ES uses this API call es:CreateElasticsearchServiceRole. Should this documentation be changed to reflect that?

https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/slr-es.html

Hello,

I am trying to use the java AwsRequestSigningApacheInterceptor provided here in my current code (which was using the v1 version of the java before) and thanks to test file also provided, I realized that the current code is removing the content of the request.
After debugging, it seems that the stream has already been read when it tries to copy back the content.
I am using a Java 11 micronaut project, and the [2.17.258] version of the software.amazon.awssdk.auth package. Could it be that a different version is needed for it to work as expected?

Thanks in advance

I received an service update with the name R20190221-P1. However, there is no description for this upgrade in the release notes: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/release-notes.html#release-table

Region: us-west-2 (Oregon)

What does R20190221-P1 provide?

In Monitoring Elasticsearch logs with Amazon CloudWatch Logs, slow logs publishing is documented to be supported in all Elasticsearch versions. However, in Features by Elasticsearch version, slow logs publishing is documented to be supported only in Elasticsearch versions 5.1 and above.

Internal reference: P50120882

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/auto-tune.html document has wrong cli command for update domain config.

aws opensearch update-domain-config \ --domain-name my-domain \ --auto-tune-options DesiredState=ENABLED,UseOffPeakWindow=true,MaintenanceSchedules=null

It throws,

Parameter validation failed:
Invalid type for parameter AutoTuneOptions.MaintenanceSchedules[0], value: null, type: <class 'str'>, valid types: <class 'dict'>

It has to be updated to,
aws opensearch update-domain-config --domain-name my-domain --auto-tune-options DesiredState=ENABLED,UseOffPeakWindow=true,MaintenanceSchedules={}

The Troubleshooting guide contains inaccurate information for controlling access to Kibana. See #4

The JavaScript sample in https://github.com/awsdocs/amazon-opensearch-service-developer-guide/blob/master/doc_source/serverless-clients.md doesn't work for OpenSeearch serverless.

Code: https://github.com/dblock/opensearch-node-client-demo/tree/opensearch-serverless-docs

export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export AWS_SESSION_TOKEN=
export AWS_REGION=
export ENDPOINT=
npm install
ts-node src/index.ts 

ResponseError: AccessDenied
    at onBody (/Users/dblock/source/dblock/opensearch-node-client-demo/node_modules/@opensearch-project/opensearch/lib/Transport.js:403:23)
    at IncomingMessage.onEnd (/Users/dblock/source/dblock/opensearch-node-client-demo/node_modules/@opensearch-project/opensearch/lib/Transport.js:318:11)
    at IncomingMessage.emit (node:events:525:35)
    at IncomingMessage.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1359:12)
    at processTicksAndRejections (node:internal/process/task_queues:82:21) {
  meta: {
    body: {
      status: 403,
      'request-id': 'b6a4bd0c-1211-9a14-85fa-5776ef5efe18',
      error: [Object]
    },
    statusCode: 403,
    headers: {
      'x-request-id': 'b6a4bd0c-1211-9a14-85fa-5776ef5efe18',
      'content-type': 'application/json',
      date: 'Thu, 12 Jan 2023 13:23:15 GMT',
      'content-length': '191',
      'x-aoss-response-hint': 'X01:gw-helper-deny',
      server: 'aoss-amazon'
    },
    meta: {
      context: null,
      request: [Object],
      name: 'opensearch-js',
      connection: [Object],
      attempts: 0,
      aborted: false
    }
  }
}

The sample signs the body and forces a content-length of 0, as per the documentation. That broke at some point since preview. Removing the content body and not signing a content-length header works around it, see https://github.com/dblock/opensearch-node-client-demo/blob/opensearch-serverless-aws-sdk2/src/index.ts.

we are using java 8 to run on local

Error Showing

java: cannot access org.opensearch.client.opensearch.OpenSearchClient
bad class file: .../org/opensearch/client/opensearch-java/2.1.0/opensearch-java-2.1.0.jar!/org/opensearch/client/opensearch/OpenSearchClient.class
class file has wrong version 55.0, should be 52.0
Please remove or make sure it appears in the correct subdirectory of the classpath.

Running sample from https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-clients.html#serverless-python as is, getting an error server-side.

Creating index:
{'status': 500, 'request-id': 'd64063a6-f54e-959e-81e7-764d8ee886eb', 'error': {'reason': 'There was a server error. Please try again or contact AWS Support', 'type': 'InternalFailure'}}
Traceback (most recent call last):
  File "/Users/dblock/source/dblock/opensearch-python-client-demo/foo.py", line 50, in <module>
    client.index(index=index_name, body=document, id='1')
  File "/Users/dblock/.local/share/virtualenvs/opensearch-python-client-demo-ejYmDdj0/lib/python3.11/site-packages/opensearchpy/client/utils.py", line 178, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dblock/.local/share/virtualenvs/opensearch-python-client-demo-ejYmDdj0/lib/python3.11/site-packages/opensearchpy/client/__init__.py", line 355, in index
    return self.transport.perform_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dblock/.local/share/virtualenvs/opensearch-python-client-demo-ejYmDdj0/lib/python3.11/site-packages/opensearchpy/transport.py", line 408, in perform_request
    raise e
  File "/Users/dblock/.local/share/virtualenvs/opensearch-python-client-demo-ejYmDdj0/lib/python3.11/site-packages/opensearchpy/transport.py", line 369, in perform_request
    status, headers_response, data = connection.perform_request(
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dblock/.local/share/virtualenvs/opensearch-python-client-demo-ejYmDdj0/lib/python3.11/site-packages/opensearchpy/connection/http_requests.py", line 210, in perform_request
    self._raise_error(
  File "/Users/dblock/.local/share/virtualenvs/opensearch-python-client-demo-ejYmDdj0/lib/python3.11/site-packages/opensearchpy/connection/base.py", line 301, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
opensearchpy.exceptions.TransportError: TransportError(500, 'InternalFailure')

In this part of the docs, it states that just using the unsigned-payload attribute for the header value should allow signing to work but it does not seem to be the case. I've had to compute the payload manually here -- opensearch-project/opensearch-py#268

The x-amz-content-sha256 header is required for all AWS Signature Version 4 requests. It provides a hash of the request payload. For OpenSearch Serverless, include it with one of the following + "/" + id values when you build the canonical request for signing:

• If there's a request payload, set the value to its Secure Hash Algorithm (SHA) cryptographic hash (SHA256).

• If there's no request payload, set the value to e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, which is the hash of an empty string.

• In either of the above two cases, you can also use the literal string UNSIGNED-PAYLOAD as the value of the x-amz-content-sha256 header.

Description

The testPostSigner() method is failing reproducibly.

org.opentest4j.AssertionFailedError: 
Expected :{"test": "val"}
Actual   :

Presumably this happens because during the signing of the request the content of the request is read. This results in the position of the InputStream to be moved to the end of the byte array. Next when the request is wrapped to make it possibly to retry here:

if (request instanceof HttpEntityEnclosingRequest) {
    HttpEntityEnclosingRequest httpEntityEnclosingRequest =
            (HttpEntityEnclosingRequest) request;
    if (httpEntityEnclosingRequest.getEntity() != null) {
        BasicHttpEntity basicHttpEntity = new BasicHttpEntity();
        basicHttpEntity.setContent(signedRequest.contentStreamProvider()
                .orElseThrow(() -> new IllegalStateException("There must be content"))
                .newStream());
        // wrap into repeatable entity to support retries
        httpEntityEnclosingRequest.setEntity(new BufferedHttpEntity(basicHttpEntity));
    }
}

The basicHttpEntity content is set to the previously-read InputStream. As a result creating the BufferedHttpEntity will result in an entity containing an empty buffer, as this will again try to read the InputStream. As the position is already at the end of the byte array it will read nothing.

Adding basicHttpEntity.getContent().reset(); before creating the BufferedHttpEntity resolves the issue.

Reproduce

Run the tests and it should fail.

Provided code is working on indices with http_compress : https://github.com/awsdocs/amazon-opensearch-service-developer-guide/blob/master/doc_source/gzip.md#sample-code-python-3

The same would result in below error for cluster settings

import boto3
from requests_aws4auth import AWS4Auth
from opensearchpy import OpenSearch, RequestsHttpConnection, AWSV4SignerAuth

host = '' # cluster endpoint, for example: my-test-domain.us-east-1.es.amazonaws.com
region = '' # e.g. us-west-1

credentials = boto3.Session().get_credentials()
auth = AWSV4SignerAuth(credentials, region)

client = OpenSearch(
    hosts = [{'host': host, 'port': 443}],
    http_auth = auth,
    use_ssl = True,
    http_compress=True,
    verify_certs = True,
    connection_class = RequestsHttpConnection
)


# document = {
# "title": "Moneyball",
#  "director": "Bennett Miller",
#  "year": "2011"
# }

# Send the request.
# print(client.index(index='movies', id='1', body=document, refresh=True))

res = client.cluster.put_settings({"persistent": {"cluster.max_shards_per_node": 1500}})
print(f'Set result: {res}')

Output:

Traceback (most recent call last):
  File "/Users/xxxx/s3-to-opensearch/OSClientModified.py", line 22, in <module>
    res = client.cluster.put_settings({"persistent": {"cluster.max_shards_per_node": 1500}})
  File "/usr/local/lib/python3.9/site-packages/opensearchpy/client/utils.py", line 177, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/opensearchpy/client/cluster.py", line 229, in put_settings
    return self.transport.perform_request(
  File "/usr/local/lib/python3.9/site-packages/opensearchpy/transport.py", line 407, in perform_request
    raise e
  File "/usr/local/lib/python3.9/site-packages/opensearchpy/transport.py", line 368, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/usr/local/lib/python3.9/site-packages/opensearchpy/connection/http_requests.py", line 203, in perform_request
    self._raise_error(
  File "/usr/local/lib/python3.9/site-packages/opensearchpy/connection/base.py", line 300, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
opensearchpy.exceptions.AuthenticationException: AuthenticationException(401, '{"Message":"Your request: \'/_cluster/settings\' payload is not allowed."}')

The same request will work by remove "http_compress" from client

client = OpenSearch(
    hosts = [{'host': host, 'port': 443}],
    http_auth = auth,
    use_ssl = True,
    verify_certs = True,
    connection_class = RequestsHttpConnection
)

After checking N. Virginia and Oregon regions, it appears that t2.micro.elasticsearch is no longer support, or at least results in an InvalidTypeException error in CloudFormation when requested. Similarly, this instance type does not appear in the drop list provided by the web console.

The latest available R20200910 SSR is being offered for update for an ES instance in us-east-1, though the release notes doc does not mention this release.

Can you please update the doc, so we know what is being delivered by the update?

I see, this is pretty common issue being reported often by customers. Is there any chance to improve the release process to improve the customer experience and ensure, the release notes doc is updated on time?

There are much simpler ways to do this now with newer clients, see https://opensearch.org/blog/aws-sigv4-support-for-clients/

I tried to use FileBeat on AWS ElasticSearch Service 6.3. I hit the following error message:

ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://elasticsearch.my-host.com:80)): Connection marked as failed because the onConnect callback failed: Error loading pipeline for fileset auditd/log: This module requires the following Elasticsearch plugins: ingest-geoip. You can install them by running the following commands on all the Elasticsearch nodes:
    sudo bin/elasticsearch-plugin install ingest-geoip 

if I run GET /_nodes/ingest?pretty I get the output below, but the geoip type is missing:

{
  "_nodes": {
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "cluster_name": "019190149510:ecg-test",
  "nodes": {
    "LRWdA_QkTdqyobK9wTcagw": {
      "name": "LRWdA_Q",
      "version": "6.3.1",
      "build_flavor": "oss",
      "build_type": "zip",
      "build_hash": "eb782d0",
      "roles": [
        "master",
        "data",
        "ingest"
      ],
      "ingest": {
        "processors": [
          {
            "type": "append"
          },
          {
            "type": "attachment"
          },
          {
            "type": "convert"
          },
          {
            "type": "date"
          },
          {
            "type": "date_index_name"
          },
          {
            "type": "dot_expander"
          },
          {
            "type": "fail"
          },
          {
            "type": "foreach"
          },
          {
            "type": "grok"
          },
          {
            "type": "gsub"
          },
          {
            "type": "join"
          },
          {
            "type": "json"
          },
          {
            "type": "kv"
          },
          {
            "type": "lowercase"
          },
          {
            "type": "remove"
          },
          {
            "type": "rename"
          },
          {
            "type": "script"
          },
          {
            "type": "set"
          },
          {
            "type": "sort"
          },
          {
            "type": "split"
          },
          {
            "type": "trim"
          },
          {
            "type": "uppercase"
          },
          {
            "type": "urldecode"
          },
          {
            "type": "user_agent"
          }
        ]
      }
    },
    "XaHFnEWuQLWoAHKpd3DAlw": {
      "name": "XaHFnEW",
      "version": "6.3.1",
      "build_flavor": "oss",
      "build_type": "zip",
      "build_hash": "eb782d0",
      "roles": [
        "master",
        "data",
        "ingest"
      ],
      "ingest": {
        "processors": [
          {
            "type": "append"
          },
          {
            "type": "attachment"
          },
          {
            "type": "convert"
          },
          {
            "type": "date"
          },
          {
            "type": "date_index_name"
          },
          {
            "type": "dot_expander"
          },
          {
            "type": "fail"
          },
          {
            "type": "foreach"
          },
          {
            "type": "grok"
          },
          {
            "type": "gsub"
          },
          {
            "type": "join"
          },
          {
            "type": "json"
          },
          {
            "type": "kv"
          },
          {
            "type": "lowercase"
          },
          {
            "type": "remove"
          },
          {
            "type": "rename"
          },
          {
            "type": "script"
          },
          {
            "type": "set"
          },
          {
            "type": "sort"
          },
          {
            "type": "split"
          },
          {
            "type": "trim"
          },
          {
            "type": "uppercase"
          },
          {
            "type": "urldecode"
          },
          {
            "type": "user_agent"
          }
        ]
      }
    }
  }
}

Is ingest-geopip not supported on AWS ElasticSearch Service?