The integration test failed at distribution level for component custom-codecs
Version: 2.10.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/5857/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.10.0/8459/linux/arm64/tar/test-results/5857/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

The integration test failed at distribution level for component custom-codecs
Version: 2.10.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/5829/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.10.0/8459/linux/arm64/tar/test-results/5829/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

Coming from opensearch-project/opensearch-build#4161

What is happening?

We are planning 2.11.1 release in upcoming few weeks. Following the semver guildelines and branching strategy, this release will be based on 2.11 as a reference branch

Why?

1. We need this release because of a critical breaking bug in the security plug-in. This bug impacts everyone using logstash and the only work around is to turn off compression (which is expensive)

2. As a refresher, our normal process is to commit changes to main, then backport to 2.x. Since the 1.0 release, we only backport bug fixes and security changes to the current 2.x branch (in this case 2.11) so that if we need to we can quickly do a patch release.

3. There are two reason we don’t back port features to the current release branch (right now 2.11):

   1. Because we want patch release to be as thin as possible, and the more changes you make, the more verification we need to do
   2. Adding features in a patch breaks semver.

However, looking at the list of changes for 2.11, we currently have a mix of features that have been merged in. After providing several options and discussing with all the component teams as well as community we came to the conclusion of reverting all feature related changes from 2.11 branch.

What do I need to do?

Please help revert all the commits that are related to new features or enhancements from 2.11 branch that were added after 2.11.0 release.
Here is a rough list of commits that went in after 2.11.0 release for each repo.

Status

Please check-mark one of the below and close this issue once the action item is completed.

• We confirm that no features or enhancements were added to 2.11 branch. We are good to go for 2.11.1 consisting only of bug fixes and security fixes.

OR

• We have reverted all the feature or ehancement related commits from 2.11 branch. We are good to go for 2.11.1 consisting only of bug fixes and security fixes.

Thank you!
cc: @bbarani @CEHENKLE @Divyaasm

What is the bug?

The release notes for 2.13.0 (https://github.com/opensearch-project/custom-codecs/blob/main/release-notes/opensearch-custom-codecs.release-notes-2.13.0.0.md) do not follow the standard guidelines mentioned here: https://github.com/opensearch-project/opensearch-plugins/blob/main/RELEASE_NOTES.md

Due to this the automation in https://github.com/opensearch-project/opensearch-build/tree/main/src/release_notes_workflow breaks and the release manager has to manually sort the PRs.

How can one reproduce the bug?

Clone build repo and run ./release_notes.sh compile manifests/2.13.0/opensearch-2.13.0.yml --output opensearch.md
See custom-codecs under NON-COMPLAINT banner:

<h2>NON-COMPLIANT</h2>
<h2>CHANGED</h2>
<h3>Opensearch Custom Codecs</h3>
<ul>
<li>Remove unneccessary ArrayUtil.grow in Zstd compression modes. (<a href="https://github.com/opensearch-project/custom-codecs/pull/121">#121</a>)</li>
</ul>

What is the expected behavior?

Should not be under non-complaint banner but appropriate headings

What is your host/environment?

Operating system, version.

Do you have any screenshots?

If applicable, add screenshots to help explain your problem.

Do you have any additional context?

Add any other context about the problem.

What is the bug?

This project is missing the generic gradle task updateVersion that is responsible to handle the version increment for the repo.

Plugin team can you please add updateVersion to this repo so that it is consistent with the process of version increment.
(More examples: https://github.com/opensearch-project/opensearch-plugin-template-java/pull/32/files#diff-3ab46ee209a127470fce3c2cf[…]badbb929a4b5b13656a4bc4ce19c0b8)

Having said this the generated RC's for 2.11.1 release has to be re-generated, as the custom-codecs still point to 2.11.0

This wont impact the RC as the opensearch.version was set to 2.11.1 during build stage, but its just that the the build.gradle will have 2.11.0 for 2.11.1 release.

Received Error: Error building custom-codecs, retry with: ./build.sh manifests/2.12.0/opensearch-2.12.0.yml --component custom-codecs.
The distribution build for custom-codecs has failed for version: 2.12.0.
Please see build log at https://build.ci.opensearch.org/job/distribution-build-opensearch/9172/display/redirect.
The failed build stage will be marked as unstable(!). Please see ./build.sh step for more details

This is a component issue for 2.12.0.
Coming from opensearch-project/opensearch-build#4115. Please follow the following checklist.
Please refer to the DATES in that post.

Preparation

• Assign this issue to a release owner.
• Finalize scope and feature set and update the Public Roadmap.
• All the tasks in this issue have been reviewed by the release owner.
• Create, update, triage and label all features and issues targeted for this release with v2.12.0.
• Finalize the code and create the the release branch 2.12 from the 2.x branch.

CI/CD

• All code changes for 2.12.0 are complete.
• Ensure working and passing CI.
• Check that this repo is included in the distribution manifest.

Pre-Release

• Increment the version on the parent branch to the next development iteration.
• Gather, review and publish release notes following the rules and back port it to the release branch.git-release-notes may be used to generate release notes from your commit history.
• Confirm that all changes for 2.12.0 have been merged.
• Add this repo to the manifest for the next developer iteration.

Release Testing

• Find/fix bugs using latest tarball and docker image provided in parent release issue and update the release notes if necessary.
• Code Complete: Test within the distribution, ensuring integration, backwards compatibility, and performance tests pass.
• Sanity Testing: Sanity testing and fixing of critical issues found.
• File issues for all intermittent test failures.

Release

• Complete documentation.
• Verify all issued labeled for this release are closed or labelled for the next release.
• Verify the release date mentioned in release notes is correct and matches actual release date.

Post Release

• Prepare for an eventual security fix development iteration by incrementing the version on the release branch to the next eventual patch version.
• Add this repo to the manifest of the next eventual security patch version.
• Suggest improvements to this template.
• Conduct a retrospective, and publish its results.

The integration test failed at distribution level for component custom-codecs
Version: 2.12.0
Distribution: zip
Architecture: x64
Platform: windows

Please check the logs: https://build.ci.opensearch.org/job/integ-test/7291/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.12.0/9272/windows/x64/zip/test-results/7291/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

OS 2.x branch recently upgraded to use Lucene 9.9.1

We need to evaluate if we require changes in custom-codecs as well to rely on the new codecs introduced.

e.g.
https://github.com/opensearch-project/custom-codecs/blob/main/src/main/java/org/opensearch/index/codec/customcodecs/Lucene95CustomCodec.java#L93C36-L93C49

The integration test failed at distribution level for component custom-codecs
Version: 2.12.0
Distribution: rpm
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/7252/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.12.0/9266/linux/x64/rpm/test-results/7252/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

PR #122 introduced hardware-accelerated compression, leveraging Intel (R) QAT technology, for DEFLATE and LZ4 compression algorithms. The implementation uses the Qat-Java library to interact with the QAT hardware.

The PR also introduced two additional values for index-codec: qat_deflate and qat_lz4. Both codecs are compatible with their corresponding software counterparts, best_compression and default respectively, but do not override them (at least for the time being).

The new setting index.codec.qatmode defines two modes of execution. A hardware mode exclusively uses QAT while an auto mode may fallback and use a software implementation in cases where hardware resources are not available.

Another approach that could be taken is to override best_compression and default such that, in systems where the hardware is available, hardware acceleration is used.

The purpose of this RFC is to initiate a discussion on the pros and cons this last approach.

@reta @sarthakaggarwal97 @wbeckler @andrross

The integration test failed at distribution level for component custom-codecs
Version: 2.12.0
Distribution: deb
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/7251/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.12.0/9266/linux/x64/deb/test-results/7251/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

The integration test failed at distribution level for component custom-codecs
Version: 2.10.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/5839/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.10.0/8459/linux/arm64/tar/test-results/5839/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

Is your feature request related to a problem?

Coming from #122 (comment)

Given custom-codecs plugin is out of sandbox, any new codecs introduced should get some bake time to be tried by users before it can be marked ready for production usage. We should add some setting to allow-list a codec to be generally available. Codecs like zstd have been existing for quite a few versions, and already have seen backward compatibility handled and working. As new codecs are added (e.g. qat based), it would be helpful to enable them for production usage only after they have been vetted through some community feedback as well.

What solution would you like?

Introduce a setting which allows a codec to be marked experimental, and enabled on demand.

What alternatives have you considered?

Introduce codecs in new sandboxed plugins

The integration test failed at distribution level for component custom-codecs
Version: 2.10.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/5835/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.10.0/8459/linux/arm64/tar/test-results/5835/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

Is your feature request related to a problem?

Currently, there is not a straightforward way to leverage the implementation of ZstdCompressionMode and ZstdNoDictCompressionMode out of the box as they are protected access modifiers. By introducing simple getters in the Lucene95CustomStoredFieldsFormat, we can provide a way for the developers to use these ztsandard compression modes out of the box and alongside the flexibility to use their own custom StoredFieldsFormat with Zstandard Compression Algorithm.

What solution would you like?

Implementing simple getters for the custom compression modes

This is a component issue for 2.14.0.
Coming from opensearch-project/opensearch-build#4562. Please follow the following checklist.
Please refer to the DATES in that post.

Preparation

• Assign this issue to a release owner.
• Finalize scope and feature set and update the Public Roadmap.
• All the tasks in this issue have been reviewed by the release owner.
• Create, update, triage and label all features and issues targeted for this release with v2.14.0.
• Finalize the code and create the the release branch 2.14 from the 2.x branch.

CI/CD

• All code changes for 2.14.0 are complete.
• Ensure working and passing CI.
• Check that this repo is included in the distribution manifest.

Pre-Release

• Increment the version on the parent branch to the next development iteration.
• Gather, review and publish release notes following the rules and back port it to the release branch.git-release-notes may be used to generate release notes from your commit history.
• Confirm that all changes for 2.14.0 have been merged.
• Add this repo to the manifest for the next developer iteration.

Release Testing

• Find/fix bugs using latest tarball and docker image provided in parent release issue and update the release notes if necessary.
• Code Complete: Test within the distribution, ensuring integration, backwards compatibility, and performance tests pass.
• Sanity Testing: Sanity testing and fixing of critical issues found.
• File issues for all intermittent test failures.

Release

• Complete documentation.
• Verify all issued labeled for this release are closed or labelled for the next release.
• Verify the release date mentioned in release notes is correct and matches actual release date.

Post Release

• Prepare for an eventual security fix development iteration by incrementing the version on the release branch to the next eventual patch version.
• Add this repo to the manifest of the next eventual security patch version.
• Suggest improvements to this template.
• Conduct a retrospective, and publish its results.

The integration test failed at distribution level for component custom-codecs
Version: 2.10.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/5899/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.10.0/8472/linux/arm64/tar/test-results/5899/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

This is a component issue for 3.0.0.
Coming from opensearch-project/opensearch-build#3747. Please follow the following checklist.
Please refer to the DATES in that post.

Preparation

• Assign this issue to a release owner.
• Finalize scope and feature set and update the Public Roadmap.
• All the tasks in this issue have been reviewed by the release owner.
• Create, update, triage and label all features and issues targeted for this release with v3.0.0.
• Finalize the code and create the the release branch 3.0 from the 3.x branch.

CI/CD

• All code changes for 3.0.0 are complete.
• Ensure working and passing CI.
• Check that this repo is included in the distribution manifest.

Pre-Release

• Increment the version on the parent branch to the next development iteration.
• Gather, review and publish release notes following the rules and back port it to the release branch.git-release-notes may be used to generate release notes from your commit history.
• Confirm that all changes for 3.0.0 have been merged.
• Add this repo to the manifest for the next developer iteration.

Release Testing

• Find/fix bugs using latest tarball and docker image provided in parent release issue and update the release notes if necessary.
• Code Complete: Test within the distribution, ensuring integration, backwards compatibility, and performance tests pass.
• Sanity Testing: Sanity testing and fixing of critical issues found.
• File issues for all intermittent test failures.

Release

• Complete documentation.
• Verify all issued labeled for this release are closed or labelled for the next release.
• Verify the release date mentioned in release notes is correct and matches actual release date.

Post Release

• Prepare for an eventual security fix development iteration by incrementing the version on the release branch to the next eventual patch version.
• Add this repo to the manifest of the next eventual security patch version.
• Suggest improvements to this template.
• Conduct a retrospective, and publish its results.

This issue is created out of opensearch-project/OpenSearch#9422. Technical discussions around the merits and tradeoffs of users choosing ZStd codec have been discussed. We need to make sure this doesn't get lost in the noise of how to react to the potential demotion of the ZStd module to a plugin. I'm opening this issue here to capture those merits and tradeoffs until we can get them into proper documentation.

ZStd Pros:

1. better / faster compression
2. cost savings

Cons:

1. force killed nodes
2. wildcard loadLibrary security hole
3. no rolling upgrade
4. libc portability issues

This is a "living" issue and the list will likely be updated as more information is learned and new protection mechanisms are injected.

Add the backport workflow and labels to automate backporting changes across branches.

The integration test failed at distribution level for component custom-codecs
Version: 2.13.0
Distribution: deb
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/7981/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9616/linux/arm64/deb/test-results/7981/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

The integration test failed at distribution level for component custom-codecs
Version: 2.10.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/5825/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.10.0/8459/linux/arm64/tar/test-results/5825/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

The integration test failed at distribution level for component custom-codecs
Version: 2.10.0
Distribution: tar
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/5846/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.10.0/8459/linux/x64/tar/test-results/5846/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

This is a component issue for 2.15.0.
Coming from opensearch-project/opensearch-build#4681. Please follow the following checklist.
Please refer to the DATES in that post.

Preparation

• Assign this issue to a release owner.
• Finalize scope and feature set and update the Public Roadmap.
• All the tasks in this issue have been reviewed by the release owner.
• Create, update, triage and label all features and issues targeted for this release with v2.15.0.
• Finalize the code and create the the release branch 2.15 from the 2.x branch.

CI/CD

• All code changes for 2.15.0 are complete.
• Ensure working and passing CI.
• Check that this repo is included in the distribution manifest.

Pre-Release

• Increment the version on the parent branch to the next development iteration.
• Gather, review and publish release notes following the rules and back port it to the release branch.git-release-notes may be used to generate release notes from your commit history.
• Confirm that all changes for 2.15.0 have been merged.
• Add this repo to the manifest for the next developer iteration.

Release Testing

• Find/fix bugs using latest tarball and docker image provided in parent release issue and update the release notes if necessary.
• Code Complete: Test within the distribution, ensuring integration, backwards compatibility, and performance tests pass.
• Sanity Testing: Sanity testing and fixing of critical issues found.
• File issues for all intermittent test failures.

Release

• Complete documentation.
• Verify all issued labeled for this release are closed or labelled for the next release.
• Verify the release date mentioned in release notes is correct and matches actual release date.

Post Release

• Prepare for an eventual security fix development iteration by incrementing the version on the release branch to the next eventual patch version.
• Add this repo to the manifest of the next eventual security patch version.
• Suggest improvements to this template.
• Conduct a retrospective, and publish its results.

What is the bug?

The pom file under /org/opensearch/opensearch-custom-codecs is missing project description. This lead to maven publication to fail.

Invalid POM: /org/opensearch/opensearch-custom-codecs/2.10.0.0/opensearch-custom-codecs-2.10.0.0.pom: Project description missing

How can one reproduce the bug?

Steps to reproduce the behavior.

What is the expected behavior?

Maven POM validations should go through

What is your host/environment?

Linux x64

Do you have any screenshots?

If applicable, add screenshots to help explain your problem.

Do you have any additional context?

We fixed the problem by manually adding the description for 2.10.0. But need this to be fixed before next release.

The integration test failed at distribution level for component custom-codecs
Version: 2.14.0
Distribution: zip
Architecture: x64
Platform: windows

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8159/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.14.0/9733/windows/x64/zip/test-results/8159/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

Is your feature request related to a problem?

We have a meta to operate on collections of plugins in https://github.com/opensearch-project/opensearch-plugins/blob/main/plugins/.meta, which is convenient for various maintenance tasks.

What solution would you like?

Please add a line to that.

What is the bug?

This is more of a question. I can see in the maven repository that we have an artifact for custom-codecs, but the last version is 2.10.0. Is there is a reason why the 2.11.0 version was not added? Is there any process that we follow currently so that artifacts are published alongside OpenSearch release.

What is the expected behavior?

A version of custom-codecs being released alongside other OpenSearch version releases.

@reta @nknize @andrross

Received Error: Error building custom-codecs, retry with: ./build.sh manifests/2.10.0/opensearch-2.10.0.yml --component custom-codecs.
The distribution build for custom-codecs has failed for version: 2.10.0.
Please see build log at https://build.ci.opensearch.org/job/distribution-build-opensearch/8427/consoleFull

The integration test failed at distribution level for component custom-codecs
Version: 2.10.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/5864/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.10.0/8459/linux/arm64/tar/test-results/5864/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

Background

Previously, when installing the security plugin demo configuration, the cluster was spun up with the default admin credentials, admin:admin. A change was made in main and backported to 2.x for the 2.12.0 release, which now requires an initial admin password to be passed in via the environment variable OPENSEARCH_INITIAL_ADMIN_PASSWORD. This will break some CI/testing that relies on OpenSearch to come up without setting this environment variable. This tracking issue is to ensure compliance with the new changes.

Coming from: opensearch-project/security#3624

Acceptance Criteria

• All documentation references to the old default credentials admin:admin are removed
• Ensure that CI/testing is working with main and 2.x branches

The integration test failed at distribution level for component custom-codecs
Version: 2.10.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/5849/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.10.0/8459/linux/arm64/tar/test-results/5849/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

The integration test failed at distribution level for component custom-codecs
Version: 2.10.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/5879/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.10.0/8472/linux/arm64/tar/test-results/5879/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

This is a component issue for 2.13.0.
Coming from opensearch-project/opensearch-build#4433. Please follow the following checklist.
Please refer to the DATES in that post.

Preparation

• Assign this issue to a release owner.
• Finalize scope and feature set and update the Public Roadmap.
• All the tasks in this issue have been reviewed by the release owner.
• Create, update, triage and label all features and issues targeted for this release with v2.13.0.
• Finalize the code and create the the release branch 2.13 from the 2.x branch.

CI/CD

• All code changes for 2.13.0 are complete.
• Ensure working and passing CI.
• Check that this repo is included in the distribution manifest.

Pre-Release

• Increment the version on the parent branch to the next development iteration.
• Gather, review and publish release notes following the rules and back port it to the release branch.git-release-notes may be used to generate release notes from your commit history.
• Confirm that all changes for 2.13.0 have been merged.
• Add this repo to the manifest for the next developer iteration.

Release Testing

• Find/fix bugs using latest tarball and docker image provided in parent release issue and update the release notes if necessary.
• Code Complete: Test within the distribution, ensuring integration, backwards compatibility, and performance tests pass.
• Sanity Testing: Sanity testing and fixing of critical issues found.
• File issues for all intermittent test failures.

Release

• Complete documentation.
• Verify all issued labeled for this release are closed or labelled for the next release.
• Verify the release date mentioned in release notes is correct and matches actual release date.

Post Release

• Prepare for an eventual security fix development iteration by incrementing the version on the release branch to the next eventual patch version.
• Add this repo to the manifest of the next eventual security patch version.
• Suggest improvements to this template.
• Conduct a retrospective, and publish its results.

Hi All,

This is coming from the campaign here:

• opensearch-project/opensearch-build#3966

Overview

We would like your CI check (specifically plugin build) in GitHub Repo to run on top of the Build Docker Images from production distribution pipeline.

This is to ensure every plugin repo will use the exact docker images we used in Jenkins build, to check their PRs and run tests before merging the code, so that issues can be detected earlier, and environment can be identical across teams.

Solutions

The Build Team has created a simple script to dynamically retrieve the current docker image name/tag, so everyone can easily pull the images for their CI checks.

We have a trial run of the above with k-NN team. The script retrieves the docker image dynamically, save output, and use it as the docker image to pull for the upcoming run:

• opensearch-project/k-NN#1099

Note that GitHub Actions only support LINUX docker container at the time of this writing, so we will add Windows containers later on as well as macOS.

Implementation Notes

We would like you to review above PR and implement similar changes. Note on line 33 of the above k-NN PR, -u and -p parameters needs to assign values accordingly.

• OpenSearch Plugin:

          CI_IMAGE_VERSION=`opensearch-build/docker/ci/get-ci-images.sh -p centos7 -u opensearch -t build | head -1`

• OpenSearch-Dashboards Plugin:

          CI_IMAGE_VERSION=`opensearch-build/docker/ci/get-ci-images.sh -p rockylinux8 -u opensearch-dashboards -t build | head -1`

Note that in the above k-NN PR, despite it being OpenSearch plugin, it still uses rockylinux8, as we initially plan to upgrade to rockylinux. We have since revert back to centos7 to support older versions of systems running k-NN lib. As a result, all OpenSearch plugins still uses centos7 for the time being, and all OpenSearch-Dashboards plugins can go to rockylinux8.

Completion Date

The above should be implemented by Nov. 1, 2023 (2023-11-01) by Plugin Owners to their repository.
And backport the changes to 2.x branch after merging in main branch.

Contacts

Please contact @peterzhuamazon for any questions on this campaign.

cc: @bbarani

Thanks.

The integration test failed at distribution level for component custom-codecs
Version: 2.10.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/5885/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.10.0/8472/linux/arm64/tar/test-results/5885/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

Is your feature request related to a problem?

Coming here from opensearch-project/OpenSearch#11605 (comment)

We are observing good benefits from enabling Hybrid Compression during merges with default codec. That being said, I believe we would see more benefits with Zstandard since it has much better compression ratio that default codec, so the trade offs on disk throughput / iops should be reduced.

Since we would not need to introduce a new codec to implement hybrid compression with Zstandard, I propose to have an opt in settings that allows users to manage the thresholds for hybrid compression.

What solution would you like?

Hybrid Compression for Zstandard where the recently added stored fields are not compressed, which enables faster indexing, look ups for recent documents, but the stored fields are compressed upon merges.

Do you have any additional context?

Detailed Description: opensearch-project/OpenSearch#13110

The integration test failed at distribution level for component custom-codecs
Version: 2.10.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/5909/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.10.0/8472/linux/arm64/tar/test-results/5909/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

The integration test failed at distribution level for component custom-codecs
Version: 2.10.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/5889/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.10.0/8472/linux/arm64/tar/test-results/5889/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

The integration test failed at distribution level for component custom-codecs
Version: 2.10.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/5895/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.10.0/8472/linux/arm64/tar/test-results/5895/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

The integration test failed at distribution level for component custom-codecs
Version: 2.12.0
Distribution: tar
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/7734/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.12.0/9367/linux/x64/tar/test-results/7734/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

The integration test failed at distribution level for component custom-codecs
Version: 2.10.0
Distribution: zip
Architecture: x64
Platform: windows

Please check the logs: https://build.ci.opensearch.org/job/integ-test/5966/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.10.0/8483/windows/x64/zip/test-results/5966/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

The integration test failed at distribution level for component custom-codecs
Version: 2.10.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/5873/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.10.0/8459/linux/arm64/tar/test-results/5873/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

Is your feature request related to a problem?

Currently, custom-codecs supports 6 compression levels with Zstandard compression codecs. ZSTD library supports compression levels from 1 to 22.

What solution would you like?

We should look into increasing the spectrum of compression levels we support currently.

The integration test failed at distribution level for component custom-codecs
Version: 2.12.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/6700/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.12.0/9061/linux/arm64/tar/test-results/6700/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

The integration test failed at distribution level for component custom-codecs
Version: 2.10.0
Distribution: tar
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/5876/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.10.0/8472/linux/x64/tar/test-results/5876/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.