varun-lodaya / opensearch-migrations Goto Github PK

OpenSearch upgrade, migration, and comparison tooling facilitates OpenSearch migrations and upgrades. With these tools, you can set up a proof-of-concept environment locally using Docker containers or deploy to AWS using a one-click deployment script. Once set up and deployed, users can redirect their production traffic from a source cluster to a provisioned target cluster, enabling a comparison of results between the two clusters. All traffic directed to the source cluster is stored for future replay. Meanwhile, traffic to the target cluster is replayed at an identical rate to ensure a direct "apple-to-apple" comparison. This toolset empowers users to fine-tune cluster configurations and manage workloads more effectively.

• Supported cluster versions and platforms
• Build and deploy
  • Local deployment
  • AWS deployment
• Developer contributions
  • Pre-commit hooks
  • Traffic Capture Proxy and Replayer
  • Running Tests
• Security
• License

There are numerous combinations of source clusters, target clusters, and platforms. While the tools provided in this repository might work with various combinations, they might not support breaking changes between different source and target versions. Below is a list of supported source and target versions and platforms.

• Elasticsearch 6.x (Coming soon...)
• Elasticsearch 7.0 - 7.17.x
• OpenSearch 1.x
• OpenSearch 2.x

• Self-managed (hosted by cloud provider)
• Self-managed (on-premises)
• Managed cloud offerings (e.g., Amazon OpenSearch, Amazon OpenSearch Serverless)

A containerized end-to-end solution can be deployed locally using the Docker Solution.

Refer to AWS Deployment to deploy this solution to AWS.

Developers must run the "install_githooks.sh" script in order to add any pre-commit hooks. Developers should run these hooks before opening a pull request to ensure checks pass and prevent potential rejection of the pull request."

The TrafficCapture directory hosts a set of projects designed to facilitate the proxying and capturing of HTTP traffic, which can then be offloaded and replayed to other HTTP(S) server(s).

More documentation on this directory including the projects within it can be found here: Traffic Capture.

The FetchMigration directory hosts tools that simplify the process of backfilling / moving data from one cluster to another.

Further documentation can be found here: Fetch Migration README.

Developers can run a test script which will verify the end-to-end Local Docker Solution.

More documentation on this test script can be found here: End-to-End Testing

See CONTRIBUTING for more information.

This project is licensed under the Apache-2.0 License.

The release process is standard across repositories in this org and is run by a release manager volunteering from amongst maintainers.

1. Create a tag, e.g. 0.1.0, and push it to this GitHub repository.
2. The release-drafter.yml will be automatically kicked off and a draft release will be created.
3. This draft release triggers the jenkins release workflow as a result of which the opensearch-migrations toolset is released and published on artifacts.opensearch.org example as https://artifacts.opensearch.org/migrations/0.1.0/opensearch-migrations-0.1.0.tar.gz.
4. Once the above release workflow is successful, the drafted release on GitHub is published automatically.