Elasticsearch supports using saml authentication, would love to be able to utilize this.

Related Docs:
https://aws.amazon.com/about-aws/whats-new/2020/10/amazon-elasticsearch-service-adds-native-saml-authentication-kibana/
https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/saml.html

Hi!
I would like to configure all the AWS ES supported log types. How can I achieve this feature using your scripts?
Can you please provide an example?
I have tried this configuration, but it takes only one configuration not all of them
log_publishing_options = { enabled = "true" log_type = "INDEX_SLOW_LOGS" } log_publishing_options= { enabled = "true" log_type = "ES_APPLICATION_LOGS" } log_publishing_options= { enabled = "true" log_type = "SEARCH_SLOW_LOGS" }

Thank you

Hi @lgallard

Thank you for this module.

I'd like to use EBS with gp3, for that the IOPS and throughput are required but not supported by the module.

See here: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/elasticsearch_domain#throughput

Thanks,
Simon

The cluster deleted and recreated after every apply

Hi,

Using the VPC example fails when trying to plan with the following error:

Error: Unsupported argument
  on .terraform/modules/elasticsearch.elasticsearch/main.tf line 38, in resource "aws_elasticsearch_domain" "es_domain":
  38:       custom_endpoint_enabled         = lookup(domain_endpoint_options.value, "custom_endpoint_enabled")
An argument named "custom_endpoint_enabled" is not expected here.
Error: Unsupported argument
  on .terraform/modules/elasticsearch.elasticsearch/main.tf line 39, in resource "aws_elasticsearch_domain" "es_domain":
  39:       custom_endpoint                 = lookup(domain_endpoint_options.value, "custom_endpoint")
An argument named "custom_endpoint" is not expected here.
Error: Unsupported argument
  on .terraform/modules/elasticsearch.elasticsearch/main.tf line 40, in resource "aws_elasticsearch_domain" "es_domain":
  40:       custom_endpoint_certificate_arn = lookup(domain_endpoint_options.value, "custom_endpoint_certificate_arn")
An argument named "custom_endpoint_certificate_arn" is not expected here.

Trying to specify version "0.8.0" in the module definition also results in an error:

Error: Unresolvable module version constraint
There is no available version of module "lgallard/elasticsearch/aws"
(../../../modules/aws/elasticsearch/main.tf:1) which matches the given version
constraint. The newest available version is 0.9.0.

Specifying 0.7.1 works and plans successfully.

Hi folks!
I'm using your module trying to create an ES with internal master user enabled, but Im receiving this error:

module.elasticsearch.aws_cloudwatch_log_resource_policy.es_aws_cloudwatch_log_resource_policy: Creating...
module.elasticsearch.aws_cloudwatch_log_group.es_cloudwatch_log_group: Creating...
module.elasticsearch.aws_cloudwatch_log_resource_policy.es_aws_cloudwatch_log_resource_policy: Creation complete after 1s [id=data-elasticsearch-policy]
module.elasticsearch.aws_cloudwatch_log_group.es_cloudwatch_log_group: Creation complete after 2s [id=data-elasticsearch-log_group]
module.elasticsearch.aws_elasticsearch_domain.es_domain: Creating...

Error: Error creating ElasticSearch domain: ValidationException: You must provide a master username and password together.

Here is a piece of my code as an example:

module "elasticsearch" {

  source  = "lgallard/elasticsearch/aws"
  version = "0.7.0"
  
  domain_name = "test"
  elasticsearch-version = "6.7"

  advanced_security_options_enabled = true
  advanced_security_options_internal_user_database_enabled = true
  advanced_security_options_master_user_username = "admin"
  advanced_security_options_master_user_password = "password"
}

When looking at the terraform plan, we can check that the user name is coming as a null (only password is being filled):

      + advanced_security_options {
          + enabled                        = true
          + internal_user_database_enabled = true

          + master_user_options {
              + master_user_password = (sensitive value)
            }
        }

If you take a look on main.tf, we can see that in line 23:

master_user_name     = lookup(lookup(advanced_security_options.value, "master_user_options"), "master_user_name", null)

its trying to find a master_user_name variable under master_user_options, which does not exists:

  master_user_options = lookup(var.advanced_security_options, "master_user_options", null) != null ? lookup(var.advanced_security_options, "master_user_options") : {
    master_user_arn      = var.advanced_security_options_internal_user_database_enabled == false ? var.advanced_security_options_master_user_arn : null
    master_user_username = var.advanced_security_options_internal_user_database_enabled == true ? var.advanced_security_options_master_user_username : null
    master_user_password = var.advanced_security_options_internal_user_database_enabled == true ? var.advanced_security_options_master_user_password : null
  }

the right name for this variable is master_user_username under master_user_options, not master_user_name.
Am I right or missing something?

PS.: This is the first fix I open for open source projects, so pardon me if I'm missing some information.

error output:

Error: Unsupported block type

  on .terraform/modules/search/main.tf line 86, in resource "aws_elasticsearch_domain" "es_domain":
  86:       dynamic "cold_storage_options" {

Blocks of type "cold_storage_options" are not expected here.

Looks like aws supports custom domain names now, would love to add that support.

AWS Post about support
https://aws.amazon.com/about-aws/whats-new/2020/11/amazon-elasticsearch-service-now-supports-defining-a-custom-name-for-your-domain-endpoint/

log_publishing_options = { enabled = "true" log_type = "INDEX_SLOW_LOGS" log_type = "ES_APPLICATION_LOGS" }

error: expected log_publishing_options.0.log_type to be one of [INDEX_SLOW_LOGS SEARCH_SLOW_LOGS ES_APPLICATION_LOGS], got INDEX_SLOW_LOGS, ES_APPLICATION_LOGS

Module fails if we create cloudwatch log group in same stack. We get the following error

│ Error: Invalid for_each argument
│
│ on .terraform/modules/aws_es/iam.tf line 8, in resource "aws_cloudwatch_log_group" "es_cloudwatch_log_group":
│ 8: for_each = local.options
│ ├────────────────
│ │ local.options will be known only after apply
│
│ The "for_each" value depends on resource attributes that cannot be
│ determined until apply, so Terraform cannot predict how many instances will
│ be created. To work around this, use the -target argument to first apply
│ only the resources that the for_each depends on.

Example

`module "aws_es" {

source = "lgallard/elasticsearch/aws"

domain_name = "elasticsearch_public"
elasticsearch_version = "7.1"

cluster_config = {
dedicated_master_enabled = true
instance_count = 3
instance_type = "r5.large.elasticsearch"
zone_awareness_enabled = true
availability_zone_count = 3
}

ebs_options = {
ebs_enabled = "true"
volume_size = "25"
}

encrypt_at_rest = {
enabled = true
kms_key_id = "arn:aws:kms:us-east-1:123456789101:key/cccc103b-4ba3-5993-6fc7-b7e538b25fd8"
}

log_publishing_options = {
index_slow_logs = {
enabled = true
cloudwatch_log_group_arn = "${aws_cloudwatch_log_group.es.arn}:"
log_publishing_options_retention = 90
}
search_slow_logs = {
enabled = true
cloudwatch_log_group_arn = "${aws_cloudwatch_log_group.es.arn}:"
}
es_application_logs = {
enabled = true
cloudwatch_log_group_arn = "${aws_cloudwatch_log_group.es.arn}:"
}
audit_logs = {
enabled = false
cloudwatch_log_group_arn = "${aws_cloudwatch_log_group.es.arn}:"
}
}

advanced_options = {
"rest.action.multi.allow_explicit_index" = true
}

domain_endpoint_options = {
enforce_https = true
custom_endpoint_enabled = true
custom_endpoint = "lgallardo.com"
custom_endpoint_certificate_arn = "arn:aws:acm:us-east-1:123456789101:certificate/abcd1234-ef11-abcd-1234-abcd1234efef"
}

node_to_node_encryption_enabled = true
snapshot_options_automated_snapshot_start_hour = 23

tags = {
Owner = "sysops"
env = "dev"
}

resource "aws_cloudwatch_log_group" "es" {
name = "test"
retention_in_days = 90
}

`

Fixed here #46

when applying I can see this from the plan

• advanced_security_options {
  + enabled = true
  + internal_user_database_enabled = true

     + master_user_options {
         + master_user_arn = "arn:aws:iam::123456:user/[email protected]"
       }
   }
  

however I get the following error

aws_elasticsearch_domain.es_domain: Creating...

Error: Error creating ElasticSearch domain: ValidationException: You must provide a master username and password when the internal user database is enabled.

Hi,

Getting below error while deploying 2 domains in the same account:

Error: error creating IAM Service Linked Role (es.amazonaws.com): InvalidInput: Service role name AWSServiceRoleForAmazonElasticsearchService has been taken in this account, please try a different suffix. │ status code: 400, request id: fb0ed8e1-352d-40d6-87bd-0d96d7ba7415 │ │ with module.aws_es.aws_iam_service_linked_role.es[0], │ on .terraform/modules/aws_es/iam.tf line 38, in resource "aws_iam_service_linked_role" "es": │ 38: resource "aws_iam_service_linked_role" "es" {

Please help.

Thanks,
RR

Would be great to see this in the module since it looks there is support now for this https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/elasticsearch_domain#auto_tune_options.

An example of the variable would be:

auto_tune_options {
    maintenance_schedule = {
      cron_expression_for_recurrence = "cron(0 2 ? * 1 *)"
      start_at                       = "2022-09-08T16:47:06Z"

      duration {
          unit  = "HOURS"
          value = 2
        }
    }
  }

Thanks!

Even the az count set to 3, it is not working, that block does not show up in the cluster_config block and ends up defaulting to 2 which is the tf default. Probably some issue in the for_each line for the dynamic zone_awareness_config block.

I'm relatively new to implementing custom terraform modules, so forgive me if I'm missing something here.

I'm attempting to use this module to create an elasticsearch cluster in a particular vpc (we have multiple within an account). When I run a plan, I see that it should be associating a vpc with the elasticsearch cluster.

When I look at the options taken by this module, however, vpc_id is not one. I looked at your main.tf, and can see that it's simply not being looked for in in the code.

# vpc_options
  dynamic "vpc_options" {
    for_each = local.vpc_options
    content {
      security_group_ids = lookup(vpc_options.value, "security_group_ids")
      subnet_ids         = lookup(vpc_options.value, "subnet_ids")
    }
  }

Is this a relatively simple update to the code, or am I overlooking something that would allow me to do this without changing your module?

Thanks!

the following construct should be supported ...

log_publishing_options = [
{
log_type = "INDEX_SLOW_LOGS"
enabled = true
},
{
log_type = "SEARCH_SLOW_LOGS"
enabled = true
},
{
log_type = "ES_APPLICATION_LOGS"
enabled = true
}
]
}

To minimise cost of aged indexes, S3/cache storage is available to move older index content.
https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/ultrawarm.html#ultrawarm-enable

Please consider adding these optional cluster properties to main.tf section dynamic "cluster_config"

"WarmEnabled": true,
"WarmCount": 6,
"WarmType": "ultrawarm1.medium.elasticsearch"

Hi,

Every time I run terraform apply, its deleting the existing domain and recreating a completely new one. Is there a way we can avoid this?

Thanks,
RR

The new module version (0.6.0) leads to a configuration drift for a warm node.

Configuration drift

      ~ cluster_config {
            dedicated_master_count   = 0
            dedicated_master_enabled = false
            instance_count           = 2
            instance_type            = "t3.small.elasticsearch"
          ~ warm_count               = 0 -> 2
            warm_enabled             = false
          + warm_type                = "ultrawarm1.medium.elasticsearch"
            zone_awareness_enabled   = true

            zone_awareness_config {
                availability_zone_count = 2
            }
        }

How to reproduce

1. create Elasticsearch using module with no warm nodes configuration (or set cluster_config_warm_enabled=false as an option).
2. terraform apply: resources were created.
3. terraform plan: shows the configuration drift above.
4. terraform apply: changes are applied.
5. terraform plan: shows the configuration drift again.

When switching back to the previous module version (0.5.1) problem disappears.

Terraform & provider version

terraform {
  required_version = "= 0.13.3"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "3.7.0"
    }
  }
}

Code to reproduce

module "es" {
  # newer module version (0.6.0) shows configuration drift for warm nodes
  source                          = "git::https://github.com/lgallard/terraform-aws-elasticsearch?ref=0.6.0"
  domain_name                     = "es_domain_name"
  elasticsearch_version           = "7.7"
  node_to_node_encryption_enabled = true
  vpc_options_security_group_ids = ["sg-03333333333333333"]
  vpc_options_subnet_ids         =  ["subnet-09999999999999999", "subnet-02222222222222222", "subnet-05555555555555555"]

  cluster_config = {
    dedicated_master_enabled = false
    instance_count           = 2
    instance_type            = "t3.small.elasticsearch"
    zone_awareness_enabled   = true
    availability_zone_count  = 2
  }

  ebs_enabled = true
  ebs_options_volume_size = 20
  encrypt_at_rest_enabled = true
}

https://aws.amazon.com/ru/blogs/big-data/introducing-cold-storage-for-amazon-opensearch-service/

When setting the option: advanced_security_options_enabled = "true"

Terraform states that:
Error: Error creating ElasticSearch domain: ValidationException: You must enable EnforceHTTPS in the domain endpoint options to use advanced security.

but there is no option to enable EnforceHTTPS on the module

On the resource "aws_elasticsearch_domain" definition it is possible to add:

domain_endpoint_options {
    enforce_https = true
  }

This could be usefull.

A variable to choose if the resources are gonna be created.

// Elasticsearch service
module "aws_es0" {
  source  = "lgallard/elasticsearch/aws"
  version = "~> 0.6.1"

  create = var.environment == "PRO" ? true : false

  domain_name                                    = "elasticsearch0"
  elasticsearch_version                          = "7.8"
  vpc_options_subnet_ids                         = module.my_vpc.private_subnets
  node_to_node_encryption_enabled                = "true"
  snapshot_options_automated_snapshot_start_hour = "23"


  cluster_config = {
    dedicated_master_enabled = "true"
    instance_count           = "3"
    instance_type            = "t3.medium.elasticsearch"
    zone_awareness_enabled   = "true"
    availability_zone_count  = "3"
  }
.
.
.
}

If the service role already existed in the account, the error will occur.
Error: Error creating service-linked role with name es.amazonaws.com: InvalidInput: Service role name AWSServiceRoleForAmazonElasticsearchService has been taken in this account, please try a different suffix.

Hi,

When setting cognito enabled
and completing variables.
cognito_options_enabled = true
cognito_options_identity_pool_id = "${local.cognito_options_identity_pool_id}"
cognito_options_role_arn = "${local.cognito_options_role_arn}"
cognito_options_user_pool_id = "${local.cognito_options_user_pool_id}"

terraform/terragrunt fails with

Error: Error creating ElasticSearch domain: ValidationException: Enable fine-grained access control or apply a restrictive access policy to your domain

on main.tf line 1, in resource "aws_elasticsearch_domain" "es_domain":
1: resource "aws_elasticsearch_domain" "es_domain" {

although I have a policy specified.

access_policies = templatefile("access_policies.tpl", {
region = "${local.aws_region}",
account = "${local.aws_account_id}",
domain_name = "${local.elastic_domain}"
})