Overview

cloud-sdk-go do not use validate_only parameter for deployment update api.
ElasticCloud's REST API can use it.
reference: https://www.elastic.co/guide/en/cloud-enterprise/current/update-deployment.html#ece_query_parameters_13

Possible Implementation

#458

Testing

actual use deployment update api with validate_only by #458

Context

I can not use validate_only parameter for deployment update api

Your Environment

Overview

Ideally, we need a make release to be able to push releases to the github repository. There's a VERSION var in the makefile but it's outdated and not used for anything.

ECE Version Info
ESS production
Bug description:

The update from 8.1.2 to 8.1.3 updates Elasticsearch, Kibana but it does not update the Integrations server

Steps to reproduce:

• Create a cluster running 8.1.2 with the Elastic terraform provider

terraform {
  required_version = ">= 0.12.29"

  required_providers {
    ec = {
      source  = "elastic/ec"
      version = "0.4.0"
    }
  }
}

provider "ec" {
  endpoint = "https://cloud.elastic.co"
  apikey = "<<REDACTED>>"

  verbose = false
}

# Create an Elastic Cloud deployment
resource "ec_deployment" "main" {
  name = "xyz"

  # Mandatory fields
  region                 = "gcp-us-west2"
  version                = "8.1.2"
  deployment_template_id = "gcp-io-optimized-v2"

  elasticsearch {
    autoscale = true
  }

  kibana {}

  integrations_server {
    topology {
      zone_count = 2
      size = "2g"
    }
  }
}

• Update the Cluster to 8.1.3 using terraform too

terraform {
  required_version = ">= 0.12.29"

  required_providers {
    ec = {
      source  = "elastic/ec"
      version = "0.4.0"
    }
  }
}

provider "ec" {
  endpoint = "https://cloud.elastic.co"
  apikey = "<<REDACTED>>"

  verbose = false
}

# Create an Elastic Cloud deployment
resource "ec_deployment" "main" {
  name = "xyz"

  # Mandatory fields
  region                 = "gcp-us-west2"
  version                = "8.1.3"
  deployment_template_id = "gcp-io-optimized-v2"

  elasticsearch {
    autoscale = true
  }

  kibana {}

  integrations_server {
    topology {
      zone_count = 2
      size = "2g"
    }
  }
}

• Check that the activities end successfully and the integration server is not updated

Readiness Checklist

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

We are running ECE 3.4.1 and using cloud-sdk-go to automate some of the management tasks like creating ES/Kibana deployments on our ECE platform.

We used to build against [email protected] , which is quite old at this time. So I am considering to update to the latest version 1.11.0 . Building against [email protected] gives me an error saying KibanaClusterPlan do not have ZoneCount field.

Since we also set the ZoneCount in KibanaClusterTopologyElement, I tried to remove the line. The code compiles but trying to create a deployment with kibana will give an error:

api error: 1 error occurred:
        * clusters.cluster_invalid_plan: Zone Count not specified but there is no default (resources.kibana[0].zone_count)

I expected it to work with ECE 3.4.1 but apparently the ECE api endpoint is still requiring this field to be set.

In the old version it had the field:

In the newer version it got removed: https://github.com/elastic/cloud-sdk-go/blob/master/pkg/models/kibana_cluster_plan.go#L38-L50

It seems that the last version that having this field is 1.7.0, and we can confirm that 1.7.0 works with our codebase.
The first version broke the behavior is 1.8.0

Is this a bug or behavior discrepancy between the ECE platform and the Elastic Cloud platform?

Your Environment

• Version used: 1.11.0 , 1.8.0
• Environment name and version (e.g. Go 1.9): Go 1.8
• Server type and version: ECE 3.4.1
• Operating System and version: CentOS7
• Link to your project: sorry, internal tool of the company

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Edited/Blocked

These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.

• fix(deps): update go-openapi packages (github.com/go-openapi/errors, github.com/go-openapi/runtime, github.com/go-openapi/spec, github.com/go-openapi/strfmt, github.com/go-openapi/swag, github.com/go-openapi/validate)

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Overview

At the moment when I want to change the template I get error messages about missing instance configuration

Possible Implementation

Now the template to be used knows about the resources that will be needed.
Can we please add this feature that the missing part will be merged into the provideded template?

Testing

Context

I did following steps to change a deployment:

> ecctl init
> ecctl deployment list
> ecctl deployment template list --region gcp-europe-west1
> ecctl deployment show ....
check used template of deployment:
> ecctl deployment show ... --generate-update-payload | jq -r '.resources.elasticsearch[].plan.deployment_template.id'
generate payload for updating the deployment
> ecctl deployment show ... --generate-update-payload > test.json
> vi test.json
---> changed template gcp-io-optimized to gcp-hot-warm
> ecctl deployment update ... -f test.json --track

--> results in error message

  cluster.invalid_instance_config: For template [Hot-Warm Architecture] the plan definition should contain instance configuration [gcp.data.highio.1,gcp.data.highstorage.1]. Only [gcp.data.highio.1] was supplied (resources.elasticsearch[0])

Your Environment

My test cloud.elastic.co tested

Overview

There are certain situations where we can have non-fatal errors in a plan; however, currently the way the tracking code works is that any and all responses along the way that might contain errors/warnings get bubbled back up as a fatal issue for that plan. In reality, there are situations where these things can happen, but the plan is still successful.

Possible Implementation

It seems to me we need monitor the stream of events coming from the response tracking in order to evaluate whether the above is true.

The relevant code paths:

• Tracking
• Streaming

How Vacates consume it (AFAICT it's the only consumer):

• VacateCluster

POC (WIP): master...Z3r0Sum:cloud-sdk-go:feat-overall-plan-success-tracking. Summary:

• This is an attempt to avoid sending what would be perceived as a 'fatal'
  set of errors back to the consumer of the TrackChange() func.
• The scenario we are trying to account for:
  • There are errors/warnings along the life of a plan as it relates
    to a vacate
  • The above do not cause the plan to halt
  • The plan eventually succeeds
  • The consumer receives these errors in a fatal manner when in
    reality there is no cause for action
• Problems with current approach:
  • Does not make the end user aware of any problems at all if the
    plan succeeded. Not sure we care?
• TODO: thorough testing and confirm assumptions about the following:
  • Do all fatal errors manifest themselves properly in the
    plan-completed step, or prevent us from getting to it. If both
    are true, this approach appears safe other than the problem
    outlined above.

Testing

• Added unit tests
• Integration testing is needed/required

Context

Vacates report a failure when in reality it was a non-issue and the plan succeeded along with the vacate of the instance.

Your Environment

Internal SaaS.

Overview

We're currently using up to 4 http.RoundTripper implementations which get nested and nested, 4 layers is enough.

It would be best if instead of segregating the code in separate objects, we instead unified them to a custom single one which does the same.

Overview

Currently, vacate operations will ignore `enterprisesearch" workloads, It should be added to the list of resources which can be vacated.

Description

There is a strong dependency between the API and the various cloud clients. The cloud clients should make sure that all commands, or endpoints, are available/work for their intended audience.
This can be achieved through integration testing.

Possible Solution

• Test cloud-sdk-go using mocks for api endpoints taken from apidocs*.json files
• Refactor and leverage existing CLI integration tests

[UPDATE]
Work on mocks has been halted while we discuss how to improve our spec in order to generate said mocks.

For the time being we will make use of the prism validation proxy against the ESS API to validate parity between it and our api spec

Overview

It would be nice to use validate_only query parameter when creating/updating a deployment to just validate the deployment definition. It looks like the elastic client does support the param, but the elastic api doesn't yet.
https://www.elastic.co/guide/en/cloud-enterprise/current/create-deployment.html

Possible Implementation

Add a property for ValidateOnly to CreateParams/UpdateParams struct and pass them when calling CreateDeployment/UpdateDeployment

type UpdateParams struct {
	*api.API

        ...

	// Optional values
	SkipSnapshot      bool
	HidePrunedOrphans bool
	ValidateOnly      bool // Added
}

func Update(params UpdateParams) (*models.DeploymentUpdateResponse, error) {
	...
	
	res, err := params.V1API.Deployments.UpdateDeployment(
		deployments.NewUpdateDeploymentParams().
			WithDeploymentID(params.DeploymentID).
			WithBody(params.Request).
			WithSkipSnapshot(&params.SkipSnapshot).
			WithHidePrunedOrphans(&params.HidePrunedOrphans).
			WithValidateOnly(&params.ValidateOnly), // Added
		params.AuthWriter,
	)

        ...

	return res.Payload, nil
}

Testing

I couldn't find any test code for params, but I'm happy to add a test case

Context

It's always nicer to allow user to validate the deployment definition first before creating/updating a deployment

Readiness Checklist

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

That deploymentapi.NewUpdateRequest() returns a valid DeploymentUpdateRequest for all version.

Current Behavior

Using a new DeploymentUpdateRequest from deploymentapi.NewUpdateRequest() gives an error for clusters >=7.10 (my test deployment was 7.11).

error updating deployment: api error: 4 errors occurred:
	* clusters.cluster_invalid_plan: Cluster must contain at least a master topology element and a data topology element. 'master' node type is missing,'data' node type is missing,'master' node type exists in more than one topology element (resources.elasticsearch[0].cluster_topology)
	* clusters.cluster_invalid_plan: Node types are not specified for the cluster topology. You must specify at least one of available node types: [data,master,ingest]. (resources.elasticsearch[0].cluster_topology[0].node_type)
	* deployments.missing_topology_element: Deployment plan definition with template [I/O Optimized] missing topology element with instance configuration: [Some(aws.data.highio.i3)], node type: [Some(ElasticsearchNodeType(Some(true),Some(true),Some(true),None))] (resources.elasticsearch[0])
	* deployments.unrecognized_topology_elements: Deployment plan definition includes topology element not supported by template [I/O Optimized]. Instance configuration: [Some(aws.data.highio.i3)], node types: [None] (resources.elasticsearch[0])

Possible Solution

Steps to Reproduce

1. Create a 7.11 deployment
2. GET a DeploymentGetResponse for this deployment.

ok, err := ifs.CloudSDKDeployments.GetDeployment(getParams, ifs.CloudSDKAuthInfo)
if err != nil {
	return err
}
res := ok.GetPayload()

3. Use NewUpdateRequest to generate a DeploymentUpdateRequest

newReq := deploymentapi.NewUpdateRequest(res)

4. Update the deployment and see errors

params := deployments.NewUpdateDeploymentParams().
		WithDeploymentID(deploymentID).
		WithBody(req)
_, err = ifs.CloudSDKDeployments.UpdateDeployment(params, ifs.CloudSDKAuthInfo)

Context

Your Environment

• Version used: 1.2.0
• Environment name and version (e.g. Go 1.9):
• Server type and version:
• Operating System and version:
• Link to your project:

cc @marclop. Pretty sure this is the same node_type v's node_role change that I saw you implementing support for in the terraform provider !

Seems like the link to the ecctl example code under get started in the root Readme.md is broken:

https://github.com/elastic/cloud-sdk-go#getting-started

I think the URL might have changed?

Description

MS39 is live. Our SDK client and models should be updated to that version.

Additionally some changes with the users API have been made, where
the PATCH method no longer takes a type models.User but a string.
This means the userapi.Update and userapi.UpdateCurrent functions
need to be modified to reflect this change.

Readiness Checklist

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

It would appear that on deployments that are of version >= 8.0.0 the APM component is now represented as the "integration server". Where as prior versions are under the "apm" umbrella. There's no representation of this in the code that handles vacates: https://github.com/elastic/cloud-sdk-go/blob/master/pkg/api/platformapi/allocatorapi/vacate.go#L596-L630. There might be other gaps in the SDK; however, vacates are probably the most important to address. This results in the following errors when trying to perform a vacate with ecctl and the latest version of the cloud-sdk-go:

{
  "errors": [
    {
      "message": "plan track change: plan track change: couldn't find a deployment containing Kind apm with ID 5e74e4fb576548c3a4bfdab7cae183e9"
    },
    {
      "message": "plan track change: plan track change: couldn't find a deployment containing Kind apm with ID 890a50f1ac5542c69315e42c859543b4"
    },
    {
      "message": "plan track change: plan track change: couldn't find a deployment containing Kind apm with ID 9415cc233ba84b69bd4b714b713cbf2f"
    },
    {
      "message": "plan track change: plan track change: couldn't find a deployment containing Kind apm with ID b00486f33d9f47eaa5bbee1a1bcb84d4"
    },
    {
      "message": "plan track change: plan track change: couldn't find a deployment containing Kind apm with ID b418aef280014aba8313bbb476de8deb"
    },
    {
      "message": "plan track change: plan track change: couldn't find a deployment containing Kind apm with ID c5aa70a1a70e4461b3a39ac288f6fd78"
    }
  ]
}

See: https://github.com/elastic/cloud-sdk-go/blob/master/pkg/models/move_clusters_details.go#L35-L59.

In the cloud-sdk-go for the vacate flow - we use the move API with the validation query param to get the contents of what needs to be moved off the allocator - this determines the kind, it's a bit peculiar to me that the integrations server pieces are manifesting as apm, but there might be a reason for that behind the scenes. Anyways, here's the rough flow of things:

https://github.com/elastic/cloud-sdk-go/blob/master/pkg/api/platformapi/allocatorapi/vacate.go#L125-L150 -> https://github.com/elastic/cloud-sdk-go/blob/master/pkg/api/platformapi/allocatorapi/vacate.go#L203-L268

https://github.com/elastic/cloud-sdk-go/blob/master/pkg/api/platformapi/allocatorapi/vacate.go#L148 maps to the move clusters details type I linked at the start, which in turn was called by: https://github.com/elastic/cloud-sdk-go/blob/master/pkg/client/platform_infrastructure/platform_infrastructure_client.go#L2114-L2148

Current Behavior

Cannot properly track integration server instances during a vacate on deployments >= 8.0.0.

Possible Solution

See above.

Steps to Reproduce

See above.

Overview

Expose the traffic filtering APIs.

Overview

Background

There seems to be an agreement that we're to move away from top level region interpolation in favour of using region as a parameter in API calls. We'll eventually have new endpoints which accept region as a parameter.
In the meantime the /api/v1/platform/... routes still need an interpolated /api/v1/regions/<region>/platform/... in the endpoint to target that specific region.

Description

To be able to provide a better experience for consumers of the Elastic Cloud SDK, it would be ideal to move away from region-bound api.API instances to a region-less instance which can accommodate region-specific calls.

Doing so will most likely require some modifications to api.CloudClientRuntime and most likely the api.API structure too. The bigger investment will come to consumers of the SDK will require crafty modifications.

Possible Implementation

Requires investigation to reach a good API

In recent ECE releases, we expose new attributes that allow us to determine the overall health of allocators based on more attributes than just the connectivity, including data around several other health checks reported by the allocator itself.

However, the API exposes a hardcoded constant for searching for unhealthy allocators that does not use those attributes and rather depends on just its connectivity, which causes that it will not display allocators failing in some specific failure modes other than lack of connectivity:

This attribute is used by ecctl:
https://github.com/elastic/ecctl/blob/056f4d673adc428362a987b697e589579580c3ee/cmd/platform/allocator/list.go#L63
and other SDK-powered applications when filtering for unhealthy allocators. We would want to update this query to allow to also target allocators that are connected but not healthy.

Readiness Checklist

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

> ecl deployment show ${deployment_id} --generate-update-payload

should work on deployments with legacy templates

Current Behavior

> ecl --config ecqa deployment show 8435137e1d624a1d9810728b8b382001 --generate-update-payload
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x18e76b3]
goroutine 1 [running]:
github.com/elastic/cloud-sdk-go/pkg/api/deploymentapi.parseKibanaGetResponse(0xc0004cbc50, 0xc00060fab0, 0xd, 0x0)
    /Users/marc/go/pkg/mod/github.com/elastic/[email protected]/pkg/api/deploymentapi/update_payload.go:121 +0x123
github.com/elastic/cloud-sdk-go/pkg/api/deploymentapi.NewUpdateRequest(0xc0004e4330, 0x7ffeefbff8b2)
    /Users/marc/go/pkg/mod/github.com/elastic/[email protected]/pkg/api/deploymentapi/update_payload.go:48 +0x31d
github.com/elastic/ecctl/cmd/deployment.glob..func8(0x3128f40, 0xc0001b4600, 0x1, 0x4, 0x0, 0x0)
    /Users/marc/go/pkg/mod/github.com/elastic/[email protected]/cmd/deployment/show.go:95 +0x527
github.com/spf13/cobra.(*Command).execute(0x3128f40, 0xc0001b45c0, 0x4, 0x4, 0x3128f40, 0xc0001b45c0)
    /Users/marc/go/pkg/mod/github.com/spf13/[email protected]/command.go:842 +0x460
github.com/spf13/cobra.(*Command).ExecuteC(0x3126540, 0x3107640, 0x2, 0x2)
    /Users/marc/go/pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x349
github.com/spf13/cobra.(*Command).Execute(...)
    /Users/marc/go/pkg/mod/github.com/spf13/[email protected]/command.go:887
github.com/elastic/cloud-cli/cmd.Execute(0x248667c, 0x5, 0x24ee440, 0x28, 0x248b610, 0x7, 0x248bf68, 0x9, 0x0)
    /Users/marc/repos/elastic/cloud-cli/cmd/root.go:79 +0x1cf
main.main()
    /Users/marc/repos/elastic/cloud-cli/main.go:35 +0x85

Steps to Reproduce

1. create a deployment
2. edit Kibana configuration in advanced edit:

              "memory_per_node": 1024,
              "node_count_per_zone": 1,
              "instance_configuration_id": "aws.kibana.classic",
              "zone_count": 1,

3. run ecl deployment show ${deployment_id} --generate-update-payload and get above parseKibanaGetResponse stack

Context

It prevented o11y team to use native ecl commands in upgrade script, so we resorted to using raw ecl api 'v1/deployment' calls

Your Environment

• Version used: 1.2.2 (Linux/MacOS)
• Link to your project: https://github.com/elastic/cloud/tree/master/found-deployment/docker/olly_upgrader

Notes

Initially was reported in https://github.com/elastic/cloud/issues/69425 - it has more example failures (parseElasticsearchGetResponse, parseApmGetResponse)

Overview

Add a doc.go file at the repository root which includes some usage help, examples and better documentation.

Possible Implementation

Examples:

• https://pkg.go.dev/cloud.google.com/go?tab=doc
• https://pkg.go.dev/github.com/aws/aws-sdk-go?tab=doc

Overview

Currently, many of our packages support outputting to JSON / text and our multierror.Error does not, it does support a custom formatter function, which is easy to replace.

The best way forward with this would be to allow a JSON format setting in the multierror.Prefixed itself, so that when Error is called, the output is JSON.

Possible Implementation

Writing a package function might be the best way forward as it can provide a good API for clients to use without the need to cast to a type and having to handle that logic in the client.

func WithFormat(error, FormatFunc) error 

Accepting an error and a format function is a good way to provide an open implementation, following that, writing a FormatFunc which implements a JSON formatted output can be handy.

Description

The stack api is no longer part of the platform api (/api/v1/regions/{region}/stack/). Our APIs should reflect that.
So everything within pkg/api/platformapi/stackapi/ should be moved to pkg/api/stackapi/.

Overview

the sync/pool package is very verbose at the moment and operating it requires a lot of boilerplate code for task specific pools.

It'd be best if we had some abstractions built on top which are purpose-specific rather than catering for all possible cases of a pool.

Possible Implementation

Adding a sync/poolutil package with at least the worker use case.

Description

MS39 is live, we should update our apidocs and client/models to reflect that.

Additionally, it might make sense to update the go-openapi deps that can't be automatically updated (#141, #140) as part of this PR, since we are generating new client/models

Overview

After the merge of #124, and #125, all newly added /platform APIs need to be modified to require a Region parameter to be set, meaning that if not specified a new error in the multi error will be appended.

Possible Implementation

The ideal implementation would open a Pull Request per API package:

• allocatorapi @marclop (#133) elastic/ecctl#302
• configurationtemplateapi @marclop (#146) elastic/ecctl#316
• constructorapi @marclop (#135) elastic/ecctl#305
• enrollmenttokenapi @marclop (#137) elastic/ecctl#307
• instanceconfigapi @marclop (#152) elastic/ecctl#331
• platformapi @marclop (#148) elastic/ecctl#320
• proxyapi @karencfv (#149) elastic/ecctl#321
• filteredgroupapi @karencfv (#144) elastic/ecctl#315
• roleapi @karencfv (#139) elastic/ecctl#309
• runnerapi @karencfv (#138) elastic/ecctl#308
• snaprepoapi @karencfv (#136) elastic/ecctl#304
• stackapi @karencfv (#134) elastic/ecctl#303

Dependabot couldn't parse the go.mod found at /go.mod.

The error Dependabot encountered was:

go: github.com/go-openapi/[email protected] requires
	gopkg.in/[email protected]: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /opt/go/gopath/pkg/mod/cache/vcs/9241c28341fcedca6a799ab7a465dd6924dc5d94044cbfabb75778817250adfc: exit status 128:
	fatal: The remote end hung up unexpectedly

View the update logs.

Overview

Our awesome pool is heavily used in cloud-deployer and it's super fast!
We would like to send dynamic slack notifications during the workers' processing like how many items have been processed so far, how many pending, errors etc.

Possible Implementation

I'd consider two different implementations

Add API so that consumers call it to get the information needed

This gives consumers the flexibility of poking the API directly and processing the results as required. Ideally, I'd expect that results are the structure that has been passed in workers during pool initialization

Inject a function inside the pool

This would require the addition of 2-3 new fields during pool creation

• Function to inject
• Frequency
• Initial timeout before injecting the function for the first time

The pool would know then when to run this function internally
This gives less flexibility to the consumers but hides all the details inside the pool

Context

Make our awesome pool more awesome

Description

Several functions and methods withinpkg/api return single errors when in fact these should be multierrors.

Overview

ECE 2.8.0 is scheduled for Tuesday the 9th, we should release the SDK accordingly.

• Merge #279
• Generate release notes #287.
• Cut a 1.2.0 branch
• Bump the master version to v1.3.0 and ECE version to 2.9.0 #285.
• Update the apidocs.json and apidocs-user.json from the MS-53 branch #285.

Description

Github Actions job modified in #360 failed because we don't have any changes so far.

To make the job more robust we need to generate a version folder in .changelog/ on a version bump. It should be implemented in https://github.com/elastic/cloud-sdk-go/blob/master/build/Makefile.version

Description

Move apivalidator CLI tool to this repository in order to be able to
run apivalidation tests on make swagger. This will ensure that
all endpoints exist on the live public API when updating our
client/models.

Overview

Currently, the default Go User-Agent will be used for all API calls.
It would be best that clients could set that header using the api.Config settings.

Possible Implementation

Adding an optional User-Agent field which is used to construct our custom http.RoundTripper.

Context

Better metrics.

Readiness Checklist

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

List deployment example in README.md should compile.

Current Behavior

$ go run main.go
# command-line-arguments
./main.go:24:58: cannot use ess.V1API (type *"github.com/elastic/cloud-sdk-go/pkg/client".Rest) as type *api.API in field value

Steps to Reproduce

1. Add code from README.md to main.go.
2. go run main.go

Context

Unable to get started with cloud-sdk.

Overview

As part of the revamp that Go has done in the errors (wrapping and unwrapping), it would great to add a new error type which implements the today's api.UnwrapError and apierror.Unwrap in that type with the addition of the Unwrap method.

Readiness Checklist

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

Tests should be reliable.

Current Behavior

The "TestPoolAdd/Add_succeeds" test sometimes fails when running it through GitHub actions with the following error:

--- FAIL: TestPoolAdd (0.02s)
    --- FAIL: TestPoolAdd/Add_succeeds (0.01s)
##[error]        pool_test.go:288: Pool.Add() = [0x7678c8], want {[] [0x7678c8 0x7678c8] [{} {}]}
##[error]        pool_test.go:291: Pool.Add() error = pool: failed adding work, queue full, want <nil>
##[error]        pool_test.go:301: Pool.queue = [0x7678c8], want [0x7678c8 0x7678c8]
##[error]        pool_test.go:305: Pool.signals.added = [{}], want [{} {}]

It always passes when running locally.

Steps to Reproduce

1. If the build fails with Github actions check the unit test section and it's most likely that.

Overview

Currently the verbose transport, dumps out all of the request / response to the specified writer, while this is great for debugging purposes, it also exposes the Authorization header which contains credentials.

It would be great to add a bool parameter in the api.VerboseSettings to allow the VerboseTransport to obscure that header.

This will allow us to ship and collect the verbose output cleanly without exposing any secrets.

Of course a step further would be to actually purge any potential from API responses not only headers.

Readiness Checklist

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

When using the stackapi.List API a list of the available stack packs should be returned
ordered from latest to oldest.

Current Behavior

With version 7.10.0 out the list is now ordered in the following way:

VERSION   DELETED   ELASTICSEARCH IMAGE                                     KIBANA IMAGE                                     APM IMAGE
7.9.3     false     docker.elastic.co/cloud-assets/elasticsearch:7.9.3-0    docker.elastic.co/cloud-assets/kibana:7.9.3-0    docker.elastic.co/cloud-assets/apm:7.9.3-0
<...>
7.2.0     false     docker.elastic.co/cloud-assets/elasticsearch:7.2.0-0    docker.elastic.co/cloud-assets/kibana:7.2.1-1    docker.elastic.co/cloud-assets/apm:7.2.0-0
7.10.0    false     docker.elastic.co/cloud-assets/elasticsearch:7.10.0-0   docker.elastic.co/cloud-assets/kibana:7.10.0-0   docker.elastic.co/cloud-assets/apm:7.10.0-0
7.1.1     false     docker.elastic.co/cloud-assets/elasticsearch:7.1.1-0    docker.elastic.co/cloud-assets/kibana:7.1.1-1    docker.elastic.co/cloud-assets/apm:7.1.1-0

Context

Both ecctl and terraform-provider-ec use the first version returned to fill in default
"lastest" values for versions.

Overview

Add implementation for traffic filter APIs.

The /api/deployments/traffic_filter... path is not implemented
and is needed for the terraform work.

A new trafficfilterapi under deploymentapi should be added.

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

Readiness Checklist

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

If a move plan calculation cannot be computed, we should know that and then bubble the appropriate error up rather than assume it was calculated. An example will be outlined below.

Current Behavior

I tried updating to the latest cloud-sdk-go version and ecctl version locally and received the exact same stack trace as older versions (both are on v1.8.0), so I don't perceive this as a new problem and perhaps a backport should be considered when we fix it. Here's the crux of the problem:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1d3103d]

goroutine 90 [running]:
github.com/elastic/cloud-sdk-go/pkg/api/platformapi/allocatorapi.ComputeVacateRequest(0xc000320280, 0xc000799cd8, 0x1, 0x1, 0x2d18240, 0x0, 0x0, 0x0, 0xc0001297cb, 0xc0004fa121, ...)
	/Users/stephenschmidt/go/pkg/mod/github.com/elastic/[email protected]/pkg/api/platformapi/allocatorapi/vacate.go:708 +0xc1d
github.com/elastic/cloud-sdk-go/pkg/api/platformapi/allocatorapi.newMoveClusterParams(0xc000497c70, 0xc0003d4920, 0x1, 0x1)
	/Users/stephenschmidt/go/pkg/mod/github.com/elastic/[email protected]/pkg/api/platformapi/allocatorapi/vacate.go:404 +0x3a7
github.com/elastic/cloud-sdk-go/pkg/api/platformapi/allocatorapi.moveClusterByType(0xc000497c70, 0xc000497c70, 0x0)
	/Users/stephenschmidt/go/pkg/mod/github.com/elastic/[email protected]/pkg/api/platformapi/allocatorapi/vacate.go:441 +0x45
github.com/elastic/cloud-sdk-go/pkg/api/platformapi/allocatorapi.VacateCluster(0xc000497c70, 0xc000488fa8, 0x135bf26)
	/Users/stephenschmidt/go/pkg/mod/github.com/elastic/[email protected]/pkg/api/platformapi/allocatorapi/vacate.go:310 +0x77
github.com/elastic/cloud-sdk-go/pkg/api/platformapi/allocatorapi.VacateClusterInPool(0x22de040, 0xc000497c70, 0xc00005f501, 0x0)
	/Users/stephenschmidt/go/pkg/mod/github.com/elastic/[email protected]/pkg/api/platformapi/allocatorapi/vacate.go:297 +0x53
github.com/elastic/cloud-sdk-go/pkg/sync/pool.Work.func2(0xc00010e180, 0xc00058e420, 0xc00058e600, 0xc00058e840, 0xc00058e960, 0xc00010e1e0, 0x21da570, 0x6fc23ac00, 0x22de040, 0xc000497c70, ...)
	/Users/stephenschmidt/go/pkg/mod/github.com/elastic/[email protected]/pkg/sync/pool/worker.go:82 +0x44
created by github.com/elastic/cloud-sdk-go/pkg/sync/pool.Work
	/Users/stephenschmidt/go/pkg/mod/github.com/elastic/[email protected]/pkg/sync/pool/worker.go:81 +0x21d

"Debugging" locally:

Computation complete payload: &{Failures:0xc0004f0080 Moves:0xc0004f0100}
Computation complete payload error: &{ApmClusters:[] AppsearchClusters:[] ElasticsearchClusters:[] EnterpriseSearchClusters:[] KibanaClusters:[]}
Computation complete payload moves: <nil>
We think this is ent search, plan config: <nil>

panic: runtime error: invalid memory address or nil pointer dereference

The problem is indeed the plan computation is returning nil and we don't bother to check for nil values here: https://github.com/elastic/cloud-sdk-go/blob/master/pkg/api/platformapi/allocatorapi/vacate.go#L708, but more specifically on the field:

https://github.com/elastic/cloud-sdk-go/blob/master/pkg/models/move_clusters_details.go#L54 -> https://github.com/elastic/cloud-sdk-go/blob/master/pkg/models/move_enterprise_search_details.go#L41

We should be checking that we have a calculated plan for everything, rather than just assuming we'll get one as a good defensive programming practice. Mym recommendation is that each stack component and cluster in its corresponding slice should have the check and bubble the error up for a a nil calculated plan here: https://github.com/elastic/cloud-sdk-go/blob/master/pkg/api/platformapi/allocatorapi/vacate.go#L633-L718.

At the end of the day it's a chicken and egg issue, so if ES is in a really bad state, something like entsearch can't be checked; thus, we can't calc a plan. Here's an example:

{
  "cluster_type": "elasticsearch",
  "details": "Could not make sure [ElasticsearchCluster(<redacted>)] is up and running",
  "caused_by": "no.found.constructor.plan.apm.ClusterNotReachable: Unexpected response [401 Unauthorized, {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [cloud-internal-enterprise_search-server] for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [cloud-internal-enterprise_search-server] for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}]"
}

Possible Solution

Noted above, but to make parsing easier:

We should be checking that we have a calculated plan for everything, rather than just assuming we'll get one as a good defensive programming practice. Mym recommendation is that each stack component and cluster in its corresponding slice should have the check and bubble the error up for a a nil calculated plan here: https://github.com/elastic/cloud-sdk-go/blob/master/pkg/api/platformapi/allocatorapi/vacate.go#L633-L718.

I also don't mind working on this in the near future, but if someone wants to tackle it within the next few weeks, feel free.

Steps to Reproduce

Get ES into a really bad state with another stack component running along side it and attempt to vacate said stack component.

Context

I think enough was provided above :).

Your Environment

• Version used: v1.8.0 for all the things
• Environment name and version (e.g. Go 1.9): 1.16

Overview

In the future the ESS API will require all requests being made against it to contain the Content-Type: application/json header.

I'm fairly sure that our SDK already does this but we should double check that's the case writing appropriate tests that ensure we're doing so.

Overview

The current implementation of pkg/plan does not use the deployment API but rather an endpoint in the /cluster API. Ideally we need a package which uses the deployment API.
Mainly for the ability to track AppSearch changes but also would provide a more wholistic approach to the changes in a deployment, even if only 1 resource is "impacted" it might change other resources indirectly.

Readiness Checklist

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

Passing a GetDeploymentResponse to NewUpdateRequest will create a valid DeploymentUpdateRequest when deployment autoscaling is enabled.

Current Behavior

Currently the NewUpdateRequest function will exclude topology elements were MemoryPerNode = 0

When operating on deployments with autoscaling : true this will remove valid topology elements, and return an error like this

"Deployment plan definition with template [I/O Optimized] missing topology elements: [coordinating,warm,cold]. Only [hot_content,frozen,master,ml] was supplied."

Possible Solution

Check for deployment autoscaling before removing topology elements.

Steps to Reproduce

1. GetDeploymentResponse from a deployment with autocaling enabled
2. Create a DeploymentUpdateRequest with NewUpdateRequest(res)
3. Attempt to no-op
4. Recieve error

Context

We use NewUpdateRequest in our tooling to easily generate update payloads.

Your Environment

• Version used: v1.6.0
• Environment name and version (e.g. Go 1.9): go1.16.3 darwin/amd64
• Server type and version:
• Operating System and version:
• Link to your project:

Overview

With MS53, we're going to ship new fields in the plan models so the API spec needs to be updated and make a bunch of changes related to these fields being bubbled up:

• Plan Tracker
• Vacate JSON errors.

Hi all

The current master version of cloud sdk supports observability settings for Elastic cloud deployments, whereas these settings are not available in the current release.

Any plans for releasing the new sdk version.

Thanks

Readiness Checklist

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

When leveraging the cloud client method for: https://github.com/elastic/cloud-sdk-go/blob/master/pkg/client/deployments/deployments_client.go#L1552-L1586 which hits the following API endpoint: https://www.elastic.co/guide/en/cloud-enterprise/current/set-deployment-resource-raw-metadata.html, we should not be escaping the string JSON body.

Current Behavior

Noted above.

Possible Solution

See PR:

Steps to Reproduce

1. Attempt to use the above method for SetDeploymentResourceRawMetadata

Context

Unable to leverage this API call via the SDK.

Description

Currently our deployment template APIs and commands are using the default "cluster" format. This is not a problem in itself, but as the API moves to deprecate the clusters api, we should do the same by moving to using the "deployment" format /platform/configuration/templates/deployments?format=deployment.

This work is also related to bringing back the flag based deployment create and update. Since the new deployment template API will use the same format (DeploymentCreateRequest), this will be a first step into migrating to the new endpoint and having that ready for when the new endpoint is available.

Readiness Checklist

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

SDK should send the correct request body without extra fields

Current Behavior

Update deployment metadata is sending field last_resource_plan_modified which is not part of the request schema.

Detected unexpected fields for [no.found.adminconsole.api.v1.routes.deployments.models.DeploymentUpdateRequest.DeploymentUpdateMetadata], not failing as strict deserialization has been disabled

io.circe.DecodingFailure$$anon$2: Unexpected field: [last_resource_plan_modified]; valid fields: system_owned, hidden, tags: DownField(metadata)

The request being made is to PUT /api/v1/deployments/{deployment_id} with user agent cloud-sdk-go/2.10.0

Possible Solution

Steps to Reproduce

Context

Your Environment

• Version used:
• Environment name and version (e.g. Go 1.9):
• Server type and version:
• Operating System and version:
• Link to your project:

Overview

The S3Config struct in the snaprepoapi is missing the PathStyleAccess field.

Possible Implementation

Add the missing PathStyleAccess field to the struct S3Config which is used for configuring S3 snapshot repositories.

Context

We need to set this property, as our S3 backend requires it.

Your Environment

ECE with a custom S3 backend.

See also https://www.elastic.co/guide/en/elasticsearch/reference/current/repository-s3.html#repository-s3-client