kayac / ecspresso Goto Github PK
View Code? Open in Web Editor NEWecspresso is a deployment tool for Amazon ECS
License: MIT License
ecspresso is a deployment tool for Amazon ECS
License: MIT License
"schedulingStrategy": "REPLICA" (アプリケーション)と "DAEMON" (Datadog) のサービスが
同居(同時稼働)する EC2 インスタンス 2台(サービス2つずつ)に対して、
画面に表示されたメッセージを可能な限り記述します。
使用が推奨できない環境に該当するようにも思われるのですが、今回の状況が
表示のみの問題なのか、何か確認しなければならない懸念事項が残っている状態なのかを知りたいです。
❯ ecspresso deploy --config app-green/app-green/app-green-config.yaml --tasks 2 --debug
2020/09/28 18:40:01 app-green/app-green Starting deploy
Service: app-green
Cluster: app-green
TaskDefinition: app-green:13
Deployments:
PRIMARY app-green:13 desired:1 pending:0 running:1
Events:
2020/09/28 18:40:02 app-green/app-green Registering a new task definition...
2020/09/28 18:40:02 app-green/app-green Task definition is registered app-green:14
2020/09/28 18:40:02 app-green/app-green service attributes will not change
2020/09/28 18:40:02 app-green/app-green desired count: 2
2020/09/28 18:40:02 app-green/app-green Updating service tasks...
2020/09/28 18:40:02 app-green/app-green {
Cluster: "app-green/app-green",
DesiredCount: 2,
ForceNewDeployment: false,
Service: "app-green/app-green",
TaskDefinition: "arn:aws:ecs:ap-southeast-1:XXXXXXXXXXX:task-definition/app-green:14"
}
2020/09/28 18:40:06 app-green/app-green Waiting for service stable...(it will take a few minutes)
2020/09/28 18:49:56 app-green/app-green PRIMARY app-green:14 desired:2 pending:0 running:1
2020/09/28 18:49:56 app-green/app-green ACTIVE app-green:13 desired:1 pending:0 running:1
2020/09/28 18:40:38 (service app-green) was unable to place a task because no container instance met all of its requirements. The closest matching (container-instance xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) has insufficient CPU units available. For more information, see the Troubleshooting section of the Amazon ECS Developer Guide.
2020/09/28 18:40:29 (service app-green) has started 1 tasks: (task xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx).
2020/09/28 18:50:01 deploy FAILED. failed to wait service stable: RequestCanceled: waiter context canceled
caused by: context deadline exceeded
~/src/.../infra/ecspresso/green feat/ecspresso-greenReduce* 10m 0s
Currently,diff
command does not cover desiredCount
.
I want to target desiredCount as well as other parameters so that we can manage changes in ecs-service-def.json
.
We can also achieve this with init --force-overwrite
, but this method will not work if you are using templates or jsonnet.
Please add an option like --enable-desired-count
.
create
command has --tasks=1
default value.
So ecspresso create
doesn't respect a desiredCount in service definition by the default.
例えば、CDK から s3EncryptionEnabled を true にしたようなクラスタに対して exec すると、以下のようなエラーがでます。
SessionId: ******:
----------ERROR-------
Encountered error while initiating handshake. KMSEncryption failed on client with status 2 error: Failed to process action KMSEncryption: Installed version of AWS CLI does not support Session Manager encryption feature. Please upgrade to latest version of AWS CLI
session-manager-plugin
を呼ぶ際、引数を 4 つ渡していますが、
https://github.com/kayac/ecspresso/blob/v1/exec.go#L93
これは session-manager-plugin
上 LegacyArgumentLength に該当するようです。
https://github.com/aws/session-manager-plugin/blob/e4ad4017df969b298c3e263f0c8ce9881c0b0a79/src/sessionmanagerplugin/session/session.go#L139
また、IsAwsCliUpgradeNeeded な場合、レガシー判定を受けて ProcessKMSEncryptionHandshakeAction を行うことができません。
https://github.com/aws/session-manager-plugin/blob/65933d1adf368d1efde7380380a19a7a691340c1/src/datachannel/streaming.go#L832
最新の awscli が扱っているようにあるいは、コメントにあるように引数を増やしてあげる必要がありそうです。
https://github.com/aws/aws-cli/blob/45b0063b2d0b245b17a57fd9eebd9fcc87c4426a/awscli/customizations/sessionmanager.py#L83
https://github.com/aws/session-manager-plugin/blob/e4ad4017df969b298c3e263f0c8ce9881c0b0a79/src/sessionmanagerplugin/session/session.go#L101
Hi,
I'm really excited with the jsonnet support. I can remove jsonnet install step 🙂
I tried it and found currently there's no way to pass --ext-str a=b
options.
Do you have plans to support extVar
related options?
Thank you for great tool!
Being able to manage an existing app by a code base is very useful and easily, but on the other hand,
I often want to deploy new service from existing conf files with same tool (= ecspresso).
For example, we have a situation where we want to create a dev, stg, prd env based on the conf files which we we have used in PoC env.
I want to make ecspresso deploy
enable the deployment of new services in addition to existing features.
Alternatively, this could be accomplished by passing a flag (such as --new
) to deploy
subcommand.
I apologize if it's already possible or if I've made suggestions that are outside of the ecspresso's philosophy.
I think we should increase the default to at least 6 events(if not more).
6 because this is the number of events I see in one deployment
First of all I'd like to say thank you for such a great tool, and keep up a good work! 👍
It's not really clear to me at first glance so I would be really thankful if someone could explain it to me what does --force-new-deployment
option do?
ecspresso deploy --force-new-deployment
Is this option perhaps going to force tasks to be stopped, skip the draining part or it is something completely different?
Thanks in advance for your answer! 👋
ecspresso version: v0.17.3
Fargate : 1.3.0
CodeDeployDefault.ECSAllAtOnce is always used instead of the DeploymentConfigName set in the DeploymentGroup.
If you have any solutions, please let me know.
DeploymentGroup
aws deploy get-deployment-group --application-name cluster --deployment-group-name cluster | jq -r '.[].deploymentConfigName'
CodeDeployDefault.ECSCanary10Percent5Minutes
Results of running ecspresso (cluster name,service name,sg, subnet are masked)
ecspresso deploy --config config.yml --debug --tasks=1 --suspend-auto-scaling --force-new-deployment
2020/08/27 23:38:27 service-name/cluster Starting deploy
Service: service-name
Cluster: cluster
TaskDefinition: cluster:817
TaskSets:
PRIMARY cluster:817 desired:1 pending:0 running:1
ACTIVE cluster:818 desired:1 pending:0 running:1
Events:
2020/08/27 23:38:28 service-name/cluster Creating a new task definition by ../task_definition.json
2020/08/27 23:38:28 service-name/cluster Registering a new task definition...
2020/08/27 23:38:28 service-name/cluster Task definition is registered cluster:819
2020/08/27 23:38:28 service-name/cluster desired count: 1
2020/08/27 23:38:28 service-name/cluster No scalable target for service/cluster/service-name
2020/08/27 23:38:28 service-name/cluster appSpecContent: version: "0.0"
Resources:
- TargetService:
Type: AWS::ECS::Service
Properties:
TaskDefinition: arn:aws:ecs:ap-northeast-1:123456789012:task-definition/cluster:819
LoadBalancerInfo:
ContainerName: application
ContainerPort: 8080
PlatformVersion: 1.3.0
NetworkConfiguration:
awsvpcconfiguration:
assignpublicip: DISABLED
securitygroups:
- sg-1234567890abcd
subnets:
- subnet-1234567890abcd
- subnet-1234567890abcd
- subnet-1234567890abcd
2020/08/27 23:38:28 service-name/cluster depoymentInfo {
ApplicationName: "cluster",
AutoRollbackConfiguration: {
Enabled: true,
Events: ["DEPLOYMENT_FAILURE"]
},
BlueGreenDeploymentConfiguration: {
DeploymentReadyOption: {
ActionOnTimeout: "STOP_DEPLOYMENT",
WaitTimeInMinutes: 5
},
TerminateBlueInstancesOnDeploymentSuccess: {
Action: "TERMINATE",
TerminationWaitTimeInMinutes: 5
}
},
CompleteTime: 2020-08-27 14:16:47 +0000 UTC,
ComputePlatform: "ECS",
CreateTime: 2020-08-27 14:13:47 +0000 UTC,
Creator: "user",
DeploymentConfigName: "CodeDeployDefault.ECSAllAtOnce",
DeploymentGroupName: "cluster",
DeploymentId: "d-HT2P9ZZR5",
DeploymentOverview: {
Failed: 0,
InProgress: 0,
Pending: 0,
Ready: 0,
Skipped: 0,
Succeeded: 1
},
DeploymentStatusMessages: [],
DeploymentStyle: {
DeploymentOption: "WITH_TRAFFIC_CONTROL",
DeploymentType: "BLUE_GREEN"
},
FileExistsBehavior: "DISALLOW",
IgnoreApplicationStopFailures: false,
InstanceTerminationWaitTimeStarted: true,
LoadBalancerInfo: {
TargetGroupPairInfoList: [{
ProdTrafficRoute: {
ListenerArns: ["arn:aws:elasticloadbalancing:ap-northeast-1:123456789012:listener/app/cluster-external/e4a3a44e88770f9c/8c707a19857045b2"]
},
TargetGroups: [{
Name: "cluster-external-8080-blue"
},{
Name: "cluster-external-8080-green"
}]
}]
},
PreviousRevision: {
AppSpecContent: {
Sha256: "4b3a0a34efd879534fb5074d1c9a89dcd2bed4468bd615f11d3f64a4615e5748"
},
RevisionType: "AppSpecContent"
},
Revision: {
AppSpecContent: {
Sha256: "3469252eba2fe38ab9eb15bd4f97d3f38928d584f47f1d7757669aeff953cf1e"
},
RevisionType: "AppSpecContent"
},
Status: "Succeeded",
UpdateOutdatedInstancesOnly: false
}
2020/08/27 23:38:28 service-name/cluster creating a deployment to CodeDeploy {
ApplicationName: "cluster",
DeploymentConfigName: "CodeDeployDefault.ECSAllAtOnce",
DeploymentGroupName: "cluster",
Revision: {
AppSpecContent: {
Content: "version: \"0.0\"\nResources:\n- TargetService:\n Type: AWS::ECS::Service\n Properties:\n TaskDefinition: arn:aws:ecs:ap-northeast-1:123456789012:task-definition/cluster:819\n LoadBalancerInfo:\n ContainerName: application\n ContainerPort: 8080\n PlatformVersion: 1.3.0\n NetworkConfiguration:\n awsvpcconfiguration:\n assignpublicip: DISABLED\n securitygroups:\n - sg-1234567890abcd\n subnets:\n - subnet-1234567890abcd\n - subnet-1234567890abcd\n - subnet-1234567890abcd\n"
},
RevisionType: "AppSpecContent"
}
}
日本語で失礼致します:bowing_man:
先日Twitterにてリプライした内容について、おこがましいですがIssueを作成致します。
s3にあるtfstateを参照するとき、 s3://<bucket-name>/<key-of-file>
形式で指定できると直感的でわかりやすいと思います。
また同時に、 Workspaces に関して現在 ecspresso ではサポートがないものと存じますが、これステージ分離等に活用している場合も(configファイルを複数設けるなどハードコーディングで)対応できるように感じます。(URLを s3://<bucket-name>/env:/${terraform.workspace}/<key-of-project>
形式にする)
これは完全に余談ですが、ユースケースによってはtfstate自体が分離されている可能性もあって、tfstateを複数指定できたら更に良いなとも思います。ただこれは単に配列で取得してJSONをマージしても事故が起こると思われるので、名前をつけてimportする必要がありますが、そこで付与した名前をserviceなどの定義ファイル内で識別子として認識するところも実装する必要がありそうで大変そうなので...自分がまともにGoのコードを開発できるようになってからPR飛ばします...:crying_cat_face:
cluster: arn:aws:ecs:ap-northeast-1:1234567890:cluster/my-cluster
ecspresso deploy
No errors happen.
Got failed to find CodeDeploy Application/DeploymentGroup
error:
2021/07/23 19:17:02 deploy FAILED. failed to find CodeDeploy Application/DeploymentGroup for ECS service <my-service-name> on cluster arn:aws:ecs:ap-northeast-1:1234567890:cluster/my-cluster
The cluster
value in config.yml compared with possible deployment group's ECS cluster value, but we can specify cluster's full ARN as cluster
value in config.yml and ecspresso works well except running ecspresso deploy
.
So if we specified the full ARN, it is never matched with the corresponding deployment group's cluster specification.
I have some ideas about this problem:
cluster
's value requirementscluster
if we extract the short name from them and pass it to ListDeploymentGroups
's argumentHello.
I want to add tag to taskdefinition.
But ecspresso doesn't support to add tags.
Because there are no tags in the argument.
Lines 108 to 122 in 53d3c28
The AWS SDK RegisterTaskDefinitionInput supports adding tags.
https://docs.aws.amazon.com/sdk-for-go/api/service/ecs/#RegisterTaskDefinitionInput
Tags []*Tag `locationName:"tags" type:"list"`
In my project, I use tags to use billing and AWS Resource Groups services.
So I want this feature.
Thank you.
Can we use values from multiple tfstate files?
I tried the following, but seems like only the last one is loaded.
# ecspresso.yml
...
plugins:
- name: tfstate
config:
url: s3://tfstate1
- name: tfstate
config:
url: s3://tfstate2
Thanks for the great tool.
Currently, ecspresso emits warning messages into stdout such as:
https://github.com/kayac/ecspresso/blob/v1/verify.go#L226-L230
This causes breaking structured outputs.
For example, running ecspresso tasks --output=json
and gets the below outputs from stdout:
# ecspresso tasks --output=json
WARNING: json: unknown field "$comment" in etc/ecspresso/production/task-definition.json
{
"attachments": [
{
"details": [
# snip
But those are invalid as either JSON or JSON lines.
ecspresso using fmt.Print*
functions to emit warning messages but those functions use os.Stdout
to output destination.
When emitting warning or error messages, the stderr may be preferred.
https://twitter.com/sogaoh/status/1390165192357466112 でお伝えした問題です
ecspresso --config config.yaml verify
2021/05/06 04:34:22 failed to read tfstate: s3://XXXXXXXXXX/YYYYY/ZZZZZ/terraform.tfstate: failed to read tfstate from s3://XXXXXXXXXX/YYYYY/ZZZZZ/terraform.tfstate: MissingRegion: could not find region configuration
状況として、AWS_DEFAULT_REGION="ap-southeast-1" を direnv で設定してますが、AWS_REGION は設定していませんでした。
他に確認事項あれば随時ご連絡ください。 @fujiwara
When taskDefinition defines secrets that stored in secrets manager,ecspresso verify
always shows error.
example definition
"secrets": [
{
"name": "MY_SECRET",
"valueFrom": "arn:aws:secretsmanager:ap-northeast-1:1234567890:secret:foo/bar/my-secret-abcdef"
}
]
error message
[NG] verify Secret[MY_SECRET] failed: ValidationException: Invalid parameter name. Please use correct syntax for referencing a version/label <name>:<version/label>
The cause is that all secrets are retrieved for the parameter store in the following:
Lines 327 to 330 in 3af4584
If valueFrom
starts with arn:aws:secretsmanager
, the secret should be retrieved from secrets manager.
task definitions require a valid log driver.
failed to register task definition: ClientException: awsvpc is not a valid log driver. Must be one of [awslogs,splunk,awsfirelens]
When running a command in a container that takes less than 1 second to complete, escpresso run
frequently determines that the task has failed, even though the command completed successfully.
$ ecspresso run --config ecspresso.yml
2022/04/12 15:22:02 ecspresso-test/api Running task
2022/04/12 15:22:02 ecspresso-test/api Registering a new task definition...
2022/04/12 15:22:04 ecspresso-test/api Task definition is registered ecspresso-test:22
2022/04/12 15:22:04 ecspresso-test/api Task definition ARN: arn:aws:ecs:us-east-1:509803364528:task-definition/ecspresso-test:22
2022/04/12 15:22:04 ecspresso-test/api Watch container: hello
2022/04/12 15:22:04 ecspresso-test/api Running task with arn:aws:ecs:us-east-1:509803364528:task-definition/ecspresso-test:22
2022/04/12 15:22:04 ecspresso-test/api Task ARN: arn:aws:ecs:us-east-1:509803364528:task/api/3c84bfc97964403cae75ec093ceb200e
2022/04/12 15:22:04 ecspresso-test/api Waiting for run task...(it may take a while)
2022/04/12 15:22:04 ecspresso-test/api Watching container: hello
2022/04/12 15:22:04 ecspresso-test/api logGroup: ecspresso-test
2022/04/12 15:22:04 ecspresso-test/api logStream: test/hello/3c84bfc97964403cae75ec093ceb200e
2022/04/12 15:22:07 ecspresso-test/api Waiting for task ID 3c84bfc97964403cae75ec093ceb200e until running
2022/04/12 15:22:07 run FAILED. failed to run task: failed to run task: ResourceNotReady: failed waiting for successful resource state
ecspresso run waits for the task to reach the RUNNNIG state with WaitUntilTasksRunningWithContext after starting the task.
Line 177 in e3dd249
Here is the task definition I used.
{
family: 'ecspresso-test',
containerDefinitions: [
{
name: 'hello',
image: 'alpine:latest',
essential: true,
cpu: 10,
memory: 64,
command: [
'sh',
'-c',
'sleep 1 && echo hello',
],
logConfiguration: {
logDriver: 'awslogs',
options: {
'awslogs-group': 'ecspresso-test',
'awslogs-region': 'us-east-1',
'awslogs-stream-prefix': 'test',
},
},
},
],
}
CodeDeploy利用時のwaitコマンドでconfigで30分以上の値を設定してもSDKのデフォルト値(120 * 15)の30分でタイムアウトします。
https://github.com/aws/aws-sdk-go/blob/v1.42.23/service/codedeploy/waiters.go#L31-L32
WaitUntilDeploymentSuccessfulWithContext
でも WaitUntilServicesStableWithContext
と同様にwaiterでdelay、attemptsを渡してあげる必要がある。
Lines 646 to 649 in 354ba4a
Lines 484 to 488 in 354ba4a
https://twitter.com/dmnlk/status/1446410781810069507
ecspresso rollback --deregister-task-definition --no-wait
does not deregister a task definition.
We have two ways to fix the confused behaivor.
Is there a plan to add plugin support for ssm parameter store? If not, are you willing to accept the PR if there's one?
I'm imagining something like below:
# ecspresso.yml
...
plugins:
- name: ssm
# anywhere in ecs-service-def.json or task-def.json
{
...
"some_key": "{{ ssm `/path/to/parameter` }}"
}
Thanks.
service_definition
if run tasks only....and so on
I write {{.ID}} in docker logging tags, but this conflicts with the function to replace the syntax of JSON file with environment variable. What should I do now?
reference:
docker logging tags,
https://docs.docker.com/config/containers/logging/log_tags/
We tried to deploy an ECS application using B/G deployment with CodeDeploy, by ecspresso v1.7.9, but it was failed with events log below.
Events:
2022/03/10 06:20:44 ... Registering a new task definition...
2022/03/10 06:20:45 ... Task definition is registered ...:33
2022/03/10 06:20:45 ... Updating service attributes...
2022/03/10 06:20:45 deploy FAILED. failed to update service attributes: InvalidParameterException: Unable to update load balancers on services with a CODE_DEPLOY deployment controller. Use AWS CodeDeploy to trigger a new deployment.
By ecspresso v1.7.8, we could deploy the application successfully.
FYI: ecs-service-def.json
like below.
And we didn't update the file, so it should not updates ECS service I think.
{
"deploymentConfiguration": {
"maximumPercent": 200,
"minimumHealthyPercent": 100
},
"deploymentController": {
"type": "CODE_DEPLOY"
},
"desiredCount": 1,
"enableECSManagedTags": true,
"enableExecuteCommand": false,
"healthCheckGracePeriodSeconds": 0,
"launchType": "FARGATE",
"loadBalancers": [
{
"containerName": "app",
"containerPort": 80,
"targetGroupArn": "..."
}
],
"networkConfiguration": {
"awsvpcConfiguration": {
"assignPublicIp": "DISABLED",
"securityGroups": ["..."],
"subnets": ["...", "..."]
}
},
"placementConstraints": [],
"placementStrategy": [],
"platformFamily": "Linux",
"platformVersion": "1.4.0",
"schedulingStrategy": "REPLICA",
"serviceRegistries": [],
"tags": []
}
watch option to status command
I tried ecspresso wait
command but doesn't work.
Code Deploy Deployment was Succeeded
.
$ ecspresso deploy --config config.yaml --no-wait
2020/04/01 21:13:12 sample/sample-cluster Starting deploy
Service: sample
Cluster: sample-cluster
TaskDefinition: sample-app:98
TaskSets:
PRIMARY sample-app:98 desired:2 pending:0 running:2
Events:
2020/04/01 21:13:12 sample/sample-cluster Creating a new task definition by ecs-task-def.json
2020/04/01 21:13:12 sample/sample-cluster Registering a new task definition...
2020/04/01 21:13:12 sample/sample-cluster Task definition is registered sample-app:101
2020/04/01 21:13:13 sample/sample-cluster Deployment d-XXXXXXXXX is created on CodeDeploy:
2020/04/01 21:13:13 sample/sample-cluster https://ap-northeast-1.console.aws.amazon.com/codesuite/codedeploy/deployments/d-XXXXXXXXX?region=ap-northeast-1
$
$ ecspresso wait --config config.yaml
2020/04/01 21:13:23 sample/sample-cluster Waiting for the service stable
2020/04/01 21:13:23 sample/sample-cluster Waiting for service stable...(it will take a few minutes)
2020/04/01 21:16:18 (service sample) has reached a steady state.27906806) updated state to
2020/04/01 21:16:08 (service sample, taskSet ecs-svc/9666680786127906806) updated state to
STEADY_STATE.
2020/04/01 21:15:58 (service sample, taskSet ecs-svc/9666680786127906806) has stopped 2 ru
nning tasks: (task 496fcdf75dd84b86a2151d5879d89fe0) (task 45c3d01e80a24f51baeda1f51d25049
d).
2020/04/01 21:15:58 (service sample, taskSet ecs-svc/1043283654285067039) updated state to
STEADY_STATE.
2020/04/01 21:15:48 (service sample, taskSet ecs-svc/9666680786127906806) has begun draini
ng connections on 2 tasks.
2020/04/01 21:15:48 (service sample) deregistered 2 targets in (target-group arn:aws:elast
icloadbalancing:ap-northeast-1:012345678901:targetgroup/sample-alb-01-tg-02/4e53119caa016b
82)
2020/04/01 21:15:48 (service sample, taskSet ecs-svc/1043283654285067039) updated state to
STABILIZING.
2020/04/01 21:15:48 (service sample) updated computedDesiredCount for taskSet ecs-svc/9666
680786127906806 to 0.
2020/04/01 21:14:25 (service sample) has reached a steady state.
2020/04/01 21:23:23 wait FAILED. the service still unstable: RequestCanceled: waiter context canceled
It worked fine when the deploymentController is ECS.
$ ecspresso deploy --config config.yaml --no-wait
2020/04/01 21:06:00 sample-app/sample-cluster Starting deploy
Service: sample-app
Cluster: sample-cluster
TaskDefinition: sample-app:99
Deployments:
PRIMARY sample-app:99 desired:1 pending:0 running:1
Events:
2020/04/01 21:06:02 sample-app/sample-cluster Creating a new task definition by ecs-task-def.json
2020/04/01 21:06:02 sample-app/sample-cluster Registering a new task definition...
2020/04/01 21:06:02 sample-app/sample-cluster Task definition is registered sample-app:100
2020/04/01 21:06:02 sample-app/sample-cluster Updating service tasks...
2020/04/01 21:06:05 sample-app/sample-cluster Service is deployed.
$
$ ecspresso wait --config config.yaml
2020/04/01 21:06:15 sample-app/sample-cluster Waiting for the service stable
2020/04/01 21:06:15 sample-app/sample-cluster Waiting for service stable...(it will take a few minutes)
2020/04/01 21:08:16 sample-app/sample-cluster PRIMARY sample-app:100 desired:1 pending:0 running:1
2020/04/01 21:07:54 (service sample-app) has stopped 1 running tasks: (task c073aeb0f71148
969b477faf66faed83).
2020/04/01 21:08:20 sample-app/sample-cluster Service is stable now. Completed!
Run ecspresso init
with required arguments except --config
.
No errors happen or enough information to fix it such as --config required but not passed
.
Got failed to write file: open : no such file or directory
error.
Full error messages are below:
[2021-07-21 13:55:58] ✘╹◡╹✘ < ecspresso init --cluster $cluster_arn --service $service_arn
2021/07/21 13:56:06 <snip> save service definition to ecs-service-def.json
2021/07/21 13:56:06 <snip> save task definition to ecs-task-def.json
2021/07/21 13:56:06 <snip> save config to
2021/07/21 13:56:06 init FAILED. failed to write file: open : no such file or directory
I think --config
argument is actually a required parameter so it must be marked as a required parameter using .Required()
, or should we set a default value such as config.yml
?
I'm sorry, but I'm making this issue in Japanese for ease and speed. Please translate comments with services such as DeepL If you are not a Japanese speaker.
先日 Twitter にて AutoScaling 管理下の Desired Count を変更した際に起こる競合について質問をした者です。その際はありがとうございました。
(ちゃんとタスクがスケールされたことの確認のポーリングを自前で実装するのは面倒だったこともあり、) ecspresso scale
でのスケールを検討しているのですが、どうせなら ecspresso deploy
コマンドのように --suspend-auto-scaling
および --no-suspend-auto-scaling
もセットになっていたら嬉しいなぁ...と思いました。
自分でも手伝えるような内容に思えるので、今夜PRの作成トライしてみますが、シンプルさなどの方針から取り込みたくないということであれば Issue 自体 Reject していただいて大丈夫です!
Currently, ecspresso exec
fails when a session-manager-plugin is not found.
exec FAILED. exec: "session-manager-plugin": executable file not found in $PATH
If the environment does not have a session-manager-plugin, ecspresso should show a helpful error message.
I have multiple deploy destinations.
I have prepared one ecspresso configuration file group.
When run ecspresso xxx
, I use the environment variables for each environment.
like this:
# for env HOGE
source HOGE.env
ecspresso verify --config=config.yaml && ecspresso diff --config=config.yaml
ecspreoss deploy --config=config.yaml
# for env FUGA
source FUGA.env
ecspresso verify --config=config.yaml && ecspresso diff --config=config.yaml
ecspreoss deploy --config=config.yaml
However, the above command overwrites the login shell environment variables.
If ecspresso supports the --envfile option, it will prevent overwriting them.
# for env HOGE
ecspresso verify --envfile=HOGE.env --config=config.yaml && ecspresso diff --envfile=HOGE.env --config=config.yaml
ecspreoss deploy --envfile=HOGE.env --config=config.yaml
There are workarounds such as the following, but this is a little complicated.
# for env HOGE
(source HOGE.env && ecspresso verify --config=config.yaml && ecspresso diff --config=config.yaml)
(source HOGE.env && ecspreoss deploy --config=config.yaml)
Furthermore, if supports multiple --envfile
options, it will be more useful.
ecspreoss deploy --envfile=HOGE_MAIN.env --envfile=HOGE_SIDECAR.env --config=config.yaml
ecspresso appspec
creates an AppSpec file based on current ECS service attributes (not based on service definition file).
Therefore we cannot update platformVersion and networkConfiguration for services that have deploymentController "CODE_DEPLOY".
Need to create an AppSpec file based on service definition file when ecspresso deploy --update-service
.
ecspresso deploy
, which was successful in v1.7.3, does not seem to succeed in v1.7.4 with the following error:
2021/12/25 18:35:43 my-fargate-service/my-fargate Starting deploy
Service: my-fargate-service
Cluster: my-fargate
TaskDefinition: my-task:77
Deployments:
PRIMARY my-task:77 desired:3 pending:0 running:3
Events:
2021/12/25 18:35:44 my-fargate-service/my-fargate Registering a new task definition...
2021/12/25 18:35:44 my-fargate-service/my-fargate Task definition is registered my-task:78
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1b71cc4]
goroutine 1 [running]:
github.com/kayac/ecspresso.(*App).Deploy(0xc000580c00, 0xc000037ab5, 0xc000037ab8, 0xc000037ac0, 0xc000037ac1, 0xc000037ac2, 0x0, 0xc00020ae80, 0xc000037ac4, 0xc000037ac5, ...)
/home/runner/work/ecspresso/ecspresso/deploy.go:81 +0xa24
main._main(0xc000100058)
/home/runner/work/ecspresso/ecspresso/cmd/ecspresso/main.go:250 +0x8e6e
main.main()
/home/runner/work/ecspresso/ecspresso/cmd/ecspresso/main.go:18 +0x25
Probably due to #365.
Thank you for making a great tool!
I would like to propose a platform verify feature.
Currently, ECS provides several runtimes.
e.g.
Fargate/EC2 with AMD64/Linux
Fargate/EC2 with ARM64/Linux
Fargate/EC2 with AMD64/Windows
The feature is to verify whether images in task definition support target platform or not.
For example, there is a following task definition.
{
"containerDefinitions": [
{
"name": "custom-image"
}
],
"requiresCompatibilities": [
"FARGATE"
],
"networkMode": "awsvpc",
"runtimePlatform": {
"operatingSystemFamily": "LINUX",
"cpuArchitecture": "ARM64"
}
}
When custom-image
does not support arm64/linux platform, ECS task will fail to start due to incompatible platform.
We need to start tasks and check container logs to be aware of failures.
I will try to implement it and create PR.
I tried to change desiredCount
field in service.json
and run ecspresso deploy --update-service --skip-task-definition
, but a count of tasks was not changed.
{
- "desiredCount": 1,
+ "desiredCount": 0,
I confirmed that --tasks
option works fine to scale-in/out, so does not ecspresso support to change desiredCount
field in JSON for scale-in/out?
Version: 0.16.0
ecspresso v1.7.8
When executing ecspresso exec
to a task definition that contains must_env
environment variables, panic occurs due to undefined environment variables.
This did not happen in v1.5.0.
Even if must_env is used in a task definition, it should be possible to ecspresso exec
without setting environment variables at runtime.
$ DEPLOY_TARGET_ENV=prod ecspresso exec --config=.ecspresso/application-config.yml --command="/bin/bash"
panic: template attach failed: template: conf:12:77: executing "conf" at <must_env `DEPLOY_IMAGE_TAG`>: error calling must_env: environment variable DEPLOY_IMAGE_TAG is not defined
goroutine 1 [running]:
github.com/kayac/go-config.readConfigBytes(0xc000372600, 0x5b9, 0x5ba, 0xc00083ed60, 0x5ba, 0x0, 0x0, 0x0, 0x0)
/home/runner/go/pkg/mod/github.com/kayac/[email protected]/config.go:149 +0x18e
github.com/kayac/go-config.(*Loader).ReadWithEnv(0xc0003d8180, 0xc0003680f0, 0x24, 0x3a7c600, 0x0, 0x23d, 0xc00083ee08, 0x10187d3)
/home/runner/go/pkg/mod/github.com/kayac/[email protected]/config.go:322 +0xc6
github.com/kayac/ecspresso.(*App).readDefinitionFile(0xc0000bac80, 0xc0003680f0, 0x24, 0x10c1245, 0x10c1d00, 0xc0003685a0, 0xc000293628, 0x0)
/home/runner/work/ecspresso/ecspresso/util.go:68 +0x6b9
github.com/kayac/ecspresso.(*App).LoadTaskDefinition(0xc0000bac80, 0xc0003680f0, 0x24, 0x25, 0x2374b08, 0xc0002935f0)
/home/runner/work/ecspresso/ecspresso/ecspresso.go:498 +0x5a
github.com/kayac/ecspresso.(*App).listTasks(0xc0000bac80, 0x236f1a0, 0xc00071d680, 0xc000719d20, 0xc00083f2f8, 0x1, 0x1, 0xc0007143c0, 0x1, 0x1, ...)
/home/runner/work/ecspresso/ecspresso/tasks.go:64 +0x48b
github.com/kayac/ecspresso.(*App).Exec(0xc0000bac80, 0xc000719d20, 0xc000719d50, 0xc000719d60, 0xc00045f7a0, 0xc00045f790, 0xc00045f798, 0x0, 0x0)
/home/runner/work/ecspresso/ecspresso/exec.go:53 +0x145
main._main(0xc000094058)
/home/runner/work/ecspresso/ecspresso/cmd/ecspresso/main.go:293 +0x8cde
main.main()
/home/runner/work/ecspresso/ecspresso/cmd/ecspresso/main.go:18 +0x25
At this time, in our task definition, we set the family, arn, and image URLs as follows. The reason we specify must_env
is that this is mainly for CD.
"family": “my_{{ must_env `DEPLOY_TARGET_ENV` }}_application”,
"taskRoleArn": "arn:aws:iam::hogefuga/components/my_{{ must_env `DEPLOY_TARGET_ENV` }}_application_task_role",
"containerDefinitions": [
{
"name": “hoge”,
"image": “hogehoge.dkr.ecr.ap-northeast-1.amazonaws.com/hoge:{{ must_env `DEPLOY_IMAGE_TAG` }}",
…
Also, we have confirmed that if must_env is replaced with env, it can be executed without any problem.
This also happens when the environment variable used by the family is left undefined, so it does not seem to depend on which element uses must_env.
Define a must_env
environment variable (in our case, DEPLOY_IMAGE_TAG
) at ecspresso exec
runtime.
Probably because the change in v1.6 to allow command execution without ECS service refers to the task definition to find the same family.
It would be very nice to have the ability to a specific version and not just for the last one.
the reason is that if you know the last version also has issued and you wish to rollback to version X-5 you will have to run rollback 5 times.
When I want to run any command without changing the task or service definition, I have to use --overrides
flag at run, and have to prepare json string for it.
So, I would like to add the flag --command
and be able to execute any command as follows.
ecspresso run --config config.yaml --command "echo hoge"
When an Amazon ECR image URL has not a tag (for example 123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/myimage
), ecspresso fails to verify the URL.
[NG] verify Image[123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/myimage] failed: aborted
Dependabot encountered the following error when parsing your .dependabot/config.yml
:
The property '#/update_configs/0/package_manager' value "go" did not match one of the following values: javascript, ruby:bundler, php:composer, java:maven, elixir:hex, rust:cargo, java:gradle, dotnet:nuget, go:dep, go:modules, docker, elm, submodules, github_actions, python, terraform
Please update the config file to conform with Dependabot's specification using our docs and online validator.
Hi, I use ecspresso with Codedeploy.
My environments have many deployment group, applications.
When deploying multiple services at the same time, throttling exception happend.
I'd like to fix this issue.
$ ecspresso version
ecspresso v1.7.7
I think it's root cause .
Lines 242 to 305 in 7ec5449
findDeploymentInfo()
take applicationName/deploymentconfigName/deploymentGroupName dynamically.
Therefore, I think that it call a lot of APIs in my environment and exceeded the API rate limit.
Below when error happened.
# log
2022/04/13 09:10:14 test-ecsporesso/test-ecspresso-cluster Starting deploy
Service: test-ecsporesso
Cluster: test-ecspresso-cluster
TaskDefinition: test-ecsporesso-api:85
TaskSets:
PRIMARY test-ecsporesso-api:85 desired:1 pending:0 running:0
ACTIVE test-ecsporesso-api:86 desired:1 pending:0 running:0
AutoScaling:
Capacity min:1 max:1
Suspended in:false out:false scheduled:false
Policy name:test-ecsporesso-policy type:TargetTrackingScaling
Events:
2022/04/13 09:10:14 test-ecsporesso/test-ecspresso-cluster Registering a new task definition...
2022/04/13 09:10:14 test-ecsporesso/test-ecspresso-cluster Task definition is registered test-ecsporesso-api:88
2022/04/13 09:10:14 test-ecsporesso/test-ecspresso-cluster Updating service attributes...
2022/04/13 09:10:17 test-ecsporesso/test-ecspresso-cluster desired count: unchanged
2022/04/13 09:10:34 deploy FAILED. ThrottlingException: Rate exceeded
deployment_definition
.cluster: default
service: test
service_definition: ecs-service-def.json
task_definition: ecs-task-def.json
deployment_definition: ecs-deployment-def.json
{
"applicationName": "test-app",
"deploymentConfigName": "CodeDeployDefault.ECSAllAtOnce",
"deploymentGroupName": "test-deployment-group"
}
I think applicationName/deploymentconfigName/deploymentGroupName take deployment_definition statically.
What do you think?
config fileでtimeout: 15mを設定したが、10分後にerrorになって失敗してしまうのですが、なぜでしょうか。error文は下記になります。
2019/08/28 06:32:18 deploy FAILED. deploy failed: ResourceNotReady: exceeded wait attempts
Build step 'Execute shell' marked build as failure
Hi and thanks for your work 👍
I've read your Advent calendar on zenn and tried ecspresso run --latest-task-definition
. I thought it would use the latest task definition but actually it uses the definition used by the service. Is that an intended behavior? If so, maybe reflect it in docs?
My use case is as follows:
It would be nice if you could specify task revision (e.g. 2) for the run command.
run --skip-task-definition and --latest-task-definition must work without an ECS service available.
#327
ecspresso
cli resolves definition files ( service_definition
and task_definition
) path from current working directory.
So we need to run ecspresso
cli on exact directory .
This causes some confusion when config files put nested directory structure.
ecspresso
cli resolves definition files path from config.yml file relatively.
In this directory structure, I need to configure and run escpresso
cli in 2 ways.
.
└── deploy
├── config.yaml
├── ecs-service-def.json
└── ecs-task-def.json
deploy
wayConfigure config.yml
's definition files following example, and run ecspresso
cli after change directory to deploy
service_definition: ecs-service-def.json
task_definition: ecs-task-def.json
Configure config.yml
's definition files following example, and run ecspresso
cli in root directory.
service_definition: ./deploy/ecs-service-def.json
task_definition: ./deploy/ecs-task-def.json
Please consider to upgrade aws-sdk-go version.
AWS Single Sign-On (SSO) service was added to v1.37.0
.
aws/aws-sdk-go#3755
Currently ecspresso cannot get credentials from aws sso login
command.
command example.
$ ecspresso version
ecspresso v1.3.2
$ aws sso login --profile my-sso-profile
Attempting to automatically open the SSO authorization page in your default browser.
If the browser does not open or you wish to use a different device to authorize this request, open the following URL:
https://device.sso.us-east-1.amazonaws.com/
Then enter the code:
ABCD-EFGH
Successully logged into Start URL: https://example.com/start
$ export AWS_PROFILE=my-sso-profile
$ ecspresso init --config config.yml --region ap-northeast-1 --cluster hoge-cluster --service hoge-service
2021/02/02 16:15:34 init FAILED. failed to describe service: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
In my use-case, deployment workflow updates multiple services and runs task (unrelated to those services).
So, I want to run ecspress init
command multiple times in one work directory.
This produces overwrite prompt like below.
021/02/02 12:01:38 xxxx/xxxx save service definition to ecs-service-def.json
Overwrite existing file ecs-service-def.json? (y/n) [n]: y
2021/02/02 12:01:44 xxxx/xxxx save task definition to ecs-task-def.json
Overwrite existing file ecs-task-def.json? (y/n) [n]: y
Or, should I create config.yml files for each ECS Services I want to update?
前日リリースされたAmazon ECR Public Galleryのイメージを指定した task definition を、ECRに未ログインで verify
したところ 401 Unauthorized
で NG となりました。
期待する動作はECRへの未ログイン時にECR Publicイメージへは匿名アクセスで存在確認が行われる、または verifySkipErr
などでverify NGとはならない形になります。
NG時の出力:
TaskDefinition [NG] verify ContainerDefinition[mackerel-container-agent] failed: verify Image[public.ecr.aws/mackerel/mackerel-container-agent:plugins] failed: 401 Unauthorized
イメージをDocker Hubのものに変えただけの差分でのOK出力:
ContainerDefinition[mackerel-container-agent]
Image[mackerel/mackerel-container-agent:plugins]
--> [OK]
public.ecr.aws/mackerel/mackerel-container-agent:plugins
は手元のマシンで未認証で docker pull 可能なことを確認しました。
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.