brabster / crucible Goto Github PK

View Code? Open in Web Editor NEW

72.0 9.0 18.0 351 KB

AWS CloudFormation templates built with Clojure

License: Eclipse Public License 1.0

Clojure 99.90% Shell 0.10%

crucible cloudformation clojure clojure-specs

crucible's People

Contributors

Stargazers

Watchers

Forkers

l0st3d garethmdavies yetanalytics tclamb rarous avisi keerts dmarjenburgh adamfrey rastandy shooit roomkey csumpter justjohn2001 yin8686 cch1 testingiacwithnewaccount andreacrotti

crucible's Issues

Unsupported lambda property: BatchSize

BatchSize is not a supported property on a lambda function. Should this be ReservedConcurrentExecutions?

The :crucible.aws.lambda/batch-size spec has a max of 10000, but the max reserved-concurrent-executions is 1000.

Add support for conditions

Add support for conditions in the template.

Use function instead of string or xref

Is there any other way to do the equivalent of:

  {:file-store-bucket (s3/bucket {::s3/bucket-name (cf/join [{:ref "AWS::StackName"} "-rfj-files"])})}

The spec allows either a string or an xref.

Add link to https://cljdoc.org/d/crucible

Generate specs from CloudFormation Resource Specification

AWS vends a JSON document containing the complete and up-to-date list of supported resources and parameters. This is probably what's used to generate AWS's HTML documentation. It should be possible to generate specs from this file.

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-resource-specification.html

Spec for internet-gateway

The following spec is playing with me:

(defresource internet-gateway (ec2 "InternetGateway") (s/? (s/keys :opt [::res/tags])))
If I read it correctly, both of the following should be conformant:

   :ig-next (ec2/internet-gateway)
   :ig-next (ec2/internet-gateway {::res/tags [{::res/key "Name" ::res/value "next"}]})

But the second form yields the following:

1. Unhandled clojure.lang.ExceptionInfo
   Invalid resource properties
   #:clojure.spec.alpha{:problems
                        ({:path [],
                          :pred map?,
                          :val
                          [:crucible.resources/tags
                           [#:crucible.resources{:key "Name",
                                                 :value "next"}]],
                          :via [],
                          :in [0]}),
                        :spec
                        {:clojure.spec.alpha/op :clojure.spec.alpha/alt,
                         :ps
                         (#object[clojure.spec.alpha$map_spec_impl$reify__1931 0x5ab09b8a "clojure.spec.alpha$map_spec_impl$reify__1931@5ab09b8a"]
                          {:clojure.spec.alpha/op
                           :clojure.spec.alpha/accept,
                           :ret :clojure.spec.alpha/nil}),
                         :ks nil,
                         :forms
                         ((clojure.spec.alpha/keys
                           :opt
                           [:crucible.resources/tags])
                          :clojure.spec.alpha/nil),
                         :maybe
                         (clojure.spec.alpha/keys
                          :opt
                          [:crucible.resources/tags])},
                        :value
                        #:crucible.resources{:tags
                                             [#:crucible.resources{:key
                                                                   "Name",
                                                                   :value
                                                                   "next"}]}}

Must be my lacking experience with spec, but I need a little help understanding the error. Any help appreciated.

Container-definition environment spec is incorrect

The spec for environment is not a vector as it should be

Custom resource double-call should just take two arguments

((custom/resource "Foo") {:foo :bar})

would be better

(custom/resource "Foo" {:foo :bar})

Sub function doesn't render correctly

(sub "foo ${bar}")

renders as a structure like

{"Type": "Sub", "SubValue": "foo ${bar}"}

Cloverage coverage report zeroed

Resource policies clunky, no function support

Implement Lambda Environment

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-function-environment.html

needs AWS::Lambda::Permission wrapper

Needs some AWS::Lambda::Permission wrapper. See pseudo-code example below:

(def s3-invoke-function-permission
  (new-template
   "AWS::Lambda::Permission"
   {:Action "lambda:InvokeFunction"
    :FunctionName (xref some-lambda-fn)
    :Principal "s3.amazonaws.com"
    :SourceAccount login-account}))

No easy way to generate templates during a build

Difficult to assemble partial templates

spec-or-ref used for most leaf nodes in a resource

crucible.values/spec-or-ref allows a union of a literal valid by the spec arg or any value generated at template build time (like a join or a ref). Most templates so far use/should use this for most leaf nodes so as not to assume usage, so needs to be constantly remembered and typed when creating resources. Should be default behaviour?

Add support for EC2::NatGateway

A PR will follow shortly

Lambda Event Source Mapping spec incorrect

There is a typo in ::starting-postition, and ::starting-position is an optional attribute. Proposed patch:

diff --git a/src/crucible/aws/lambda.clj b/src/crucible/aws/lambda.clj
index 38bd8c9..9b12cfa 100644
--- a/src/crucible/aws/lambda.clj
+++ b/src/crucible/aws/lambda.clj
@@ -65,13 +65,13 @@

 (s/def ::event-source-arn (spec-or-ref string?))

-(s/def ::starting-postition (spec-or-ref #{"TRIM_HORIZON" "LATEST"}))
+(s/def ::starting-position (spec-or-ref #{"TRIM_HORIZON" "LATEST"}))

 (s/def ::event-source-mapping (s/keys :req [::event-source-arn
-                                            ::function-name
-                                            ::starting-position]
+                                            ::function-name]
                                       :opt [::batch-size
-                                            ::enabled]))
+                                            ::enabled
+                                            ::starting-position]))

 (defresource event-source-mapping "AWS::Lambda::EventSourceMapping" ::event-source-mapping)

Lambda environment variables get CamelCased

It looks like the configs get camel cased by default. This looks like a source of bugs.

Validation errors don't pretty-print

spec-or-ref in policie spec causes circular dependency error

In the policy spec I have a need for a parameter to be able to take a spec-or-ref. This is because I want to use an xref parameter. However I can't simply use spec-or-ref in the crucible.policies namespace. This is because the spec-or-ref function is defined in the crucible.resources namespace. Referencing it in the crucible.policies namespace or any required namespaces in the crucible.policies causes a circular dependency.

We could move the spec-or-ref function out of the crucible.resources namespace but that would be a rather large change since the spec-or-ref function is used often. Also, it would mean a breaking change for anyone who has defined custom resources. I can't come up with a simple solution so I'm looking for any ideas. However, I think the right solution may need to move the spec-or-ref out of the crucible.resources.

ECS service container port spec must be integer

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-service-loadbalancers.html#cfn-ecs-service-loadbalancers-containerport

Parameter options aren't being used

Too many requires

Writing a template or a resource need several requires for basic functionality.

->key doesn't work

Keys already PascalCased. Fool.

Handling of non-string values during encoding

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-cors-corsrule.html specifies max-age as an integer, but the CORS example at http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket.html gives it as a string. Would be nice to specify specs with correct literal validation, eg. an integer. Need to check whether non-string values are accepted in the affected templates.

Camel casing for aws-vpc-configuration is wrong

Add AWS::S3::BucketPolicy

lambda vpc config security group IDs should be sequences

resource= doesn't check that output is correct

resource= doesn't seem to be checking that the output is valid anymore. Check it!

Pseudo parameters doesn't work

Structure of pseudo params is old and doesn't match templating expectation

eg [:pseudo ::account-id]

should be

{:crucible.values/type :crucible.values/pseudo :crucible.values/param :account-id}

Fn::GetAZs implementation

Not sure how to implement Fn::GetAZs function so that i can have something like

AvailabilityZone: !Select [ 0, !GetAZs '' ]

AvailabilityZone:
  Fn::Select:
  - '0'
  - {'Fn::GetAZs': ''}

Tried this one

(s/def ::get-azs (s/keys :req [::values/type ::region]))
(defmethod values/value-type ::get-azs [_] ::get-azs)
(defmethod values/encode-value ::get-azs [{:keys [::region]}]
  {"Fn::GetAZs" region})
(defn get-azs [region]
  {::values/type ::get-azs
   ::region region})

:subnet-a (ec2/subnet {
              ::ec2/availability-zone (select 0 (get-azs ""))
              ::ec2/vpc-id (xref :vpc)
              ::ec2/cidr-block (cidr-block 28)
              ::ec2/map-public-ip-on-launch "true"})

but getting this error

Invalid resource properties-- Spec failed --------------------

  {:crucible.aws.ec2/availability-zone {:crucible.values/type :crucible.values/select,
                                        :crucible.values/index 0,
                                        :crucible.values/fn-values
                                        {:crucible.values/type :infra.ecs/get-azs, :infra.ecs/region ""}},
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   :crucible.aws.ec2/vpc-id ...,
   :crucible.aws.ec2/cidr-block ...,
   :crucible.aws.ec2/map-public-ip-on-launch ...}

should satisfy

  string?

or value

  {:crucible.aws.ec2/availability-zone
   {:crucible.values/type ...,
    :crucible.values/index ...,
    :crucible.values/fn-values {:crucible.values/type :infra.ecs/get-azs, :infra.ecs/region ""}},
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   :crucible.aws.ec2/vpc-id ...,
   :crucible.aws.ec2/cidr-block ...,
   :crucible.aws.ec2/map-public-ip-on-launch ...}

should satisfy

  vector?

However it doesn't fail if i change subnet code to

:subnet-a (ec2/subnet {
              ::ec2/availability-zone (select 0 [(get-azs "")])
              ::ec2/vpc-id (xref :vpc)
              ::ec2/cidr-block (cidr-block 28)
              ::ec2/map-public-ip-on-launch "true"})

but in that case it produces code like

AvailabilityZone:
  Fn::Select:
  - '0'
  - - {'Fn::GetAZs': ''}

which is not correct.

needs AWS::EC2::SecurityGroup wrapper

Needs some AWS::EC2::SecurityGroup wrapper. See example below:

(def https-vpc-security-group
  (new-template
   "AWS::EC2::SecurityGroup"
   {:GroupDescription "Allow access to https servers"
    :VpcId (xref :vpc)
    :SecurityGroupEgress [{:IpProtocol "tcp" :FromPort "443" :ToPort "443" :CidrIp "0.0.0.0/0"}]}))

Set up minimal IAM resources

For firehose assume-role, lambda assume-role

Value spec should not allow string?

Most leaf nodes in a template can be specified by a referenced value instead of a literal one. Value spec currently allows string? - should it?

Namespaced keyword maps more trouble than they're worth?

Having namespaced keyword maps for everything seems like a lot of effort to go to in every template for no obvious value - solving a problem that is unlikely to happen, and if it did it would be easy enough to solve? Maybe switch to non-namespaced keywords by default?