Code Monkey home page Code Monkey logo

aws-cfn-template-flip's Introduction

NOTICE: For CLI usage, AWS CloudFormation Template Flip is now deprecated.

You should use rain instead. rain fmt can convert CloudFormation templates between JSON and YAML.

See the rain fmt documentation for details.

This deprecation notice does not affect the API in this package which may continue to be used.


Build Status PyPI version Codecov Test Coverage

AWS CloudFormation Template Flip

About

AWS CloudFormation Template Flip is a tool that converts AWS CloudFormation templates between JSON and YAML formats, making use of the YAML format's short function syntax where possible.

The term "Flip" is inspired by the well-known Unix command-line tool flip which converts text files between Unix, Mac, and MS-DOS formats.

Installation

AWS CloudFormation Template Flip can be installed using pip:

pip install cfn-flip

Usage

AWS CloudFormation Template Flip is both a command line tool and a python library.

Note that the command line tool is spelled cfn-flip with a hyphen, while the python package is cfn_flip with an underscore.

Command line tool

Usage: cfn-flip [OPTIONS] [INPUT] [OUTPUT]

  AWS CloudFormation Template Flip is a tool that converts AWS
  CloudFormation templates between JSON and YAML formats, making use of the
  YAML format's short function syntax where possible."

Options:
  -i, --input [json|yaml]   Specify the input format. Overrides -j and -y
                            flags.
  -o, --output [json|yaml]  Specify the output format. Overrides -j, -y, and
                            -n flags.
  -j, --json                Convert to JSON. Assume the input is YAML.
  -y, --yaml                Convert to YAML. Assume the input is JSON.
  -c, --clean               Performs some opinionated cleanup on your
                            template.
  -l, --long                Use long-form syntax for functions when converting
                            to YAML.
  -n, --no-flip             Perform other operations but do not flip the
                            output format.
  --version                 Show the version and exit.
  --help                    Show this message and exit.

cfn-flip will detect the format of the input template and convert JSON to YAML and YAML to JSON, respectively.

Examples:

  • Reading from stdin and outputting to stdout:

    cat examples/test.json | cfn-flip
  • Reading from a file and outputting to stdout:

    cfn-flip examples/test.yaml
  • Reading from a file and outputting to another file:

    cfn-flip examples/test.json output.yaml
  • Reading from a file and cleaning up the output

    cfn-flip -c examples/test.json

Python package

To use AWS CloudFormation Template Flip from your own python projects, import one of the functions flip, to_yaml, or to_json as needed.

from cfn_flip import flip, to_yaml, to_json

"""
All functions expect a string containing serialised data
and return a string containing serialised data
or raise an exception if there is a problem parsing the input
"""

# flip takes a best guess at the serialisation format
# and returns the opposite, converting json into yaml and vice versa
some_yaml_or_json = flip(some_json_or_yaml)

# to_json expects serialised yaml as input, and returns serialised json
some_json = to_json(some_yaml)

# to_yaml expects serialised json as input, and returns serialised yaml
some_yaml = to_yaml(some_json)

# The clean_up flag performs some opinionated, CloudFormation-specific sanitation of the input
# For example, converting uses of Fn::Join to Fn::Sub
# flip, to_yaml, and to_json all support the clean_up flag
clean_yaml = to_yaml(some_json, clean_up=True)

Configuration paramters

You can configure some parameters like:

max_col_width: Maximum columns before breakline. Default value is 200 To change the configuration you can use:

Environment Variable

Linux/Unix: export CFN_MAX_COL_WIDTH=120

Windows: SET CFN_MAX_COL_WIDTH=120

Python

from cfn_tools._config import config
from cfn_flip import flip, to_yaml, to_json

"""
All functions expect a string containing serialised data
and return a string containing serialised data
or raise an exception if there is a problem parsing the input
"""

# Change the default number of columns to break line to 120
config['max_col_width'] = "120"

# flip takes a best guess at the serialisation format
# and returns the opposite, converting json into yaml and vice versa
some_yaml_or_json = flip(some_json_or_yaml)

# to_json expects serialised yaml as input, and returns serialised json
some_json = to_json(some_yaml)

# to_yaml expects serialised json as input, and returns serialised yaml
some_yaml = to_yaml(some_json)

# The clean_up flag performs some opinionated, CloudFormation-specific sanitation of the input
# For example, converting uses of Fn::Join to Fn::Sub
# flip, to_yaml, and to_json all support the clean_up flag
clean_yaml = to_yaml(some_json, clean_up=True)

aws-cfn-template-flip's People

Contributors

azatoth avatar chenrui333 avatar ctd avatar daldei avatar dtrodrigues avatar gcampionpae avatar gliptak avatar hyandell avatar jonathanmorley avatar koiker avatar markpeek avatar michael-k avatar nickeckert avatar pm990320 avatar ryansb avatar santiagocardenas avatar sanyer avatar stilvoid avatar svozza avatar thomasrockhu avatar u6768 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aws-cfn-template-flip's Issues

Long values are split badly

Very long values, such as:

"ConstraintDescription": "Must be 'foo', 'bar', 'baz', 'foo', 'bar', 'baz', 'foo', 'bar', 'baz', 'foo', 'bar', 'baz'..."

Becomes the yaml:

Constraint Description: Must be 'foo', 'bar', 'baz', 'foo', 'bar', 'baz', 'foo', 'bar', 'baz',
    foo', 'bar', 'baz'...

Note the missing quotation on the second line of the yaml there.

Syntax error messages could be improved

Error messages could be improved to make trouble shooting easier:

cfn-flip ./template.yaml 
Error: Could not determine the input format

Turned out ot be due to lingering commas in the yaml file.
Running the same template through https://www.json2yaml.com/ correctly identified the source of the error::

Did Not Find Expected Key While Parsing A Block Mapping At Line 83 Column 13`

Once the offending comma was removed, cfn-flip worked perfectly. I wouldn't have found that issue just using cfn-flip however.

Node support

Hi ,

Do you have any similar tool for nodejs?

Regards,
Sri

Setuptools fails to install 1.1.0.post1 (utf8 character in README.md decoded as ASCII)

Setuptools fails to install the latest release (1.1.0.post1), it fails to decode a (utf8 encoded) character using the default ascii codec.

The problem is the โ€™ (U+2019) in formatโ€™s on line 8 of the README.md, which is encoded as 0xE2 0x80 0x99:

AWS CloudFormation Template Flip is a tool that converts [AWS CloudFormation](https://aws.amazon.com/cloudformation/) templates between [JSON](http://json.org/) and [YAML](http://yaml.org) formats, making use of the YAML formatโ€™s short function syntax where possible.

This stems from the read of README.md performed in setup.py:

long_description=open("README.md").read(),

Stacktrace:

Running cfn_flip-1.1.0.post1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-sceqptzl/cfn_flip-1.1.0.post1/egg-dist-tmp-tuve9xpi
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/setuptools/sandbox.py", line 154, in save_modules
     yield saved
   File "/usr/lib/python3/dist-packages/setuptools/sandbox.py", line 195, in setup_context
     yield
   File "/usr/lib/python3/dist-packages/setuptools/sandbox.py", line 243, in run_setup
     DirectorySandbox(setup_dir).run(runner)
   File "/usr/lib/python3/dist-packages/setuptools/sandbox.py", line 273, in run
     return func()
   File "/usr/lib/python3/dist-packages/setuptools/sandbox.py", line 242, in runner
     _execfile(setup_script, ns)
   File "/usr/lib/python3/dist-packages/setuptools/sandbox.py", line 46, in _execfile
     exec(code, globals, locals)
   File "/tmp/easy_install-sceqptzl/cfn_flip-1.1.0.post1/setup.py", line 17, in <module>
   File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
     return codecs.ascii_decode(input, self.errors)[0]
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 516: ordinal not in range(128)

Empty assignment line in json join leads to poor 'clean' result

When a line in the json Join ends in a space e.g. x = , this leads the result of -c to leave in line breaks. In the working example below, the yaml has the lines split apart as desired, but in the failing example, it does not:

Working example

{
  "content": { "Fn::Join": [ "", [
    "line1 = g\n",
    "line2 = h\n"
  ] ] }
}

correctly becomes:

content: 'line1 = g

  line2 = h

  '

Failing example:

{
  "content": { "Fn::Join": [ "", [
    "line1 = g\n",
    "line2 = \n"
  ] ] }
}

becomes

content: "line1 = g\nline2 = \n"

The example here is from one an Amazon blog post here

cfn-flip takes a json and outputs a json...

This might be because I'm on WLS (Windows Ubuntu). I noticed other people said they had success running it on WLS.

levi@La-Tower:/mnt/c/linux/repos/rho-cf-modules/galaxy$ head -50 galaxy.json
{
  "AWSTemplateFormatVersion" : "2010-09-09",
  "Description" : "AWS CloudFormation stack that bring up a cfncluster with galaxy docker running on master.",
  "Metadata" : {
    "AWS::CloudFormation::Interface" : {
      "ParameterGroups" : [
        {
          "Label" : {
            "default" : "Cluster - Basic Setting"
          },
          "Parameters" : [
            "KeyName",
            "MasterInstanceType",
            "ComputeInstanceType",
            "BaseOS",
            "Scheduler",
            "Environment",
            "DevopsBucket"
          ]
        },
        {
          "Label" : {
            "default" : "EBS - Basic Settings"
          },
          "Parameters" : [
            "VolumeSize"
          ]
        },
        {
          "Label" : {
            "default" : "Network - Advanced Settings"
          },
          "Parameters" : [
            "AccessFrom",
            "ComputeSubnetId",
            "ComputeSubnetCidr",
            "UsePublicIps",
            "AdditionalSG",
            "VPCSecurityGroupId"
          ]
        },
        {
          "Label" : {
            "default" : "Cluster - Advanced Setting"
          },
          "Parameters" : [
            "InitialQueueSize",
            "ComputeWaitConditionCount",
            "MaxQueueSize",
            "SpotPrice",
levi@La-Tower:/mnt/c/linux/repos/rho-cf-modules/galaxy$ cfn
cfncluster  cfn-flip
levi@La-Tower:/mnt/c/linux/repos/rho-cf-modules/galaxy$ cfn-flip galaxy.json galaxy.yml
levi@La-Tower:/mnt/c/linux/repos/rho-cf-modules/galaxy$ head -50 galaxy.yml
{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "AWS CloudFormation stack that bring up a cfncluster with galaxy docker running on master.",
    "Metadata": {
        "AWS::CloudFormation::Interface": {
            "ParameterGroups": [
                {
                    "Label": {
                        "default": "Cluster - Basic Setting"
                    },
                    "Parameters": [
                        "KeyName",
                        "MasterInstanceType",
                        "ComputeInstanceType",
                        "BaseOS",
                        "Scheduler",
                        "Environment",
                        "DevopsBucket"
                    ]
                },
                {
                    "Label": {
                        "default": "EBS - Basic Settings"
                    },
                    "Parameters": [
                        "VolumeSize"
                    ]
                },
                {
                    "Label": {
                        "default": "Network - Advanced Settings"
                    },
                    "Parameters": [
                        "AccessFrom",
                        "ComputeSubnetId",
                        "ComputeSubnetCidr",
                        "UsePublicIps",
                        "AdditionalSG",
                        "VPCSecurityGroupId"
                    ]
                },
                {
                    "Label": {
                        "default": "Cluster - Advanced Setting"
                    },
                    "Parameters": [
                        "InitialQueueSize",
                        "ComputeWaitConditionCount",
                        "MaxQueueSize",
                        "SpotPrice",

Am I doing something wrong?

Inconsistent GetAtt YAML->JSON convertion

!GetAtt YAML->JSON convertion doesn't change YAML's short form to proper JSON function form:

Input YAML:

...
Outputs:
  LambdaTrustRole:
    Value: !GetAtt 'LambdaTrustRole.Arn'
  LambdaTrustRole2:
    Value: !GetAtt LambdaTrustRole.Arn
Policies:
  - PolicyDocument:
      Statement:
        - Resource:
          - !GetAtt [LambdaTrustRole, Arn]
  - PolicyDocument:
      Statement:
        - Resource:
          - Fn::GetAtt:
            - LambdaTrustRole
            - Arn
...

Output JSON:

...
{
    "Outputs": {
        "LambdaTrustRole": {
            "Value": {
                "Fn::GetAtt": "LambdaTrustRole.Arn"
            }
        },
        "LambdaTrustRole2": {
            "Value": {
                "Fn::GetAtt": "LambdaTrustRole.Arn"
            }
        }
    },
    "Policies": [{
            "PolicyDocument": {
                "Statement": [{
                    "Resource": [{
                        "Fn::GetAtt": [
                            "LambdaTrustRole",
                            "Arn"
                        ]
                    }]
                }]
            }
        },
        {
            "PolicyDocument": {
                "Statement": [{
                    "Resource": [{
                        "Fn::GetAtt": [
                            "LambdaTrustRole",
                            "Arn"
                        ]
                    }]
                }]
            }
        }
    ]
}

...

cannot pip install cfn-flip in RHEL 7.5

I attempted to install cfn-flip in RHEL 7.5 but got some error due to python_requires

I have this python version-
[user@workstation ~]$ python --version
Python 2.7.5

error on install-
[user@workstation ~]$ pip install cfn-flip
Collecting cfn-flip
Downloading https://files.pythonhosted.org/packages/37/5e/ba630ddfe0685c2935b8a8141b74b4a476b3f3d339b1eef6d4bcc7d9dc82/cfn_flip-1.1.0.post1.tar.gz
Complete output from command python setup.py egg_info:
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
warnings.warn(msg)
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-FtpxiS/cfn-flip/setup.py", line 33, in
"console_scripts": ["cfn-flip=cfn_flip.main:main"],
File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
_setup_distribution = dist = klass(attrs)
File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 265, in init
self.fetch_build_eggs(attrs.pop('setup_requires'))
File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 289, in fetch_build_eggs
parse_requirements(requires), installer=self.fetch_build_egg
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 618, in resolve
dist = best[req.key] = env.best_match(req, self, installer)
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 862, in best_match
return self.obtain(req, installer) # try and download/install
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 874, in obtain
return installer(requirement)
File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 339, in fetch_build_egg
return cmd.easy_install(req)
File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 623, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 653, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 849, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1130, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1115, in run_setup
run_setup(setup_script, args)
File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 69, in run_setup
lambda: execfile(
File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 120, in run
return func()
File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 71, in
{'file':setup_script, 'name':'main'}
File "setup.py", line 76, in

  File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
    _setup_distribution = dist = klass(attrs)
  File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 269, in __init__
    _Distribution.__init__(self,attrs)
  File "/usr/lib64/python2.7/distutils/dist.py", line 287, in __init__
    self.finalize_options()
  File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 302, in finalize_options
    ep.load()(self, ep.name, value)
  File "/tmp/easy_install-BkUDxA/pytest-runner-4.2/setuptools_scm-3.1.0-py2.7.egg/setuptools_scm/integration.py", line 10, in version_keyword
  File "/tmp/easy_install-BkUDxA/pytest-runner-4.2/setuptools_scm-3.1.0-py2.7.egg/setuptools_scm/version.py", line 66, in _warn_if_setuptools_outdated
setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too old (<12)

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-FtpxiS/cfn-flip/

PyYaml conflict

The requirement for PyYaml 3.1.1 causes fatal conflict with awscli and aws-sam-cli. This dependency should be upgraded to 5.1.1

Multi-level Fn::GetAtt JSON-to-YAML conversion broken

With the following test.json

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Resources": {
        "TestBucket": {
            "Type": "AWS::S3::Bucket",
            "Properties": {
                "BucketName": {
                    "Fn::GetAtt": [
                        "TestDatabase",
                        "Endpoint",
                        "Address"
                    ]
                }
            }
        }
    }
}

running

cfn-flip test.json

yields

AWSTemplateFormatVersion: '2010-09-09'
Resources:
  TestBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !GetAtt 'TestDatabase.Endpoint'

i.e. the ".Address" part is lost. This was working in 0.2.1.

No support for specifying output format

The flip program linked in the README supports flags for specifying the output format. Beyond flipping, this would support the additional use cases of:

  • Clean/validate a file and keep the data format
  • Flip mixed/unknown files into one type

This would be useful in build-time validation and when using CF templates from dependencies.

Command line version doesn't seem to work with Bash Ubuntu on windows

I run the pip install command, which runs without error, but cfn-flip is not found when I try running it. Maybe it's installed somewhere that isn't on the default path or something:

mark@MyComp:/$ pip install cfn_flip
Requirement already satisfied: cfn_flip in /home/mark/.local/lib/python2.7/site-packages
Requirement already satisfied: PyYAML in /home/mark/.local/lib/python2.7/site-packages (from cfn_flip)
Requirement already satisfied: six in /home/mark/.local/lib/python2.7/site-packages (from cfn_flip)

mark@MyComp:/$ cfn-flip
cfn-flip: command not found

mark@MyComp:/$ locate cfn-flip

mark@MyComp:/$

Pinned dependency versions causes incompatibilities

v1.0.5 has dependencies pinned to absolute versions. This causes conflicts and pip check errors when cfn-flip is being installed as a development utility alongside other dependencies.

Can 7ccbf10 be reverted to use compatible version ranges instead of absolute versions?

Python package: ability for to_yaml() to receive input YAML data when performing cleanup of the YAML input

As per the usage guide, the command line tool can be used to perform opinionated cleanup operations with the -c (or --clean) option. The -n (or --no-flip) option can be used to perform operations whilst keeping the same input data format. With the command line tool, I can then perform cleanup operations without switching input format, for example: cfn-flip -n -c my-test.json my-test1.json or cfn-flip -n -c my-test.yaml my-test1.yaml.

With the Python package instead, it does not seem to be possible to perform a cleanup if input data is YAML and to_yaml() is used; JSON input with to_json() seems to be working as expected. Assuming my_json_input_data is a CloudFormation template content in JSON format, to_json(my_json_input_data, clean_up=True) for example returns a JSON output processed with the cleanup logic. Assuming my_yaml_input_data is a CloudFormation template content in YAML format instead, to_yaml(my_yaml_input_data, clean_up=True) for example results in a json.decoder.JSONDecodeError.

Can you explore, if feasible, the possibility of having to_yaml() take YAML data as input?

Thank you for your time!

ODict not working properly after deep copied

I created a pull request for this bug, basically the following test failed before the fix:

def test_post_deepcopy_repr():
    dct = ODict([("a", 1)])
    dct2 = deepcopy(dct)
    assert repr(dct) == repr(dct2)
    dct2["b"] = 2
    assert repr(dct) != repr(dct2)

Error: 'int' object has no attribute 'replace'

Trying to convert a json template into yaml results in the error Error: 'int' object has no attribute 'replace'

Converting the same template in the cloud formation designer in the AWS console results in a properly formatted YAML file.

Taking this formatted YAML file and doing a --clean --no-flip results in a file with the value "null
..."

The template in question is an AWS supplied template which can be obtained from here:
https://s3.amazonaws.com/solutions-reference/aws-instance-scheduler/latest/instance-scheduler.template

It is referenced in the following document:
https://docs.aws.amazon.com/solutions/latest/instance-scheduler/deployment.html

Tabs inside cloudformation YML templates

I found that AWS Cloudformation service accepts YML templates that contain tab characters at the end of the line. (Which I think this is not the standard of YML that AWS follows https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-formats.html). But this tool correctly flags this as a problem. I shared this using AWS support chat and they told me to create an issue here.
So shall the tool be updated to support the same behaviour?
(I would be more than happy to check it if that's the case ๐Ÿ™‚)

Example error:

Error: while scanning for the next token
found character '\t' that cannot start any token
  in "<unicode string>", line 6, column 10:
          - a
             ^

Version: AWS Cloudformation Template Flip, Version 1.2.1
Cloudformation Template: test.txt

cfn-flip mangles the attribute specified in my Fn::GetAtt calls

Hello, I've been testing cfn-flip this week with our JSON templates and we've hit a snag with a template containing Fn::GetAtt. cfn-flip seems to place a . in-between each character of the attribute name. I have created a test template that demonstrates the issue.

{
  "AWSTemplateFormatVersion" : "2010-09-09",
  "Description" : "Test cfn-flip template",
  "Metadata" : {
    "HostedZones" : {
      "123456789012" : {
        "ZoneId" : "ABCDEFGHIG1234",
        "ZoneName" : "testsub.testdomain.com."
      }
    }
  },
  "Parameters" : {
  },
  "Conditions" : {
  },
  "Resources" : {
    "TestRoute53Record" : {
      "Type" : "AWS::Route53::RecordSet",
      "Properties" : {
        "HostedZoneId": { "Fn::FindInMap" : [ "HostedZones", { "Ref" : "AWS::AccountId" }, "ZoneId" ] },
        "Comment" : "Test DNS entry",
        "Name" : {
          "Fn::Join" : [
            "",
            [
              { "Ref" : "ParentStackName" },
              ".",
              { "Ref" : "AWS::Region" },
              ".",
              { "Fn::FindInMap" : [ "HostedZones", { "Ref" : "AWS::AccountId" }, "ZoneName" ]
              }
            ]
          ]
        },
        "Type" : "CNAME",
        "TTL" : "300",
        "ResourceRecords" : [
          { "Fn::GetAtt" : "FakeResource.FakeAttribute" }
        ]
      }
    }
  },
  "Outputs" : {
  }
}

Generates the following output:

AWSTemplateFormatVersion: '2010-09-09'
Description: Test cfn-flip template
Metadata:
  HostedZones:
    '123456789012':
      ZoneId: ABCDEFGHIG1234
      ZoneName: testsub.testdomain.com.
Parameters: {}
Conditions: {}
Resources:
  TestRoute53Record:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !FindInMap [HostedZones, !Ref 'AWS::AccountId', ZoneId]
      Comment: Test DNS entry
      Name: !Join ['', [!Ref 'ParentStackName', ., !Ref 'AWS::Region', ., !FindInMap [
            HostedZones, !Ref 'AWS::AccountId', ZoneName]]]
      Type: CNAME
      TTL: '300'
      ResourceRecords:
        - !GetAtt 'F.a.k.e.R.e.s.o.u.r.c.e...F.a.k.e.A.t.t.r.i.b.u.t.e'
Outputs: {}

Python and cfn-flip versions:

$ python --version
Python 2.7.14

$ pip show cfn-flip
Name: cfn-flip
Version: 0.2.4.post2
Summary: Convert AWS CloudFormation templates between JSON and YAML formats
Home-page: https://github.com/awslabs/aws-cfn-template-flip
Author: Steve Engledow
Author-email: [email protected]
License: Apache2
Location: /Users/sdejean/.virtualenvs/cfnflip-wujEZljp/lib/python2.7/site-packages
Requires: six, PyYAML

Inconsistent conversion of strings from json to yaml

I am converting a document from json to yaml as part of a CloudFormation Template, and am noticing an odd error where some Id's that are marked as strings are being converted to strings, and other times not.

Here's a json snippet I'm working with right now which are the mappings for some of the Generic Elastic Load Balancer ID's for AWS:

        "Regions": {
            "us-east-1": {
                "ELBID": "127311923021",
                "Name": "ue1"
            },
            "us-east-2": {
                "ELBID": "033677994240",
                "Name": "ue2"
            },
            "us-west-1": {
                "ELBID": "027434742980",
                "Name": "uw1"
            },
            "us-west-2": {
                "ELBID": "797873946194",
                "Name": "uw2"
            }
        }
    }

And This is the resulting yaml I'm getting after calling to_yaml:

Mappings:
  Regions:
    us-east-1:
      ELBID: '127311923021'
      Name: ue1
    us-east-2:
      ELBID: 033677994240
      Name: ue2
    us-west-1:
      ELBID: 027434742980
      Name: uw1
    us-west-2:
      ELBID: '797873946194'
      Name: uw2

Strangely enough, any number beginning with 0 is converted, but the ones beginning with other numbers do not. I'm not sure what the expected behavior should be in this case, (either fully converted or not) but having it half and half is inconsistent, and I would believe is a bug.

Currently I'm having errors with using this yaml with sceptre/CloudFormation due to some of the Elastic Load Balancer ID's not being strings.

strips comments

I would use the -c cleanup arg as a pre-commit hook to ensure minimal variation in git commits, but it strips comments.
Could these be preserved?

Nested collections dumped in flow style

First of all, thank you very much for making this happen. I was very close to starting my own project, but created only a bunch of tests based on my own CF JSON->YAML experience.

Anyway, back to issue.

As of now, cfn-flip dumps nested collections in flow instead of block style.
Would be nice to make it dump in block style.

Expected result:

Statement:
- Effect: Allow
  Action:
  - s3:AbortMultipartUpload
  - s3:CopyObject
  - s3:DeleteObject
  - s3:DeleteObjectVersion
  - s3:GetLifecycleConfiguration
  - s3:GetObject
  - s3:GetObjectAccessControlPolicy
  - s3:GetObjectAcl
  - s3:GetObjectExtended
  - s3:GetObjectVersion
  - s3:GetObjectVersionAcl
  - s3:ListBucket
  - s3:ListBucketMultipartUploads
  - s3:ListBucketVersions
  - s3:ListMultipartUploadParts
  - s3:PutLifecycleConfiguration
  - s3:PutObject
  - s3:PutObjectAcl
  - s3:PutObjectInline
  - s3:PutObjectVersionAcl
  - s3:SetObjectAccessControlPolicy

Actual result:

Statement:
- Effect: Allow
  Action: ['s3:AbortMultipartUpload', 's3:CopyObject', 's3:DeleteObject',
    's3:DeleteObjectVersion', 's3:GetLifecycleConfiguration', 's3:GetObject',
    's3:GetObjectAccessControlPolicy', 's3:GetObjectAcl', 's3:GetObjectExtended',
    's3:GetObjectVersion', 's3:GetObjectVersionAcl', 's3:ListBucket', 's3:ListBucketMultipartUploads',
    's3:ListBucketVersions', 's3:ListMultipartUploadParts', 's3:PutLifecycleConfiguration',
    's3:PutObject', 's3:PutObjectAcl', 's3:PutObjectInline', 's3:PutObjectVersionAcl',
    's3:SetObjectAccessControlPolicy']

Might be done by setting the parameter default_flow_style of yaml.dump() to False (ref: Dictionaries without nested collections are not dumped correctly)

Importing cfn_flip causes yaml to use OrderedDict

If a program imports cfn_flip (or does so through troposphere), it affects the way yaml.load() works. Here's my test script:

from pprint import pprint
import yaml
import cfn_flip
yaml_string = '''
base:
  - name: everyone
    ips: [ 0.0.0.0/0 ]
'''
config = yaml.load(yaml_string)
pprint(config)

config has an OrderedDict. If I don't import cfn_flip, it's just a vanilla dict.

Escaped newlines in json result in de-escaped newlines in yml

For example:

>>> a='[1, 2, "hello\\n\\n"]'
>>> print cfn_flip.flip(a)
- 1
- 2
- 'hello


'

I guess these should either remain escaped within double quotes or be turned into literals. This is a common pattern for joining large blocks of text as e.g. an embedded script.

Cleaning a template that contains a lamba function makes the code unreadable

The lambda function

LambdaDeployFunction:
    Type: AWS::Lambda::Function
    DependsOn: LambdaDeployFunctionExecutionRole
    Properties:
      Handler: "index.handler"
      Role: !GetAtt LambdaDeployFunctionExecutionRole.Arn
      Code:
        ZipFile: !Sub |
          import boto3
          import json
          import logging
          import cfnresponse
          import time
          from botocore.exceptions import ClientError

          def handler(event, context):
            logger = logging.getLogger()
            logger.setLevel(logging.INFO)
            logger.info (f"Input parameters from cloud formation: {event}")
            responseData = {}
            if (event["RequestType"] == 'Delete'):
              logger.info("Responding to delete event...")
              cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData)
            
            try:            
              lambdaClient = boto3.client('lambda')
              s3Bucket = event['ResourceProperties']['S3Bucket']
              s3Key = event['ResourceProperties']['S3Key']
              functionName = event['ResourceProperties']['FunctionName']
              logger.info("Updating the function code for Lambda function '{}' to use the code stored in S3 bucket '{}' at key location '{}'".format(functionName, s3Bucket, s3Key))
              logger.info("Sleeping for 5 seconds to allow IAM permisisons to take effect")
              time.sleep(5)             
              response = lambdaClient.update_function_code(
                FunctionName=functionName,
                S3Bucket='{}'.format(s3Bucket),
                S3Key='{}'.format(s3Key),
                Publish=True)
              responseValue = "Function: {}, Version: {}, Last Modified: {}".format(response["FunctionName"],response["Version"],response["LastModified"])
              responseData['Data'] = responseValue
              cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, response["FunctionArn"])
            except ClientError as e:
              errorMessage = e.response['Error']['Message']
              logger.error(errorMessage)
              cfnresponse.send(event, context, cfnresponse.FAILED, responseData)
      Runtime: "python3.6"
      Timeout: "30"

Was converted to:

LambdaDeployFunction:
    Type: AWS::Lambda::Function
    DependsOn: LambdaDeployFunctionExecutionRole
    Properties:
      Handler: index.handler
      Role: !GetAtt 'LambdaDeployFunctionExecutionRole.Arn'
      Code:
        ZipFile: !Sub "import boto3\nimport json\nimport logging\nimport cfnresponse\n\
          import time\nfrom botocore.exceptions import ClientError\n\ndef handler(event,\
          \ context):\n  logger = logging.getLogger()\n  logger.setLevel(logging.INFO)\n\
          \  logger.info (f\"Input parameters from cloud formation: {event}\")\n \
          \ responseData = {}\n  if (event[\"RequestType\"] == 'Delete'):\n    logger.info(\"\
          Responding to delete event...\")\n    cfnresponse.send(event, context, cfnresponse.SUCCESS,\
          \ responseData)\n  \n  try:            \n    lambdaClient = boto3.client('lambda')\n\
          \    s3Bucket = event['ResourceProperties']['S3Bucket']\n    s3Key = event['ResourceProperties']['S3Key']\n\
          \    functionName = event['ResourceProperties']['FunctionName']\n    logger.info(\"\
          Updating the function code for Lambda function '{}' to use the code stored\
          \ in S3 bucket '{}' at key location '{}'\".format(functionName, s3Bucket,\
          \ s3Key))\n    logger.info(\"Sleeping for 5 seconds to allow IAM permisisons\
          \ to take effect\")\n    time.sleep(5)             \n    response = lambdaClient.update_function_code(\n\
          \      FunctionName=functionName,\n      S3Bucket='{}'.format(s3Bucket),\n\
          \      S3Key='{}'.format(s3Key),\n      Publish=True)\n    responseValue\
          \ = \"Function: {}, Version: {}, Last Modified: {}\".format(response[\"\
          FunctionName\"],response[\"Version\"],response[\"LastModified\"])\n    responseData['Data']\
          \ = responseValue\n    cfnresponse.send(event, context, cfnresponse.SUCCESS,\
          \ responseData, response[\"FunctionArn\"])\n  except ClientError as e:\n\
          \    errorMessage = e.response['Error']['Message']\n    logger.error(errorMessage)\n\
          \    cfnresponse.send(event, context, cfnresponse.FAILED, responseData)\n"
      Runtime: python3.6
      Timeout: '30'

I think it should leave the code as it is because it is unreadable now and impossible to maintain.

Unit test failure on macOS with newer version of PyYAML

How to reproduce the unit test failure?

  1. Use a MacBook and issue happens only on macOS with python 2 and newer PyYAML (e.g., latest 5.1).
  2. Run py27 test in tox and test_flip_to_yaml_with_newlines will fail for the last value "d\r". The actual value returned by to_yaml() call is not enclosed in double quotes. Please see detailed error message below.
  3. There seems to be known issue just fixed in PyYAML (Version 5.1 dumps strings differently on different platforms). To confirm, please change setup.py to use older 3.13b1 version (PyYAML==3.13b1) and recreate tox env (tox -r -e py27), which would pass this unit test case again.
```bash
 tests/test_flip.py โœ“โœ“โœ“โœ“โœ“โœ“โœ“โœ“โœ“โœ“โœ“โœ“โœ“โœ“โœ“โœ“โœ“โœ“โœ“                                                                                                                                                                                         59% โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‰    

โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€• test_flip_to_yaml_with_newlines โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•โ€•

    def test_flip_to_yaml_with_newlines():
        """
        Test that strings containing newlines are quoted
        """
    
        source = r'["a", "b\n", "c\r\n", "d\r"]'
    
        expected = "".join([
            '- a\n',
            '- "b\\n"\n',
            '- "c\\r\\n"\n',
            '- "d\\r"\n',
        ])
    
>       assert cfn_flip.to_yaml(source) == expected
E       assert '- a\n- "b\\n...\\n"\n- d\r\n' == '- a\n- "b\\n"..."\n- "d\\r"\n'
E           - a
E           - "b\
E           "
E           - "c\
E           \
E           "
...       - - d
E         
E         ...Full output truncated (3 lines hidden), use '-vv' to show

tests/test_flip.py:413: AssertionError

Combining cfn-flip with cfn-lint when cleaning code

There are a few cases where cleaning the code does not agree with warnings that cfn-lint exposes.

An example is the Security Group ingress 'FromPort' and 'ToPort' properties which should be integers but cfn-flip either makes no judgement on this or converts them to strings.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-security-group-rule.html

Also, cfn-flip puts the parameter for intrinsic functions such as Condition, Ref and GetAtt as strings whereas cfn-lint makes no judgement on this and according to the spec they do not need to be strings.

e.g. PrivateIp:
Description: PrivateIp of the newly created EC2 instance
Value: !GetAtt EC2Instance.PrivateIp

Becomes

PrivateIp:
Description: PrivateIp of the newly created EC2 instance
Value: !GetAtt 'EC2Instance.PrivateIp'

After going through a cfn-flip clean. I cannot see why the parameter needs to be wrapped in quotes as it works perfectly well without them and in my opinion better represents what it is which is an object with a property.

Ideally cfn-flip would use the linting rules that cfn-lint has when it is doing it's cleaning.

Clean Join replacement can produce invalid template

it replaces valid template input

      Value:
        Fn::Join:
          - ","
          - - Fn::If:
                - HasZoneA
                - Ref: PublicSubnetA
                - Ref: "AWS::NoValue"
            - Fn::If:
                - HasZoneB
                - Ref: PublicSubnetB
                - Ref: "AWS::NoValue"
            - Fn::If:
                - HasZoneC
                - Ref: PublicSubnetC
                - Ref: "AWS::NoValue"

by invalid output

      Value: !Sub
      - ${Param1},${Param2},${Param3}
      - Param1: !If
          - HasZoneA
          - !Ref 'PublicSubnetA'
          - !Ref 'AWS::NoValue'
        Param2: !If
          - HasZoneB
          - !Ref 'PublicSubnetB'
          - !Ref 'AWS::NoValue'
        Param3: !If
          - HasZoneC
          - !Ref 'PublicSubnetC'
          - !Ref 'AWS::NoValue'

and then CloudFormation fails with:

Template error: every value of the context object of every Fn::Sub object must be a string or a function that returns a string

I temporarily fixed this with the following:

import cfn_clean

def clean_without_join_replacement(source: Any) -> Any:
    if isinstance(source, dict):
        for key, value in source.items():
            source[key] = clean_without_join_replacement(value)

    elif isinstance(source, list):
        return [clean_without_join_replacement(item) for item in source]

    return source

cfn_clean.clean = clean_without_join_replacement

Originally posted by @christianbeland in cloudtools/troposphere#976 (comment)

yaml ordereddict fix breaks when yaml is a .egg

The fix for #14 doesn't work in all cases. When PyYAML is installed in a .egg file the load of yaml as custom_yaml fails. For troposphere, here is an example of the tests failing. The issue is the imp module does not know how how to handle hooks to zipimport. I have yet to find a good alternate solution to this code.

Slight formatting issue cause wrong template format detection.

A CFN template like this:

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Simple Custom Resource",
    "Metadata": {
        "foo": [
            "bar",
            "bar2"
        ],
    },
    "Resources": {
        "myresource": {
            "Properties": {
                "ServiceToken": "arn:aws:lambda:eu-west-2:112233445566:function:test"
            },
            "Type": "Custom::String"
        }
    }
}

Which contains a JSON formatting warning, but still consider a valid JSON

Warning: Strict JSON does not allow a final comma in an object (dictionary) literal
   |  At line 9, column 4, offset 181
   |  Object started at line 4, column 16, offset 109
/tmp/malformated.json: ok, with warnings

due to the way how the template format is detected cause the cfn-flip to return json from json.

$ cfn-flip /tmp/malformated.json
{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Simple Custom Resource",
    "Metadata": {
        "foo": [
            "bar",
            "bar2"
        ]
    },
    "Resources": {
        "myresource": {
            "Properties": {
                "ServiceToken": "arn:aws:lambda:eu-west-2:112233445566:function:test"
            },
            "Type": "Custom::String"
        }
    }
}

But still works fine if -n parameter is used.

$ cfn-flip -n /tmp/malformated.json
AWSTemplateFormatVersion: '2010-09-09'
Description: Simple Custom Resource
Metadata:
  foo:
    - bar
    - bar2
Resources:
  myresource:
    Properties:
      ServiceToken: arn:aws:lambda:eu-west-2:112233445566:function:test
    Type: Custom::String

What is the type of "some_json_or_yaml" ?

In the docs it says:
[...]
from cfn_flip import flip, to_yaml, to_json

some_yaml_or_json = flip(some_json_or_yaml)
some_json = to_json(some_yaml)
some_yaml = to_yaml(some_json)
clean_yaml = to_yaml(some_json, clean_up=True)
[...]
What is some_json_or_yaml? Is it a file, a python representation of json/yaml, a string, or other?

to_yaml converts entire template, including json values

Hello and thank you so much for this package!

I was wondering if anyone had come across the issue of trying to convert a json template to yaml that includes some JSON values. For example, the AWS::StepFunctions::StateMachine resource has a property DefinitionString which requires the Amazon States Language JSON-based syntax. Currently the converter will convert the value of DefinitionString from JSON to yaml format, which is invalid syntax for that CloudFormation resource.

I was considering a potential solution would be to convert the entire template to yaml, and then load the yaml file as an object and iterate each of the CloudFormation Resources and use the to_json method on the values that should remain JSON (like the DefinitionString for AWS::StepFunctions::StateMachine). Would love to hear of any other solutions you've used or thought of or if you have any feedback on this approach! Thanks!

'!Condition' in YAML incorrectly converted to 'Fn::Condition'

For example

MyAndCondition: !And
  - !Equals ["sg-mysggroup", !Ref ASecurityGroup]
  - !Condition SomeOtherCondition

should convert to

"MyAndCondition": {
   "Fn::And": [
      {"Fn::Equals": ["sg-mysggroup", {"Ref": "ASecurityGroup"}]},
      {"Condition": "SomeOtherCondition"}
   ]
}

but instead is incorrectly being converted to

"MyAndCondition": {
   "Fn::And": [
      {"Fn::Equals": ["sg-mysggroup", {"Ref": "ASecurityGroup"}]},
      {"Fn::Condition": "SomeOtherCondition"}
   ]
}

cfn_clean breaks DefinitionString with Sub function usage

It produces yaml in form:

      DefinitionString: |-
        {
          "Fn::Sub": [
            "\n        {\n           \"QueueUrl\": \"${feedbackQueueUrl}\"\n   ...    }\n        ",
            {
              "feedbackQueueUrl": {
                "Ref": "FeedbackRequestsQueue"
              }
            }
          ]
        }

when should be

      DefinitionString: !Sub
        - "\n         \"QueueUrl\": \"${feedbackQueueUrl}\"\n                    },\n\
          \                    \"End\": true\n                }\n            }\n \
          \       }\n        "
        - feedbackQueueUrl: !Ref 'FeedbackRequestsQueue'

originally found an issue here: cloudtools/troposphere#1423

json to test on

{
    "Resources": {
        "FeedbackRequestStateMachine": {
            "Properties": {
                "DefinitionString": {
                    "Fn::Sub": [
                        "\n        {\n            \"Comment\": \"State machine which awaits until defined date to submit a feedback request\",\n            \"StartAt\": \"WaitForDueDate\",\n            \"States\": {\n                \"WaitForDueDate\": {\n                    \"Type\": \"Wait\",\n                    \"TimestampPath\": \"$.plannedAt\",\n                    \"Next\": \"SubmitFeedbackRequestToSQS\"\n                },\n                \"SubmitFeedbackRequestToSQS\": {\n                    \"Type\": \"Task\",\n                    \"Resource\": \"arn:aws:states:::sqs:sendMessage\",\n                    \"Parameters\": {\n                        \"MessageBody.$\": \"$\",\n                        \"QueueUrl\": \"${feedbackQueueUrl}\"\n                    },\n                    \"End\": true\n                }\n            }\n        }\n        ",
                        {
                            "feedbackQueueUrl": {
                                "Ref": "FeedbackRequestsQueue"
                            }
                        }
                    ]
                },
                "RoleArn": {
                    "Fn::Join": [
                        "/",
                        [
                            {
                                "Fn::Sub": "arn:aws:iam::${AWS::AccountId}:role"
                            },
                            {
                                "Ref": "FeedbackRequestStateMachineRole"
                            }
                        ]
                    ]
                },
                "StateMachineName": {
                    "Fn::Sub": "${AWS::StackName}-FeedbackRequestStateMachine"
                }
            },
            "Type": "AWS::StepFunctions::StateMachine"
        },
        "FeedbackRequestStateMachineRole": {
            "Properties": {
                "AssumeRolePolicyDocument": {
                    "Statement": [
                        {
                            "Action": [
                                "sts:AssumeRole"
                            ],
                            "Effect": "Allow",
                            "Principal": {
                                "Service": [
                                    {
                                        "Fn::Sub": "states.${AWS::Region}.amazonaws.com"
                                    }
                                ]
                            }
                        }
                    ]
                },
                "Policies": [
                    {
                        "PolicyDocument": {
                            "Statement": [
                                {
                                    "Action": [
                                        "sqs:SendMessage"
                                    ],
                                    "Effect": "Allow",
                                    "Resource": {
                                        "Fn::GetAtt": [
                                            "FeedbackRequestsQueue",
                                            "Arn"
                                        ]
                                    }
                                }
                            ]
                        },
                        "PolicyName": "submit_request_to_sqs"
                    }
                ]
            },
            "Type": "AWS::IAM::Role"
        },
        "FeedbackRequestsQueue": {
            "Properties": {
                "MessageRetentionPeriod": 172800,
                "VisibilityTimeout": 600
            },
            "Type": "AWS::SQS::Queue"
        }
    }
}

reverting to 1.2.0 with pip install -U 'cfn_flip==1.2.0' solves the issue

pip install may fail because of the setup_requires dependency for those who have pip configurations

pip install fails because distutils tries to do the install and ignores any pip configurations that have be set. For those of us who have pip configurations set, this might cause a SSL error like this:

distutils.errors.DistutilsError: Download error for https://files.pythonhosted.org/packages/72/a4/d7a5738a3096f22a98bec1609e237b250ebff04e5ea2930305d485337263/pytest_runner-4.2-py2.py3-none-any.whl#sha256=d987fec1e31287592ffe1cb823a8c613c533db4c6aaca0ee1191dbc91e2fcc61: EOF occurred in violation of protocol (_ssl.c:833)

It would be better to at least have a try/except statement around the dependencies with the appropriate error raised to install them.

New release soon?

I'd like to pin a release for use by troposphere. Will you be creating a new release soon? If possible, it would be nice to have git tags applied and pushed to PyPI. Thanks.

install error on pytest-runner on mac osx

Problem/Error:

$ pip install cfn_flip
Collecting cfn_flip
Downloading https://files.pythonhosted.org/packages/ba/c8/a928c55457441c87366eb2423efca9aa0f46380994fd8a476153493c319a/cfn_flip-1.0.3.tar.gz
Complete output from command python setup.py egg_info:
Download error on https://pypi.org/simple/pytest-runner/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726) -- Some packages may not be found!
Couldn't find index page for 'pytest-runner' (maybe misspelled?)
Download error on https://pypi.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726) -- Some packages may not be found!
No local packages or working download links found for pytest-runner
...
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pytest-runner')
...

Fix:

pip install pytest-runner

setup.py is using setup_requires which is causing issues with pip in certain environments

We use pip to install packages for our build pipelines and for vendoring packages prior to deployment onto our cloud environment. The use of setup_requires bypasses the pip installation and means we have to hack around with our pipelines in order to fix these up. Is the dependency on pytest-runner for setup of cfn-flip actually required and if not can it be removed? pip user guide actively discourages use of setup_requires.

Could not determine the input format. Perhaps it's malformed?

I get the above error on most of my cloudformation yaml templates.
These templates are not malformed, in the sense that CloudFormation will parse and launch them as a stack.

What gives?

Here's an example template with some literals redacted.

---
AWSTemplateFormatVersion: 2010-09-09
Description: Template for creating Eden.Garden Worker Cluster

Resources:

  Garden:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      AvailabilityZones:
        - 'us-east-1a'
      VPCZoneIdentifier:
        - 'XXXXXXXXXXXX'
      LaunchConfigurationName:
        Ref: WorkerConfig
      MinSize:
        Ref: NumWorkers
      MaxSize:
        Ref: NumWorkers
      Tags:
        - Key: Name
          Value: 'Garden Worker'
          PropagateAtLaunch: 'true'
    CreationPolicy:
      ResourceSignal:
        Timeout: PT15M
        Count:
          Ref: NumWorkers

  WorkerConfig:
    Type: AWS::AutoScaling::LaunchConfiguration
    Metadata:
      AWS::CloudFormation::Authentication:
        S3AccessCreds:
          type: S3
          roleName:
            Ref: WorkerInstanceRole
      AWS::CloudFormation::Init:
        configSets:
          garden_worker_bootstrap:
          - install_ansible
          - download_ansible_bootstrap
          - run_ansible_bootstrap
        install_ansible:
          commands:
            01_pip_install_ansible:
              command:
                Fn::Join:
                - ' '
                - - "pip"
                  - "install"
                  - "ansible"
              cwd: '/home/ec2-user/'
              ignoreErrors: 'false'
        download_ansible_bootstrap:
          files:
            "/home/ec2-user/.eden/bootstrap/worker.bootstrap.yaml":
              source: http://s3.amazonaws.com/xxxxxxxxxxxxxxxxx
              mode: '000744'
              owner: ec2-user
              group: ec2-user
              authentication: S3AccessCreds
              context:
                rabbitmq_host:
                  Ref: RabbitMQHost
                rabbitmq_port:
                  Ref: RabbitMQPort
        run_ansible_bootstrap:
          commands:
            01_run_ansible_bootstrap:
              command:
                Fn::Join:
                - ' '
                - - "/usr/local/bin/ansible-playbook"
                  - "/home/ec2-user/.eden/bootstrap/worker.bootstrap.yaml"
                  - "--connection=local"
              cwd: "/usr/local/bin/"
              ignoreErrors: 'false'
    Properties:
      AssociatePublicIpAddress: 'true'
      KeyName: 'XXXXXX'
      ImageId: XXXXXXXX
      SecurityGroups:
        - Ref: WorkerSecurityGroup
      InstanceType:
        Ref: WorkerInstanceType
      IamInstanceProfile:
        Ref: WorkerInstanceProfile
      BlockDeviceMappings:
        # this is only for AMZ linux, all else use /dev/sda1. Jesus.
        - DeviceName: "/dev/xvda"
          Ebs:
            VolumeSize:
              Ref: WorkerRootVolumeSize
            VolumeType: 'gp2'
      UserData:
        Fn::Base64:
          Fn::Join:
          - ''
          - - "#!/bin/bash -xe\n"
            - "/opt/aws/bin/cfn-init -v "
            - "         --stack "
            - Ref: AWS::StackName
            - "         --resource WorkerConfig "
            - "         --configsets garden_worker_bootstrap "
            - "         --region "
            - Ref: AWS::Region
            - "\n"
            - "/opt/aws/bin/cfn-signal -e $? "
            - "         --stack "
            - Ref: AWS::StackName
            - "         --resource Garden "
            - "         --region "
            - Ref: AWS::Region
            - "\n"

Parameters:

  NumWorkers:
    Description: Number of workers in the Garden autoscaling group
    Type: String
    Default: 2

  RabbitMQHost:
    Description: DNS name of RabbitMQ server
    Default: 'localhost'
    Type: String

  RabbitMQPort:
    Description: Port of RabbitMQ server
    Default: '5672'
    Type: String

  WorkerInstanceRole:
    Description: Role defining permissions for instances
    Type: String

  WorkerInstanceProfile:
    Description: Profile for instances needing AWS perms
    Type: String

  WorkerSecurityGroup:
    AllowedPattern: 'sg-[a-zA-Z0-9]*'
    ConstraintDescription: must begin with sg- and end with alphanumeric, maybe hex.
    Default: XXXXXXXXXX
    Description: The VPC security group for the RDS instance, specifying ingress rules
    MaxLength: '64'
    MinLength: '4'
    Type: String

  WorkerRootVolumeSize:
    Description: Size in GiB of the worker root volume
    Type: String
    Default: '800'

  WorkerKeyName:
    Description: Name of an existing EC2 KeyPair to enable SSH access to the instances
    Type: AWS::EC2::KeyPair::KeyName
    Default: XXXXXX
    ConstraintDescription: must be the name of an existing EC2 KeyPair.

  WorkerInstanceType:
    Description: Garden Worker EC2 instance type
    Type: String
    Default: t2.micro
    AllowedValues:
    - t2.micro
    - t2.large
    - m3.medium
    - m3.large
    - m3.xlarge
    - m3.2xlarge
    - m4.large
    - m4.xlarge
    - m4.2xlarge
    - m4.4xlarge
    ConstraintDescription: must be a valid EC2 instance type.

Outputs:
  GardenASGName:
    Description: Eden.Garden AutoScaling Group Name
    Value:
      Ref: Garden

using clean produces invalid template

we are using new_template = to_yaml( to_json( orig_template ) ) to cleanup the yaml we are producing with jinja2

clean_up=False we see

  PublicSubnetIds:
    Description: The public subnetids
    Condition: HasPublicSubnet1AZ1
    Value:
      Fn::Join:
        - ','
        -
          - Fn::If:
            - HasPublicSubnet1AZ1
            - !Ref PublicSubnet1AZ1
            - !Ref AWS::NoValue
          - Fn::If:
            - HasPublicSubnet1AZ2
            - !Ref PublicSubnet1AZ2
            - !Ref AWS::NoValue
          - Fn::If:
            - HasPublicSubnet1AZ3
            - !Ref PublicSubnet1AZ3
            - !Ref AWS::NoValue
    Export:
      Name: !Sub vpc-${Environment}-PublicSubnetIds

but with clean_up=True we get

  PublicSubnetIds:
    Description: The public subnetids
    Condition: HasPublicSubnet1AZ1
    Value: !Sub
      - ${Param1},${Param2},${Param3}
      - Param1: !If
          - HasPublicSubnet1AZ1
          - !Ref 'PublicSubnet1AZ1'
          - !Ref 'AWS::NoValue'
        Param2: !If
          - HasPublicSubnet1AZ2
          - !Ref 'PublicSubnet1AZ2'
          - !Ref 'AWS::NoValue'
        Param3: !If
          - HasPublicSubnet1AZ3
          - !Ref 'PublicSubnet1AZ3'
          - !Ref 'AWS::NoValue'
    Export:
      Name: !Sub 'vpc-${Environment}-PublicSubnetIds'

Which CloudFormation complains about .. suggestions ??

Conversion of Fn::GetAtt from YML to JSON for short form syntax

Fn::GetAtt requires two arguments as input, Resource and Attribute.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-getatt.html

The following entry is a valid YML reference, provided in AWS Documentation:

S3SubBucket:
    Type: "AWS::S3::Bucket"
    Properties:
      Tags:
        -
          Key: "MainBucket"
          Value: !GetAtt S3Stack.Outputs.BucketName

When this YML block is converted to json via $ cfn-flip -y , although it converts to a valid JSON syntax, it adds a third argument which makes the Fn::GetAtt be invalid:

"S3SubBucket": {
            "Type": "AWS::S3::Bucket", 
            "Properties": {
                "Tags": [
                    {
                        "Key": "MainBucket", 
                        "Value": {
                            "Fn::GetAtt": [
                                "S3Stack", 
                                "Outputs", 
                                "BucketName"
                            ]
                        }
                    }
                ]
            }
        }

The correct conversion should be:

"Fn::GetAtt": [
     "S3Stack", 
      "Outputs.BucketName"
]

In order to prevent this bug from happening, the template needs to be implemented using full syntax instead of short form:
Fn::GetAtt: [ S3Stack, Outputs.BucketName ]
It is very likely that when the cfn-flip finds !GetAtt S3Stack.Outputs.BucketName, it is simply using '.' as delimiter and breaking in multiple items.

cfn_tools.dump_yaml strips quotes, causing loaded YAML template invalid

Description

When parsing YAML files containing string scalar longer than 8 chars and start with 0, the function cfn_tools.dump_yaml strips single (and double) quotes, which converts string scalar to number scalar. When CFN templates refers these string values in Ref or Sub functions, it causes template error.

Package version: 1.2.0

$ pip show cfn_flip
Name: cfn-flip
Version: 1.2.0
Summary: Convert AWS CloudFormation templates between JSON and YAML formats
Home-page: https://github.com/awslabs/aws-cfn-template-flip
Author: Steve Engledow
Author-email: [email protected]
License: Apache2
Location: /usr/local/lib/python2.7/site-packages
Requires: six, PyYAML, Click
Required-by:

Test-cases

1. Test case '012345678'

import cfn_tools
content = "Parameters:\n  MyAccountId:\n    Type: String\n    Default: '012345678'"
body = cfn_tools.load_yaml(content)
template_body = cfn_tools.dump_yaml(body)
print template_body

Expected output:

Parameters:
  MyAccountId:
    Type: String
    Default: '012345678'

Actual output:

Parameters:
  MyAccountId:
    Type: String
    Default: 012345678

2. Test-case '01234567'

import cfn_tools
content = "Parameters:\n  MyAccountId:\n    Type: String\n    Default: '01234567'"
body = cfn_tools.load_yaml(content)
template_body = cfn_tools.dump_yaml(body)
print template_body

Output:

Parameters:
  MyAccountId:
    Type: String
    Default: '01234567'

Improve conversion output for multiline strings

I have some code in yaml

        UserData:
          Fn::Base64: !Sub |
            write_files:
              - path: /etc/yum.repos.d/nix.repo
                content: |
                   [nix]
                   name=nix Repository
....

But after converting to json (corporate standart) i've got something terrible

               "UserData": {
                    "Fn::Base64": {
                        "Fn::Sub": "write_files:\n  - path: /etc/yum.repos.d/nix.repo\n    content: |\n       [nix]\n       name=nix Repository\n.....

I think - need replace Sub to Find and var names inside to {"Sub":"var"}
Or at least multiline strings, not 1 string

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.