Code Monkey home page Code Monkey logo

container-transform's Introduction

image

image

Documentation Status

container-transform

container-transform is a small utility to transform various docker container formats to one another.

Currently, container-transform can parse and convert:

  • Kubernetes Pod specs
  • ECS task definitions
  • Docker-compose configuration files
  • Marathon Application Definitions or Groups of Applications
  • Chronos Task Definitions

and it can output to:

  • Systemd unit files

Examples

Compose to Kubernetes

$ cat docker-compose.yaml
version: '2'
services:
  etcd:
    cpu_shares: 102.4
    entrypoint: /usr/local/bin/etcd -data-dir /var/etcd/data -listen-client-urls http://127.0.0.1:2379,http://127.0.0.1:4001
      -advertise-client-urls http://127.0.0.1:2379,http://127.0.0.1:4001 -initial-cluster-token
      skydns-etcd
    image: gcr.io/google_containers/etcd-amd64:2.2.1
    mem_limit: 524288000b
  healthz:
    command: -cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
      -port=8080
    cpu_shares: 10.24
    image: gcr.io/google_containers/exechealthz:1.0
    mem_limit: 20971520b
    ports:
    - '8080'
  kube2sky:
    command: --kubecfg-file=/etc/kubernetes/worker-kubeconfig.yaml --domain=cluster.local
    cpu_shares: 102.4
    image: gcr.io/google_containers/kube2sky:1.14
    mem_limit: 209715200b
    volumes:
    - /usr/share/ca-certificates:/etc/ssl/certs
    - /etc/kubernetes/worker-kubeconfig.yaml:/etc/kubernetes/worker-kubeconfig.yaml:ro
    - /etc/kubernetes/ssl:/etc/kubernetes/ssl:ro
  skydns:
    command: -machines=http://127.0.0.1:4001 -addr=0.0.0.0:53 -ns-rotate=false -domain=cluster.local.
    cpu_shares: 102.4
    image: gcr.io/google_containers/skydns:2015-10-13-8c72f8c
    mem_limit: 209715200b
    ports:
    - 53/udp
    - '53'
$ container-transform -i compose -o kubernetes docker-compose.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: null
    version: latest
  name: null
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: null
      version: latest
  template:
    metadata:
      labels:
        app: null
        version: latest
    spec:
      containers:
      - command:
        - /usr/local/bin/etcd
        - -data-dir
        - /var/etcd/data
        - -listen-client-urls
        - http://127.0.0.1:2379,http://127.0.0.1:4001
        - -advertise-client-urls
        - http://127.0.0.1:2379,http://127.0.0.1:4001
        - -initial-cluster-token
        - skydns-etcd
        image: gcr.io/google_containers/etcd-amd64:2.2.1
        name: etcd
        resources:
          limits:
            cpu: 100.0m
            memory: 500Mi
      - args:
        - -cmd=nslookup
        - kubernetes.default.svc.cluster.local
        - 127.0.0.1
        - '>/dev/null'
        - -port=8080
        image: gcr.io/google_containers/exechealthz:1.0
        name: healthz
        ports:
        - containerPort: 8080
          protocol: TCP
        resources:
          limits:
            cpu: 10.0m
            memory: 20Mi
      - args:
        - --kubecfg-file=/etc/kubernetes/worker-kubeconfig.yaml
        - --domain=cluster.local
        image: gcr.io/google_containers/kube2sky:1.14
        name: kube2sky
        resources:
          limits:
            cpu: 100.0m
            memory: 200Mi
        volumeMounts:
        - mountPath: /etc/ssl/certs
          name: usr-share-ca-certificates
        - mountPath: /etc/kubernetes/worker-kubeconfig.yaml
          name: etc-kubernetes-worker-kubeconfig.yaml
          readOnly: true
        - mountPath: /etc/kubernetes/ssl
          name: etc-kubernetes-ssl
          readOnly: true
      - args:
        - -machines=http://127.0.0.1:4001
        - -addr=0.0.0.0:53
        - -ns-rotate=false
        - -domain=cluster.local.
        image: gcr.io/google_containers/skydns:2015-10-13-8c72f8c
        name: skydns
        ports:
        - containerPort: 53
          protocol: UDP
        - containerPort: 53
          protocol: TCP
        resources:
          limits:
            cpu: 100.0m
            memory: 200Mi
      volumes:
      - hostPath:
          path: /etc/kubernetes/ssl
        name: etc-kubernetes-ssl
      - hostPath:
          path: /etc/kubernetes/worker-kubeconfig.yaml
        name: etc-kubernetes-worker-kubeconfig.yaml
      - hostPath:
          path: /usr/share/ca-certificates
        name: usr-share-ca-certificates

Compose to ECS

$ cat docker-compose.yml | container-transform  -v
{
    "family": "python-app",
    "volumes": [
        {
            "name": "host_logs",
            "host": {
                "sourcePath": "/var/log/myapp"
            }
        }
    ],
    "containerDefinitions": [
        {
            "memory": 1024,
            "image": "postgres:9.3",
            "name": "db",
            "essential": true
        },
        {
            "memory": 128,
            "image": "redis:latest",
            "name": "redis",
            "essential": true
        },
        {
            "name": "web",
            "memory": 64,
            "command": [
                "uwsgi",
                "--json",
                "uwsgi.json"
            ],
            "mountPoints": [
                {
                    "sourceVolume": "host_logs",
                    "containerPath": "/var/log/uwsgi/"
                }
            ],
            "environment": [
                {
                    "name": "AWS_ACCESS_KEY_ID",
                    "value": "AAAAAAAAAAAAAAAAAAAA"
                },
                {
                    "name": "AWS_SECRET_ACCESS_KEY",
                    "value": "1111111111111111111111111111111111111111"
                }
            ],
            "essential": true
        }
    ]
}
Container web is missing required parameter "image".
Container web is missing required parameter "cpu".

Quick Help

Usage: container-transform [OPTIONS] [INPUT_FILE]

  container-transform is a small utility to transform various docker
  container formats to one another.

  Default input type is compose, default output type is ECS

  Default is to read from STDIN if no INPUT_FILE is provided

  All options may be set by environment variables with the prefix "CT_"
  followed by the full argument name.

Options:
  -i, --input-type [ecs|compose|marathon|chronos|kubernetes]
  -o, --output-type [ecs|compose|systemd|marathon|chronos|kubernetes]
  -v, --verbose / --no-verbose    Expand/minify json output
  -q, --quiet                     Silence error messages
  --version                       Show the version and exit.
  -h, --help                      Show this message and exit.

Docker Image

To get the docker image, run:

docker pull micahhausler/container-transform:latest

To run the docker image:

docker run --rm -v $(pwd):/data/ micahhausler/container-transform  docker-compose.yml

# or
cat docker-compose.yml | docker run --rm -i micahhausler/container-transform

Installation

To install the latest release (Python 3 only), type:

pip install container-transform

To install the latest code directly from source, type:

pip install git+git://github.com/micahhausler/container-transform.git

Documentation

Full documentation is available at http://container-transform.readthedocs.org

License

MIT License (see LICENSE)

container-transform's People

Contributors

cnadiminti avatar mahayash315 avatar micahhausler avatar opepin avatar philipn avatar samuelkarp avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

container-transform's Issues

use log volumes awseb-logs-...

Hi,
I'd like to use the log volumes as described here:
https://docs.aws.amazon.com/de_de/elasticbeanstalk/latest/dg/create_deploy_docker_v2config.html

I am converting a docker-compose.yml, but don't know how to get such a awseb-logs- volume.

if I use such a docker-compose:

version: '2'

services:
    
    php:
        image: php:latest
        mem_limit: 950m
        restart: always
        links:
            - redis
            - db
        volumes:
            - awseb-logs-mylog:/var/log

it'll give me

{
            "essential": true,
            "image": "php:latest",
            "links": [
                "redis",
                "db"
            ],
            "memory": 950,
            "mountPoints": [
                {
                    "containerPath": "/var/log",
                    "sourceVolume": "Awseb-Logs-mylog"
                }
            ],
            "name": "php"
        },

Support relative volume definition in systemd output

If a volume is defined relative with './mypath' e.g. [Cachet compose](https://github.com/CachetHQ/Docker/blob/v2.2.4/docker-compose.yml#L9] it would be useful to define the path to the docker-compose file. This path would be added as a prefix to the volume definition in the systemd output.

Actual output

# nginx.service #######################################################################
[Unit]
Description=Nginx
After=docker.service cachet.service 
Requires=docker.service cachet.service 

[Service]
ExecStartPre=-/usr/bin/docker kill nginx
ExecStartPre=-/usr/bin/docker rm nginx
ExecStartPre=/usr/bin/docker pull nginx
ExecStart=/usr/bin/docker run \
    --name nginx \
    -p 80:8000 \
    -v ./docker/nginx-site.conf:/etc/nginx/conf.d/default.conf \
    --link cachet \
    --volumes-from cachet \
    nginx 
ExecStop=/usr/bin/docker stop nginx

Expected

# nginx.service #######################################################################
[Unit]
Description=Nginx
After=docker.service cachet.service 
Requires=docker.service cachet.service 

[Service]
ExecStartPre=-/usr/bin/docker kill nginx
ExecStartPre=-/usr/bin/docker rm nginx
ExecStartPre=/usr/bin/docker pull nginx
ExecStart=/usr/bin/docker run \
    --name nginx \
    -p 80:8000 \
    -v /path/to/my/project/docker/nginx-site.conf:/etc/nginx/conf.d/default.conf \
    --link cachet \
    --volumes-from cachet \
    nginx 
ExecStop=/usr/bin/docker stop nginx

I use the micahhausler/container-transform docker image to generate a systemd output. By passing a environment variable to the docker container could fix the problem:

docker run --rm -e COMPOSE_BASE_PATH='$(pwd)' -v $(pwd):/data/ micahhausler/container-transform -o systemd docker-compose.yml

Support .env file for docker compose

Similar to #44 I think, but maybe the situation is different 2 years later?

I am trying to convert https://github.com/jfrog/artifactory-docker-examples/tree/master/docker-compose/xray

$ container-transform -i compose -o marathon xray.yml
Traceback (most recent call last):
(SNIP)
ValueError: invalid literal for int() with base 10: '$XRAY_SERVER_PORT'

Apparently they are using the ".env" file as per https://docs.docker.com/compose/environment-variables/#the-env-file

It seems like container-transform would need to parse the compose file the same way docker compose does to produce the same results?

I tried appending the variables to the "docker-compose.yaml" (xray.yml) file, but the results were not positive either.

$ echo '' >> xray.yml # add a newline to the end of the file
$ sed -e 's/=/: /' .env >> xray.yml

... same error :(

~tommy

Installing on Ubuntu

The installation instructions don't work on Ubuntu, where thepython command points to Python 2.7.

This is what I did to get it working:

  1. Update pip:
wget https://bootstrap.pypa.io/get-pip.py
sudo python3 get-pip.py
  1. Install container-transform to user directory
pip3 install --user container-transform

Now you will have $HOME/.local/bin/container-transform available, and it should be on your path:

$ container-transform --version
container-transform, version 1.1.5

Should we put these instructions in the README? Do they work for someone else?

Related Links:

Failing on empty args when running from docker

Running
docker run --rm -v $(pwd):/data/ micahhausler/container-transform

Returns:

$ docker run --rm -v $(pwd):/data/ micahhausler/container-transform
Traceback (most recent call last):
  File "/usr/local/bin/container-transform", line 11, in <module>
    load_entry_point('container-transform==1.1.5', 'console_scripts', 'container-transform')()
  File "/usr/local/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/container_transform-1.1.5-py3.6.egg/container_transform/client.py", line 64, in transform
    output = converter.convert(verbose)
  File "/usr/local/lib/python3.6/site-packages/container_transform-1.1.5-py3.6.egg/container_transform/converter.py", line 46, in convert
    input_transformer = self._input_class(self._filename)
  File "/usr/local/lib/python3.6/site-packages/container_transform-1.1.5-py3.6.egg/container_transform/compose.py", line 32, in __init__
    self.stream_version = float(stream.get('version', '1'))
AttributeError: 'NoneType' object has no attribute 'get'
$

Choking on ECS input mountPoints

Firstly, thanks for such a useful tool.

I'm having trouble converting the ECS input of:

            "mountPoints": [
                {
                    "sourceVolume": "awseb-logs-api",
                    "containerPath": "/var/log/api"
                }
            ],

The error I get is:

  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/container_transform/converter.py", line 99, in _convert_container
    output[output_name] = emit_func(ingest_func(container.get(input_name)))
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/container_transform/ecs.py", line 216, in ingest_volumes
    return [self._ingest_volume(volume) for volume in volumes]
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/container_transform/ecs.py", line 209, in _ingest_volume
    'host': self.volumes_in.get(volume.get('sourceVolume')).get('path'),
AttributeError: 'NoneType' object has no attribute 'get'

If look further into the problem, would you be happy to accept a pull request?

Thanks

Add support for multiple docker-compose files

Lets say you have docker-compose.yml, docker-compose.sandbox.yml and docker-compose.production.yml.So in your sandbox environment you run
docker-compose -f docker-compose.yml - f docker-compose.sandbox.yml...
Is it doable?

Compose properties

Stacktrace

$ cat docker-compose.yml | docker run --rm -i micahhausler/container-transform
Traceback (most recent call last):
  File "/usr/local/bin/container-transform", line 11, in <module>
    load_entry_point('container-transform==1.1.5', 'console_scripts', 'container-transform')()
  File "/usr/local/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/container_transform-1.1.5-py3.6.egg/container_transform/client.py", line 64, in transform
    output = converter.convert(verbose)
  File "/usr/local/lib/python3.6/site-packages/container_transform-1.1.5-py3.6.egg/container_transform/converter.py", line 57, in convert
    output_transformer
  File "/usr/local/lib/python3.6/site-packages/container_transform-1.1.5-py3.6.egg/container_transform/converter.py", line 90, in _convert_container
    output[output_name] = emit_func(ingest_func(container.get(input_name)))
  File "/usr/local/lib/python3.6/site-packages/container_transform-1.1.5-py3.6.egg/container_transform/compose.py", line 138, in ingest_port_mappings
    return [self._parse_port_mapping(mapping) for mapping in port_mappings]
  File "/usr/local/lib/python3.6/site-packages/container_transform-1.1.5-py3.6.egg/container_transform/compose.py", line 138, in <listcomp>
    return [self._parse_port_mapping(mapping) for mapping in port_mappings]
  File "/usr/local/lib/python3.6/site-packages/container_transform-1.1.5-py3.6.egg/container_transform/compose.py", line 116, in _parse_port_mapping
    'host_port': int(parts[0]),
ValueError: invalid literal for int() with base 10: "{'published'"

I believe this is occurring because of the lack of treatment of the following properties:

version: '3.4'

services:
  mycontainer:
    ports:
      - published: 80
      target: 80
      - published: 443
      target: 443
      - published: 8080
      target: 8080
      - published: 9090
      target: 9090
      - published: 9093
      target: 9093

Regards,
SuperMock

Publish Docker Hub Image

Hey, first of all, awesome tool, thanks a lot for putting it together.

Second, would you mind publishing it as a docker image?

My usecase: we have jenkins with slaves that run docker images (all of our apps are dockerized so we run tests inside containers). The slaves can only do 1 thing, run docker. They don't have any other dependancies.

I want to keep the image that runs the slaves as minimal as I can, so I don't want to install a ton of tooling, but running tools in images is very cheap. I can of course fork and publish it myself, but I figure you want to have control of that image/project. After all, work is yours and is fantastic.

No big rush, as I can just build it once on a slave and then its cached.

Error after installation

I get this error on ubuntu 14.04.04 and python 2.7.6

File "/usr/local/bin/container-transform", line 9, in
load_entry_point('container-transform==1.0.0', 'console_scripts', 'container-transform')()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 351, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2363, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2088, in load
entry = import(self.module_name, globals(),globals(), ['name'])
File "/usr/local/lib/python2.7/dist-packages/container_transform/init.py", line 3, in
from .converter import Converter
File "/usr/local/lib/python2.7/dist-packages/container_transform/converter.py", line 1, in
from .schema import TransformationTypes, ARG_MAP
File "/usr/local/lib/python2.7/dist-packages/container_transform/schema.py", line 28, in
'required': True,
AttributeError: 'str' object has no attribute 'value'

Terraform support?

This is obviously a lot of work but I've been toying around with the idea of writing dev environments which work in Docker compose and then having a tool which can translate these into AWS resources created via Terraform. For example compose could run your application and a DB for development. When it comes time to push to prod, convert those into actual networked resources inside a VPC.

Thoughts?

Issue with ports that have udp

Traceback (most recent call last):
  File "/usr/local/bin/container-transform", line 9, in <module>
    load_entry_point('container-transform==1.0.0', 'console_scripts', 'container-transform')()
  File "/usr/local/lib/python3.5/site-packages/click-6.2-py3.5.egg/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/click-6.2-py3.5.egg/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.5/site-packages/click-6.2-py3.5.egg/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.5/site-packages/click-6.2-py3.5.egg/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/container_transform-1.0.0-py3.5.egg/container_transform/client.py", line 42, in transform
    output = converter.convert(v)
  File "/usr/local/lib/python3.5/site-packages/container_transform-1.0.0-py3.5.egg/container_transform/converter.py", line 52, in convert
    output_transformer
  File "/usr/local/lib/python3.5/site-packages/container_transform-1.0.0-py3.5.egg/container_transform/converter.py", line 95, in _convert_container
    output[output_name] = emit_func(ingest_func(container.get(input_name)))
  File "/usr/local/lib/python3.5/site-packages/container_transform-1.0.0-py3.5.egg/container_transform/compose.py", line 132, in ingest_port_mappings
    return [self._parse_port_mapping(mapping) for mapping in port_mappings]
  File "/usr/local/lib/python3.5/site-packages/container_transform-1.0.0-py3.5.egg/container_transform/compose.py", line 132, in <listcomp>
    return [self._parse_port_mapping(mapping) for mapping in port_mappings]
  File "/usr/local/lib/python3.5/site-packages/container_transform-1.0.0-py3.5.egg/container_transform/compose.py", line 95, in _parse_port_mapping
    'container_port': int(parts[0])
ValueError: invalid literal for int() with base 10: '53/udp'

Any Updates?

Just writing to see if there are any updates happening to this project? It seems like nothing has been happening since January, 2017; however, the last pip release was in June of last year for version 1.1.4. I'm interested in a new version as currently the volume parsing is broken for read-write volume parsing from docker-compose, but it appears to be fixed in master.

Container web is missing required parameter "image".

I want to convert Docker compose file to JSON file by using:
cat docker-compose.prod.yml | container-transform -v command.
For the services with the image parameter is works, but for the services with build parameter gives this error.

Successful:

db:
        image: postgres

Failed:

 db:
        build: 
            context: .
            dockerfile: Dockerfile

How can I solve this problem?

missing required parameter "memory".

Great tool thanks!

Which docker-compose value is the script looking for for the memory value?

I tried mem_limit but no joy:

I get Container db is missing required parameter "memory"..

Error while running `container-transform -h`

root@shahb-e7440:/# python --version
Python 2.7.12
root@shahb-e7440:/# pip -V
pip 8.1.2 from /usr/local/lib/python2.7/dist-packages (python 2.7)
root@shahb-e7440:/# container-transform -h
Traceback (most recent call last):
File "/usr/local/bin/container-transform", line 9, in
load_entry_point('container-transform==1.1.4', 'console_scripts', 'container-transform')()
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 542, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 2569, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 2229, in load
return self.resolve()
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 2235, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "/usr/local/lib/python2.7/dist-packages/container_transform/init.py", line 3, in
from .converter import Converter
File "/usr/local/lib/python2.7/dist-packages/container_transform/converter.py", line 3, in
from .compose import ComposeTransformer
File "/usr/local/lib/python2.7/dist-packages/container_transform/compose.py", line 6, in
from .transformer import BaseTransformer
File "/usr/local/lib/python2.7/dist-packages/container_transform/transformer.py", line 37
class BaseTransformer(object, metaclass=ABCMeta):
^
SyntaxError: invalid syntax

Error when trying to translate the k8s example to Marathon

When trying to use the k8s example as input, and specifying Marathon as output like this

docker run --rm -v /tmp:/data/ micahhausler/container-transform -i kubernetes -o marathon k8s.yml

I see the following error:

docker run --rm -v /tmp:/data/ micahhausler/container-transform -i kubernetes -o marathon k8s.yml
Traceback (most recent call last):
  File "/usr/local/bin/container-transform", line 9, in <module>
    load_entry_point('container-transform==1.1.4', 'console_scripts', 'container-transform')()
  File "/usr/local/lib/python3.5/site-packages/click-6.6-py3.5.egg/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/click-6.6-py3.5.egg/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.5/site-packages/click-6.6-py3.5.egg/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.5/site-packages/click-6.6-py3.5.egg/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/container_transform-1.1.4-py3.5.egg/container_transform/client.py", line 64, in transform
    output = converter.convert(verbose)
  File "/usr/local/lib/python3.5/site-packages/container_transform-1.1.4-py3.5.egg/container_transform/converter.py", line 57, in convert
    output_transformer
  File "/usr/local/lib/python3.5/site-packages/container_transform-1.1.4-py3.5.egg/container_transform/converter.py", line 90, in _convert_container
    output[output_name] = emit_func(ingest_func(container.get(input_name)))
  File "/usr/local/lib/python3.5/site-packages/container_transform-1.1.4-py3.5.egg/container_transform/kubernetes.py", line 321, in ingest_cpu
    cpu = float(int(cpu[:-1]) / 1000)
ValueError: invalid literal for int() with base 10: '100.0'

conversion fails with entrypoint and command using the exec notation

conversion fails with entrypoint and command using the "exec" notation
entrypoint: ["executable", "param1"]
command: ["executable", "param1", "param2"]

Traceback (most recent call last):
  File "/usr/local/bin/container-transform", line 9, in <module>
    load_entry_point('container-transform==0.6.0', 'console_scripts', 'container-transform')()
  File "/Library/Python/2.7/site-packages/click/core.py", line 664, in __call__
    return self.main(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/click/core.py", line 644, in main
    rv = self.invoke(ctx)
  File "/Library/Python/2.7/site-packages/click/core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Library/Python/2.7/site-packages/click/core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/container_transform/client.py", line 42, in transform
    output = converter.convert(v)
  File "/Library/Python/2.7/site-packages/container_transform/converter.py", line 56, in convert
    output_transformer
  File "/Library/Python/2.7/site-packages/container_transform/converter.py", line 99, in _convert_container
    output[output_name] = emit_func(ingest_func(container.get(input_name)))
  File "/Library/Python/2.7/site-packages/container_transform/ecs.py", line 180, in emit_entrypoint
    return entrypoint.split()

Container names are not unique

It would be great if you could add a project-name parameter to container-transform.

Currently the container names created by container-transform don't include the project name of the docker-compose file. This leads to collisions with other project using the same container name (e.g. "nginx").

I believe that usually one would not try to make the service names (or container names) globally unique like "projectxynginx" inside a docker-compose.yml.

Example

Run two similar projects on the same server: Both projects have a frontend named web and a backend named phpfpm:

Project A

version: '2'

services:
  web:
    image: nginx:1-alpine
    volumes:
      - ./config/nginx/conf.d:/etc/nginx/conf.d:ro
      - ./config/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    volumes_from:
      - phpfpm:ro
    networks:
      - front
      - back
    labels:
      - "traefik.port=80"
      - "traefik.frontend.rule=Host:wambo.co,www.wambo.co"
      - "traefik.protocol=http"
      - "traefik.frontend.entryPoints=http,https"
      - "traefik.frontend.passHostHeader=true"

  phpfpm:
    image: php:7-fpm-alpine
    volumes:
      - ./web:/var/www/html
    networks:**phpfpm**
      - back

networks:
  front:
    driver: bridge
  back:
    driver: bridge

Project B

version: '2'

services:
  web:
    image: nginx:1-alpine
    volumes:
      - ./config/nginx/conf.d:/etc/nginx/conf.d:ro
      - ./config/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    volumes_from:
      - phpfpm:ro
    networks:
      - front
      - back
    labels:
      - "traefik.port=80"
      - "traefik.frontend.rule=Host:beta.wambo.co"
      - "traefik.protocol=http"
      - "traefik.frontend.entryPoints=http,https"
      - "traefik.frontend.passHostHeader=true"

  phpfpm:
    image: php:7-fpm-alpine
    volumes:
      - ./web:/var/www/html
    networks:
      - back

networks:
  front:
    driver: bridge
  back:
    driver: bridge

If I now create systemd services for both projects using container-tranform I will get collisions because the docker containers of both projects have the same names:

Project A

# web.service #######################################################################
[Unit]
Description=Web
After=docker.service 
Requires=docker.service 

[Service]
ExecStartPre=-/usr/bin/docker kill web
ExecStartPre=-/usr/bin/docker rm web
ExecStartPre=/usr/bin/docker pull nginx:1-alpine
ExecStart=/usr/bin/docker run \
    --name web \
    -p 9080:80 \
    --net ['front', 'back'] \
    -v ./config/nginx/conf.d:/etc/nginx/conf.d:ro \
    -v ./config/nginx/nginx.conf:/etc/nginx/nginx.conf:ro \
    --label traefik.frontend.entryPoints="http,https" \
    --label traefik.frontend.passHostHeader="true" \
    --label traefik.frontend.rule="Host:wambo.co,www.wambo.co" \
    --label traefik.port="80" \
    --label traefik.protocol="http" \
    --volumes-from phpfpm:ro \
    nginx:1-alpine 
ExecStop=/usr/bin/docker stop web
# phpfpm.service #######################################################################
[Unit]
Description=Phpfpm
After=docker.service 
Requires=docker.service 

[Service]
ExecStartPre=-/usr/bin/docker kill phpfpm
ExecStartPre=-/usr/bin/docker rm phpfpm
ExecStartPre=/usr/bin/docker pull php:7-fpm-alpine
ExecStart=/usr/bin/docker run \
    --name phpfpm \
    --net ['back'] \
    -v ./web:/var/www/html \
    php:7-fpm-alpine 
ExecStop=/usr/bin/docker stop phpfpm

Project B

# server.service #######################################################################
[Unit]
Description=Server**phpfpm**
After=docker.service 
Requires=docker.service 

[Service]
ExecStartPre=-/usr/bin/docker kill server
ExecStartPre=-/usr/bin/docker rm server
ExecStartPre=/usr/bin/docker pull nginx:1-alpine
ExecStart=/usr/bin/docker run \
    --name server \
    --net ['front', 'back'] \
    -v ./config/nginx/conf.d:/etc/nginx/conf.d:ro \
    -v ./config/nginx/nginx.conf:/etc/nginx/nginx.conf:ro \
    --label traefik.frontend.entryPoints="http,https" \
    --label traefik.frontend.passHostHeader="true" \
    --label traefik.frontend.rule="Host:beta.wambo.co" \
    --label traefik.port="80" \
    --label traefik.protocol="http" \
    --volumes-from phpfpm:ro \
    nginx:1-alpine 
ExecStop=/usr/bin/docker stop server
# phpfpm.service #######################################################################
[Unit]
Description=Phpfpm
After=docker.service 
Requires=docker.service 

[Service]
ExecStartPre=-/usr/bin/docker kill phpfpm
ExecStartPre=-/usr/bin/docker rm phpfpm
ExecStartPre=/usr/bin/docker pull php:7-fpm-alpine
ExecStart=/usr/bin/docker run \
    --name phpfpm \
    --net ['back'] \
    -v ./web:/var/www/html \
    php:7-fpm-alpine 
ExecStop=/usr/bin/docker stop phpfpm

... I always wanted to learn Python. Maybe this is the time for me so I can create a pull request for your project ๐Ÿ˜„

container-transform 1.1.5 and/or Latest container doesn't appear to be working, at least for ECS -> Compose

$ <snipped> | docker run --rm -i micahhausler/container-transform --input-type ecs --output-type compose
Unable to find image 'micahhausler/container-transform:latest' locally
latest: Pulling from micahhausler/container-transform
79650cf9cc01: Pull complete
581a2604819e: Pull complete
ada98fe44e3a: Pull complete
70b4de28adfd: Pull complete
bcd83fe4359e: Pull complete
a79ed293bf7c: Pull complete
ca7ba4450342: Pull complete
bdc934014ded: Pull complete
Digest: sha256:3836dc18396e463afce186372f715137007cf4f19680a130833ff8763332038e
Status: Downloaded newer image for micahhausler/container-transform:latest
Traceback (most recent call last):
  File "/usr/local/bin/container-transform", line 11, in <module>
    load_entry_point('container-transform==1.1.5', 'console_scripts', 'container-transform')()
  File "/usr/local/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/container_transform-1.1.5-py3.6.egg/container_transform/client.py", line 64, in transform
    output = converter.convert(verbose)
  File "/usr/local/lib/python3.6/site-packages/container_transform-1.1.5-py3.6.egg/container_transform/converter.py", line 57, in convert
    output_transformer
  File "/usr/local/lib/python3.6/site-packages/container_transform-1.1.5-py3.6.egg/container_transform/converter.py", line 83, in _convert_container
    if container.get(input_name) and \
AttributeError: 'str' object has no attribute 'get'

Add "docker run" output option

I didn't see this listed as one of the output options, so I'd like to request this feature:

Sometimes it can be useful to be able to run a docker run command that is equivalent to you docker-compose.yml. So the environment section would be converted to --env options, etc.

Doesn't seem to support and enumeration of ports for DBs

I have a docker-compose.yml that looks as follows:

version: '2'
services:
  oracle:
    image: wnameless/oracle-xe-11g:latest
    hostname: adb
    ports:
      - "49160:22"
      - "49161:1521"
      - "49080:8080"
    environment:
      - processes=200
    volumes:
      - data:/u01/app/oracle
  web:
    build: .
    ports:
     - "8080:8080"
    volumes:
     - .:/app
    depends_on:
     - oracle
    links:
     - oracle
volumes:
  data: {}

When I run the command to convert this it fails on what it calls an enum:

โžœ  cas-acronyms-db git:(develop) โœ—  cat docker-compose-aws.yml | container-transform -v
Traceback (most recent call last):
  File "/usr/local/bin/container-transform", line 7, in <module>
    from container_transform.client import transform
  File "/usr/local/lib/python2.7/site-packages/container_transform/__init__.py", line 3, in <module>
    from .converter import Converter
  File "/usr/local/lib/python2.7/site-packages/container_transform/converter.py", line 1, in <module>
    from .schema import TransformationTypes, ARG_MAP
  File "/usr/local/lib/python2.7/site-packages/container_transform/schema.py", line 1, in <module>
    from enum import Enum
ImportError: No module named enum

I tried using the original docker-compose version 1 and the same issue.

how to generate outer json properties?

When I run container-transform on my docker-compose.yml, I only get the array for containerDefinitions generated.

docker-compose.yml

runner:
    image: ceroic/scrapyd
    ports:
        - "6800:6800"
    environment:
        - CLOSESPIDER_ITEMCOUNT=20
        - DOWNLOAD_DELAY=10
        - ELASTICSEARCH_SERVER=localhost
        - ELASTICSEARCH_PORT=9200

translated to:

[
    {
        "environment": [
            {
                "name": "CLOSESPIDER_ITEMCOUNT", 
                "value": "1"
            }, 
            {
                "name": "ELASTICSEARCH_PORT", 
                "value": "9200"
            }, 
            {
                "name": "DOWNLOAD_DELAY", 
                "value": "2"
            }, 
            {
                "name": "ELASTICSEARCH_SERVER", 
                "value": "localhost"
            }
        ], 
        "portMappings": [
            {
                "containerPort": 6800, 
                "hostPort": 6800
            }
        ], 
        "image": "ceroic/scrapyd", 
        "name": "runner", 
        "essential": true
    }
]

What do I have to change to get the whole ECS task definition to be generated, including family and containerDefinitions key?

Add required option

Similar to cpu_shares and mem_limit (which aren't official options for docker-compose.yml, but were added to container transform so they can be outputted in the ECS task JSON definition), we should add the required option, which translates directly to the required option in the TASK definition JSON

Should default to true when it doesn't exist (as it currently does)

Error container-transform 1.1.5 compose > ECS

command:
container-transform -i compose -o ecs docker-compose.yaml

Traceback (most recent call last):
  File "/home/leonardo/.local/bin/container-transform", line 11, in <module>
    load_entry_point('container-transform==1.1.5', 'console_scripts', 'container-transform')()
  File "/home/leonardo/.local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/leonardo/.local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/leonardo/.local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/leonardo/.local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/leonardo/.local/lib/python3.6/site-packages/container_transform/client.py", line 64, in transform
    output = converter.convert(verbose)
  File "/home/leonardo/.local/lib/python3.6/site-packages/container_transform/converter.py", line 57, in convert
    output_transformer
  File "/home/leonardo/.local/lib/python3.6/site-packages/container_transform/converter.py", line 90, in _convert_container
    output[output_name] = emit_func(ingest_func(container.get(input_name)))
  File "/home/leonardo/.local/lib/python3.6/site-packages/container_transform/compose.py", line 305, in ingest_volumes
    in volumes
  File "/home/leonardo/.local/lib/python3.6/site-packages/container_transform/compose.py", line 306, in <listcomp>
    if self._ingest_volume(volume) is not None
  File "/home/leonardo/.local/lib/python3.6/site-packages/container_transform/compose.py", line 271, in _ingest_volume
    parts = volume.split(':')
AttributeError: 'dict' object has no attribute 'split'

backwards compatibility with python 2.7

container-transform is currently not backwards compatible with python 2.7:

Traceback (most recent call last): File "/usr/local/bin/container-transform", line 9, in <module> load_entry_point('container-transform==1.1.5', 'console_scripts', 'container-transform')() File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 565, in load_entry_point return get_distribution(dist).load_entry_point(group, name) File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 2697, in load_entry_point return ep.load() File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 2370, in load return self.resolve() File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 2376, in resolve module = __import__(self.module_name, fromlist=['__name__'], level=0) File "/Library/Python/2.7/site-packages/container_transform/__init__.py", line 3, in <module> from .converter import Converter File "/Library/Python/2.7/site-packages/container_transform/converter.py", line 3, in <module> from .compose import ComposeTransformer File "/Library/Python/2.7/site-packages/container_transform/compose.py", line 6, in <module> from .transformer import BaseTransformer File "/Library/Python/2.7/site-packages/container_transform/transformer.py", line 38 class BaseTransformer(object, metaclass=ABCMeta): ^ SyntaxError: invalid syntax

adding support to `ulimits`

we need the supports for ulimits section

docker-compose example:

    ulimits:
      nproc: 65535
      nofile:
        soft: 65535
        hard: 65535

Doesn't support soft memory limition (memory reversation) or docker compose V3

currently this project only supports docker compose v2 and doesn't tallow us to specify the soft memory limitation.

suggestion: adding supports to docker compose file v3 resource section, which supports both hard and soft limitation

resources:
  limits:
    cpus: '0.001'
    memory: 50M
  reservations:
    cpus: '0.0001'
    memory: 20M

Support for "9995:9995/tcp"

Traceback (most recent call last):
  File "/usr/local/bin/container-transform", line 9, in <module>
    load_entry_point('container-transform==1.1.4', 'console_scripts', 'container-transform')()
  File "/usr/local/lib/python3.5/site-packages/click-6.6-py3.5.egg/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/click-6.6-py3.5.egg/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.5/site-packages/click-6.6-py3.5.egg/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.5/site-packages/click-6.6-py3.5.egg/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/container_transform-1.1.4-py3.5.egg/container_transform/client.py", line 64, in transform
    output = converter.convert(verbose)
  File "/usr/local/lib/python3.5/site-packages/container_transform-1.1.4-py3.5.egg/container_transform/converter.py", line 57, in convert
    output_transformer
  File "/usr/local/lib/python3.5/site-packages/container_transform-1.1.4-py3.5.egg/container_transform/converter.py", line 90, in _convert_container
    output[output_name] = emit_func(ingest_func(container.get(input_name)))
  File "/usr/local/lib/python3.5/site-packages/container_transform-1.1.4-py3.5.egg/container_transform/compose.py", line 138, in ingest_port_mappings
    return [self._parse_port_mapping(mapping) for mapping in port_mappings]
  File "/usr/local/lib/python3.5/site-packages/container_transform-1.1.4-py3.5.egg/container_transform/compose.py", line 138, in <listcomp>
    return [self._parse_port_mapping(mapping) for mapping in port_mappings]
  File "/usr/local/lib/python3.5/site-packages/container_transform-1.1.4-py3.5.egg/container_transform/compose.py", line 105, in _parse_port_mapping
    'container_port': int(parts[1]),
ValueError: invalid literal for int() with base 10: '9995/tc'

Issue with version container-transform 1.0 and docker-compose file version 1

My docker-compose.yml file

rumifico:
  image: id.dkr.ecr.us-east-1.amazonaws.com/rumifico:latest
  cpu_shares: 1024
  mem_limit: 2147483648b
  environment:
    RACK_ENV: xolo
  ports:
   - "1701"

When i transform and send the output to aws with version 0.6.1, it's OK ๐Ÿ‘

sudo -H pip install --upgrade "container-transform==0.6.1"
cat docker-compose.yml | container-transform  -v > tmp/task.json
aws ecs register-task-definition --cli-input-json file://tmp/task.json --family rumifico-master
{
    "taskDefinition": {
        "status": "ACTIVE",
        "family": "rumifico-master",
        "requiresAttributes": [
            {
                "name": "com.amazonaws.ecs.capability.ecr-auth"
            }
        ],
        "volumes": [],
        "taskDefinitionArn": "arn:aws:ecs:us-east-1:520600400730:task-definition/rumifico-master:18",
        "containerDefinitions": [
            {
                "environment": [
                    {
                        "name": "RACK_ENV",
                        "value": "xolo"
                    }
                ],
                "name": "rumifico",
                "mountPoints": [],
                "image": "520600400730.dkr.ecr.us-east-1.amazonaws.com/rumifico:latest",
                "cpu": 1024,
                "portMappings": [
                    {
                        "protocol": "tcp",
                        "containerPort": 1701,
                        "hostPort": 0
                    }
                ],
                "memory": 2048,
                "essential": true,
                "volumesFrom": []
            }
        ],
        "revision": 18
    }
}

And with version 1.0.0

sudo -H pip install --upgrade "container-transform==1.0.0"
cat docker-compose.yml | container-transform  -v > tmp/task.json
Traceback (most recent call last):
  File "/usr/local/bin/container-transform", line 7, in <module>
    from container_transform.client import transform
  File "/Library/Python/2.7/site-packages/container_transform/__init__.py", line 3, in <module>
    from .converter import Converter
  File "/Library/Python/2.7/site-packages/container_transform/converter.py", line 3, in <module>
    from .compose import ComposeTransformer
  File "/Library/Python/2.7/site-packages/container_transform/compose.py", line 6, in <module>
    from .transformer import BaseTransformer
  File "/Library/Python/2.7/site-packages/container_transform/transformer.py", line 25
    class BaseTransformer(object, metaclass=ABCMeta):
                                           ^
SyntaxError: invalid syntax
make: *** [ecs-task-definition] Error 1

I was mistaken when I opened this issue

Never mind what this used to say - I thought I found a bug, but it was my mistake.

Since this is already here, I will say: you are a hero. This is an awesome tool; thanks for making it.

Support 'extends' syntax

When converting from a docker-compose file which relies on 'extends' keyword, then container-transform does not follows the definition from that service.

It would be useful to see it being supported.

Invalid volume.name of format xxxx.xxx

When specifying volumes like this

  volumes:
    - '/etc/certs/cert.crt:/etc/nginx/certs/cert.crt'
    - '/etc/certs/cert.key:/etc/nginx/certs/cert.key'

The container-transform will output something like this

    "volumes": [
        {
            "host": {
                "sourcePath": "/etc/certs/cert.crt"
            },
            "name": "EtcCertsCert.Crt"
        },
        {
            "host": {
                "sourcePath": "/etc/certs/cert.key"
            },
            "name": "EtcCertsCert.Key"
        }
    ]

and

            "mountPoints": [
                {
                    "containerPath": "somePath.crt",
                    "sourceVolume": "EtcCertsCert.Crt"
                },
                {
                    "containerPath": "somePath.key",
                    "sourceVolume": "EtcCertsCert.Key"
                }
            ],

Unfortunately, EtcCertsCert.Crt and EtcCertsCert.Key are not valid volume names (Amazon ECS will complain with Invalid volume.name). I had to manually replace EtcCertsCert.Crt with EtcCertsCert_Crt and EtcCertsCert.Key with EtcCertsCert_Key.

If a volume.label happens to be a direct filename, the . needs to be replaced with some other valid character

Support the ':Z' access option for volumes

When passing the following:

version: "3"

services:
  local-registry:
    name: local-registry
    image: registry
    environment:
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /var/lib/registry
      REGISTRY_PROXY_REMOTEURL: https://registry-1.docker.io
    ports:
      - "127.0.0.1:5000:5000"
    networks:
      - docker-registry
    volumes:
      - "docker-registry:/var/lib/registry:Z"

networks:
  docker-registry:
    external: true

volumes:
  docker-registry:

I get:

# local-registry.service #######################################################################
[Unit]
Description=Local-Registry
After=docker.service 
Requires=docker.service 

[Service]
ExecStartPre=-/usr/bin/docker kill local-registry
ExecStartPre=-/usr/bin/docker rm local-registry
ExecStartPre=/usr/bin/docker pull registry
ExecStart=/usr/bin/docker run \
    --name local-registry \
    -p 127.0.0.1:5000:5000 \
    --net ['docker-registry'] \
    -e "REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io" \
    -e "REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/var/lib/registry" \
    registry 
ExecStop=/usr/bin/docker stop local-registry

Removing the :Z option makes the volume flag appear again. Some quick code-diving suggests that the :z and :Z options are completely ignored - a little unfortunate, I use selinux :|

Port mapping error while converting ecs to compose

Consider I have an 'ecs' script snippet

{
    "containerDefinitions": [
        {
            "essential": true, 
            "image": "ctlc/serf", 
            "name": "serf", 
            "portMappings": [
                {
                    "containerPort": 7373
                }, 
                {
                    "containerPort": 7946, 
                    "hostPort": 80
                }
            ]
        }
  ]
}

I tried to convert the above configuration to compose format using the command cat tranform.txt | container-transform --input-type ecs --output-type compose. I got an error like this.

 File "/local/lib/python2.7/site-packages/container_transform/ecs.py", line 113, in _parse_port_mapping
    'host_port': int(mapping['hostPort']),
KeyError: 'hostPort'

After exploring the source code, I could find that you are accessing the hostPort key without checking its presence.

Docker compose to multiple marathon json

Hi - Is there a plan to support docker-compose v3 .
I tried to convert marathon.json to docker compose but many definitions got lost . eg the placement constraint.
I think those are supported in dokcer compose v3 .

Also , if a compose has multiple app definitions, is there a way to create multiple marathon.json from a single compose file

Container svc is missing required parameter "memory"

Docker Compose version 2 supports setting soft limits on memory using mem_reservation, and ECS accepts this as memoryReservation.

I received the error Container svc is missing required parameter "memory". (but exit code 0) whether I had or didn't have a memory parameter in my YAML.

If you can point me to where I can correct this, I'm happy to submit a PR.

Input

docker-compose.yml like this:

services:
  svc:
    cpu_shares: 512
    mem_reservation: 512m
    environment:
      MYSQL_ROOT_PASSWORD: secret
    image: mariadb
    ports:
    - '3306'
version: '2'

Expected

I expected the output to contain memoryReservation ECS Task Definition key:

{
    "containerDefinitions": [
        {
            "cpu": 512,
            "environment": [
                {
                    "name": "MYSQL_ROOT_PASSWORD",
                    "value": "secret"
                }
            ],
            "essential": true,
            "image": "mariadb",
            "memoryReservation": 512,
            "name": "svc",
            "portMappings": [
                {
                    "containerPort": 3306
                }
            ]
        }
    ],
    "family": "",
    "volumes": []
}

Actual

Instead, I received JSON output without the memoryReservation key, and got a warning about memory:

{
    "containerDefinitions": [
        {
            "cpu": 512,
            "environment": [
                {
                    "name": "MYSQL_ROOT_PASSWORD",
                    "value": "secret"
                }
            ],
            "essential": true,
            "image": "mariadb",
            "name": "svc",
            "portMappings": [
                {
                    "containerPort": 3306
                }
            ]
        }
    ],
    "family": "",
    "volumes": []
}
Container svc is missing required parameter "memory".

Python error when trying to transform docker-compose

I'm installing it as shown in docs:

pip install container-transform
Collecting container-transform
  Downloading container-transform-1.1.4.tar.gz
Requirement already satisfied: PyYAML<4,>=3.10 in /Library/Python/2.7/site-packages (from container-transform)
Requirement already satisfied: Jinja2>=2.7.0 in /Library/Python/2.7/site-packages (from container-transform)
Collecting click>=3.3 (from container-transform)
  Downloading click-6.6-py2.py3-none-any.whl (71kB)
    100% |โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 71kB 2.2MB/s
Requirement already satisfied: MarkupSafe in /Library/Python/2.7/site-packages (from Jinja2>=2.7.0->container-transform)
Building wheels for collected packages: container-transform
  Running setup.py bdist_wheel for container-transform ... done
  Stored in directory: /Users/danny.heinrich/Library/Caches/pip/wheels/f3/44/e8/c8034b152a761279d741568098c3eee2ccbbcf776a49c161a8
Successfully built container-transform
Installing collected packages: click, container-transform
Successfully installed click-6.6 container-transform-1.1.4

It installs successfully but after running it as shown in docs it gives me:

cat docker-compose.yml |container-transform -v
Traceback (most recent call last):
  File "/usr/local/bin/container-transform", line 7, in <module>
    from container_transform.client import transform
  File "/usr/local/lib/python2.7/site-packages/container_transform/__init__.py", line 3, in <module>
    from .converter import Converter
  File "/usr/local/lib/python2.7/site-packages/container_transform/converter.py", line 3, in <module>
    from .compose import ComposeTransformer
  File "/usr/local/lib/python2.7/site-packages/container_transform/compose.py", line 6, in <module>
    from .transformer import BaseTransformer
  File "/usr/local/lib/python2.7/site-packages/container_transform/transformer.py", line 36
    class BaseTransformer(object, metaclass=ABCMeta):
                                           ^
SyntaxError: invalid syntax

System: macOS Sierra
Version: 10.12.1
HW: MacBook pro early 2015

Invalid YAML output

Not all string values are quoted in the YAML output when I convert ecs to compose. Some work but I've got an AWS ARN with some slashes that causes docker-compose to not accept the file.

installation error

after the python install :
Traceback (most recent call last):
File "/usr/local/bin/container-transform", line 9, in
load_entry_point('container-transform==1.1.5', 'console_scripts', 'container-transform')()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/init.py", line 565, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/init.py", line 2697, in load_entry_point
return ep.load()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/init.py", line 2370, in load
return self.resolve()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/init.py", line 2376, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "/Library/Python/2.7/site-packages/container_transform/init.py", line 3, in
from .converter import Converter
File "/Library/Python/2.7/site-packages/container_transform/converter.py", line 1, in
from .schema import TransformationTypes, ARG_MAP
File "/Library/Python/2.7/site-packages/container_transform/schema.py", line 38, in
'required': True,
AttributeError: 'str' object has no attribute 'value'

env_file support

First of all: I love your tool!

I am converting docker-compose.yml to Dockerrun.aws.json. This has been working very well for almost 3 years now.

Recently I added for one service an env_file directive:

version: '2'

services:
    php:
        env_file:
            - /var/app/current/myenv

so, besides my docker-compose there is a file myenv that should be used as env file. During transfer to ECS, it will be stored in /var/app/current/myenv from where it could be used.

I am unsure if such an option exists in the Dockerrun format at all.

If not, theoretically it would be possible to transform each of the defined variables to

"containerDefinitions": [
    {
      "name": "php-app",
      "image": "php:fpm",
      "environment": [
        {
          "name": "var1",
          "value": "val1"
        },
        {
          "name": "var2",
          "value": "val2"
        }, ...
      ],

see the docs here

I would be very grateful for your help

can't run container-transform (KeyError: 'M')

Hi, I am trying to use container-transform to convert a working docker-compose file to ecs format and I am getting

cat docker-compose.yml | docker run --rm -i micahhausler/container-transform

Traceback (most recent call last):
File "/usr/local/bin/container-transform", line 9, in
load_entry_point('container-transform==1.1.4', 'console_scripts', 'container-transform')()
File "/usr/local/lib/python3.5/site-packages/click-6.6-py3.5.egg/click/core.py", line 716, in call
return self.main(_args, *_kwargs)
File "/usr/local/lib/python3.5/site-packages/click-6.6-py3.5.egg/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.5/site-packages/click-6.6-py3.5.egg/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, *_ctx.params)
File "/usr/local/lib/python3.5/site-packages/click-6.6-py3.5.egg/click/core.py", line 534, in invoke
return callback(_args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/container_transform-1.1.4-py3.5.egg/container_transform/client.py", line 64, in transform
output = converter.convert(verbose)
File "/usr/local/lib/python3.5/site-packages/container_transform-1.1.4-py3.5.egg/container_transform/converter.py", line 57, in convert
output_transformer
File "/usr/local/lib/python3.5/site-packages/container_transform-1.1.4-py3.5.egg/container_transform/converter.py", line 100, in _convert_container
output[output_name] = emit_func(ingest_func(container.get(input_name)))
File "/usr/local/lib/python3.5/site-packages/container_transform-1.1.4-py3.5.egg/container_transform/compose.py", line 192, in ingest_memory
return bit_shift[unit]['func'](number, bit_shift[unit]['shift'])
KeyError: 'M'

any ideas?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.