merll / docker-map Goto Github PK

View Code? Open in Web Editor NEW

74.0 74.0 12.0 1.43 MB

Managing Docker images, containers, and their dependencies in Python.

License: MIT License

Python 100.00%

docker docker-configurations docker-containers dockerfile python

docker-map's People

Contributors

Stargazers

Watchers

Forkers

scast pahaz zalan-axis vijayshan pombredanne caputomarcos 1600 ronozoro e-j kaiszuttor vincentwei2021

docker-map's Issues

Please provide imports necessary in docs

It took me 15 minutes to figure out which parts of dockermap to import to re-create your examples in the introduction. Could you add the correct import statements so that getting started is easier?

CLI: multiple values for keyword argument 'cmd' when using 'exec_commands'

I'm trying to run a build script using exec_commands. I'm getting the following error:

Traceback (most recent call last):
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/main.py", line 756, in main
    *args, **kwargs
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 426, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 173, in run
    return self.wrapped(*args, **kwargs)
  File "/data/apex-server/build_deploy/src/fabfile.py", line 612, in app_setup
    apex.workflow_add(config=config, tag=tag)
  File "/data/apex-server/build_deploy/src/helpers/apex.py", line 66, in workflow_add
    workflow_containers_add(config, tag)
  File "/data/apex-server/build_deploy/src/helpers/apex.py", line 402, in workflow_containers_add
    docker.startup_required_containers(docker_map)
  File "/data/apex-server/build_deploy/src/helpers/docker.py", line 125, in startup_required_containers
    container_fabric_inst.startup(container)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/client.py", line 224, in startup
    return self.run_actions('startup', container, instances=instances, map_name=map_name, **kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/client.py", line 110, in run_actions
    results.extend(runner.run_actions(*actions))
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/runner/__init__.py", line 94, in run_actions
    res = c_method(action_config, container_name, **action.extra_data)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/runner/cmd.py", line 54, in exec_container_commands
    self.exec_commands(config, c_name, run_cmds=config_cmds)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/runner/cmd.py", line 37, in exec_commands
    e_id = client.exec_create(**ec_kwargs)['Id']
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockerfabric/cli.py", line 122, in exec_create
    cmd_str = self._out.get_cmd('exec_create', *args, **kwargs)
TypeError: get_cmd() got multiple values for keyword argument 'cmd'

I've seen something similar before and assumed it was something wrong with my legacy code. This time I seem to be using 100% internal functionality.

Option to run docker commands using sudo

The discussion started on docker-fabric. TL;DR

This article argues that sudo docker (with a very restrictive sudoers line and an alias to docker) is a better security practice than adding users to the docker group.
This approach also makes it easy to manage docker execution privileges from LDAP since sudoers supports LDAP groups.
@merll indicated that the key constraint is the way docker-map interacts with docker -- using the docker-py API client through socat. docker-map would need to issue shell commands instead.

Since these commands would need to be issued by docker-map, I'm moving the ticket here.

python 3.x support

Traceback (most recent call last):
  File "test_docker.py", line 3, in <module>
    from dockermap.build.dockerfile import DockerFile
  File "/venv/lib/python3.2/site-packages/dockermap/build/dockerfile.py", line 10, in <module>
    from .buffer import DockerStringBuffer
  File "/venv/lib/python3.2/site-packages/dockermap/build/buffer.py", line 5, in <module>
    import cStringIO
ImportError: No module named cStringIO

CLI Response Issues

Unlike the API, the sudo CLI sometimes returns spurious data. For example, my LDAP password is about to expire so I'm actually seeing:

[172.24.6.11] sudo: docker version --format="{{json .}}"
[172.24.6.11] out: sudo password:
[172.24.6.11] out: Your password will expire in 4 day(s).
[172.24.6.11] out: {"Client":{"Platform":{"Name":""}, ...

Presumably, other notifications could be sent this way as well. Perhaps all CLI parsing code (e.g. dockermap/client/cli.py", line 347, in parse_version_output) should be wrapped with a decorator that catches errors and writes the full output to the terminal/logging. I only discovered the issue by manually suppressing the quiet kwarg on the docker version call. If I wasn't familiar with docker-map, this would have been an extremely difficult issue to diagnose.

Validate Names

Due to a change in docker, container names can no longer include an underscore. A legacy name with an underscore leads to an invalid hostname (merll/docker-fabric#14) error that is hard to trace down. I suggest an assertion in the constructor for maps/containers (and/or their configs) that validates names and raises an informative exception.

CLI: exec_commands issue

Assume this is another issue related to #16

...
  File "/data/apex-server/build_deploy/src/helpers/docker.py", line 125, in startup_required_containers
    container_fabric_inst.startup(container)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/client.py", line 224, in startup
    return self.run_actions('startup', container, instances=instances, map_name=map_name, **kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/client.py", line 110, in run_actions
    results.extend(runner.run_actions(*actions))
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/runner/__init__.py", line 94, in run_actions
    res = c_method(action_config, container_name, **action.extra_data)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/runner/cmd.py", line 54, in exec_container_commands
    self.exec_commands(config, c_name, run_cmds=config_cmds)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/runner/cmd.py", line 37, in exec_commands
    e_id = client.exec_create(**ec_kwargs)['Id']
TypeError: 'NoneType' object has no attribute '__getitem__'

... and for reference, this is how we're calling it:

    # because we don't add users to the docker group, we need to use_sudo
    docker_client = docker_fabric(use_sudo=True)
    # we inject the map directly rather than use env
    container_fabric_inst = container_fabric(docker_client=docker_client, container_maps=docker_map)
    for container, config in docker_map.containers:
        container_fabric_inst.startup(container)

Errors when using a client not based on docker-py

Problem occurs in Docker-Fabric, but is caused by a check if the client is a subclass of docker.client.Client. This check should probably be replaced with something more flexible in order to support non-API clients such as the CLI.

Hyphen in container names

Got confused and thought I had reported it in a different issue when it was actually new. I ran into a linking error and it turns out that it was related to #15. I changed the container name to eliminate a hyphen the problem resolved itself. If you go back to the same thread referenced in #15, it looks like there's a toxic interaction between EVN variables (no hyphen) and network names (no underscore) that's working its way through the docker consciousness.

I'm not sure if there's a local fix -- except raise a useful exception -- until they get the conflict sorted out internally.

Docker Networking Support

According to the documentation on links, the feature has been deprecated in favor of a new networking option. Have you looked at this new feature to determine whether (and how) it's supported by docker-map (and docker-fabric)?

No way to set environment variables on the container

container configuration like

'db': { # Configure container creation and startup
'image': 'postgres:9.4.4',
'exposes': {
5432: 5432
},
'binds': ['dbdata', "dbinit"],
'Env': {
"PGDATA": "/dbdata"
}
},

would need the PGDATA environment variable set in the container that is started but after some debugging i realized that the vvariable was being completely ignored. i tried changing it to environment too but that was being ignored too.

Wait timeout for starting containers

In some cases containers fail to start, if they link to containers that are still in the process of starting up. One occurrence is described in a Docker-Fabric issue. An optional startup delay should be added to the container configuration.

swallows errors during build_from_file

For example

(Pdb) 
> /Users/dylanjay/Projects/download-cache/eggs/docker_map-0.3.0-py2.7.egg/dockermap/map/base.py(59)_docker_log_stream()
-> for e in response:
(Pdb) output
{u'errorDetail': {u'message': u'Cannot create container with more than 127 parents'}, u'error': u'Cannot create container with more than 127 parents'}
(Pdb)

CLI args formating

This is a (sanitized, and wrapped for readability) create command generated by client.cli.get_cmd:

 docker create 
--name="graylog_map.graylog" 
--links="(u'graylog_map.mongo', 'mongo')" 
--links="(u'graylog_map.elasticsearch', 'elasticsearch')" 
--hostname="graylog_map.graylog" 
--env="GRAYLOG_PASSWORD_SECRET=<pepper>" 
--env="GRAYLOG_ROOT_PASSWORD_SHA2=<password>" 
--env="GRAYLOG_REST_TRANSPORT_URI=https://<url>:12900" 
--volume="/data/graylog/data:/usr/share/graylog/data/journal:rw" 
--volume="/data/graylog/config:/usr/share/graylog/data/config:rw" 
graylog2/server:2.0.3-2

Here are the issues (and resolutions) I've identified so far:

links is actually link
- in client.cli.KWARGS_MAP add 'links': 'link'
the value for link is not collapsed correctly and should be <name>:<name>
- The code block below resolves a couple issues: recursive calls for e.g. a list of tuples and cases where the kw (i.e. the outer flags) are None

def _transform_kwargs(ka):
    if ka is None:
        yield
    for key, value in iteritems(ka):
        cmd_arg = KWARG_MAP.get(key, key.replace('_', '-'))
        if value is None:
            continue
        for response in _transform_kwarg(cmd_arg, value):
            yield response


def _transform_kwarg(cmd_arg, value):
    if isinstance(value, list):
        for vi in value:
            for response in _transform_kwarg(cmd_arg, vi):
                yield response
    elif isinstance(value, dict):
        for ki, vi in iteritems(value):
            yield _mapping_format(cmd_arg, ki, vi)
    elif isinstance(value, tuple):
        yield _mapping_format(cmd_arg, value[0], value[1])
    elif isinstance(value, bool):
        yield _arg_format(cmd_arg, 'true' if value else 'false')
    else:
        yield _quoted_arg_format(cmd_arg, value)

graylog_map.graylog is not a valid hostname. As mentioned here valid hostnames need to be sanitized to [a-zA-Z0-9\-]+ or left out. Replacing non-alphanumeric characters with hyphens could be a valid option. For my testing, I used a code block to skip it.