- What is it?
- Install it
- Run it
- What you can do with it
- How to write DockerMake.yml
- Example
- Command line usage
A command line tool to build and manage stacks of docker images. You can mix and match different sets of build instructions as a dependency graph to create maintainable and extensible stacks of docker images.
Requires Docker and Python 2.7 or 3.5+.
pip install DockerMake
This will install the command line tool, docker-make, and its supporting python package, which you can import as import dockermake.
To build some illustrative examples, try running the example in this repository:
git clone https://github.com/avirshup/DockerMake
cd DockerMake/example
docker-make --list
docker-make finalThe punchline: define small pieces of configuration or functionality, then mix them together into production container images.
A DockerMake.yml file contains discrete build "steps". These steps can depend on each other, forming a dependency graph. DockerMake solves the dependency graph and drives building of each image as necessary. This makes it easy to keep your images up-to-date while still taking advantage of docker's shared fileystem layers and build cache.
- new: beta support for dockerfile build arguments
- new: specify custom
.dockerignorefiles for any given build step - Automated registry login and image pushes
- squash arbitrary parts of your build (using
squash: truein a step definition) without busting the cache - Designate
secret_filesto erase them from intermediate layers (In the step definition, usesecret_files: ['path1', 'path2', ...])
WARNING: these features are in alpha - use with extreme caution
- Create builds that ADD or COPY files from anywhere on your file system
- Build artifacts in one image, then copy them into smaller images for deployment
- Invalidate docker's build cache at a specific step in the build using
--bust-cache [stepname] - new: Use specific images to resolve docker's build cache (using
--cache-repo [repo]and/or--cache-tag [tag]) - Force a clean rebuild without using the cache (using
--no-cache)
DockerMake.yml lets you split your image build up into discrete, easy to manage steps that can be mixed together (as a dependency graph) to produce your desired container image.
The DockerMake.yml file is a YAML-formatted file that lists build steps. To create an extremely basic image, define a step with a base image and a series of dockerfile commands:
FIRST_STEP_NAME:
FROM: BASE_IMAGE_NAME
build: |
RUN [something]
ADD [something else]
[Dockerfile commands go here]Use the requires field to define additional steps that extend the first one:
FIRST_STEP_NAME:
[...]
NEXT_STEP_NAME:
requires:
- FIRST_STEP_NAME
build: |
[additional Dockerfile instructions]Image definitions can include any of the following fields:
FROM/FROM_DOCKERFILEbuildrequiresbuild_directoryignore/ignorefiledescriptioncopy_fromsquashsecret_files
The docker image to use as a base for this image (and those that require it). This can be either the name of an image (using FROM) or the path to a local Dockerfile (using FROM_DOCKERFILE).
Example:
baseimage:
FROM: python:3.6-slimor
baseimage:
FROM_DOCKERFILE: ../myproject/DockerfileMulti-line string defining dockerfile commands to build this step. Note that these commands CANNOT contain 'FROM'. See also Notes on multi-line strings below.
Example:
build-image:
requires:
- baseimage
build: |
RUN apt-get update \
&& apt-get install -y gcc vi
ENV editor=viList of other image definitions to include in this one. docker-make will create a new image from a single DockerFile that includes an amalgamation of all image definitions.
Example:
my-tools:
build: |
RUN pip install numpy jupyter pandas
[...]
data-sci-environment:
requires:
- baseimage
- my-toolsPath to a directory on your filesystem. This will be used to locate files for ADD and COPY commands in your dockerfile. See Notes on relative paths below.
Example:
data-image:
build_directory: ./datafiles
build: |
COPY data /opt/data
[...]A custom .dockerignore for this step. This overrides any existing .dockerignore file in the build context. Only relevant for ADD or COPY commands when the build_directory is specified. This can either be a multi-line string (using the ignore field) or the path to a file (using the ignorefile field).
Example:
data-image:
build_directory: ./datafiles
build: |
ADD [...]
ignore: |
*.pyc
*~
*.tmpAn arbitrary comment (ignored by docker-make)
Used to copy files into this image from other images (to copy from your filesystem or a URL, use the standard ADD and COPY dockerfile commands). This is a mapping of mappings of the form:
[image-name]:
[...]
copy_from:
[source_image1]:
[source path 1]: [dest path 1]
[source path 2]: [dest path 2]
[source image2]:
[...]Note that, for historical reasons, these copies are performed after any build instructions are executed.
NOTE: this feature requires that your docker daemon's experimental features be enabled.
Used to squash all layers produced in a given step. This can be helfpul both for keeping image sizes low, especially when it's necessary to add a lot of data via the ADD or COPY dockerfile commands.
Note that setting squash: True for a step only squashes the layers generated by that step. All layers in the base image are left intact.
Additionally, unlike the vanilla docker build --squash command, downstream image builds can use the squashed image in their cache, so that squashing doesn't force you to repeatedly re-run the same downstream build steps.
Example: In this example, we create a huge file in the image, do something with it, then erase it.
count-a-big-file:
FROM: alpine
build: |
RUN dd if=/dev/zero of=/root/bigfile count=16384 bs=1024
RUN wc /root/bigfile > /root/numbiglines
RUN rm /root/bigfileLet's build it and check the size:
$ docker-make count-a-big-file
[...]
docker-make finished.
Built:
* count-a-big-file
$ docker images count-a-big-file
REPOSITORY ... SIZE
count-a-big-file ... 20.9MBBut, take them same definition and add a squash: true to it:
count-a-big-file:
FROM: alpine
squash: true
build: |
RUN dd if=/dev/zero of=/root/bigfile count=16384 bs=1024
RUN wc /root/bigfile > /root/numbiglines
RUN rm /root/bigfileAnd we find that the deleted file is no longer taking up space:
$ docker-make count-a-big-file
[...]
docker-make finished.
Built:
* count-a-big-file
$ docker images count-a-big-file
REPOSITORY ... SIZE
count-a-big-file ... 4.15MBRead these caveats first
- This is an alpha-stage feature. DO NOT rely on it as a security tool. You must carefully verify that your final image, and all its layers AND its history, are free of sensitive information before deploying or publishing them.
- It relies on experimental docker daemon features.
- Although your final image won't contain your secrets, they will be present in intermediate images on your build machine. Your secrets will be exposed to all
dockerusers on your build machine. - When you define
secret_filesfor a step, it only erases files that are added in thebuilddefinition for that step. Files added in other steps will remain exposed in your image's layers.
Background
It's often necessary to perform some form of authentication during a build - for instance, you might need to clone a private git repository or download dependencies from a private server. However, it's quite challenging to do so without leaving your credentials inside a layer of the final docker image or its history.
Files added or created in a given step can be designated as secret_files in DockerMake.yml. These files will be automatically erased at the end of the step, and the step's layers will be squashed to keep the files out of the history.
Example
my-secret-steps:
FROM: python:3.6
build: |
ADD my-credentials /opt/credentials
RUN some-process --credentials /opt/credentials
secret_files:
- /opt/credentialsYou can include step definitions from other DockerMake.yml files by listing them in the _SOURCES_. For example:
_SOURCES_:
- ~/mydefinitions/DockerMake.yml
- ./other/file.yml
[...]Please note that relative file paths in each file are always interpreted relative to the directory containing that file.
By default, running docker-make --all will build all well-defined images defined in a file (and any files included via _SOURCES_). Images without a FROM or FROM_DOCKERFILE field in any of their requirements will be ignored.
Alternatively, you can use the _ALL_ field to designate specific images to build. For example, in the following definition, docker-make --all will only build imgone and imgtwo, not baseimage:
_ALL_:
- imgone
- imgtwo
baseimage:
FROM: [...]
[...]
imgone: [...]
imgtwo: [...]Note that the _ALL_ fields from any files included via _SOURCES_ are ignored.
Relative paths: Several of these fields include paths on your local filesystem. They may be absolute or relative; relative paths are resolved relative to the DockerMake.yml file they appear in. Use of ~ is allowed to denote the home directory.
Multiline strings: You'll usually want to express the build and ignore fields as multiline strings. To do so, use the following YML "literal block scalar" style, as in all examples above.
field-name: |
[line 1]
[line 2]
[...]
next field: [...](See also this production example)
This example builds a single docker image called data_science. It does this by mixing together three components: devbase (the base image), airline_data (a big CSV file), and python_image (a python installation). docker-make will create an image that combines all of these components.
Here's the DockerMake.yml file:
devbase:
FROM: phusion/baseimage
build: |
RUN apt-get -y update && apt-get -y install
build-essential
&& mkdir -p /opt
airline_data:
build_directory: sample_data/airline_data
build: |
ADD AirPassengers.csv /data
plant_data:
build_directory: sample_data/plant_growth
build: |
ADD Puromycin.csv /data
python_image:
requires:
- devbase
build: |
RUN apt-get install -y python python-pandas
data_science:
requires:
- python_image
- airline_data
- plant_data
To build an image called alice/data_science, you can run:
docker-make data_science --repository alicewhich will create an image with all the commands in python_image and airline_data.
This works by dynamically generating a new Dockerfile every time you ask to build something. However, most of the commands will be cached, especially if you have a large hierarchy of base images. This actually leads to less rebuilding than if you had a series of Dockerfiles linked together with FROM commands.
Here's the dependency graph and generated Dockerfiles:
usage: docker-make [-h] [-f MAKEFILE] [-a] [-l] [--build-arg BUILD_ARG]
[--requires [REQUIRES [REQUIRES ...]]] [--name NAME] [-p]
[-n] [--dockerfile-dir DOCKERFILE_DIR] [--pull]
[--cache-repo CACHE_REPO] [--cache-tag CACHE_TAG]
[--no-cache] [--bust-cache BUST_CACHE] [--clear-copy-cache]
[--keep-build-tags] [--repository REPOSITORY] [--tag TAG]
[--push-to-registry] [--registry-user REGISTRY_USER]
[--registry-token REGISTRY_TOKEN] [--version] [--help-yaml]
[--debug]
[TARGETS [TARGETS ...]]
NOTE: Docker environmental variables must be set. For a docker-machine, run
`eval $(docker-machine env [machine-name])`
optional arguments:
-h, --help show this help message and exit
Choosing what to build:
TARGETS Docker images to build as specified in the YAML file
-f MAKEFILE, --makefile MAKEFILE
YAML file containing build instructions
-a, --all Print or build all images (or those specified by
_ALL_)
-l, --list List all available targets in the file, then exit.
--build-arg BUILD_ARG
Set build-time variables (used the same way as docker
build --build-arg), e.g., `... --build-arg VAR1=val1
--build-arg VAR2=val2`
--requires [REQUIRES [REQUIRES ...]]
Build a special image from these requirements.
Requires --name
--name NAME Name for custom docker images (requires --requires)
Dockerfiles:
-p, --print-dockerfiles, --print_dockerfiles
Print out the generated dockerfiles named
`Dockerfile.[image]`
-n, --no_build Only print Dockerfiles, don't build them. Implies
--print.
--dockerfile-dir DOCKERFILE_DIR
Directory to save dockerfiles in (default:
./docker_makefiles)
Image caching:
--pull Always try to pull updated FROM images
--cache-repo CACHE_REPO
Repository to use for cached images. This allows you
to invoke the `docker build --build-from` option for
each image.For instance, running `docker-make foo bar
--cache-repo docker.io/cache` will use
docker.io/cache/foo as a cache for `foo` and
docker.io/cache/bar as a cachefor `bar`.
--cache-tag CACHE_TAG
Tag to use for cached images; can be used with the
--cache-repo option (see above).
--no-cache Rebuild every layer
--bust-cache BUST_CACHE
Force docker to rebuilt all layers in this image. You
can bust multiple image layers by passing --bust-cache
multiple times.
--clear-copy-cache, --clear-cache
Remove docker-make's cache of files for `copy-from`.
--keep-build-tags Don't untag intermediate build containers when build
is complete
Repositories and tags:
--repository REPOSITORY, -r REPOSITORY, -u REPOSITORY
Prepend this repository to all built images, e.g.
`docker-make hello-world -u quay.io/elvis` will tag
the image as `quay.io/elvis/hello-world`. You can add
a ':' to the end to image names into tags: `docker-
make -u quay.io/elvis/repo: hello-world` will create
the image in the elvis repository:
quay.io/elvis/repo:hello-world
--tag TAG, -t TAG Tag all built images with this tag. If image names are
ALREADY tags (i.e., your repo name ends in a ":"),
this will append the tag name with a dash. For
example: `docker-make hello-world -u elvis/repo: -t
1.0` will create the image "elvis/repo:hello-world-1.0
--push-to-registry, -P
Push all built images to the repository specified
(only if image repository contains a URL) -- to push
to dockerhub.com, use index.docker.io as the registry)
--registry-user REGISTRY_USER, --user REGISTRY_USER
For pushes: log into the registry using this username
--registry-token REGISTRY_TOKEN, --token REGISTRY_TOKEN
Token or password to log into registry (optional; uses
$HOME/.dockercfg or $HOME/.docker/config.json if not
passed)
Help:
--version Print version and exit.
--help-yaml Print summary of YAML file format and exit.
--debug
Copyright (c) 2015-2017, Autodesk Inc. Copyright (c) 2017-2018, Docker-Make contributors. Released under the Apache 2.0 License.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent