Usage

docker-compose lets you define multiple services in a YAML file, then build and start them together. It's particularly useful for development or running tests in a CI/CD environment which depend on a database.

# Registry for mirrored source images, default is Docker Hub if not set
export REGISTRY=123456789.dkr.ecr.us-east-1.amazonaws.com/

# Destination repository for app final image
export REPO_URL=${REGISTRY}foo/app

# Login to registry, needed to push to repo or use mirrored base images
# Docker Hub
# docker login --username cogini --password <access-token>
# AWS ECR
aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $REGISTRY

# Build all images (dev, test and app prod, local Postgres db)
docker compose build

# Run tests, talking to db in container
docker compose up test
docker compose run test mix test

# Push final app image to repo REPO_URL
docker compose push app

You can also run the docker build commands directly, which give more control over caching and cross builds. build.sh is a wrapper on docker buildx build which sets various options.

DOCKERFILE=deploy/debian.Dockerfile ecs/build.sh

To run the prod app locally, talking to the db container:

# Create prod db schema via test stage
DATABASE_DB=app docker compose run test mix ecto.create

export SECRET_KEY_BASE="JBGplDAEnheX84quhVw2xvqWMFGDdn0v4Ye/GR649KH2+8ezr0fAeQ3kNbtbrY4U"
DATABASE_DB=app docker compose up app

# Make request to app running in Docker
curl -v http://localhost:4000/

To develop the app in a container:

# Start dev instance
docker compose up dev

# Create dev db schema by running mix
docker compose run dev mix ecto.create

# Make request to app running in Docker
curl -v http://localhost:4000/

# Open a shell on the running dev environment
docker compose run dev bash

Building for multiple platforms

Building in emulation is considerably slower, mainly due to lack of precompiled packages for Arm. The key in any case is getting caching optimized.

PLATFORM="--platform linux/amd64,linux/arm64" ecs/build.sh
docker buildx imagetools inspect $REPO_URL

It can also be configured in docker compose.yml.

Environment vars

The new BuildKit features are enabled with environment vars:

DOCKER_BUILDKIT=1 enables the new experimental Dockerfile caching syntax with the standard docker build command. It requires Docker version 18.09.

DOCKER_CLI_EXPERIMENTAL=enabled enables the new Docker buildx CLI command (and the new file syntax). It is built in with Docker version 19.03, but can be installed manually before that.

COMPOSE_DOCKER_CLI_BUILD=1 tells docker-compose to use buildx.

The prod deploy container uses the following env vars:

• DATABASE_URL defines the db connection, e.g. DATABASE_URL=ecto://user:pass@host/database You can configure the number of db connections in the pool, e.g. POOL_SIZE=10.

• SECRET_KEY_BASE protects Phoenix cookies from tampering. Generate it with the command mix phx.gen.secret.

• PORT defines the port that Phoenix will listen on, default is 4000.

Mirroring source images

You can make a mirror of the base images that your build depends on to your own registry. This is particularly useful since Docker started rate limiting requests to public images.

See https://earthly.dev/blog/how-to-setup-and-use-amazons-elastic-container-registry/

To use the mirror registry, set the REGISTRY env variable:

export REGISTRY=1234567890.dkr.ecr.ap-northeast-1.amazonaws.com/
docker compose build

skopeo

docker.io:
  images:
    earthly/buildkitd:
      - 'v0.6.2'

    earthly/dind:
      - 'alpine'

    ubuntu:
      # CodeBuild base image
      - 'focal'

    # Target base image, choose one
    alpine:
      - '3.12.1'
      - '3.13.2'
      - '3.15.0'

    debian:
      - 'buster-slim'

    postgres:
      - '12'
      - '14'
      - '14.1-alpine'

    mysql:
      - '8'

    # Build base images
    hexpm/elixir:
      # Choose one
      # - '1.11.2-erlang-23.1.2-alpine-3.12.1'
      # - '1.11.2-erlang-23.1.2-debian-buster-20201012'
      - '1.11.2-erlang-23.2.1-alpine-3.12.1'
      - '1.11.2-erlang-23.2.1-debian-buster-20201012'
      - '1.11.2-erlang-23.2.4-alpine-3.13.1'
      - '1.11.2-erlang-23.2.4-debian-buster-20201012'
      - '1.11.3-erlang-23.2.6-alpine-3.13.2'
      - '1.11.3-erlang-23.2.6-debian-buster-20210208'
      - '1.13.1-erlang-24.2-alpine-3.15.0'

    node:
      - '14.4-stretch'
      - '14.15.1-stretch'
      - '14.4-buster'
      - '14.15.1-buster'

  mcr.microsoft.com:
    mssql/server:
      - '2019-latest'

aws ecr create-repository --repository-name centos
aws ecr get-login-password | skopeo login -u AWS --password-stdin $REGISTRY_NOSLASH
skopeo sync --all --src yaml --dest docker skopeo-sync.yml $REGISTRY

https://polyverse.com/blog/skopeo-the-best-container-tool-you-need-to-know-about/ https://github.com/onfido/ecr-mirror https://alexwlchan.net/2020/11/copying-images-from-docker-hub-to-amazon-ecr/

https://shazi.info/docker-build-x86-arm-multi-arch-images-with-aws-ecr/

Dregsy is a utility which mirrors repositories from one registry to another.

NOTE: dregsy does not currently support multi-arch images https://github.com/xelalexv/dregsy/issues/43

dregsy.yml

relay: skopeo

# relay config sections
skopeo:
  # path to the skopeo binary; defaults to 'skopeo', in which case it needs to
  # be in PATH
  binary: skopeo

# list of sync tasks
tasks:
  - name: docker

    # interval in seconds at which the task should be run; when omitted,
    # the task is only run once at start-up
    # interval: 60

    # determines whether for this task, more verbose output should be
    # produced; defaults to false when omitted
    verbose: true

    source:
      registry: docker.io
      # Authenticate with Docker Hub to get higher rate limits
      # echo '{"username":"cogini","password":"sekrit"}' | base64
      # auth: xxx
    target:
      registry: 1234567890.dkr.ecr.ap-northeast-1.amazonaws.com
      auth-refresh: 10h

    # 'mappings' is a list of 'from':'to' pairs that define mappings of image
    # paths in the source registry to paths in the destination; 'from' is
    # required, while 'to' can be dropped if the path should remain the same as
    # 'from'. Additionally, the tags being synced for a mapping can be limited
    # by providing a 'tags' list. When omitted, all image tags are synced.
    # mappings:
    #   - from: test/image
    #     to: archive/test/image
    #     tags: ['0.1.0', '0.1.1']
    mappings:
      # - from: moby/buildkit
      #   tags: ['latest']

      # CodeBuild base image
      - from: ubuntu
        tags: ['bionic', 'focal']

      # Target base image, choose one
      - from: alpine
        tags:
          - '3.12.1'
          - '3.13.2'
          - '3.15.0'

      - from: debian
        tags: ['buster-slim']

      - from: postgres
        tags:
          - '12'
          - '14.1-alpine'

      # Build base images
      # - from: hexpm/erlang
      - from: hexpm/elixir
        tags:
          # Choose one
          - '1.11.3-erlang-23.2.6-alpine-3.13.2'
          - '1.11.3-erlang-23.2.6-debian-buster-20210208'
          - '1.13.1-erlang-24.2-alpine-3.15.0'
      - from: node
        tags:
          - '14.4-buster'
          - '14.15.1-buster'

  - name: microsoft
    verbose: true

    source:
      registry: mcr.microsoft.com

    target:
      registry: 770916339360.dkr.ecr.ap-northeast-1.amazonaws.com
      auth-refresh: 10h

    mappings:
      - from: mssql/server
        tags: ['2019-latest']

Run dregsy to sync docker images, using AWS credentials from your ~/.aws/config file specified by AWS_PROFILE environment var:

docker run --rm -v $(pwd)/dregsy.yml:/config.yaml -v $HOME/.aws:/root/.aws -e AWS_PROFILE xelalex/dregsy

or get AWS credentials from AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars:

docker run --rm -v $(pwd)/dregsy.yml:/config.yaml -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY xelalex/dregsy

Developing in a Docker container

Visual Studio Code has support for developing in a Docker container.

• https://code.visualstudio.com/docs/remote/containers-tutorial
• https://code.visualstudio.com/docs/remote/remote-overview
• https://code.visualstudio.com/docs/remote/containers
• https://code.visualstudio.com/docs/remote/devcontainerjson-reference
• https://code.visualstudio.com/docs/containers/docker-compose
• https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack

See .devcontainer/devcontainer.json.

It uses the docker compose.yml plus an .env file to set environment variables.

The default .env file is picked up from the root of the project, but you can use env_file in docker compose.yml file to specify an alternate location.

.env

IMAGE_NAME="foo-app"
IMAGE_OWNER="cogini"
IMAGE_TAG=latest

SECRET_KEY_BASE="JBGplDAEnheX84quhVw2xvqWMFGDdn0v4Ye/GR649KH2+8ezr0fAeQ3kNbtbrY4U"
DATABASE_URL=ecto://postgres:postgres@db/app

After the containe