docker devops workflow

Efficient development with Docker and docker-compose

Lars de Ridder · 9 November 2018

We are going to set up a development environment for a project consisting of various services. All of these services will be containerized with Docker, and they will all run simultaneously during development using docker-compose.

Our environment will feature instant code reloading, test-driven development, database connectivity, dependency management, and more. It will be possible to easily deploy it to production with docker-compose or with Rancher. And as a bonus, we’ll set up continuous integration on Gitlab.

The article is about efficiency, so I’ll get straight to the point.

The goals

We want to:

Type code and observe changes as soon as possible in our Docker containers, preferably without manual actions;
Have a local environment that is representative of the actual deployment environment;
Support a number of workflows.

Let’s make it happen.

The prerequisites

You are going to need to install the following tools:

docker (CE is fine)
docker-compose

The premise

We’ll set up a project consisting of a Python and Java service, together with a Postgres database. The Postgres database will run on our own machine in the development environment, but is assumed to be external during production (it might use Amazon RDS for example).

The Python service contains unit tests supported by Pytest, for which we will set up test-driven development. The Java service uses Maven for its build process.

Finally, we will use Gitlab’s container registry and Gitlab’s CI service. The code as described below is also available in a Github or Gitlab repository.

This setup should demonstrate most essential concepts. However, the approach described below should work regardless of technology.

The setup

The file structure:

|/myproject
| /python
|  /mypackage
|   run.py
|  /tests
|   my test.py
|  Dockerfile
|  setup.py
|  requirements.txt
|
| /java
|  Dockerfile
|  pom.xml
|  /src
|   /main
|    /java
|     /com
|      /example
|       /Main.java
|
| docker-compose.common.yml
| docker-compose.dev.yml
| docker-compose.prod.yml
| Makefile
| python-tests.sh
| .gitlab-ci.yml

The Dockerfile for the Python service is as follows:

FROM python:3.6-slim

COPY . /code
WORKDIR /code

RUN pip install --no-cache-dir -r requirements.txt
RUN pip install -e .

ENTRYPOINT python ./mypackage/run.py

This adds the service code to the container, installs its dependencies (contained in the requirements.txt, which in this example will contain pytest and watchdog), and installs the Python service itself. It also defines the command to be executed when the container is started.

The Dockerfile for the Java service can be found below:

FROM maven:3.5-jdk-8

COPY . /usr/src/app
WORKDIR /usr/src/app

RUN apt-get update && apt-get install entr -y

RUN mvn clean package --batch-mode
ENTRYPOINT java -jar target/docker-compose-java-example-1.0-SNAPSHOT.jar

Like the Python Dockerfile, this also first adds the code to the container. It then proceeds to install the Unix utility entr which we will need later. Maven is used to create a JAR file, after which we define the container command to execute the JAR file.

Finally, the docker-compose.common.yml file forms the basis for our environment, and contains all configuration that is important for the application, regardless of environment in which it is being executed:

version: '2'

services:
  python:
    build: ./python
    environment:
      - POSTGRES_USER
      - POSTGRES_PASSWORD
      - POSTGRES_DB
      - POSTGRES_HOST

  java:
    build: ./java
    environment:
      - POSTGRES_USER
      - POSTGRES_PASSWORD
      - POSTGRES_DB
      - POSTGRES_HOST

The development configuration

Let’s have a look at the docker-compose.dev.yml file:

version: '2'

services:
  python:
    image: registry.gitlab.com/mycompany/myproject/python:dev
    volumes:
      - ./python/:/code
    entrypoint: watchmedo auto-restart --recursive --pattern="*.py" --directory="." python mypackage/run.py
    depends_on:
      - postgres
    links:
      - postgres
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=myproject
      - POSTGRES_HOST=postgres

  python-tests:
    image: registry.gitlab.com/mycompany/myproject/python:dev
    volumes:
      - ./python/:/code
    entrypoint: watchmedo auto-restart --recursive --pattern="*.py" --directory="." pytest
    depends_on:
      - python

  java:
    image: registry.gitlab.com/mycompany/myproject/java:dev
    volumes:
      - ./java/:/usr/src/app
    entrypoint: sh -c 'find src/ | entr mvn clean compile exec:java --batch-mode --quiet'
    depends_on:
      - postgres
    links:
      - postgres
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=myproject
      - POSTGRES_HOST=postgres

  postgres:
    image: postgres:9.6
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=myproject
    volumes:
      - /data/aedspy/postgres:/var/lib/postgresql/data

  pgadminer:
    image: clue/adminer
    ports:
      - "99:80"

The development configuration - Python

The key part for Python is the volume mapping and entrypoint:

volumes:
  - ./python/:/code

Without the volumes statement, the contents of the python subdirectory would simply be added to the container. If a change is made on the host machine, the container has to be rebuilt before we can see those changes. With the volumes statement, any changes you make will immediately be reflected inside the container.

The entrypoint then uses this:

entrypoint: watchmedo auto-restart --recursive --pattern="*.py" --directory="." python mypackage/run.py

The watchmedo command is part of the watchdog package. It monitors files with the provided pattern (all *.py files) in a given directory, and if any of them is modified, restarts the running process.

This means every modification of any Python file in the ./python directory on our host machine will restart our application. If you open any Python file and modify it, you will see that every change you make will immediately be reflected in the running container.

The development configuration - Python unit tests and TDD

The python-tests service uses the same image as the Python service, but runs pytest on every file change:

entrypoint: watchmedo auto-restart --recursive --pattern="*.py" --directory="." pytest

The result: every code change automatically executes all tests, giving you instant feedback on the status of your tests. This makes test-driven development with Docker trivial.

The development configuration - Java

Java, being a compiled language, is a little more complicated. The entr utility (installed in the Dockerfile) handles watching:

entrypoint: sh -c 'find src/ | entr mvn clean compile exec:java --batch-mode --quiet'

This says: “whenever any file in the directory src/ changes, ask Maven to clean, compile and then execute the Java project.” Combined with the volume mapping, every change to a Java source file restarts the application, compiles the files, installs new dependencies, and restarts the service.

Building and executing

First, build all containers for development:

docker-compose -f docker-compose.common.yml -f docker-compose.dev.yml build

And to start all services:

docker-compose -f docker-compose.common.yml -f docker-compose.dev.yml up

As this is a lot to type, I tend to define these in a Makefile:

dev-build:
	docker-compose -f docker-compose.common.yml -f docker-compose.dev.yml build --no-cache

dev:
	docker-compose -f docker-compose.common.yml -f docker-compose.dev.yml up

The production configuration

The docker-compose.prod.yml is minimal:

version: '2'

services:
  python:
    image: $IMAGE/python:$TAG
    restart: always

  java:
    image: $IMAGE/java:$TAG
    restart: always

Most of the configuration lives in docker-compose.common.yml, and the commands are all in the Dockerfiles. You need to pass in the environment variables that have no value yet from an external source, which should be handled by your build script or secrets management toolchain.

Gitlab CI

The .gitlab-ci.yml to build and test:

stages:
  - build
  - test

variables:
  TAG: $CI_BUILD_REF
  IMAGE: $CI_REGISTRY_IMAGE

services:
  - docker:dind

image: docker

before_script:
  - apk add --update py-pip
  - pip install docker-compose
  - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY

build:
  stage: build
  script:
    - docker-compose -f docker-compose.common.yml -f docker-compose.prod.yml build
    - docker-compose -f docker-compose.common.yml -f docker-compose.prod.yml push

test-python:
  stage: test
  script:
    - docker-compose -f docker-compose.common.yml -f docker-compose.prod.yml pull python
    - docker-compose -f docker-compose.common.yml -f docker-compose.prod.yml run --rm --entrypoint pytest python

Wrapping up

So there you have it; an efficient but not very complicated setup to orchestrate and develop most projects inside Docker.

If you’re interested, I’ve also written an article on setting up simple deployment with docker-compose on Gitlab. That should take you from a development environment that builds on Gitlab CI right to continuous deployment with docker-compose.