👋 Hey there, I’m Dheeraj Choudhary an AI/ML educator, cloud enthusiast, and content creator on a mission to simplify tech for the world.
After years of building on YouTube and LinkedIn, I’ve finally launched TechInsight Neuron a no-fluff, insight-packed newsletter where I break down the latest in AI, Machine Learning, DevOps, and Cloud.
What to expect: actionable tutorials, tool breakdowns, industry trends, and career insights all crafted for engineers, builders, and the curious.
If you're someone who learns by doing and wants to stay ahead in the tech game you're in the right place.

Introduction

Managing a single container with docker run is straightforward. Managing three containers where a web application depends on a database and a cache, all wired together with the right networks, volumes, and environment variables, becomes a wall of commands you have to remember, execute in the right order, and share with your team somehow. It breaks down fast.

Docker Compose replaces that entire manual workflow with a single YAML file and a single command. You describe every service, volume, and network your application needs in a compose.yaml file, commit it to version control alongside your code, and anyone on your team can bring up the entire stack with docker compose up. Tear it all down with docker compose down. That's the core value proposition.

This guide covers how Compose works, how to write a complete compose file, the critical distinction between depends_on and actual readiness, how to handle environment variables and secrets safely, and how to use multiple Compose files for environment-specific configuration.

The Problem Compose Solves

Here's what it looks like to manually start a three-service application stack without Compose:

# Step 1: Create the network
docker network create app-net

# Step 2: Create the volume
docker volume create db-data

# Step 3: Start the database (must come first)
docker run -d \
  --name db \
  --network app-net \
  -v db-data:/var/lib/mysql \
  -e MYSQL_ROOT_PASSWORD=secret \
  -e MYSQL_DATABASE=myapp \
  mysql:8

# Step 4: Start the cache
docker run -d \
  --name cache \
  --network app-net \
  redis:7-alpine

# Step 5: Start the web app
docker run -d \
  --name web \
  --network app-net \
  -p 8080:3000 \
  -e DB_HOST=db \
  -e REDIS_HOST=cache \
  my-app:1.0

Five commands, manual ordering, all the configuration scattered across the terminal. Now imagine doing this after every code change, or handing it to a new teammate, or trying to reproduce it in CI.

The same stack in Compose:

services:
  db:
    image: mysql:8
    environment:
      MYSQL_ROOT_PASSWORD: secret
      MYSQL_DATABASE: myapp
    volumes:
      - db-data:/var/lib/mysql

  cache:
    image: redis:7-alpine

  web:
    image: my-app:1.0
    ports:
      - "8080:3000"
    environment:
      DB_HOST: db
      REDIS_HOST: cache

volumes:
  db-data:

One file. docker compose up. Done.

Compose v1 vs Compose v2: What You Need to Know

There are two versions of Docker Compose that behave differently, and it's worth being explicit about which one this guide covers.

  • Compose v1 was a standalone Python tool invoked as docker-compose (with a hyphen). It reached end of life in July 2023 and no longer receives updates.

  • Compose v2 is a Go-based plugin built directly into the Docker CLI, invoked as docker compose (with a space). It is the only version that receives updates and the one you should use for any new project. If you installed Docker Desktop recently, you already have Compose v2.

# v1 (deprecated, end of life July 2023 - don't use)
docker-compose up -d

# v2 (current - use this)
docker compose up -d

  • The file format is largely compatible between versions, but with one important change: the version: field at the top of the file is no longer required in v2 and is ignored if present. Compose v2 follows the open Compose Specification standard. Drop version: '3.8' from new files. It's unnecessary noise.

  • Compose v2 also uses hyphens in container naming (myproject-web-1) instead of v1's underscores (myproject_web_1), uses BuildKit by default for faster image builds, and handles dependency ordering and build contexts more reliably.

The compose.yaml File Structure

The file is named compose.yaml (the preferred name) or docker-compose.yml (still supported). It has three top-level sections:

services:    # Required. Your containers.
  web: ...
  db: ...

volumes:     # Optional. Named volumes.
  db-data:

networks:    # Optional. Custom networks.
  frontend:
  backend:

If you don't define a networks section, Compose automatically creates a default bridge network for the project and connects all services to it. All services can reach each other by service name. This is why a minimal Compose file doesn't need explicit network configuration.

The project name is inferred from the directory containing the compose file. A project in /home/user/myapp gets the project name myapp. All resources (containers, networks, volumes) are prefixed with this name, so they don't collide with resources from other Compose projects. You can override it with --project-name or the COMPOSE_PROJECT_NAME environment variable.

Defining Services

Each entry under services defines a container. Here are the most commonly used service attributes:

services:
  api:
    # Image to use (pull from registry)
    image: node:18-alpine

    # OR build from a Dockerfile
    build:
      context: ./api
      dockerfile: Dockerfile

    # Override the default CMD
    command: ["node", "server.js"]

    # Ports to publish (host:container)
    ports:
      - "3000:3000"
      - "127.0.0.1:9229:9229"  # bind to localhost only

    # Environment variables
    environment:
      NODE_ENV: production
      DB_HOST: db

    # Load env vars from a file
    env_file:
      - .env

    # Named volumes and bind mounts
    volumes:
      - app-data:/data           # named volume
      - ./src:/app/src           # bind mount

    # Networks to connect to
    networks:
      - backend

    # Startup dependency
    depends_on:
      db:
        condition: service_healthy

    # Restart policy
    restart: unless-stopped

    # Health check
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 15s

The restart policy controls what happens when a container exits. unless-stopped restarts the container on any exit except when you explicitly stop it with docker compose stop or docker compose down. Other options are no (default, never restart), always (always restart including after daemon restart), and on-failure[:max-retries].

Networks in Compose

When you don't define any networks, Compose creates one default network for the whole project and connects every service to it. All services reach each other by service name.

When you need more control, like keeping a database inaccessible from a public-facing service, define networks explicitly:

services:
  nginx:
    image: nginx
    networks:
      - frontend
      - backend

  api:
    build: ./api
    networks:
      - backend

  db:
    image: postgres:16
    networks:
      - backend

networks:
  frontend:
  backend:

nginx is on both networks and can bridge traffic between them. db is only on backend, so it's unreachable from anything on frontend. This mirrors the network segmentation pattern covered in the previous article, but declared in YAML instead of wired up manually.

Volumes in Compose

Named volumes declared at the top-level volumes section are created and managed by Docker. Services reference them by name:

services:
  db:
    image: postgres:16
    volumes:
      - postgres-data:/var/lib/postgresql/data

  app:
    build: .
    volumes:
      - ./src:/app/src        # bind mount for development
      - /app/node_modules     # anonymous volume to prevent node_modules overwrite

volumes:
  postgres-data:              # Docker manages this

The empty postgres-data: declaration is all that's needed. Docker creates the volume the first time you run docker compose up if it doesn't already exist. On subsequent runs it reuses the existing volume, so data persists across docker compose down and docker compose up cycles.

docker compose down does not remove volumes by default. To remove volumes along with containers and networks, use docker compose down -v. This is intentional: you shouldn't accidentally wipe a database just by stopping a stack.

Essential Compose Commands

# Start all services (foreground, shows logs)
docker compose up

# Start all services in background (detached)
docker compose up -d

# Stop and remove containers and networks (volumes preserved)
docker compose down

# Stop and remove containers, networks, AND volumes
docker compose down -v

# List running services and their status
docker compose ps

# View logs for all services
docker compose logs

# Follow logs for a specific service
docker compose logs -f web

# Execute a command inside a running service container
docker compose exec web sh
docker compose exec db psql -U postgres

# Run a one-off command in a new container for a service
docker compose run --rm web node migrate.js

# Restart a specific service
docker compose restart web

# Pull latest images for all services
docker compose pull

# Show resource usage (CPU, memory) for running services
docker compose top

The difference between docker compose exec and docker compose run matters. exec runs a command inside an already running container. run starts a brand new container for that service, runs the command, and exits. The --rm flag on run removes the temporary container after it finishes, which is almost always what you want for one-off tasks like running migrations.

Building Custom Images with Compose

Services can either pull a pre-built image or build one from a Dockerfile. The build key handles the latter:

services:
  # Pull a pre-built image from a registry
  cache:
    image: redis:7-alpine

  # Build from a Dockerfile in the current directory
  api:
    build: .

  # Build with full control over context and Dockerfile
  frontend:
    build:
      context: ./frontend        # directory sent as build context
      dockerfile: Dockerfile.prod  # which Dockerfile to use
      target: production         # multi-stage build target
      args:
        NODE_ENV: production      # build arguments

To rebuild images:

# Build all services that use build:
docker compose build

# Build a specific service
docker compose build api

# Start and force rebuild (even if image exists)
docker compose up --build

# Build without using cache
docker compose build --no-cache

One common mistake: if you change your application code and run docker compose up -d, Compose won't automatically rebuild the image. It uses the existing cached image. You need docker compose up -d --build to pick up source code changes when building images. This is not an issue when using bind mounts for development (the host files are mounted directly), but matters for production-style builds.

depends_on: Startup Order and the Readiness Problem

depends_on tells Compose which services need to start before others:

services:
  web:
    image: my-app
    depends_on:
      - db
      - cache

  db:
    image: postgres:16

  cache:
    image: redis:7-alpine

Here db and cache start before web. This sounds like it solves the ordering problem, but there's a critical catch: depends_on only waits for the container to start, not for the service inside it to be ready.

PostgreSQL takes several seconds to initialize after the container starts. MySQL can take even longer. A web application that connects to the database on startup will fail if it tries to connect during that initialization window, even with depends_on in place.

The naive depends_on with just a service name is not enough for any service that needs its dependency to actually be accepting connections. You need health checks.

Health Checks: The Right Way to Handle Dependencies

Health checks let Compose verify that a service is genuinely ready before starting services that depend on it. Combine a healthcheck on the dependency with condition: service_healthy in depends_on:

services:
  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_PASSWORD: secret
      POSTGRES_DB: myapp
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
      interval: 5s
      timeout: 5s
      retries: 5
      start_period: 10s

  api:
    build: ./api
    depends_on:
      db:
        condition: service_healthy   # waits until pg_isready passes
    environment:
      DATABASE_URL: postgresql://postgres:secret@db:5432/myapp

The healthcheck fields:

  • test: the command that runs inside the container. Exit code 0 means healthy, non-zero means unhealthy

  • interval: how often to run the check

  • timeout: how long to wait for the check to complete before marking it as failed

  • retries: how many consecutive failures before marking the container unhealthy

  • start_period: grace period during startup where failures don't count toward the retry limit

The condition: service_healthy in depends_on tells Compose to wait until the health check passes before starting the api container. Without it, the api might try to connect to postgres before it's accepting connections and crash on startup.

Three depends_on conditions are available in Compose v2:

depends_on:
  db:
    condition: service_started    # default, just waits for container start
  migrations:
    condition: service_completed_successfully  # waits for exit code 0
  cache:
    condition: service_healthy    # waits for health check to pass

service_completed_successfully is useful for one-shot setup tasks like database migrations. The migration container runs, exits with code 0, and only then does the API start.

Environment Variables and .env Files

Compose provides several ways to pass environment variables to services, with a clear priority order when multiple sources exist.

Inline in compose.yaml

services:
  api:
    environment:
      NODE_ENV: production
      PORT: 3000

Interpolated from the shell or .env file

services:
  api:
    environment:
      API_KEY: ${API_KEY}           # pulled from shell or .env
      DB_PASSWORD: ${DB_PASSWORD:-default}  # with fallback

Compose automatically loads a .env file from the same directory as the compose file. Variables defined there are available for interpolation in the compose file itself.

# .env file
API_KEY=abc123
DB_PASSWORD=secret
NODE_ENV=production

Per-service env_file

services:
  api:
    env_file:
      - .env
      - .env.local        # overrides .env values

The env_file attribute loads variables directly into the container's environment. Multiple files are loaded in order, with later files overriding earlier ones.

Priority order (highest to lowest)

  1. Values set with docker compose run -e VAR=value on the command line

  2. Shell environment variables on the host

  3. Values in .env file (for compose file interpolation)

  4. Values in env_file files

  5. Values in environment: in the compose file

  6. Defaults set in the Dockerfile ENV instruction

One important rule from the official Docker docs: do not use environment variables to pass sensitive information like passwords into containers. Use secrets instead.

Never commit .env files containing real secrets to version control. Add .env to .gitignore. Use .env.example with placeholder values as a template for teammates.

Secrets: Handling Sensitive Data Properly

Environment variables are visible to all processes inside a container, can appear in logs, and are exposed in docker inspect output. For passwords, API keys, and other sensitive values, Compose secrets are a safer mechanism.

Secrets are mounted as files inside the container at /run/secrets/<secret_name>. The application reads the secret from the file rather than from an environment variable.

services:
  db:
    image: postgres:16
    secrets:
      - db_password
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password

  api:
    build: ./api
    secrets:
      - db_password
      - api_key
    environment:
      DB_PASSWORD_FILE: /run/secrets/db_password
      API_KEY_FILE: /run/secrets/api_key

secrets:
  db_password:
    file: ./secrets/db_password.txt   # contents of this file become the secret
  api_key:
    file: ./secrets/api_key.txt

The secret files are small text files containing only the secret value. They live outside version control. The compose file references them by path, but only the container mounts their contents, and only services explicitly listed in a secret's secrets attribute can access it.

For services that support _FILE environment variable variants (PostgreSQL's POSTGRES_PASSWORD_FILE, MySQL's MYSQL_ROOT_PASSWORD_FILE), the application reads the password from the file path rather than from a plain environment variable. Many official Docker images support this pattern.

Multiple Compose Files for Different Environments

A common pattern is to have a base compose.yaml with shared configuration and separate override files for development and production:

compose.yaml           # base configuration, shared across all environments
compose.override.yaml  # development overrides (loaded automatically)
compose.prod.yaml      # production overrides

Compose automatically merges compose.override.yaml with compose.yaml when you run docker compose up without specifying files. For production, specify both files explicitly:

# Development (compose.yaml + compose.override.yaml loaded automatically)
docker compose up -d

# Production (explicit file selection)
docker compose -f compose.yaml -f compose.prod.yaml up -d

Example split:

# compose.yaml (base)
services:
  api:
    build: ./api
    environment:
      NODE_ENV: production

  db:
    image: postgres:16
    volumes:
      - postgres-data:/var/lib/postgresql/data

volumes:
  postgres-data:
# compose.override.yaml (development - loaded automatically)
services:
  api:
    volumes:
      - ./api/src:/app/src    # bind mount source for live reload
    environment:
      NODE_ENV: development
    ports:
      - "9229:9229"           # expose debugger port

  db:
    ports:
      - "5432:5432"           # expose db port for local access with DB client
# compose.prod.yaml (production - specified explicitly)
services:
  api:
    image: myregistry/api:1.2.3    # use a specific published image, not build
    restart: unless-stopped
    deploy:
      resources:
        limits:
          memory: 512m

This keeps your development setup (bind mounts, debug ports, exposed databases) completely separate from production config (specific image tags, restart policies, resource limits) without duplicating the shared configuration.

Service Profiles: Selective Service Startup

Compose v2 introduced profiles, which let you define services that only start when explicitly requested. This is useful for optional services like monitoring tools, database admin UIs, or test runners that you don't want running all the time.

services:
  api:
    build: ./api
    # No profile = always starts

  db:
    image: postgres:16
    # No profile = always starts

  adminer:
    image: adminer
    profiles:
      - tools            # only starts when "tools" profile is active
    ports:
      - "8080:8080"

  prometheus:
    image: prom/prometheus
    profiles:
      - monitoring       # only starts when "monitoring" profile is active
# Start just the core services (api + db)
docker compose up -d

# Start core services + tools profile
docker compose --profile tools up -d

# Start everything
docker compose --profile tools --profile monitoring up -d

A Complete Production-Ready Example

Here's a full three-service stack: a Node.js API, a PostgreSQL database, and a Redis cache, with health checks, proper dependency ordering, secrets handling, volume persistence, and network segmentation.

# compose.yaml

services:
  db:
    image: postgres:16-alpine
    restart: unless-stopped
    secrets:
      - db_password
    environment:
      POSTGRES_USER: appuser
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
      POSTGRES_DB: myapp
    volumes:
      - postgres-data:/var/lib/postgresql/data
    networks:
      - backend
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "appuser", "-d", "myapp"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s

  cache:
    image: redis:7-alpine
    restart: unless-stopped
    networks:
      - backend
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 3

  api:
    build:
      context: ./api
      dockerfile: Dockerfile
    restart: unless-stopped
    secrets:
      - db_password
    environment:
      NODE_ENV: production
      DB_HOST: db
      DB_USER: appuser
      DB_NAME: myapp
      DB_PASSWORD_FILE: /run/secrets/db_password
      REDIS_URL: redis://cache:6379
    ports:
      - "3000:3000"
    networks:
      - backend
      - frontend
    depends_on:
      db:
        condition: service_healthy
      cache:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 15s

networks:
  frontend:
  backend:

volumes:
  postgres-data:

secrets:
  db_password:
    file: ./secrets/db_password.txt

Walk through the key decisions: the database and cache have health checks so the API only starts when both are genuinely ready. The database password is managed as a secret mounted at /run/secrets/db_password, not passed as a plain environment variable. The database is on backend only with no published ports. The API is on both networks and publishes port 3000 for external access. restart: unless-stopped keeps all services running after unexpected exits or host reboots. The postgres volume persists data across stack restarts.

Key Takeaways

  • Docker Compose replaces multiple docker run commands with a single declarative compose.yaml file. docker compose up -d starts everything, docker compose down tears it all down

  • Use docker compose (space, v2) not docker-compose (hyphen, v1). Compose v1 reached end of life in July 2023. Drop the version: field from new files, it's no longer needed

  • The three top-level sections are services (containers), volumes (persistent storage), and networks (custom connectivity). If you don't define networks, Compose creates a default one automatically and connects all services to it

  • docker compose down preserves volumes by default. Use docker compose down -v to also remove volumes. This protects database data from accidental deletion

  • docker compose exec service cmd runs a command inside a running container. docker compose run --rm service cmd starts a fresh temporary container for one-off tasks like migrations

  • depends_on controls startup order but does NOT wait for a service to be ready, only for its container to start. Use healthcheck plus condition: service_healthy in depends_on to wait for genuine readiness

  • The start_period field in health checks gives slow-starting services a grace period before failures count toward the retry limit. Always set this for databases

  • Never put secrets in plain environment variables. Use the secrets top-level block, mount them at /run/secrets/<name>, and use _FILE env var variants that read from files

  • Compose automatically loads a .env file from the project directory for variable interpolation in the compose file. Add .env to .gitignore, commit .env.example with placeholder values

  • Use multiple compose files for environment-specific config: a base compose.yaml, an auto-loaded compose.override.yaml for development, and explicit compose.prod.yaml for production

  • Service profiles let you define optional services (like admin UIs or monitoring tools) that only start when you pass --profile name to compose commands

Conclusion

  • Docker Compose transforms multi-container application management from a manual, error-prone process into something reproducible, shareable, and version-controlled. The compose.yaml file becomes the single source of truth for how your application stack is wired together, the same way a Dockerfile is the single source of truth for how a single image is built.

  • The patterns covered here, particularly health checks with condition: service_healthy, secrets management, and multiple compose files per environment, are what separate a compose file that works on your laptop from one that holds up in a real team and production context.

  • From here, the concepts in this series (containers, images, volumes, networks, Compose) form the complete foundation for understanding container orchestration at scale with Kubernetes, where many of the same ideas apply but across clusters of machines rather than a single host.

🔗Let’s Stay Connected

📱 Join Our WhatsApp Community
Get early access to AI/ML, Cloud & Devops resources, behind-the-scenes updates, and connect with like-minded learners.
➡️ Join the WhatsApp Group

✅ Follow Me for Daily Tech Insights
➡️ LinkedIN
➡️ YouTube
➡️ X (Twitter)
➡️ Website

Keep Reading

© 2026 TechInsightNeuron.
Report abusePrivacy policyTerms of use
beehiivPowered by beehiiv