Is there a maximum number of containers running on a Docker host?

by

There are a number of system limits you can run into (and work around) but there’s a significant amount of grey area depending on

  1. How you are configuring your docker containers.
  2. What you are running in your containers.
  3. What kernel, distribution and docker version you are on.

The figures below are from the boot2docker 1.11.1 vm image which is based on Tiny Core Linux 7. The kernel is 4.4.8

Docker

Docker creates or uses a number of resources to run a container, on top of what you run inside the container.

  • Attaches a virtual ethernet adaptor to the docker0 bridge (1023 max per bridge)
  • Mounts an AUFS and shm file system (1048576 mounts max per fs type)
  • Create’s an AUFS layer on top of the image (127 layers max)
  • Forks 1 extra docker-containerd-shim management process (~3MB per container on avg and sysctl kernel.pid_max)
  • Docker API/daemon internal data to manage container. (~400k per container)
  • Creates kernel cgroups and name spaces
  • Opens file descriptors (~15 + 1 per running container at startup. ulimit -n and sysctl fs.file-max )

Docker options

  • Port mapping -p will run a extra process per port number on the host (~4.5MB per port on avg pre 1.12, ~300k per port > 1.12 and also sysctl kernel.pid_max)
  • --net=none and --net=host would remove the networking overheads.

Container services

The overall limits will normally be decided by what you run inside the containers rather than dockers overhead (unless you are doing something esoteric, like testing how many containers you can run 🙂

If you are running apps in a virtual machine (node,ruby,python,java) memory usage is likely to become your main issue.

IO across a 1000 processes would cause a lot of IO contention.

1000 processes trying to run at the same time would cause a lot of context switching (see vm apps above for garbage collection)

If you create network connections from a 1000 containers the hosts network layer will get a workout.

It’s not much different to tuning a linux host to run a 1000 processes, just some additional Docker overheads to include.

Example

1023 Docker busybox images running nc -l -p 80 -e echo host uses up about 1GB of kernel memory and 3.5GB of system memory.

1023 plain nc -l -p 80 -e echo host processes running on a host uses about 75MB of kernel memory and 125MB of system memory

Starting 1023 containers serially took ~8 minutes.

Killing 1023 containers serially took ~6 minutes

More Related Contents:

  1. How to copy files from host to Docker container?
  2. Why docker container exits immediately
  3. Use of Supervisor in docker
  4. Locating data volumes in Docker Desktop (Windows)
  5. Docker Compose – Share named volume between multiple containers
  6. In a Dockerfile, How to update PATH environment variable?
  7. How to remove old Docker containers
  8. How to keep Docker container running after starting services?
  9. How to show the run command of a docker container
  10. How to make sure docker’s time syncs with that of the host?
  11. what is docker run -it flag?
  12. Cannot login to Docker account
  13. Linux machine with docker deploy windows container
  14. In Docker, what’s the difference between a container and an image? [duplicate]
  15. docker data volume vs mounted host directory
  16. Docker Compose – How to execute multiple commands?
  17. Is there a way to combine Docker images into 1 container?
  18. How do I access a spring app running in a docker container?
  19. Are you trying to mount a directory onto a file (or vice-versa)?
  20. How to get ENV variable when doing Docker Inspect
  21. How to pass ARG value to ENTRYPOINT?
  22. Multiple commands on docker ENTRYPOINT
  23. How to link containers in docker?
  24. How to restart a single container with docker-compose
  25. Error “Get https://registry-1.docker.io/v2/: net/http: request canceled” while building image
  26. What work does the process in container “gcr.io/google_containers/pause:0.8.0” do?
  27. Dockerfile: Output of RUN instruction into a Variable
  28. How to write data to host file system from Docker container
  29. Is there a way to force docker-machine to create vm with a specific ip?
  30. how to set ulimit / file descriptor on docker container the image tag is phusion/baseimage-docker

Leave a Comment