As I hinted in my 37c3 post, I was using a bit of time at ‘Congress’ to improve the structure of my containers in the cloud. While I have quite a number of projects running in dedicated virtual machines, I have containerized others such as this blog, my MediaWiki, an OnlyOffice instance, Etherpads and a number of internal projects. Each app typically requires two or three containers, usually one for the app itself, and another one for a MariaDB or other kind of database. And, in addition, there’s a reverse web proxy on each host, so I can share a public IP address and have a central place that automatically gets and updates TLS certificates.
This is all nice and well but one thing that has been bugging me a bit is that each container can communicate with all other containers. Surely, there must be a way to isolate them!?
Theoretically that huge container mesh is not a problem, as each container should be robust enough to be directly connected to the Internet. However, there is really no need that the database container of one app should be reachable from containers of other apps. It could thus even happen by mistake, that a new project accidentally writes its data into the database container of another app. It takes a bit of misconfiguration to actually make this happen, but well…, last famous words…
Anyway, so since I wanted to have a closer look at how Docker networking works anyway, I set myself the goal to see if I could structure my setup in a way that database containers would only be reachable over an internal network by containers of the same project, but not by containers of other projects. Turns out that this is much simpler than I thought.
The Straight Forward Approach
Let’s make a practical example, the MediaWiki I recently set-up for myself. Without anything fancy, the relevant parts for networking in the docker-compose.yml file look as follows:
version: '3.8'
services:
martin-wiki-db:
image: mariadb:latest
volumes:
- ./mysql:/var/lib/mysql
restart: always
environment:
[...]
martin-mediawiki:
image: mediawiki:${WIKI_VERSION}
depends_on:
- martin-wiki-db
restart: always
[...]
ports:
- ${WIKI_HTTP_PORT}:80
[...]
There are actually only two lines that are important for networking. The ‘ports’ parameter at the end of the config file excerpt exposes the internal TCP port 80 to an external port, e.g. also port 80, or any port you like. This external port is then reachable on the outside. If connected to the Internet, the world can come and visit.
The second line that has something to do with networking here is a bit more subtle, it’s actually the service name of the the MariaDB database container ‘martin-wiki-db‘! During initial configuration of the MediaWiki, the software wants to know the hostname or IP address of the database server. Since Docker assigns new IP addresses whenever a docker-compose project is started, there is no fixed IP address, However, the container name is also the hostname in the internal network, so one can give the container name to the installation routine. The installation routine knows nothing about containers, it just treats it as a hostname and will resolve it. Docker’s DNS resolver will then return the ephemeral IP address of the database container.
Note that only the internal port 80 of the MediaWiki Container is visible to the outside world. The MediaWiki container and the MariaDB container then use an internal IP subnet to talk with each other. All that is required to set this up is those two lines, so it’s really easy!
The Reverse Proxy Approach
In most cases, apps in containers do not handle TLS certificates for https connections themselves but leave the task to a (containerized) reverse proxy. Have a look here for details. To talk to the world via the reverse proxy, the docker-compose.yml file needs to be extended as follows. Have a look at the original MediaWiki post for the full config file:
[...]
martin-mediawiki:
image: mediawiki:${WIKI_VERSION}
[...]
environment:
TZ: ${TZ}
VIRTUAL_HOST: MY_DOMAIN_NAME
LETSENCRYPT_HOST: MY_DOMAIN_NAME
LETSENCRYPT_EMAIL: noreply@test.com
#ports:
# - ${WIKI_HTTP_PORT}:80
networks:
default:
external:
name: proxyBasically, the first three blue lines tell the reverse proxy which domain name the application should be reachable at from the Internet. The domain name has to be given twice, once for the reverse proxy itself, and once for the Letsencrypt setup that is also managed by the reverse proxy. This is NOT a network specific configuration, however, this deals with the certificates only!
The important addition is the networks section with the external network that is called proxy. This is a Docker internal network that is separate from the default internal Docker network. It is created by the Reverse Proxy docker container configuration, so have a look at the reverse proxy post I linked to above for details.
Also important: The port mapping of the internal port 80 to the outside world has to be removed or commented out with a # character. This is because we don’t want the Mediawiki to receive requests directly, but only via the reverse proxy setup and the internal network called ‘proxy‘. Note: The internal network of the reverse proxy could by called anything, it is configurable!
Have a look at the indentation of the networks line! It is right at the beginning and not indented. This means that it applies to ALL containers that are described in the docker-compose.yml file!
Note that the MediaWiki itself is totally unaware of any of this. The three environment variables are only useful to the reverse proxy and the networks section tells Docker to which internal network to connect the containers of the project. No impact on the MediaWiki, it doesn’t notice any of this!
Reverse Proxy + Separation
The shortcoming I wanted to get rid of in the next step is that ALL containers of a project are connected to the ‘proxy’ network. However, the MariaDB container of the WikiMedia project doesn’t need to be reachable from there, it only needs to talk to the WikiMedia container in the project. The way this can be done is to create an internal Docker network for the project, so the containers can talk with each other, and only connect the MediaWiki container to the ‘proxy’ network. Here’s how the config for this looks like:
services:
martin-wiki-db:
image: mariadb:latest
[...]
networks:
- internal_network
[...]
martin-mediawiki:
image: mediawiki:${WIKI_VERSION}
depends_on:
- martin-wiki-db
[...]
networks:
- internal_network
- default
networks:
internal_network:
driver: bridge
default:
external:
name: proxy
In the ‘networks’ section in the code snippet above, you will still see the ‘proxy‘ network configuration as before. In addition there is an ‘internal_network‘ now (give it any name you want) that uses Docker’s (Ethernet) ‘bridge’ driver. The MariaDB container is then connected to the ‘internal_network‘ only, while the MediaWiki container is connected both to the ‘internal_network‘ and the ‘default‘ network, which maps to the ‘proxy‘ network.
Useful Docker Commands
O.k. so far so good, but how can you see if containers are connected to the networks as you intended? Fortunately, there are Docker commands for this. Here’s how to get an overview of all internal Docker networks:
docker network lsEven if the name ‘internal_network‘ is used in several docker_compose.yml files, it still works, because Docker prepends the project name, so each project has its own ‘internal_network‘. Great!
To see which containers are connected to a particular network, use the following command:
docker network inspect <network name>This will produce quite some output and give you, among other things, the subnet address and a list of all containers connected to this network and their IP addresses in this subnet.
Summary
OK, I’ve put quite a bit of text around it, but keeping network traffic private between containers of a single project is actually straight forward, only a few extra lines of configuration is required in a docker-compose.yml file. In the meantime I have adapted all my Docker’ized apps to this scheme and I’m very happy with the result!