Fixing Docker MTU for private networks and VPNs

Fixing Docker MTU for private networks and VPNs
Photo by Cameron Venti / Unsplash

Recently while settings up a Docker swarm cluster I noticed something strange happening, a lot of the requests being processed by the swarm service were failing even though the cluster was live and the nodes were able to communicate to each other. After a few hours of debugging, scratching heads, googling etc I finally found this and that was indeed the problem.

In a nutshell, the problem was the swarm ingress routing mesh with a bad MTU configuration, so even though the nodes could communicate to each other, some of the TCP packets could not be sent over the network because of its size.

💡
The internet's transmission control protocol (TCP) uses the MTU to determine the maximum size of each packet in any transmission. MTU is usually associated with the Ethernet protocol, where a 1500-byte packet is the largest allowed in it (and hence over most of the internet).

The Docker Daemon does not check the MTU of the outgoing connection and is by default set to 1500, my private network (using Tailscale) on the hand only supports an MTU of 1280, you can find this by running ip link command.

The solution is fairly straightforward, we have to update Docker network configuration to use the lower MTU and we can do this in two ways:

Create new ingress

  • Stop all services and delete the current ingress using docker network rm ingress
  • Create new ingress network with updated MTU, docker network create --driver overlay --ingress --opt com.docker.network.driver.mtu=1280 --subnet 10.11.0.0/24 --gateway 10.11.0.1 ingress
  • Restart docker and deploy your services again.

Configure using docker compose

Another solution is to provide this configuration in your docker-compose file.

networks:                                
  default:                               
    driver: bridge                       
    driver_opts:                         
      com.docker.network.driver.mtu: 1280

References:

Use swarm mode routing mesh
Use the routing mesh to publish services externally to a swarm
Docker swarm load balancing not working over private network · Issue #36689 · moby/moby
Description Problem is probably similar to #25325. Docker can't reach containers from hostB when I query hostA public address. I'm using Docker swarm with 2 hosts, they are connected via wi...
Docker MTU issues and solutions - Matthias Lohr
If you want to use Docker on servers or virtual machines, technical limitations can sometimes lead to a situation in which – even without intentional limitation – it is not possible to access the outer world from a docker container. Docker MTU configuration A common problem when operating dockers wi…