Overlay Networks Across Multiple Docker Hosts

1. Why bridge networks end at the host boundary

A bridge network exists exclusively inside the Linux kernel of a single Docker host. Containers on two different machines can never reach each other in a bridge network by container name, even if both networks share the same name, because technically they are two completely separate virtual switches. As soon as a setup spans more than one host, for example for high availability or horizontal scaling, it needs a different solution than a regular bridge network.

This is exactly where an overlay network comes in. It encapsulates container traffic into additional network packets that get transported over the regular network between hosts and decapsulated again at the destination host. None of this is visible to the application inside the container; it keeps communicating over ordinary IP addresses and service names, regardless of which physical host the target container actually runs on. An overlay network makes the host boundary transparent for container communication.

2. Fundamentals of Docker Swarm and overlay networks

Docker ships its multi-host orchestration in what is called Swarm mode, included directly in the Docker Engine with no extra software required. A swarm consists of manager nodes that maintain cluster state, and worker nodes that run containers. An overlay network is defined on a manager node and automatically replicated to every node that actually runs containers in that network, instead of being distributed globally to every node in the cluster.

The control plane for an overlay network runs over Swarm's gossip-protocol-based cluster store, which keeps information about container IPs and names in sync across all participating nodes. This fundamentally distinguishes an overlay network from a bridge network, where the embedded DNS server only knows local information. Without active Swarm mode, no regular overlay network can be created; a single standalone Docker host remains limited to bridge, host and macvlan.

For the high availability of the cluster itself, manager nodes use a Raft consensus algorithm, which keeps guaranteeing a consistent view of the cluster state, and therefore of every overlay network, even if individual managers fail. Odd numbers of managers are recommended, usually three or five, so that a majority for valid decisions is always achievable in case of a network split. This consensus layer is why an overlay network keeps working correctly even after a single manager fails, as long as the majority of managers stays reachable.

# List all nodes in the cluster with their role and availability
docker node ls

# Example output:
# ID       HOSTNAME   STATUS   AVAILABILITY   MANAGER STATUS
# abc123   node-1     Ready    Active         Leader
# def456   node-2     Ready    Active         Reachable
# ghi789   node-3     Ready    Active         Reachable

3. Initializing Swarm and creating an overlay network

The first step toward an overlay network is initializing a swarm cluster on a first node, which automatically becomes the manager. Additional nodes join the cluster through a join token that the manager issues at initialization, separated by manager and worker role. Only after several nodes have successfully joined does an overlay network make practical sense, because it enables exactly the communication between these nodes.

An overlay network is then created with the overlay driver and assigned to a service via docker service create. Docker Swarm automatically places the service's containers on the available worker nodes and ensures that every container, regardless of its actual physical host, appears in the same logical network and is reachable by the service name.

# Initialize the swarm on the first node (becomes the manager)
docker swarm init --advertise-addr 10.0.1.10

# Retrieve the join token to add worker nodes
docker swarm join-token worker

# On a second machine, join the swarm as a worker
docker swarm join --token SWMTKN-1-xxxxx 10.0.1.10:2377

# Create an overlay network for cross-host communication
docker network create -d overlay --attachable shop-overlay

# Deploy a service using the overlay network, replicated across nodes
docker service create --name shop-app \
  --network shop-overlay \
  --replicas 3 \
  shop-app:latest

4. VXLAN encapsulation: the technology behind it

Technically, an overlay network in Docker is based on VXLAN, Virtual Extensible LAN, an encapsulation protocol that wraps Ethernet frames in UDP packets. Every packet that a container wants to send to another physical network segment gets a VXLAN header attached at the source host, is transported over the physical network to the destination host, and is unwrapped there before being forwarded to the target container. This encapsulation runs entirely in the kernel and is invisible to applications inside the container.

The decisive advantage of VXLAN over classic VLAN tagging: an overlay network is not limited to 4094 possible segments like 802.1Q, but uses a 24-bit identifier that allows over 16 million distinct logical networks. For operation, it is enough for the participating hosts to exchange UDP packets on the standard VXLAN port over the physical network; deeper configuration of the physical network is usually not required.

5. Service discovery and load balancing

Within an overlay network, name resolution works exactly like in a local bridge network: every service is reachable by its name, regardless of which node the individual replica containers actually run on. Docker Swarm additionally integrates a virtual-IP-based load balancer that automatically distributes requests to a service name across all running replicas, without requiring an external load balancer to be configured.

This built-in load balancer works at layer 4 and uses IPVS in the Linux kernel for distribution, which produces significantly less overhead for communication within an overlay network than a separate reverse proxy per service. For incoming traffic from outside the cluster, an additional ingress network is used, a special, automatically created overlay network that accepts published ports on every node of the cluster and forwards them to the correct container.

Which node actually runs which replica of a service can be checked at any time with docker service ps. This is especially useful when debugging, when one particular replica is noticeably slow to respond and you want to check whether that replica happens to run on an overloaded node of the overlay network, while other replicas on quieter nodes work fine.

# Show which node each replica of a service actually runs on
docker service ps shop-app

# Example output:
# NAME          NODE     CURRENT STATE
# shop-app.1    node-2   Running 2 hours ago
# shop-app.2    node-3   Running 2 hours ago
# shop-app.3    node-1   Running 2 hours ago

6. Encrypting overlay traffic with --opt encrypted

By default, traffic in an overlay network is unencrypted once it crosses the physical network between hosts. For environments where this physical network is not fully trusted, for example hosts in different data centers or cloud regions, encryption can be enabled with the --opt encrypted option when creating the network. Docker uses IPsec for this, with automatically generated and rotated keys managed through Swarm's internal certificate store.

Encrypting an overlay network has a measurable, but usually moderate, performance overhead from encrypting and decrypting every packet. For sensitive data, for example connections between application containers and databases across multiple data centers, the security benefit outweighs this overhead in almost every case. For purely internal clusters in a single, physically secured data center, encryption is often not necessary.

# Create an encrypted overlay network for sensitive cross-host traffic
docker network create -d overlay \
  --opt encrypted \
  --attachable \
  secure-overlay

# Verify encryption is active by inspecting the network options
docker network inspect secure-overlay --format '{{.Options}}'

7. Firewall ports and infrastructure requirements

An overlay network requires certain ports to be open between all swarm nodes, without which the encapsulation does not work. TCP port 2377 is needed for cluster management between manager nodes, TCP and UDP port 7946 for the gossip protocol used in node discovery, and UDP port 4789 for the actual VXLAN data traffic. If any of these ports is missing from the firewall configuration between hosts, hard-to-diagnose problems arise where individual containers in the overlay network exist but are unreachable.

Cloud environments with security groups or network segmentation between availability zones require special attention, because these rules are often more restrictive than a classic on-premise firewall. Before running an overlay network in production across multiple cloud regions or availability zones, it is worth explicitly testing connectivity on the three ports mentioned, instead of relying on implicit default firewall rules.

A simple but effective test before setting up an overlay network checks with tools like nc directly between two future swarm nodes whether the relevant ports are actually reachable, instead of dealing with cryptic error messages only after joining the swarm. UDP port 4789 in particular is often overlooked in firewall rule sets, because TCP ports dominate many checklists while the actual VXLAN data traffic runs entirely over UDP.

# Test manager port connectivity before joining the swarm
nc -zv 10.0.1.10 2377

# Test the gossip protocol port, both TCP and UDP
nc -zv 10.0.1.10 7946
nc -zuv 10.0.1.10 7946

# Test the VXLAN data port, UDP only, often overlooked in firewalls
nc -zuv 10.0.1.10 4789

8. Attachable networks for standalone containers

Without the --attachable option, only Swarm services can join an overlay network; regular containers started with docker run remain excluded. This is impractical in many cases, for example when a temporary debug container or a single legacy service outside Swarm orchestration needs access to services in the overlay network. Adding --attachable when creating the network lifts this restriction.

An attachable overlay network lets you connect standalone containers directly with docker run --network, without declaring them as a Swarm service at all. This is especially useful for diagnostic tools that get plugged into the overlay network briefly to test connectivity between services across multiple hosts, without needing to define a full service for it.

9. Bridge, overlay and host network compared

The following table compares the three network drivers with regard to multi-host capability, to put the choice of an overlay network versus the alternatives in perspective.

This comparison shows that an overlay network is not a general alternative to bridge networks, but a specific answer to the requirement of letting containers communicate across multiple physical hosts. Anyone running just a single host simply has no use case for an overlay network and should stick with bridge.

10. Summary

An overlay network solves exactly the problem where bridge networks fail: container communication beyond the boundary of a single Docker host. Through VXLAN encapsulation, Docker Swarm transports traffic between hosts while applications keep communicating over ordinary service names, regardless of where the target container actually runs. Built-in load balancing over virtual IPs automatically distributes requests across all replicas of a service.

Encryption with --opt encrypted protects sensitive traffic between data centers, attachable networks let standalone containers join in as well, and the three key firewall ports 2377, 7946 and 4789 must be open between all nodes for an overlay network to work at all. Anyone running just a single host does not need an overlay network, but as soon as a cluster with multiple nodes emerges, it is the only native Docker solution for cross-container communication.

11. FAQ: Overlay Networks Across Multiple Docker Hosts