We recently had a question asked of us in the community slack channel, and it’s worth sharing.
The user runs Docker Swarm and was asking how they can configure the Portainer agent when their swarm nodes have differing configurations. In his case, some of his swarm nodes had NVMe drives, and he redirected the docker data-root (normally /var/lib/docker) to the NVMe drive.
Example docker daemon.json showing the changed docker data-root
Why does this even matter?
Well, ordinarily, it wouldn’t. The Portainer agent talks to the docker socket on each node of the cluster, and we then talk to the Docker API over that socket. The reason this matters is that in the standard agent deployment, which runs on all nodes in the swarm cluster, we also bind mount to /var/lib/docker/volumes. When you have redirected your Docker data-root path, the agent will not correctly bind to the missing path on those non-standard nodes and will not function.
So how do you fix this? How can you use the Portainer agent on a Swarm Cluster with different docker data-root settings?
The first step is to create labels on the nodes, with the labels indicating which are “default” vs which are “custom”. You use the docker command “docker node update --label-add <LABEL>=<VALUE> nodename“(eg docker node update --label-add data-root=default node1), and you do this for each node, but setting the value to be different on the nodes that are different eg –label-add data-root=custom node3.
Once you have that, you can deploy the below compose file, which will deploy the agent across the cluster, with placement constraints ensuring that the correct bind mounts are made on the applicable nodes.
version: '3.9'
services: portainer_agent_default: image: portainer/agent:2.20.3 ports: - target: 9001 published: 9001 protocol: tcp mode: host volumes: - /var/run/docker.sock:/var/run/docker.sock - /var/lib/docker/volumes:/var/lib/docker/volumes networks: - portainer_agent_network deploy: mode: global placement: constraints: - node.labels.data-root == default portainer_agent_custom: image: portainer/agent:2.20.3 command: --cluster-addr="agent_portainer_agent_default" ports: - target: 9001 published: 9001 protocol: tcp mode: host volumes: - /var/run/docker.sock:/var/run/docker.sock - /dockerdata/volumes:/var/lib/docker/volumes networks: - portainer_agent_network deploy: mode: global placement: constraints: - node.labels.data-root == custom
networks: portainer_agent_network:
You deploy this with the command: docker stack deploy --compose-file=<WHATEVER_YOU_NAMED_IT.yml> agent
Dont forget to change the BOLD RED font area in the compose to whatever your docker-root path is.
Once you have this running, go ahead and add the agent to your existing Portainer instance. When you go into that environment, and click on “services” you will see the two services defined. You can now fully interact with Portainer, and all features work as expected.
