[Linux] Running multiple commands on multiple servers in parallel

I have a cluster of Raspberry Pi’s at home running Docker Swarm. To keep up with things, they’re using the same configuration – users, Docker versions, etc.

Sometimes some of the updates require running the same thing on all the nodes of the cluster – for example – creating a new Docker macvlan network.

To do this, one has to either run the same commands on all the nodes manually, by ssh-ing in each and every one, running the commands and moving on to the next one.

Since I’m trying to reduce my work in the long run and to make things as organized as possible, I started looking into ways to run commands on all the nodes at the same time.

Iteration 1: Custom Python script

The first iteration of doing this was to create a custom Python script. One side of this was to learn Python a bit better, the other side was that I couldn’t find a good way to run commands in more native way.

As one can expect (hey, still having little experience in the language 🙂 ), it became bloated mess pretty quickly and was very prone to breakage.

I had to find another way to do this, without having to re-learn what I did 3 weeks ago and what broke the last time.

Iteration 2: pssh

Googling around I stumbled upon pssh that can run things in parallel via SSH.

All things considered, it was pretty easy to get it up and running.

The problem was that to run some of the things I needed to be able to do them via sudo. Passing passwords around wasn’t the best, so I had to search around again to figure out how to send the sudo password when the remote server asked for it.

Turns out it’s pretty easy if you stumble upon the right StackOverflow comment.

The TL;DR; version is:

stty -echo; printf "Password: "; read PASS; stty echo; echo "${PASS}" | \
pssh -h <HOSTFILE> -o /tmp/output -t 0 -I "`cat ./commands.sh`"

The -I "`cat ./commands.sh`" is required to be done like this (surrounded in quotes and backcquotes like this), otherwise it doesn’t properly run the commands. The password is going to be sent as a command separately and not picked up as an input to sudo.

The other catch is that because of the way this is set up, one needs to run sudo with the -S parameter, to pick the password input properly.