We had written two articles in the past to run commands on multiple remote server in parallel.
Those are Parallel SSH (PSSH) or Distributed Shell (DSH).
Today we are going to discuss one more topic similar to those, but it allows us to perform the same on cluster nodes as well.
You may think, we can write a small shell script to achieve this instead of installing these third party packages.Of course you are right and if you are going to run some commands in 10-15 remote systems then you don’t need to use this.
However, the scripts will take some time to complete this task as it runs in a sequential order.Think about if you would like to run some commands on 1000+ servers what will be the options?
In this case your script won’t help y. Also, it would take good amount of time to complete a task.So, to overcome this kind of situation. We need to run the command in parallel on remote machines.
For that, we need use one of the Parallel applications. I hope this explanation might clarify your doubts about parallel utilities.
What is ClusterShell?
clush stands for ClusterShell. ClusterShell is an event-driven open source Python library, designed to run local or distant commands in parallel on server farms or on large Linux clusters.
It will take care of common issues encountered on HPC clusters, such as operating on groups of nodes, running distributed commands using optimized execution algorithms, as well as gathering results and merging identical outputs, or retrieving return codes.
ClusterShell takes advantage of existing remote shell facilities already installed on your systems, like SSH.
ClusterShell’s primary goal is to improve the administration of high- performance clusters by providing a lightweight but scalable Python API for developers. It also provides clush, clubak and cluset/nodeset, convenient command-line tools that allow traditional shell scripts to benefit from some of the library features.
ClusterShell’s written in Python and it requires Python (v2.6+ or v3.4+) to run on your system.
How to install ClusterShell on Linux?
ClusterShell package is available in most of the distribution official package manager. So, use the distribution package manager tool to install it.
For Fedora system, use DNF Command to install clustershell.
$ sudo dnf install clustershell
Python 2 module and tools are installed and if it’s default on your system then run the following command to install Python 3 development on Fedora System.
$ sudo dnf install python3-clustershell
Make sure you should have enabled the EPEL repository on your system before performing clustershell installation.
For RHEL/CentOS systems, use YUM Command to install clustershell.
$ sudo yum install clustershell
Python 2 module and tools are installed and if it’s default on your system then run the following command to install Python 3 development on CentOS/RHEL System.
$ sudo yum install python34-clustershell
For openSUSE Leap system, use Zypper Command to install clustershell.
$ sudo zypper install clustershell
Python 2 module and tools are installed and if it’s default on your system then run the following command to install Python 3 development on OpenSUSE System.
$ sudo zypper install python3-clustershell
For Debian/Ubuntu systems, use APT-GET Command or APT Command to install clustershell.
$ sudo apt install clustershell
How to install ClusterShell in Linux using PIP?
Use PIP to install ClusterShell because it’s written in Python.
Make sure you have enabled the Python and PIP on your system before performing clustershell installation.
$ sudo pip install ClusterShell
How to use ClusterShell on Linux?
It’s straight forward and awesome tool compared with other utilities such as pssh and dsh. It has so many options to perform the remote execution in parallel.
Make sure you should have enabled the password less login on your system before start using clustershell.
The following configuration file defines system-wide default values. You no need to modify anything here.
$ cat /etc/clustershell/clush.conf
If you would like to create a servers group. Here you can go. By default some examples were available so, do the same for your requirements.
$ cat /etc/clustershell/groups.d/local.cfg
Just run the clustershell command in the following format to get the information from the given nodes.
$ clush -w 192.168.1.4,192.168.1.9 cat /proc/version 192.168.1.9: Linux version 4.15.0-45-generic (buildd@lgw01-amd64-031) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 192.168.1.4: Linux version 3.10.0-957.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Thu Nov 8 23:39:32 UTC 2018
Option:
-w:nodes where to run the command.
You can use the regular expressions instead of using full hostname and IPs.
$ clush -w 192.168.1.[4,9] uname -r 192.168.1.9: 4.15.0-45-generic 192.168.1.4: 3.10.0-957.el7.x86_64
Alternatively you can use the following format if you have the servers in the same IP series.
$ clush -w 192.168.1.[4-9] date 192.168.1.6: Mon Mar 4 21:08:29 IST 2019 192.168.1.7: Mon Mar 4 21:08:29 IST 2019 192.168.1.8: Mon Mar 4 21:08:29 IST 2019 192.168.1.5: Mon Mar 4 09:16:30 CST 2019 192.168.1.9: Mon Mar 4 21:08:29 IST 2019 192.168.1.4: Mon Mar 4 09:16:30 CST 2019
clustershell allow us to run the command in batch mode. Use the following format to achieve this.
$ clush -w 192.168.1.4,192.168.1.9 -b
Enter 'quit' to leave this interactive mode
Working with nodes: 192.168.1.[4,9]
clush> hostnamectl
---------------
192.168.1.4
---------------
Static hostname: CentOS7.2daygeek.com
Icon name: computer-vm
Chassis: vm
Machine ID: 002f47b82af248f5be1d67b67e03514c
Boot ID: f9b37a073c534dec8b236885e754cb56
Virtualization: kvm
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-957.el7.x86_64
Architecture: x86-64
---------------
192.168.1.9
---------------
Static hostname: Ubuntu18
Icon name: computer-vm
Chassis: vm
Machine ID: 27f6c2febda84dc881f28fd145077187
Boot ID: f176f2eb45524d4f906d12e2b5716649
Virtualization: oracle
Operating System: Ubuntu 18.04.2 LTS
Kernel: Linux 4.15.0-45-generic
Architecture: x86-64
clush> free -m
---------------
192.168.1.4
---------------
total used free shared buff/cache available
Mem: 1838 641 217 19 978 969
Swap: 2047 0 2047
---------------
192.168.1.9
---------------
total used free shared buff/cache available
Mem: 1993 352 1067 1 573 1473
Swap: 1425 0 1425
clush> w
---------------
192.168.1.4
---------------
09:21:14 up 3:21, 3 users, load average: 0.00, 0.01, 0.05
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
daygeek :0 :0 06:02 ?xdm? 1:28 0.30s /usr/libexec/gnome-session-binary --session gnome-classic
daygeek pts/0 :0 06:03 3:17m 0.06s 0.06s bash
daygeek pts/1 192.168.1.6 06:03 52:26 0.10s 0.10s -bash
---------------
192.168.1.9
---------------
21:13:12 up 3:12, 1 user, load average: 0.08, 0.03, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
daygeek pts/0 192.168.1.6 20:42 29:41 0.05s 0.05s -bash
clush> quit
If you would like to run the command on a group of nodes then use the following format.
$ clush -w @dev uptime or $ clush -g dev uptime or $ clush --group=dev uptime 192.168.1.9: 21:10:10 up 3:09, 1 user, load average: 0.09, 0.03, 0.01 192.168.1.4: 09:18:12 up 3:18, 3 users, load average: 0.01, 0.02, 0.05
If you would like to run the command on more than one group of nodes then use the following format.
$ clush -w @dev,@uat uptime or $ clush -g dev,uat uptime or $ clush --group=dev,uat uptime 192.168.1.7: 07:57:19 up 59 min, 1 user, load average: 0.08, 0.03, 0.00 192.168.1.9: 20:27:20 up 1:00, 1 user, load average: 0.00, 0.00, 0.00 192.168.1.5: 08:57:21 up 59 min, 1 user, load average: 0.00, 0.01, 0.05
clustershell allow us to copy a file to remote machines. To copy local file or directory to the remote nodes in the same location.
$ clush -w 192.168.1.[4,9] --copy /home/daygeek/passwd-up.sh
We can verify the same by running the following command.
$ clush -w 192.168.1.[4,9] ls -lh /home/daygeek/passwd-up.sh 192.168.1.4: -rwxr-xr-x. 1 daygeek daygeek 159 Mar 4 09:00 /home/daygeek/passwd-up.sh 192.168.1.9: -rwxr-xr-x 1 daygeek daygeek 159 Mar 4 20:52 /home/daygeek/passwd-up.sh
To copy local file or directory to the remote nodes in the different location.
$ clush -g uat --copy /home/daygeek/passwd-up.sh --dest /tmp
We can verify the same by running the following command.
$ clush --group=uat ls -lh /tmp/passwd-up.sh 192.168.1.7: -rwxr-xr-x. 1 daygeek daygeek 159 Mar 6 07:44 /tmp/passwd-up.sh
To copy file or directory from remote nodes to local system.
$ clush -w 192.168.1.7 --rcopy /home/daygeek/Documents/magi.txt --dest /tmp
We can verify the same by running the following command.
$ ls -lh /tmp/magi.txt.192.168.1.7 -rw-r--r-- 1 daygeek daygeek 35 Mar 6 20:24 /tmp/magi.txt.192.168.1.7
