Linux system, preferably with X86_64 processor, is required for installation. In principle, it is possible to use also 32 bit system, however, there will be problems during cross-compilation of 64 bit slave programs and the whole clusters will have to be composed of 32 bit processors. The possibility to use 32 bit processors is nowadays only theoretical and probably no one will use it. For proper function of the cluster 1 Gbit/s network is recommended. If a slower 100 Mbit/s network is used, its bandwidth is relatively quickly saturated and computational speed is bottlenecked. Also, newAD2 requires user to have the same user name on all computers in the cluster.
Installation on the main computer (Master) for more users
Group newad
Installation is done into common directories accessible by group newad
. Therefore, it is necessary to create this group and to add all users developing models, who will have permission to install newAD2 (thus usually only you). Other newAD2 users do not have to be included in the group. Log in with root permissions or ask system administrator and type following:
groupadd newad
usermod -a -G newad username
newAD2 directory
Now create directory for programs and data:
mkdir -p /usr/local/share/newAD2
chown root:newad /usr/local/share/newAD2
chmod 2775 /usr/local/share/newAD2
Install packages
- c/c++ : compilers
- boost-devel : c++ extension
- gsl-devel : library for scientific calculations
- pvm-devel : program and library for parallelization
- readline-devel : library allowing users to edit command lines as they are typed in
- gwyddion-devel : library of SPM data analysis program
Together with listed packages, around 100 more will be installed depending on your Linux distribution. Installation commands for some distribution are given below:
openSUSE Leap 15.4
zypper addrepo https://download.opensuse.org/repositories/science/15.4/science.repo
zypper refresh
zypper -n install gcc gcc-c++ make boost-devel gsl-devel readline-devel gwyddion-devel libtirpc-devel
Baličky pvm a pvm-devel po stažení nainstalujeme pomocí příkazu:
rpm -ihv pvm-3.4.6-lp154.0.0.x86_64.rpm pvm-devel-3.4.6-lp154.0.0.x86_64.rpm
Ubuntu 20.04
sudo apt-get install gcc g++ libboost-all-dev libgsl0-dev pvm-dev libreadline-dev libgwyddion20-dev libgtkglext1-dev libfftw3-dev libgtk2.0-dev libtirpc-dev gawk
sudo ln -s /usr/lib/x86_64-linux-gnu/libGL.so.1 /usr/lib/x86_64-linux-gnu/libGL.so
sudo mkdir /usr/lib/pvm3/lib/LINUX64
sudo ln -s /usr/lib/x86_64-linux-gnu/libpvm3.a /usr/lib/pvm3/lib/LINUX64/libpvm3.a
When finished, check the installation in user regime by:
echo halt | pvm
If it returns:
pvm> halt
Terminated
Everything is alright and you can proceed to next step on installation. If terminal returns error "Console: Can't start pvmd" something is wrong:
- Check your computer name by command
hostname
. You should receive the host name without domain. Otherwise it is necessary to set the system to return the host name alone. In SuSE change the corresponding line in file/etc/HOSTNAME
. - Check if network is working by command
ping hostname
. It should return positive response, because pvm requires working network even if pvm will be used for parallelization only locally. If your computer is not connected to any network set hostname on loopback (add127.0.0.1 hostname
into/etc/hosts
).
Download
Extract
tar jxf newAD2-1.tar.bz2
Create file clusterAD2.ini
If you are going to install newAD2 only to one computer you can skip this step.
In directory newAD2-1
create file clusterAD2.ini
with one line in the following format:
hostname:pvm_arch/proc_arch
If this file does not exist, it will be created automatically after first calling of make
. In extracted directory execute following command:
make clean
If make clean
writes error "gensub function never defined", install gawk
.
The file clusterAD2.ini
can stay as it is generated if you are going to use newAD2 on only one computer with X86_64 processor. Otherwise edit the file:
- Value
hostname
(name of the computer) is set automatically by program hostname and it is not necessary to change it. - Value
pvm_arch
is set toLINUX64
, orLINUXX86_64
on Fedora distribution and it is not necessary to change it. In case that you want to install newAD2 on 32 bit system, change the value toLINUX
. - Value
proc_arch
is set by script tonative
, which makes the compiler to automatically detect compiling processor and to set the correct value. In case that you will use newAD2 on more computers, it is better to set this value to architecture which best corresponds to your processor. This prevents double compilation for the given architecture, if the processor in main computer is the same as on Slave. Possible values can be found in Makefile. For processors X86_64 allowed values arek8
,amdfam10
,bdver1
,core2
,corei7
,coreavx2
,haswell
. The names are based on options-march
in gcc compiler without dashes ("-"). For example optioncore-avx2
is shortened tocoreavx2
(see man gcc). If some option is mission it is possible to use value for older processor (the code optimization will be worse) or to add it into Makefile.
Note that the file clusterAD2.ini
defines computer cluster on which newAD2 will be installed. Thus, so far there is only one computer in the cluster.
Compilation
Compilation of the parts necessary on main computer is done by command:
make
Installation
Installation on everything essential on main computer is done by command:
make local
If everything went alright, newAD2 is now installed on the computer in directory /usr/local/share/newAD2/bin
.
In Ubuntu, make
may have a problem locating pvm library. Probably the easiest workaround is to make the expected directory and include there a link to the pvm library:
sudo mkdir -p /usr/lib/pvm3/lib/LINUX64/
sudo ln -s /usr/lib/libpvm3.a /usr/lib/pvm3/lib/LINUX64/libpvm3.a
Also, in directory /usr/local/share/newAD2
file hostfileAD2
was generated, which contains information about the cluster in the following format:
&hostname sp=corenumber ep=/usr/local/share/newAD2/bin
where corenumber
in number of cores on the respective computer and is generated automatically.
Set path
In order to avoid calling the program by full path, it is convenient to set directory /usr/local/share/newAD2/bin
into system variable PATH
. In the home directory, users should edit file .bashrc
and add following line:
export PATH=$PATH:/usr/local/share/newAD2/bin
Log off and on to apply the changes. Text console of newAD2 can be started by command:
newAD2
or you can start newAD2 with graphical user interface by command:
newAD2gui
Installation on Slave computers for more users
Group newad and newAD2 directories
In order to extend your cluster to another computer ask administrator to create user accounts, newAD2 directories and group newad with users with installation permissions (same as for Master). Also it is essential to have package pvm installed. Other developer packages are not necessary, compilation was already done on the main computer.
Key generation
Next, it is necessary for both installation and running newAD2 to generate on Slave computers public and private key for all users (every user has to do this on his own). This allows to log Slave computers without entering password. In home directory run command:
ssh-keygen -t rsa
For requests of keygen command press three times by Enter, which will generate in directory .ssh
public key id_rsa.pub
and private key id_rsa
which will be usable without password. If you don't have any keys on the Slave computer with name slavehostname
you can copy key from your hostname
to file authorized_keys
in directory .ssh
on both Master and Slave computer:
cp .ssh/id_rsa.pub .ssh/authorized_keys
scp .ssh/id_rsa.pub slavehostname:.ssh/authorized_keys
Finally, check if it is possible to log without password or any other requests by command:
ssh slavehostname
Modification of clusterAD2.ini file
In directory newAD2-1
it is necessary to add lines with information about Slave computers into clusterAD2.ini
file. This serves compilation of corresponding parts of newAD2 and generation hostfileAD2
file. Format of line for Slave computer is the same as for Master, but on the end of the definition line path to pvmd on the Slave computer is added:
masterhostname:master_pvm_arch/master_proc_arch
slave1hostname:slave1_pvm_arch/slave1_proc_arch:slave1_pvmd_path
slave2hostname:slave2_pvm_arch/slave2_proc_arch:slave2_pvmd_path
slave3hostname:slave3_pvm_arch/slave3_proc_arch:slave3_pvmd_path
...
Values ..._proc_arch
will differ according to the processor type and values ..._pvmd_path
will depend on the particular system distribution:
-
/usr/lib/pvm3/lib/pvmd
SuSe, Ubuntu -
/usr/share/pvm3/lib/pvmd
Fedora
Same as for Master, ..._pvm_arch
can have on modern X86_64 platforms values LINUX64 or LINUXX86_64.
Compilation
For compilation of relevant parts of newAD2 that will be executed on Slave computers run command:
make
Installation of the whole cluster is done by command:
make cluster
This will copy required files into directories on Slave computers and will generate updated hostfileAD2
,
which contains all necessary information about the cluster in format:
&masterhostname sp=mastercorenumber dx= ep=/usr/local/pvm3/bin
&slave1hostname sp=slave1corenumber dx=slave1_pvmd_path ep=/usr/local/pvm3/bin
&slave2hostname sp=slave2corenumber dx=slave2_pvmd_path ep=/usr/local/pvm3/bin
&slave3hostname sp=slave3corenumber dx=slave3_pvmd_path ep=/usr/local/pvm3/bin
...
Note that dx=
in the first line (path to pvdm in Master computer) is not set, because it is not required. If some value in clusterAD2.ini
is edited, it is ignored because pvdm is already running on the Master computer.
Also it must be noted that compilation and installation of whole newAD2 can be done only by command make cluster
or make install
which is synonym. If only Master computer is listed in slaveAD2.ini
, then make local
and make cluster
(make install
) are equivalent. Logically, even if clusterAD2.ini
lists more computers, command make local
will make cluster only with Master computer. Furthermore, this commands differ by method of compiled programs coping into target directories. Command make local
uses install
command which doesn't require authentication. Command make cluster
(make install
) uses for file coping slaveAD2-OTF
and inspectorAD2
command rsync
which requires password in case that authentication on Master computer is done without password (even if you are technically copying from computer on the same computer). Lets sum up. Command make has several targets set:
-
make master
- compiles all parts necessary on Master computer exceptslaveAD2-OTF
andinspectorAD2
-
make slave
- compilesslaveAD2-OTF
andinspectorAD2
for Master computer -
make slaves
- compilesslaveAD2-OTF
andinspectorAD2
for all architectures present in the cluster -
make
ormake all
- compiles everything -
make local
- installs cluster only on the Master computer -
make cluster
ormake install
- installs cluster on all computers listed inclusterAD2.ini
If pvm runs on Slave computer (can be checked by echo halt | pvm
) and it is possible to log on from Master computer without password, it is necessary to check if pvm can add Slave computer into cluster (start pvm daemon). This can be done by following commands:
pvm /usr/local/share/newAD2/hostfileAD2
pvm> add slavehostname
1 successful
HOST DTID
... ...
pvm> halt
If command add
in pvm console fails, you can resolve the problem by following:
- Turn off firewall or make sure by other mean that it doesn't block pvm ports.
- If on some computers
pvm3d
is frozen, kill it and delete/tmp/pvmd.UID
file which would otherwise block its re-starting. - If you are starting pvm using
add
from the Master, pvm must not be running on the Slave.
Although, cluster can be started by command add slavehostname
, then quiting pvm console by quit
and subsequently starting newAD2, it is not necessary. Program newAD2 can start cluster by similar commands:
newAD2
newAD2> add slavehostname
...
newAD2> quit
Installation for model developer
Program newAD2 has three different levels of users:
- Standard user - this user utilizes newAD2 as installed with fixed documented models (user is not present in group
newad
). - Model developer - this user utilizes newAD2 as installed, except part where physical models are implemented
(processes
slaveAD2-OTF...
). User is in groupnewad
and can develop models and compile cluster (generateshostfileAD2...
file). - newAD2 developer - this user develops also other parts of newAD2, is in
newad
group and has access into subversion system.
If you are interested in development of your own models which you would like to compile, it is necessary to make sure that you will not affect other users. This is ensured by defining variable CLUSTERNAME
in .bashrc
, for example to value of your user name:
export CLUSTERNAME=$USER
Variable CLUSTERNAME
defines cluster (file hostfileAD2$CLUSTERNAME
)
and models (processes slaveAD2-OTF$CLUSTERNAME
), which are used during newAD2 start.
Variable SUFFIX = $(CLUSTERNAME)
defined in file Makefile
choses which cluster and models will be compiled. In this way, any number of clusters can be compiled, including unnamed with stable models, which can serve to standard users. If the variable CLUSTERNAME
is set we can compile unnamed cluster and run commands:
export CLUSTERNAME=
make install
newAD2gui