What is PBS Pro?
Installation
and Setting up of environment (cluster with multiple nodes)
Creating vnode in PBS Pro:
We can create multiple vnodes in a single node which contain some part of resources in a node. We can execute job on this vnodes with specified allocated resources. We can create vnode using qmgr command which is command line interface to PBS server. We can use command given below to create vnode using qmgr.
Portable
Batch System (PBS) is a software which is used in cluster computing to schedule
jobs on multiple nodes. PBS was started as contract project by NASA.
PBS
is available in three different versions as below
1)
Torque: Terascale Open-source Resource and QUEue Manager (Torque) is developed
from OpenPBS. It is developed and maintain by Adaptive Computing Enterprises.
It is used as a distributed resource manager can perform well when integrated
with Maui cluster scheduler to improve performance.
2)
PBS Professional (PBS Pro): It is commercial version of PBS offered by Altair
Engineering.
3) OpenPBS: It is open source version released in 1998 developed by NASA. It is not actively developed.
In
this article we are going to concentrate on tutorial of PBS Pro it is similar
to some extent with Torque.
Fig. 1.1 PBS complex cluster with
eight execution host
|
PBS contain three basic
units server, MoM (execution host), scheduler.
1) Server: It is heart of the PBS, with executable named “pbs_server”. It
uses IP network to communicate with the MoMs. PBS server create a batch job,
modify the job requested from different MoMs. It keeps track of all resources
available, assigned in the PBS complex from different MoMs. It will also
monitor the PBS license for jobs. If your license expires it will throw an error.
2) Scheduler: PBS scheduler uses various algorithms to decide when job
should get executed on which node or vnode by using detail of resources
available from server. It has executable as “pbs_sched”.
3) MoM: MoM is the mother of all execution job with executable “pbs_mom”.
When MoM gets job from server it will actually execute that job on the host.
Each node must have MoM running to get participate in execution.
Extract
compressed software of PBS Pro and go the path of extracted folder it
contain “INSTALL” file, make that file executable you may use command like “chmod
+x ./INSTALL”. As shown in the image below run this executable. It will ask for
the “execution directory” where you want to store the executable (such as qsub,
pbsnodes, qdel etc.) used for different PBS operations and “home directory”
which contain different configuration files. Keep both as default for
simplicity.
There are three kind of installation available as shown
in figure:![]() |
Fig. 1.2 PBS installation |
1)
Server node: PBS server, scheduler, MoM and commands are installed on this
node. PBS server will keep track of all execution MoMs present in the cluster.
It will schedule jobs on this execution nodes. As MoM and commands are also
installed on server node it can be used to submit and execute the jobs.
2)
Execution node: This type installs MoM and commands. This nodes are added as
available nodes for execution in a cluster. They are also allowed to submit the
jobs at server side with specific permission by server as we are going to see
below. They are not involved in scheduling. This kind of installation ask for PBS
server which is used to submit jobs, get status of jobs etc.
3) Client node: This are the
nodes which are only allowed to submit a PBS job at server with specific
permission by the server and allowed to see the status of the jobs. They are
not involved in execution or scheduling.
Creating vnode in PBS Pro:
We can create multiple vnodes in a single node which contain some part of resources in a node. We can execute job on this vnodes with specified allocated resources. We can create vnode using qmgr command which is command line interface to PBS server. We can use command given below to create vnode using qmgr.
Qmgr: create node Vnode1,Vnode2 resources_available.ncpus=8, resources_available.mem=10gb, resources_available.ngpus=1, sharing=default_excl
The command above will create two vnodes named Vnode1 and Vnode2 with 8 cpus cores, 10gb of memory and 1 GPU with sharing mode as default_excl which means this vnode can execute exclusively only one job at a time independent of number of resources free. This sharing mode can be default_shared which means any number of jobs can run on that vnode until all resources are busy. To know more about all attributes which can be used with vnode creation are available in PBS Pro reference guide.
You can also create a file in "/var/spool/PBS/mom_priv/config.d/" this folder with any name you want I prefer hostname-vnode with sample given below. It will select all files even temporary files with (~) and replace configuration for same vnode so delete unnecessary files to get proper configuration of vnodes.
e.g.
$configversion 2 hostname:resources_available.ncpus=0hostname:resources_available.mem=0hostname:resources_available.ngpus=0hostname[0]:resources_available.ncpus=8Here in this example we assigned default node configuration to resource available as 0 because by default it will detect and allocate all available resources to default node with sharing attribute as is default_shared. Which cause problem as all the jobs will by default get scheduled on that default vnode because its sharing type is default_shared. If you want to schedule jobs on your customized vnodes you should allocate resources available as 0 on default vnode. Every time whenever you restart the PBS server this vnodes get create unlike vnodes created manually using command line.
hostname[0]:resources_available.mem=16gbhostname[0]:resources_available.ngpus=1
hostname[0]:sharing=default_exclhostname[1]:resources_available.ncpus=8hostname[1]:resources_available.mem=16gbhostname[1]:resources_available.ngpus=1
hostname[1]:sharing=default_exclhostname[2]:resources_available.ncpus=8hostname[2]:resources_available.mem=16gbhostname[2]:resources_available.ngpus=1
hostname[2]:sharing=default_exclhostname[3]:resources_available.ncpus=8hostname[3]:resources_available.mem=16gbhostname[3]:resources_available.ngpus=1
hostname[3]:sharing=default_excl
PBS get status:
get status of Jobs:
qstat will give details about jobs there states etc.
useful options:
To print detail about all jobs which are running or in hold state: qstat -a
To print detail about subjobs in JobArray which are running or in hold state: qstat -ta
To print all finished jobs: qstat -x
get status of PBS nodes and vnodes:
"pbsnode -a" command will provide list of all nodes present in PBS complex with there resources available, assigned, status etc.
To get details of all nodes and vnodes you created use "pbsnodes -av" command.
You can also specify node or vnode name to get detail information of that specific node or vnode.
e.g.
pbsnodes wolverine (here wolverine is hostname of the node in PBS complex which is mapped with IP address in /etc/hosts file)
Job
submission (qsub):
If you specify resources to Job Array each subjob will require resources specified. JobId in Job Array start with JobArrayID[0], JobArrayID[1], .....,, JobArrayID[n-1]
Submit dependant jobs:
In some cases you may require job which should run after successful or unsuccessful completion of some specified jobs for that PBS provide some options such as
This specified job will start only after successful completion of job with job ID "316.megamind". Like afterok PBS has other options such as beforeok, beforenotok to , afternotok. You may find all this details in the man page of qsub.
Submit Job with priority:
There are two ways using which we can set priority to jobs which are going to execute.
1) Using single queue with different jobs with different priority:
To change sequence of jobs queued in a execution queue open "$PBS_HOME/sched_priv/sched_config" file, normally $PBS_HOME is present in "/var/spool/PBS/" folder. Open this file and uncomment the line below if present otherwise add it.
job_sort_key : "job_priority HIGH"
After saving this file you will need to restart the pbs_sched daemon on head node you may use command below
service pbs restart
After completing this task you have to submit the job with -p option to specify priority of job within queue. This value may range between (-1024) to 1023, where -1024 is the lowest priority and 1023 is the highest priority in the queue.
e.g.
2) Using different queues with specified priority: We are going to discuss this point in PBS Queue section.
PBS Pro can manage multiple queue as per users requirement. By default every job is queued in "workq" for execution. There are two types of queue are available execution and routing queue. Jobs in execution queue are used by pbs server for execution. Jobs in routing queue can not be executed they can be redirected to execution queue or another routing queue by using command qmove command. By default queue "workq" is an execution queue. The sequence of job in queue may change by using priority defined while job submission as specified above in job submission section.
Useful qmgr commands:
First type qmgr which is Manager interface of PBS Pro.
To create queue:
To set type of queue you created:
OR
To enable queue:
To set priority of queue:
Jobs in queue with higher priority will get first preference. After completion of all jobs in the queue with higher priority jobs in lower priority queue are scheduled. There is huge probability of job starvation in queue with lower priority.
To start queue:
To activate all queue (present at particular node):
To set queue for specified users: You require to set acl_user_enable attribute to true which indicate PBS to only allow user present in acl_users list to submit the job.
To set users permitted (to submit job in a queue):
To delete queues we created:
To see details of all queue status:
You may specify specific queue name: qstat -Q test2
To see full details of all queue: qstat -Q -f
You may specify specific queue name: qstat -Q -f test2
PBS MoM will submit jobs to the PBS server. Server
maintain queue of jobs by default all jobs are submitted to default queue named
“workq”. You may create multiple queues by using “qmgr” command which is administrator
interface mainly used to create, delete & modify queues and vnodes. PBS
server will decide which job to be scheduled on which node or vnode based on
scheduling policy and privileges set by user. To schedule jobs server will
continuously ping to all MoMs in the PBS complex to get detail of resources
available and assigned. PBS assigns unique job identifier to each and every job called JobID.
For job submission PBS uses “qsub” command. It has
syntax as shown below
qsub
script
Here script may be a shell (sh, csh, tchs, ksh,
bash) script. PBS by default uses /bin/sh. You may refer simple script given
below
#!/bin/sh
echo “This is PBS job”
echo “This is PBS job”
sleep 100
When PBS completes execution of job it will store errors in file name with JobName.e{JobID} e.g. Job1.e1492
Output with file name
JobName.o{JobID} e.g. Job1.o1492
By default it will store this files in the current working directory (can be seen by pwd command). You can change this location by giving path with -o option.
you may specify job name with -N option while submitting the job
qsub -N firstJob ./test.sh
If you don't specify job name it will store files by replacing JobName with script name.
When PBS completes execution of job it will store errors in file name with JobName.e{JobID} e.g. Job1.e1492
Output with file name
JobName.o{JobID} e.g. Job1.o1492
By default it will store this files in the current working directory (can be seen by pwd command). You can change this location by giving path with -o option.
you may specify job name with -N option while submitting the job
qsub -N firstJob ./test.sh
If you don't specify job name it will store files by replacing JobName with script name.
e.g. qsub ./test.sh this command will store results in file with test.sh.e1493 and test.sh.o.1493 in current working directory.
OR
qsub -N firstJob -o /home/user1/ ./test.sh this command will store results in file with test.sh.e1493 and test.sh.o.1493 in /home/user1/ directory.
If submitted job terminate abnormally (errors in job is not abnormal, this errors get stored in JobName.e{JobID} file) it will store its error and output files in "/var/spool/PBS/undelivered/" folder.
Useful Options:
Select resources:
e.g.
This Job will selects 2 copies with 3 cpus, 1 gpu and 2gb memory which mean it will select 6 cpus, 2 gpus and 4 gb ram.
This job will select one node specified with hostname.
To select multiple nodes you may use command given below
Submit multiple jobs with same script (JobArray):OR
qsub -N firstJob -o /home/user1/ ./test.sh this command will store results in file with test.sh.e1493 and test.sh.o.1493 in /home/user1/ directory.
If submitted job terminate abnormally (errors in job is not abnormal, this errors get stored in JobName.e{JobID} file) it will store its error and output files in "/var/spool/PBS/undelivered/" folder.
Useful Options:
Select resources:
qsub -l select="chunks":ncpus=3:ngpus=1:mem=2gb script
e.g.
qsub -l select=2:ncpus=3:ngpus=1:mem=2gb /home/titan/PBS/scripts/in.sh
This Job will selects 2 copies with 3 cpus, 1 gpu and 2gb memory which mean it will select 6 cpus, 2 gpus and 4 gb ram.
qsub -l nodes=megamind:ncpus=3 /home/titan/PBS/input/in.sh
This job will select one node specified with hostname.
To select multiple nodes you may use command given below
qsub -l nodes=megamind+titan:ncpus=3 /home/titan/PBS/input/in.sh
qsub -J 1-20 script
If you specify resources to Job Array each subjob will require resources specified. JobId in Job Array start with JobArrayID[0], JobArrayID[1], .....,, JobArrayID[n-1]
Submit dependant jobs:
In some cases you may require job which should run after successful or unsuccessful completion of some specified jobs for that PBS provide some options such as
qsub -W depend=afterok:316.megamind /home/titan/PBS/input/in.sh
This specified job will start only after successful completion of job with job ID "316.megamind". Like afterok PBS has other options such as beforeok, beforenotok to , afternotok. You may find all this details in the man page of qsub.
Submit Job with priority:
There are two ways using which we can set priority to jobs which are going to execute.
1) Using single queue with different jobs with different priority:
To change sequence of jobs queued in a execution queue open "$PBS_HOME/sched_priv/sched_config" file, normally $PBS_HOME is present in "/var/spool/PBS/" folder. Open this file and uncomment the line below if present otherwise add it.
job_sort_key : "job_priority HIGH"
After saving this file you will need to restart the pbs_sched daemon on head node you may use command below
service pbs restart
After completing this task you have to submit the job with -p option to specify priority of job within queue. This value may range between (-1024) to 1023, where -1024 is the lowest priority and 1023 is the highest priority in the queue.
e.g.
qsub -p 100 ./X.shIn this case PBS will execute jobs as explain in the diagram given below
qsub -p 101 ./Y.sh
qsub -p 102 ./Z.sh
2) Using different queues with specified priority: We are going to discuss this point in PBS Queue section.
In this example all jobs in queue 2 will complete first
then queue 3 then queue 1, since priority of queue 2 > queue 3 > queue 1.
Because of this job execution flow is as shown
below
J4=> J5=>
J6=>J7=> J8=> J9=> J1=> J2=> J3
PBS Queue:PBS Pro can manage multiple queue as per users requirement. By default every job is queued in "workq" for execution. There are two types of queue are available execution and routing queue. Jobs in execution queue are used by pbs server for execution. Jobs in routing queue can not be executed they can be redirected to execution queue or another routing queue by using command qmove command. By default queue "workq" is an execution queue. The sequence of job in queue may change by using priority defined while job submission as specified above in job submission section.
Useful qmgr commands:
First type qmgr which is Manager interface of PBS Pro.
To create queue:
Qmgr: create queue test2
To set type of queue you created:
Qmgr: set queue test2 queue_type=execution
OR
Qmgr: set queue test2 queue_type=route
To enable queue:
Qmgr: set queue test2 enabled=True
To set priority of queue:
Qmgr: set queue test2 priority=50
Jobs in queue with higher priority will get first preference. After completion of all jobs in the queue with higher priority jobs in lower priority queue are scheduled. There is huge probability of job starvation in queue with lower priority.
To start queue:
Qmgr: set queue test2 started = True
To activate all queue (present at particular node):
Qmgr: active queue @default
To set queue for specified users: You require to set acl_user_enable attribute to true which indicate PBS to only allow user present in acl_users list to submit the job.
Qmgr: set queue test2 acl_user_enable=True
To set users permitted (to submit job in a queue):
Qmgr: set queue test2 acl_users="user1@..,user2@..,user3@.."(in place of .. you have to specify hostname of compute node in PBS complex. Only user name without hostname will allow users (with same name) to submit job from all nodes (permitted to submit job) in a PBS Complex).
To delete queues we created:
Qmgr: delete queue test2
To see details of all queue status:
qstat -Q
You may specify specific queue name: qstat -Q test2
To see full details of all queue: qstat -Q -f
You may specify specific queue name: qstat -Q -f test2
0 comments:
Post a comment