I have a working mpi compile of Rosetta 3.2, and intend to use AbinitioRelax and relax on a cluster.
Things are currently working:
- with NO EXTRA PARAMETERS in my flags file (so same flags file as for single processor). Note that I have "-run:constant_seed" and "-run:jran 1111111" in my flags file.
- using "mpirun -np 8 scriptfile" (looks like one core is used for management, and 7 used for computations, and speed improvement is ~6X versus non-mpi version)
My question relates to potential JD2 parameters I could use in my flags file.
From the 3.2 "relax documentation" : "All JD2 options apply (see JD2 Documentation ofr more details)"
From the 3.2 "AbinitioRelax documentation" : there is actually no mention of JD2
From the 3.2 "How To Use The New Job Distributor" :
"There are four MPI flavors to choose from.
1) MPIWorkPartitionJobDistributor: this one is MPI in name only. Activate it with jd2::mpi_work_partition_job_distributor. Here, each processor looks at the job list and determines which jobs are its own from how many other processors there are (in other words, divide number of jobs by number of processors; each processor does that many). Its files contain an example. This KD has the best potential efficiency with MPI but isn't useful for big stuff, because you have to load-balance yourself by setting it up so that (njobs % nprocs) is either zero or close to nprocs. It is recommended only for small jobs, or jobs with tightly controlled runtimes. I use it for generating a small number of trajectories (100 jobs on 100 processors) to determine where to set my filters, then run large jobs under the next jd.
2) MPIWorkPoolJobDistributor: this is the default in MPI mode. Here, one node does no Rosetta work and instead controls all the other nodes. All other nodes request jobs from the head node, do them, report, and then get a new one until all the jobs are done. If you only have one processor, then you get only a head node, and no work is done: make sure you're using MPI properly! This one has "blocking" output - only one job can write to disk at a time, to prevent silent files from being corrupted. It's a good choice if the runtime of each individual job is relatively large, so that output happens sparsely (because then the blocking is a non-issue). If your jobs are very short and write to disk often, or you have a huge number of processors and they write often just because there's a ton of them, this job distributor will be inefficient. As of this writing, this is the job distributor of choice when writing PDB files.
3) MPIFileBufJobDistributor: this is the other default for MPI. It allocates two nodes to non-rosetta jobs. One head node as before and one node dedicated to managing output. It is nonblocking, so it is the best choice if job completion occurs often relative to how long filesystem writes take (short job times or huge numbers of processors or both). At the moment it works only with silent file output. It is the best choice if you have short job times or large (many thousands) of processors."
1. I assume I am using MPIWorkPoolJobDistributor (based on documentation and the fact that I get 7 silent files with "mpirun -np 8")
2. Do flags like "jd2::mpi_work_partition_job_distributor" apply to AbinitioRelax and relax?
3. If so, what are the other flags I could try ? (i.e. where do I find additional docs... no mention of available jd2 flags from "relax -help"
4. What is the fourth MPI flavour mentioned in the docs ? ( "How To Use The New Job Distributor" )
5. Any suggestions for improved setting of distributed Abinitio and relax runs (I have attached my flags and mpi scripts) ?