Hey, I am a relatively new user and my University recently got an academic license and installed Rosetta on our high performance computing for my PhD in de novo design after requesting. I am trying to run the abinitio demo contained within "/rosetta/3.13/gcc-9.3.0+openmpi-4.0.4/rosetta_src_2021.16.61629_bundle/main/demos/public/abinitio" but I am running into a strange error.
I used cp to copy the contents of input_files of the abinitio directory and also copied the ignore_list file (which I suppressed as I wanted to to see the log) and tried to use slurm #sbatch to run the demo. However, the log file is returning this and is essentially trapped (after hours too) at the core.scoring.AtomVDW:
core.init: (0) Checking for fconfig files in pwd and ./rosetta/flags
core.init: (0) Rosetta version: rosetta.source.release-280 r280 2021.16+release.8ee4f02 8ee4f02ac5768a8a339ffada74cb0ff5f778b3e6 https://www.rosettacommons.org 2021-04-20T20:57:38.289184
core.init: (0) command: /opt/apps/rosetta/3.13/gcc-9.3.0+openmpi-4.0.4/rosetta_src_2021.16.61629_bundle/main/source/bin/AbinitioRelax.mpi.linuxgccrelease @flags -database /opt/apps/rosetta/3.13/gcc-9.3.0+openmpi-4.0.4/rosetta_src_2021.16.61629_bundle/main/database/ -run:constant_seed -nodelay
basic.random.init_random_generator: (0) Constant seed mode, seed=1111111 seed_offset=0 real_seed=1111111
basic.random.init_random_generator: (0) RandomGenerator:init: Normal mode, seed=1111111 RG_type=mt19937
core.chemical.GlobalResidueTypeSet: (0) Finished initializing fa_standard residue type set. Created 984 residue types
core.chemical.GlobalResidueTypeSet: (0) Total time to initialize 0.82 seconds.
core.import_pose.import_pose: (0) File './input_files/1elw.pdb' automatically determined to be of type PDB
core.conformation.Conformation: (0) [ WARNING ] missing heavyatom: CG on residue GLU:NtermProteinFull 1
core.conformation.Conformation: (0) [ WARNING ] missing heavyatom: CD on residue GLU:NtermProteinFull 1
core.conformation.Conformation: (0) [ WARNING ] missing heavyatom: OE1 on residue GLU:NtermProteinFull 1
core.conformation.Conformation: (0) [ WARNING ] missing heavyatom: OE2 on residue GLU:NtermProteinFull 1
core.pack.pack_missing_sidechains: (0) packing residue number 1 because of missing atom number 6 atom name CG
core.pack.task: (0) Packer task: initialize from command line()
core.scoring.ScoreFunctionFactory: (0) SCOREFUNCTION: ref2015
core.scoring.etable: (0) Starting energy table calculation
core.scoring.etable: (0) smooth_etable: changing atr/rep split to bottom of energy well
core.scoring.etable: (0) smooth_etable: spline smoothing lj etables (maxdis = 6)
core.scoring.etable: (0) smooth_etable: spline smoothing solvation etables (max_dis = 6)
core.scoring.etable: (0) Finished calculating energy tables.
basic.io.database: (0) Database file opened: scoring/score_functions/hbonds/ref2015_params/HBPoly1D.csv
basic.io.database: (0) Database file opened: scoring/score_functions/hbonds/ref2015_params/HBFadeIntervals.csv
basic.io.database: (0) Database file opened: scoring/score_functions/hbonds/ref2015_params/HBEval.csv
basic.io.database: (0) Database file opened: scoring/score_functions/hbonds/ref2015_params/DonStrength.csv
basic.io.database: (0) Database file opened: scoring/score_functions/hbonds/ref2015_params/AccStrength.csv
basic.io.database: (0) Database file opened: scoring/score_functions/rama/fd/all.ramaProb
basic.io.database: (0) Database file opened: scoring/score_functions/rama/fd/prepro.ramaProb
basic.io.database: (0) Database file opened: scoring/score_functions/omega/omega_ppdep.all.txt
basic.io.database: (0) Database file opened: scoring/score_functions/omega/omega_ppdep.gly.txt
basic.io.database: (0) Database file opened: scoring/score_functions/omega/omega_ppdep.pro.txt
basic.io.database: (0) Database file opened: scoring/score_functions/omega/omega_ppdep.valile.txt
basic.io.database: (0) Database file opened: scoring/score_functions/P_AA_pp/P_AA
basic.io.database: (0) Database file opened: scoring/score_functions/P_AA_pp/P_AA_n
core.scoring.P_AA: (0) shapovalov_lib::shap_p_aa_pp_smooth_level of 1( aka low_smooth ) got activated.
basic.io.database: (0) Database file opened: scoring/score_functions/P_AA_pp/shapovalov/10deg/kappa131/a20.prop
basic.io.database: (0) Database file opened: scoring/score_functions/elec_cp_reps.dat
core.scoring.elec.util: (0) Read 40 countpair representative atoms
core.pack.dunbrack.RotamerLibrary: (0) shapovalov_lib_fixes_enable option is true.
core.pack.dunbrack.RotamerLibrary: (0) shapovalov_lib::shap_dun10_smooth_level of 1( aka lowest_smooth ) got activated.
core.pack.dunbrack.RotamerLibrary: (0) Binary rotamer library selected: /users/40344278/.rosetta/database/rotamer/shapovalov/StpDwn_0-0-0/Dunbrack10.lib.bin
core.pack.dunbrack.RotamerLibrary: (0) Using Dunbrack library binary file '/users/40344278/.rosetta/database/rotamer/shapovalov/StpDwn_0-0-0/Dunbrack10.lib.bin'.
core.pack.dunbrack.RotamerLibrary: (0) Dunbrack 2010 library took 0.11 seconds to load from binary
core.pack.pack_rotamers: (0) built 17 rotamers at 1 positions.
core.pack.interaction_graph.interaction_graph_factory: (0) Instantiating DensePDInteractionGraph
core.chemical.GlobalResidueTypeSet: (0) Finished initializing centroid residue type set. Created 69 residue types
core.chemical.GlobalResidueTypeSet: (0) Total time to initialize 0.02 seconds.
protocols.abinitio.AbrelaxApplication: (0) read fasta sequence: 117 residues
EQVNELKEKGNKALSVGNIDDALQCYSEAIKLDPHNHVLYSNRSAAYAKKGDYQKAYEDGCKTVDLKPDWGKGYSRKAAALEFLNRFEEAKRTYEEGLKHEANNPQLKEGLQNMEAR
core.import_pose.import_pose: (0) File './input_files/1elw.pdb' automatically determined to be of type PDB
core.conformation.Conformation: (0) [ WARNING ] missing heavyatom: CG on residue GLU:NtermProteinFull 1
core.conformation.Conformation: (0) [ WARNING ] missing heavyatom: CD on residue GLU:NtermProteinFull 1
core.conformation.Conformation: (0) [ WARNING ] missing heavyatom: OE1 on residue GLU:NtermProteinFull 1
core.conformation.Conformation: (0) [ WARNING ] missing heavyatom: OE2 on residue GLU:NtermProteinFull 1
core.pack.pack_missing_sidechains: (0) packing residue number 1 because of missing atom number 6 atom name CG
core.pack.task: (0) Packer task: initialize from command line()
core.scoring.ScoreFunctionFactory: (0) SCOREFUNCTION: ref2015
core.pack.pack_rotamers: (0) built 17 rotamers at 1 positions.
core.pack.interaction_graph.interaction_graph_factory: (0) Instantiating DensePDInteractionGraph
protocols.evaluation.ChiWellRmsdEvaluatorCreator: (0) Evaluation Creator active ...
core.io.fragments: (0) reading fragments from file: ./input_files/aa1elwA09_05.200_v1_3 ...
core.io.fragments: (0) rosetta++ fileformat detected! Calling legacy reader...
core.fragments.ConstantLengthFragSet: (0) finished reading top 25 9mer fragments from file ./input_files/aa1elwA09_05.200_v1_3
core.io.fragments: (0) reading fragments from file: ./input_files/aa1elwA03_05.200_v1_3 ...
core.io.fragments: (0) rosetta++ fileformat detected! Calling legacy reader...
core.fragments.ConstantLengthFragSet: (0) finished reading top 200 3mer fragments from file ./input_files/aa1elwA03_05.200_v1_3
core.fragment: (0) compute strand/loop fractions for 117 residues...
protocols.abinitio.AbrelaxApplication: (0) run ClassicAbinitio.....
basic.io.database: (0) Database file opened: scoring/score_functions/EnvPairPotential/env_log.txt
basic.io.database: (0) Database file opened: scoring/score_functions/EnvPairPotential/cbeta_den.txt
basic.io.database: (0) Database file opened: scoring/score_functions/EnvPairPotential/pair_log.txt
basic.io.database: (0) Database file opened: scoring/score_functions/EnvPairPotential/cenpack_log.txt
basic.io.database: (0) Database file opened: scoring/score_functions/SecondaryStructurePotential/phi.theta.36.HS.resmooth
basic.io.database: (0) Database file opened: scoring/score_functions/SecondaryStructurePotential/phi.theta.36.SS.resmooth
core.scoring.ScoreFunction: (0) ATOM_VDW set to CENTROID_ROT_MIN
core.scoring.ramachandran: (0) shapovalov_lib::shap_rama_smooth_level of 4( aka highest_smooth ) got activated.
basic.io.database: (0) Database file opened: scoring/score_functions/rama/shapovalov/kappa25/all.ramaProb
basic.io.database: (0) Database file opened: scoring/score_functions/centroid_smooth/cen_rot_pair_params.txt
basic.io.database: (0) Database file opened: scoring/score_functions/centroid_smooth/cen_rot_env_params.txt
basic.io.database: (0) Database file opened: scoring/score_functions/centroid_smooth/cen_rot_cbeta_params.txt
basic.io.database: (0) Database file opened: scoring/score_functions/centroid_smooth/cen_rot_pair_ang_params.txt
core.scoring.AtomVDW: (0) Openning alternative vdw file: /opt/apps/rosetta/3.13/gcc-9.3.0+openmpi-4.0.4/rosetta_src_2021.16.61629_bundle/main/database/chemical/atom_
When I run the same demo input @flag file from the command line (using \$ROSETTA/source/bin/AbinitioRelax.mpi.linuxgccrelease @flags -database $ROSETTA3_DB -run:constant_seed -nodelay 2>&1) I can see that it has no issues with accessing the core.scoring.AtomVDW file:
core.init: (0) Checking for fconfig files in pwd and ./rosetta/flags
core.init: (0) Rosetta version: rosetta.source.release-280 r280 2021.16+release.8ee4f02 8ee4f02ac5768a8a339ffada74cb0ff5f778b3e6 https://www.rosettacommons.org 2021-04-20T20:57:38.289184
core.init: (0) command: /opt/apps/rosetta/3.13/gcc-9.3.0+openmpi-4.0.4/rosetta_src_2021.16.61629_bundle/main/source/bin/AbinitioRelax.mpi.linuxgccrelease @flags -database /opt/apps/rosetta/3.13/gcc-9.3.0+openmpi-4.0.4/rosetta_src_2021.16.61629_bundle/main/database/ -run:constant_seed -nodelay
basic.random.init_random_generator: (0) Constant seed mode, seed=1111111 seed_offset=0 real_seed=1111111
basic.random.init_random_generator: (0) RandomGenerator:init: Normal mode, seed=1111111 RG_type=mt19937
core.chemical.GlobalResidueTypeSet: (0) Finished initializing fa_standard residue type set. Created 984 residue types
core.chemical.GlobalResidueTypeSet: (0) Total time to initialize 0.76 seconds.
core.import_pose.import_pose: (0) File './input_files/1elw.pdb' automatically determined to be of type PDB
core.conformation.Conformation: (0) [ WARNING ] missing heavyatom: CG on residue GLU:NtermProteinFull 1
core.conformation.Conformation: (0) [ WARNING ] missing heavyatom: CD on residue GLU:NtermProteinFull 1
core.conformation.Conformation: (0) [ WARNING ] missing heavyatom: OE1 on residue GLU:NtermProteinFull 1
core.conformation.Conformation: (0) [ WARNING ] missing heavyatom: OE2 on residue GLU:NtermProteinFull 1
core.pack.pack_missing_sidechains: (0) packing residue number 1 because of missing atom number 6 atom name CG
core.pack.task: (0) Packer task: initialize from command line()
core.scoring.ScoreFunctionFactory: (0) SCOREFUNCTION: ref2015
core.scoring.etable: (0) Starting energy table calculation
core.scoring.etable: (0) smooth_etable: changing atr/rep split to bottom of energy well
core.scoring.etable: (0) smooth_etable: spline smoothing lj etables (maxdis = 6)
core.scoring.etable: (0) smooth_etable: spline smoothing solvation etables (max_dis = 6)
core.scoring.etable: (0) Finished calculating energy tables.
basic.io.database: (0) Database file opened: scoring/score_functions/hbonds/ref2015_params/HBPoly1D.csv
basic.io.database: (0) Database file opened: scoring/score_functions/hbonds/ref2015_params/HBFadeIntervals.csv
basic.io.database: (0) Database file opened: scoring/score_functions/hbonds/ref2015_params/HBEval.csv
basic.io.database: (0) Database file opened: scoring/score_functions/hbonds/ref2015_params/DonStrength.csv
basic.io.database: (0) Database file opened: scoring/score_functions/hbonds/ref2015_params/AccStrength.csv
basic.io.database: (0) Database file opened: scoring/score_functions/rama/fd/all.ramaProb
basic.io.database: (0) Database file opened: scoring/score_functions/rama/fd/prepro.ramaProb
basic.io.database: (0) Database file opened: scoring/score_functions/omega/omega_ppdep.all.txt
basic.io.database: (0) Database file opened: scoring/score_functions/omega/omega_ppdep.gly.txt
basic.io.database: (0) Database file opened: scoring/score_functions/omega/omega_ppdep.pro.txt
basic.io.database: (0) Database file opened: scoring/score_functions/omega/omega_ppdep.valile.txt
basic.io.database: (0) Database file opened: scoring/score_functions/P_AA_pp/P_AA
basic.io.database: (0) Database file opened: scoring/score_functions/P_AA_pp/P_AA_n
core.scoring.P_AA: (0) shapovalov_lib::shap_p_aa_pp_smooth_level of 1( aka low_smooth ) got activated.
basic.io.database: (0) Database file opened: scoring/score_functions/P_AA_pp/shapovalov/10deg/kappa131/a20.prop
basic.io.database: (0) Database file opened: scoring/score_functions/elec_cp_reps.dat
core.scoring.elec.util: (0) Read 40 countpair representative atoms
core.pack.dunbrack.RotamerLibrary: (0) shapovalov_lib_fixes_enable option is true.
core.pack.dunbrack.RotamerLibrary: (0) shapovalov_lib::shap_dun10_smooth_level of 1( aka lowest_smooth ) got activated.
core.pack.dunbrack.RotamerLibrary: (0) Binary rotamer library selected: /users/40344278/.rosetta/database/rotamer/shapovalov/StpDwn_0-0-0/Dunbrack10.lib.bin
core.pack.dunbrack.RotamerLibrary: (0) Using Dunbrack library binary file '/users/40344278/.rosetta/database/rotamer/shapovalov/StpDwn_0-0-0/Dunbrack10.lib.bin'.
core.pack.dunbrack.RotamerLibrary: (0) Dunbrack 2010 library took 0.11 seconds to load from binary
core.pack.pack_rotamers: (0) built 17 rotamers at 1 positions.
core.pack.interaction_graph.interaction_graph_factory: (0) Instantiating DensePDInteractionGraph
core.chemical.GlobalResidueTypeSet: (0) Finished initializing centroid residue type set. Created 69 residue types
core.chemical.GlobalResidueTypeSet: (0) Total time to initialize 0.02 seconds.
protocols.abinitio.AbrelaxApplication: (0) read fasta sequence: 117 residues
EQVNELKEKGNKALSVGNIDDALQCYSEAIKLDPHNHVLYSNRSAAYAKKGDYQKAYEDGCKTVDLKPDWGKGYSRKAAALEFLNRFEEAKRTYEEGLKHEANNPQLKEGLQNMEAR
core.import_pose.import_pose: (0) File './input_files/1elw.pdb' automatically determined to be of type PDB
core.conformation.Conformation: (0) [ WARNING ] missing heavyatom: CG on residue GLU:NtermProteinFull 1
core.conformation.Conformation: (0) [ WARNING ] missing heavyatom: CD on residue GLU:NtermProteinFull 1
core.conformation.Conformation: (0) [ WARNING ] missing heavyatom: OE1 on residue GLU:NtermProteinFull 1
core.conformation.Conformation: (0) [ WARNING ] missing heavyatom: OE2 on residue GLU:NtermProteinFull 1
core.pack.pack_missing_sidechains: (0) packing residue number 1 because of missing atom number 6 atom name CG
core.pack.task: (0) Packer task: initialize from command line()
core.scoring.ScoreFunctionFactory: (0) SCOREFUNCTION: ref2015
core.pack.pack_rotamers: (0) built 17 rotamers at 1 positions.
core.pack.interaction_graph.interaction_graph_factory: (0) Instantiating DensePDInteractionGraph
protocols.evaluation.ChiWellRmsdEvaluatorCreator: (0) Evaluation Creator active ...
core.io.fragments: (0) reading fragments from file: ./input_files/aa1elwA09_05.200_v1_3 ...
core.io.fragments: (0) rosetta++ fileformat detected! Calling legacy reader...
core.fragments.ConstantLengthFragSet: (0) finished reading top 25 9mer fragments from file ./input_files/aa1elwA09_05.200_v1_3
core.io.fragments: (0) reading fragments from file: ./input_files/aa1elwA03_05.200_v1_3 ...
core.io.fragments: (0) rosetta++ fileformat detected! Calling legacy reader...
core.fragments.ConstantLengthFragSet: (0) finished reading top 200 3mer fragments from file ./input_files/aa1elwA03_05.200_v1_3
core.fragment: (0) compute strand/loop fractions for 117 residues...
protocols.abinitio.AbrelaxApplication: (0) run ClassicAbinitio.....
basic.io.database: (0) Database file opened: scoring/score_functions/EnvPairPotential/env_log.txt
basic.io.database: (0) Database file opened: scoring/score_functions/EnvPairPotential/cbeta_den.txt
basic.io.database: (0) Database file opened: scoring/score_functions/EnvPairPotential/pair_log.txt
basic.io.database: (0) Database file opened: scoring/score_functions/EnvPairPotential/cenpack_log.txt
basic.io.database: (0) Database file opened: scoring/score_functions/SecondaryStructurePotential/phi.theta.36.HS.resmooth
basic.io.database: (0) Database file opened: scoring/score_functions/SecondaryStructurePotential/phi.theta.36.SS.resmooth
core.scoring.ScoreFunction: (0) ATOM_VDW set to CENTROID_ROT_MIN
core.scoring.ramachandran: (0) shapovalov_lib::shap_rama_smooth_level of 4( aka highest_smooth ) got activated.
basic.io.database: (0) Database file opened: scoring/score_functions/rama/shapovalov/kappa25/all.ramaProb
basic.io.database: (0) Database file opened: scoring/score_functions/centroid_smooth/cen_rot_pair_params.txt
basic.io.database: (0) Database file opened: scoring/score_functions/centroid_smooth/cen_rot_env_params.txt
basic.io.database: (0) Database file opened: scoring/score_functions/centroid_smooth/cen_rot_cbeta_params.txt
basic.io.database: (0) Database file opened: scoring/score_functions/centroid_smooth/cen_rot_pair_ang_params.txt
core.scoring.AtomVDW: (0) Openning alternative vdw file: /opt/apps/rosetta/3.13/gcc-9.3.0+openmpi-4.0.4/rosetta_src_2021.16.61629_bundle/main/database/chemical/atom_type_sets/centroid_rot//min.txt
core.scoring.ScoreFunction: (0) ATOM_VDW set to CENTROID_ROT_MIN
core.scoring.ScoreFunctionFactory: (0) SCOREFUNCTION: ref2015
protocols.jobdist.JobDistributors: (0) Node: 0 next_job()
protocols.jobdist.JobDistributors: (0) Master Node -- Waiting for job request; tag_ = 1
I return errors when setting the number of nodes from srun and the command line and would prefer to use sbatch.
The paths that have been set are (with an appropriate conda environment setup afterwards):
- module load rosetta/3.13/gcc-9.3.0+openmpi-4.0.4
- export PATH=/opt/apps/rosetta/3.13/gcc-9.3.0+openmpi-4.0.4/rosetta_src_2021.16.61629_bundle/main/source/bin
- export PYTHONPATH=/opt/apps/rosetta/3.13/gcc-9.3.0+openmpi-4.0.4/rosetta_src_2021.16.61629_bundle/main/pyrosetta_scripts/apps
- export ROSETTA=/opt/apps/rosetta/3.13/gcc-9.3.0+openmpi-4.0.4/rosetta_src_2021.16.61629_bundle/main
-
export ROSETTA3_DB=/opt/apps/rosetta/3.13/gcc-9.3.0+openmpi-4.0.4/rosetta_src_2021.16.61629_bundle/main/database/
My slurm job input file contains:
#!/bin/bash
#SBATCH --job-name=ros_ab
#SBATCH --mem-per-cpu=4G
#SBATCH --nodes=3
#SBATCH --ntasks-per-node=12
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --partition=k2-hipri
cd /users/[studentID]/
source set_Rosetta3.13.sh
cd /users/[studentID]/abinitio_test/
\$ROSETTA/source/bin/AbinitioRelax.mpi.linuxgccrelease @flags -database \$ROSETTA3_DB -run:constant_seed -nodelay 2>&1
# | egrep -vf ignore_list \
# >> log
What am I doing wrong for the file not to be able to be accessed from slurm but can be accessed from the command line? Sorry if this is a trivial issue, I am not experienced in this so far. The build and installation appears to have went well and I can run .py files, but using the actual demo files has went wrong for whatever reason. Thanks for any help and no worries if you are all busy.
When you run under MPI, you need to use the mpi job launcher for your particular MPI system (this program is often called `mpirun` or `mpiexec`). That's what launches the mulitple copies of the program which then use MPI to talk to each other.
Right now it looks like you're only launching a single process. When run MPI-style, the first process is the coordinator ("master") node, and doesn't do any scientific computation itself. Instead, it coordinates the other nodes and makes sure jobs are processed and allocated correctly. That's the issue you're running into. You have the coordinator node coordinating the activities, but you don't have any processing nodes actually doing the processing.
If you use mpirun/mpiexec (with the proper settings for your MPI system) to launch the AbinitioRelax application, then you should have enough nodes for the coordinator node to distribute work.
(Note that which command you use and what arguments you need to give it depend on the particular flavor of MPI your cluster system is running -- talk to your cluster administrators to learn the proper way to launch MPI jobs on your particular cluster.)
That makes perfect sense for why it would not run, I am glad you listed `mpiexec` as that is the one our system is using and other `mpi` such as `mpirun` do not seem to exist.
Thank you for the detailed explanation that has resolved the issue. I am not sure what is the correct etiquette on replying when your issues have been solved but I have changed the question type from "unsolved" to "solved" and hope this brief thank you is acceptable for this particular forum.