Rosetta  2020.46
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
Public Member Functions | Protected Member Functions | Private Attributes | List of all members
protocols::jd3::JobQueen Class Referenceabstract

The JobQueen class (think of a queen bee) has quite a few responsibilities: More...

#include <JobQueen.hh>

Inheritance diagram for protocols::jd3::JobQueen:
Inheritance graph
[legend]

Public Member Functions

 JobQueen ()
 
 ~JobQueen () override
 
JobDigraphOP create_and_set_initial_job_dag ()
 Creates the initial job dag using create_initial_job_dag function and sets it to the JQ. More...
 
void note_job_completed_and_track (LarvalJobCOP job, JobStatus status, core::Size nresults)
 Call note_job_completed in derived classes and track the job using the JobTracker. More...
 
LarvalJobs determine_job_list_and_track (JobDAGNodeID job_dag_node_index, core::Size max_njobs)
 Call determine_job_list in derived classes and track the jobs using the JobTracker. More...
 
virtual std::string job_definition_xsd () const =0
 All JobQueens must describe their job input XML format in the form of an XSD (XML Schema Definition), and they must validate their job input files against their XSDs. If the JobDistributor is awakened with the flag "jd3::output_job_xsd <output file>" on the command line, then the JobDistributor will write out the XSD to the output file. More...
 
virtual std::string resource_definition_xsd () const =0
 JobQueens may optionally define an XSD for their resource definition file which is fed to their resource manager (if they control one). If the JobDistributor is awakened with the flag "jd3::output_resource_xsd <output file>" on the command line, and if the derived JobQueen defines an XSD for resources she uses, then the JobDistributor will write out the resource definition XSD to the output file. The derived queen may return an empty string to indicate that no resources are definable for the resource manager. More...
 
virtual void update_job_dag (JobDigraphUpdater &updater)=0
 The JobQueen is allowed to update the JobDigraph over the course of execution but only in a particular way: by adding new nodes, and then by adding edges that land on those new nodes. The JobQueen is not allowed to add edges that land on an old node. More...
 
virtual LarvalJobs determine_job_list (JobDAGNodeID job_dag_node_index, core::Size max_njobs)=0
 The JobDistributor asks the JobQueen for a list of jobs that should be performed. This function neither knows nor cares what jobs are already complete on disk/memory. More...
 
virtual bool has_job_previously_been_output (LarvalJobCOP job)=0
 
virtual JobOP mature_larval_job (LarvalJobCOP job, utility::vector1< JobResultCOP > const &input_job_results)=0
 Mature the input larval job into a full fledged job that will be run within this process (i.e. on this CPU), and thus can hold pointers to data that may persist inside the ResourceManager or may even be used by another process running in this thread. To pull that off, the Job object must share no non-bitwise-const data with any other Job object. That means if the Job were to contain a Pose and a Mover (as the MoverJob does), then so each Mover should be freshly constructed, and if the Pose had been copied from an existing Pose, must use the Poses "deep_copy" method (since Pose's sometimes share non-constant data between them, e.g. the AtomTree observer system, and sometimes Constraints ). The JobResults vector, which is supplied by the Jobdistributor, has entries corresponding to the JobResults from JobIDs in its input_job_result_indices vector. More...
 
virtual void note_job_completed (LarvalJobCOP job, JobStatus status, core::Size nresults)=0
 The JobDistributor will call this function to inform the JobQueen that a job has "completed" – in the sense that it will not be run in the future. It does not guarantee that the job was run on this CPU or that it successfully completed anywhere. The JobStatus indicates whether the job completed or whether it failed. The nresults count lists how many JobSummary/JobResult pairs were produced by the completed job. More...
 
virtual void completed_job_summary (LarvalJobCOP job, ResultIndex result_index, JobSummaryOP summary)=0
 The JobDistributor guarnatees that exactly one JobQueen will see every JobSummary generated within a Job batch. This guarantee allows the JobQueen to aggregate data across all of the Jobs so that Rosetta is able to compute data from structures instead of forcing that computation into accessory scripts. More...
 
virtual std::list
< output::OutputSpecificationOP
jobs_that_should_be_output ()=0
 The JobDistributor asks the JobQueen which JobResults should be queued for output. The JobQueen should reply by returning a list of OutputSpecification objects. Each OutputSpecification indicates a particular JobResult using the JobResultID object that the Specification holds. It also indicates how the JobResult should be output by whatever object will be doing to job outputting. The JobQueen should ask for a given JobResult to be output only once. After a job result is output, the JobDistributor will discard the JobResult, so the JobQueen should not tell the JobDistributor to output a job if it will be used as an input to another job in the future. Note that this method will only be called on the master-node JobQueen (aka JQ0), but that she will not necessarily be the JobQueen to actually return the ResultOutputter for the jobs that will be output. The JobDistributor holds the option to distribute output to other nodes. More...
 
virtual std::list< JobResultIDjob_results_that_should_be_discarded ()=0
 The JobDistributor, to manage memory use, asks the JobQueen which JobResults may be discarded because they will not be used in the future. The JobDistributor will exit with an error message if the JobQueen gives it a LarvalJob that lists one of these discarded JobResults as a required input for that LarvalJob. More...
 
virtual output::ResultOutputterOP result_outputter (output::OutputSpecification const &output_specification)=0
 The JobDistributor will ask the JobQueen to provide an outputter for a result giving it an annotated-with-job-distributor-specific-prefix OutputSpecification. The JobQueen must guarantee that a distinct ResultOutputter is given for each distinct JD prefix and that no two ResultOutputters share non-(bitwise-)constant data as the JobDistributor may use multiple threads to perform output. Note that the JobQueen that creates the outputter may not have been the JobQueen to request that output be performed. This function is thus similar to mature_larval_job: it requires that the JobQueen make few assumptions about the state of her data in order for this behavior to translate from the VanillaJobDistributor to the MPIWorkPoolJobDistributor. More...
 
virtual void flush ()=0
 Send all buffered output to disk – called by the JobDistributor right before it shuts down if it hits an error or catches an exception that it cannot ignore. More...
 
virtual std::list
< deallocation::DeallocationMessageOP
deallocation_messages ()=0
 Are there any deallocation messages that the JobQueen would like to have delivered to the JobQueens living on remote hosts? If there are none, the derived JobQueen may return an empty list. The JobDistributor guarantees to send the deallocation message to all remote hosts. The JobDistributor will ask the JobQueen on the head node for a set of deallocation messages after each set of jobs from determine_job_list have been distributed. (i.e., before the next call to determine_job_list is made). More...
 
virtual void process_deallocation_message (deallocation::DeallocationMessageOP message)=0
 A deallocation message first sent to the JobDistributor on this host originating from a remote JobQueen. More...
 

Protected Member Functions

virtual JobDigraphOP create_initial_job_dag ()=0
 The JobQueen creates a directed acyclic graph (DAG) describing the sets of all jobs and the interdependencies between them to the JobDistributor. The JobDistributor will allow the JobQueen to update the DAG by adding new nodes and adding edges to new nodes if the JobQueen does not know up front how many nodes will be in the graph. More...
 
JobDigraph const & get_job_graph () const
 Get read access to the current job graph. More...
 
JobTrackerget_job_tracker ()
 Get write access to the current job tracker. More...
 
JobTracker const & get_job_tracker () const
 Get read access to the current job tracker. More...
 

Private Attributes

JobDigraphCOP job_graph_ = nullptr
 
JobTrackerOP job_tracker_ = nullptr
 

Detailed Description

The JobQueen class (think of a queen bee) has quite a few responsibilities:

The JobQueen is responsible for creating all of the LarvalJobs, grouped into "batches" which are dispatched by the JobDistributor; the queen then matures the larval jobs into the mature form (a Job, e.g. a Mover/Pose pair) and gives them up to the JobDistributor to actually execute the work. The JobDistributor then actually calls the Job's "run" method, which produces a JobResult. It then hands this JobResult (e.g. a Pose or several Poses) back to the JobQueen for output. In fact, in MPI contexts, the JobDistributor guarantees that it will hand all of the completed JobResults for a single batch to a single JobQueen, so that she can compute aggregate statistics (e.g. averaging energies) or can look at the results from that round of jobs and decide which to carry forward into the next round of execution.

Constructor & Destructor Documentation

protocols::jd3::JobQueen::JobQueen ( )

References job_tracker_.

protocols::jd3::JobQueen::~JobQueen ( )
overridedefault

Member Function Documentation

virtual void protocols::jd3::JobQueen::completed_job_summary ( LarvalJobCOP  job,
ResultIndex  result_index,
JobSummaryOP  summary 
)
pure virtual

The JobDistributor guarnatees that exactly one JobQueen will see every JobSummary generated within a Job batch. This guarantee allows the JobQueen to aggregate data across all of the Jobs so that Rosetta is able to compute data from structures instead of forcing that computation into accessory scripts.

Implemented in protocols::jd3::standard::StandardJobQueen, protocols::multistage_rosetta_scripts::MRSJobQueen, protocols::jd3::chunk_library::ChunkLibraryJobQueen, and protocols::jd3::full_model::FullModelJobQueen.

JobDigraphOP protocols::jd3::JobQueen::create_and_set_initial_job_dag ( )

Creates the initial job dag using create_initial_job_dag function and sets it to the JQ.

Decorator to create_initial_job_dag

References create_initial_job_dag(), and job_graph_.

Referenced by protocols::multistage_rosetta_scripts::MRSJobQueen::complete_larval_job_maturation().

virtual JobDigraphOP protocols::jd3::JobQueen::create_initial_job_dag ( )
protectedpure virtual

The JobQueen creates a directed acyclic graph (DAG) describing the sets of all jobs and the interdependencies between them to the JobDistributor. The JobDistributor will allow the JobQueen to update the DAG by adding new nodes and adding edges to new nodes if the JobQueen does not know up front how many nodes will be in the graph.

Implemented in protocols::jd3::standard::StandardJobQueen, protocols::multistage_rosetta_scripts::MRSJobQueen, protocols::jd3::chunk_library::ChunkLibraryJobQueen, protocols::jd3::full_model::FullModelJobQueen, protocols::rosetta_scripts::RosettaScriptsJobQueen, and protocols::multistage_rosetta_scripts::MRSJobQueenChecker.

Referenced by create_and_set_initial_job_dag().

virtual std::list< deallocation::DeallocationMessageOP > protocols::jd3::JobQueen::deallocation_messages ( )
pure virtual

Are there any deallocation messages that the JobQueen would like to have delivered to the JobQueens living on remote hosts? If there are none, the derived JobQueen may return an empty list. The JobDistributor guarantees to send the deallocation message to all remote hosts. The JobDistributor will ask the JobQueen on the head node for a set of deallocation messages after each set of jobs from determine_job_list have been distributed. (i.e., before the next call to determine_job_list is made).

Implemented in protocols::jd3::standard::StandardJobQueen, protocols::jd3::chunk_library::ChunkLibraryJobQueen, protocols::jd3::full_model::FullModelJobQueen, and protocols::rosetta_scripts::RosettaScriptsJobQueen.

virtual LarvalJobs protocols::jd3::JobQueen::determine_job_list ( JobDAGNodeID  job_dag_node_index,
core::Size  max_njobs 
)
pure virtual

The JobDistributor asks the JobQueen for a list of jobs that should be performed. This function neither knows nor cares what jobs are already complete on disk/memory.

  • it just figures out what ones should exist given the input. The JobDistributor tells the JobQueen which node in the JobDAG it is requesting jobs for, and puts a limit on the number of LarvalJobs that the JobQueen should return. The JobDistributor will call this function repeatedly for a single node until the JobQueen returns an empty list, at which point, the JobDistributor will consider the node's jobs exhausted.

Implemented in protocols::jd3::standard::StandardJobQueen, protocols::multistage_rosetta_scripts::MRSJobQueen, protocols::jd3::chunk_library::ChunkLibraryJobQueen, protocols::jd3::full_model::FullModelJobQueen, and protocols::multistage_rosetta_scripts::MRSJobQueenChecker.

Referenced by determine_job_list_and_track().

LarvalJobs protocols::jd3::JobQueen::determine_job_list_and_track ( JobDAGNodeID  job_dag_node_index,
core::Size  max_njobs 
)

Call determine_job_list in derived classes and track the jobs using the JobTracker.

Decorator to determine_job_list

References determine_job_list(), and job_tracker_.

virtual void protocols::jd3::JobQueen::flush ( )
pure virtual

Send all buffered output to disk – called by the JobDistributor right before it shuts down if it hits an error or catches an exception that it cannot ignore.

Implemented in protocols::jd3::standard::StandardJobQueen, protocols::jd3::chunk_library::ChunkLibraryJobQueen, and protocols::jd3::full_model::FullModelJobQueen.

JobDigraph const & protocols::jd3::JobQueen::get_job_graph ( ) const
protected

Get read access to the current job graph.

References job_graph_.

JobTracker & protocols::jd3::JobQueen::get_job_tracker ( )
protected
JobTracker const & protocols::jd3::JobQueen::get_job_tracker ( ) const
protected

Get read access to the current job tracker.

References job_tracker_.

virtual bool protocols::jd3::JobQueen::has_job_previously_been_output ( LarvalJobCOP  job)
pure virtual

The JobQueen must be able to determine if a particular job has already been output. The JobQueen returns "true" if the job has already been written, and "false" otherwise. Used for overwrite behavior.

Implemented in protocols::jd3::standard::StandardJobQueen, protocols::jd3::chunk_library::ChunkLibraryJobQueen, and protocols::jd3::full_model::FullModelJobQueen.

virtual std::string protocols::jd3::JobQueen::job_definition_xsd ( ) const
pure virtual

All JobQueens must describe their job input XML format in the form of an XSD (XML Schema Definition), and they must validate their job input files against their XSDs. If the JobDistributor is awakened with the flag "jd3::output_job_xsd <output file>" on the command line, then the JobDistributor will write out the XSD to the output file.

Implemented in protocols::jd3::standard::StandardJobQueen, protocols::jd3::chunk_library::ChunkLibraryJobQueen, and protocols::jd3::full_model::FullModelJobQueen.

virtual std::list< JobResultID > protocols::jd3::JobQueen::job_results_that_should_be_discarded ( )
pure virtual

The JobDistributor, to manage memory use, asks the JobQueen which JobResults may be discarded because they will not be used in the future. The JobDistributor will exit with an error message if the JobQueen gives it a LarvalJob that lists one of these discarded JobResults as a required input for that LarvalJob.

Implemented in protocols::jd3::standard::StandardJobQueen, protocols::multistage_rosetta_scripts::MRSJobQueen, protocols::jd3::chunk_library::ChunkLibraryJobQueen, and protocols::jd3::full_model::FullModelJobQueen.

virtual std::list< output::OutputSpecificationOP > protocols::jd3::JobQueen::jobs_that_should_be_output ( )
pure virtual

The JobDistributor asks the JobQueen which JobResults should be queued for output. The JobQueen should reply by returning a list of OutputSpecification objects. Each OutputSpecification indicates a particular JobResult using the JobResultID object that the Specification holds. It also indicates how the JobResult should be output by whatever object will be doing to job outputting. The JobQueen should ask for a given JobResult to be output only once. After a job result is output, the JobDistributor will discard the JobResult, so the JobQueen should not tell the JobDistributor to output a job if it will be used as an input to another job in the future. Note that this method will only be called on the master-node JobQueen (aka JQ0), but that she will not necessarily be the JobQueen to actually return the ResultOutputter for the jobs that will be output. The JobDistributor holds the option to distribute output to other nodes.

Implemented in protocols::jd3::standard::StandardJobQueen, protocols::multistage_rosetta_scripts::MRSJobQueen, protocols::jd3::chunk_library::ChunkLibraryJobQueen, and protocols::jd3::full_model::FullModelJobQueen.

virtual JobOP protocols::jd3::JobQueen::mature_larval_job ( LarvalJobCOP  job,
utility::vector1< JobResultCOP > const &  input_job_results 
)
pure virtual

Mature the input larval job into a full fledged job that will be run within this process (i.e. on this CPU), and thus can hold pointers to data that may persist inside the ResourceManager or may even be used by another process running in this thread. To pull that off, the Job object must share no non-bitwise-const data with any other Job object. That means if the Job were to contain a Pose and a Mover (as the MoverJob does), then so each Mover should be freshly constructed, and if the Pose had been copied from an existing Pose, must use the Poses "deep_copy" method (since Pose's sometimes share non-constant data between them, e.g. the AtomTree observer system, and sometimes Constraints ). The JobResults vector, which is supplied by the Jobdistributor, has entries corresponding to the JobResults from JobIDs in its input_job_result_indices vector.

Implemented in protocols::jd3::standard::StandardJobQueen, protocols::jd3::chunk_library::ChunkLibraryJobQueen, and protocols::jd3::full_model::FullModelJobQueen.

virtual void protocols::jd3::JobQueen::note_job_completed ( LarvalJobCOP  job,
JobStatus  status,
core::Size  nresults 
)
pure virtual

The JobDistributor will call this function to inform the JobQueen that a job has "completed" – in the sense that it will not be run in the future. It does not guarantee that the job was run on this CPU or that it successfully completed anywhere. The JobStatus indicates whether the job completed or whether it failed. The nresults count lists how many JobSummary/JobResult pairs were produced by the completed job.

Implemented in protocols::jd3::standard::StandardJobQueen, protocols::multistage_rosetta_scripts::MRSJobQueen, protocols::jd3::chunk_library::ChunkLibraryJobQueen, and protocols::jd3::full_model::FullModelJobQueen.

Referenced by note_job_completed_and_track().

void protocols::jd3::JobQueen::note_job_completed_and_track ( LarvalJobCOP  job,
JobStatus  status,
core::Size  nresults 
)

Call note_job_completed in derived classes and track the job using the JobTracker.

Decorator to note_job_completed

References job_tracker_, and note_job_completed().

virtual void protocols::jd3::JobQueen::process_deallocation_message ( deallocation::DeallocationMessageOP  message)
pure virtual

A deallocation message first sent to the JobDistributor on this host originating from a remote JobQueen.

Implemented in protocols::jd3::standard::StandardJobQueen, protocols::jd3::chunk_library::ChunkLibraryJobQueen, and protocols::jd3::full_model::FullModelJobQueen.

virtual std::string protocols::jd3::JobQueen::resource_definition_xsd ( ) const
pure virtual

JobQueens may optionally define an XSD for their resource definition file which is fed to their resource manager (if they control one). If the JobDistributor is awakened with the flag "jd3::output_resource_xsd <output file>" on the command line, and if the derived JobQueen defines an XSD for resources she uses, then the JobDistributor will write out the resource definition XSD to the output file. The derived queen may return an empty string to indicate that no resources are definable for the resource manager.

Implemented in protocols::jd3::standard::StandardJobQueen, protocols::jd3::chunk_library::ChunkLibraryJobQueen, and protocols::jd3::full_model::FullModelJobQueen.

virtual output::ResultOutputterOP protocols::jd3::JobQueen::result_outputter ( output::OutputSpecification const &  output_specification)
pure virtual

The JobDistributor will ask the JobQueen to provide an outputter for a result giving it an annotated-with-job-distributor-specific-prefix OutputSpecification. The JobQueen must guarantee that a distinct ResultOutputter is given for each distinct JD prefix and that no two ResultOutputters share non-(bitwise-)constant data as the JobDistributor may use multiple threads to perform output. Note that the JobQueen that creates the outputter may not have been the JobQueen to request that output be performed. This function is thus similar to mature_larval_job: it requires that the JobQueen make few assumptions about the state of her data in order for this behavior to translate from the VanillaJobDistributor to the MPIWorkPoolJobDistributor.

Implemented in protocols::jd3::standard::StandardJobQueen, protocols::jd3::chunk_library::ChunkLibraryJobQueen, and protocols::jd3::full_model::FullModelJobQueen.

virtual void protocols::jd3::JobQueen::update_job_dag ( JobDigraphUpdater updater)
pure virtual

The JobQueen is allowed to update the JobDigraph over the course of execution but only in a particular way: by adding new nodes, and then by adding edges that land on those new nodes. The JobQueen is not allowed to add edges that land on an old node.

Implemented in protocols::jd3::standard::StandardJobQueen, protocols::jd3::chunk_library::ChunkLibraryJobQueen, and protocols::jd3::full_model::FullModelJobQueen.

Member Data Documentation

JobDigraphCOP protocols::jd3::JobQueen::job_graph_ = nullptr
private
JobTrackerOP protocols::jd3::JobQueen::job_tracker_ = nullptr
private

The documentation for this class was generated from the following files: