dagparts module

DAG construction tools.

class dagparts.DAG(name, logpath='/tmp')

Bases: CondorDAG

! A thin subclass of pipeline.CondorDAG.

Extra features include an add_node() method and a cache writing method. Also includes some standard setup, e.g., log file paths etc.

add_node(node, retry=3)

Add a CondorDAGNode to this DAG. The CondorJob that the node uses is also added to the list of Condor jobs in the DAG so that a list of the submit files needed by the DAG can be maintained. Each unique CondorJob will be added once to prevent duplicate submit files being written. @param node: CondorDAGNode to add to the CondorDAG.

write_cache()

class dagparts.DAGJob(executable, tag_base=None, universe='vanilla', condor_commands={})

Bases: CondorDAGJob

! A job class that subclasses pipeline.CondorDAGJob and adds some extra boiler plate items for gstlal jobs which tends to do the “right” thing when given just an executable name.

class dagparts.DAGNode(job, dag, parent_nodes, opts={}, input_files={}, output_files={}, input_cache_files={}, output_cache_files={}, input_cache_file_name=None)

Bases: CondorDAGNode

! A node class that subclasses pipeline.CondorDAGNode that automates adding the node to the dag, makes sensible names and allows a list of parent nodes to be provided.

It tends to do the “right” thing when given a job, a dag, parent nodes, dictionary options relevant to the job, a dictionary of options related to input files and a dictionary of options related to output files.

NOTE and important and subtle behavior - You can specify an option with an empty argument by setting it to “”. However options set to None are simply ignored.

dagparts.T050017_filename(instruments, description, seg, extension, path=None)

! A function to generate a T050017 filename.

dagparts.breakupseg(seg, maxextent, overlap)

dagparts.breakupseglists(seglists, maxextent, overlap)

dagparts.breakupsegs(seglist, maxextent, overlap)

dagparts.cache_to_instruments(cache)

! Given a cache, returns back a string containing all the IFOs that are contained in each of its cache entries, sorted by IFO name.

dagparts.condor_command_dict_from_opts(opts, defaultdict=None)

! A function to turn a list of options into a dictionary of condor commands, e.g.,

>>> condor_command_dict_from_opts(["+Online_CBC_SVD=True", "TARGET.Online_CBC_SVD =?= True"])
{'+Online_CBC_SVD': 'True', 'TARGET.Online_CBC_SVD ': '?= True'}
>>> condor_command_dict_from_opts(["+Online_CBC_SVD=True", "TARGET.Online_CBC_SVD =?= True"], {"somecommand":"somevalue"})
{'somecommand': 'somevalue', '+Online_CBC_SVD': 'True', 'TARGET.Online_CBC_SVD ': '?= True'}
>>> condor_command_dict_from_opts(["+Online_CBC_SVD=True", "TARGET.Online_CBC_SVD =?= True"], {"+Online_CBC_SVD":"False"})
{'+Online_CBC_SVD': 'True', 'TARGET.Online_CBC_SVD ': '?= True'}

dagparts.condor_scratch_space()

! A way to standardize the condor scratch space even if it changes >>> condor_scratch_space() ‘_CONDOR_SCRATCH_DIR’

dagparts.flatten(lst)

! Flatten a list by one level of nesting.

dagparts.group_T050017_filename_from_T050017_files(cache_entries, extension, path=None)

! A function to return the name of a file created from multiple files following the T050017 convention. In addition to the T050017 requirements, this assumes that numbers relevant to organization schemes will be the first entry in the description, e.g. 0_DIST_STATS, and that all files in a given cache file are from the same group of ifos and either contain data from the same segment or from the same background bin. Note, that each file doesn’t have to be from the same IFO, for example the template bank cache could contain template bank files from H1 and template bank files from L1.

dagparts.groups(l, n)

! Given a list, returns back sublists with a maximum size n.

dagparts.log_path()

! The stupid pet tricks to find log space on the LDG. Defaults to checking TMPDIR first.

dagparts.pipeline_dot_py_append_opts_hack(opt, vals)

! A way to work around the dictionary nature of pipeline.py which can only record options once.

>>> pipeline_dot_py_append_opts_hack("my-favorite-option", [1,2,3])
'1 --my-favorite-option 2 --my-favorite-option 3'

dagparts.which(prog)