glue.pipeline module

This modules contains objects that make it simple for the user to create python scripts that build Condor DAGs to run code on the LSC Data Grid.

This file is part of the Grid LSC User Environment (GLUE)

GLUE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class glue.pipeline.AnalysisChunk(start, end, trig_start=0, trig_end=0)

Bases: object

An AnalysisChunk is the unit of data that a node works with, usually some subset of a ScienceSegment.

dur()

Returns the length (duration) of the chunk in seconds.

end()

Returns the GPS end time of the chunk.

set_trig_end(end)

Set the last GPS time at which triggers for this chunk should be generated.

set_trig_start(start)

Set the first GPS time at which triggers for this chunk should be generated.

start()

Returns the GPS start time of the chunk.

trig_end()

Return the last GPS time at which triggers for this chunk should be generated.

trig_start()

Return the first GPS time at which triggers for this chunk should be generated.

class glue.pipeline.AnalysisJob(cp)

Bases: object

Describes a generic analysis job that filters LIGO data as configured by an ini file.

channel()

Returns the name of the channel that this job is filtering. Note that channel is defined to be IFO independent, so this may be LSC-AS_Q or IOO-MC_F. The IFO is set on a per node basis, not a per job basis.

get_config(sec, opt)

Get the configration variable in a particular section of this jobs ini file. @param sec: ini file section. @param opt: option from section sec.

set_channel(channel)

Set the name of the channel that this job is filtering. This will overwrite the value obtained at initialization.

class glue.pipeline.AnalysisNode

Bases: object

Contains the methods that allow an object to be built to analyse LIGO data in a Condor DAG.

calibration()

Set the path to the calibration cache file for the given IFO. During S2 the Hanford 2km IFO had two calibration epochs, so if the start time is during S2, we use the correct cache file.

calibration_cache_path()

Determine the path to the correct calibration cache file to use.

get_calibration()

Return the calibration cache file to be used by the DAG.

get_data_end()

Get the GPS end time of the data needed by this node.

get_data_start()

Get the GPS start time of the data needed by this node.

get_end()

Get the GPS end time of the node.

get_ifo()

Returns the two letter IFO code for this node.

get_ifo_tag()

Returns the IFO tag string

get_input()

Get the file that will be passed as input.

get_output()

Get the file that will be passed as output.

get_pad_data()

Get the GPS start time of the data needed by this node.

get_start()

Get the GPS start time of the node.

get_trig_end()

Get the trig end time of the node.

get_trig_start()

Get the trig start time of the node.

get_user_tag()

Returns the usertag string

set_cache(filename)

Set the LAL frame cache to to use. The frame cache is passed to the job with the –frame-cache argument. @param filename: calibration file to use.

set_data_end(time)

Set the GPS end time of the data needed by this analysis node. @param time: GPS end time of job.

set_data_start(time)

Set the GPS start time of the data needed by this analysis node. @param time: GPS start time of job.

set_end(time, pass_to_command_line=True)

Set the GPS end time of the analysis node by setting a –gps-end-time option to the node when it is executed. @param time: GPS end time of job. @bool pass_to_command_line: add gps-end-time as variable option.

set_ifo(ifo)

Set the ifo name to analyze. If the channel name for the job is defined, then the name of the ifo is prepended to the channel name obtained from the job configuration file and passed with a –channel-name option. @param ifo: two letter ifo code (e.g. L1, H1 or H2).

set_ifo_tag(ifo_tag, pass_to_command_line=True)

Set the ifo tag that is passed to the analysis code. @param ifo_tag: a string to identify one or more IFOs @bool pass_to_command_line: add ifo-tag as a variable option.

set_input(filename, pass_to_command_line=True)

Add an input to the node by adding a –input option. @param filename: option argument to pass as input. @bool pass_to_command_line: add input as a variable option.

set_output(filename, pass_to_command_line=True)

Add an output to the node by adding a –output option. @param filename: option argument to pass as output. @bool pass_to_command_line: add output as a variable option.

set_pad_data(pad)

Set the GPS start time of the data needed by this analysis node. @param time: GPS start time of job.

set_start(time, pass_to_command_line=True)

Set the GPS start time of the analysis node by setting a –gps-start-time option to the node when it is executed. @param time: GPS start time of job. @bool pass_to_command_line: add gps-start-time as variable option.

set_trig_end(time, pass_to_command_line=True)

Set the trig end time of the analysis node by setting a –trig-end-time option to the node when it is executed. @param time: trig end time of job. @bool pass_to_command_line: add trig-end-time as a variable option.

set_trig_start(time, pass_to_command_line=True)

Set the trig start time of the analysis node by setting a –trig-start-time option to the node when it is executed. @param time: trig start time of job. @bool pass_to_command_line: add trig-start-time as a variable option.

set_user_tag(usertag, pass_to_command_line=True)

Set the user tag that is passed to the analysis code. @param user_tag: the user tag to identify the job @bool pass_to_command_line: add user-tag as a variable option.

class glue.pipeline.CondorDAG(log)

Bases: object

A CondorDAG is a Condor Directed Acyclic Graph that describes a collection of Condor jobs and the order in which to run them. All Condor jobs in the DAG must write their Codor logs to the same file. NOTE: The log file must not be on an NFS mounted system as the Condor jobs must be able to get an exclusive file lock on the log file.

add_maxjobs_category(categoryName, maxJobsNum)

Add a category to this DAG called categoryName with a maxjobs of maxJobsNum. @param node: Add (categoryName,maxJobsNum) tuple to CondorDAG.__maxjobs_categories.

add_node(node)

Add a CondorDAGNode to this DAG. The CondorJob that the node uses is also added to the list of Condor jobs in the DAG so that a list of the submit files needed by the DAG can be maintained. Each unique CondorJob will be added once to prevent duplicate submit files being written. @param node: CondorDAGNode to add to the CondorDAG.

get_dag_file()

Return the path to the DAG file.

get_jobs()

Return a list containing all the jobs in the DAG

get_maxjobs_categories()

Return an array of tuples containing (categoryName,maxJobsNum)

get_nodes()

Return a list containing all the nodes in the DAG

set_dag_file(path)

Set the name of the file into which the DAG is written. @param path: path to DAG file.

set_integer_node_names()

Use integer node names for the DAG

write_concrete_dag()

Write all the nodes in the DAG to the DAG file.

write_dag()

Write a dag.

write_maxjobs(fh, category)

Write the DAG entry for this category’s maxjobs to the DAG file descriptor. @param fh: descriptor of open DAG file. @param category: tuple containing type of jobs to set a maxjobs limit for

and the maximum number of jobs of that type to run at once.

write_script()

Write the workflow to a script (.sh instead of .dag).

Assuming that parents were added to the DAG before their children, dependencies should be handled correctly.

write_sub_files()

Write all the submit files used by the dag to disk. Each submit file is written to the file name set in the CondorJob.

exception glue.pipeline.CondorDAGError(args=None)

Bases: CondorError

class glue.pipeline.CondorDAGJob(universe, executable)

Bases: CondorJob

A Condor DAG job never notifies the user on completion and can have variable options that are set for a particular node in the DAG. Inherits methods from a CondorJob.

add_var_arg(arg_index, quote=False)

Add a command to the submit file to allow variable (macro) arguments to be passed to the executable.

add_var_condor_cmd(command)

Add a condor command to the submit file that allows variable (macro) arguments to be passes to the executable.

add_var_opt(opt, short=False)

Add a variable (or macro) option to the condor job. The option is added to the submit file and a different argument to the option can be set for each node in the DAG. @param opt: name of option to add.

create_node()

Create a condor node from this job. This provides a basic interface to the CondorDAGNode class. Most jobs in a workflow will subclass the CondorDAGNode class and overwrite this to give more details when initializing the node. However, this will work fine for jobs with very simp input/output.

exception glue.pipeline.CondorDAGJobError(args=None)

Bases: CondorError

class glue.pipeline.CondorDAGManJob(dag, dir=None)

Bases: object

Condor DAGMan job class. Appropriate for setting up DAGs to run within a DAG.

create_node()

Create a condor node from this job. This provides a basic interface to the CondorDAGManNode class. Most jobs in a workflow will subclass the CondorDAGManNode class and overwrite this to give more details when initializing the node. However, this will work fine for jobs with very simp input/output.

get_dag()

Return the name of any associated dag file

get_dag_directory()

Get the directory where the dag will be run

get_sub_file()

Return the name of the dag as the submit file name for the SUBDAG EXTERNAL command in the uber-dag

set_dag_directory(dir)

Set the directory where the dag will be run @param dir: the name of the directory where the dag will be run

set_notification(value)

Set the email address to send notification to. @param value: email address or never for no notification.

write_sub_file()

Do nothing as there is not need for a sub file with the SUBDAG EXTERNAL command in the uber-dag

class glue.pipeline.CondorDAGManNode(job)

Bases: CondorDAGNode

Condor DAGMan node class. Appropriate for setting up DAGs to run within a DAG. Adds the user-tag functionality to condor_dagman processes running in the DAG. May also be used to extend dagman-node specific functionality.

add_maxjobs_category(categoryName, maxJobsNum)

Add a category to this DAG called categoryName with a maxjobs of maxJobsNum. @param node: Add (categoryName,maxJobsNum) tuple to CondorDAG.__maxjobs_categories.

get_maxjobs_categories()

Return an array of tuples containing (categoryName,maxJobsNum)

get_user_tag()

Returns the usertag string

set_user_tag(usertag)

Set the user tag that is passed to the analysis code. @param user_tag: the user tag to identify the job

class glue.pipeline.CondorDAGNode(job)

Bases: object

A CondorDAGNode represents a node in the DAG. It corresponds to a particular condor job (and so a particular submit file). If the job has variable (macro) options, they can be set here so each nodes executes with the correct options.

add_checkpoint_file(filename)

Add filename as a checkpoint file for this DAG node @param filename: checkpoint filename to add

add_checkpoint_macro(filename)

add_file_arg(filename)

Add a variable (or macro) file name argument to the condor job. The argument is added to the submit file and a different value of the argument can be set for each node in the DAG. The file name is also added to the list of input files. @param filename: name of option to add.

add_file_opt(opt, filename, file_is_output_file=False)

Add a variable (macro) option for this node. If the option specified does not exist in the CondorJob, it is added so the submit file will be correct when written. The value of the option is also added to the list of input files. @param opt: option name. @param value: value of the option for this node in the DAG. @param file_is_output_file: A boolean if the file will be an output file instead of an input file. The default is to have it be an input.

add_input_file(filename)

Add filename as a necessary input file for this DAG node.

@param filename: input filename to add

add_input_macro(filename)

Add a variable (macro) for storing the input files associated with this node. @param filename: filename of input file

add_io_macro(io, filename)

Add a variable (macro) for storing the input/output files associated with this node. @param io: macroinput or macrooutput @param filename: filename of input/output file

add_macro(name, value)

Add a variable (macro) for this node. This can be different for each node in the DAG, even if they use the same CondorJob. Within the CondorJob, the value of the macro can be referenced as ‘$(name)’ – for instance, to define a unique output or error file for each node. @param name: macro name. @param value: value of the macro for this node in the DAG

add_output_file(filename)

Add filename as a output file for this DAG node.

@param filename: output filename to add

add_output_macro(filename)

Add a variable (macro) for storing the output files associated with this node. @param filename: filename of output file

add_parent(node)

Add a parent to this node. This node will not be executed until the parent node has run sucessfully. @param node: CondorDAGNode to add as a parent.

add_post_script_arg(arg)

Adds an argument to the post script that is executed before the DAG node is run.

add_pre_script_arg(arg)

Adds an argument to the pre script that is executed before the DAG node is run.

add_var_arg(arg, quote=False)

Add a variable (or macro) argument to the condor job. The argument is added to the submit file and a different value of the argument can be set for each node in the DAG. @param arg: name of option to add.

add_var_condor_cmd(command, value)

Add a variable (macro) condor command for this node. If the command specified does not exist in the CondorJob, it is added so the submit file will be correct. PLEASE NOTE: AS with other add_var commands, the variable must be set for all nodes that use the CondorJob instance. @param command: command name @param value: Value of the command for this node in the DAG.

add_var_opt(opt, value, short=False)

Add a variable (macro) option for this node. If the option specified does not exist in the CondorJob, it is added so the submit file will be correct when written. @param opt: option name. @param value: value of the option for this node in the DAG.

finalize()

The finalize method of a node is called before the node is finally added to the DAG and can be overridden to do any last minute clean up (such as setting extra command line arguments)

get_args()

Return the arguments for this node. Note that this returns only the arguments for this instance of the node and not those associated with the underlying job template.

get_category()

Get the category for this node in the DAG.

get_checkpoint_files()

Return a list of checkpoint files for this DAG node and its job.

get_cmd_line()

Return the full command line that will be used when this node is run by DAGman.

get_cmd_tuple_list()

Return a list of tuples containg the command line arguments

get_input_files()

Return list of input files for this DAG node and its job.

get_name()

Get the name for this node in the DAG.

get_opts()

Return the opts for this node. Note that this returns only the options for this instance of the node and not those associated with the underlying job template.

get_output_files()

Return list of output files for this DAG node and its job.

get_post_script()

returns the name of the post script that is executed before the DAG node is run. @param script: path to script

get_post_script_arg()

Returns and array of arguments to the post script that is executed before the DAG node is run.

get_priority()

Get the priority for this node in the DAG.

get_retry()

Return the number of times that this node in the DAG should retry. @param retry: number of times to retry node.

job()

Return the CondorJob that this node is associated with.

set_category(category)

Set the category for this node in the DAG.

set_log_file(log)

Set the Condor log file to be used by this CondorJob. @param log: path of Condor log file.

set_name(name)

Set the name for this node in the DAG.

set_post_script(script)

Sets the name of the post script that is executed before the DAG node is run. @param script: path to script

set_pre_script(script)

Sets the name of the pre script that is executed before the DAG node is run. @param script: path to script

set_priority(priority)

Set the priority for this node in the DAG.

set_retry(retry)

Set the number of times that this node in the DAG should retry. @param retry: number of times to retry node.

write_category(fh)

Write the DAG entry for this node’s category to the DAG file descriptor. @param fh: descriptor of open DAG file.

write_input_files(fh)

Write as a comment into the DAG file the list of input files for this DAG node.

@param fh: descriptor of open DAG file.

write_job(fh)

Write the DAG entry for this node’s job to the DAG file descriptor. @param fh: descriptor of open DAG file.

write_output_files(fh)

Write as a comment into the DAG file the list of output files for this DAG node.

@param fh: descriptor of open DAG file.

write_parents(fh)

Write the parent/child relations for this job to the DAG file descriptor. @param fh: descriptor of open DAG file.

write_post_script(fh)

Write the post script for the job, if there is one @param fh: descriptor of open DAG file.

write_pre_script(fh)

Write the pre script for the job, if there is one @param fh: descriptor of open DAG file.

write_priority(fh)

Write the DAG entry for this node’s priority to the DAG file descriptor. @param fh: descriptor of open DAG file.

write_vars(fh)

Write the variable (macro) options and arguments to the DAG file descriptor. @param fh: descriptor of open DAG file.

exception glue.pipeline.CondorDAGNodeError(args=None)

Bases: CondorError

exception glue.pipeline.CondorError(args=None)

Bases: Exception

Error thrown by Condor Jobs

class glue.pipeline.CondorJob(universe, executable, queue)

Bases: object

Generic condor job class. Provides methods to set the options in the condor submit file for a particular executable

add_arg(arg)

Add an argument to the executable. Arguments are appended after any options and their order is guaranteed. @param arg: argument to add.

add_checkpoint_file(filename)

Add filename as a checkpoint file for this DAG job.

add_condor_cmd(cmd, value)

Add a Condor command to the submit file (e.g. a class add or evironment). @param cmd: Condor command directive. @param value: value for command.

add_file_arg(filename)

Add a file argument to the executable. Arguments are appended after any options and their order is guaranteed. Also adds the file name to the list of required input data for this job. @param filename: file to add as argument.

add_file_opt(opt, filename)

Add a command line option to the executable. The order that the arguments will be appended to the command line is not guaranteed, but they will always be added before any command line arguments. The name of the option is prefixed with double hyphen and the program is expected to parse it with getopt_long(). @param opt: command line option to add. @param value: value to pass to the option (None for no argument).

add_ini_opts(cp, section)

Parse command line options from a given section in an ini file and pass to the executable. @param cp: ConfigParser object pointing to the ini file. @param section: section of the ini file to add to the options.

add_input_file(filename)

Add filename as a necessary input file for this DAG node.

@param filename: input filename to add

add_opt(opt, value)

Add a command line option to the executable. The order that the arguments will be appended to the command line is not guaranteed, but they will always be added before any command line arguments. The name of the option is prefixed with double hyphen and the program is expected to parse it with getopt_long(). @param opt: command line option to add. @param value: value to pass to the option (None for no argument).

add_output_file(filename)

Add filename as a output file for this DAG node.

@param filename: output filename to add

add_short_opt(opt, value)

Add a command line option to the executable. The order that the arguments will be appended to the command line is not guaranteed, but they will always be added before any command line arguments. The name of the option is prefixed with single hyphen and the program is expected to parse it with getopt() or getopt_long() (if a single character option), or getopt_long_only() (if multiple characters). Long and (single-character) short options may be mixed if the executable permits this. @param opt: command line option to add. @param value: value to pass to the option (None for no argument).

get_args()

Return the list of arguments that are to be passed to the executable.

get_checkpoint_files()

Return a list of checkpoint files for this DAG node

get_condor_cmds()

Return the dictionary of condor keywords to add to the job

get_executable()

Return the name of the executable for this job.

get_grid_scheduler()

Return the grid scheduler.

get_grid_server()

Return the grid server on which the job will run.

get_grid_type()

Return the grid type of the job.

get_input_files()

Return list of input files for this DAG node.

get_opt(opt)

Returns the value associated with the given command line option. Returns None if the option does not exist in the options list. @param opt: command line option

get_opts()

Return the dictionary of opts for the job.

get_output_files()

Return list of output files for this DAG node.

get_short_opts()

Return the dictionary of short options for the job.

get_stderr_file()

Get the file to which Condor directs the stderr of the job.

get_stdin_file()

Get the file from which Condor directs the stdin of the job.

get_stdout_file()

Get the file to which Condor directs the stdout of the job.

get_sub_file()

Get the name of the file which the Condor submit file will be written to when write_sub_file() is called.

get_universe()

Return the condor universe that the job will run in.

set_executable(executable)

Set the name of the executable for this job.

set_grid_scheduler(grid_scheduler)

Set the grid scheduler. @param grid_scheduler: grid scheduler on which to run.

set_grid_server(grid_server)

Set the grid server on which to run the job. @param grid_server: grid server on which to run.

set_grid_type(grid_type)

Set the type of grid resource for the job. @param grid_type: type of grid resource.

set_log_file(path)

Set the Condor log file. @param path: path to log file.

set_notification(value)

Set the email address to send notification to. @param value: email address or never for no notification.

set_stderr_file(path)

Set the file to which Condor directs the stderr of the job. @param path: path to stderr file.

set_stdin_file(path)

Set the file from which Condor directs the stdin of the job. @param path: path to stdin file.

set_stdout_file(path)

Set the file to which Condor directs the stdout of the job. @param path: path to stdout file.

set_sub_file(path)

Set the name of the file to write the Condor submit file to when write_sub_file() is called. @param path: path to submit file.

set_universe(universe)

Set the condor universe for the job to run in. @param universe: the condor universe to run the job in.

write_sub_file()

Write a submit file for this Condor job.

exception glue.pipeline.CondorJobError(args=None)

Bases: CondorError

exception glue.pipeline.CondorSubmitError(args=None)

Bases: CondorError

class glue.pipeline.DeepCopyableConfigParser(defaults=None, dict_type=<class 'dict'>, allow_no_value=False, *, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section='DEFAULT', interpolation=<object object>, converters=<object object>)

Bases: ConfigParser

The standard SafeConfigParser no longer supports deepcopy() as of python 2.7 (see http://bugs.python.org/issue16058). This subclass restores that functionality.

class glue.pipeline.LSCDataFindJob(cache_dir, log_dir, config_file, lsync_cache_file=None, lsync_type_regex=None)

Bases: CondorDAGJob, AnalysisJob

An LSCdataFind job used to locate data. The static options are read from the section [datafind] in the ini file. The stdout from LSCdataFind contains the paths to the frame files and is directed to a file in the cache directory named by site and GPS start and end times. The stderr is directed to the logs directory. The job always runs in the scheduler universe. The path to the executable is determined from the ini file.

get_cache_dir()

returns the directroy that the cache files are written to.

get_config_file()

return the configuration file object

lsync_cache()

class glue.pipeline.LSCDataFindNode(job)

Bases: CondorDAGNode, AnalysisNode

A DataFindNode runs an instance of LSCdataFind in a Condor DAG.

get_end()

Return the start time of the datafind query

get_observatory()

Return the start time of the datafind query

get_output()

Return the output file, i.e. the file containing the frame cache data.

get_output_cache()

get_start()

Return the start time of the datafind query

get_type()

gets the frame type that we are querying

set_end(time)

Set the end time of the datafind query. @param time: GPS end time of query.

set_observatory(obs)

Set the IFO to retrieve data for. Since the data from both Hanford interferometers is stored in the same frame file, this takes the first letter of the IFO (e.g. L or H) and passes it to the –observatory option of LSCdataFind. @param obs: IFO to obtain data for.

set_start(time, pad=None)

Set the start time of the datafind query. @param time: GPS start time of query.

set_type(type)

sets the frame type that we are querying

class glue.pipeline.LigolwAddJob(log_dir, cp)

Bases: CondorDAGJob, AnalysisJob

A ligolw_add job can be used to concatenate several ligo lw files

class glue.pipeline.LigolwAddNode(job)

Bases: CondorDAGNode, AnalysisNode

Runs an instance of ligolw_add in a Condor DAG.

class glue.pipeline.LigolwCutJob(log_dir, cp)

Bases: CondorDAGJob, AnalysisJob

A ligolw_cut job can be used to remove parts of a ligo lw file

class glue.pipeline.LigolwCutNode(job)

Bases: CondorDAGNode, AnalysisNode

Runs an instance of ligolw_cut in a Condor DAG.

class glue.pipeline.LigolwSqliteJob(cp)

Bases: SqliteJob

A LigolwSqlite job. The static options are read from the section [ligolw_sqlite] in the ini file.

set_replace()

Sets the –replace option. This will cause the job to overwrite existing databases rather than add to them.

class glue.pipeline.LigolwSqliteNode(job)

Bases: SqliteNode

A LigolwSqlite node.

get_input_cache()

Gets input cache.

get_output()

Override standard get_output to return xml-file if xml-file is specified. Otherwise, will return database.

set_input_cache(input_cache)

Sets input cache.

set_xml_input(xml_file)

Sets xml input file instead of cache

set_xml_output(xml_file)

Tell ligolw_sqlite to dump the contents of the database to a file.

class glue.pipeline.LsyncCache(path)

Bases: object

get_lfns(site, frameType, gpsStart, gpsEnd)

group(lst, n)

Group an iterable into an n-tuples iterable. Incomplete tuples are discarded

parse(type_regex=None)

Each line of the frame cache file is like the following:

/frames/E13/LHO/frames/hoftMon_H1/H-H1_DMT_C00_L2-9246,H,H1_DMT_C00_L2,1,16 1240664820 6231 {924600000 924646720 924646784 924647472 924647712 924700000}

The description is as follows:

1.1) Directory path of files 1.2) Site 1.3) Type 1.4) Number of frames in the files (assumed to be 1) 1.5) Duration of the frame files.

2. UNIX timestamp for directory modification time.

3. Number of files that that match the above pattern in the directory.

4. List of time range or segments [start, stop)

We store the cache for each site and frameType combination as a dictionary where the keys are (directory, duration) tuples and the values are segment lists.

Since the cache file is already coalesced we do not have to call the coalesce method on the segment lists.

class glue.pipeline.NoopJob(log_dir, cp)

Bases: CondorDAGJob, AnalysisJob

A Noop Job does nothing.

class glue.pipeline.NoopNode(job)

Bases: CondorDAGNode, AnalysisNode

Run an noop job in a Condor DAG.

class glue.pipeline.ScienceData

Bases: object

An object that can contain all the science data used in an analysis. Can contain multiple ScienceSegments and has a method to generate these from a text file produces by the LIGOtools segwizard program.

append_from_tuple(seg_tuple)

coalesce()

Coalesces any adjacent ScienceSegments. Returns the number of ScienceSegments in the coalesced list.

intersect_3(second, third)

Intersection routine for three inputs. Built out of the intersect, coalesce and play routines

intersect_4(second, third, fourth)

Intersection routine for four inputs.

intersection(other)

Replaces the ScienceSegments contained in this instance of ScienceData with the intersection of those in the instance other. Returns the number of segments in the intersection. @param other: ScienceData to use to generate the intersection

invert()

Inverts the ScienceSegments in the class (i.e. set NOT). Returns the number of ScienceSegments after inversion.

make_chunks(length, overlap=0, play=0, sl=0, excl_play=0, pad_data=0)

Divide each ScienceSegment contained in this object into AnalysisChunks. @param length: length of chunk in seconds. @param overlap: overlap between segments. @param play: if true, only generate chunks that overlap with S2 playground data. @param sl: slide by sl seconds before determining playground data. @param excl_play: exclude the first excl_play second from the start and end of the chunk when computing if the chunk overlaps with playground.

make_chunks_from_unused(length, trig_overlap, play=0, min_length=0, sl=0, excl_play=0, pad_data=0)

Create an extra chunk that uses up the unused data in the science segment. @param length: length of chunk in seconds. @param trig_overlap: length of time start generating triggers before the start of the unused data. @param play:

• 1 : only generate chunks that overlap with S2 playground data.

• 2as 1 plus compute trig start and end times to coincide

  with the start/end of the playground

@param min_length: the unused data must be greater than min_length to make a chunk. @param sl: slide by sl seconds before determining playground data. @param excl_play: exclude the first excl_play second from the start and end of the chunk when computing if the chunk overlaps with playground. @param pad_data: exclude the first and last pad_data seconds of the segment when generating chunks

make_optimised_chunks(min_length, max_length, pad_data=0)

Splits ScienceSegments up into chunks, of a given maximum length. The length of the last two chunks are chosen so that the data utilisation is optimised. @param min_length: minimum chunk length. @param max_length: maximum chunk length. @param pad_data: exclude the first and last pad_data seconds of the segment when generating chunks

make_short_chunks_from_unused(min_length, overlap=0, play=0, sl=0, excl_play=0)

Create a chunk that uses up the unused data in the science segment @param min_length: the unused data must be greater than min_length to make a chunk. @param overlap: overlap between chunks in seconds. @param play: if true, only generate chunks that overlap with S2 playground data. @param sl: slide by sl seconds before determining playground data. @param excl_play: exclude the first excl_play second from the start and end of the chunk when computing if the chunk overlaps with playground.

play()

Keep only times in ScienceSegments which are in the playground

read(filename, min_length, slide_sec=0, buffer=0)

Parse the science segments from the segwizard output contained in file. @param filename: input text file containing a list of science segments generated by segwizard. @param min_length: only append science segments that are longer than min_length. @param slide_sec: Slide each ScienceSegment by:

delta > 0:
  [s, e] -> [s+delta, e].
delta < 0:
  [s, e] -> [s, e-delta].

@param buffer: shrink the ScienceSegment:

[s, e] -> [s+buffer, e-buffer]

split(dt)

Split the segments in the list is subsegments at least as long as dt

tama_read(filename)

Parse the science segments from a tama list of locked segments contained in

file.

@param filename: input text file containing a list of tama segments.

union(other)

Replaces the ScienceSegments contained in this instance of ScienceData with the union of those in the instance other. Returns the number of ScienceSegments in the union. @param other: ScienceData to use to generate the intersection

class glue.pipeline.ScienceSegment(segment)

Bases: object

A ScienceSegment is a period of time where the experimenters determine that the inteferometer is in a state where the data is suitable for scientific analysis. A science segment can have a list of AnalysisChunks asscociated with it that break the segment up into (possibly overlapping) smaller time intervals for analysis.

add_chunk(start, end, trig_start=0, trig_end=0)

Add an AnalysisChunk to the list associated with this ScienceSegment. @param start: GPS start time of chunk. @param end: GPS end time of chunk. @param trig_start: GPS start time for triggers from chunk

dur()

Returns the length (duration) in seconds of this ScienceSegment.

end()

Returns the GPS end time of this ScienceSegment.

get_df_node()

Returns the DataFind node for this ScienceSegment.

id()

Returns the ID of this ScienceSegment.

make_chunks(length=0, overlap=0, play=0, sl=0, excl_play=0, pad_data=0)

Divides the science segment into chunks of length seconds overlapped by overlap seconds. If the play option is set, only chunks that contain S2 playground data are generated. If the user has a more complicated way of generating chunks, this method should be overriden in a sub-class. Any data at the end of the ScienceSegment that is too short to contain a chunk is ignored. The length of this unused data is stored and can be retrieved with the unused() method. @param length: length of chunk in seconds. @param overlap: overlap between chunks in seconds. @param play: 1 : only generate chunks that overlap with S2 playground data.

2as play = 1 plus compute trig start and end times to

coincide with the start/end of the playground

@param sl: slide by sl seconds before determining playground data. @param excl_play: exclude the first excl_play second from the start and end of the chunk when computing if the chunk overlaps with playground. @param pad_data: exclude the first and last pad_data seconds of the segment when generating chunks

set_df_node(df_node)

Set the DataFind node associated with this ScienceSegment to df_node. @param df_node: the DataFind node for this ScienceSegment.

set_end(t)

Override the GPS end time (and set the duration) of this ScienceSegment. @param t: new GPS end time.

set_start(t)

Override the GPS start time (and set the duration) of this ScienceSegment. @param t: new GPS start time.

set_unused(unused)

Set the length of data in the science segment not used to make chunks.

start()

Returns the GPS start time of this ScienceSegment.

unused()

Returns the length of data in the science segment not used to make chunks.

exception glue.pipeline.SegmentError(args=None)

Bases: Exception

class glue.pipeline.SqliteJob(cp, sections, exec_name)

Bases: CondorDAGJob, AnalysisJob

A cbc sqlite job adds to CondorDAGJob and AnalysisJob features common to jobs which read or write to a sqlite database. Of note, the universe is always set to local regardless of what’s in the cp file, the extension is set to None so that it may be set by individual SqliteNodes, log files do not have macrogpsstarttime and endtime in them, and get_env is set to True.

get_exec_name()

Get the exec_name name

set_exec_name(exec_name)

Set the exec_name name

class glue.pipeline.SqliteNode(job)

Bases: CondorDAGNode, AnalysisNode

A cbc sqlite node adds to the standard AnalysisNode features common to nodes which read or write to a sqlite database. Specifically, it adds the set_tmp_space_path and set_database methods.

get_database()

Gets database option.

get_tmp_space()

Gets tmp-space path.

set_database(database)

Sets database option.

set_tmp_space(tmp_space)

Sets temp-space path. This should be on a local disk.

glue.pipeline.s2play(t)

Return True if t is in the S2 playground, False otherwise t = GPS time to test if playground