================ Using bilby_pipe ================ Basics ------ The primary user-interface for this code is a command line tool :code:`bilby_pipe` which is available after following the `installation instructions `_. To see the help for this tool, run .. code-block:: console $ bilby_pipe --help (the complete output is given in `bilby_pipe help`_) To run :code:`bilby_pipe`, you first need to define an define `_; examples for different types of ini files can be found below. Once you have an ini file (for the purpose of clarity, lets say :code:`my-run.ini`, you initialize your job with .. code-block:: console $ bilby_pipe my-run.ini This will produce a directory structure as follows: .. code-block:: console my-run.ini outdir/ -> data/ -> log_data_analysis/ -> log_data_generation/ -> log_results_page/ -> result/ -> results_page/ -> submit/ The first six of these folders will initially be empty, but as the job progresses will be populated. The :code:`data` directory will contain all the data to be analyses while :code:`result` will contain the :code:`*result.json` result files generated by bilby along with any plots. Note that the location to the `log` and `results_page` folders can be modified. The final folder, :code:`submit`, contains all the of the DAG submission scripts. To submit your job, run :code:`condor_submit_dag` giving as a first argument the file prepended with :code:`dag` under :code:`outdir/submit` (instructions to do this are printed to the terminal after). Alternatively, you can initialise and submit your jobs with .. code-block:: console $ bilby_pipe my-run.ini --submit Running all or part of the job directly --------------------------------------- In some cases, you may need to run all or part of the job directly (e.g., not through a scheduler). This can be done by using the file prepended with `bash` in the `submit/` directory. This file is a simple bash script that runs all commands in sequence. One simple way to run part of the job is to open the bash file and copy the commands you require to another script and then run that. For convenience, we also add if statements to the bash script to enable you to run parts of the analysis by providing a pattern as a command line. For example, to run the data generation step, you can call the bash script with `generation` in the arguments, e.g.: .. code-block:: console $ bash outdir/submit/bash_my_label.sh generation If you want to run the analysis step and `n-parallel=1`, then you would use .. code-block:: console $ bash outdir/submit/bash_my_label.sh analysis Note, if `n-parallel > 1` this will run all the parallel jobs. To run just one, run (replacing `par0` with the analysis you want to run): .. code-block:: console $ bash outdir/submit/bash_my_label.sh par0 Finally to merge the analyses, run .. code-block:: console $ bash outdir/submit/bash_my_label.sh merge Internally, the bash script is simply matching the given argument to the job name. This works in simple cases, but will likely fail or need inspection of the base file itself in complicated cases. Moreover, if you use any of the special key words (generation, analysis, par, or merge) in your label, the ability to filter to single jobs will be lost. Using the slurm batch scheduler ------------------------------- By default, :code:`bilby_pipe` runs under a HTCondor environment (the default for the LIGO data grid). It can also be used on a slurm-based cluster. Here we give a brief description of the steps required to run under slurm, but a full list of available options, see the output of `bilby_pipe --help`. To use slurm, add :code:`scheduler=slurm` to your ini file. Typically, slurm needs you to configure the correct environment, you can do this by passing it in to :code:`scheduler-env=my-environment`. This will add the following line to your submit scripts. .. code-block:: console $ source activate my-environment (Note: for conda users, this is equivalent to :code:`conda activate my-environment`). If the cluster you are using does not provide network access on the compute nodes, the data generation step may fail if an attempt is made to remotely access the data. (If you are creating simulated data, or have local copies of the data, this is, of course, not a problem). To resolve this issue, you can set `local-generation=True` in your ini file. The generation steps will then be run on the head node when you invoke :code:`bilby_pipe` after which you simply submit the job. Slurm modules can be loaded using :code:`scheduler-modules`, a space-separated list of modules to load. Additional commands to :code:`sbatch` can be given using the :code:`scheduler-args` command. Putting all this together, adding these lines to your ini file .. code-block:: console scheduler = slurm scheduler-args = arg1=val1 arg2=val2 scheduler-modules = git python scheduler-env = my-environment scheduler-analysis-time = 1-00:00:00 # Limit job to 1 day Will produce a :code:`slurm` submit files which contains .. code-block:: console #SBATCH --arg1=val1 #SBATCH --arg2=val2 module load git python and individual bash scripts containing .. code-block:: console module load git python source activate my-environment Summary webpage --------------- :code:`bilby_pipe` allows the user to visualise the posterior samples through a 'summary' webpage. This is implemented using `PESummary `_ (documentation `here `_). To generate a summary webpage, the :code:`create-summary` option must be passed in the configuration file. Additionally, you can specify a web directory where you would like the output from :code:`PESummary` to be stored; by default this is placed in :code:`outdir/results_page`. If you are working on an LDG cluster, then the web directory should be in your public_html. Below is an example of the additional lines to put in your configuration file to generate 'summary' webpages: .. code-block:: text create-summary = True email = albert.einstein@ligo.org webdir = /home/albert.einstein/public_html/project If you have already generated a webpage in the past using :code:`PESummary`, then you are able to pass the :code:`existing-dir` options to add further results files to a single webpage. This includes all histograms for each results file as well as comparison plots. Below is an example of the additional lines in the configuration file that will add to an existing webpage: .. code-block:: text create-summary = True email = albert.einstein@ligo.org existing-dir = /home/albert.einstein/public_html/project bilby_pipe help --------------- For reference, here is the full output of .. code-block:: console $ bilby_pipe --help .. highlight:: none .. argparse:: :ref: bilby_pipe.main.create_parser :prog: bilby_pipe :noepilog: