gstlal_fake_frames
==================

This program will make fake data in a variety of ways.  Its input is anything
supported by datasource.py.  You can additionally whiten the data or apply a
frequency domain coloring filter.  It writes its output to frame files.

Overview graph of the pipeline
------------------------------

Gray boxes are optional and depend on the command line given

.. graphviz::

   digraph G {

       // graph properties

       rankdir=LR;
       compound=true;
       node [shape=record fontsize=10 fontname="Verdana"];
       edge [fontsize=8 fontname="Verdana"];

       // nodes

       "mkbasicsrc()" [URL="\ref datasource.mkbasicsrc()"];
       "whitened_multirate_src()" [label="whitened_multirate_src()", URL="\ref multirate_datasource.mkwhitened_multirate_src()", style=filled, color=lightgrey];
       capsfilter [style=filled, color=lightgrey, URL="\ref pipeparts.mkcapsfilter()"];
       resample [style=filled, color=lightgrey, URL="\ref pipeparts.mkresample()"];
       audioamplify [style=filled, color=lightgrey, URL="\ref pipeparts.mkaudioamplify()", label="audioamplify \n if --data-source=white \n in order to have unit variance \n after resampling"];
       lal_shift[style=filled, color=lightgrey, URL="\ref pipeparts.mkshift()", label="lal_shift \n [iff --shift provided]"];
       progressreport1 [URL="\ref pipeparts.mkprogressreport()", label="progressreport \n [iff --verbose provided]"];
       progressreport2 [style=filled, color=lightgrey, URL="\ref pipeparts.mkprogressreport()", label="progressreport \n [iff --verbose provided]"];
       lal_firbank [style=filled, color=lightgrey, URL="\ref pipeparts.mkfirbank()", label="lal_firbank \n [iff --color-psd provided]"];
       taginject [URL="\ref pipeparts.mktaginject()"];
       lal_simulation [style=filled, color=lightgrey, URL="\ref pipeparts.mkinjections()", label="lal_simulation \n [iff --injections provided]"];
       framecppchannelmux [URL="\ref pipeparts.mkframecppchannelmux()"];
       framecppfilesink [URL="\ref pipeparts.mkframecppfilesink()"];

       // connect it up

       "mkbasicsrc()" -> progressreport1-> lal_shift  -> progressreport2;
       progressreport2 -> "whitened_multirate_src()" [label="if --whiten given"];
       "whitened_multirate_src()" -> lal_firbank;
       progressreport2 -> resample [label="if --whiten not given"];
       resample -> capsfilter;
       capsfilter -> audioamplify
       audioamplify -> lal_firbank;
       lal_firbank -> taginject -> lal_simulation -> framecppchannelmux -> framecppfilesink;
   }


Usage cases
-----------

1. Making initial LIGO colored noise, Advanced LIGO noise, etc for different
instruments.   See datasource.append_options() for other choices::

   $ gstlal_fake_frames --data-source=LIGO --channel-name=H1=FAKE-STRAIN \
   --frame-type=H1_FAKE --gps-start-time=900000000 --gps-end-time=900005000 \
   --output-path=testframes  --verbose

   $ gstlal_fake_frames --data-source=AdvLIGO --channel-name=L1=FAKE-STRAIN \
   --frame-type=L1_FAKE --gps-start-time=900000000 --gps-end-time=900005000 \
   --output-path=testframes  --verbose


2. Custom colored noise.

Obviously it is not practical to code up every possible noise curve to
simulate as a custom data source.  However, it is possible to provide your
own custom noise curve as an ASCII file with frequency in one column and
strain/Hz in the second. e.g., early advanced ligo noise curve
https://git.ligo.org/lscsoft/gstlal/raw/master/gstlal/share/early_aligo_asd.txt
You will need to convert ASD text files to PSD xml files using
gstlal_psd_xml_from_asd_txt first.::

	$ gstlal_fake_frames --data-source=white --color-psd v1psd.xml.gz \
	--channel-name=V1=FAKE-STRAIN --frame-type=V1_FAKE --gps-start-time=900000000 \
	--gps-end-time=900005000 --output-path=testframes  --verbose


3. Recoloring existing frame data

This procedure is very similar to the above except that instead of
using white noise to drive the fake frame generation, we start with
real frame data and whiten it.  Recoloring can be thought of as first
whitening data and then applying a coloring filter.  You must first
make a reference PSD of the data with e.g. gstlal_reference_psd.  You
will need to make a frame cache of the frames to recolor.::

	$ gstlal_fake_frames --whiten-reference-psd reference_psd.xml.gz \
	--color-psd recolor_psd.xml.gz --data-source frames \
	--output-path /path/to/output --output-channel-name h_16384Hz \
	--gps-start-time 966384031 --frame-type T1300121_V1_EARLY_RECOLORED_V2 \
	--gps-end-time 966389031 --frame-duration 16 --frames-per-file 256 \
	--frame-cache frame.cache --channel-name=V1=h_16384Hz --verbose


4. Creating injections into silence (assumes you have an injection xml file from e.g. lalapps_inspinj)::

	$ gstlal_fake_frames --data-source silence --output-path /path/to/output \
	--gps-start-time 966384031 --frame-type V1_INJECTIONS \
	--gps-end-time 966389031 --frame-duration 16 --frames-per-file 256 \
	--verbose --channel-name=V1=INJECTIONS --injections injections.xml


5. Other things are certainly possible, please add some!

Debug
-----

It is possible to check the pipeline graph by interupting the running process
with ctrl+c if you have the GST_DEBUG_DUMP_DOT_DIR evironment set.  A dot
graph will be written to gstlal_fake_frames.  Here is an example::

	$ GST_DEBUG_DUMP_DOT_DIR="." gstlal_fake_frames --data-source=LIGO \
	--channel-name=H1=FAKE-STRAIN --frame-type=H1_FAKE \
	--gps-start-time=900000000 --gps-end-time=900005000 \
	--output-path=testframes  --verbose``

You should see::

	Wrote pipeline to ./gstlal_fake_frames_PLAYING.dot

After Ctrl+c you should see::

	^C*** SIG 2 attempting graceful shutdown (this might take several minutes) ... ***
		Wrote pipeline to ./gstlal_fake_frames.dot

You can then turn the pipeline graph into an image with dot, e.g.,::

	$ dot -Tpng gstlal_fake_frames_PLAYING.dot > test.png

Review
------

Command line options
--------------------



.. code-block:: none

   Usage: gstlal_fake_frames [options]
   
   Options:
     -h, --help            show this help message and exit
     --shift=ns            Number of nanoseconds to delay (negative) or advance
                           (positive) the time stream
     --sample-rate=Hz      Sample rate at which to generate the data, should be
                           less than or equal to the sample rate of the measured
                           psds provided, default = 16384 Hz, max 16384 Hz
     --whiten-reference-psd=name
                           Set the name of psd xml file to whiten the data with
     --whiten-track-psd    Whiten the data by tracking PSD changes, can be
                           combined with --whiten-reference-psd to seed the
                           process.
     --color-psd=name      Set the name of psd xml file to color the data with
     --output-path=name    Path to output frame files (default = ".").
     --output-channel-name=name
                           The name of the channel in the output frames. The
                           default is the same as the channel name
     --output-frame-type=name
                           Frame type, required
     --output-frame-duration=s
                           Set the duration of the output frames.  The duration
                           of the frame file will be multiplied by --frames-per-
                           file.  Default: 16s
     --output-frames-per-file=n
                           Set the number of frames per file.  Default: 256
     -v, --verbose         Be verbose (optional).
   
     Data source options:
       Use these options to set up the appropriate data source
   
       --data-source=source
                           Set the data source from
                           [frames|framexmit|lvshm|nds|silence|white].  Required.
       --block-size=bytes  Data block size to read in bytes. Default 16384 * 8 *
                           512 (512 seconds of double precision data at 16384 Hz.
                           This parameter is only used if --data-source is one of
                           white, silence, AdvVirgo, LIGO, AdvLIGO, nds.
       --frame-cache=filename
                           Set the name of the LAL cache listing the LIGO-Virgo
                           .gwf frame files (optional).
       --frame-type=name   Set the frame type for a given instrument.  Can be
                           given multiple times as --frame-type=IFO=FRAME-TYPE.
                           Used with --data-source=frames
       --data-find-server=url
                           Set the data find server for LIGO data discovery. Used
                           with --data-source=frames
       --gps-start-time=seconds
                           Set the start time of the segment to analyze in GPS
                           seconds. Required unless --data-source=lvshm
       --gps-end-time=seconds
                           Set the end time of the segment to analyze in GPS
                           seconds.  Required unless --data-source=lvshm
       --injection-file=filename
                           Set the name of the LIGO light-weight XML file from
                           which to load injections (optional).
       --channel-name=name
                           Set the name of the channels to process.  Can be given
                           multiple times as --channel-name=IFO=CHANNEL-NAME
       --idq-channel-name=idqname
                           iDQ channel names to process. Must also provide idq-
                           state-channel-name. Can be given multiple times as
                           --idq-channel-name=IFO=IDQ-CHANNEL-NAME
       --idq-state-channel-name=idqstatename
                           iDQ state channel names to process. Can be given
                           multiple times as --idq-state-channel-name=IFO=IDQ-
                           STATE-CHANNEL-NAME
       --nds-host=hostname
                           Set the remote host or IP address that serves nds
                           data. This is required iff --data-source=nds
       --nds-port=portnumber
                           Set the port of the remote host that serves nds data.
                           This is required iff --data-source=nds
       --nds-channel-type=type
                           Set the port of the remote host that serves nds data.
                           This is required only if --data-source=nds.
                           default==online
       --framexmit-addr=name
                           Set the address of the framexmit service.  Can be
                           given multiple times as --framexmit-
                           addr=IFO=xxx.xxx.xxx.xxx:port
       --framexmit-iface=name
                           Set the multicast interface address of the framexmit
                           service.
       --state-channel-name=name
                           Set the name of the state vector channel.  This
                           channel will be used to control the flow of data via
                           the on/off bits.  Can be given multiple times as
                           --channel-name=IFO=CHANNEL-NAME
       --dq-channel-name=name
                           Set the name of the data quality channel.  This
                           channel will be used to control the flow of data via
                           the on/off bits.  Can be given multiple times as
                           --channel-name=IFO=CHANNEL-NAME
       --shared-memory-partition=name
                           Set the name of the shared memory partition for a
                           given instrument.  Can be given multiple times as
                           --shared-memory-partition=IFO=PARTITION-NAME
       --shared-memory-dir=name
                           Set the name of the shared memory directory for a
                           given instrument.  Can be given multiple times as
                           --shared-memory-dir=IFO=DIR-NAME
       --shared-memory-assumed-duration=SHARED_MEMORY_ASSUMED_DURATION
                           Set the assumed span of files in seconds. Default = 4.
       --shared-memory-block-size=SHARED_MEMORY_BLOCK_SIZE
                           Set the byte size to read per buffer. Default = 4096.
       --frame-segments-file=filename
                           Set the name of the LIGO light-weight XML file from
                           which to load frame segments.  Optional iff --data-
                           source=frames
       --frame-segments-name=name
                           Set the name of the segments to extract from the
                           segment tables.  Required iff --frame-segments-file is
                           given
       --state-vector-on-bits=bits
                           Set the state vector on bits to process (optional).
                           The default is 0x7 for all detectors. Override with
                           IFO=bits can be given multiple times.  Only currently
                           has meaning for online (lvshm) data.
       --state-vector-off-bits=bits
                           Set the state vector off bits to process (optional).
                           The default is 0x160 for all detectors. Override with
                           IFO=bits can be given multiple times.  Only currently
                           has meaning for online (lvshm) data.
       --dq-vector-on-bits=bits
                           Set the DQ vector on bits to process (optional).  The
                           default is 0x7 for all detectors. Override with
                           IFO=bits can be given multiple times.  Only currently
                           has meaning for online (lvshm) data.
       --dq-vector-off-bits=bits
                           Set the DQ vector off bits to process (optional).  The
                           default is 0x160 for all detectors. Override with
                           IFO=bits can be given multiple times.  Only currently
                           has meaning for online (lvshm) data.