Detailed Description

Header file for cleaning routines.

Module containing routines for dealing with spectral disturbances in SFTs.

Author: Badri Krishnan, Alicia Sintes, Greg Mendell

Routines for cleaning SFT files using known spectral disturbances.

Synopsis

#include <lal/SFTClean.h>

Format for list of known spectral disturbances and using them to clean SFT data

Error conditions

Test program. %%

Author: Badri Krishnan, Alicia Sintes, Greg Mendell

Description

This module contains routines for dealing with lists of known spectral disturbances in the frequency domain, and using them to clean SFTs.

The basic input is a text file containing a list of known spectral lines. NOTE: the legacy format used here is not identical with that of modern Advanced LIGO linefiles. To run this code with newer linefiles, they should first be converted into the legacy format. An example of the supported format is the following:

* 0.0      0.25     4000     0.0        0.0   0.25Hzlines
* 0.0      60.0     20       0.5        0.5   60Hzlines
* 0.0      16.0     100      0.0        0.0   16Hzlines
* 166.7    0.0      1        0.0        0.0   Calibrationline
* 345.0    0.0      1        3.0        3.0   violinmodes
*

The file consists of rows with 6 columns each. Each row has information about a set of spectral lines of the form \( f_n = f_0 + n\Delta f \) . The first column is the start frequency \( f_0 \) , the second column is the spacing \( \Delta f \) , the third column is the total number of lines, the fourth column is the left-width of each line (in Hz), the fifth column is the width on the right, and the last column is a brief comment string (no spaces). If this file is meant to be used for cleaning SFTs, then certain features which the user must be aware of are explained in the documentation of the function LALCleanCOMPLEX8SFT().

Uses

void LALFindNumberHarmonics (LALStatus, LineHarmonicsInfo, CHAR)
void LALReadHarmonicsInfo (LALStatus, LineHarmonicsInfo, CHAR)
void LALHarmonics2Lines (LALStatus, LineNoiseInfo, LineHarmonicsInfo)
void LALChooseLines (LALStatus, LineNoiseInfo, LineNoiseInfo, REAL8, REAL8)
void LALCheckLines ( LALStatus, INT4, LineNoiseInfo, REAL8)
void LALFindNumberLines (LALStatus, LineNoiseInfo, CHAR)
void LALReadLineInfo (LALStatus, LineNoiseInfo, CHAR)
void LALCleanCOMPLEX8SFT (LALStatus, SFTtype, INT4, INT4, LineNoiseInfo, RandomParams)

Prototypes
void	LALFindNumberHarmonics (LALStatus status, LineHarmonicsInfo harmonicInfo, CHAR *fname)
	Looks into the input file containing list of lines, does some checks on the file format, and calculates the number of harmonic sets in this file. More...

void	LALReadHarmonicsInfo (LALStatus status, LineHarmonicsInfo lineInfo, CHAR *fname)
	Reads in the contents of the input line-info file and fills up the LineHarmonicsInfo structure. More...

void	LALHarmonics2Lines (LALStatus status, LineNoiseInfo lineInfo, LineHarmonicsInfo *harmonicsInfo)
	Converts the list of harmonic sets into an explicit list of spectral lines. More...

void	LALChooseLines (LALStatus status, LineNoiseInfo outLine, LineNoiseInfo *inLine, REAL8 freqMin, REAL8 freqMax)
	Takes a set of spectral lines and a frequency range as input and returns those lines which lie within the specified frequency range. More...

void	LALCheckLines (LALStatus status, INT4 flag, LineNoiseInfo *lines, REAL8 freq)
	Function to count how many lines affect a given frequency. More...

void	LALFindNumberLines (LALStatus status, LineNoiseInfo lineInfo, CHAR *fname)
	Finds total number of spectral-lines contained in case the input file is a list of explicit spectral lines – obsolete. More...

void	LALReadLineInfo (LALStatus status, LineNoiseInfo lineInfo, CHAR *fname)
	Reads information from file containing list of explicit lines - obsolete. More...

void	LALCleanCOMPLEX8SFT (LALStatus status, SFTtype sft, INT4 width, INT4 window, LineNoiseInfo lineInfo, RandomParams randPar)
	Function for cleaning a SFT given a set of known spectral disturbances. More...

void	LALCleanSFTVector (LALStatus status, SFTVector sftVect, INT4 width, INT4 window, LineNoiseInfo lineInfo, RandomParams randPar)
	Function to clean a sft vector – calls LALCleanCOMPLEX8SFT repeatedly for each sft in vector. More...

void	LALCleanMultiSFTVect (LALStatus status, MultiSFTVector multVect, INT4 width, INT4 window, LineNoiseInfo lineInfo, RandomParams randPar)
	Function to clean a sft vector – calls LALCleanCOMPLEX8SFT repeatedly for each sft in vector. More...

void	LALRemoveKnownLinesInSFTVect (LALStatus status, SFTVector sftVect, INT4 width, INT4 window, CHAR linefile, RandomParams randPar)
	function to remove lines from a sft vector given a file containing list of lines More...

void	LALRemoveKnownLinesInMultiSFTVector (LALStatus status, MultiSFTVector multiSFTVect, INT4 width, INT4 window, LALStringVector linefiles, RandomParams randPar)
	top level function to remove lines from a multi sft vector given a list of files containing list of known spectral lines More...

Data Structures
struct	LineNoiseInfo
	structure for storing list of spectral lines – constructed by expanding list of harmonics More...

struct	LineHarmonicsInfo
	structure for storing the contents of the input list of known spectral disturbances More...

Error Codes
#define	SFTCLEANH_ENULL 1

#define	SFTCLEANH_EFILE 2

#define	SFTCLEANH_EHEADER 3

#define	SFTCLEANH_EENDIAN 4

#define	SFTCLEANH_EVAL 5

#define	SFTCLEANH_ELINENAME 6

#define	SFTCLEANH_ESEEK 9

#define	SFTCLEANH_EREAD 10

#define	SFTCLEANH_EWRITE 11

#define	SFTCLEANH_MSGENULL "Null pointer"

#define	SFTCLEANH_MSGEFILE "Could not open file"

#define	SFTCLEANH_MSGEHEADER "Incorrect header in file"

#define	SFTCLEANH_MSGEENDIAN "Incorrect endian type"

#define	SFTCLEANH_MSGEVAL "Invalid value"

#define	SFTCLEANH_MSGELINENAME "Invalid linefile name"

#define	SFTCLEANH_MSGESEEK "fseek failed"

#define	SFTCLEANH_MSGEREAD "fread failed"

#define	SFTCLEANH_MSGEWRITE "fwrite failed"

Function Documentation

◆ LALFindNumberHarmonics()

void LALFindNumberHarmonics	(	LALStatus *	status,
		LineHarmonicsInfo *	harmonicInfo,
		CHAR *	fname
	)

Looks into the input file containing list of lines, does some checks on the file format, and calculates the number of harmonic sets in this file.

Parameters

status	pointer to LALStatus structure
harmonicInfo	list of harmonics
fname	input filename

Definition at line 92 of file lib/SFTclean.c.

◆ LALReadHarmonicsInfo()

void LALReadHarmonicsInfo	(	LALStatus *	status,
		LineHarmonicsInfo *	harmonicsInfo,
		CHAR *	fname
	)

Reads in the contents of the input line-info file and fills up the LineHarmonicsInfo structure.

Appropriate memory must be allocated for this structure before this function is called.

Parameters

status	pointer to LALStatus structure
harmonicsInfo	list of harmonics
fname	input file

Definition at line 149 of file lib/SFTclean.c.

◆ LALHarmonics2Lines()

void LALHarmonics2Lines	(	LALStatus *	status,
		LineNoiseInfo *	lineInfo,
		LineHarmonicsInfo *	harmonicsInfo
	)

Converts the list of harmonic sets into an explicit list of spectral lines.

Parameters

status	pointer to LALStatus structure
lineInfo	output list of explicit lines
harmonicsInfo	input list of harmonics

Definition at line 210 of file lib/SFTclean.c.

◆ LALChooseLines()

void LALChooseLines	(	LALStatus *	status,
		LineNoiseInfo *	outLine,
		LineNoiseInfo *	inLine,
		REAL8	fMin,
		REAL8	fMax
	)

Takes a set of spectral lines and a frequency range as input and returns those lines which lie within the specified frequency range.

The output is a reduced list of spectral lines which either lie completely within the frequency range or whose wings overlap with the frequency range. This is useful for discarding unnecessary lines to save computational cost and memory.

Parameters

status	pointer to LALStatus structure
outLine	reduced list of lines
inLine	input list of lines
fMin	start of frequency band
fMax	end of frequency band

Definition at line 386 of file lib/SFTclean.c.

◆ LALCheckLines()

void LALCheckLines	(	LALStatus *	status,
		INT4 *	countL,
		LineNoiseInfo *	lines,
		REAL8	freq
	)

Function to count how many lines affect a given frequency.

Input is a list of lines and a frequency. The output is an integer which is equal to the number of lines which intersect this frequency. An output of zero indicates that the frequencty is not influenced by the lines. Note that the doppler broadening of the lines is taken into account while deciding whether the frequency is affected or not.

Parameters

status	pointer to LALStatus structure
countL	number of lines affecting frequency
lines	list of lines
freq	frequency to be checked

Definition at line 464 of file lib/SFTclean.c.

◆ LALFindNumberLines()

void LALFindNumberLines	(	LALStatus *	status,
		LineNoiseInfo *	lineInfo,
		CHAR *	fname
	)

Finds total number of spectral-lines contained in case the input file is a list of explicit spectral lines – obsolete.

Use instead LALFindNumberHarmonics().

Definition at line 282 of file lib/SFTclean.c.

◆ LALReadLineInfo()

void LALReadLineInfo	(	LALStatus *	status,
		LineNoiseInfo *	lineInfo,
		CHAR *	fname
	)

Reads information from file containing list of explicit lines - obsolete.

Use instead LALReadHarmonicsInfo()

Definition at line 329 of file lib/SFTclean.c.

◆ LALCleanCOMPLEX8SFT()

void LALCleanCOMPLEX8SFT	(	LALStatus *	status,
		SFTtype *	sft,
		INT4	width,
		INT4	window,
		LineNoiseInfo *	lineInfo,
		RandomParams *	randPar
	)

Function for cleaning a SFT given a set of known spectral disturbances.

The algorithm is the following. For each spectral line, first the central frequency of the line is converted to a bin index by round(tBase * freq). If the wings are set to zero, then only this central bin is cleaned. Note that if the frequency lies between two exactly resolved frequencies, then only one of these bins is cleaned. The user must know about the SFT timebase and make sure that the central frequency is an exactly resolved one. The wingsize is calculated in bins according to floor(tBase * wingsize). This is done separately for the left and right wings. Note the use of the floor function. If the wingsize corresponds to say 2.5 bins, then only 2 bins will be cleaned in addition to the central frequency.

Having decided which bins are to be cleaned, the next step is to produce random noise to replace the data in these bins. The fake random noise must mimic the behavior of the true noise in the vicinity of the spectral disturbance, and we must therefore estimate the noise floor in the vicinity of the disturbance. The user specifies a "window size" for cleaning, and this determines how many data points from the SFT are to be used for estimating this noise floor. Consider the number of frequency bins on each side of the spectral disturbance (excluding the wings) given by the user specified window size. The function calculates the SFT power in these bins and calculates their median. The median is then converted to a standard deviation for the real and imaginary parts of the SFT data, taking into account the median bias. There is also a width parameter. This is an upper limit on the number of bins that will be cleaned on either wing. Thus, if width is specified to say 3, then only a maximum 3 bins will be cleaned on either side irrespective of the actual wing size specified. This parameter is present for historical reasons, and should probably be removed. Currently, it is recommended to set this sufficiently large (larger that any wing size in bins) so that it has no effect.

Some "features" that must be kept in mind

The following is part of a email sent to pulgroup on 4/10/05. Some of these points have already been mentioned above.

i) If the width of the line to be cleaned is specified to be zero and if the central frequency is midway between two exactly resolved bins, then the code will only clean one frequency bin and not both as it perhaps should.

ii) The wings size is converted from Hz to frequency bins using the floor function. So some bins are being dropped at the edges due to rounding off.

iii) The cleaning uses data from neighboring bins to generate random noise. This is a problem if there are other spectral disturbances in the neighborhood or if the frequency bin is at the edge of the SFT. Shouldn't one make sure that the data used to generate the random number are un-contaminated?

Regarding the first two points, the user is supposed to know the properties of the SFTs which are being cleaned and the list of lines is meant to be tailored for a particular set of SFTs. Therefore, the user should know the timebaseline of the SFT to make sure that the central frequency value being specified is a resolved frequency bin and the wings should be specified correctly. In many cases, the size of the wings, when it is non-zero, is somewhat uncertain, and this uncertainty is often larger that this rounding off error.

Perhaps one should specify the frequency bin index of the central frequency value and the wing size also in bins, but this makes the code more cumbersome and non-intuitive to use.

Both of these points can be handled by documenting the code better, so that the user knows exactly what is being done and chooses the input appropriately . I will do this as soon as possible.

The third point is more difficult. Currently, the code calculates the median of the power in the neighboring bins and uses that to generate a random number with the appropriate standard deviation.

Let us first consider the cases when there are other spectral disturbances in the neighborhood so that the median estimate might be corrupted. When there are only a small number of narrow (only few bins) disturbances in the neighborhood, the use of the median is meant to eliminate the effects of these lines. However, this might not work for the cases when there is a broad spectral disturbance or when there are a large number of narrow disturbances.

If there are a large number of narrow spectral disturbances very close to each other, it might make more sense to group them together and consider them to be one single broad disturbance; we probably won't trust a detection near these lines anyway.

In the case of a broad spectral feature, it will certainly corrupt the median value and the random number generated won't be correct. The alternative is to skip the broad feature and take only undisturbed bins further away. This might be ok, but recall that the purpose of the cleaning is to replace a spectral disturbance with a random number whose distribution is similar to the noise in the neighborhood of the disturbance. For colored noise, if we have to go very far away in frequency to get a undisturbed frequency bin, then the random number distribution won't be correct either. Edge effects are taken care by making sure that the SFT being read in is large enough. There must be extra wings to the SFT equal to the window size used for cleaning. If the edge effects are important, then the code using the cleaning routines must read in the extra bins.

Parameters

status	pointer to LALStatus structure
sft	SFT to be cleaned
width	maximum width to be cleaned -- set sufficiently large if all bins in each line are to be cleaned
window	window size for noise floor estimation in vicinity of a line
lineInfo	list of lines
randPar	parameters for generating random noise

Definition at line 582 of file lib/SFTclean.c.

◆ LALCleanSFTVector()

void LALCleanSFTVector	(	LALStatus *	status,
		SFTVector *	sftVect,
		INT4	width,
		INT4	window,
		LineNoiseInfo *	lineInfo,
		RandomParams *	randPar
	)

Function to clean a sft vector – calls LALCleanCOMPLEX8SFT repeatedly for each sft in vector.

Parameters

status	pointer to LALStatus structure
sftVect	SFTVector to be cleaned
width	maximum width to be cleaned -- set sufficiently large if all bins in each line are to be cleaned
window	window size for noise floor estimation in vicinity of a line
lineInfo	list of lines
randPar	parameters for generating random noise

Definition at line 775 of file lib/SFTclean.c.

◆ LALCleanMultiSFTVect()

void LALCleanMultiSFTVect	(	LALStatus *	status,
		MultiSFTVector *	multVect,
		INT4	width,
		INT4	window,
		LineNoiseInfo *	lineInfo,
		RandomParams *	randPar
	)

Function to clean a sft vector – calls LALCleanCOMPLEX8SFT repeatedly for each sft in vector.

Parameters

status	pointer to LALStatus structure
multVect	SFTVector to be cleaned
width	maximum width to be cleaned -- set sufficiently large if all bins in each line are to be cleaned
window	window size for noise floor estimation in vicinity of a line
lineInfo	list of lines
randPar	parameters for generating random noise

Definition at line 814 of file lib/SFTclean.c.

◆ LALRemoveKnownLinesInSFTVect()

void LALRemoveKnownLinesInSFTVect	(	LALStatus *	status,
		SFTVector *	sftVect,
		INT4	width,
		INT4	window,
		CHAR *	linefile,
		RandomParams *	randPar
	)

function to remove lines from a sft vector given a file containing list of lines

Parameters

status	pointer to LALStatus structure
sftVect	SFTVector to be cleaned
width	maximum width to be cleaned -- set sufficiently large if all bins in each line are to be cleaned
window	window size for noise floor estimation in vicinity of a line
linefile	file with list of lines
randPar	for creating random numbers

Definition at line 853 of file lib/SFTclean.c.

◆ LALRemoveKnownLinesInMultiSFTVector()

void LALRemoveKnownLinesInMultiSFTVector	(	LALStatus *	status,
		MultiSFTVector *	multiSFTVect,
		INT4	width,
		INT4	window,
		LALStringVector *	linefiles,
		RandomParams *	randPar
	)

top level function to remove lines from a multi sft vector given a list of files containing list of known spectral lines

Parameters

status	pointer to LALStatus structure
multiSFTVect	SFTVector to be cleaned
width	maximum width to be cleaned
window	window size for noise floor estimation in vicinity of a line
linefiles	list of per-detector files with list of lines (the basename of each file must start with a canonical IFO name)
randPar	for creating random numbers

Definition at line 953 of file lib/SFTclean.c.

Macro Definition Documentation

◆ SFTCLEANH_ENULL

#define SFTCLEANH_ENULL 1

Definition at line 93 of file SFTClean.h.

◆ SFTCLEANH_EFILE

#define SFTCLEANH_EFILE 2

Definition at line 94 of file SFTClean.h.

◆ SFTCLEANH_EHEADER

#define SFTCLEANH_EHEADER 3

Definition at line 95 of file SFTClean.h.

◆ SFTCLEANH_EENDIAN

#define SFTCLEANH_EENDIAN 4

Definition at line 96 of file SFTClean.h.

◆ SFTCLEANH_EVAL

#define SFTCLEANH_EVAL 5

Definition at line 97 of file SFTClean.h.

◆ SFTCLEANH_ELINENAME

#define SFTCLEANH_ELINENAME 6

Definition at line 98 of file SFTClean.h.

◆ SFTCLEANH_ESEEK

#define SFTCLEANH_ESEEK 9

Definition at line 99 of file SFTClean.h.

◆ SFTCLEANH_EREAD

#define SFTCLEANH_EREAD 10

Definition at line 100 of file SFTClean.h.

◆ SFTCLEANH_EWRITE

#define SFTCLEANH_EWRITE 11

Definition at line 101 of file SFTClean.h.

◆ SFTCLEANH_MSGENULL

#define SFTCLEANH_MSGENULL "Null pointer"

Definition at line 103 of file SFTClean.h.

◆ SFTCLEANH_MSGEFILE

#define SFTCLEANH_MSGEFILE "Could not open file"

Definition at line 104 of file SFTClean.h.

◆ SFTCLEANH_MSGEHEADER

#define SFTCLEANH_MSGEHEADER "Incorrect header in file"

Definition at line 105 of file SFTClean.h.

◆ SFTCLEANH_MSGEENDIAN

#define SFTCLEANH_MSGEENDIAN "Incorrect endian type"

Definition at line 106 of file SFTClean.h.

◆ SFTCLEANH_MSGEVAL

#define SFTCLEANH_MSGEVAL "Invalid value"

Definition at line 107 of file SFTClean.h.

◆ SFTCLEANH_MSGELINENAME

#define SFTCLEANH_MSGELINENAME "Invalid linefile name"

Definition at line 108 of file SFTClean.h.

◆ SFTCLEANH_MSGESEEK

#define SFTCLEANH_MSGESEEK "fseek failed"

Definition at line 109 of file SFTClean.h.

◆ SFTCLEANH_MSGEREAD

#define SFTCLEANH_MSGEREAD "fread failed"

Definition at line 110 of file SFTClean.h.

◆ SFTCLEANH_MSGEWRITE

#define SFTCLEANH_MSGEWRITE "fwrite failed"

Definition at line 111 of file SFTClean.h.

Detailed Description

Synopsis

Error conditions

Description

Uses

Prototypes

Data Structures

Error Codes

Function Documentation

◆ LALFindNumberHarmonics()

◆ LALReadHarmonicsInfo()

◆ LALHarmonics2Lines()

◆ LALChooseLines()

◆ LALCheckLines()

◆ LALFindNumberLines()

◆ LALReadLineInfo()

◆ LALCleanCOMPLEX8SFT()

Some "features" that must be kept in mind

◆ LALCleanSFTVector()

◆ LALCleanMultiSFTVect()

◆ LALRemoveKnownLinesInSFTVect()

◆ LALRemoveKnownLinesInMultiSFTVector()

Macro Definition Documentation

◆ SFTCLEANH_ENULL

◆ SFTCLEANH_EFILE

◆ SFTCLEANH_EHEADER

◆ SFTCLEANH_EENDIAN

◆ SFTCLEANH_EVAL

◆ SFTCLEANH_ELINENAME

◆ SFTCLEANH_ESEEK

◆ SFTCLEANH_EREAD

◆ SFTCLEANH_EWRITE

◆ SFTCLEANH_MSGENULL

◆ SFTCLEANH_MSGEFILE

◆ SFTCLEANH_MSGEHEADER

◆ SFTCLEANH_MSGEENDIAN

◆ SFTCLEANH_MSGEVAL

◆ SFTCLEANH_MSGELINENAME

◆ SFTCLEANH_MSGESEEK

◆ SFTCLEANH_MSGEREAD

◆ SFTCLEANH_MSGEWRITE