Testing

Unit tests

This package has a unit test suite that is set up based on the Astropy Testing Guidelines. The unit tests are run automatically on every git push using GitLab Continuous Integration (CI). See the repository’s .gitlab-ci.yml file for configuration details.

You can also run the unit tests manually by running these commands in the source directory:

$ pip install -e .[test]
$ pytest

There are many options available to adjust what tests are run or how test results are reported; see Astropy’s documentation on running tests for details.

Unit tests in ligo.skymap come in three forms:

  • Test modules: Generally, each Python subpackage of ligo.skymap has a tests directory containing Pytest-style unit tests. For example, unit tests for ligo.skymap.plot are in ligo/skymap/plot/tests.

  • Doctests: The documentation for many modules contains inline examples with expected console output called doctests. When you run the test suite, it checks that the output matches. An example of a module with many doctests is ligo.skymap.distance (see source).

  • C unit tests: The critical sections of C code have unit tests that are written in C using GSL’s test framework (see gsl_test.h). These tests are defined in the functions bayestar_test() in src/bayestar_sky_map.c and cubic_interp_test() in src/cubic_interp_test.c.

Coverage analysis

The CI pipeline does code coverage analysis when it runs the unit tests. See the coverage report for the most recent build.

Acceptance tests

There is a suite of weekly acceptance tests for BAYESTAR that check that the code reproduces localizations for past gravitational-wave events including GW170814 and GW170817 as well as populations of simulated events.

Review

This package comprises an analysis pipeline of the Compact Binary Coalescence (CBC) Working Group of the LIGO Scientific Collaboration (LSC). As such, it is required to undergo an LSC scientific review (private wiki link). Minutes of ligo.skymap’s LSC review team are kept in the project’s private wiki.

Scope

The essential science capabilities that undergo LSC review are:

Roles

All review tests (the unit tests and acceptance tests described above) should be automated, take no more than 2 hours to run, and indicate success or failure in a self-evident way. The task of the developers is to create and maintain the automated tests. The task of the reviewers is to provide oversight to ensure that the tests are necessary and sufficient to cover the scientific functionality that is under review.

Changes

ligo.skymap is a stable, mature package. Most changes are conservative and maintenance-oriented. The developers will not usually contact the reviewers about these kinds of changes before merging into the main branch. Examples of these kinds of change are adjustments to track API changes in Python, Numpy, and Astropy.

However, the developers will flag potential changes that might need extra scrutiny because they could science results by adding the Review label to merge requests and requiring an approval from a reviewer before merging. Examples of these changes are extracting new parameters from BAYESTAR (e.g. inclination angles) or making significant changes to algorithm inner loops that could affect floating point accuracy.

Releases

All stable releases (versions that are triples of numbers of the form 1.2.3) of ligo.skymap have been approved by the review team. We create one or more release candidates (versions that are of the form 1.2.3rcN for some number N) until the latest release candidate satisfies all of the tests and is verbally approved by the review team. Then we do a stable release. The review team indicates its formal assent to the release by approving the corresponding ticket in the LSC Software Change Control Board (SCCB) issue tracker (private link).