In the command line help for
bayestar-localize-coincsand in the
COMMENTcard in the output FITS file, explain that the integer value in the
OBJECTcard in the FITS header is a row ID that refers to a coinc_event table row in the input LIGO-LW document.
bayestar-realize-coincsin order to simulate early warning triggers.
In sky maps produced by
bayestar-localize-coincs, the FITS headers now contain
OBJECTidentifiers that are integer event IDs (such as
1) rather than strings (such as
ligo-skymap-statstool now recognizes FITS headers with either integer or string
Use Astropy rather than LAL for GPS to UTC time conversion in FITS headers so that LALSuite is not a requirement for reading and writing FITS files.
ligo-skymap-statsto unify its multiprocessing and progress bar implementation with other command line tools.
Update the compiler version that is used to build Linux wheels to icc 188.8.131.52 from Intel Parallel Studio XE 2020u2.
Port the Python C extension to the limited stable Python API so that one binary wheel works for all supported Python versions for any given operating system. See PEP 384.
Eliminate global static variables from the Python C extension to enable compatibility with Python subinterpreters. See PEP 3121.
Improve the numerical stability of the method
ligo.skymap.distance.conditional_ppf()by reparametrizing the equation that is being solved. This method, which calculates the inverse of the distance CDF, works by solving the equation \(f(x) - p = 0\) for \(x\), where \(f(x)\) is the distance CDF, and \(p\) is the desired probability.
The reparametrized equation is \(log(1 - f(x)) - log(1 - p) = 0\) if \(p > 1/2\) and \(log(f(x)) - log(p) = 0\) otherwise. This reparametrization is effective because it improves the dynamic range in the tails of the distribution. This same reparametrization had already proven effective in the related method
This change also fixes some rare corner cases where
marginal_ppf()returned silly values becauses it uses
conditional_ppf()internally to create its own initial guess. One example was the median distance for the binary neutron star candidate S191205ah. Before this patch, the result was negative and invalid:
>>> from ligo.skymap.distance import marginal_ppf >>> from ligo.skymap.moc import uniq2pixarea >>> from ligo.skymap.io import read_sky_map >>> url = 'https://gracedb.ligo.org/apiweb/superevents/S191205ah/files/bayestar.multiorder.fits' >>> s = read_sky_map(url, moc=True) >>> marginal_ppf(0.5, s['PROBDENSITY'] * uniq2pixarea(s['UNIQ']), ... s['DISTMU'], s['DISTSIGMA'], s['DISTNORM']) /Users/lpsinger/src/ligo.skymap/ligo/skymap/util/numpy.py:46: RuntimeWarning: invalid value encountered in marginal_ppf return func(*args, **kwargs) -223357.8508233767
After this patch, the result is positive and sensible:
>>> marginal_ppf(0.5, s['PROBDENSITY'] * uniq2pixarea(s['UNIQ']), ... s['DISTMU'], s['DISTSIGMA'], s['DISTNORM']) 362.7485740018039
Increase the range of validity of the solver used in
ligo.skymap.distance.moments_to_parameters()for low-probability pixels that are very prior dominated. Sky maps that have many such pixels could have credible volumes repoted as infinity. The incidence of such cases should now be decreased.
Correct the alignment of Numpy record arrays passed to
ligo.skymap.moc.rasterize()in order to avoid possibly undefined behavior that was detected by UBSan.
Normalize column names when an ASCII file is passed to
Migrate LIGO-LW XML support from the
glue.ligolwmodule to the newer and better maintained
Teach BAYESTAR to accept either string row IDs (such as
sngl_inspiral:event_id:1) or integer row IDs (such as
map()implementation that is used by a number of the package’s command line tools will now yield results in order as quickly as they arrive, rather than sorting all of the results at the end. This should provide a very modest speedup in some command line tools.
Replace a call to the
aligned_allocfunction with the
aligned_allocfunction is part of the C11 standard library, but is missing on some platforms, particularly very old versions of macOS.
This fixes an issue with building Conda packages.
Fix an out of bounds access in the bicubic interpolation function that BAYESTAR uses to evaluate the integral over distance. Due to the relationship between the lookup table bounds and BAYESTAR’s distance limits of integration, the corner case that caused out of bounds access was never triggered. This bug had no impact on localizations generated by BAYESTAR.
More performance improvements in BAYESTAR providing a 2x speedup. For benchmark results, see the new How fast is BAYESTAR? section in the manual.
bicubic_interp_evalhad not being effectively autovectorized by the compiler. Rewrite it in explicitly vector form using the GCC vector extension (which is also supported by clang and icc) and selected vector intrinsics. In x86_64 builds, gcc, clang, and icc will now emit SSE2, SSE4.1, and FMA instructions for this code.
Pre-evaluate the SNR=0 limit of the distance integral to move some conditionals and logarithms out of BAYESTAR’s innermost loop.
Add loop count hints to improve the efficacy of loop unrolling.
Perform manual loop fission in
Update ligo.skymap to the latest version of the Astropy affiliated package template. Migrate package infrastructure from APE 4 to APE 17. The astropy-helpers submodule has been removed, and the package now includes a pyproject.toml file (see PEP 517 and PEP 518).
As a consequence of migrating to APE 17 and switching to setuptools_scm, the version of ligo.skymap will be reported slightly differently. The
ligo.skymap.__githash__variable has been removed, and instead the git hash will be part of the
ligo.skymap.__version__version string for unreleased, local versions.
ligo.skymaptools that generate FITS files (
ligo-skymap-from-samples) will no longer populate the
DATE-BLDkeys in FITS headers.
Fix incorrect legends on histograms generated by
bayestar-localize-lvalertscripts are called with
--loglevel=infoor higher, they will now output additional runtime measurements. Specifically, they will output the “real” time (wall clock time), “user” time (total time spent in userland across all threads), and “sys” time (total time spent in kernel land across all threads), similar to the UNIX time(1) tool. Here is an example of the formatting:
2020-05-12 18:57:12,024 INFO finished computationally-intensive section in real=0.918s, user=36.339s, sys=0.293s
import ligo.skymapby up to a second by replacing uses of
pkg_resourceswith the new Python standard library module
importlib.resources(or, for Python < 3.7, the backport
importlib_resources). The old
pkg_resourcesmodule is known to be slow because it does a lot of work on startup. (See, for example, https://github.com/pypa/setuptools/issues/926 and https://github.com/pypa/setuptools/issues/510.)
Drop dependency on seaborn.
Move some rarely used imports (
astropy.convolution) from module scope to function scope to speed up imports by up to half a second on NFS filesystems.
Update installation instructions to state that installation with pip requires pip 19.3 or newer. This has been the case since ligo.skymap 0.1.16.
Teach BAYESTAR to respect the
f_finalcolumn in the
sngl_inspiraltable for pre-merger, early warning templates.
Ensure that BAYESTAR’s arrival time prior is long enough to contain at least half a cycle of the template autocorrelation sequence. Previously, the duration of the arrival time prior was calculated solely from the light travel times between the participating detectors. This fixes an issue where SNR time series for early-warning events could have been cropped to only 1-3 samples.
Change BAYESTAR’s strategy for evaluating SNR time series from Catmull-Rom interpolation of the real and imaginary parts to Catmull-Rom interpolation of the amplitude and phase. The old interpolation method could produce oscillatory artifacts in the SNR amplitude if the data are nearly critically sampled, as is the case for early-warning BNS events. The new interpolation method is immune to this kind of artifact, and also has much faster convergence as a function of sample rate.
Lift the code to apply time shifts to SNR series outside of BAYESTAR’s inner loop because there are no data dependencies on the variables of integration. This is seen to speed up BAYESTAR by 30%.
Add software version and command line arguments metadata to the output of
Fix a bug in the Lanczos sub-sample arrival time interpolant: the Lanczos kernel should be zero for
abs(t) >= a.
requirements.txtfile and list dependencies in
bayestar-localize-coincswill no longer create HTCondor user log files because the large number of open log files could strain the filesystem if submitting from an NFS mount. This should reduce issues with held jobs on certain LIGO Data Grid clusters.
Fix deprecation warning in
Remove the deprecated
ligo.skymap.postprocess.find_injection_mocmethod, which has been renamed to
Update the compiler version that is used to build Linux wheels to icc 184.108.40.206 from Intel Parallel Studio XE 2020u0. Due to C ABI requirements, the wheels are now built for the manylinux2014 standard.
Fix a unit test failure with astropy < 4.0.0.
Add support for all combinations of map projection options, including
geo degrees globeand
geo degrees zoom. Also,
astroby itself is shorthand for
astro hours, and
geoby itself is short for
ligo-skymap-plotnow supports a variety of projections using the
Turn on continuous integration testing for Python 3.8.
Change the license for the project as a whole to GPL 3.0 or later (GPLv3+). Previously, the source files had been a mix of GPLv2+ and GPLv3+.
ligo-skymap-contour-moccommand line to create a credible region in a MOC (Multi Order Coverage) data structure. The input can be either a multiresolution or a flattened HEALPix probability map.
Add support for the
--detector-disabledcommand line option to the
bayestar-localize-coincstool, for consistency with
Remove installation dependency on astroquery, because it is only needed for the unit tests.
Add a monkey patch to work around a regression in Astropy 3.2 that broke WCS transformations from ITRS to ICRS coordinates. See https://github.com/astropy/astropy/pull/9609.
Fix a bug in the Python C extension code that could cause out-of-memory errors to be misreported as a SystemError with the message
<built-in function rasterize> returned NULL without setting an error, instead of as a MemoryError.
bayestar-injectscript now assumes that the source distribution is specified per unit comoving volume per unit proper time, rather than per unit comoving volume per unit observer time. This is in agreement with the conventional definition for LIGO/Virgo astrophysical rates.
ligo-skymap-from-samplesscripts now accept an optional integer value for the
-jflag to set the number of subprocesses.
ligo-skymap-from-sampleswill use all posterior samples if the value of the
--maxptsargument is greater than or equal to the number of posterior samples.
billiardpackage is present, then use it instead of the
multiprocessingstandard library module to parallelize
ligo-skymap-from-samplesso that the script’s Python entry point can be called from daemon processes (for example, inside Celery tasks).
Switch from WMAP9 to Planck15 cosmological parameters.
ligo.skymap.kde.Clustered2DSkyKDE.as_healpix()has an optional
top_nsideto allow for better initial grid, before refinement.
ligo-skymap-from-sampleshas an additional
Build macOS wheels with OpenMP.
Record the command line with which
ligo-skymap-statswas called by writing it to the ASCII table output as a comment line starting with
Fix a regression that caused
ligo-skymap-flattento fail for 2D sky maps.
Add installation instructions for both pip and conda.
ligo.skymap.postprocess.crossmatchmodule for fast cross-matching of sky maps with galaxy redshift catalogs.
This module used to be named
ligo.skymap.postprocess.find_injectionbecause it was originally designed for recovering injections (simulated signals) from sky localization simulations. We changed the name because galaxy cross matching is probably a more common use case than injection finding.
crossmatch()method also got some performance improvements for cross matching of large numbers of targets. Previously, to process \(n\) targets, it took about \((4 + 0.008 n)\) seconds — for a catalog of 300k targets, about 40 minutes. Now, it takes about 4 seconds total regardless of the number of targets.
Note that the
ligo.skymap.postprocess.crossmatchAPI is likely to change as documentation for it improves.
Several performance improvements for BAYESTAR:
Add GCC branch prediction hints.
Exploit nested parallelism in radial integrator lookup table generation.
Calculate signal amplitudes using single-precision floating point.
Add tracepoints for Intel’s Instrumentation and Tracing Technology (ITT) API, which can be enabled at build time by passing the
python setup.py build.
Switch from using the GNU Compiler Collection (gcc) to the Intel C Compiler (icc) for building optimized Linux binaries. On Intel Skylake machines, this can speed up BAYESTAR by 1.3x or more.
Due to icc’s C ABI requirements, Linux wheels now target the manylinux2010 platform tag.
In BAYESTAR, change the OpenMP scheduling kind from
static(the default) to
guided. This improves CPU utilization by load-balancing work across threads more efficiently.
ligo-skymap-constellations, an easter egg program to list the most probable constellations for a localization, for fun and for public outreach purposes.
Switch the implementation of the
astropy.convolution.convolve_fftin order to correctly handle points near the projection boundary where invalid values must be masked out.
AutoScaledWCSAxesas a Matplotlib projection with the name
astro wcsso that subclasses can be created using
plt.axes(..., projection='astro wcs', header='...').
Suppress Numpy warnings for HEALPix reprojection operations in WCS plots because it is normal for invalid values to occur when transforming pixels that lie outside of the projection.
geo globe, and
astro zoomto rotate the plot in the plane of the screen about the center of the projection.
Pass through keyword arguments from
AutoScaledWCSAxes.scalebar().label()to Matplotlib so that plot styles can be adjusted easily.
Bump matplotlib version to >= 3.0.2 because of a bug that affected
ligo-skymap-unflattentool will now write multiresolution sky maps with pixels sorted by the
UNIQcolumn, as required by the standard multi-order coverage map serialization in FITS.
All functions in
ligo.skymap.mocnow assume that
uniqis a signed integer. This makes it easier to call these functions with Numpy indexing routines, which work with signed integers. Also, saved multi-order sky maps will now be read correctly by tools such as
fvfrom HEASOFT, which do not correctly handle unsigned integer columns.
Add timestamps to the command line tools’ default logging configuration in order to start characterizing the latency of BAYESTAR’s data handling stages.
Increase precision of BAYESTAR’s run time measurement for the FITS headers.
ligo-skymap-plot-observabilitytool to plot observability windows for many sites at once. Conceptually, this tool is a variation of
ligo-skymap-plot-airmassin which the sky position is integrated out.
ligo-skymap-plot-airmasstool will now use the color map’s full dynamic range.
ligo-skymap-flattento support flattening multi-resolution HEALPix datasets to specified resolutions.
ligo-skymap-statsnow ignores skymaps with no corresponding entries in the inspinjfind database, instead of failing.
Add options to
ligo-skymap-plot-airmassto specify site coordinates explicitly rather than by a site nickname.
Fix a bug caused by improper floating point comparison that caused some contours to be missing from the output of
ligo-skymap-contourby skipping pixels that lie completely on the interior or exterior of the contour. For a typical LIGO/Virgo HEALPix map with a resolution of nside=512, the run time has decreased from about 42 seconds to 3 seconds.
ligo-skymap-from-samplestools will now generate multiresolution FITS files by default.
ligo-skymap-from-samplesto support storing metadata about which detectors contributed data.
Fix a bug in
ligo-skymap-plot-airmassthat caused the airmass chart to be blank if the lower and upper credible levels were always in opposite hemispheres. The root cause was that
plt.fill_betweendoes not clip infinities to the plot’s data range.
Require lalsuite >6.53 and lscsoft-glue >=2.0.0 due to breaking changes in API and behavior for LIGO-LW XML reading.
Pin lalsuite at <=6.52 and lscsoft-glue at <=1.60.0 due to breaking changes in API and behavior for LIGO-LW XML reading.
ligo-skymap-unflattentool to convert flat, fixed resolution, implicitly indexed HEALPix files to multi-resolution HEALPix files. This tools is the inverse of
Migrate from glue.segments to ligo.segments.
bayestar-localize-lvalertto control the limits of the isotropic prior over the inclination angle.
Un-pin ligo-segments and require version >= 1.2.0 due to packaging bugfixes.
Fix a bug that prevented the output of ligo-skymap-flatten from being gzip-compressed if the output filename ended in .gz.
Require astropy >= 3.1 because some code that we previously had to monkeypatch went upstream. See https://github.com/astropy/astropy-healpix/pull/106.
In the KDE clustering and
ligo-skymap-from-samples, disable OpenMP parallelism if Python mulitprocessing parallelism is enabled. This will prevent the program from spawning an excessive number of threads.
ligo-skymap-plotno longer requires a DATE-OBS entry in the FITS header when plotting in astronomical coordinates.
Fix a typo that caused ligo.skymap to always compile the bundled copy of chealpix instead of searching for a system version using pkgconfig.
Un-pin Numpy version now that Numpy 1.15.4 is out.
ligo-skymap-from-samplestools can now natively output multi-resolution HEALPix files, although they still natively output flat, fixed-resolution HEALPix files.
ligo-skymap-flattentool to convert multi-resolution HEALPix files to flat, fixed-resolution, implicitly indexed HEALPix files.
bayestar_samples_ppplotfrom LALInference as
ligo-skymap-plot-pp-samples, a tool for making P-P plots to compare a sky map with posterior samples.
ligo-skymap-statsto calculate comoving volumes.
bayestar-mcmc, correct a mistake in setting fixed parameters that undergo sampling transformations.
simulation_idvalues so that their integer values match the corresponding events’
coinc_event_idvalues. The option
--preserve-idsswitches back to the old behavior of preserving the original
Track rename of
ligo.gracedb.rest.GraceDb._service_urlin ligo-gracedb >= 2.0.1.
Update common files and submodules from the Astropy package template.
Work around a change (possibly a regression?) in Numpy 1.15.3 that broke Astropy by requiring numpy <= 1.15.2. See <https://github.com/astropy/astropy/issues/7943>.
Work around a bug introduced in ligo-segments 1.1.0 by requiring an earlier version of that package: its dependency on ligo-common, which does not correctly implement the namespace package
ligo, broke the continuous integration build.
Depend on astropy-healpix >= 0.3 to pick up a bug fix related to HEALPix bilinear interpolation that affected
ligo-skymap-plot. See <https://github.com/astropy/astropy-healpix/pull/106>.
Drop support for Python 3.5.
--condor-submitoption of the
bayestar-mcmctools now passes the submit file directives to
condor_submitvia stdin rather than on the command line, so that the number of jobs is not limited by the operating system’s maximum number of command line arguments.
Print warnings from
ligo.skymap.io.events.ligolw.open()only once per file to avoid excessive terminal output when reading large files.
bayestar-realize-coincsnow copies the process table from the injection file and fills in the SimInspiral table and associates coincidences with found injections. As a result, it is no longer necessary to run
lalapps_inspinjfindon the output to find injections.
bayestar-realize-coincsnow prints a running count of the number of injections that have been found and saved.
Add unit tests and binary wheels for Python 3.7.
Increase lifetime of continuous integration artifacts. The unit tests take longer now because they are more complete.
bayestar-mcmctool for pure Markov Chain Monte Carlo parameter estimation, without sky map postprocessing but with options for holding parameters at fixed values.
Fix a corner case in the initialization of the
distance.marginal_ppfsolver that could cause NaN return values.
numpy.genfromtxtUnicode deprecation warning in
ligo-skymap-plot-statsand update the minimum version of Numpy to 1.14. See the related Numpy changelog entry.
Silence deprecation warning in
ligo-skymap-plot-statsdue to Matplotlib renaming the
histmethod’s keyword argument from
bayestar-realize-coincstool now copies over spins from the input
sim_inspiraltable to the output
Some reasonable sub-sample trigger interpolation schemes can return peak times that are almost a full sample away from the maximum sample if the SNR time series has a pronounced skew in one direction in the vicinity of the maximum. Such an example occurs for the
catmull-rominterpolation method for the new unit tests in
ligo.skymap.bayestar.tests.test_interpolation. Because of this, relax the tolerance of BAYESTAR’s sanity check on single-detector trigger times and SNR series timestamps to a full sample.
ligo-skymap-plot-statsto reduce code duplication.
--measurement-error gaussian-noiseoption to
bayestar-realize-coincsto simulate a matched filter in Gaussian noise.
Remove deprecated module
bayestar_localize_lvalertwill now write the correct GraceDb URL to FITS file headers in the case that it is run with a non-default GraceDb server.
BAYESTAR’s SNR series time stamp assertions now include a bit more detail.
Add phase convention for gstlal-spiir, which needs to be confirmed upstream.
Fix datatype of simulated SNR time series produced by
Make lalsuite and lscsoft-glue required dependencies.
The Python code is now required to pass linting by Flake8.
On reading, rename columns from Fermi GBM HEALPix files to match the LIGO/Virgo convention. In particular, rename any column named
Reduce the memory footprint of
ligo-skymap-plot-airmassby transposing two nested loops.
Make some cosmetic improvements to
Add altitude and local time axes.
Center plot on local solar midnight.
Adjust blending and z-order of twilight shading.
ligo-skymap-plot-airmasswill now write an airmass table to stdout.
Rewrite the MCMC mode of BAYESTAR using
ligo.skymap.ez_emcee, a new reusable, fire-and-forget, parallel-tempering, MCMC sampler that features automated convergence testing and progress monitoring.
Update common files from Astropy package template.
ligo-skymap-combine, a tool to combine sky localizations from different observations into a joint skymap.
Add a new Astropy coordinate frame
ligo.skymap.coordinates.DetectorFrameto visualize triangulation rings with pairs of detectors.
Deprecate all functions in
Overhaul documentation so that all essential functionality is presented on the front page.
Require version 0.3.2 of the
reprojectpackage because of a regression that was caused by improper handling of nans in the
astropy-healpixpackage. See <https://github.com/astropy/astropy-healpix/pull/77>.
Declare the top-level
ligomodule as a namespace package.
Update common files from Astropy package template.
Enable Python version check in
setup.pyand top-level namespace package.
ligo-skymap-statswithout injections, instead of writing
nanvalues for irrelevant columns, don’t write the columns in the first place.
Start process of switching to tqdm for progress bars so that long-running operations show time estimates.
ligo-skymap-stats, disable OpenMP parallelism if running with
-jto avoid creating a huge number of threads on machines with very many cores.
ligo-skymap-plot-stats. The new
ligo-skymap-statsprogram can generate summary statistics for skymaps, with or without injection-finding.
This is the first version that has been tested and shown to reproduce the results in the “First Two Years” paper, which is the review benchmark.
Bring back simulation tools from LALSuite.
ligo-skymap-plot-airmass, a tool for probabilistic airmass charts.
Adjust CI configuration for uploading to PyPI.