Anomaly Detection and Approximate Similarity Searches of Transients in Real-time Data Streams

Abstract We present Lightcurve Anomaly Identification and Similarity Search ( LAISS ), an automated pipeline to detect anomalous astrophysical transients in real-time data streams. We deploy our anomaly detection model on the nightly Zwicky Transient Facility (ZTF) Alert Stream via the ANTARES broker, identifying a manageable ∼1–5 candidates per night for expert vetting and coordinating follow-up observations. Our method leverages statistical light-curve and contextual host galaxy features within a random forest classifier, tagging transients of rare classes ( spectroscopic anomalies), of uncommon host galaxy environments ( contextual anomalies), and of peculiar or interaction-powered phenomena ( behavioral anomalies). Moreover, we demonstrate the power of a low-latency (∼ms) approximate similarity search method to find transient analogs with similar light-curve evolution and host galaxy environments. We use analogs for data-driven discovery, characterization, (re)classification, and imputation in retrospective and real-time searches. To date, we have identified ∼50 previously known and previously missed rare transients from real-time and retrospective searches, including but not limited to superluminous supernovae (SLSNe), tidal disruption events, SNe IIn, SNe IIb, SNe I-CSM, SNe Ia-91bg-like, SNe Ib, SNe Ic, SNe Ic-BL, and M31 novae. Lastly, we report the discovery of 325 total transients, all observed between 2018 and 2021 and absent from public catalogs (∼1% of all ZTF Astronomical Transient reports to the Transient Name Server through 2021). These methods enable a systematic approach to finding the “needle in the haystack” in large-volume data streams. Because of its integration with the ANTARES broker, LAISS is built to detect exciting transients in Rubin data.

Authors

Institutions

Publication Details

Journal
The Astrophysical Journal
Published
2024-10-01
DOI
https://doi.org/10.3847/1538-4357/ad6869
Citations
7
Primary Topic
Gamma-ray bursts and supernovae
Type
article
Field-Weighted Citation Impact
2.32

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Anomaly Detection and Approximate Similarity Searches of Transients in Real-time Data Streams

R. J. Foley, D. O. Jones, E. A. Magnier, Christa Gall et al.
7 citations
The Astrophysical Journal
Gamma-ray bursts and supernovae
2.32
article

Anomaly Detection and Approximate Similarity Searches of Transients in Real-time Data Streams

R. J. Foley, D. O. Jones, E. A. Magnier, Christa Gall, Gautham Narayan, C. -C. Lin, Andy Berres, Vivienne Baldassare, Thomas de Boer, A. W. Engel, V. G. Shah, T. Matheson, Kaylee de Soto, Kyle W. Davis, K. S. Mandel, D. Farias, Benjamin M. Boyd, Matthew Grayling, C. R. Angus, Konstantin Malanchev, S. I. Raimundo, Alexander Gagliano, K. Auchettl, V. A. Villar, Monika Soraisam, K. C. Chambers, P. Aleo, Nicolas Esquivel, Sebastián Gómez, Hua Gao, R. J. Wainscoat, Sebastián Vicencio
article en
7 citations

Abstract

Abstract We present Lightcurve Anomaly Identification and Similarity Search ( LAISS ), an automated pipeline to detect anomalous astrophysical transients in real-time data streams. We deploy our anomaly detection model on the nightly Zwicky Transient Facility (ZTF) Alert Stream via the ANTARES broker, identifying a manageable ∼1–5 candidates per night for expert vetting and coordinating follow-up observations. Our method leverages statistical light-curve and contextual host galaxy features within a random forest classifier, tagging transients of rare classes ( spectroscopic anomalies), of uncommon host galaxy environments ( contextual anomalies), and of peculiar or interaction-powered phenomena ( behavioral anomalies). Moreover, we demonstrate the power of a low-latency (∼ms) approximate similarity search method to find transient analogs with similar light-curve evolution and host galaxy environments. We use analogs for data-driven discovery, characterization, (re)classification, and imputation in retrospective and real-time searches. To date, we have identified ∼50 previously known and previously missed rare transients from real-time and retrospective searches, including but not limited to superluminous supernovae (SLSNe), tidal disruption events, SNe IIn, SNe IIb, SNe I-CSM, SNe Ia-91bg-like, SNe Ib, SNe Ic, SNe Ic-BL, and M31 novae. Lastly, we report the discovery of 325 total transients, all observed between 2018 and 2021 and absent from public catalogs (∼1% of all ZTF Astronomical Transient reports to the Transient Name Server through 2021). These methods enable a systematic approach to finding the “needle in the haystack” in large-volume data streams. Because of its integration with the ANTARES broker, LAISS is built to detect exciting transients in Rubin data.

The Astrophysical JournalVol. 974(2)
University of Hawaiʻi at Mānoa (US), University of Copenhagen (DK), Queen's University Belfast (GB), Space Telescope Science Institute (US), University of Hawaii System (US), Pacific Northwest National Laboratory (US), University of Illinois Urbana-Champaign (US), The University of Melbourne (AU), University of California, Santa Cruz (US), Center for Astrophysics Harvard & Smithsonian (US), National Center for Supercomputing Applications (US), NSF NOIRLab (US), Niels Brock (DK), The NSF AI Institute for Artificial Intelligence and Fundamental Interactions (US), University of Southampton (GB), Washington State University (US), Carnegie Mellon University (US)
National Science Foundation, David and Lucile Packard Foundation, Smithsonian Institution, U.S. Department of Energy, National Aeronautics and Space Administration, Alfred P. Sloan Foundation, Gordon and Betty Moore Foundation, Battelle, Brinson Foundation, University of California, Santa Cruz, California Institute of Technology, Northwestern University, University of Washington, Johns Hopkins University, Villum Fonden, National Centre for Supercomputing Applications, Space Telescope Science Institute, Heising-Simons Foundation, University of Warwick, European Commission, Queen's University Belfast, Durham University, Trinity College Dublin, Weizmann Institute of Science, Queen's University, Nuclear Safety and Security Commission, National Central University, Stockholms Universitet, Eötvös Loránd Tudományegyetem, Max-Planck-Institut für Astronomie, Directorate for Computer and Information Science and Engineering, University of Illinois at Urbana-Champaign, Smithsonian Astrophysical Observatory, Science Mission Directorate, Science and Technology Facilities Council, Deutsches Elektronen-Synchrotron, Institut National de Physique Nucléaire et de Physique des Particules, Laboratory Directed Research and Development, Planetary Science Division, Los Alamos National Laboratory, Pacific Northwest National Laboratory
Life in Land
Openalex Percentile: Top 12%
Gamma-ray bursts and supernovae
2.32
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.