X-Pypeline

Porting LIGO's burst detection pipeline to Python

 

Angus Gray-Weale

ADACS Swinburne

X-Pipeline Users Meeting — March 2026

Goal

Port X-Pipeline from MATLAB to Python

  • Access the Python scientific ecosystem (numpy, scipy, matplotlib)
  • Remove MATLAB licence dependency
  • Enable community contributions and CI/CD
  • Modern deployment: pip install, containers, OSG

Change of Approach

Since the last talk

Previously: Traditional Parsers

  • smop and other MATLAB-to-Python transpilers
  • Produced syntactically-valid but semantically-broken code
  • Required extensive manual correction
  • Could not handle MATLAB idioms, cell arrays, struct arrays

Now: AI Coding Agents

  • Claude Code reads MATLAB and writes Python
  • Handles idioms: 1-based indexing, column-major layout, cell arrays
  • Test frameworks detect agent errors and constrain development
  • Testing framework allows continued conversion

Three-Repository Strategy

RepoPurpose
xpfreshUnchanged reference — the "truth"
xprefactorRefactored into small functions for conversion
xpinstrumentInstrumented to generate test cases

All three verified to produce identical results

The Port: rosella

176
Python modules
52k
lines of source
76k
lines of tests
10
C++ extensions
763
commits

C++ Extensions (fastcluster)

Clustering

  • fastlabel — connected components
  • fastclusterprop — region properties
  • fastsupercluster — O(n log n) sweep-line

Analysis

  • fastcoincidence2 — temporal coincidence
  • clustertopixel — label → mask
  • statisticSumLabelledMap

Same C++ code as MATLAB MEX, compiled via nanobind for Python

Three Layers of Testing

Instrument Tests
pre-recorded MATLAB
→
Engine Tests
live MATLAB comparison
→
Integration Tests
full pipeline

 

Every output, every field, every return value compared for both shape and value

Instrument Tests

MATLAB function
runs on real data
→
Inputs + outputs
saved to MAT/JSON
→
Python function
called with same inputs
→
Compare
rtol=10-10

 

  • 441 test functions across 152 files
  • No MATLAB runtime needed — uses pre-recorded cases
  • Captures real edge cases from production MATLAB runs

Engine Tests

Test inputs
↓
MATLAB Engine
matlab.engine API
↘
↓
Python function
↙
Compare

 

  • 286 test functions across 166 files
  • Both run side-by-side with identical inputs
  • Tests current MATLAB behaviour, not pre-recorded

Integration Tests

Shared noise
Python generates
→
xdetection
140 jobs
→
merge
→
veto test
21 combos
→
tune
→
closedbox

 

  • Full GRB pipeline: 6 stages, 178+ MAT files compared
  • Shared noise ensures identical inputs to both pipelines
  • Field-by-field comparison of every output file

Integration Results

CheckpointStatus
RNG / job split✓ Identical
Injection masks✓ Identical
Loudest background✓ ~10-7
On-source selection✓ same job
Detection threshold✓ ~10-9
Injection pass flags✓ Identical
Efficiency (8 scales)✓ Identical
UL 50%/90%/95%✓ Identical
Vela run7 efficiency curve

Vela run7 efficiency curve (timtam)

GRB Pipeline Architecture

Frame data
H1, L1, V1
→
Conditioning
whiten, filter
→
TF maps
spectrogram
→
Likelihood
coherent network
→
Clustering
fastcluster
→
Triggers
vetoes, UL

 

All stages ported and validated: xdetection → xmerge → xmakegrbwebpage → xtunegrbwebpage

Results Viewer: timtam

  • Modern web interface for GRB analysis results
  • Reads MAT file output directly — no MATLAB needed
  • 25+ publication-quality plots
  • Light/dark mode, collapsible sections
  • Comment system for collaborative review

timtam Features

Plots

  • Antenna patterns & globe projections
  • ASD & noise spectra
  • Trigger scatter plots & histograms
  • Efficiency curves per waveform
  • Rate vs significance

Analysis

  • GRB parameters table
  • Loudest events with FAP
  • Veto performance comparison
  • Detection statistics
  • Injection recovery summary

timtam: Live Demo Pages

  • GRB mini closedbox — integration test output rendered by timtam
  • Vela run7 closedbox — MATLAB O4b production run rendered by timtam

 

Both pages generated from the same MAT file format — timtam reads Python or MATLAB output identically

Example: GRB Sensitivity

sample_grb_notebook.ipynb — Python port of sample_grb_script.m

from rosella.xmakeskygrid import xmakeskygrid
from rosella.antennaPatterns import antennaPatterns
from rosella.gwbenergy import gwbenergy
from rosella.SRD import SRD

# Build sky grid, compute antenna response, estimate SNR
ra_search, dec_search, prob, area, cov = xmakeskygrid(
    '231.7', '-34.1', '1454920000', '5.0', '2', 'H1~L1~V1', '5e-4')

Fp, Fc, _ = antennaPatterns(network, sky_ctr)
hrss = hrss_nominal * (Egw / Egw_nominal)**0.5
SNR = np.sqrt(Fp**2 + Fc**2) * hrss / np.sqrt(S)

All values verified identical to MATLAB to 15 significant figures

Existing Notebooks

  • sample_grb_notebook.ipynb — GRB sensitivity analysis (live demo)
  • demo_xdetection.ipynb — full detection pipeline
  • demo_signal_processing.ipynb — filtering & resampling
  • demo_xconditionSmall2.ipynb — data conditioning
  • demo_xtimefrequencymapSmall3.ipynb — time-frequency maps
  • demo_xdetection_processInjection.ipynb — injections

Deployment Plans

  • X-Pypeline — lightweight release package from rosella
  • PyPI / pip install — standard Python packaging
  • Open Science Grid — distributed computing
  • LIGO clusters — HTCondor integration
  • Containers — reproducible environments

Status

ComponentStatus
Core detection (xdetection)✓ Complete
C++ extensions (fastcluster)✓ Complete
GRB post-processing pipeline✓ Complete
Integration tests passing✓ Complete
Results viewer (timtam)✓ Complete
Example notebooks● In progress
X-Pypeline release package● Upcoming
OSG / LIGO deployment● Upcoming

Next Steps

  • Create X-Pypeline release package
  • Complete sample notebooks for users
  • Deploy on OSG and LIGO clusters
  • All-sky and supernova search ports
  • GPU acceleration for clustering

Questions?

 

rosella: 176 modules, 52k lines source + 76k lines tests, 763 commits
fastcluster: 10 C++ extensions via nanobind
Testing: 727 test functions across 3 layers
Integration: field-by-field match