Switch survey calibration from l0-python to SparseCalibrator (microplex)

[MaxGhenis]

Summary

Frontier comparison notebook shows that microplex.calibration.SparseCalibrator (L1 penalty, FISTA solver) dominates l0-python's SparseCalibrationWeights (L0 penalty, Hard Concrete gates) on every metric:

Property	SparseCalibrator (L1)	HardConcrete / l0-python (L0)
Solver	FISTA (convex)	Adam SGD (non-convex)
Deterministic	Yes	No (seed-dependent)
Dominates frontier	Yes	No
Runtime (5K records)	~0.02s	~1.8s
Runtime (200K records)	~0.5s	~6s
Loss function	Relative MSE (via normalization)	Relative MSE

Key findings

1. SparseCalibrator dominates the entire accuracy-sparsity frontier — lower or equal out-of-sample error at every sparsity level
2. HardConcrete variance explodes at high sparsity — ±10-14% SE below 100 records, vs zero variance for SparseCalibrator (convex = deterministic)
3. 12-90x faster depending on configuration and dataset size
4. Already supports relative loss — normalize_targets=True (default) divides each constraint row by |b_i|, equivalent to loss_type="relative" in l0-python

What this means for us-data

policyengine-us-data currently uses l0-python's SparseCalibrationWeights in fit_national_weights.py with ~200K households and ~300-500 targets. Switching to SparseCalibrator would:

• Eliminate seed-dependent calibration results (reproducibility)
• Speed up calibration (~0.5s vs ~6s at 200K records for a single fit)
• Improve out-of-sample target accuracy at every sparsity level
• Remove the PyTorch dependency for calibration (SparseCalibrator uses only NumPy/SciPy)

Migration path

The API is similar. Current l0-python usage:

from l0.calibration import SparseCalibrationWeights
model = SparseCalibrationWeights(n_features=n_households, ...)
model.fit(M=M, y=targets, lambda_l0=1e-6, loss_type="relative", ...)
weights = model.get_weights()

Proposed:

from microplex.calibration import SparseCalibrator
cal = SparseCalibrator(sparsity_weight=0.01)  # tune for desired sparsity
cal.fit(data, marginal_targets, continuous_targets)
weights = cal.weights_

Note: SparseCalibrator currently takes (data, marginal_targets, continuous_targets) rather than a raw (M, y) matrix. We may want to add a lower-level fit_matrix(M, y) method or adapt the us-data code to use the higher-level API. This is tracked in microplex — when it supersedes us-data's calibration pipeline, SparseCalibrator will be the default method.

Notebook

Full reproducible comparison: https://gist.github.com/MaxGhenis/9f9a31a156eba8a8cbf041710ce31213

[baogorek]

l0-python laywered up and got me, so here I go:

• Lower Relative MSE loss alone is not necessarily better
• Solutions with many very small weights <.01 or many very large weights >1000 are undesirable solutions
• Solutions that zero out too many households are undesireable solutions
• By giving us L0, L2, and early stopping knobs, we can shape very different solutions, and we have practice with this
• I have not seen evidence that our algorithm for getting good weights has ever been the bottleneck. In the case of medicaid, for example, we had a bug in the targets logic, and then we found ways to improve the data (via simulation)

So in closing, your honor, l0-python is working pretty well, we understand how to shape solutions, and it may not be the best time to switch horses.