Semi-Synthetic Y(0) Bakeoff

All estimators target the missing untreated potential outcome

Executive Read

This branch reframes the TROP semi-synthetic comparison as a pure missing-Y(0) prediction problem. Treated post-period cells are held out from each model fit. Every estimator returns an estimate of

\[ \mu_0^\mathcal{M} = \frac{1}{|\mathcal{M}|}\sum_{(i,t)\in\mathcal{M}} Y_{it}(0), \]

where \(\mathcal{M}\) is the set of treated post-period cells. Accuracy is measured against the known semi-synthetic untreated mean on those held-out cells:

\[ \widehat e = \widehat \mu_0^\mathcal{M} - \mu_0^\mathcal{M}. \]

This is deliberately not a scalar-treatment-effect bakeoff. Treatment effects can still be recovered afterward by subtracting \(\widehat\mu_0^\mathcal{M}\) from the observed treated-post mean, but no method uses a treatment-effect regressor while fitting the missing surface.

Run metadata:

  • rows: 420 design-replications
  • designs: 21
  • reps requested: 20
  • MTGP steps: 500
  • fixed MTGP rank: 4
  • ordered-shrinkage maximum rank: 5
  • ordered-amplitude prior: {'additive_fixed_effects': True, 'gap_shape': 10.0, 'gap_rate': 4.0, 'prior_weight': 200.0}
  • Laplacian graph kernel: matern
  • GP-whitened nuclear row: {'lambda_scale': 1.0, 'unit_kernel': 'pre-treatment-path graph Laplacian', 'time_kernel': 'RBF lengthscale 0.9', 'penalty': 'nuclear norm of K_u^{-1/2} F K_t^{-1/2}', 'post_fe_constraints': 'zero row and column sums on F'}
  • output directory: docs/figures/y0_semisynthetic/

Key Takeaways

  • The option-3 ordered-amplitude row is now included. OA_FE_MTGPAX_Y0 has mean RMSE 0.2890, versus 0.2669 for the earlier gamma/precision ordered-shrinkage row and 0.3257 for fixed-rank FE MTGPAX. The best overall row is SDID Y(0) at 0.1317.
  • The rank diagnostic moved in the intended direction. Gamma/precision shrinkage has mean effective rank 3.05 with mean scales 0.95, 0.43, 0.19, 0.08, 0.03; ordered-amplitude shrinkage has mean effective rank 1.94 with mean scales 1.24, 0.16, 0.02, 0.00, 0.00.
  • The Laplacian-prior variants are feasible but not better in this run. Ordered-amplitude Laplacian FE MTGPAX has mean RMSE 0.2894 and effective rank 1.96; gamma/precision Laplacian FE MTGPAX has mean RMSE 0.2818.
  • The option-5 GP-whitened nuclear row is much faster in this implementation because it uses a SoftImpute-style proximal approximation rather than an exact CVX solve. The TROP-weighted version has mean RMSE 0.1423, close to TROP_Y0 at 0.1397; the unweighted version is weaker at 0.2235.
  • GP-whitened singular-value shrinkage is not yet producing post-FE rank one or two. The unweighted nuclear row has mean effective rank 18.07 and the TROP-weighted row has 12.59.
  • The fixed-rank MTGPAX baselines remain in the table for continuity. MTGPAX_Y0 has mean RMSE 0.3130 and FE_MTGPAX_Y0 has mean RMSE 0.3257, versus 0.1397 for TROP_Y0.
  • The main MTGPAX problem is variance, not average bias. MTGPAX_Y0 has mean absolute bias 0.0603, lower than TROP_Y0’s 0.0695, but its mean error variance is 0.2438 versus 0.0214 for TROP_Y0.
  • The Laplacian row is a kernel-regularized factor model, not a full-panel exact GP. That keeps the Y0 simulation feasible on the larger CPS/PWT panels while still making unit similarity part of the MTGPAX prior.

Design-level RMSE winners:

method design wins
SDID Y(0) 6
TROP-weighted GP-whitened nuclear Y(0) 5
SC Y(0) 3
TROP Y(0) 3
Ordered-amplitude FE MTGPAX Y(0) 1
MC Y(0) 1
DIFP Y(0) 1
FE MTGPAX Y(0) 1

Estimator Definitions

Let \(\mathcal{O}\) be all observed untreated cells after masking treated post-period observations. All estimators below are fit only to \(\mathcal{O}\) and then evaluated on \(\mathcal{M}\).

TROP Y(0)

TROP_Y0 is the TROP-style weighted matrix-completion imputer without a treatment-effect term. It solves a weighted two-way fixed-effect plus nuclear-norm problem:

\[ (\widehat\mu,\widehat\alpha,\widehat\gamma,\widehat L) = \arg\min_{\mu,\alpha,\gamma,L} \sum_{(i,t)\in\mathcal{O}} \left[ \delta_{it} \{Y_{it}-\mu-\alpha_i-\gamma_t-L_{it}\} \right]^2 + \lambda_{\mathrm{nn}}\|L\|_*. \]

The prediction is the average fitted value over missing cells:

\[ \widehat\mu_{0,\mathrm{TROP}}^\mathcal{M} = \frac{1}{|\mathcal{M}|}\sum_{(i,t)\in\mathcal{M}} (\widehat\mu+\widehat\alpha_i+\widehat\gamma_t+\widehat L_{it}). \]

The localization weights \(\delta_{it}\) use the same TROP unit and time distance parameters selected for the corresponding semi-synthetic design, but the model part contains no \(\tau W_{it}\) term.

MC Y(0)

MC_Y0 uses the same two-way fixed-effect plus nuclear-norm imputation problem as TROP_Y0, but sets all observed-cell weights to one. It is therefore a global matrix-completion baseline for the untreated surface:

\[ \min_{\mu,\alpha,\gamma,L} \sum_{(i,t)\in\mathcal{O}} \{Y_{it}-\mu-\alpha_i-\gamma_t-L_{it}\}^2 + \lambda_{\mathrm{nn}}\|L\|_*. \]

DID Y(0)

DID_Y0 is an unweighted two-way fixed-effect imputer. It fits

\[ Y_{it}=\mu+\alpha_i+\gamma_t+\varepsilon_{it} \qquad (i,t)\in\mathcal{O}, \]

and averages \(\widehat\mu+\widehat\alpha_i+\widehat\gamma_t\) over \(\mathcal{M}\).

SDID Y(0)

SDID_Y0 uses a two-way fixed-effect imputer with product weights derived from synthetic-difference-in-differences style unit and time balancing. The fitted surface is still

\[ \widehat Y_{it}(0)=\widehat\mu+\widehat\alpha_i+\widehat\gamma_t, \]

but the observed-cell loss is weighted by the SDID-style product weights.

SC Y(0)

SC_Y0 constructs convex weights on donor units using only pre-treatment outcomes. It matches the treated-group pre-period mean with

\[ \min_{\omega\ge 0,\ \mathbf{1}'\omega=1} \left\| \bar Y_{\mathrm{treated},pre} - Y_{\mathrm{controls},pre}'\omega \right\|_2^2, \]

then predicts the missing untreated mean by averaging the donor-weighted post-period controls.

DIFP Y(0)

DIFP_Y0 is the same donor-weighted imputer as SC_Y0, but includes an intercept in the pre-period fit. The intercept is carried into the treated-post counterfactual prediction.

MTGPAX Y(0)

MTGPAX_Y0 fits the MTGPAX low-rank multi-task GP to observed untreated cells only:

\[ m_{it} = \nu_i + \sum_{r=1}^R \beta_{ir}u_{rt}, \qquad u_r\sim\mathcal{GP}(0,K_t). \]

It uses scalar_effect=False, so there is no treatment-effect parameter in the likelihood. The reported estimate is the average counterfactual mean over treated post-period cells.

FE MTGPAX Y(0)

FE_MTGPAX_Y0 is the same missing-Y(0) MTGPAX imputer with centered unpenalized additive unit and time fixed effects:

\[ Y_{it}=a_i+b_t+m_{it}+\varepsilon_{it}, \qquad (i,t)\in\mathcal{O}. \]

Again, scalar_effect=False; the fixed effects and GP surface impute the untreated mean directly.

Ordered-Shrinkage FE MTGPAX Y(0)

OS_FE_MTGPAX_Y0 keeps the same additive fixed-effect interpretation but treats the factor dimension as overcomplete. The simulation uses maximum rank R_{\max}=5 and multiplicative ordered shrinkage:

\[ Y_{it}=a_i+b_t+\sum_{r=1}^{R_{\max}}\tilde\beta_{ir}u_{rt}+\varepsilon_{it}, \qquad \tilde\beta_{ir}=\beta_{ir}/\sqrt{\prod_{s\le r}\delta_s}. \]

The prior uses stronger shrinkage after the first column, pushing the effective rank toward one or two unless the observed untreated cells support additional columns. The raw output records the posterior MAP effective-rank diagnostic, factor scales, and column signals for this row.

Ordered-Amplitude FE MTGPAX Y(0)

OA_FE_MTGPAX_Y0 is the option-3 parameterization. It keeps the same additive-FE factor model but introduces explicit ordered factor amplitudes:

\[ Y_{it}=a_i+b_t+\sum_{r=1}^{R_{\max}}s_r\beta_{ir}u_{rt}+\varepsilon_{it}, \qquad \log s_r = \log s_{r-1} - g_r,\quad g_r>0. \]

The simulation uses a Gamma-style MAP prior on the positive gaps. The shape, rate, and prior weight are recorded in y0_meta.json and printed in the run metadata above. This is more direct than the gamma/precision row because later factors must have smaller amplitudes unless the likelihood pays to keep them alive.

Ordered-Shrinkage Laplacian FE MTGPAX Y(0)

OS_FE_LAPLACIAN_MTGPAX_Y0 uses the same ordered-shrinkage, additive-FE factor model as OS_FE_MTGPAX_Y0, but replaces the iid prior over loading columns with a graph-Laplacian unit-kernel prior:

\[ \beta_{\cdot r}\sim N(0,K_{\mathrm{unit,Lap}}), \qquad K_{\mathrm{unit,Lap}} = (\kappa^2 I + L)^{-1} \]

for the default Matérn/resolvent graph kernel. The unit graph is built from pre-treatment outcome-path similarity. This keeps the estimator in the scalable MAP factor-model family while making similar units share loading information.

Ordered-Amplitude Laplacian FE MTGPAX Y(0)

OA_FE_LAPLACIAN_MTGPAX_Y0 combines the option-3 ordered amplitudes with the same graph-Laplacian loading prior used in OS_FE_LAPLACIAN_MTGPAX_Y0.

GP-Whitened Nuclear Y(0)

GP_NUCLEAR_Y0 is the option-5 row. It keeps additive fixed effects, but puts the low-rank shrinkage directly on the residual untreated surface:

\[ Y_{it}(0)=a_i+b_t+F_{it}+\varepsilon_{it}, \qquad \|K_u^{-1/2} F K_t^{-1/2}\|_*. \]

Here \(K_u\) is the pre-treatment-path graph-Laplacian unit kernel and \(K_t\) is an RBF time kernel. The residual \(F\) is double-centered so the nuclear component is interpreted after additive unit and time effects. This is the closest row to a kernelized nuclear-norm/TROP bridge: if both kernels are identity matrices, the penalty reduces to an ordinary post-FE nuclear norm.

The exact convex version was too slow for the all-design simulation. The row in this report uses a fast SoftImpute-style approximation: fit weighted additive fixed effects, fill observed residuals, whiten by \(K_u^{-1/2}\) and \(K_t^{-1/2}\), soft-threshold singular values, unwhiten, re-center, and iterate. The reported singular values and effective rank are diagnostics from the final GP-whitened residual.

TROP-Weighted GP-Whitened Nuclear Y(0)

TROP_GP_NUCLEAR_Y0 uses the same GP-whitened nuclear residual model as GP_NUCLEAR_Y0, but the additive-FE fit and observed-residual fill step use the TROP localization weights. This keeps the singular-value shrinkage in the GP-whitened residual space while borrowing TROP’s unit/time localization.

Overall Results

method mean RMSE median RMSE mean bias
TROP Y(0) 0.1397 0.0917 0.0695 0.0214 1.0000
SDID Y(0) 0.1317 0.0843 0.0598 0.0230 1.0000
SC Y(0) 0.1559 0.0714 0.0831 0.0247 1.0000
DID Y(0) 0.2241 0.1603 0.1233 0.0458 1.0000
MC Y(0) 0.1516 0.0839 0.0742 0.0249 1.0000
DIFP Y(0) 0.1508 0.0726 0.0657 0.0335 1.0000
MTGPAX Y(0) 0.3130 0.1415 0.0603 0.2438 1.0000
FE MTGPAX Y(0) 0.3257 0.1661 0.0749 0.2366 1.0000
Ordered-shrinkage FE MTGPAX Y(0) 0.2669 0.1525 0.0714 0.1475 0.9929
Ordered-shrinkage Laplacian FE MTGPAX Y(0) 0.2818 0.1295 0.0657 0.1764 0.9952
Ordered-amplitude FE MTGPAX Y(0) 0.2890 0.1014 0.1082 0.2474 1.0000
Ordered-amplitude Laplacian FE MTGPAX Y(0) 0.2894 0.1196 0.0981 0.2473 1.0000
GP-whitened nuclear Y(0) 0.2235 0.1603 0.1232 0.0457 1.0000
TROP-weighted GP-whitened nuclear Y(0) 0.1423 0.0916 0.0727 0.0209 1.0000

Design-Level RMSE

dataset key TROP Y(0) SDID Y(0) SC Y(0) DID Y(0) MC Y(0) DIFP Y(0) MTGPAX Y(0) FE MTGPAX Y(0) Ordered-shrinkage FE MTGPAX Y(0) Ordered-shrinkage Laplacian FE MTGPAX Y(0) Ordered-amplitude FE MTGPAX Y(0) Ordered-amplitude Laplacian FE MTGPAX Y(0) GP-whitened nuclear Y(0) TROP-weighted GP-whitened nuclear Y(0)
Basque basque_random 0.0917 0.0767 0.1465 0.1872 0.1362 0.0831 0.1175 0.1195 0.0755 0.1096 0.0688 0.1196 0.1872 0.0916
Basque basque_simulated 0.1056 0.0843 0.1136 0.2061 0.1157 0.1075 0.1588 0.1693 0.1929 0.1295 0.2160 0.1889 0.2061 0.0988
Basque basque_treated_unit 0.1431 0.1442 0.0453 0.1293 0.0431 0.0611 0.2074 0.3186 0.2551 0.2608 0.2944 0.2557 0.1293 0.1497
Boatlift boatlift_random 0.1426 0.1226 0.1862 0.1297 0.1378 0.2085 0.3963 0.3881 0.3696 0.4473 0.4297 0.4868 0.1302 0.1573
Boatlift boatlift_simulated 0.3673 0.4697 0.5409 0.4384 0.4170 0.5771 1.3194 1.0637 1.0208 1.1746 2.0930 2.0230 0.4361 0.3584
Boatlift boatlift_treated_unit 0.3037 0.3088 0.1829 0.2326 0.2659 0.3854 0.9536 1.0657 0.8292 0.8008 0.6768 0.5816 0.2296 0.3855
CPS cps_hours_minwage 0.1126 0.1130 0.1942 0.2365 0.1605 0.1425 0.2836 0.2682 0.3318 0.2054 0.2336 0.2340 0.2358 0.1255
CPS cps_logwage_abortion 0.0241 0.0301 0.0361 0.0466 0.0350 0.0370 0.0527 0.1264 0.0335 0.0442 0.0285 0.0286 0.0477 0.0238
CPS cps_logwage_gunlaw 0.0290 0.0252 0.0260 0.0440 0.0275 0.0249 0.0939 0.0679 0.0456 0.0439 0.0311 0.0294 0.0437 0.0298
CPS cps_logwage_minwage 0.0279 0.0232 0.0444 0.0471 0.0263 0.0286 0.0860 0.1045 0.0502 0.0588 0.0351 0.0373 0.0444 0.0286
CPS cps_logwage_random 0.0216 0.0214 0.0226 0.0558 0.0340 0.0351 0.0496 0.0772 0.0478 0.0490 0.0250 0.0253 0.0526 0.0211
CPS cps_urate_minwage 0.2478 0.1873 0.2200 0.3935 0.2349 0.2103 0.2235 0.1687 0.1923 0.2456 0.1739 0.1725 0.3928 0.2412
Germany germany_random 0.0301 0.0354 0.0714 0.0696 0.0432 0.0375 0.0561 0.0588 0.0553 0.0516 0.0699 0.0672 0.0696 0.0311
Germany germany_simulated 0.0760 0.0858 0.0589 0.1492 0.0839 0.0726 0.1076 0.1061 0.0982 0.0914 0.0809 0.1011 0.1492 0.0702
Germany germany_treated_unit 0.0862 0.0792 0.0413 0.1038 0.0787 0.0725 0.1415 0.1661 0.1525 0.1528 0.1014 0.0996 0.1038 0.0915
PWT pwt_loggdp_democracy 0.0326 0.0377 0.0351 0.2283 0.0569 0.0349 0.0595 0.0734 0.0719 0.0578 0.0622 0.0581 0.2278 0.0299
PWT pwt_loggdp_education 0.0329 0.0547 0.0688 0.1717 0.0491 0.0469 0.0813 0.0977 0.0772 0.1079 0.0922 0.0912 0.1708 0.0340
PWT pwt_loggdp_random 0.0523 0.0451 0.0521 0.1137 0.0607 0.0500 0.0722 0.0771 0.0669 0.0717 0.0637 0.0682 0.1137 0.0519
Smoking smoking_random 0.0919 0.1052 0.1332 0.1603 0.1082 0.1345 0.2173 0.2807 0.3250 0.2692 0.1800 0.1643 0.1603 0.0916
Smoking smoking_simulated 0.3596 0.2904 0.3729 0.6909 0.4135 0.3255 1.1093 1.1441 0.6020 0.7432 0.6480 0.6371 0.6914 0.3417
Smoking smoking_treated_unit 0.5559 0.4256 0.6808 0.8726 0.6556 0.4914 0.7857 0.8987 0.7115 0.8036 0.4648 0.6072 0.8721 0.5357

Output Files

The generated outputs for this branch are:

  • docs/figures/y0_semisynthetic/y0_raw.csv: one row per design-replication.
  • docs/figures/y0_semisynthetic/y0_summary.csv: wide design-level summary.
  • docs/figures/y0_semisynthetic/y0_metrics_long.csv: long design × method metrics.
  • docs/figures/y0_semisynthetic/y0_meta.json: run metadata and method list.
  • docs/y0_semisynthetic.html: rendered self-contained memo.