study_file = "analyses/full-archive/configs/S20-P001.json"
config = {}
comparison_file = "templates/expensive/configs/credit-markets.json"
comparison = {}
reuse_results = True
public_results_url = ""Panel-estimator comparison
import json
import sys
from pathlib import Path
ROOT = next(
p
for p in [Path.cwd(), *Path.cwd().parents]
if (p / "templates/expensive/runner.py").exists()
)
sys.path.insert(0, str(ROOT / "templates/expensive"))
from runner import Comparison, markdown_table # noqa: E402
settings = json.loads((ROOT / study_file).read_text()) | config
options = json.loads((ROOT / comparison_file).read_text()) | comparison
run = Comparison(settings, options, ROOT)Estimation source
Comparison is a project-specific orchestration class, defined in runner.py, not a black-box package estimator. prepare() constructs the analysis sample; estimate() invokes the adapters below or validates and loads their saved fits. The source companion exposes the complete implementations, with line links and file hashes; the downloadable code bundle includes the same files.
| Fits | Estimation implementation |
|---|---|
| Static/dynamic TWFE, BJS, Sun–Abraham | replication_engine.py: PyFixest feols, did2s, and saturated event-study regression; sparse implementation for large panels |
| CS-DID | csdid_model.py: ATTgt; csdid_aggregations.py: aggte, aggregation weights and influence-function checks |
| SDID, matrix completion, horizontal ridge | matrix_models.py: the corresponding crabbymetrics classes, including the SDID event-path correction |
| Both multisynth modes | multisynth_model.R: augsynth::multisynth, donor masks, cohort aggregation and pooling diagnostics |
The supplied Credit example loads the four validated S20-P001 regression fits, rather than estimating them anew on each render. Other saved fits are reused only after matching data, tuning, adapter hashes and recorded Python versions. To refit all models, remove regression_results from the comparison configuration and set reuse_results=False; the latter alone does not override the supplied regression file. Estimation and reporting source are kept separate so a figure-style change does not trigger new fits.
sys.path.insert(0, str(ROOT / "templates/reporting"))
from comparison_report import ( # noqa: E402
attach_intervals,
export_report,
forest_plots,
)Data and analysis sample
run.base.describe()
run.base.df.head()Credit Markets Property Rights and the Commons — Noack and Costello (2024), JPE. https://doi.org/10.1086/729065
CMPRC-CSA-NOTYET
Archive specification: CMPRC-CSA-NOTYET. Input: 2.111 MB.
| Role | Column | Definition |
|---|---|---|
| Y | y | |
| W | d | ; values [0, 1]; cardinality 2 |
| i | unit | unit |
| t | time | time |
| X_it | None | Plus unit and period fixed effects |
| Cluster | cluster | cluster |
shape: (5, 9)
| obs_id | unit | time | assignment | cluster | y | d | weight | cohort |
|---|---|---|---|---|---|---|---|---|
| i64 | i64 | i64 | i64 | i64 | f64 | i64 | f64 | i64 |
| 1327 | 1686 | 1960 | 1686 | 5 | 1.137982 | 0 | 1.0 | 1967 |
| 1328 | 1686 | 1961 | 1686 | 5 | 1.137982 | 0 | 1.0 | 1967 |
| 1329 | 1686 | 1962 | 1686 | 5 | 1.719052 | 0 | 1.0 | 1967 |
| 1330 | 1686 | 1963 | 1686 | 5 | 1.078451 | 0 | 1.0 | 1967 |
| 1331 | 1686 | 1964 | 1686 | 5 | 3.163037 | 0 | 1.0 | 1967 |
Every model uses Y_{it}/s_Y, with the unweighted source-sample standard deviation before exclusions (ddof=1). Outcomes are not centered. Adoption is the first period with treatment equal to one; relative time is period minus adoption. The regression engine removes initially treated units, units without period −1, and unsupported fixed-effect observations, recording every exclusion below.
sample = run.prepare()
markdown_table(sample.T.reset_index().to_dict("records"))| index | 0 |
|---|---|
| outcome | y |
| treatment | d |
| source_rows | 39432 |
| source_sd | 1.329171187176219 |
| rows | 38319 |
| units | 723 |
| clusters | 47 |
| treated_units | 529 |
| never_treated_units | 194 |
| treated_cohorts | 15 |
| latest_cohort_control_index | |
| unique_unit_period_cells | 38319 |
| unsupported_prediction_rows_removed | 0 |
| rows_without_never_control_time_fe_removed | 0 |
| initially_treated_units_removed | 21 |
| post_adoption_untreated_rows_removed | 0 |
| units_without_minus_one_removed | 0 |
| additional_rows_removed | 0 |
run.base.panelview()
y__d: 100 of 723 units shown (seed 20261008). All eligible units enter estimation. Purple: untreated; yellow: treated; white: unobserved.
The counterfactual adapters require a balanced unit-period panel, absorbing treatment, never-treated donors, ordinary unit/time fixed effects, no additional covariates and uniform observation weights. Unsupported adapters remain in the table with a reason. They never silently trim the sample or omit configured controls. Regression models retain the full configured specification.
Estimators and targets
| Estimator | Construction | Scalar summary |
|---|---|---|
| Static TWFE | Treatment plus unit/time effects and controls; already-treated comparisons can enter | Regression coefficient, not a common-weight ATT |
| Dynamic TWFE | Event-time indicators; period −1 omitted | Configured post-event weighting |
| BJS / did2s | All untreated observations, including pre-adoption histories; imputation residuals | Configured post-event weighting |
| Sun–Abraham | Cohort-by-event-time saturated regression with a never-treated reference | Configured post-event weighting |
| CS-DID (two summaries) | One group-time DID fit; not-yet-treated controls including never-treated units | Dynamic: equal nonnegative event times; simple: equal treated cells |
| SDID | Separate adoption-cohort fits with never-treated donors and cohort-specific unit/time weights | All treated cells, subtracting the time-weighted pre-gap |
| Matrix completion | One joint untreated-cell low-rank fit with unit/time effects | All treated cells |
| Horizontal ridge | Separate cohort fits using pre-treatment histories and never-treated donors | All treated cells |
| Multisynth (cohort-averaged) | Partially pooled adoption-cohort fits; time_cohort=TRUE |
All treated cells |
| Multisynth (individual) | Partially pooled individual-unit fits; time_cohort=FALSE |
All treated cells |
The table reports the actual aggregation for each row. Equal event-time and equal treated-cell averages are different estimands. By default the three regression event studies use treated-cell weights, and CS-DID now reports both dynamic and simple aggregation. Simple matches the all-post treated-cell target on an eligible balanced panel; group aggregation instead weights cohort post-period means by cohort size, without multiplying by post-period count. Thus group is not the matching option when cohorts have different post-treatment follow-up lengths. Matching scalar weights does not equate identifying assumptions; dynamic TWFE need not recover a clean ATT under heterogeneous effects. Horizon and weighting overrides for the regression engine do not change the counterfactual adapters’ all-post target. Event-time averages may change cohort composition with exposure length.
results_file = run.estimate(reuse=reuse_results)
for spec in run.base.results["specifications"].values():
for result in spec["estimates"]:
reason = spec.get("method_status", {}).get(result["method"])
if reason:
for field in ["estimate", "se", "pvalue", "lower", "upper"]:
result[field] = None
result.update(status="unvalidated", reason=reason)
spec["event_studies"].pop(result["method"], None)attach_intervals(run)Point estimates
This is the single combined results table. A missing SE means that no uncertainty estimate was computed for that fit; it is not zero.
rows = run.rows()
for row in rows:
row["SE"] = "—" if row["SE"] is None else f"{row['SE']:.4f}"
if row["Aggregation"] == "observations":
row["Aggregation"] = "Treated cells"
if all(row["Status"] == "estimated" for row in rows):
for row in rows:
row.pop("Status")
row.pop("Note")
if len({row["Outcome / treatment"] for row in rows}) == 1:
for row in rows:
row.pop("Outcome / treatment")
markdown_table(rows)| Estimator | Estimate (SDs) | SE | Aggregation | Event study |
|---|---|---|---|---|
| Static TWFE | -0.45497 | 0.0694 | Coefficient | No |
| Dynamic TWFE | -0.50982 | 0.0852 | Treated cells | Yes |
| BJS / did2s | -0.52248 | 0.0856 | Treated cells | Yes |
| Sun–Abraham | -0.51382 | 0.0641 | Treated cells | Yes |
| CS-DID | -0.39297 | 0.0933 | Equal post-event times | Yes |
| CS-DID (treated cells) | -0.49006 | 0.0781 | Treated cells | Same CS-DID path |
| SDID | -0.50282 | 0.0777 | Treated cells | Yes |
| Matrix completion | -0.51087 | 0.0877 | Treated cells | Yes |
| Horizontal ridge | 0.032141 | 0.0812 | Treated cells | Yes |
| Multisynth (cohort-averaged) | -0.47334 | 0.1155 | Treated cells | Yes |
| Multisynth (individual) | -0.35737 | 0.1477 | Treated cells | Yes |
Forest plot
Points reproduce the combined table; horizontal ranges are available 95% confidence intervals, not a common inference procedure imposed on all estimators. Hollow points explicitly mark missing intervals; unsupported fits remain labeled without invented point estimates. Aggregation labels distinguish regression coefficients, equal event-time averages and treated-cell averages. Each outcome–treatment pair gets its own figure.
forest_files = forest_plots(run)
The Credit example has intervals for all eleven scalar summaries: four regression cluster-robust intervals, two CS-DID influence-function intervals including cohort-share uncertainty, three percentile intervals from 999 full country-cluster bootstrap refits, and two multisynth intervals from 9,999 country-clustered Mammen wild-bootstrap draws. The full-refit matrix procedure resamples whole countries, refits weights and data-dependent tuning, and recomputes cohort shares, holding the source outcome-SD denominator fixed. The multisynth procedure instead reuses fitted weights and tuning with common within-country multipliers; every draw uses treated-cell aggregation, with no jackknife. Its basic bootstrap intervals condition on the fitted weights and observed design and are not full-refit intervals. All inference methods and cluster counts are recorded in JSON.
The optional inference_files configuration imports saved bootstrap inference only when its prepared-panel fingerprint, point-fit settings, package version and fitted event path match this run. Changed data or tuning require new inference, not recycling Credit’s intervals. Omit that option for a new study; native verified bounds still appear automatically. The bootstrap source and forest-plot source are public. For matrix completion, bootstrap fits allow 5,000 iterations rather than the point fit’s 1,000, with the same objective and convergence tolerance; all 999 draws converged without exclusions.
Credit forest downloads: PNG · PDF. Each run also saves its figures alongside the combined JSON.
CS-DID aggregation diagnostics
The two main CS-DID rows share one fitted event-study path. The group summary below is an additional diagnostic, not the matched treated-cell target. All three intervals include estimated cohort-share uncertainty and use the configured clusters.
| Aggregation | Estimate (SDs) | SE | 95% CI |
|---|---|---|---|
| Equal post-event times | -0.39297 | 0.093306 | [-0.5759, -0.2101] |
| Equal treated unit-years | -0.49006 | 0.078134 | [-0.6432, -0.3369] |
| Cohort-size-weighted post-period means | -0.49445 | 0.077859 | [-0.6470, -0.3418] |
Event studies
Each supported estimator gets its own full-width figure on its native effect scale. No additional period −1 recentering is applied, so post-period paths reproduce their reported scalar under the stated weights. Regression and CS-DID ribbons are pointwise 95% intervals using the configured clusters. Multisynth paths have pointwise fixed-weight bootstrap bands when computed; the other counterfactual paths have no event-study intervals. Their fitted pre-treatment gaps are fit diagnostics, not held-out pretrend tests. Static TWFE and scalar-only estimators have no event-study curve.
run.event_plots()
Pooling and tuning
multisynth reports its fitted shared ν separately for the individual and cohort specifications; λ is a regularization parameter, not ν. Both retain every observed post-treatment period and use never-treated donors. SDID and horizontal ridge fit cohorts separately, without cross-cohort ν. Matrix completion shares a low-rank outcome model. Model defaults and any overrides are retained in the JSON, together with per-cohort diagnostics when available.
run.tuning_table()| Specification | Estimator | Settings | Pooling |
|---|---|---|---|
| y__d | CS-DID | {“est_method”: “dr”, “control_group”: “notyettreated”, “xformla”: “y ~ 1”, “base_period”: “universal”, “anticipation”: 0, “panel”: true, “allow_unbalanced_panel”: false, “clustervar”: “cluster”, “clusters”: 47, “bstrap”: false, “cband”: false, “aggregate”: “dynamic”, “event_aggregate”: “dynamic”, “balance_e”: null, “post_event_min”: 0, “post_event_max”: 45} | {“parameter”: null, “value”: null, “status”: “not_applicable”, “fit_scope”: “Group-time DIDs; no synthetic-control pooling parameter”} |
| y__d | SDID | {} | {“parameter”: null, “value”: null, “status”: “not_applicable”, “fit_scope”: “Separate adoption-cohort fits”, “within_cohort_target”: “Mean treated outcome history”, “cross_cohort_partial_pooling”: false, “shared_structure”: null, “note”: “Regularization penalties are not partial-pooling parameters.”} |
| y__d | Matrix completion | {“lambda_fraction”: 0.25, “max_iterations”: 1000, “tolerance”: 1e-12} | {“parameter”: null, “value”: null, “status”: “not_applicable”, “fit_scope”: “Joint untreated-cell fit”, “within_cohort_target”: null, “cross_cohort_partial_pooling”: false, “shared_structure”: “Low-rank matrix and unit/time effects”, “note”: “Regularization penalties are not partial-pooling parameters.”} |
| y__d | Horizontal ridge | {“penalty”: 1.0} | {“parameter”: null, “value”: null, “status”: “not_applicable”, “fit_scope”: “Separate adoption-cohort fits”, “within_cohort_target”: “Mean treated outcome history”, “cross_cohort_partial_pooling”: false, “shared_structure”: null, “note”: “Regularization penalties are not partial-pooling parameters.”} |
| y__d | Multisynth (cohort-averaged) | {“nu”: 0.605112942944002, “nu_selection”: “Package data-derived default”, “lambda”: 0, “fixedeff”: true, “scm”: true, “time_cohort”: true, “n_leads”: 46, “n_lags”: 50, “eps_abs”: 1e-06, “eps_rel”: 1e-06, “V”: “Identity (package default)”, “matrix_assembly”: “Unmodified package default”} | {“parameter”: “nu”, “value”: 0.605112942944002, “status”: “used”, “fit_scope”: “Joint optimization of 15 cohort fits”, “cross_cohort_partial_pooling”: true, “note”: “nu=0 weights separate cohort imbalance only; nu=1 weights pooled imbalance only. One shared nu, not a cohort-specific tuning parameter.”} |
| y__d | Multisynth (individual) | {“nu”: 0.151637936983281, “nu_selection”: “Package data-derived default”, “lambda”: 0, “fixedeff”: true, “scm”: true, “time_cohort”: false, “n_leads”: 46, “n_lags”: 50, “eps_abs”: 1e-06, “eps_rel”: 1e-06, “V”: “Identity (package default)”, “matrix_assembly”: “Equivalent sparse matrix construction”} | {“parameter”: “nu”, “value”: 0.151637936983281, “status”: “used”, “fit_scope”: “Joint optimization of 529 unit fits”, “cross_cohort_partial_pooling”: true, “note”: “nu=0 weights separate cohort imbalance only; nu=1 weights pooled imbalance only. One shared nu, not a cohort-specific tuning parameter.”} |
Cohort-deviation F-test
The imported TestingInEventStudies design parameterizes the saturated regression as a common event-time path plus cohort deviations. The earliest cohort supplies the reference path. The null sets identified cohort deviations at positive event times to zero; common effects, leads, and period-zero deviations remain unrestricted. This is the homoskedastic nested-model F-test, separate from the clustered estimator intervals.
run.base.f_report()**y__d: F(379, 36764) = 2.9413; p = 5.812e-72.** Homoskedastic nested-model F. 780 design columns.
Each line is a cohort deviation. The joint test restricts positive-event-time deviations to zero; event time 0 remains unrestricted.
Reproducibility
Each estimator is checkpointed independently. Cache receipts contain the prepared-panel fingerprint, settings, adapter hashes and Python package versions. Model outputs record their own package/tuning details. The JSON also exports the forest plot’s exact scalar ranges, interval methods and reporting-source hash. R uses the project’s .Rlib; raw panels and local fit matrices are not public downloads.
Editable Jot source · Download QMD · Helpers, configurations and instructions
Choose a study JSON for outcomes, treatments, IDs, controls, weights and period coding. Choose a comparison JSON for estimator toggles, tuning, output directory and timeout. Every outcome–treatment pair is estimated separately. For a new study, omit regression_results to fit its regressions afresh; the supplied Credit example may reuse validated archive regression results after source, sample and configuration checks.
uv sync --locked
uv sync --locked --project TestingInEventStudies
QUARTO_PYTHON="$PWD/.venv/bin/python" OPENBLAS_NUM_THREADS=1 \
quarto render templates/expensive/replication-template.qmd \
--execute-dir "$PWD"