a dynamic identification argument converts a long-run policy derivative into a weighted contrast of cumulative future rewards;
the feasible estimator is a small, static covariate-balancing problem over order occasions.
The first step is where the substantive assumptions live. Once it is accepted, the computational core is indeed simple. This vignette uses crabbymetrics.MPE_CBPS, a native Rust implementation checked against the authors’ released A/B and switchback code, then compares it with two direct entropy-calibration fits from crabbymetrics.BalancingWeights.
1 Setup and target
For customer \(i\) at time \(t\), let
\(U_{it}\) be a latent state such as satisfaction, beliefs, or intent;
\(X_{it}\) be observed order characteristics, with \(X_{it}=\varnothing\) when no order is placed;
\(D_{it}\in\{0,1\}\) indicate a delay;
\(Y_{it}\) be a reward such as spend, order frequency, or retention value.
Thus \(0.01\tau\) is the effect of increasing the delay probability by one percentage point. A reliability improvement has \(\Delta\text{Delay}<0\), so its predicted value effect is \(\Delta\text{Delay}\times\tau\).
Delays play the mathematical role of MDP actions, but are not actions in the operational sense: the platform does not deliberately choose to delay an order. The intervention modifies their conditional distribution.
2 Identification is the hard part
The paper assumes:
\((U_{it},X_{it},D_{it})\) is Markov and rewards depend on this current state;
delays are sequentially unconfounded after observed order characteristics are fixed, \[D_{it}\perp (S_{i,t-1},U_{it})\mid X_{it};\]
the effect of a delay on rewards \(k\) periods later decays no more slowly than \(Ce^{-k/\nu}\);
delay probabilities are bounded away from zero and one.
This result is what permits hidden consumer states: \(U_{it}\) need not be measured, provided it does not still confound the contemporaneous delay after conditioning on \(X_{it}\). Increasing \(K\) reduces truncation bias but increases noise because each observation carries a longer cumulative reward.
3 Why it becomes a balancing problem
Inverse-probability weights balance every chosen basis \(b(X)\) in expectation:
The paper estimates two coefficient vectors. In the authors’ released code, let \(W_i^{(a)}=1\{D_i=a\}\), \(z_i=(1,b(X_i)')'\), and \(\dot\pi_i\) be the derivative of the treatment probability with respect to the policy perturbation. For each arm \(a\in\{0,1\}\), it minimizes the convex loss
Thus \(w_i^{(a)}=1+e^{-z_i^\top\widehat\theta_a}\) balances each arm separately to the complete sample, including its total mass through the intercept. MPE_CBPS solves both convex programs with analytic gradients and Hessians in Rust and reports the resulting weights, moments, effective sample sizes, and optimization diagnostics.
3.1 Matching the authors’ repository
The released repository contains the same implementation twice:
CrabbyMetrics reproduces those equations rather than treating a nearby calibration estimator as canonical. The parity tests transcribe the released MIT-licensed SciPy/BFGS reference at commit 06c29f4, solve the same deterministic samples with MPE_CBPS, and compare both coefficient vectors, all observation-level weights, and the final normalized policy-gradient estimate. Coefficients agree within \(2\times10^{-7}\) and the exponentially transformed weights within \(5\times10^{-6}\); the normalized policy-gradient comparison is tighter than \(2\times10^{-9}\) in absolute value.
3.2 Exact CBPS versus BalancingWeights
The paper’s weights use the inverse-logit links \(1+e^{-b^\top\beta_1}\) and \(1+e^{b^\top\beta_0}\). Entropy calibration in BalancingWeights instead uses an exponential tilt. These are not algebraically identical weight families.
If the panel includes periods with no order, the normalized weighted-mean contrast must additionally reflect the empirical order rate \(N_{\mathrm{orders}}/(nT)\). The executable example below treats every recorded row as an order occasion, so this factor is one.
4 A dynamic stress test with known truth
The simulation deliberately has an unobserved persistent state:
Current delays have a direct effect \(c\) and change all future rewards through \(U\). Delay assignment depends only on an observed nonlinear basis \(b(X_{it})\), so sequential unconfoundedness holds even though \(U\) is hidden. Under an additive perturbation to the delay probability, the true horizon-\(K\) effect is
5 Fit the exact and generic CrabbyMetrics estimators
MPE_CBPS internally adds an intercept and standardizes the supplied basis with the same population-standard-deviation convention as the released code. It returns the canonical raw weights: each arm’s active weights sum to the full sample size. estimate() applies the arm masks, policy derivative, and requested denominator without re-normalizing.
Both balancing methods remove the chosen moment discrepancies to numerical precision. MPE_CBPS exposes the exact released objective’s two sets of diagnostics; the generic calibration fits expose their own effective sample sizes and maximum residual imbalance through summary().
The naive contrast has the wrong sign because orders with characteristics associated with higher baseline value are also more likely to be delayed. Both balanced estimators reverse that confounding. Their estimates become more negative with \(K\), approaching the long-run effect as the hidden-state response propagates.
The oracle IPW estimate is not automatically the least noisy line. Long-horizon outcomes and inverse probabilities can be volatile; exact finite-sample balance is a stabilization device, not merely a route to asymptotic consistency. This is the central empirical motivation for the paper’s preference for CBPS over a prediction-tuned propensity model.
7 Unit-clustered bootstrap
The paper resamples customers, not customer-period rows, and re-estimates the weights inside each bootstrap draw. The same rule is used here for the exact \(K=28\)MPE_CBPS estimator.
In the paper’s marketplace simulation, a naive future-outcome contrast overstates the long-run reliability effect by roughly an order of magnitude. IPW with a prediction-tuned MLP propensity model removes much of the bias but remains about three times too large and produces a wide interval. CBPS-IPW predicts 92–97% of the long-run randomized-experiment effect at 56- and 70-day horizons.
The switchback experiment identifies how added supply changes the market-level delay rate; Chronos estimates how a marginal delay-rate change maps into long-run customer value. This product predicts persistent-rollout value much better than reading customer value directly from the short switchback.
9 Practical recipe
For an applied panel:
Define an order-level delay or service-defect indicator and a reward measured on a stable time grid.
Construct \(\Gamma_{it}^K\) for several substantively meaningful horizons rather than choosing one horizon after seeing results.
Build a balance basis from variables determined before the delay: the paper uses rolling delay exposure, order frequency, spend, recency, lifetime orders, same-day orders, distance, preparation time, tip, day of week, and hour of day.
Fit MPE_CBPS once to the pre-delay basis and delay indicator. Use BalancingWeights only when a generic calibration family is substantively preferred.
Inspect maximum imbalance, effective sample size, and weight tails. Relax balance or restrict the target population when overlap is weak.
Form the weighted cumulative-reward contrast, multiplying by the order rate if the denominator is customer-periods rather than order occasions.
Resample customers or other independent units and repeat the full weighting procedure for inference.
Report a horizon curve. Stability across large \(K\) is evidence about mixing; it is not proof of sequential unconfoundedness.
10 Limits of the simple translation
The dynamic theorem is indispensable. A successful balance solver does not justify treating cumulative future outcomes as causal.
Entropy calibration is an analogue, not the paper’s exact inverse-logit family.MPE_CBPS is the exact family; BalancingWeights(objective="entropy") remains useful as a sensitivity check.
This is a local policy derivative. It supports adjacent changes to delay rates, not arbitrary large policy changes without further approximation error.
Observed state must be rich enough. Hidden states may mediate long-run effects, but cannot continue to predict delay after \(X\) is fixed.
Order timing matters. Features contaminated by the current delay or post-delay behavior invalidate the balance design.
Spillovers are separate. The estimator targets long-run direct customer effects. Market-wide feedback through congestion or shared supply requires another design, such as the paper’s switchback component.
A doubly robust extension is possible. The paper points to a debiased machine-learning alternative. The present vignette isolates the transparent weighting estimator.
