Ridge

L2-regularized least squares with optional CV

1 Where it fits

Group: Regression

Ridge solves

\[ \min_{\alpha,\beta} \sum_i (y_i - \alpha - x_i'\beta)^2 + \lambda \|\beta\|_2^2. \]

A scalar penalty gives one ridge fit. A penalty grid with cv selects a penalty by cross-validation, stores the coefficient path, and refits on the full sample.

2 Criterion and tuning

With \(\tilde X=[\mathbf 1,X]\) and \(P_\lambda=\operatorname{diag}(0,\lambda,\ldots,\lambda)\), the fitted parameter solves

\[ \hat\theta_\lambda = \arg\min_\theta \sum_i w_i(y_i-\tilde x_i'\theta)^2+\theta'P_\lambda\theta. \]

The penalty is not divided by \(n\), so its scale depends on the sample weights and sample size. The intercept is never penalized. The implementation solves an augmented least-squares problem with \(\sqrt{\lambda}I\) rows, avoiding an explicit normal-equation inverse for the point estimate.

When given a grid, the class fits every penalty and chooses the lowest mean validation MSE. Folds are deterministic, unshuffled assignments \(i\bmod K\); weighted fits use weighted validation MSE. It then selects the corresponding full-sample path coefficient. The grid search does not standardize features, so penalty effects depend directly on column scale.

3 Implementation walkthrough

The point estimate is implemented as augmented least squares rather than through \((X'X+P_\lambda)^{-1}X'y\).

  1. The wrapper validates the penalty array and data, prepends a column of ones, and, for a weighted fit, multiplies each design row and outcome by \(\sqrt{w_i}\). Zero weights therefore contribute zero rows to the numerical problem.
  2. For \(\lambda>0\), it appends one synthetic row for each penalized coefficient. The slope block of those rows is \(\sqrt{\lambda}I\) and their synthetic outcomes are zero; the intercept column is omitted from this block. Ordinary QR least squares on the augmented system exactly minimizes the stated ridge criterion. At \(\lambda=0\), it solves the original design directly.
  3. Multi-penalty paths center \(X\) and \(y\) using the optional analytic weights, factor the square-root-weighted centered design once by SVD, and reuse \(U,D,V'\) across positive penalties. Zero penalty retains the original QR path. Intercepts are reconstructed from the weighted means.
  4. Cross-validation assigns row \(i\) to fold \(i\bmod K\). Each fold constructs its arrays once and computes the whole path from one factorization. Held-out ordinary or weight-normalized MSE selects the first minimum in grid order. The corresponding full-sample path column becomes the fitted model. Weight shape, sign, finiteness, and positive mass are validated before splitting.
  5. Prediction is a dense \(X\beta\) plus intercept. The training design, response, optional weights, selected penalty, and complete path are retained because summary(), robust covariance calculations, and the pairs bootstrap need them.

This QR construction is more numerically defensible than explicitly solving the normal equations for coefficients, but the inferential code still forms dense cross-products and inverses. The deterministic interleaved folds make a run reproducible without a seed; they are inappropriate when row order itself encodes time, clusters, or another dependence structure.

4 Inference

Let \(B=\tilde X'\tilde X+P_\lambda\) after any square-root weight transformation and define the effective degrees of freedom

\[ d_{\mathrm{eff}} = \operatorname{tr}\{\tilde X'\tilde X B^{-1}\}. \]

The model-based covariance is

\[ \widehat V_{\mathrm{vanilla}} = \hat\sigma^2B^{-1}\tilde X'\tilde X B^{-1}, \qquad \hat\sigma^2=\frac{e'e}{n-d_{\mathrm{eff}}}. \]

HC1, Newey-West, and cluster options use parameter scores \(e_i\tilde x_i'B^{-1}\) and the same finite-sample corrections as OLS, replacing \(n-p\) by \(n-d_{\mathrm{eff}}\). These are conditional linearization variances around the penalized estimator. They do not remove ridge shrinkage bias or account for selecting \(\lambda\) by cross-validation. The pairs bootstrap refits at the already selected penalty; it does not rerun the grid search.

5 Performance and numerical behavior

A scalar fit uses augmented QR. A grid pays for one centered SVD per fold plus the full sample, then smaller spectral solves per positive penalty; zero-penalty entries use QR separately. Dense covariance construction remains quadratic in the number of coefficients. Positive analytic weights determine the weighted criterion; zero-weight rows are excluded from inference counts and clusters, but the existing row-modulo CV assignment is unchanged. Covariance is conditional on the selected penalty and does not correct shrinkage or tuning bias.

6 Python API

Constructor: cm.Ridge

The main methods mirror OLS: fit, fit_weighted, predict, summary, and bootstrap. summary() includes the selected penalty and, for grid fits, cross-validation diagnostics and coefficient paths.

Code 
print(inspect.signature(cm.Ridge))
(penalty=None, cv=5)
Code 
cls = cm.Ridge
display(HTML(html_table(["Public method"], public_methods(cls))))
Public method
Ridge(penalty=None, cv=5)
bootstrap(self, /, n_bootstrap, seed=None)
fit(self, /, x, y)
fit_weighted(self, /, x, y, sample_weight)
predict(self, /, x)
summary(self, /, vcov='hc1', lags=None, clusters=None)

7 Minimal example

rng = np.random.default_rng(2)
x = rng.normal(size=(240, 5))
y = 0.3 + x @ np.array([1.0, -0.8, 0.0, 0.25, 0.1]) + rng.normal(scale=0.6, size=240)
model = cm.Ridge(penalty=np.array([0.0, 0.05, 0.2, 1.0]), cv=4)
model.fit(x, y)
print(model.summary()['penalty'])
print(model.summary()['coef'])
print(model.predict(x[:3]))
1.0
[ 0.96303432 -0.80048681 -0.1007246   0.29078042  0.11061705]
[0.38294603 1.55660946 1.29480827]

8 summary() contract

The table below is generated by fitting the live class in this repository and then inspecting summary(). Shapes are shown because most values are plain NumPy arrays or scalars.

Code 
rng = np.random.default_rng(102)
x = rng.normal(size=(90, 4))
y = 0.3 + x @ np.array([1, -0.5, 0.2, 0]) + rng.normal(size=90)
model = cm.Ridge(penalty=np.array([0.0, 0.1, 1.0]), cv=3)
model.fit(x, y)
summary = model.summary()
display(HTML(html_table(["summary() key", "shape"], summary_shape_rows(summary))))
summary() key shape
intercept ()
coef (4,)
intercept_se ()
coef_se (4,)
penalty ()
penalties (3,)
vcov_type ()
best_penalty_index ()
cv_mse (3,)
intercept_path (3,)
coef_path (4, 3)