Optimizers

Static optimization routines for Python callbacks

1 Where it fits

Group: Estimation interfaces

Optimizers is a small namespace for callback-driven numerical optimization. It is not an estimator; it exposes reusable routines for smooth objectives, nonlinear least squares, and a simple stochastic global search.

2 Solver mechanics

All methods bridge Python callbacks into Rust and normalize the result into the same dictionary, but their step construction and success rules differ.

2.1 BFGS and L-BFGS

Both call fun(theta) and grad(theta) separately, use a More-Thuente line search, and apply tolerance to gradient and cost stopping. BFGS initializes a dense inverse-Hessian approximation at identity and updates the full matrix. L-BFGS retains seven correction pairs instead. Both return Argmin’s best parameter and classify success only from its termination status; reaching the iteration budget returns a dictionary with success=False rather than raising.

2.2 Nonlinear conjugate gradient

This method uses Polak-Ribiere-plus directions and an Armijo backtracking line search with sufficient-decrease constant \(0.2\). It supports periodic and orthogonality restarts. Because the underlying solver does not use the public tolerance directly, the wrapper reevaluates the final Python gradient and upgrades the result to success when its Euclidean norm is at most tolerance.

2.3 Gauss-Newton least squares

The package owns this loop. At each iterate it calls the residual and Jacobian functions, requires Jacobian shape (n_residuals, n_parameters), forms \(g=J'r\) and \(J'J\), explicitly inverts \(J'J\), and proposes \((J'J)^{-1}g\). Gradient or step norm below tolerance stops. Otherwise a halving line search requires strict decrease in \(\|r\|^2/2\) down to scale \(10^{-8}\). A small objective change succeeds; a singular normal matrix, callback error, or malformed shape raises, while line-search or iteration exhaustion returns success=False.

2.4 Simulated annealing

The proposal modifies at least one randomly selected coordinate, adds a uniform perturbation in [-step_size, step_size], and clamps it to optional elementwise bounds. Proposal and acceptance RNGs use separate Xoshiro streams; with a seed the second stream uses seed + 1. Argmin’s Boltzmann temperature schedule runs until its stopping status or 1,000 consecutive stalls in best or accepted states. This is the only method that can use OS entropy when no seed is supplied.

3 Performance and failure behavior

Every objective, gradient, residual, or Jacobian evaluation crosses the Python-Rust boundary and copies NumPy results. BFGS stores \(O(p^2)\) curvature state, L-BFGS and nonlinear CG use \(O(p)\) state aside from callback data, and native Gauss-Newton forms and inverts a dense \(p\times p\) normal matrix. The wrappers return the best parameter, then reevaluate the public objective to populate fun; a callback that is stochastic or stateful can therefore report a value different from the one attached to the optimizer’s best state.

4 Python API

Constructor: cm.Optimizers

The methods are static and return plain dictionaries with scipy-like keys: x, fun, nit, success, message, and method. Smooth minimizers require objective and gradient callbacks; Gauss-Newton requires residual and Jacobian callbacks; simulated annealing only requires the objective.

print(inspect.signature(cm.Optimizers))

()

cls = cm.Optimizers
display(HTML(html_table(["Public method"], public_methods(cls))))

Public method
Optimizers()
minimize_bfgs(fun, x0, grad, max_iterations=100, tolerance=1e-06)
minimize_gauss_newton_ls(residual_fn, x0, jacobian_fn, max_iterations=100, tolerance=1e-06)
minimize_lbfgs(fun, x0, grad, max_iterations=100, tolerance=1e-06)
minimize_nonlinear_cg(fun, x0, grad, max_iterations=100, restart_iters=10, restart_orthogonality=0.1, tolerance=1e-06)
minimize_simulated_annealing(fun, x0, lower=None, upper=None, temp=15.0, step_size=0.1, max_iterations=5000, seed=None)

5 Minimal example

def fun(theta):
    return float(np.sum((theta - np.array([1.0, -2.0])) ** 2))
def grad(theta):
    return 2.0 * (theta - np.array([1.0, -2.0]))
res = cm.Optimizers.minimize_bfgs(fun, np.zeros(2), grad, max_iterations=100)
print(res)

{'x': array([ 1., -2.]), 'fun': 0.0, 'nit': 1, 'success': True, 'message': 'Solver converged', 'method': 'bfgs'}

6 summary() contract

The table below is generated by fitting the live class in this repository and then inspecting summary(). Shapes are shown because most values are plain NumPy arrays or scalars.

class _Dummy:

    def summary(self):
        return {'note': 'Optimizers has static methods, not fitted state'}
model = _Dummy()
summary = model.summary()
display(HTML(html_table(["summary() key", "shape"], summary_shape_rows(summary))))

summary() key	shape
note	()