clophfit.fitting.plate_lm#

Fit a whole plate at once, classically, with plate-wide noise scales.

Every classical arm in this campaign fits one well at a time: fit_binding_glob takes a single well’s Dataset, and the control groups are only pooled afterwards, by averaging per-well K estimates. The Bayesian multi-well model does something the classical side cannot answer to - it shares one K across a control group during the fit, and learns a per-label noise scale from the whole plate - so the two have never been compared like for like on the control holdout.

This closes that gap. Residuals from every well are concatenated into one least-squares problem: K is shared within each control group and free elsewhere, S0 and S1 stay per well and label, and the per-label scale is profiled rather than fitted.

Profiling is not an optimisation detail. Multiplying every weight by a constant leaves the argmin of a least-squares objective untouched - only chi-square and the covariance move - so a global noise scale is not identifiable that way at all. For fixed structural parameters the maximum-likelihood scale has a closed form, the root-mean-square of that label’s standardised residuals, so the fit alternates: solve with current scales, update the scales, repeat. That is what lmfit’s scale_covar does with one global factor, generalised to one per label, which is the structure section 1 found to be the right amount.

Classes#

PlateLMResult

Outcome of one plate-wide classical fit.

Functions#

`fit_plate_lm`(datasets, groups, *[, max_iter, tol, ...])	Fit every well of a plate jointly, profiling one noise scale per label.
`profile_k_intervals`(datasets, groups, wells, *[, ...])	Profile-likelihood interval on K, for wells the quadratic error fails.
`apply_excluded_points`(datasets, excluded_points, *[, ...])	Mask the points a screen rejected, so the next fit inherits its verdict.
`screening_sigma_floor`(residuals, *[, fraction])	Smallest sigma the screen may judge a point by, per well and label.
`fractional_outliers`(residuals, *[, frac_threshold, ...])	Points whose fractional deviation is too large to be ordinary noise.
`ratiometric_exempt`(residuals, *[, frac_threshold, ...])	Points no screen should take, because the ratio already cancels them.
`fit_plate_lm_screened`(datasets, groups, *[, ...])	Find outliers with a calibrated ruler, then fit K with the plain one.

Module Contents#

class clophfit.fitting.plate_lm.PlateLMResult#

Outcome of one plate-wide classical fit.

Parameters:

k (dict[str, float]) – Fitted K per well. Wells sharing a control group hold the identical value, not an average of separate fits.
k_stderr (dict[str, float]) – Standard error on each K, from the Jacobian at the solution, scaled by the profiled noise. Conditional on the profiled scales, so mildly optimistic.
ye_mag (dict[str, float]) – Profiled noise multiplier per label, relative to the supplied y_err.
params (dict[str, dict[str, float]]) – Per well, the whole fitted curve - K, sK and the S0_/S1_ plateaus per label - not only K. Exporting K alone made the plate fit a second-class output: it could be tabulated but not plotted, while every other fitter in the same run produced a K plot and per-well figures.
n_points (int) – Unmasked observations entering the fit.
n_params (int) – Free structural parameters.
success (bool) – Whether the final least-squares solve converged.
n_excluded (int) – Observations dropped by a screening pass, 0 for a single fit.
noise (dict[str, dict[str, float]]) – Per-label sigma_floor, gain and alpha actually used. When calibration ran these are what it estimated, and they are the whole point of asking for it - a run that fits the noise and reports only ye_mag has thrown the answer away.
excluded_points (dict[str, dict[str, list[int]]]) – Well to label to the original indices a screening pass dropped. The screen builds its own masked copies internally, so without this a caller plotting the input datasets would draw the discarded points as though they had been fitted.
residuals (list[dict[str, Any]]) – One record per unmasked observation, on the library’s canonical columns: well, label, step, yhat, raw_res, sigma and std_res. sigma includes the profiled scale, so std_res has per-label SD ~1 by construction and only its shape - tails, step dependence - carries information; raw_res and yhat are the signal-scale pair the noise-calibration estimators read.
budget_reached (bool) – Whether any solve stopped on its evaluation cap rather than on scipy’s own convergence tests. Kept apart from success: the caps are safety nets, and a caller deciding whether to trust a fit needs to know which of the two it is looking at.

clophfit.fitting.plate_lm.fit_plate_lm(datasets, groups, *, max_iter=6, tol=0.001, loss='linear', noise_model=None, calibrate_noise=False, noise_free=('gain', 'alpha'))#

Fit every well of a plate jointly, profiling one noise scale per label.

Parameters:

datasets (Mapping[str, Any]) – Well identifier to Dataset.
groups (Mapping[str, Sequence[str]]) – Control group name to member wells; those wells share one K.
max_iter (int) – Maximum alternations between solving and rescaling.
tol (float) – Stop when every scale moves by less than this, relatively.
loss (Literal["linear", "huber", "soft_l1", "cauchy"]) – Loss handed to the solver. "linear" is ordinary least squares. Anything else down-weights large residuals, which matters when the fit is about to be used to find outliers: least squares drags the curve toward a bad point, shrinking its own residual and inflating its neighbours’, and the profiled scale then inflates too because it is the RMS of those residuals. Screening on such a fit decides with a bent curve and a stretched ruler. With a robust loss the scale is taken from the median absolute deviation for the same reason.
noise_model (Mapping[str, Any] | None) – Per-label noise parameters. When given, a signal-dependent y_err is re-evaluated at the model prediction between passes instead of being taken at the observation, where a point’s own fluctuation would set its own weight. A flat floor makes this a no-op.
calibrate_noise (bool) – Estimate gain and alpha per label from each pass’s residuals, floor pinned, and refit under the result. Off by default: it describes the residuals better and fits K worse, because down-weighting high-signal points down-weights the plateaus that pin S0 and S1.
noise_free (Sequence[str]) – Which noise terms calibrate_noise may move: any of "floor", "gain" and "alpha". The default frees both signal terms with the floor pinned, which is the only case the estimator could express before and the one where gain and alpha are collinear. Holding one of them lets the iteration converge instead of wandering along that ridge.

Returns:

Fitted K per well, its standard error, and the profiled scales.

Return type:

PlateLMResult

clophfit.fitting.plate_lm.profile_k_intervals(datasets, groups, wells, *, level=0.94, loss='linear', noise_model=None)#

Profile-likelihood interval on K, for wells the quadratic error fails.

k_stderr comes from the curvature at the optimum, which describes the likelihood only where it is close to a parabola. That assumption breaks on exactly the wells worth checking: a flat basin reports a standard error of 1e5, and a well whose K is genuinely pinned down can still sit in an asymmetric valley. Profiling makes no such assumption - it fixes K, lets every other parameter re-optimise, and reads off where the cost has risen by the chi-square quantile for one degree of freedom.

One refit per profile point, so this is deliberately not run for a whole plate. Pass the wells that need it.

Parameters:

datasets (Mapping[str, Any]) – Well identifier to Dataset, as passed to fit_plate_lm().
groups (Mapping[str, Sequence[str]]) – Control group name to member wells; those wells share one K.
wells (Sequence[str]) – Wells to profile. Grouped wells share a K, so profiling any member profiles the group.
level (float) – Interval mass, 0.94 to match the HDI this project reports elsewhere.
loss (Literal["linear", "huber", "soft_l1", "cauchy"]) – Loss for the refits; use the one the reported fit used.
noise_model (Mapping[str, Any] | None) – Signal-dependent noise model, if the reported fit used one. The profile has to weight the data the same way the fit did or its criterion refers to a different likelihood.

Returns:

Well to (lower, upper). A bound that runs into the edge of the K range without the cost rising enough comes back as -inf or inf: the data does not bound K on that side, which is the honest answer and the one a standard error cannot express.

Return type:

dict[str, tuple[float, float]]

clophfit.fitting.plate_lm.apply_excluded_points(datasets, excluded_points, *, min_keep=5)#

Mask the points a screen rejected, so the next fit inherits its verdict.

fit_plate_lm_screened() builds its masked copies internally and hands the caller’s datasets back untouched, which is right for it and wrong for a pipeline: a Bayesian fit that runs afterwards would otherwise see every point the screen threw out. This applies the screen’s excluded_points to a fresh copy, under the same min_keep guard the screen itself uses, so the two agree about which drops were actually taken.

Parameters:

datasets (Mapping[str, Any]) – Well key to dataset. Not modified.
excluded_points (Mapping[str, Mapping[str, Sequence[int]]]) – Well to label to positional indices, as PlateLMResult.excluded_points records them. Unknown wells, labels and out-of-range indices are ignored: a screen may have run over a superset of what is fitted here.
min_keep (int) – Never leave a label with fewer unmasked points than this.

Returns:

A copy with the rejected points masked out.

Return type:

dict[str, Any]

clophfit.fitting.plate_lm.screening_sigma_floor(residuals, *, fraction=_SCREEN_AMPLITUDE_FRACTION)#

Smallest sigma the screen may judge a point by, per well and label.

The amplitude is the span of the fitted curve over the observed points, max(yhat) - min(yhat), not the span of the data. An outlier inflates the observed span – on these plates 31 cells of 1904 have a fitted span below half the observed one – which would raise the floor and make the screen inert on exactly the curve that needed it. It is also not |S1 - S0|, which is the range extrapolated beyond the titrated window and can far exceed anything the curve actually traverses.

Parameters:

residuals (Sequence[Mapping[str, Any]]) – Rows from a first-pass fit, carrying well, label and yhat.
fraction (float) – Fraction of the amplitude to floor sigma at. 0 disables the floor and restores judging purely on the physical sigma.

Returns:

(well, label) to the floor, 0.0 where the fitted curve is flat and there is no amplitude to take a fraction of.

Return type:

dict[tuple[str, str], float]

clophfit.fitting.plate_lm.fractional_outliers(residuals, *, frac_threshold=_FRACTIONAL_THRESHOLD, ratiometric_ratio=_RATIOMETRIC_RATIO, label=None)#

Points whose fractional deviation is too large to be ordinary noise.

Applied to the first label only. The second channel’s plateau falls to zero, where a fraction of the prediction stops being meaningful; its outliers are left to the z-screen with its amplitude floor.

Parameters:

residuals (Sequence[Mapping[str, Any]]) – Rows from a fit, carrying well, label, step, raw_i, raw_res and yhat – the schema fit_plate_lm produces, which has no separate y.
frac_threshold (float) – Deficits deeper than this fraction of the prediction are screened; raw_res / yhat must be below -frac_threshold. Positive excursions are never screened.
ratiometric_ratio (float) – Spare a point when the other channel moves the same way by at least this fraction of the first channel’s move.
label (str | None) – Label to screen. None takes the lowest label present, which is the 400 nm channel on these plates.

Returns:

(well, label) to the raw_i values to drop.

Return type:

dict[tuple[str, str], set[int]]

clophfit.fitting.plate_lm.ratiometric_exempt(residuals, *, frac_threshold=_FRACTIONAL_THRESHOLD, ratiometric_ratio=_RATIOMETRIC_RATIO)#

Points no screen should take, because the ratio already cancels them.

When both channels of a well shift the same way at the same step by a comparable fraction, the cause is a well-level multiplicative artefact – a volume or read error – and a ratiometric measurement is immune to it. Removing such points biases the plateau instead of cleaning it.

This is a veto over every screening criterion, not a clause of one. On L3 C06 the z-screen takes label 2 at |z| 6.8 and 6.7 while both channels are simply reading 15% high then 15% low; sparing those points only inside the fractional criterion leaves the z-screen free to take them anyway.

Parameters:

residuals (Sequence[Mapping[str, Any]]) – Rows from a fit, carrying well, label, step, raw_i, raw_res and yhat.
frac_threshold (float) – Only shifts larger than this are treated as artefacts; ordinary noise moves both channels a little and is not exempted.
ratiometric_ratio (float) – How closely the two channels must agree, as a fraction of the larger move.

Returns:

(well, label) to the raw_i values to spare, for every label involved in the shared shift.

Return type:

dict[tuple[str, str], set[int]]

clophfit.fitting.plate_lm.fit_plate_lm_screened(datasets, groups, *, noise_model=None, threshold=3.0, min_keep=5, amplitude_fraction=_SCREEN_AMPLITUDE_FRACTION, frac_threshold=None, calibrate_screen=True, calibrate_refit=False)#

Find outliers with a calibrated ruler, then fit K with the plain one.

A standardised residual is only as good as the sigma it is divided by, and the default sigma is one number per label - the read-noise floor - while the noise plainly grows with signal. On L6a the binned |std residual| runs from 0.065 at low signal to 1.113 at high, so a fixed threshold is a far harsher test of a bright point than a dim one. Screening on that ruler discards the plateaus, which are what pin S0 and S1 and hence where the midpoint sits: over eleven plates it threw away 947 points and made K worse.

So the two jobs are split, each given the model measured to win at it. The screening pass calibrates gain and alpha per label from its own residuals, which puts every point on a comparable scale; the refit then uses the plain weighting, because calibrated weights describe residuals better and fit K worse. Same threshold, right ruler: 99 points dropped instead of 947, and sum_log -5.92 against -5.49 for no screening at all, improving nine plates of eleven.

The threshold is not a free knob. At 2.5 the screen turns harmful again (-4.42), so 3.0 is not a rounding of “about three sigma” but the value that separates a useful screen from a destructive one.

Parameters:

datasets (Mapping[str, Any]) – Well identifier to Dataset.
groups (Mapping[str, Sequence[str]]) – Control group name to member wells; those wells share one K.
noise_model (Mapping[str, Any] | None) – Per-label noise parameters supplying the floors the calibration pins. Without it there is nothing to calibrate and the screening pass falls back to the plain fit, which is the behaviour this function exists to avoid - pass it.
threshold (float) – Standardised-residual magnitude above which a point is dropped, judged on the calibrated scale.
min_keep (int) – Never leave a label with fewer points than this, whatever their residuals. Four cannot locate a midpoint and two plateaus: on L4, E03 lost three of seven, landed exactly on the old default of four, and its fit collapsed to a flat line - taking the well’s other label with it, since K is shared between them.
amplitude_fraction (float) – Floor the screening sigma at this fraction of each curve’s fitted span, so a point is judged against model error and not only against measurement noise. 0 restores judging on the physical sigma alone. See screening_sigma_floor().
frac_threshold (float | None) – When given, also screen the first label on |y - yhat| / yhat exceeding this, sparing points the second channel moves with. None leaves the z-screen alone. See fractional_outliers().
calibrate_screen (bool) – Judge points on the calibrated ruler (the default, and the reason this function exists) or on the weights as built.
calibrate_refit (bool) – Fit K with calibrated weights instead of the plain ones. The two jobs used to be fixed to one choice each, and the CLI then reused --plate-noise calibrated to mean calibrated weights without a screen but plain weights after one – a confound worth ~1.1 in the eleven-plate bench score, credited to the screen. Each is now its own switch; the defaults keep the split described above.

Returns:

The refit, carrying the count of dropped observations in n_excluded. When nothing crosses the threshold this is the single-pass fit.

Return type:

PlateLMResult