clophfit.fitting.utils#

Utility functions for fitting modules.

Functions#

`parse_remove_outliers`(spec)	Parse outlier specification `"method:threshold:min_keep"`.
`robust_scale`(residuals)	Estimate a robust standard deviation from residuals via the MAD.
`robust_z_scores`(residuals[, sigma])	Score residuals by a robust z built on the median and the MAD.
`studentized_scores`(residuals, jacobian)	Score residuals by the externally studentized (deleted) residual.
`bonferroni_threshold`(n, dof[, alpha])	Two-sided Student-t cutoff, Bonferroni-corrected for testing n points.
`identify_outliers_mad`(residuals[, threshold])	Identify outliers by robust z-score, using the median and the MAD.
`cap_by_min_keep`(flagged, scores, min_keep)	Drop the weakest flags until at least min_keep points survive.
`reweight_from_residuals`(ds, residuals)	Update y_errc in a Dataset from the mean absolute residuals of each label.
`flag_trend_outliers`(x, y[, threshold])	Flag outliers using robust Theil-Sen regression of y on x.
`fit_trendline`(x, y)	Fit a robust Theil-Sen regression line.
`smoothness`(y)	Calculate the smoothness of a curve.
`roughness`(y)	Calculate the roughness of a curve.
`outlier_scores_extended`(x, y)	Compute outlier scores for each point using geometric deviation.
`apply_outlier_mask`(ds[, threshold, min_keep])	Mask outlier points iteratively in each DataArray of a Dataset.
`add_robust_scores`(residuals, *[, levels, residual_col])	Add robust scale and z-score columns at several pooling levels.

Module Contents#

clophfit.fitting.utils.parse_remove_outliers(spec)#

Parse outlier specification "method:threshold:min_keep".

Parameters:: spec (str) – The string to parse.
Returns:: A tuple of method, threshold, min_keep.
Return type:: tuple[str, float, int]

Examples

"mad:4.0:5" -> (“mad”, 4.0, 5)
"mad" -> (“mad”, 3.5, 1)
"studentized" -> (“studentized”, 0.05, 1)

clophfit.fitting.utils.robust_scale(residuals)#

Estimate a robust standard deviation from residuals via the MAD.

The MAD is scaled to be a consistent estimator of the standard deviation under normality. It collapses to zero when more than half the residuals are identical, so the scale falls back to a normal-consistent IQR and then to the (non-robust) standard deviation.

Parameters:: residuals (np.ndarray) – The residuals to estimate a scale from. NaNs are ignored.
Returns:: A positive scale, or 0.0 if no positive scale can be found.
Return type:: float

clophfit.fitting.utils.robust_z_scores(residuals, sigma=None)#

Score residuals by a robust z built on the median and the MAD.

A mean/standard-deviation z-score is inflated by the very outlier it is meant to expose, which caps the attainable score at sqrt(n - 1): at n = 7 nothing can score above 2.45, and the score saturates rather than growing with the outlier. The median and MAD do not respond to the outlier, so this score has no such ceiling.

Parameters:

residuals (np.ndarray) – The residuals to score. NaNs are ignored when locating the centre and scale, and score as NaN.
sigma (float | None) – Scale to divide by. None estimates it from residuals themselves. Pass a pooled scale (e.g. estimated plate-wide) when the per-fit sample is too small for a stable MAD.

Returns:

Absolute robust z-scores, all zeros if no positive scale is available.

Return type:

np.ndarray

clophfit.fitting.utils.studentized_scores(residuals, jacobian)#

Score residuals by the externally studentized (deleted) residual.

Ordinary residuals are not comparable across points: a high-leverage point pulls the fit towards itself, so its residual is shrunk by \(\sqrt{1 - h_{ii}}\) and it can hide from any test applied to raw residuals. This score divides that shrinkage out and rescales by a leave-one-out variance, so the point under test contributes nothing to the scale used to judge it:

\[t_i = \frac{r_i}{s_{(i)}\sqrt{1 - h_{ii}}}\]

Under the null it follows a Student-t with \(n - p - 1\) degrees of freedom, which makes a calibrated threshold possible (see bonferroni_threshold()) rather than a hand-picked constant.

Parameters:

residuals (np.ndarray) – Residual vector actually minimized (weighted, if the fit was weighted), so that it is homoscedastic and matches jacobian.
jacobian (np.ndarray) – The (n, p) Jacobian of those residuals at the solution.

Returns:

Absolute studentized residuals, and the degrees of freedom n - p - 1. Scores are zeros when there is no residual freedom left.

Return type:

tuple[np.ndarray, int]

clophfit.fitting.utils.bonferroni_threshold(n, dof, alpha=0.05)#

Two-sided Student-t cutoff, Bonferroni-corrected for testing n points.

Testing every point for outlyingness is n simultaneous tests, so the per-test level is alpha / n. Without the correction, a plate of 90 wells would flag points at the nominal rate by chance alone.

Parameters:

n (int) – Number of points being tested simultaneously.
dof (int) – Degrees of freedom of the studentized residual (n - p - 1).
alpha (float) – Family-wise error rate. Default 0.05.

Returns:

The cutoff, or inf when there is no residual freedom to test with.

Return type:

float

clophfit.fitting.utils.identify_outliers_mad(residuals, threshold=3.5)#

Identify outliers by robust z-score, using the median and the MAD.

Parameters:

residuals (np.ndarray) – The residuals to analyze. Use raw (unweighted) residuals, so that the scale is estimated from the data rather than from y_err.
threshold (float) – The robust z-score beyond which a point is considered an outlier.

Returns:

A boolean mask where True indicates an outlier.

Return type:

ArrayMask

clophfit.fitting.utils.cap_by_min_keep(flagged, scores, min_keep)#

Drop the weakest flags until at least min_keep points survive.

Parameters:

flagged (ArrayMask) – Boolean mask, True where a point is a candidate for removal.
scores (np.ndarray) – Per-point outlier scores used to rank candidates worst-first.
min_keep (int) – Minimum number of points that must remain unflagged.

Returns:

A mask flagging at most len(flagged) - min_keep of the worst points.

Return type:

ArrayMask

clophfit.fitting.utils.reweight_from_residuals(ds, residuals)#

Update y_errc in a Dataset from the mean absolute residuals of each label.

Parameters:

ds (Dataset) – The input dataset.
residuals (np.ndarray) – The combined 1D array of residuals for all labels in the dataset, in the order of ds.values().

Returns:

A new dataset with updated y_err.

Return type:

Dataset

clophfit.fitting.utils.flag_trend_outliers(x, y, threshold=3.0)#

Flag outliers using robust Theil-Sen regression of y on x.

A point is flagged if its residual is far from the trendline (Z-score < -threshold) OR if its x-value is extremely low compared to the population (Z-score < -threshold).

Parameters:

x (pd.Series) – The independent variable (e.g., maximum signal, mean).
y (pd.Series) – The dependent variable (e.g., signal span, std, or dynamic range).
threshold (float) – The Z-score threshold for flagging an outlier.

Returns:

A boolean Series of the same length as x, True for outliers.

Return type:

pd.Series

clophfit.fitting.utils.fit_trendline(x, y)#

Fit a robust Theil-Sen regression line.

Parameters:

x (pd.Series) – The independent variable.
y (pd.Series) – The dependent variable.

Returns:

Slope and intercept.

Return type:

tuple[float, float]

clophfit.fitting.utils.smoothness(y)#

Calculate the smoothness of a curve.

Sum of |consecutive diffs| / total span. = 1 for perfectly monotone, > 1 for noisy/non-monotone.

Parameters:: y (np.ndarray) – The signal array.
Returns:: The smoothness value.
Return type:: float

clophfit.fitting.utils.roughness(y)#

Calculate the roughness of a curve.

Excess path fraction: 0 = perfectly monotone, 1 = all noise, flat-safe. roughness = (consec - span) / consec.

Parameters:: y (np.ndarray) – The signal array.
Returns:: The roughness value.
Return type:: float

clophfit.fitting.utils.outlier_scores_extended(x, y)#

Compute outlier scores for each point using geometric deviation.

Uses a hybrid approach for edge points: - If edge_step > 2 * local_step: anomalously large jump → use full projection deviation - Elif wrong direction (reversal): use projection deviation - Else (correct direction / plateau approach): score = 0

For internal points: triangle inequality score.

Parameters:

x (np.ndarray) – x-values (e.g. pH or concentration).
y (np.ndarray) – Observed y-values.

Returns:

Per-point outlier scores (non-negative; higher = more anomalous).

Return type:

np.ndarray

Examples

>>> import numpy as np
>>> x = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
>>> y = np.array([10.0, 8.0, 15.0, 4.0, 2.0])
>>> scores = outlier_scores_extended(x, y)
>>> bool(scores[2] > 0.4)
True

clophfit.fitting.utils.apply_outlier_mask(ds, threshold=0.2, min_keep=3)#

Mask outlier points iteratively in each DataArray of a Dataset.

Removes the single worst outlier (if above threshold) and recomputes scores, repeating until no score exceeds the threshold or fewer than min_keep unmasked points remain.

Parameters:

ds (Dataset) – Dataset to process (deep-copied; input is not modified).
threshold (float, optional) – Outlier score above which a point is masked. Default is 0.2.
min_keep (int, optional) – Minimum number of unmasked points to retain. Default is 3.

Returns:

A new Dataset with outlier points masked.

Return type:

Dataset

clophfit.fitting.utils.add_robust_scores(residuals, *, levels=('well_label', 'well', 'label', 'global'), residual_col='raw_res')#

Add robust scale and z-score columns at several pooling levels.

For each requested level this adds robust_sigma_{level} (a normal-consistent MAD estimate of the noise scale over that grouping) and robust_z_{level} (|r - median| / sigma, the median always taken per well and label, so only the scale is pooled).

When "well_label" and "label" are both present it also adds ye_mag_est, their scale ratio. That is a classical, sampler-free estimate of the per-well noise inflation the hierarchical model learns as ye_mag_{lbl}, useful for cross-checking it cheaply.

A level that is not identifiable from residuals – a pooled level on a single-well table, say – yields all-NaN columns rather than a silently degenerate duplicate of a finer level. Which levels those were is recorded in df.attrs["robust_score_degenerate_levels"].

Parameters:

residuals (pd.DataFrame) – Canonical residual table carrying well, label and residual_col.
levels (tuple[str, ...]) – Any of "well_label", "well", "label", "global". "well" pools the labels of one well, so it only means something when the labels share a noise scale; on plates where they do not (a bright band and a dim one, say) its scale is dominated by the noisier label and "well_label" is the one to use.
residual_col (str) – Residual column to score. Defaults to the raw (unweighted) residual, so the scale is estimated from the data rather than from an assumed y_err.

Returns:

Copy of residuals with the added columns.

Return type:

pd.DataFrame

Raises:

ValueError – If levels names an unknown level, or residual_col is missing.