AxoBench API

Core metric functions

Signatures, parameters, return values, and source for core metric functions.

Overview

Array-level implementations of Voltage SERA, the subthreshold voltage RMSE, Response SERA, Dynamics SERA, and the SERA skill scores. Predictions and targets are (trace, time, 2) arrays in the AxoBench target coordinate; voltages are converted to millivolts as soma/soma_scale + soma_bias_mv, and raw_soma_mv replaces the reconstructed teacher voltage when supplied. Core metrics gives the definitions.

Source revision: 496e0c1a2dfd. Export index.

compute_core_metric_set

axobench.metrics.compute_core_metric_set(prediction: np.ndarray, target: np.ndarray, *, raw_soma_mv: np.ndarray | None=None, trace_ids: list[str] | None=None, soma_scale: float=0.1, soma_bias_mv: float=-67.7, ignore_start: int=0, spike_threshold: float | None=None, spike_threshold_policy: str='prevalence', spike_tolerances_steps: tuple[int, ...] | list[int]=(0, 1, 2, 3, 4, 5), spike_refractory_steps: int | None=2, timestep_ms: float=1.0, intervention_prediction: np.ndarray | None=None, intervention_target: np.ndarray | None=None, intervention_raw_soma_mv: np.ndarray | None=None) -> dict[str, Any]

Source

Compute the core metrics on aligned arrays without a calibration split. Without spike_threshold, the default policy chooses the threshold on the same traces so that the predicted event count equals the teacher spike count.

Parameters

prediction np.ndarray
required. Model outputs shaped (trace, time, 2).
target np.ndarray
required. Teacher targets shaped (trace, time, 2): spike indicator and soma target coordinate.
raw_soma_mv np.ndarray | None
keyword-only, default=None. Optional unclipped teacher soma voltage in millivolts, shaped (trace, time); replaces the voltage reconstructed from the target.
trace_ids list[str] | None
keyword-only, default=None. Sample IDs, one per trace; the text before the first ':' is the morphology or cell-type group.
soma_scale float
keyword-only, default=0.1. Scale of the soma target coordinate; millivolts are soma / soma_scale + soma_bias_mv.
soma_bias_mv float
keyword-only, default=-67.7. Millivolt offset of the soma target coordinate.
ignore_start int
keyword-only, default=0. Initial timesteps excluded from scoring.
spike_threshold float | None
keyword-only, default=None. Fixed spike threshold; recommended when calibrated on other traces.
spike_threshold_policy str
keyword-only, default='prevalence'. 'prevalence' for the event-count-matched threshold or 'zero' for 0, used when spike_threshold is None.
spike_tolerances_steps tuple[int, ...] | list[int]
keyword-only, default=(0, 1, 2, 3, 4, 5). F1 timing tolerances in steps.
spike_refractory_steps int | None
keyword-only, default=2. Suppression radius of spike events; None treats every supra-threshold step as an event.
timestep_ms float
keyword-only, default=1.0. Duration of one timestep in milliseconds.
intervention_prediction np.ndarray | None
keyword-only, default=None. Optional model outputs for edited inputs; target and prediction then act as the baseline.
intervention_target np.ndarray | None
keyword-only, default=None. Teacher targets for the edited inputs; required with intervention_prediction.
intervention_raw_soma_mv np.ndarray | None
keyword-only, default=None. Optional unclipped teacher voltage for the edited inputs.

Returns

result dict[str, Any]
voltage, dynamics, subthreshold, spike, and optionally response sub-results, plus voltage_sera_iron_per_spike_mv2, subthreshold_rmse_mv, subthreshold_skill, dynamics_sera_iron_per_spike_mv2_per_ms2, spike_mean_f1, spike_f1_area, spike_threshold, spike_threshold_kind, and response_sera_iron_per_spike_mv2 when available.

iron_voltage_sera

axobench.metrics.iron_voltage_sera(prediction: np.ndarray, target: np.ndarray, *, raw_soma_mv: np.ndarray | None=None, soma_scale: float=0.1, soma_bias_mv: float=-67.7, ignore_start: int=0) -> dict[str, Any]

Source

IRon-default Voltage SERA per teacher spike, with relevance fitted on the pooled scored teacher voltage.

Parameters

prediction np.ndarray
required. Model outputs shaped (trace, time, 2).
target np.ndarray
required. Teacher targets shaped (trace, time, 2): spike indicator and soma target coordinate.
raw_soma_mv np.ndarray | None
keyword-only, default=None. Optional unclipped teacher soma voltage in millivolts, shaped (trace, time); replaces the voltage reconstructed from the target.
soma_scale float
keyword-only, default=0.1. Scale of the soma target coordinate; millivolts are soma / soma_scale + soma_bias_mv.
soma_bias_mv float
keyword-only, default=-67.7. Millivolt offset of the soma target coordinate.
ignore_start int
keyword-only, default=0. Initial timesteps excluded from scoring.

Returns

result dict[str, Any]
voltage_sera_iron_mv2 (area), voltage_sera_iron_per_spike_mv2 (None without teacher spikes), teacher_spikes, and relevance_mean.

sera_skill_trace_terms

axobench.metrics.sera_skill_trace_terms(prediction: np.ndarray, target: np.ndarray, *, reference_mv: float, raw_soma_mv: np.ndarray | None=None, soma_scale: float=0.1, soma_bias_mv: float=-67.7, timestep_ms: float=1.0, ignore_start: int=0) -> dict[str, np.ndarray]

Source

Per-trace IRon SERA areas behind the SERA skill scores. Relevance is fitted once on the pooled scored teacher, as in iron_voltage_sera and iron_dynamics_sera, so sums over any subset of traces give that subset’s skills.

Parameters

prediction np.ndarray
required. Scored model outputs shaped (trace, time, 2).
target np.ndarray
required. Scored teacher targets of the same shape.
reference_mv float
keyword-only, required. Constant reference voltage in mV, usually sera_reference_voltage of the calibration traces.
raw_soma_mv np.ndarray | None
keyword-only, default=None. Optional unclipped teacher voltage (trace, time); replaces the voltage reconstructed from the target.
soma_scale float
keyword-only, default=0.1. Scale of the soma target coordinate; millivolts are soma / soma_scale + soma_bias_mv.
soma_bias_mv float
keyword-only, default=-67.7. Millivolt offset of the soma target coordinate.
timestep_ms float
keyword-only, default=1.0. Duration of one timestep in milliseconds.
ignore_start int
keyword-only, default=0. Initial timesteps excluded from scoring.

Returns

terms dict[str, np.ndarray]
Per trace: voltage (prediction area), voltage_reference (constant reference), voltage_ceiling (teacher clipped at -55 mV), dynamics and dynamics_reference (dV/dt areas of the prediction and of the constant, whose rate is zero), and teacher_spikes.

sera_skills

axobench.metrics.sera_skills(terms: dict[str, np.ndarray], indices: np.ndarray | None=None) -> dict[str, float | None]

Source

SERA skill scores from sera_skill_trace_terms, pooled over all or selected traces. Each skill is 1 - area / reference area; prediction and reference share relevance and spike count, so this equals 1 - SERA / SERA(reference).

Parameters

terms dict[str, np.ndarray]
required. Result of sera_skill_trace_terms.
indices np.ndarray | None
default=None. Trace indices to pool; None pools all, and repeated indices count repeatedly as in a bootstrap draw.

Returns

skills dict[str, float | None]
voltage_sera_skill, voltage_sera_ceiling_skill (the skill of the teacher clipped at -55 mV, the best prediction without action potentials), voltage_sera_skill_of_ceiling (their ratio), and dynamics_sera_skill; None where a reference area or the ceiling is zero.

subthreshold_voltage_error

axobench.metrics.subthreshold_voltage_error(prediction: np.ndarray, target: np.ndarray, *, predicted_spikes: np.ndarray, raw_soma_mv: np.ndarray | None=None, trace_ids: list[str] | None=None, ap_post_steps: np.ndarray | None=None, soma_scale: float=0.1, soma_bias_mv: float=-67.7, ignore_start: int=0, timestep_ms: float=1.0) -> dict[str, Any]

Source

Somatic voltage RMSE outside action-potential windows of teacher and predicted spikes, with both voltages clipped at -55 mV, and its skill against the per-trace mean clipped teacher voltage.

Parameters

prediction np.ndarray
required. Model outputs covering the whole trace, (trace, time, 2).
target np.ndarray
required. Teacher targets covering the whole trace.
predicted_spikes np.ndarray
keyword-only, required. Predicted spike events of the scored steps only, (trace, time - ignore_start).
raw_soma_mv np.ndarray | None
keyword-only, default=None. Optional unclipped teacher voltage covering the whole trace.
trace_ids list[str] | None
keyword-only, default=None. Sample IDs, one per trace; the text before the first ':' is the morphology or cell-type group.
ap_post_steps np.ndarray | None
keyword-only, default=None. Per-trace window ends; None computes them with ap_window_ends from the scored teacher steps.
soma_scale float
keyword-only, default=0.1. Scale of the soma target coordinate; millivolts are soma / soma_scale + soma_bias_mv.
soma_bias_mv float
keyword-only, default=-67.7. Millivolt offset of the soma target coordinate.
ignore_start int
keyword-only, default=0. Initial timesteps excluded from scoring.
timestep_ms float
keyword-only, default=1.0. Duration of one timestep in milliseconds.

Returns

result dict[str, Any]
subthreshold_rmse_mv, subthreshold_skill, null_subthreshold_rmse_mv, and retained_points.

subthreshold_voltage_point_terms

axobench.metrics.subthreshold_voltage_point_terms(prediction: np.ndarray, target: np.ndarray, *, predicted_spikes: np.ndarray, raw_soma_mv: np.ndarray | None=None, trace_ids: list[str] | None=None, ap_post_steps: np.ndarray | None=None, soma_scale: float=0.1, soma_bias_mv: float=-67.7, ignore_start: int=0, timestep_ms: float=1.0) -> dict[str, np.ndarray]

Source

Per-step terms of the subthreshold voltage error on the scored steps, for breaking the error down by time or frequency. Steps in the action-potential windows of teacher or predicted spikes are excluded and both voltages are clipped at -55 mV.

Parameters

prediction np.ndarray
required. Model outputs covering the whole trace, (trace, time, 2).
target np.ndarray
required. Teacher targets covering the whole trace.
predicted_spikes np.ndarray
keyword-only, required. Predicted spike events of the scored steps only, (trace, time - ignore_start).
raw_soma_mv np.ndarray | None
keyword-only, default=None. Optional unclipped teacher voltage covering the whole trace.
trace_ids list[str] | None
keyword-only, default=None. Sample IDs, one per trace; the text before the first ':' is the morphology or cell-type group.
ap_post_steps np.ndarray | None
keyword-only, default=None. Per-trace window ends; None computes them with ap_window_ends from the scored teacher steps.
soma_scale float
keyword-only, default=0.1. Scale of the soma target coordinate; millivolts are soma / soma_scale + soma_bias_mv.
soma_bias_mv float
keyword-only, default=-67.7. Millivolt offset of the soma target coordinate.
ignore_start int
keyword-only, default=0. Initial timesteps excluded from scoring.
timestep_ms float
keyword-only, default=1.0. Duration of one timestep in milliseconds.

Returns

terms dict[str, np.ndarray]
keep (boolean mask of retained steps), error (clipped prediction minus clipped teacher), and deviation (clipped teacher minus its per-trace mean over retained steps), each (trace, time - ignore_start) and zero where not kept.

subthreshold_voltage_trace_terms

axobench.metrics.subthreshold_voltage_trace_terms(prediction: np.ndarray, target: np.ndarray, **kwargs: Any) -> dict[str, np.ndarray]

Source

Per-trace sums behind subthreshold_voltage_error, so that subsets of traces can be pooled with fixed windows, as the paired bootstrap does. Keyword arguments are those of subthreshold_voltage_point_terms.

Parameters

prediction np.ndarray
required. Model outputs covering the whole trace, (trace, time, 2).
target np.ndarray
required. Teacher targets covering the whole trace.

Returns

terms dict[str, np.ndarray]
squared_error, null_squared_error, and retained_points per trace.

ap_window_ends

axobench.metrics.ap_window_ends(raw_soma_mv: np.ndarray | None, teacher_spikes: np.ndarray, *, trace_ids: list[str] | None=None, timestep_ms: float=1.0) -> dict[str, Any]

Source

Cell-type-specific end of the action-potential window. For each trace-ID prefix, the mean waveform of isolated teacher spikes (lags -6 to +30, no other spike in the next 30 steps) is computed, and the end is the first lag >= 1 at which it changes by less than 1 mV per ms. Groups with fewer than 20 isolated spikes, no flat lag, or no voltage use 4.

Parameters

raw_soma_mv np.ndarray | None
required. Unclipped teacher voltage (trace, time), or None to use the fallback for every group.
teacher_spikes np.ndarray
required. Teacher spike indicators, (trace, time).
trace_ids list[str] | None
keyword-only, default=None. Sample IDs, one per trace; the text before the first ':' is the morphology or cell-type group.
timestep_ms float
keyword-only, default=1.0. Duration of one timestep in milliseconds.

Returns

result dict[str, Any]
post_steps per trace, end_steps_by_cell_type, and fallback_cell_types.

ap_window_mask

axobench.metrics.ap_window_mask(spikes: np.ndarray, post_steps: np.ndarray | None=None) -> np.ndarray

Source

Boolean mask of the steps from 2 before to post_steps after each spike, inclusive.

Parameters

spikes np.ndarray
required. Spike indicators, (trace, time).
post_steps np.ndarray | None
default=None. Window end per trace; None uses AP_POST_STEPS.

Returns

mask np.ndarray
Boolean (trace, time) mask.

response_sera

axobench.metrics.response_sera(baseline_prediction: np.ndarray, intervention_prediction: np.ndarray, baseline_target: np.ndarray, intervention_target: np.ndarray, *, baseline_raw_soma_mv: np.ndarray | None=None, intervention_raw_soma_mv: np.ndarray | None=None, soma_scale: float=0.1, soma_bias_mv: float=-67.7, ignore_start: int=0) -> dict[str, Any]

Source

IRon-default Response SERA per teacher spike. The error is the predicted minus the teacher voltage change (intervention minus baseline); relevance is fitted on the pooled teacher change, and the area is divided by the teacher spikes of baseline and intervention traces.

Parameters

baseline_prediction np.ndarray
required. Model outputs for the baseline inputs, (pair, time, 2).
intervention_prediction np.ndarray
required. Model outputs for the edited inputs.
baseline_target np.ndarray
required. Teacher targets for the baseline inputs.
intervention_target np.ndarray
required. Teacher targets for the edited inputs.
baseline_raw_soma_mv np.ndarray | None
keyword-only, default=None. Optional unclipped baseline teacher voltage.
intervention_raw_soma_mv np.ndarray | None
keyword-only, default=None. Optional unclipped intervention teacher voltage.
soma_scale float
keyword-only, default=0.1. Scale of the soma target coordinate; millivolts are soma / soma_scale + soma_bias_mv.
soma_bias_mv float
keyword-only, default=-67.7. Millivolt offset of the soma target coordinate.
ignore_start int
keyword-only, default=0. Initial timesteps excluded from scoring.

Returns

result dict[str, Any]
response_sera_iron_mv2, response_sera_iron_per_spike_mv2, the zero-response control in both forms, teacher_spikes, response_rmse_mv, teacher_response_abs_mean_mv, and relevance_mean.

response_sera_trace_terms

axobench.metrics.response_sera_trace_terms(baseline_prediction: np.ndarray, intervention_prediction: np.ndarray, baseline_target: np.ndarray, intervention_target: np.ndarray, *, baseline_raw_soma_mv: np.ndarray | None=None, intervention_raw_soma_mv: np.ndarray | None=None, soma_scale: float=0.1, soma_bias_mv: float=-67.7, ignore_start: int=0) -> dict[str, np.ndarray]

Source

Per-pair terms of Response SERA with the relevance fitted once on all supplied pairs, so that subsets can be pooled with fixed relevance.

Parameters

baseline_prediction np.ndarray
required. Model outputs for the baseline inputs, (pair, time, 2).
intervention_prediction np.ndarray
required. Model outputs for the edited inputs.
baseline_target np.ndarray
required. Teacher targets for the baseline inputs.
intervention_target np.ndarray
required. Teacher targets for the edited inputs.
baseline_raw_soma_mv np.ndarray | None
keyword-only, default=None. Optional unclipped baseline teacher voltage.
intervention_raw_soma_mv np.ndarray | None
keyword-only, default=None. Optional unclipped intervention teacher voltage.
soma_scale float
keyword-only, default=0.1. Scale of the soma target coordinate; millivolts are soma / soma_scale + soma_bias_mv.
soma_bias_mv float
keyword-only, default=-67.7. Millivolt offset of the soma target coordinate.
ignore_start int
keyword-only, default=0. Initial timesteps excluded from scoring.

Returns

terms dict[str, np.ndarray]
area, zero_response_area, and teacher_spikes per pair, plus the error, teacher_delta, and weights arrays.

iron_dynamics_sera

axobench.metrics.iron_dynamics_sera(prediction: np.ndarray, target: np.ndarray, *, raw_soma_mv: np.ndarray | None=None, soma_scale: float=0.1, soma_bias_mv: float=-67.7, timestep_ms: float=1.0, ignore_start: int=0) -> dict[str, Any]

Source

IRon-default SERA of the one-step voltage change per teacher spike, with relevance fitted on the teacher’s rate of change. Requires at least two scored steps.

Parameters

prediction np.ndarray
required. Model outputs shaped (trace, time, 2).
target np.ndarray
required. Teacher targets shaped (trace, time, 2): spike indicator and soma target coordinate.
raw_soma_mv np.ndarray | None
keyword-only, default=None. Optional unclipped teacher soma voltage in millivolts, shaped (trace, time); replaces the voltage reconstructed from the target.
soma_scale float
keyword-only, default=0.1. Scale of the soma target coordinate; millivolts are soma / soma_scale + soma_bias_mv.
soma_bias_mv float
keyword-only, default=-67.7. Millivolt offset of the soma target coordinate.
timestep_ms float
keyword-only, default=1.0. Duration of one timestep in milliseconds.
ignore_start int
keyword-only, default=0. Initial timesteps excluded from scoring.

Returns

result dict[str, Any]
dynamics_sera_iron_mv2_per_ms2, dynamics_sera_iron_per_spike_mv2_per_ms2, teacher_spikes, and relevance_mean.

Constants

AP_PRE_STEPS = 2
Steps before a spike included in its action-potential window. Source
AP_POST_STEPS = 4
Fallback window end after a spike. Source
AP_FLAT_SLOPE_MV_PER_MS = 1.0
Slope below which the mean spike waveform counts as flat. Source
AP_WAVEFORM_PRE_STEPS = 6
Steps before a spike in the averaged waveform. Source
AP_ISOLATION_STEPS = 30
Steps after a spike that must be free of other spikes, and the waveform length after the spike. Source
AP_MIN_ISOLATED_SPIKES = 20
Isolated spikes required to estimate a cell type's window end. Source
SOMA_CLIP_MV = -55.0
Clipping level of the subthreshold RMSE and the NeuronIO target, in mV. Source

Search guides, examples, and API signatures.