AxoBench API

Model evaluation

Signatures, parameters, return values, and source for model evaluation.

Overview

The model-iteration evaluator selects validation traces and intervention pairs, calls a prediction function in batches, calibrates the spike threshold on a held-out block, and returns a fixed report of metrics and 24 attention rows. evaluate_predictions applies the same protocol to stored arrays. The guide Benchmarking with AxoBench describes the protocol.

Source revision: 496e0c1a2dfd. Export index.

EvaluationConfig

axobench.benchmark.iteration_evaluator.EvaluationConfig(soma_scale: float = 0.1, soma_bias_mv: float = -67.7, ignore_start: int = 500, timestep_ms: float = 1.0, calibration_fraction: float = 0.25, default_spike_threshold: float = 0.5, f1_tolerances_steps: tuple[int, ...] = (0, 1, 2, 3, 4, 5), selected_f1_steps: tuple[int, int] = (0, 5), physiological_voltage_bounds_mv: tuple[float, float] = (-120.0, 60.0), physiological_slew_bound_mv_per_ms: float = 120.0, refractory_period_ms: float = 2.0, state_error_ratio_watch: float = 1.1, state_error_floor_mv: float = 0.5)

Source

Frozen settings of the metric protocol and attention thresholds. Its fields are part of every cache protocol, so paired comparisons require equal configurations. Exported from axobench.benchmark.

Parameters

soma_scale float
default=0.1. Scale of the soma target coordinate.
soma_bias_mv float
default=-67.7. Millivolt offset of the soma target coordinate.
ignore_start int
default=500. Initial timesteps excluded from every metric and diagnostic.
timestep_ms float
default=1.0. Duration of one timestep in milliseconds.
calibration_fraction float
default=0.25. Share of traces, in (0, 1), used only to calibrate the spike threshold.
default_spike_threshold float
default=0.5. Uncalibrated threshold checked by the default_threshold_count attention row; it does not set the scoring threshold.
f1_tolerances_steps tuple[int, ...]
default=(0, 1, 2, 3, 4, 5). Timing tolerances of the F1 curve in steps; Mean F1 averages over them.
selected_f1_steps tuple[int, int]
default=(0, 5). Tolerances reported as spike_f1_exact and spike_f1_5ms; both must be in f1_tolerances_steps.
physiological_voltage_bounds_mv tuple[float, float]
default=(-120.0, 60.0). Envelope of the voltage_bounds attention row.
physiological_slew_bound_mv_per_ms float
default=120.0. One-step slope bound of the voltage_slew attention row.
refractory_period_ms float
default=2.0. Minimum inter-spike interval of the refractory_violations attention row.
state_error_ratio_watch float
default=1.1. State-to-overall RMSE ratio that triggers a state attention row.
state_error_floor_mv float
default=0.5. Minimum state RMSE for a state attention row to trigger.

Attributes

Constructor fields are retained as read-only attributes.

PairedInterventionPredictions

axobench.benchmark.iteration_evaluator.PairedInterventionPredictions(name: str, baseline_prediction: np.ndarray, intervention_prediction: np.ndarray, baseline_target: np.ndarray, intervention_target: np.ndarray, baseline_raw_soma_mv: np.ndarray | None = None, intervention_raw_soma_mv: np.ndarray | None = None, morphology_ids: list[str] | np.ndarray | None = None)

Source

Predictions and teacher traces for one intervention condition, used for Response SERA and the intervention attention rows. Exported from axobench.benchmark.

Parameters

name str
required. Condition name reported with the results.
baseline_prediction np.ndarray
required. Model outputs for the baseline inputs, (pair, time, 2).
intervention_prediction np.ndarray
required. Model outputs for the edited inputs, (pair, time, 2).
baseline_target np.ndarray
required. Teacher targets for the baseline inputs.
intervention_target np.ndarray
required. Teacher targets for the edited inputs.
baseline_raw_soma_mv np.ndarray | None
default=None. Optional unclipped baseline teacher voltage in millivolts.
intervention_raw_soma_mv np.ndarray | None
default=None. Optional unclipped intervention teacher voltage in millivolts.
morphology_ids list[str] | np.ndarray | None
default=None. Optional morphology per pair, used to stratify the paired bootstrap.

Attributes

Constructor fields are retained as read-only attributes.

evaluate_model

axobench.benchmark.iteration_evaluator.evaluate_model(model_fn: Callable[[np.ndarray], np.ndarray], *, dataset_root: str | Path, model_id: str='model', max_samples: int | None=None, batch_size: int=DEFAULT_BATCH_SIZE, max_intervention_samples: int | None=None, max_interventions: int=8, intervention_workers: int=DEFAULT_INTERVENTION_WORKERS, intervention_names: Iterable[str] | None=None, multiple_morphologies: bool=False, morphology_id: str | None=None, validation_sample_start: int=0, cache_shards: int=1, config: EvaluationConfig | None=None) -> dict[str, Any]

Source

Run batched inference on an evaluation root and return the evaluation report. Selected traces are reordered round-robin by morphology before calibration, so the calibration block covers every selected morphology. Exported from axobench.benchmark.

Parameters

model_fn Callable[[np.ndarray], np.ndarray]
required. Prediction function mapping float32 inputs (batch, time, 1278) to (batch, time, 2) outputs: a spike score and the soma target coordinate. With supports_morphology_ids=True it is called with a morphology_ids keyword.
dataset_root str | Path
keyword-only, required. Evaluation root containing val/ and optionally interventions/<condition>/.
model_id str
keyword-only, default='model'. Model name in the report.
max_samples int | None
keyword-only, default=None. Validation traces to select; None uses default_iteration_sample_limits (120).
batch_size int
keyword-only, default=DEFAULT_BATCH_SIZE = 128. Traces per prediction call.
max_intervention_samples int | None
keyword-only, default=None. Pairs per intervention condition; None uses the default of 40.
max_interventions int
keyword-only, default=8. Maximum number of condition directories read, in sorted order.
intervention_workers int
keyword-only, default=DEFAULT_INTERVENTION_WORKERS = 4. Threads that load intervention shards in parallel.
intervention_names Iterable[str] | None
keyword-only, default=None. Condition directory names to evaluate; None discovers all. A missing name raises FileNotFoundError.
multiple_morphologies bool
keyword-only, default=False. Balance traces and pairs across all morphologies instead of using one.
morphology_id str | None
keyword-only, default=None. Morphology for single-morphology mode; None uses the alphabetically first. Cannot be combined with multiple_morphologies=True.
validation_sample_start int
keyword-only, default=0. Start of the selected block within the morphology, or within the balanced order in multiple-morphology mode.
cache_shards int
keyword-only, default=1. Decoded shards kept in memory by the reader.
config EvaluationConfig | None
keyword-only, default=None. Evaluation settings; None uses EvaluationConfig().

Returns

report dict[str, Any]
The evaluate_predictions report plus evaluation_seconds, morphology_mode, morphology_ids, standard_sample_limit, intervention_sample_limit_per_condition, validation_sample_start, and sample_profile.

evaluate_predictions

axobench.benchmark.iteration_evaluator.evaluate_predictions(prediction: np.ndarray, target: np.ndarray, *, inputs: np.ndarray | None=None, raw_soma_mv: np.ndarray | None=None, trace_ids: list[str] | None=None, interventions: Iterable[PairedInterventionPredictions]=(), config: EvaluationConfig | None=None, model_id: str='model') -> dict[str, Any]

Source

Evaluate stored predictions. The first round(calibration_fraction * traces) traces, at least one and at most all but one, calibrate the spike threshold by prevalence matching; all metrics and diagnostics use the remaining traces. At least two traces are required. Exported from axobench.benchmark.

Parameters

prediction np.ndarray
required. Model outputs shaped (trace, time, 2), in calibration-then-scoring order.
target np.ndarray
required. Teacher targets in the same order.
inputs np.ndarray | None
keyword-only, default=None. Optional per-step input summary shaped (trace, time, channels); its absolute sum per step is the input load of the low-input and post-burst diagnostics. Excitatory and inhibitory event counts suffice.
raw_soma_mv np.ndarray | None
keyword-only, default=None. Optional unclipped teacher soma voltage in millivolts, shaped (trace, time); replaces the voltage reconstructed from the target.
trace_ids list[str] | None
keyword-only, default=None. Sample IDs, one per trace; the text before the first ':' is the morphology or cell-type group.
interventions Iterable[PairedInterventionPredictions]
keyword-only, default=(). Per-condition paired predictions for Response SERA and the intervention rows.
config EvaluationConfig | None
keyword-only, default=None. Evaluation settings; None uses EvaluationConfig().
model_id str
keyword-only, default='model'. Model name in the report.

Returns

report dict[str, Any]
model_id, samples, calibration_samples, scored_samples, spike_threshold, spike_f1_curve, subthreshold_ap_window, metrics (id, value, unit, meaning, core; seven core and six secondary rows, including the SERA skill rows), diagnostics (24 rows with area, id, attention, value, unit, primary, trigger, basis), and interventions.

calibrate_spike_threshold

axobench.benchmark.iteration_evaluator.calibrate_spike_threshold(prediction: np.ndarray, target: np.ndarray, config: EvaluationConfig) -> tuple[int, float]

Source

Calibration trace count and spike threshold of evaluate_predictions. The first round(calibration_fraction * traces) traces, at least one and leaving at least one to score, set the threshold whose supra-threshold step count over their scored steps matches their teacher spike count.

Parameters

prediction np.ndarray
required. Model outputs shaped (trace, time, 2), in calibration-then-scoring order.
target np.ndarray
required. Teacher targets in the same order.
config EvaluationConfig
required. Evaluation settings; calibration_fraction and ignore_start are used.

Returns

result tuple[int, float]
Number of calibration traces and the calibrated threshold.

sera_reference_voltage

axobench.benchmark.iteration_evaluator.sera_reference_voltage(target: np.ndarray, raw_soma_mv: np.ndarray | None, n_calibration: int, config: EvaluationConfig) -> float

Source

Constant reference voltage of the SERA skill scores: the mean teacher voltage clipped at -55 mV over the scored steps of the calibration traces, so no scored trace enters it.

Parameters

target np.ndarray
required. Teacher targets shaped (trace, time, 2), in calibration-then-scoring order.
raw_soma_mv np.ndarray | None
required. Optional unclipped teacher voltage (trace, time); otherwise the voltage is reconstructed from the target.
n_calibration int
required. Number of leading calibration traces.
config EvaluationConfig
required. Evaluation settings; ignore_start and the voltage coordinate are used.

Returns

reference_mv float
Reference voltage in mV.

default_iteration_sample_limits

axobench.benchmark.iteration_evaluator.default_iteration_sample_limits(*, multiple_morphologies: bool=False) -> tuple[int, int]

Source

Default ordinary and per-condition intervention sample limits for a morphology mode. Exported from axobench.benchmark.

Parameters

multiple_morphologies bool
keyword-only, default=False. Select the multiple-morphology defaults.

Returns

limits tuple[int, int]
(120, 40) in both modes.

format_evaluation_report

axobench.benchmark.iteration_evaluator.format_evaluation_report(report: dict[str, Any]) -> str

Source

Render the bounded plain-text report: header, core and secondary metrics, action-potential window ends, and the attention panel with its summary line. Exported from axobench.benchmark.

Parameters

report dict[str, Any]
required. Report returned by evaluate_model or evaluate_predictions.

Returns

text str
Deterministic plain-text report.

write_evaluation_report

axobench.benchmark.iteration_evaluator.write_evaluation_report(report: dict[str, Any], path: str | Path) -> Path

Source

Write a report as strict JSON or tidy CSV, selected by the suffix; non-finite values become null or empty. Exported from axobench.benchmark.

Parameters

report dict[str, Any]
required. Report to write.
path str | Path
required. Destination ending in .json or .csv; parent directories are created.

Returns

path Path
The written path.

Search guides, examples, and API signatures.