AxoBench API
Model evaluation
Signatures, parameters, return values, and source for model evaluation.
Overview
The model-iteration evaluator selects validation traces and intervention pairs, calls a prediction function in batches, calibrates the spike threshold on a held-out block, and returns a fixed report of metrics and 24 attention rows. evaluate_predictions applies the same protocol to stored arrays. The guide Benchmarking with AxoBench describes the protocol.
Source revision: 496e0c1a2dfd. Export index.
EvaluationConfig
axobench.benchmark.iteration_evaluator.EvaluationConfig(soma_scale: float = 0.1, soma_bias_mv: float = -67.7, ignore_start: int = 500, timestep_ms: float = 1.0, calibration_fraction: float = 0.25, default_spike_threshold: float = 0.5, f1_tolerances_steps: tuple[int, ...] = (0, 1, 2, 3, 4, 5), selected_f1_steps: tuple[int, int] = (0, 5), physiological_voltage_bounds_mv: tuple[float, float] = (-120.0, 60.0), physiological_slew_bound_mv_per_ms: float = 120.0, refractory_period_ms: float = 2.0, state_error_ratio_watch: float = 1.1, state_error_floor_mv: float = 0.5)
Frozen settings of the metric protocol and attention thresholds. Its fields are part of every cache protocol, so paired comparisons require equal configurations. Exported from axobench.benchmark.
Parameters
soma_scalefloat- default=0.1. Scale of the soma target coordinate.
soma_bias_mvfloat- default=-67.7. Millivolt offset of the soma target coordinate.
ignore_startint- default=500. Initial timesteps excluded from every metric and diagnostic.
timestep_msfloat- default=1.0. Duration of one timestep in milliseconds.
calibration_fractionfloat- default=0.25. Share of traces, in (0, 1), used only to calibrate the spike threshold.
default_spike_thresholdfloat- default=0.5. Uncalibrated threshold checked by the default_threshold_count attention row; it does not set the scoring threshold.
f1_tolerances_stepstuple[int, ...]- default=(0, 1, 2, 3, 4, 5). Timing tolerances of the F1 curve in steps; Mean F1 averages over them.
selected_f1_stepstuple[int, int]- default=(0, 5). Tolerances reported as spike_f1_exact and spike_f1_5ms; both must be in f1_tolerances_steps.
physiological_voltage_bounds_mvtuple[float, float]- default=(-120.0, 60.0). Envelope of the voltage_bounds attention row.
physiological_slew_bound_mv_per_msfloat- default=120.0. One-step slope bound of the voltage_slew attention row.
refractory_period_msfloat- default=2.0. Minimum inter-spike interval of the refractory_violations attention row.
state_error_ratio_watchfloat- default=1.1. State-to-overall RMSE ratio that triggers a state attention row.
state_error_floor_mvfloat- default=0.5. Minimum state RMSE for a state attention row to trigger.
Attributes
Constructor fields are retained as read-only attributes.
PairedInterventionPredictions
axobench.benchmark.iteration_evaluator.PairedInterventionPredictions(name: str, baseline_prediction: np.ndarray, intervention_prediction: np.ndarray, baseline_target: np.ndarray, intervention_target: np.ndarray, baseline_raw_soma_mv: np.ndarray | None = None, intervention_raw_soma_mv: np.ndarray | None = None, morphology_ids: list[str] | np.ndarray | None = None)
Predictions and teacher traces for one intervention condition, used for Response SERA and the intervention attention rows. Exported from axobench.benchmark.
Parameters
namestr- required. Condition name reported with the results.
baseline_predictionnp.ndarray- required. Model outputs for the baseline inputs, (pair, time, 2).
intervention_predictionnp.ndarray- required. Model outputs for the edited inputs, (pair, time, 2).
baseline_targetnp.ndarray- required. Teacher targets for the baseline inputs.
intervention_targetnp.ndarray- required. Teacher targets for the edited inputs.
baseline_raw_soma_mvnp.ndarray | None- default=None. Optional unclipped baseline teacher voltage in millivolts.
intervention_raw_soma_mvnp.ndarray | None- default=None. Optional unclipped intervention teacher voltage in millivolts.
morphology_idslist[str] | np.ndarray | None- default=None. Optional morphology per pair, used to stratify the paired bootstrap.
Attributes
Constructor fields are retained as read-only attributes.
evaluate_model
axobench.benchmark.iteration_evaluator.evaluate_model(model_fn: Callable[[np.ndarray], np.ndarray], *, dataset_root: str | Path, model_id: str='model', max_samples: int | None=None, batch_size: int=DEFAULT_BATCH_SIZE, max_intervention_samples: int | None=None, max_interventions: int=8, intervention_workers: int=DEFAULT_INTERVENTION_WORKERS, intervention_names: Iterable[str] | None=None, multiple_morphologies: bool=False, morphology_id: str | None=None, validation_sample_start: int=0, cache_shards: int=1, config: EvaluationConfig | None=None) -> dict[str, Any]
Run batched inference on an evaluation root and return the evaluation report. Selected traces are reordered round-robin by morphology before calibration, so the calibration block covers every selected morphology. Exported from axobench.benchmark.
Parameters
model_fnCallable[[np.ndarray], np.ndarray]- required. Prediction function mapping float32 inputs (batch, time, 1278) to (batch, time, 2) outputs: a spike score and the soma target coordinate. With supports_morphology_ids=True it is called with a morphology_ids keyword.
dataset_rootstr | Path- keyword-only, required. Evaluation root containing val/ and optionally interventions/<condition>/.
model_idstr- keyword-only, default='model'. Model name in the report.
max_samplesint | None- keyword-only, default=None. Validation traces to select; None uses default_iteration_sample_limits (120).
batch_sizeint- keyword-only, default=DEFAULT_BATCH_SIZE = 128. Traces per prediction call.
max_intervention_samplesint | None- keyword-only, default=None. Pairs per intervention condition; None uses the default of 40.
max_interventionsint- keyword-only, default=8. Maximum number of condition directories read, in sorted order.
intervention_workersint- keyword-only, default=DEFAULT_INTERVENTION_WORKERS = 4. Threads that load intervention shards in parallel.
intervention_namesIterable[str] | None- keyword-only, default=None. Condition directory names to evaluate; None discovers all. A missing name raises FileNotFoundError.
multiple_morphologiesbool- keyword-only, default=False. Balance traces and pairs across all morphologies instead of using one.
morphology_idstr | None- keyword-only, default=None. Morphology for single-morphology mode; None uses the alphabetically first. Cannot be combined with multiple_morphologies=True.
validation_sample_startint- keyword-only, default=0. Start of the selected block within the morphology, or within the balanced order in multiple-morphology mode.
cache_shardsint- keyword-only, default=1. Decoded shards kept in memory by the reader.
configEvaluationConfig | None- keyword-only, default=None. Evaluation settings; None uses EvaluationConfig().
Returns
reportdict[str, Any]- The evaluate_predictions report plus evaluation_seconds, morphology_mode, morphology_ids, standard_sample_limit, intervention_sample_limit_per_condition, validation_sample_start, and sample_profile.
evaluate_predictions
axobench.benchmark.iteration_evaluator.evaluate_predictions(prediction: np.ndarray, target: np.ndarray, *, inputs: np.ndarray | None=None, raw_soma_mv: np.ndarray | None=None, trace_ids: list[str] | None=None, interventions: Iterable[PairedInterventionPredictions]=(), config: EvaluationConfig | None=None, model_id: str='model') -> dict[str, Any]
Evaluate stored predictions. The first round(calibration_fraction * traces) traces, at least one and at most all but one, calibrate the spike threshold by prevalence matching; all metrics and diagnostics use the remaining traces. At least two traces are required. Exported from axobench.benchmark.
Parameters
predictionnp.ndarray- required. Model outputs shaped (trace, time, 2), in calibration-then-scoring order.
targetnp.ndarray- required. Teacher targets in the same order.
inputsnp.ndarray | None- keyword-only, default=None. Optional per-step input summary shaped (trace, time, channels); its absolute sum per step is the input load of the low-input and post-burst diagnostics. Excitatory and inhibitory event counts suffice.
raw_soma_mvnp.ndarray | None- keyword-only, default=None. Optional unclipped teacher soma voltage in millivolts, shaped (trace, time); replaces the voltage reconstructed from the target.
trace_idslist[str] | None- keyword-only, default=None. Sample IDs, one per trace; the text before the first ':' is the morphology or cell-type group.
interventionsIterable[PairedInterventionPredictions]- keyword-only, default=(). Per-condition paired predictions for Response SERA and the intervention rows.
configEvaluationConfig | None- keyword-only, default=None. Evaluation settings; None uses EvaluationConfig().
model_idstr- keyword-only, default='model'. Model name in the report.
Returns
reportdict[str, Any]- model_id, samples, calibration_samples, scored_samples, spike_threshold, spike_f1_curve, subthreshold_ap_window, metrics (id, value, unit, meaning, core; seven core and six secondary rows, including the SERA skill rows), diagnostics (24 rows with area, id, attention, value, unit, primary, trigger, basis), and interventions.
calibrate_spike_threshold
axobench.benchmark.iteration_evaluator.calibrate_spike_threshold(prediction: np.ndarray, target: np.ndarray, config: EvaluationConfig) -> tuple[int, float]
Calibration trace count and spike threshold of evaluate_predictions. The first round(calibration_fraction * traces) traces, at least one and leaving at least one to score, set the threshold whose supra-threshold step count over their scored steps matches their teacher spike count.
Parameters
predictionnp.ndarray- required. Model outputs shaped (trace, time, 2), in calibration-then-scoring order.
targetnp.ndarray- required. Teacher targets in the same order.
configEvaluationConfig- required. Evaluation settings; calibration_fraction and ignore_start are used.
Returns
resulttuple[int, float]- Number of calibration traces and the calibrated threshold.
sera_reference_voltage
axobench.benchmark.iteration_evaluator.sera_reference_voltage(target: np.ndarray, raw_soma_mv: np.ndarray | None, n_calibration: int, config: EvaluationConfig) -> float
Constant reference voltage of the SERA skill scores: the mean teacher voltage clipped at -55 mV over the scored steps of the calibration traces, so no scored trace enters it.
Parameters
targetnp.ndarray- required. Teacher targets shaped (trace, time, 2), in calibration-then-scoring order.
raw_soma_mvnp.ndarray | None- required. Optional unclipped teacher voltage (trace, time); otherwise the voltage is reconstructed from the target.
n_calibrationint- required. Number of leading calibration traces.
configEvaluationConfig- required. Evaluation settings; ignore_start and the voltage coordinate are used.
Returns
reference_mvfloat- Reference voltage in mV.
default_iteration_sample_limits
axobench.benchmark.iteration_evaluator.default_iteration_sample_limits(*, multiple_morphologies: bool=False) -> tuple[int, int]
Default ordinary and per-condition intervention sample limits for a morphology mode. Exported from axobench.benchmark.
Parameters
multiple_morphologiesbool- keyword-only, default=False. Select the multiple-morphology defaults.
Returns
limitstuple[int, int]- (120, 40) in both modes.
format_evaluation_report
axobench.benchmark.iteration_evaluator.format_evaluation_report(report: dict[str, Any]) -> str
Render the bounded plain-text report: header, core and secondary metrics, action-potential window ends, and the attention panel with its summary line. Exported from axobench.benchmark.
Parameters
reportdict[str, Any]- required. Report returned by evaluate_model or evaluate_predictions.
Returns
textstr- Deterministic plain-text report.
write_evaluation_report
axobench.benchmark.iteration_evaluator.write_evaluation_report(report: dict[str, Any], path: str | Path) -> Path
Write a report as strict JSON or tidy CSV, selected by the suffix; non-finite values become null or empty. Exported from axobench.benchmark.
Parameters
reportdict[str, Any]- required. Report to write.
pathstr | Path- required. Destination ending in .json or .csv; parent directories are created.
Returns
pathPath- The written path.