AxoBench metrics
Diagnostics
Read the 24-row attention panel, test metric sensitivity with controlled defects, and write teacher and model diagnostic rows.
Read the attention panel
Every evaluation report ends with the same 24 rows. A row reads YES when its value crosses the trigger, NO when it does not, and NA when the evidence is unavailable, for example without intervention pairs or with a single morphology. YES is an early watch that points at a failure mode to inspect; it is not a pass or fail decision. Each row’s basis field in the returned report states the reason for its trigger.
| Area | Row | Trigger | Basis |
|---|---|---|---|
| validity | nonfinite_outputs |
> 0 | Every emitted score and voltage must be finite |
| physiology | voltage_bounds |
> 0 | Fraction of predicted voltages outside −120 to +60 mV |
| physiology | voltage_slew |
> 0 | Fraction of one-step changes above 120 mV/ms |
| voltage | voltage_null_skill |
≤ 0 | 1 − MSE / MSE of the per-trace teacher mean; non-positive skill does not beat that null |
| voltage | voltage_bias |
> 0.10 SD | Mean offset in teacher standard deviations |
| voltage | voltage_gain |
outside 0.90–1.10 | Least-squares gain of predicted on teacher voltage |
| dynamics | residual_lag1 |
|r| > 0.10 | Lag-1 autocorrelation of the residual; above 0.10, more than 1% of residual variance is linearly predictable |
| states | low_input_state_error |
> 1.10× and > 0.5 mV | RMSE in the lowest 10% of input load, away from teacher spikes, relative to the overall RMSE |
| states | hyperpolarized_state_error |
> 1.10× and > 0.5 mV | Same ratio in the lowest 10% of teacher voltage |
| states | depolarized_state_error |
> 1.10× and > 0.5 mV | Same ratio in the highest 10% of teacher voltage |
| states | high_dvdt_error |
> 1.10× and > 0.5 mV | Same ratio in the highest 10% of teacher slope |
| states | near_spike_error |
> 1.10× and > 0.5 mV | Same ratio within 5 ms of teacher spikes |
| states | post_input_burst_error |
> 1.10× and > 0.5 mV | Same ratio in the 5 ms after the highest 10% of input load |
| spikes | spike_exact_fidelity |
< 0.90 | F1 at 0 ms |
| spikes | spike_5ms_fidelity |
< 0.95 | F1 at 5 ms |
| spikes | spike_timing_tolerance_gap |
> 0.05 | F1 gain from 0 to 5 ms tolerance |
| spikes | spike_count_error |
> 0.10 | Per-trace count error relative to the mean teacher count |
| spikes | spike_timing_bias |
|bias| > 0.5 ms | Median displacement of matched spikes |
| calibration | default_threshold_count |
outside 0.80–1.25 | Predicted-to-teacher count ratio at the uncalibrated default threshold |
| physiology | refractory_violations |
> 0 | Fraction of predicted inter-spike intervals below 2 ms |
| intervention | intervention_response_skill |
< 0.80 | Response skill of the worst condition against a zero-response prediction |
| intervention | intervention_direction |
< 0.95 | Sign agreement of the worst condition on teacher changes of at least 0.5 mV |
| intervention | intervention_gain |
outside 0.80–1.20 | Response gain of the condition furthest from 1 |
| transfer | morphology_error_disparity |
> 1.25× median | Worst morphology’s voltage RMSE relative to the median morphology |
A 1.10 RMSE ratio corresponds to at least 21% higher mean squared error in that state. The voltage and state rows use the unclipped teacher voltage over all scored steps. The count, timing-bias, default-threshold, and refractory rows count supra-threshold timesteps rather than the score-peak events of spike F1, so a spike score that stays above threshold for two steps registers there as a short interval. Intervention and transfer rows name the condition or morphology that triggered them in brackets.
Test metric sensitivity
A fidelity metric should worsen steadily as a known defect grows. axobench.benchmark.defect_injection corrupts teacher traces with one controlled defect at a time and scores the corrupted traces against the original teacher. Spike-timing defects move the action-potential waveform with the spike, so voltage and spike train stay consistent; voltage defects leave the spike train exact.
| Defect | Dose | Default doses |
|---|---|---|
spike_jitter |
Gaussian spike-time jitter, standard deviation in steps | 0, 0.5, 1, 2, 4, 8 |
spike_deletion |
Fraction of spikes deleted | 0, 0.1, 0.25, 0.5, 0.75, 1 |
spike_insertion |
Spurious spikes added, as a fraction of the spike count | 0, 0.1, 0.25, 0.5, 1, 2 |
global_lag |
Whole output delayed by this many steps | 0, 1, 2, 3, 5, 10 |
ap_clip |
Fraction of the voltage range above −55 mV removed by clipping; 1 is the NeuronIO target | 0, 0.25, 0.5, 0.75, 0.9, 1 |
ap_smoothing |
Gaussian smoothing of the action-potential window, sigma in steps | 0, 0.5, 1, 2, 4 |
voltage_lowpass |
Gaussian low-pass of the whole trace, sigma in steps | 0, 0.5, 1, 2, 4, 8 |
voltage_offset |
Constant voltage offset in mV | 0, 0.5, 1, 2, 4, 8 |
voltage_gain |
Fractional gain error around each trace’s mean | 0, 0.05, 0.1, 0.25, 0.5 |
post_spike |
Fraction by which the 50 steps after each action potential are replaced by a straight line | 0, 0.25, 0.5, 0.75, 1 |
from axobench.data import ShardedNeuronIODataset
from axobench.benchmark.defect_injection import defect_dose_rows
val = ShardedNeuronIODataset("data/axobench/val", cache_shards=1)
indices = range(16)
_, targets = val.get_batch(indices)
soma_mv = val.get_optional_array_batch("soma_mv", indices)
rows = defect_dose_rows(
targets[..., 0],
soma_mv,
doses={"spike_jitter": (0, 1, 2, 4), "voltage_offset": (0, 1, 2, 4)},
ignore_start=500,
)
for row in rows:
print(row["defect"], row["dose"], row["mean_f1"],
row["voltage_sera"], row["subthreshold_rmse_mv"], row["auc"])
Each row contains the AxoBench scores, Mean F1 with its tolerance curve, the coincidence factor at 2 ms, Voltage and Dynamics SERA with their skill against the per-trace mean, and the subthreshold RMSE and skill, next to the historical NeuronIO scores: timestep AUC of a smoothed spike train and the clipped voltage RMSE with and without rescaling. Some responses follow from the definitions: a constant offset leaves Mean F1 unchanged, and spike jitter leaves the subthreshold RMSE near zero because the moved action potentials fall inside excluded windows. The informative comparisons are the doses at which a score stops responding or responds to a defect it should ignore. The random defects use a seed derived from the defect name, so rows are reproducible.
Check the teacher
Teacher QC rows describe the dataset before any model is involved:
axobench benchmark-rows --mode teacher-qc \
--dataset-root data/axobench \
--max-samples 128 \
--morphology-id specimen_479770916 \
--output rows/teacher-qc.jsonl \
--summary-output rows/teacher-qc.md
The rows report the teacher voltage distribution, spike counts and rates, the input event fraction, and the distortion that clipping at −55 mV introduces, overall and near spikes. The command reads the first --max-samples traces of val in manifest order; --morphology-id labels the rows. --mode intervention writes the corresponding teacher response rows for one intervention condition.
axobench benchmark-v1 runs the whole teacher bundle: QC rows for val, routing rows for the five morphologies, intervention rows for every condition and morphology, and morphology-contrast rows across morphologies. It writes JSONL rows, Markdown summaries, and a manifest.json to --output-dir. --suite selects teacher-qc, paper-summary, or full.
Write model diagnostic rows
Beyond the bounded panel, evaluate_model_diagnostic_rows scores a model on supplied arrays and returns tidy rows for trace fidelity, state-conditioned dynamics, spike behavior, feature fidelity, and mechanistic response:
from axobench.benchmark import (
DiagnosticContext,
evaluate_model_diagnostic_rows,
write_diagnostic_rows,
)
indices = range(16)
inputs, targets = val.get_batch(indices)
rows = evaluate_model_diagnostic_rows(
predict,
inputs,
targets,
context=DiagnosticContext(
suites=("paper-summary",),
model_id="candidate-17",
morphology_id="specimen_479770916",
calibrated_spike_threshold=0.0,
),
trace_ids=val.get_sample_ids(indices),
raw_soma_mv=val.get_optional_array_batch("soma_mv", indices),
)
write_diagnostic_rows(rows, "rows/candidate-17.csv")
Pass a threshold calibrated on separate traces. run_v1_benchmark_bundle(model_fn=predict, ...) adds these model rows and model intervention rows to the teacher bundle. morphology_contrast_diagnostic_rows turns rows from several morphologies into mean, spread, and range across morphologies.
Every row has the columns diagnostic, suite, model_id, teacher_id, morphology_id, split, condition, intervention, trace_id, state, metric, value, unit, and n, plus the target view, ignore_start, and calibrated threshold. write_diagnostic_rows writes .json, .jsonl, or .csv, and write_diagnostic_summary writes a Markdown or JSON summary. The rows load directly into a dataframe.
Break errors down
The attention panel and the diagnostic rows flag where a model fails. The error anatomy, new in AxoBench 0.2.1, explains a model’s core-metric errors on the evaluator’s own scored traces: hits, misses, and false spikes with their timing errors; the spike-triggered voltage of teacher and model; subthreshold error by time since and until a teacher spike; and skill by firing rate, frequency band, and morphology. Run it on a saved evaluation cache with axobench error-anatomy.
Select diagnostic suites
A suite is a named set of diagnostics; a row is kept when its diagnostic belongs to one of the requested suites.
| Diagnostic | Suites | Content |
|---|---|---|
teacher_distribution_qc |
teacher-qc, paper-summary, full | Teacher voltage and spike distribution |
clipping_distortion |
teacher-qc, trace-core, paper-summary, full | Raw versus clipped target distortion |
morphology_routing_qc |
teacher-qc, paper-summary, full | Morphology and routing features |
trace_core |
trace-core, paper-summary, full | Trace fidelity, SERA, subthreshold RMSE, spike discrimination, count, and timing |
state_dynamics |
state-dynamics, paper-summary, full | State- and regime-conditioned errors |
spike_behavior |
spike-behavior, paper-summary, full | Spike calibration, counts, F1 timing, and threshold behavior |
feature_fidelity |
feature-fidelity, full | Electrophysiology-style features |
mechanistic_response |
mechanistic-response, paper-summary, full | Phase plane, event-triggered responses, history-conditioned error |
intervention_response |
teacher-qc, intervention-response, paper-summary, full | Paired response deltas |
morphology_contrast |
morphology-contrast, paper-summary, full | Differences across morphologies |
axobench dataset-schema --kind diagnostics prints the complete catalog, including audit diagnostics derived from saved comparison artifacts and the plotters attached to each diagnostic.
Plot diagnostics
axobench diagnostic-plots writes the audit figures and tables from saved audit summaries, as SVG, PDF, and PNG by default; the CLI reference lists its inputs. axobench error-anatomy writes the error-anatomy figure. AxoBench 0.2.1 draws all figures in one paper style, apply_paper_plot_style: STIX serif text at 7 to 8 pt, despined axes without grid, and the Okabe–Ito palette, sized for one or two columns. Set AXOBENCH_PAPER_STYLE to a Matplotlib style file to apply your own style first. The error-anatomy figure is also written as a standalone pgfplots .tex file for LaTeX documents.
Next steps
Explain a model’s errors with the error anatomy, return to benchmarking for the evaluation protocol, or see the diagnostics API for every argument.