AxoBench metrics

Benchmarking with AxoBench

Evaluate any prediction function under the AxoBench protocol, read the report, and compare models on paired traces.

Install AxoBench

AxoBench needs Python 3.10 or newer and depends only on NumPy, Matplotlib, and Requests; it does not import a model framework.

python -m pip install "axobench>=0.2.1"

These pages describe AxoBench 0.2.1. Since 0.2.0, it defines SERA as the IRon-default relevance area per teacher spike and replaces Dynamics SERA in the core set with the subthreshold voltage RMSE; earlier versions report different quantities under similar names. Record the installed version, for example with importlib.metadata.version("axobench"), alongside every result. A source checkout installs with python -m pip install -e .; the source repository currently requires access.

Supply a prediction function

The evaluator calls a function on batches of NumPy inputs and expects NumPy outputs of the same batch and time length:

Shape Content
Input float32[batch, time, 1278] Signed events, excitatory channels first
Output channel 0 [batch, time] Spike score; any score that increases with spike evidence, such as a logit
Output channel 1 [batch, time] Somatic voltage in the target coordinate (V + 67.7 mV) / 10

The evaluator converts channel 1 back to millivolts with V = 10 y − 67.7. Predictions above the −55 mV clipping level are allowed and scored. A model that needs morphology identity declares it with an attribute and receives one sample-ID prefix per trace:

import numpy as np

def predict(inputs: np.ndarray, *, morphology_ids) -> np.ndarray:
    # inputs: (batch, time, 1278); morphology_ids: e.g. ["specimen_479770916", ...]
    ...
    return outputs  # (batch, time, 2)

predict.supports_morphology_ids = True

Without the attribute, the evaluator calls predict(inputs). AxoSim checkpoints become prediction functions through make_checkpoint_predictor, which handles device placement, dtype, and the morphology vocabulary; see the AxoSim prediction adapter.

Evaluate a model

Assemble an evaluation root with val and, optionally, interventions as described in choose a dataset. Then:

from axobench.benchmark import (
    evaluate_model,
    format_evaluation_report,
    write_evaluation_report,
)

report = evaluate_model(
    predict,
    dataset_root="data/axobench",
    model_id="candidate-17",
    multiple_morphologies=True,
)
print(format_evaluation_report(report))
write_evaluation_report(report, "runs/candidate-17.json")

evaluate_model streams inputs in batches and retains only predictions, targets, unclipped voltage, and per-step excitatory and inhibitory event counts. Its defaults define the iteration profile:

Setting Default Effect
max_samples 120 Validation traces of the alphabetically first morphology, or of morphology_id. With multiple_morphologies=True, 24 from each of the five
max_intervention_samples 40 Pairs per intervention condition, divided among morphologies in multiple-morphology mode
max_interventions 8 Condition directories read from <root>/interventions, in sorted order
EvaluationConfig.ignore_start 500 Initial steps excluded from every metric
EvaluationConfig.calibration_fraction 0.25 Share of the selected traces used only to set the spike threshold
EvaluationConfig.f1_tolerances_steps (0, 1, 2, 3, 4, 5) Timing tolerances of the spike F1 curve
batch_size 128 Traces per prediction call

Trace-level variation differs between morphologies, so confirmation runs can raise max_samples and max_intervention_samples.

Calibration and scoring

The evaluator orders the selected traces round-robin by morphology and splits them in that order. The first quarter, here 30 traces, is the calibration block; the remaining 90 are scored. On the calibration block it sets the spike threshold to the score of the k-th highest timestep after ignore_start, where k is the number of teacher spike timesteps in the same block. Every metric is then computed on the scored traces only, with that fixed threshold. The action-potential windows of the subthreshold RMSE are computed from the teacher traces of the whole block, so they are identical for every model scored on it.

EvaluationConfig.default_spike_threshold does not set the threshold. It enters only the default_threshold_count diagnostic, which checks how many spikes an uncalibrated threshold would produce. axosim-evaluate-model sets it to 0, a logit of zero.

For model selection followed by an untouched confirmation, evaluate the selection block with validation_sample_start=0 and the confirmation block with validation_sample_start=120, using the same max_samples. In multiple-morphology mode the start indexes the balanced order, so both blocks remain balanced. The start is recorded in the report and in cache protocols, so paired comparisons cannot mix blocks.

Score stored predictions

evaluate_predictions applies the same protocol to arrays. Use it for predictions computed elsewhere or for a split other than val. Order the traces so that the calibration block contains every morphology. This example scores the first 24 test traces of each morphology:

import numpy as np
from axobench.data import ShardedNeuronIODataset
from axobench.benchmark import evaluate_predictions, format_evaluation_report

dataset = ShardedNeuronIODataset("data/axobench/test", cache_shards=1)
ids = dataset.get_sample_ids(range(len(dataset)))
groups = {}
for index, trace_id in enumerate(ids):
    groups.setdefault(trace_id.split(":", 1)[0], []).append(index)
selected = sorted(index for members in groups.values() for index in members[:24])

predictions, targets, soma_mv, event_counts, trace_ids = [], [], [], [], []
for start in range(0, len(selected), 32):
    batch = selected[start : start + 32]
    inputs, batch_targets = dataset.get_batch(batch)
    predictions.append(predict(inputs))
    targets.append(batch_targets)
    soma_mv.append(dataset.get_optional_array_batch("soma_mv", batch))
    event_counts.append(np.stack([(inputs > 0).sum(-1), (inputs < 0).sum(-1)], axis=-1))
    trace_ids += dataset.get_sample_ids(batch)

# Interleave morphologies so the calibration block contains each of them.
position, keys = {}, []
for trace_id in trace_ids:
    morphology = trace_id.split(":", 1)[0]
    keys.append((position.get(morphology, 0), morphology))
    position[morphology] = position.get(morphology, 0) + 1
order = sorted(range(len(trace_ids)), key=keys.__getitem__)

report = evaluate_predictions(
    np.concatenate(predictions)[order],
    np.concatenate(targets)[order],
    inputs=np.concatenate(event_counts)[order],
    raw_soma_mv=np.concatenate(soma_mv)[order],
    trace_ids=[trace_ids[index] for index in order],
    model_id="candidate-17",
)
print(format_evaluation_report(report))

Selecting indices in sorted order keeps each shard to one read. inputs takes the compact per-step event counts used by the input-conditioned diagnostics; full input arrays are not needed. Pass interventions= a list of PairedInterventionPredictions to include Response SERA.

Read the report

The terminal report has a header with the model, sample counts, calibrated threshold, morphologies, and runtime, followed by a fixed table of metrics, the action-potential window ends, and the 24-row attention panel.

Metric ID Unit Role
voltage_sera_iron_per_spike_mv2 mV²/spike Core: Voltage SERA
subthreshold_rmse_mv mV Core: subthreshold voltage RMSE
spike_mean_f1_0_5ms fraction Core: Mean F1 over 0–5 ms
spike_f1_area_0_5ms fraction Trapezoidal F1 area, a compatibility view
spike_f1_exact, spike_f1_5ms fraction F1 at 0 ms and 5 ms
response_sera_iron_per_spike_mv2 mV²/spike Core: Response SERA, pooled over conditions
subthreshold_skill skill Secondary: improvement over the per-trace mean voltage
dynamics_sera_iron_per_spike_mv2_per_ms2 mV²/ms²/spike Secondary: Dynamics SERA
voltage_sera_skill skill Secondary: Voltage SERA skill against a constant voltage
voltage_sera_ceiling_skill skill Secondary: skill of the teacher clipped at −55 mV, the AP-free ceiling
voltage_sera_skill_of_ceiling ratio Secondary: Voltage SERA skill as a fraction of the ceiling
dynamics_sera_skill skill Secondary: Dynamics SERA skill against the constant voltage

The returned dictionary also contains the full F1 curve with precision, recall, and event counts per tolerance, one record per intervention condition, every diagnostic with its trigger and basis, and the selection settings. write_evaluation_report writes it as strict JSON or as tidy CSV rows of type metric, diagnostic, and intervention, selected by the file suffix.

Compare with a baseline

Evaluate the baseline once, save its predictions, and compare every candidate against that cache:

from axobench.benchmark import (
    create_model_evaluation_cache,
    evaluate_model_against_cache,
    format_model_comparison,
    write_model_comparison,
    write_model_evaluation_cache,
)

cache = create_model_evaluation_cache(
    baseline_predict,
    dataset_root="data/axobench",
    model_id="baseline",
    multiple_morphologies=True,
)
write_model_evaluation_cache(cache, "runs/baseline-cache.npz")

comparison = evaluate_model_against_cache(
    predict,
    baseline_cache="runs/baseline-cache.npz",
    dataset_root="data/axobench",
    candidate_id="candidate-17",
    multiple_morphologies=True,
)
print(format_model_comparison(comparison))
write_model_comparison(comparison, "runs/candidate-17-vs-baseline.csv")

The cache is a compressed, pickle-free .npz with the baseline predictions, intervention predictions and targets, ordered trace IDs, a SHA-256 hash of the targets, the baseline report, and the protocol: sample limits, morphology selection, intervention names, sample start, and the complete EvaluationConfig. The comparison fails unless the candidate’s protocol, ordered trace IDs, and target hash all match. compare_predictions computes the same paired comparison from arrays that you have already aligned.

Each metric row reports the candidate value, the baseline value, their difference, a 95% interval, and an assessment. The interval comes from a paired bootstrap with 250 replicates and seed 20260711 by default:

  • Scored traces are resampled with replacement within each morphology, and both models are evaluated on the same resample.
  • Each model keeps its calibrated threshold, and the relevance weights and action-potential windows stay at their full-sample values.
  • Response SERA resamples pairs within each condition and morphology and pools conditions as summed relevance area per summed teacher spike count.
  • The interval spans the 2.5% and 97.5% quantiles of the replicate differences.

A metric is improved or regressed when the interval excludes zero in the preferred direction, which is lower for SERA and RMSE and higher for F1 and skill, and uncertain otherwise. The interval measures trace-sampling uncertainty for fixed predictions. It does not include variation between training runs, so compare several seeds when that is the question. The labels are evidence for a decision, not a promotion rule.

Use the command line

The axobench command converts and inspects data and writes diagnostic rows; it does not load models. For AxoSim checkpoints, axosim-evaluate-model wraps evaluate_model, create_model_evaluation_cache, and evaluate_model_against_cache; evaluate models shows the commands. For other frameworks, call the Python functions above with your prediction function.

Report a result

State the AxoBench version, dataset and split, sample limits and start, morphology mode, intervention conditions, ignore_start, calibration fraction, and the bootstrap settings. Report Voltage SERA, subthreshold RMSE, Response SERA, and Mean F1 with the tolerance grid. No single metric captures every biologically relevant error; the metric definitions explain what each one isolates.

Next steps

Read the metric definitions, explain a model’s errors with the error anatomy, inspect the attention panel and diagnostics, or look up exact signatures in the AxoBench API.

Search guides, examples, and API signatures.