AxoBench API

Paired comparison

Signatures, parameters, return values, and source for paired comparison.

Overview

A baseline cache freezes one model’s predictions together with the trace selection, target hash, and protocol. Candidates are compared against it on identical traces with a morphology-stratified paired bootstrap; see compare with a baseline.

Source revision: 496e0c1a2dfd. Export index.

ModelEvaluationCache

axobench.benchmark.iteration_evaluator.ModelEvaluationCache(model_id: str, report: dict[str, Any], prediction: np.ndarray, trace_ids: tuple[str, ...], target_sha256: str, interventions: tuple[PairedInterventionPredictions, ...], protocol: dict[str, Any], source_path: Path | None = None)

Source

A frozen baseline: predictions, intervention predictions, trace selection, target hash, report, and protocol. Exported from axobench.benchmark.

Parameters

model_id str
required. Baseline name.
report dict[str, Any]
required. Baseline evaluation report.
prediction np.ndarray
required. Baseline outputs for the selected traces, in report order.
trace_ids tuple[str, ...]
required. Ordered sample IDs of the selected traces.
target_sha256 str
required. SHA-256 of the target array, including dtype and shape.
interventions tuple[PairedInterventionPredictions, ...]
required. Baseline predictions and teacher traces per condition.
protocol dict[str, Any]
required. Sample limits, morphology selection, intervention names, sample start, and EvaluationConfig fields.
source_path Path | None
default=None. File the cache was loaded from, if any.

Attributes

Constructor fields are retained as read-only attributes.

create_model_evaluation_cache

axobench.benchmark.iteration_evaluator.create_model_evaluation_cache(model_fn: Callable[[np.ndarray], np.ndarray], *, dataset_root: str | Path, model_id: str='baseline', max_samples: int | None=None, batch_size: int=DEFAULT_BATCH_SIZE, max_intervention_samples: int | None=None, max_interventions: int=8, intervention_workers: int=DEFAULT_INTERVENTION_WORKERS, intervention_names: Iterable[str] | None=None, multiple_morphologies: bool=False, morphology_id: str | None=None, validation_sample_start: int=0, cache_shards: int=1, config: EvaluationConfig | None=None) -> ModelEvaluationCache

Source

Evaluate a baseline once and retain everything needed to compare later candidates on the same traces. Exported from axobench.benchmark.

Parameters

model_fn Callable[[np.ndarray], np.ndarray]
required. Prediction function mapping float32 inputs (batch, time, 1278) to (batch, time, 2) outputs: a spike score and the soma target coordinate. With supports_morphology_ids=True it is called with a morphology_ids keyword.
dataset_root str | Path
keyword-only, required. Evaluation root containing val/ and optionally interventions/<condition>/.
model_id str
keyword-only, default='baseline'. Baseline name stored in the cache and report.
max_samples int | None
keyword-only, default=None. Validation traces to select; None uses default_iteration_sample_limits (120).
batch_size int
keyword-only, default=DEFAULT_BATCH_SIZE = 128. Traces per prediction call.
max_intervention_samples int | None
keyword-only, default=None. Pairs per intervention condition; None uses the default of 40.
max_interventions int
keyword-only, default=8. Maximum number of condition directories read, in sorted order.
intervention_workers int
keyword-only, default=DEFAULT_INTERVENTION_WORKERS = 4. Threads that load intervention shards in parallel.
intervention_names Iterable[str] | None
keyword-only, default=None. Condition directory names to evaluate; None discovers all. A missing name raises FileNotFoundError.
multiple_morphologies bool
keyword-only, default=False. Balance traces and pairs across all morphologies instead of using one.
morphology_id str | None
keyword-only, default=None. Morphology for single-morphology mode; None uses the alphabetically first. Cannot be combined with multiple_morphologies=True.
validation_sample_start int
keyword-only, default=0. Start of the selected block within the morphology, or within the balanced order in multiple-morphology mode.
cache_shards int
keyword-only, default=1. Decoded shards kept in memory by the reader.
config EvaluationConfig | None
keyword-only, default=None. Evaluation settings; None uses EvaluationConfig().

Returns

cache ModelEvaluationCache
Baseline predictions, report, trace IDs, target hash, interventions, and protocol.

write_model_evaluation_cache

axobench.benchmark.iteration_evaluator.write_model_evaluation_cache(cache: ModelEvaluationCache, path: str | Path) -> Path

Source

Write a cache as a compressed, pickle-free NPZ file with JSON metadata. Exported from axobench.benchmark.

Parameters

cache ModelEvaluationCache
required. Cache to write.
path str | Path
required. Destination ending in .npz; parent directories are created.

Returns

path Path
The written path.

load_model_evaluation_cache

axobench.benchmark.iteration_evaluator.load_model_evaluation_cache(path: str | Path) -> ModelEvaluationCache

Source

Load a cache written by write_model_evaluation_cache, without pickle. Exported from axobench.benchmark.

Parameters

path str | Path
required. Cache file.

Returns

cache ModelEvaluationCache
The cache, with source_path set.

evaluate_model_against_cache

axobench.benchmark.iteration_evaluator.evaluate_model_against_cache(candidate_model_fn: Callable[[np.ndarray], np.ndarray], *, baseline_cache: ModelEvaluationCache | str | Path, dataset_root: str | Path, candidate_id: str='candidate', max_samples: int | None=None, batch_size: int=DEFAULT_BATCH_SIZE, max_intervention_samples: int | None=None, max_interventions: int=8, intervention_workers: int=DEFAULT_INTERVENTION_WORKERS, intervention_names: Iterable[str] | None=None, multiple_morphologies: bool=False, morphology_id: str | None=None, validation_sample_start: int=0, cache_shards: int=1, config: EvaluationConfig | None=None, bootstrap_replicates: int=250, bootstrap_seed: int=20260711) -> dict[str, Any]

Source

Evaluate a candidate and compare it with a cached baseline. Raises ValueError unless the protocol, ordered trace IDs, and target hash match the cache. Exported from axobench.benchmark.

Parameters

candidate_model_fn Callable[[np.ndarray], np.ndarray]
required. Candidate prediction function, with the same contract as model_fn of evaluate_model.
baseline_cache ModelEvaluationCache | str | Path
keyword-only, required. Cache object or path to a cache file.
dataset_root str | Path
keyword-only, required. Evaluation root containing val/ and optionally interventions/<condition>/.
candidate_id str
keyword-only, default='candidate'. Candidate name in the comparison.
max_samples int | None
keyword-only, default=None. Validation traces to select; None uses default_iteration_sample_limits (120).
batch_size int
keyword-only, default=DEFAULT_BATCH_SIZE = 128. Traces per prediction call.
max_intervention_samples int | None
keyword-only, default=None. Pairs per intervention condition; None uses the default of 40.
max_interventions int
keyword-only, default=8. Maximum number of condition directories read, in sorted order.
intervention_workers int
keyword-only, default=DEFAULT_INTERVENTION_WORKERS = 4. Threads that load intervention shards in parallel.
intervention_names Iterable[str] | None
keyword-only, default=None. Condition directory names to evaluate; None discovers all. A missing name raises FileNotFoundError.
multiple_morphologies bool
keyword-only, default=False. Balance traces and pairs across all morphologies instead of using one.
morphology_id str | None
keyword-only, default=None. Morphology for single-morphology mode; None uses the alphabetically first. Cannot be combined with multiple_morphologies=True.
validation_sample_start int
keyword-only, default=0. Start of the selected block within the morphology, or within the balanced order in multiple-morphology mode.
cache_shards int
keyword-only, default=1. Decoded shards kept in memory by the reader.
config EvaluationConfig | None
keyword-only, default=None. Evaluation settings; None uses EvaluationConfig().
bootstrap_replicates int
keyword-only, default=250. Paired bootstrap replicates.
bootstrap_seed int
keyword-only, default=20260711. Seed of the paired bootstrap.

Returns

comparison dict[str, Any]
The compare_predictions result plus evaluation_seconds and baseline_cache, the cache path.

compare_predictions

axobench.benchmark.iteration_evaluator.compare_predictions(candidate_prediction: np.ndarray, baseline_prediction: np.ndarray, target: np.ndarray, *, inputs: np.ndarray | None=None, raw_soma_mv: np.ndarray | None=None, trace_ids: list[str] | None=None, candidate_interventions: Iterable[PairedInterventionPredictions]=(), baseline_interventions: Iterable[PairedInterventionPredictions]=(), config: EvaluationConfig | None=None, candidate_id: str='candidate', baseline_id: str='baseline', baseline_report: dict[str, Any] | None=None, bootstrap_replicates: int=250, seed: int=20260711) -> dict[str, Any]

Source

Compare two models on the same traces. Scored traces are resampled within each morphology; each model keeps its calibrated threshold, and relevance weights and action-potential windows stay at their full-sample values. Intervals are the 2.5% and 97.5% quantiles of the replicate differences. Exported from axobench.benchmark.

Parameters

candidate_prediction np.ndarray
required. Candidate outputs, (trace, time, 2).
baseline_prediction np.ndarray
required. Baseline outputs for the same traces.
target np.ndarray
required. Teacher targets for the same traces.
inputs np.ndarray | None
keyword-only, default=None. Optional per-step input summary, as for evaluate_predictions.
raw_soma_mv np.ndarray | None
keyword-only, default=None. Optional unclipped teacher soma voltage in millivolts, shaped (trace, time); replaces the voltage reconstructed from the target.
trace_ids list[str] | None
keyword-only, default=None. Sample IDs, one per trace; the text before the first ':' is the morphology or cell-type group.
candidate_interventions Iterable[PairedInterventionPredictions]
keyword-only, default=(). Candidate paired predictions per condition.
baseline_interventions Iterable[PairedInterventionPredictions]
keyword-only, default=(). Baseline paired predictions in the same condition order and with identical teacher targets.
config EvaluationConfig | None
keyword-only, default=None. Evaluation settings; None uses EvaluationConfig().
candidate_id str
keyword-only, default='candidate'. Candidate name.
baseline_id str
keyword-only, default='baseline'. Baseline name; must equal baseline_report['model_id'] when a report is supplied.
baseline_report dict[str, Any] | None
keyword-only, default=None. Optional cached baseline report; recomputed when its metric set or action-potential windows differ from the candidate's.
bootstrap_replicates int
keyword-only, default=250. Paired bootstrap replicates; must be positive.
seed int
keyword-only, default=20260711. Seed of the paired bootstrap.

Returns

comparison dict[str, Any]
candidate and baseline reports, comparison rows (id, candidate, baseline, delta, ci95_low, ci95_high, unit, core, preferred_direction, assessment), bootstrap_replicates, bootstrap_seed, and confidence_level 0.95.

format_model_comparison

axobench.benchmark.iteration_evaluator.format_model_comparison(comparison: dict[str, Any]) -> str

Source

Render the paired comparison table with candidate, baseline, difference, 95% interval, and assessment per metric. Exported from axobench.benchmark.

Parameters

comparison dict[str, Any]
required. Result of compare_predictions or evaluate_model_against_cache.

Returns

text str
Plain-text table.

write_model_comparison

axobench.benchmark.iteration_evaluator.write_model_comparison(comparison: dict[str, Any], path: str | Path) -> Path

Source

Write a complete JSON comparison or a tidy CSV of the paired differences, selected by the suffix. Exported from axobench.benchmark.

Parameters

comparison dict[str, Any]
required. Comparison to write.
path str | Path
required. Destination ending in .json or .csv.

Returns

path Path
The written path.

Search guides, examples, and API signatures.