AxoBench API
Paired comparison
Signatures, parameters, return values, and source for paired comparison.
Overview
A baseline cache freezes one model’s predictions together with the trace selection, target hash, and protocol. Candidates are compared against it on identical traces with a morphology-stratified paired bootstrap; see compare with a baseline.
Source revision: 496e0c1a2dfd. Export index.
ModelEvaluationCache
axobench.benchmark.iteration_evaluator.ModelEvaluationCache(model_id: str, report: dict[str, Any], prediction: np.ndarray, trace_ids: tuple[str, ...], target_sha256: str, interventions: tuple[PairedInterventionPredictions, ...], protocol: dict[str, Any], source_path: Path | None = None)
A frozen baseline: predictions, intervention predictions, trace selection, target hash, report, and protocol. Exported from axobench.benchmark.
Parameters
model_idstr- required. Baseline name.
reportdict[str, Any]- required. Baseline evaluation report.
predictionnp.ndarray- required. Baseline outputs for the selected traces, in report order.
trace_idstuple[str, ...]- required. Ordered sample IDs of the selected traces.
target_sha256str- required. SHA-256 of the target array, including dtype and shape.
interventionstuple[PairedInterventionPredictions, ...]- required. Baseline predictions and teacher traces per condition.
protocoldict[str, Any]- required. Sample limits, morphology selection, intervention names, sample start, and EvaluationConfig fields.
source_pathPath | None- default=None. File the cache was loaded from, if any.
Attributes
Constructor fields are retained as read-only attributes.
create_model_evaluation_cache
axobench.benchmark.iteration_evaluator.create_model_evaluation_cache(model_fn: Callable[[np.ndarray], np.ndarray], *, dataset_root: str | Path, model_id: str='baseline', max_samples: int | None=None, batch_size: int=DEFAULT_BATCH_SIZE, max_intervention_samples: int | None=None, max_interventions: int=8, intervention_workers: int=DEFAULT_INTERVENTION_WORKERS, intervention_names: Iterable[str] | None=None, multiple_morphologies: bool=False, morphology_id: str | None=None, validation_sample_start: int=0, cache_shards: int=1, config: EvaluationConfig | None=None) -> ModelEvaluationCache
Evaluate a baseline once and retain everything needed to compare later candidates on the same traces. Exported from axobench.benchmark.
Parameters
model_fnCallable[[np.ndarray], np.ndarray]- required. Prediction function mapping float32 inputs (batch, time, 1278) to (batch, time, 2) outputs: a spike score and the soma target coordinate. With supports_morphology_ids=True it is called with a morphology_ids keyword.
dataset_rootstr | Path- keyword-only, required. Evaluation root containing val/ and optionally interventions/<condition>/.
model_idstr- keyword-only, default='baseline'. Baseline name stored in the cache and report.
max_samplesint | None- keyword-only, default=None. Validation traces to select; None uses default_iteration_sample_limits (120).
batch_sizeint- keyword-only, default=DEFAULT_BATCH_SIZE = 128. Traces per prediction call.
max_intervention_samplesint | None- keyword-only, default=None. Pairs per intervention condition; None uses the default of 40.
max_interventionsint- keyword-only, default=8. Maximum number of condition directories read, in sorted order.
intervention_workersint- keyword-only, default=DEFAULT_INTERVENTION_WORKERS = 4. Threads that load intervention shards in parallel.
intervention_namesIterable[str] | None- keyword-only, default=None. Condition directory names to evaluate; None discovers all. A missing name raises FileNotFoundError.
multiple_morphologiesbool- keyword-only, default=False. Balance traces and pairs across all morphologies instead of using one.
morphology_idstr | None- keyword-only, default=None. Morphology for single-morphology mode; None uses the alphabetically first. Cannot be combined with multiple_morphologies=True.
validation_sample_startint- keyword-only, default=0. Start of the selected block within the morphology, or within the balanced order in multiple-morphology mode.
cache_shardsint- keyword-only, default=1. Decoded shards kept in memory by the reader.
configEvaluationConfig | None- keyword-only, default=None. Evaluation settings; None uses EvaluationConfig().
Returns
cacheModelEvaluationCache- Baseline predictions, report, trace IDs, target hash, interventions, and protocol.
write_model_evaluation_cache
axobench.benchmark.iteration_evaluator.write_model_evaluation_cache(cache: ModelEvaluationCache, path: str | Path) -> Path
Write a cache as a compressed, pickle-free NPZ file with JSON metadata. Exported from axobench.benchmark.
Parameters
cacheModelEvaluationCache- required. Cache to write.
pathstr | Path- required. Destination ending in .npz; parent directories are created.
Returns
pathPath- The written path.
load_model_evaluation_cache
axobench.benchmark.iteration_evaluator.load_model_evaluation_cache(path: str | Path) -> ModelEvaluationCache
Load a cache written by write_model_evaluation_cache, without pickle. Exported from axobench.benchmark.
Parameters
pathstr | Path- required. Cache file.
Returns
cacheModelEvaluationCache- The cache, with source_path set.
evaluate_model_against_cache
axobench.benchmark.iteration_evaluator.evaluate_model_against_cache(candidate_model_fn: Callable[[np.ndarray], np.ndarray], *, baseline_cache: ModelEvaluationCache | str | Path, dataset_root: str | Path, candidate_id: str='candidate', max_samples: int | None=None, batch_size: int=DEFAULT_BATCH_SIZE, max_intervention_samples: int | None=None, max_interventions: int=8, intervention_workers: int=DEFAULT_INTERVENTION_WORKERS, intervention_names: Iterable[str] | None=None, multiple_morphologies: bool=False, morphology_id: str | None=None, validation_sample_start: int=0, cache_shards: int=1, config: EvaluationConfig | None=None, bootstrap_replicates: int=250, bootstrap_seed: int=20260711) -> dict[str, Any]
Evaluate a candidate and compare it with a cached baseline. Raises ValueError unless the protocol, ordered trace IDs, and target hash match the cache. Exported from axobench.benchmark.
Parameters
candidate_model_fnCallable[[np.ndarray], np.ndarray]- required. Candidate prediction function, with the same contract as model_fn of evaluate_model.
baseline_cacheModelEvaluationCache | str | Path- keyword-only, required. Cache object or path to a cache file.
dataset_rootstr | Path- keyword-only, required. Evaluation root containing val/ and optionally interventions/<condition>/.
candidate_idstr- keyword-only, default='candidate'. Candidate name in the comparison.
max_samplesint | None- keyword-only, default=None. Validation traces to select; None uses default_iteration_sample_limits (120).
batch_sizeint- keyword-only, default=DEFAULT_BATCH_SIZE = 128. Traces per prediction call.
max_intervention_samplesint | None- keyword-only, default=None. Pairs per intervention condition; None uses the default of 40.
max_interventionsint- keyword-only, default=8. Maximum number of condition directories read, in sorted order.
intervention_workersint- keyword-only, default=DEFAULT_INTERVENTION_WORKERS = 4. Threads that load intervention shards in parallel.
intervention_namesIterable[str] | None- keyword-only, default=None. Condition directory names to evaluate; None discovers all. A missing name raises FileNotFoundError.
multiple_morphologiesbool- keyword-only, default=False. Balance traces and pairs across all morphologies instead of using one.
morphology_idstr | None- keyword-only, default=None. Morphology for single-morphology mode; None uses the alphabetically first. Cannot be combined with multiple_morphologies=True.
validation_sample_startint- keyword-only, default=0. Start of the selected block within the morphology, or within the balanced order in multiple-morphology mode.
cache_shardsint- keyword-only, default=1. Decoded shards kept in memory by the reader.
configEvaluationConfig | None- keyword-only, default=None. Evaluation settings; None uses EvaluationConfig().
bootstrap_replicatesint- keyword-only, default=250. Paired bootstrap replicates.
bootstrap_seedint- keyword-only, default=20260711. Seed of the paired bootstrap.
Returns
comparisondict[str, Any]- The compare_predictions result plus evaluation_seconds and baseline_cache, the cache path.
compare_predictions
axobench.benchmark.iteration_evaluator.compare_predictions(candidate_prediction: np.ndarray, baseline_prediction: np.ndarray, target: np.ndarray, *, inputs: np.ndarray | None=None, raw_soma_mv: np.ndarray | None=None, trace_ids: list[str] | None=None, candidate_interventions: Iterable[PairedInterventionPredictions]=(), baseline_interventions: Iterable[PairedInterventionPredictions]=(), config: EvaluationConfig | None=None, candidate_id: str='candidate', baseline_id: str='baseline', baseline_report: dict[str, Any] | None=None, bootstrap_replicates: int=250, seed: int=20260711) -> dict[str, Any]
Compare two models on the same traces. Scored traces are resampled within each morphology; each model keeps its calibrated threshold, and relevance weights and action-potential windows stay at their full-sample values. Intervals are the 2.5% and 97.5% quantiles of the replicate differences. Exported from axobench.benchmark.
Parameters
candidate_predictionnp.ndarray- required. Candidate outputs, (trace, time, 2).
baseline_predictionnp.ndarray- required. Baseline outputs for the same traces.
targetnp.ndarray- required. Teacher targets for the same traces.
inputsnp.ndarray | None- keyword-only, default=None. Optional per-step input summary, as for evaluate_predictions.
raw_soma_mvnp.ndarray | None- keyword-only, default=None. Optional unclipped teacher soma voltage in millivolts, shaped (trace, time); replaces the voltage reconstructed from the target.
trace_idslist[str] | None- keyword-only, default=None. Sample IDs, one per trace; the text before the first ':' is the morphology or cell-type group.
candidate_interventionsIterable[PairedInterventionPredictions]- keyword-only, default=(). Candidate paired predictions per condition.
baseline_interventionsIterable[PairedInterventionPredictions]- keyword-only, default=(). Baseline paired predictions in the same condition order and with identical teacher targets.
configEvaluationConfig | None- keyword-only, default=None. Evaluation settings; None uses EvaluationConfig().
candidate_idstr- keyword-only, default='candidate'. Candidate name.
baseline_idstr- keyword-only, default='baseline'. Baseline name; must equal baseline_report['model_id'] when a report is supplied.
baseline_reportdict[str, Any] | None- keyword-only, default=None. Optional cached baseline report; recomputed when its metric set or action-potential windows differ from the candidate's.
bootstrap_replicatesint- keyword-only, default=250. Paired bootstrap replicates; must be positive.
seedint- keyword-only, default=20260711. Seed of the paired bootstrap.
Returns
comparisondict[str, Any]- candidate and baseline reports, comparison rows (id, candidate, baseline, delta, ci95_low, ci95_high, unit, core, preferred_direction, assessment), bootstrap_replicates, bootstrap_seed, and confidence_level 0.95.
format_model_comparison
axobench.benchmark.iteration_evaluator.format_model_comparison(comparison: dict[str, Any]) -> str
Render the paired comparison table with candidate, baseline, difference, 95% interval, and assessment per metric. Exported from axobench.benchmark.
Parameters
comparisondict[str, Any]- required. Result of compare_predictions or evaluate_model_against_cache.
Returns
textstr- Plain-text table.
write_model_comparison
axobench.benchmark.iteration_evaluator.write_model_comparison(comparison: dict[str, Any], path: str | Path) -> Path
Write a complete JSON comparison or a tidy CSV of the paired differences, selected by the suffix. Exported from axobench.benchmark.
Parameters
comparisondict[str, Any]- required. Comparison to write.
pathstr | Path- required. Destination ending in .json or .csv.
Returns
pathPath- The written path.