Brain-Score

Brain-Score is a collection of benchmarks and models. The benchmarks consist of data and metrics that score any model on how brain-like it is. By following a unified BrainModel interface, all models can be treated as an experimental subject and tested on all benchmarks.

brainscore_vision.score(model_identifier: str, benchmark_identifier: str, conda_active: bool = False) → Score

Score the model referenced by the model_identifier on the benchmark referenced by the benchmark_identifier. The model needs to implement the BrainModel interface so that the benchmark can interact with it. The benchmark will be looked up from the benchmarks and evaluates the model (looked up from models) on how brain-like it is under that benchmark’s experimental paradigm, primate measurements, comparison metric, and ceiling. This results in a quantitative Score ranging from 0 (least brain-like) to 1 (most brain-like under this benchmark).

Parameters:

• model_identifier – the identifier for the model

• benchmark_identifier – the identifier for the benchmark to test the model against

Returns:

a Score of how brain-like the candidate model is under this benchmark. The score is normalized by this benchmark’s ceiling such that 1 means the model matches the data to ceiling level.

Contents:

• Examples
• Model Tutorial
  • Quickstart
  • Common Errors: Setup
  • Common Errors: Submission
  • Frequently Asked Questions
• Benchmark Tutorial
  • 1. Package stimuli and data
  • 2. Create the benchmark
  • 3. Submit the benchmark and iterate to finalize
• BrainModel interface
  • BrainModel
• Benchmarks
  • Benchmark
  • BenchmarkBase
  • ceil_score()
• Metrics
  • Ceiling
• Submission System
  • Submission System Components
  • What to do
  • Scoring Process Description
• Helpers and Utils
  • Defaults
  • PrecomputedFeatures
  • bound_score()
  • check_standard_format()
  • LazyLoad
  • combine_fields()
  • fullname()
  • map_fields()
  • recursive_dict_merge()
• API Reference