coco_pipe.report.decoding_sweep#

Decoding sweep report: cross-result comparison engine + sweep orchestration.

Owns both the comparison primitives (ResultCollection, result loading, comparison figures, build_comparison_section(), build_result_tabs(), make_experiment_results_report()) and the sweep-report layer built on top of them: the scientific overview, leaderboards, failures and feature-selection sections, the taxonomy recipe builders (flat/sensor/grouped/descriptor), the shared make_decoding_sweep_report() skeleton, and the head-to-head comparison report.

Attributes#

`CLASSICAL_MODE_TITLES`
`DEFAULT_CLASSICAL_SECTION_BUILDERS`
`HEAD_TO_HEAD_DISPLAY_COLUMNS`

Classes#

ResultCollection

A labelled collection of decoding results and their tidy summaries.

Functions#

`collect_results`(items, *, by[, metrics])	Load labelled results and concatenate their scalar summaries.
`build_comparison_section`(collection, *, kind[, axis, ...])	Build one reusable cross-result comparison section.
`build_result_tabs`(collection, *[, sections, ...])	Build nested compact per-result sections from a collection.
`make_experiment_results_report`(items, *, by[, ...])	Build a comparison report over many labelled decoding results.
`collect_mode_results`(mode_frame, *[, context_columns])	Collect the successful results of a sweep result frame, without loading them.
`best_result_tabs`(frame, primary_metric, *[, ...])	Render diagnostics for only the top-N ranked sweep rows.
`hp_tuning_section`(frame, *[, primary_metric, ...])	Surface hyperparameter-search diagnostics for the top-ranked runs.
`summary_collection`(mode_frame[, primary_metric, ...])	Build a lightweight comparison collection from persisted sweep metrics.
`add_scientific_overview`(report, frame, *, ...[, ...])	Add high-level decoding-sweep coverage cards and strategy text.
`leaderboard_section`(frame, *, title, table_title, ...)	Build a primary leaderboard plus a compact model/bar comparison.
`failures_section`(frame)	Build a collapsible skipped/failed unit table.
`feature_selection_section`(frame, *, feature_metadata)	Render SFS / non-baseline feature-selection diagnostics.
`flat_section`(mode_frame, *, scope_label, ...)	Build comparison and diagnostics tabs for flat decoding units.
`sensor_section`(mode_frame, *, scope_label[, montage])	Build sensor-level comparison views and optional topomap.
`grouped_section`(mode_frame, *, scope_label, group_col, ...)	Build a grouped taxonomy comparison over an existing sweep frame.
`descriptor_section`(mode_frame, *, scope_label, ...[, ...])	Build descriptor-level summary comparisons plus top diagnostics.
`paired_delta_vs_baseline`(frame, *, group_columns, ...)	Compute paired metric deltas against the best baseline row per group.
`prepare_sweep_frame`(records, *[, add_cv_signature, ...])	Build the tidy sweep frame shared by every decoding sweep report.
`make_decoding_sweep_report`(records, *, title, kind, ...)	Assemble the skeleton every decoding sweep report shares.
`build_capability_matrix_section`(capability_records)	Build the foundation preflight capability matrix, or None when empty.
`build_foundation_comparison_sections`(report, frame, *)	Add foundation model/train-mode comparison figures + per-result tabs.
`build_classical_mode_elements`(mode_frame, table_title, ...)	Build the per-mode result table plus its taxonomy comparison element.
`build_classical_taxonomy_sections`(report, frame, *[, ...])	Render one taxonomy section per scope: the classical report body.
`collect_comparison_runs`(sources)	Read, filter and tag per-run result CSVs into one comparison frame.
`enrich_head_to_head_frame`(comparison_frame)	Add the derived columns a head-to-head comparison frame needs.
`normalize_head_to_head_frame`(frame, *[, display_columns])	Sort an enriched comparison frame and project the display columns.
`make_head_to_head_report`(comparison_frame, *, ...[, ...])	Build the head-to-head comparison report from a tagged comparison frame.
`make_decoding_report`(records, *[, title, ...])	Build the classical decoding sweep report.

Module Contents#

class coco_pipe.report.decoding_sweep.ResultCollection#

A labelled collection of decoding results and their tidy summaries.

Variables:

by – Ordered axis names that label the collection. The order is significant: the first axis is treated as the primary comparison axis by helpers that need a default (e.g. it seeds the default comparison axis in make_experiment_results_report()). results/contexts are keyed by tuples aligned to this order.
results – Result objects, or the paths they are saved at, keyed by the axis-value tuple. Use result() to get one loaded.
contexts – Per-result context metadata keyed by the same axis-value tuple.
summary – Concatenated tidy per-result summary rows.

by: tuple[str, ...]#

results: dict[tuple[Any, ...], Any]#

contexts: dict[tuple[Any, ...], dict[str, Any]]#

summary: pandas.DataFrame#

result(key)#

Return the result for key, loading it if only its path is held.

Parameters:: key (tuple[Any, ...])
Return type:: Any

filter(**criteria)#

Return a collection restricted to exact context matches.

Parameters:: criteria (Any)
Return type:: ResultCollection

property successful_summary: pandas.DataFrame#

Return only successfully loaded result summaries.

Return type:: pandas.DataFrame

coco_pipe.report.decoding_sweep.collect_results(items, *, by, metrics=None)#

Load labelled results and concatenate their scalar summaries.

Parameters:

items (collections.abc.Iterable[tuple[collections.abc.Mapping[str, Any], Any]])
by (collections.abc.Sequence[str])
metrics (collections.abc.Sequence[str] | None)

Return type:

ResultCollection

coco_pipe.report.decoding_sweep.build_comparison_section(collection, *, kind, axis=None, row=None, column='Model', value='accuracy_mean', group_by=(), center=None, info=None, coords=None, title=None, include_table=False, on_error='warn')#

Build one reusable cross-result comparison section.

Notes

Unlike build_decoding_sections(), on_error here accepts only "raise" or "warn" (not "placeholder"). A comparison renders many per-group figures into one section, so a single group’s failure is warned and skipped rather than replaced with a placeholder card.

Parameters:

collection (ResultCollection)
kind (str)
axis (str | None)
row (str | None)
column (str)
value (str)
group_by (collections.abc.Sequence[str])
center (float | None)
info (Any)
coords (Any)
title (str | None)
include_table (bool)
on_error (str)

Return type:

coco_pipe.report.core.Section | None

coco_pipe.report.decoding_sweep.build_result_tabs(collection, *, sections='compact', feature_metadata=None, info=None, coords=None, interactive=False, on_error='warn')#

Build nested compact per-result sections from a collection.

Parameters:

collection (ResultCollection)
sections (str | collections.abc.Sequence[str])
feature_metadata (pandas.DataFrame | None)
info (Any)
coords (Any)
interactive (bool)
on_error (str)

Return type:

coco_pipe.report.elements.TabsElement | coco_pipe.report.elements.AccordionElement | None

coco_pipe.report.decoding_sweep.make_experiment_results_report(items, *, by, comparisons='default', per_result='compact', nest=True, feature_metadata=None, info=None, coords=None, interactive=False, title='Decoding Comparison', config=None, asset_urls=None, qc_result=None, output_path=None, on_error='warn')#

Build a comparison report over many labelled decoding results.

Parameters:

comparisons (str | collections.abc.Sequence[collections.abc.Mapping[str, Any]]) – "default" (a single model-comparison view), "showcase" (model comparison plus per-axis heatmap, score spread, and metric matrix keyed on the first by axis), a single comparison kind string, or an explicit sequence of comparison specs.
interactive (bool) – If True, per-result diagnostic sections render interactive Plotly figures (cross-result comparison figures remain static Matplotlib images).
items (collections.abc.Iterable[tuple[collections.abc.Mapping[str, Any], Any]])
by (collections.abc.Sequence[str])
per_result (str | collections.abc.Sequence[str] | None)
nest (bool)
feature_metadata (pandas.DataFrame | None)
info (Any)
coords (Any)
title (str)
config (collections.abc.Mapping[str, Any] | None)
asset_urls (collections.abc.Mapping[str, str] | str | None)
qc_result (Any)
output_path (str | pathlib.Path | None)
on_error (str)

Return type:

coco_pipe.report.core.Report

coco_pipe.report.decoding_sweep.collect_mode_results(mode_frame, *, context_columns=DEFAULT_MODE_RESULT_CONTEXT_COLUMNS)#

Collect the successful results of a sweep result frame, without loading them.

Parameters:

mode_frame (pandas.DataFrame)
context_columns (collections.abc.Sequence[str])

Return type:

ResultCollection | None

coco_pipe.report.decoding_sweep.best_result_tabs(frame, primary_metric, *, group_columns=DEFAULT_BEST_RESULT_GROUP_COLUMNS, context_columns=DEFAULT_MODE_RESULT_CONTEXT_COLUMNS, feature_metadata=None, sections='compact', limit=MAX_INLINE_DIAGNOSTIC_RESULTS, on_error='placeholder')#

Render diagnostics for only the top-N ranked sweep rows.

Parameters:

frame (pandas.DataFrame)
primary_metric (str)
group_columns (collections.abc.Sequence[str])
context_columns (collections.abc.Sequence[str])
feature_metadata (pandas.DataFrame | None)
sections (str | collections.abc.Sequence[str])
limit (int)
on_error (str)

Return type:

coco_pipe.report.elements.TabsElement | coco_pipe.report.elements.AccordionElement | None

coco_pipe.report.decoding_sweep.hp_tuning_section(frame, *, primary_metric='primary_metric', feature_metadata=None, title='Hyperparameter Tuning', limit=MAX_INLINE_DIAGNOSTIC_RESULTS)#

Surface hyperparameter-search diagnostics for the top-ranked runs.

The classical and foundation reports both include this section so HP tuning is reported the same way regardless of model family. Loads the best runs’ artifacts and renders their search / best-parameter diagnostics; returns None when no run carries a hyperparameter search (on_error="warn" drops non-tuned runs instead of emitting placeholders).

Parameters:

frame (pandas.DataFrame)
primary_metric (str)
feature_metadata (pandas.DataFrame | None)
title (str)
limit (int)

Return type:

coco_pipe.report.core.Section | None

coco_pipe.report.decoding_sweep.summary_collection(mode_frame, primary_metric='primary_metric', *, descriptor_family=None, include_sensor=False, context_columns=DEFAULT_MODE_RESULT_CONTEXT_COLUMNS)#

Build a lightweight comparison collection from persisted sweep metrics.

Parameters:

mode_frame (pandas.DataFrame)
primary_metric (str)
descriptor_family (collections.abc.Mapping[str, str] | None)
include_sensor (bool)
context_columns (collections.abc.Sequence[str])

Return type:

ResultCollection | None

coco_pipe.report.decoding_sweep.add_scientific_overview(report, frame, *, dataset_name, kind, strategy_note, empty_message=None)#

Add high-level decoding-sweep coverage cards and strategy text.

Parameters:

report (coco_pipe.report.core.Report)
frame (pandas.DataFrame)
dataset_name (str)
kind (str)
strategy_note (str)
empty_message (str | None)

Return type:

None

coco_pipe.report.decoding_sweep.leaderboard_section(frame, *, title, table_title, filters, comparison_axis, group_by)#

Build a primary leaderboard plus a compact model/bar comparison.

Parameters:

frame (pandas.DataFrame)
title (str)
table_title (str)
filters (collections.abc.Mapping[str, Any])
comparison_axis (str)
group_by (collections.abc.Sequence[str])

Return type:

coco_pipe.report.core.Section | None

coco_pipe.report.decoding_sweep.failures_section(frame)#

Build a collapsible skipped/failed unit table.

Parameters:: frame (pandas.DataFrame)
Return type:: coco_pipe.report.core.Section | None

coco_pipe.report.decoding_sweep.feature_selection_section(frame, *, feature_metadata, baseline_label='baseline', selection_title='Feature-Selection Diagnostics')#

Render SFS / non-baseline feature-selection diagnostics.

Parameters:

frame (pandas.DataFrame)
feature_metadata (pandas.DataFrame | None)
baseline_label (str)
selection_title (str)

Return type:

coco_pipe.report.core.Section | None

coco_pipe.report.decoding_sweep.flat_section(mode_frame, *, scope_label, feature_metadata, kind, title_prefix, group_by, **axis_kwargs)#

Build comparison and diagnostics tabs for flat decoding units.

Parameters:

mode_frame (pandas.DataFrame)
scope_label (str)
feature_metadata (pandas.DataFrame | None)
kind (str)
title_prefix (str)
group_by (tuple[str, ...])
axis_kwargs (Any)

Return type:

Any

coco_pipe.report.decoding_sweep.sensor_section(mode_frame, *, scope_label, montage='colin27_1020', **kwargs)#

Build sensor-level comparison views and optional topomap.

Parameters:

mode_frame (pandas.DataFrame)
scope_label (str)
montage (str)
kwargs (Any)

Return type:

Any

coco_pipe.report.decoding_sweep.grouped_section(mode_frame, *, scope_label, group_col, kind, title_prefix, group_by, **axis_kwargs)#

Build a grouped taxonomy comparison over an existing sweep frame.

Parameters:

mode_frame (pandas.DataFrame)
scope_label (str)
group_col (str)
kind (str)
title_prefix (str)
group_by (tuple[str, ...])
axis_kwargs (Any)

Return type:

Any

coco_pipe.report.decoding_sweep.descriptor_section(mode_frame, *, scope_label, feature_metadata, kind, title_prefix, group_by, descriptor_family=None, include_sensor=False, **axis_kwargs)#

Build descriptor-level summary comparisons plus top diagnostics.

Parameters:

mode_frame (pandas.DataFrame)
scope_label (str)
feature_metadata (pandas.DataFrame | None)
kind (str)
title_prefix (str)
group_by (tuple[str, ...])
descriptor_family (collections.abc.Mapping[str, str] | None)
include_sensor (bool)
axis_kwargs (Any)

Return type:

Any

coco_pipe.report.decoding_sweep.paired_delta_vs_baseline(frame, *, group_columns, baseline_family, family_column='comparison_family', label_column='model_label', metric_column='primary_metric', pair_column='comparison_pair', delta_column='comparison_delta', compatible_lookup=None)#

Compute paired metric deltas against the best baseline row per group.

Parameters:

frame (pandas.DataFrame)
group_columns (collections.abc.Sequence[str])
baseline_family (str)
family_column (str)
label_column (str)
metric_column (str)
pair_column (str)
delta_column (str)
compatible_lookup (collections.abc.Mapping[tuple[Any, ...], bool] | None)

Return type:

pandas.DataFrame

coco_pipe.report.decoding_sweep.prepare_sweep_frame(records, *, add_cv_signature=True, scope_from='condition', default_scope='all')#

Build the tidy sweep frame shared by every decoding sweep report.

Concatenates records, stamps the primary metric, attaches a readable cv_signature when derivable, and guarantees a scope column: taken from scope_from when present, else filled with default_scope.

Parameters:

records (collections.abc.Iterable[collections.abc.Mapping[str, Any]])
add_cv_signature (bool)
scope_from (str | None)
default_scope (str | None)

Return type:

pandas.DataFrame

coco_pipe.report.decoding_sweep.make_decoding_sweep_report(records, *, title, kind, strategy_note, dataset_name='dataset', leaderboards=(), body=None, pre_sections=(), empty_message=None, frame=None, scope_from='condition', default_scope='all', add_cv_signature=True, config=None, asset_urls='inline', output_path=None)#

Assemble the skeleton every decoding sweep report shares.

Runs: frame prep -> scientific overview -> pre_sections (e.g. QC or a foundation capability matrix) -> each leaderboard_section(**spec) in leaderboards -> the domain body hook (only when the frame is non-empty) -> failures section -> optional save. The body callable owns the per-axis layout that differs between classical and foundation reports.

Parameters:

records (collections.abc.Iterable[collections.abc.Mapping[str, Any]])
title (str)
kind (str)
strategy_note (str)
dataset_name (str)
leaderboards (collections.abc.Sequence[collections.abc.Mapping[str, Any]])
body (collections.abc.Callable[[coco_pipe.report.core.Report, pandas.DataFrame], None] | None)
pre_sections (collections.abc.Iterable[coco_pipe.report.core.Section])
empty_message (str | None)
frame (pandas.DataFrame | None)
scope_from (str | None)
default_scope (str | None)
add_cv_signature (bool)
config (collections.abc.Mapping[str, Any] | None)
asset_urls (dict[str, str] | str | None)
output_path (str | pathlib.Path | None)

Return type:

coco_pipe.report.core.Report

coco_pipe.report.decoding_sweep.build_capability_matrix_section(capability_records)#

Build the foundation preflight capability matrix, or None when empty.

Parameters:: capability_records (collections.abc.Iterable[collections.abc.Mapping[str, Any]])
Return type:: coco_pipe.report.core.Section | None

coco_pipe.report.decoding_sweep.build_foundation_comparison_sections(report, frame, *, group_by=('condition', 'target'), per_result_sections='compact')#

Add foundation model/train-mode comparison figures + per-result tabs.

A body hook for make_decoding_sweep_report(): loads successful result artifacts, renders linear-probe bars, a training-mode heatmap, a metric matrix and a spread plot, then the coco-pipe per-result diagnostics tabs.

Parameters:

report (coco_pipe.report.core.Report)
frame (pandas.DataFrame)
group_by (tuple[str, ...])
per_result_sections (str | collections.abc.Sequence[str])

Return type:

None

coco_pipe.report.decoding_sweep.CLASSICAL_MODE_TITLES: dict[str, str]#

coco_pipe.report.decoding_sweep.DEFAULT_CLASSICAL_SECTION_BUILDERS: dict[str, collections.abc.Callable[..., Any]]#

coco_pipe.report.decoding_sweep.build_classical_mode_elements(mode_frame, table_title, analysis_modes, scope_label, feature_metadata, *, section_builders=None)#

Build the per-mode result table plus its taxonomy comparison element.

Parameters:

mode_frame (pandas.DataFrame)
table_title (str)
analysis_modes (tuple[str, ...])
scope_label (str)
feature_metadata (pandas.DataFrame | None)
section_builders (collections.abc.Mapping[str, collections.abc.Callable[..., Any]] | None)

Return type:

list[Any]

coco_pipe.report.decoding_sweep.build_classical_taxonomy_sections(report, frame, *, feature_metadata=None, section_builders=None, mode_titles=None, scope_order=None, scope_intro=None)#

Render one taxonomy section per scope: the classical report body.

A body hook for make_decoding_sweep_report(), mirroring build_foundation_comparison_sections(). For each scope it lays out the analysis-unit taxonomy from the broadest flat analysis down to the narrowest sensor/descriptor views, flags a missing flat baseline, and appends feature-selection diagnostics. Pass scope_order to control scope sequencing (e.g. a study’s configured conditions with a pooled scope last).

Parameters:

report (coco_pipe.report.core.Report)
frame (pandas.DataFrame)
feature_metadata (pandas.DataFrame | None)
section_builders (collections.abc.Mapping[str, collections.abc.Callable[..., Any]] | None)
mode_titles (collections.abc.Mapping[str, str] | None)
scope_order (collections.abc.Sequence[str] | None)
scope_intro (str | None)

Return type:

None

coco_pipe.report.decoding_sweep.collect_comparison_runs(sources)#

Read, filter and tag per-run result CSVs into one comparison frame.

Each entry in sources is a spec describing one family of runs to pull:

paths: Iterable of CSV paths (the caller owns the globbing policy).
filters: Mapping of column -> value equality filters applied to each CSV. The filter columns are also required: a CSV missing any of them is skipped rather than raising.
family: The comparison_family tag: either a string, or a callable (frame, path) -> str evaluated per surviving CSV.
space / source: Optional overrides for the comparison_space / source_run tags; callables of path. Default to path.parent.name and str(path.parent).

The read/filter/tag/concat boilerplate lives here; callers keep only their study-specific glob patterns, row filters and family policy. Empty or malformed CSVs are skipped. Returns an empty frame when nothing survives.

Parameters:: sources (collections.abc.Iterable[collections.abc.Mapping[str, Any]])
Return type:: pandas.DataFrame

coco_pipe.report.decoding_sweep.HEAD_TO_HEAD_DISPLAY_COLUMNS = ('comparison_family', 'comparison_space', 'source_run', 'scope', 'condition', 'target',...#

coco_pipe.report.decoding_sweep.enrich_head_to_head_frame(comparison_frame)#

Add the derived columns a head-to-head comparison frame needs.

Expects a concatenated frame already tagged with comparison_family (the caller’s globbing/tagging policy). Stamps the primary metric, attaches a cv_signature, fills scope from condition, and derives model_label/comparison_label/metric_name used by the table and the head-to-head plot.

Parameters:: comparison_frame (pandas.DataFrame)
Return type:: pandas.DataFrame

coco_pipe.report.decoding_sweep.normalize_head_to_head_frame(frame, *, display_columns=HEAD_TO_HEAD_DISPLAY_COLUMNS)#

Sort an enriched comparison frame and project the display columns.

The single source of truth for the head-to-head leaderboard layout: used both for the persisted comparison CSV and for the report table, so the two never drift. frame must already be enriched (see enrich_head_to_head_frame()).

Parameters:

frame (pandas.DataFrame)
display_columns (collections.abc.Sequence[str])

Return type:

pandas.DataFrame

coco_pipe.report.decoding_sweep.make_head_to_head_report(comparison_frame, *, baseline_family, group_columns=('scope', 'target'), title, table_title='Descriptors vs. Embeddings vs. Linear Probe', intro=None, paired_delta_title=None, display_columns=HEAD_TO_HEAD_DISPLAY_COLUMNS, config=None, asset_urls='inline', extra_sections=(), output_path=None, paired_test_permutations=1000)#

Build the head-to-head comparison report from a tagged comparison frame.

comparison_frame must carry comparison_family (baseline vs. each challenger) plus per-run metrics; the caller owns how it is assembled. The report shows a normalized leaderboard table, a cohort/CV compatibility panel, a head-to-head accuracy plot, and paired deltas versus baseline_family for groups whose CV/cohort signatures are compatible, each with a paired permutation p-value from the saved predictions.

Parameters:

comparison_frame (pandas.DataFrame)
baseline_family (str)
group_columns (collections.abc.Sequence[str])
title (str)
table_title (str)
intro (str | None)
paired_delta_title (str | None)
display_columns (collections.abc.Sequence[str])
config (collections.abc.Mapping[str, Any] | None)
asset_urls (dict[str, str] | str | None)
extra_sections (collections.abc.Sequence[coco_pipe.report.core.Section])
output_path (str | pathlib.Path | None)
paired_test_permutations (int)

Return type:

coco_pipe.report.core.Report

coco_pipe.report.decoding_sweep.make_decoding_report(records, *, title='Classical Decoding', dataset_name='dataset', strategy_note=None, leaderboards=None, body=None, pre_sections=(), feature_metadata=None, section_builders=None, scope_order=None, include_hp_tuning=True, frame=None, scope_from=None, default_scope='all', config=None, output_path=None, asset_urls='inline')#

Build the classical decoding sweep report.

The classical counterpart to make_foundation_decoding_report(): the same shared skeleton (scientific overview -> primary leaderboard -> body -> failures) with the analysis-unit taxonomy (flat/sensor/family/descriptor) as the default body. records are the per-unit classical decoding rows. Pass scope_order, section_builders or a custom body to override the study-specific layout; everything else falls back to the generic classical defaults.

Parameters:

records (collections.abc.Iterable[collections.abc.Mapping[str, Any]])
title (str)
dataset_name (str)
strategy_note (str | None)
leaderboards (collections.abc.Sequence[collections.abc.Mapping[str, Any]] | None)
body (collections.abc.Callable[[coco_pipe.report.core.Report, pandas.DataFrame], None] | None)
pre_sections (collections.abc.Iterable[coco_pipe.report.core.Section])
feature_metadata (pandas.DataFrame | None)
section_builders (collections.abc.Mapping[str, collections.abc.Callable[..., Any]] | None)
scope_order (collections.abc.Sequence[str] | None)
include_hp_tuning (bool)
frame (pandas.DataFrame | None)
scope_from (str | None)
default_scope (str | None)
config (collections.abc.Mapping[str, Any] | None)
output_path (str | pathlib.Path | None)
asset_urls (dict[str, str] | str | None)

Return type:

coco_pipe.report.core.Report