coco_pipe.report.decoding_sweep =============================== .. py:module:: coco_pipe.report.decoding_sweep .. autoapi-nested-parse:: Decoding sweep report: cross-result comparison engine + sweep orchestration. Owns both the comparison primitives (:class:`ResultCollection`, result loading, comparison figures, :func:`build_comparison_section`, :func:`build_result_tabs`, :func:`make_experiment_results_report`) and the sweep-report layer built on top of them: the scientific overview, leaderboards, failures and feature-selection sections, the taxonomy recipe builders (flat/sensor/grouped/descriptor), the shared :func:`make_decoding_sweep_report` skeleton, and the head-to-head comparison report. Attributes ---------- .. autoapisummary:: coco_pipe.report.decoding_sweep.CLASSICAL_MODE_TITLES coco_pipe.report.decoding_sweep.DEFAULT_CLASSICAL_SECTION_BUILDERS coco_pipe.report.decoding_sweep.HEAD_TO_HEAD_DISPLAY_COLUMNS Classes ------- .. autoapisummary:: coco_pipe.report.decoding_sweep.ResultCollection Functions --------- .. autoapisummary:: coco_pipe.report.decoding_sweep.collect_results coco_pipe.report.decoding_sweep.build_comparison_section coco_pipe.report.decoding_sweep.build_result_tabs coco_pipe.report.decoding_sweep.make_experiment_results_report coco_pipe.report.decoding_sweep.collect_mode_results coco_pipe.report.decoding_sweep.best_result_tabs coco_pipe.report.decoding_sweep.hp_tuning_section coco_pipe.report.decoding_sweep.summary_collection coco_pipe.report.decoding_sweep.add_scientific_overview coco_pipe.report.decoding_sweep.leaderboard_section coco_pipe.report.decoding_sweep.failures_section coco_pipe.report.decoding_sweep.feature_selection_section coco_pipe.report.decoding_sweep.flat_section coco_pipe.report.decoding_sweep.sensor_section coco_pipe.report.decoding_sweep.grouped_section coco_pipe.report.decoding_sweep.descriptor_section coco_pipe.report.decoding_sweep.paired_delta_vs_baseline coco_pipe.report.decoding_sweep.prepare_sweep_frame coco_pipe.report.decoding_sweep.make_decoding_sweep_report coco_pipe.report.decoding_sweep.build_capability_matrix_section coco_pipe.report.decoding_sweep.build_foundation_comparison_sections coco_pipe.report.decoding_sweep.build_classical_mode_elements coco_pipe.report.decoding_sweep.build_classical_taxonomy_sections coco_pipe.report.decoding_sweep.collect_comparison_runs coco_pipe.report.decoding_sweep.enrich_head_to_head_frame coco_pipe.report.decoding_sweep.normalize_head_to_head_frame coco_pipe.report.decoding_sweep.make_head_to_head_report coco_pipe.report.decoding_sweep.make_decoding_report Module Contents --------------- .. py:class:: ResultCollection A labelled collection of decoding results and their tidy summaries. :ivar by: Ordered axis names that label the collection. The order is significant: the **first** axis is treated as the primary comparison axis by helpers that need a default (e.g. it seeds the default comparison axis in :func:`make_experiment_results_report`). ``results``/``contexts`` are keyed by tuples aligned to this order. :ivar results: Result objects, or the paths they are saved at, keyed by the axis-value tuple. Use :meth:`result` to get one loaded. :ivar contexts: Per-result context metadata keyed by the same axis-value tuple. :ivar summary: Concatenated tidy per-result summary rows. .. py:attribute:: by :type: tuple[str, ...] .. py:attribute:: results :type: dict[tuple[Any, ...], Any] .. py:attribute:: contexts :type: dict[tuple[Any, ...], dict[str, Any]] .. py:attribute:: summary :type: pandas.DataFrame .. py:method:: result(key) Return the result for ``key``, loading it if only its path is held. .. py:method:: filter(**criteria) Return a collection restricted to exact context matches. .. py:property:: successful_summary :type: pandas.DataFrame Return only successfully loaded result summaries. .. py:function:: collect_results(items, *, by, metrics = None) Load labelled results and concatenate their scalar summaries. .. py:function:: build_comparison_section(collection, *, kind, axis = None, row = None, column = 'Model', value = 'accuracy_mean', group_by = (), center = None, info = None, coords = None, title = None, include_table = False, on_error = 'warn') Build one reusable cross-result comparison section. .. rubric:: Notes Unlike :func:`build_decoding_sections`, ``on_error`` here accepts only ``"raise"`` or ``"warn"`` (not ``"placeholder"``). A comparison renders many per-group figures into one section, so a single group's failure is warned and skipped rather than replaced with a placeholder card. .. py:function:: build_result_tabs(collection, *, sections = 'compact', feature_metadata = None, info = None, coords = None, interactive = False, on_error = 'warn') Build nested compact per-result sections from a collection. .. py:function:: make_experiment_results_report(items, *, by, comparisons = 'default', per_result = 'compact', nest = True, feature_metadata = None, info = None, coords = None, interactive = False, title = 'Decoding Comparison', config = None, asset_urls = None, qc_result = None, output_path = None, on_error = 'warn') Build a comparison report over many labelled decoding results. :param comparisons: ``"default"`` (a single model-comparison view), ``"showcase"`` (model comparison plus per-axis heatmap, score spread, and metric matrix keyed on the first ``by`` axis), a single comparison ``kind`` string, or an explicit sequence of comparison specs. :param interactive: If True, per-result diagnostic sections render interactive Plotly figures (cross-result comparison figures remain static Matplotlib images). .. py:function:: collect_mode_results(mode_frame, *, context_columns = DEFAULT_MODE_RESULT_CONTEXT_COLUMNS) Collect the successful results of a sweep result frame, without loading them. .. py:function:: best_result_tabs(frame, primary_metric, *, group_columns = DEFAULT_BEST_RESULT_GROUP_COLUMNS, context_columns = DEFAULT_MODE_RESULT_CONTEXT_COLUMNS, feature_metadata = None, sections = 'compact', limit = MAX_INLINE_DIAGNOSTIC_RESULTS, on_error = 'placeholder') Render diagnostics for only the top-N ranked sweep rows. .. py:function:: hp_tuning_section(frame, *, primary_metric = 'primary_metric', feature_metadata = None, title = 'Hyperparameter Tuning', limit = MAX_INLINE_DIAGNOSTIC_RESULTS) Surface hyperparameter-search diagnostics for the top-ranked runs. The classical and foundation reports both include this section so HP tuning is reported the same way regardless of model family. Loads the best runs' artifacts and renders their search / best-parameter diagnostics; returns None when no run carries a hyperparameter search (``on_error="warn"`` drops non-tuned runs instead of emitting placeholders). .. py:function:: summary_collection(mode_frame, primary_metric = 'primary_metric', *, descriptor_family = None, include_sensor = False, context_columns = DEFAULT_MODE_RESULT_CONTEXT_COLUMNS) Build a lightweight comparison collection from persisted sweep metrics. .. py:function:: add_scientific_overview(report, frame, *, dataset_name, kind, strategy_note, empty_message = None) Add high-level decoding-sweep coverage cards and strategy text. .. py:function:: leaderboard_section(frame, *, title, table_title, filters, comparison_axis, group_by) Build a primary leaderboard plus a compact model/bar comparison. .. py:function:: failures_section(frame) Build a collapsible skipped/failed unit table. .. py:function:: feature_selection_section(frame, *, feature_metadata, baseline_label = 'baseline', selection_title = 'Feature-Selection Diagnostics') Render SFS / non-baseline feature-selection diagnostics. .. py:function:: flat_section(mode_frame, *, scope_label, feature_metadata, kind, title_prefix, group_by, **axis_kwargs) Build comparison and diagnostics tabs for flat decoding units. .. py:function:: sensor_section(mode_frame, *, scope_label, montage = 'colin27_1020', **kwargs) Build sensor-level comparison views and optional topomap. .. py:function:: grouped_section(mode_frame, *, scope_label, group_col, kind, title_prefix, group_by, **axis_kwargs) Build a grouped taxonomy comparison over an existing sweep frame. .. py:function:: descriptor_section(mode_frame, *, scope_label, feature_metadata, kind, title_prefix, group_by, descriptor_family = None, include_sensor = False, **axis_kwargs) Build descriptor-level summary comparisons plus top diagnostics. .. py:function:: paired_delta_vs_baseline(frame, *, group_columns, baseline_family, family_column = 'comparison_family', label_column = 'model_label', metric_column = 'primary_metric', pair_column = 'comparison_pair', delta_column = 'comparison_delta', compatible_lookup = None) Compute paired metric deltas against the best baseline row per group. .. py:function:: prepare_sweep_frame(records, *, add_cv_signature = True, scope_from = 'condition', default_scope = 'all') Build the tidy sweep frame shared by every decoding sweep report. Concatenates *records*, stamps the primary metric, attaches a readable ``cv_signature`` when derivable, and guarantees a ``scope`` column: taken from *scope_from* when present, else filled with *default_scope*. .. py:function:: make_decoding_sweep_report(records, *, title, kind, strategy_note, dataset_name = 'dataset', leaderboards = (), body = None, pre_sections = (), empty_message = None, frame = None, scope_from = 'condition', default_scope = 'all', add_cv_signature = True, config = None, asset_urls = 'inline', output_path = None) Assemble the skeleton every decoding sweep report shares. Runs: frame prep -> scientific overview -> *pre_sections* (e.g. QC or a foundation capability matrix) -> each ``leaderboard_section(**spec)`` in *leaderboards* -> the domain *body* hook (only when the frame is non-empty) -> failures section -> optional save. The ``body`` callable owns the per-axis layout that differs between classical and foundation reports. .. py:function:: build_capability_matrix_section(capability_records) Build the foundation preflight capability matrix, or None when empty. .. py:function:: build_foundation_comparison_sections(report, frame, *, group_by = ('condition', 'target'), per_result_sections = 'compact') Add foundation model/train-mode comparison figures + per-result tabs. A `body` hook for :func:`make_decoding_sweep_report`: loads successful result artifacts, renders linear-probe bars, a training-mode heatmap, a metric matrix and a spread plot, then the coco-pipe per-result diagnostics tabs. .. py:data:: CLASSICAL_MODE_TITLES :type: dict[str, str] .. py:data:: DEFAULT_CLASSICAL_SECTION_BUILDERS :type: dict[str, collections.abc.Callable[..., Any]] .. py:function:: build_classical_mode_elements(mode_frame, table_title, analysis_modes, scope_label, feature_metadata, *, section_builders = None) Build the per-mode result table plus its taxonomy comparison element. .. py:function:: build_classical_taxonomy_sections(report, frame, *, feature_metadata = None, section_builders = None, mode_titles = None, scope_order = None, scope_intro = None) Render one taxonomy section per scope: the classical report body. A ``body`` hook for :func:`make_decoding_sweep_report`, mirroring :func:`build_foundation_comparison_sections`. For each scope it lays out the analysis-unit taxonomy from the broadest flat analysis down to the narrowest sensor/descriptor views, flags a missing flat baseline, and appends feature-selection diagnostics. Pass *scope_order* to control scope sequencing (e.g. a study's configured conditions with a pooled scope last). .. py:function:: collect_comparison_runs(sources) Read, filter and tag per-run result CSVs into one comparison frame. Each entry in *sources* is a spec describing one family of runs to pull: ``paths`` Iterable of CSV paths (the caller owns the globbing policy). ``filters`` Mapping of ``column -> value`` equality filters applied to each CSV. The filter columns are also required: a CSV missing any of them is skipped rather than raising. ``family`` The ``comparison_family`` tag: either a string, or a callable ``(frame, path) -> str`` evaluated per surviving CSV. ``space`` / ``source`` Optional overrides for the ``comparison_space`` / ``source_run`` tags; callables of ``path``. Default to ``path.parent.name`` and ``str(path.parent)``. The read/filter/tag/concat boilerplate lives here; callers keep only their study-specific glob patterns, row filters and family policy. Empty or malformed CSVs are skipped. Returns an empty frame when nothing survives. .. py:data:: HEAD_TO_HEAD_DISPLAY_COLUMNS :value: ('comparison_family', 'comparison_space', 'source_run', 'scope', 'condition', 'target',... .. py:function:: enrich_head_to_head_frame(comparison_frame) Add the derived columns a head-to-head comparison frame needs. Expects a concatenated frame already tagged with ``comparison_family`` (the caller's globbing/tagging policy). Stamps the primary metric, attaches a ``cv_signature``, fills ``scope`` from ``condition``, and derives ``model_label``/``comparison_label``/``metric_name`` used by the table and the head-to-head plot. .. py:function:: normalize_head_to_head_frame(frame, *, display_columns = HEAD_TO_HEAD_DISPLAY_COLUMNS) Sort an enriched comparison frame and project the display columns. The single source of truth for the head-to-head leaderboard layout: used both for the persisted comparison CSV and for the report table, so the two never drift. *frame* must already be enriched (see :func:`enrich_head_to_head_frame`). .. py:function:: make_head_to_head_report(comparison_frame, *, baseline_family, group_columns = ('scope', 'target'), title, table_title = 'Descriptors vs. Embeddings vs. Linear Probe', intro = None, paired_delta_title = None, display_columns = HEAD_TO_HEAD_DISPLAY_COLUMNS, config = None, asset_urls = 'inline', extra_sections = (), output_path = None, paired_test_permutations = 1000) Build the head-to-head comparison report from a tagged comparison frame. ``comparison_frame`` must carry ``comparison_family`` (baseline vs. each challenger) plus per-run metrics; the caller owns how it is assembled. The report shows a normalized leaderboard table, a cohort/CV compatibility panel, a head-to-head accuracy plot, and paired deltas versus *baseline_family* for groups whose CV/cohort signatures are compatible, each with a paired permutation p-value from the saved predictions. .. py:function:: make_decoding_report(records, *, title = 'Classical Decoding', dataset_name = 'dataset', strategy_note = None, leaderboards = None, body = None, pre_sections = (), feature_metadata = None, section_builders = None, scope_order = None, include_hp_tuning = True, frame = None, scope_from = None, default_scope = 'all', config = None, output_path = None, asset_urls = 'inline') Build the classical decoding sweep report. The classical counterpart to :func:`~coco_pipe.report.foundation.make_foundation_decoding_report`: the same shared skeleton (scientific overview -> primary leaderboard -> body -> failures) with the analysis-unit taxonomy (flat/sensor/family/descriptor) as the default body. ``records`` are the per-unit classical decoding rows. Pass *scope_order*, *section_builders* or a custom *body* to override the study-specific layout; everything else falls back to the generic classical defaults.