coco_pipe.io.dataset#

Specialized Dataset classes that produce standardized DataContainer objects.

Attributes#

logger

Classes#

BaseDataset

Helper class that provides a standard way to create an ABC using

TabularDataset

Dataset for loading tabular feature data (CSV, TSV, Excel).

EmbeddingDataset

Generic Dataset for loading embedding files (Pickle, NPY, JSON, H5).

BIDSDataset

Dataset for loading epoched M/EEG data from a BIDS dataset.

Module Contents#

coco_pipe.io.dataset.logger#
class coco_pipe.io.dataset.BaseDataset#

Bases: abc.ABC

Helper class that provides a standard way to create an ABC using inheritance.

abstractmethod load()#
Return type:

coco_pipe.io.structures.DataContainer

class coco_pipe.io.dataset.TabularDataset(path, target_col=None, index_col=None, sep='\t', header=0, sheet_name=0, columns_to_dims=None, col_sep='_', meta_columns=None, clean=False, clean_kwargs=None, select_kwargs=None)#

Bases: BaseDataset

Dataset for loading tabular feature data (CSV, TSV, Excel).

This class handles loading, optional clearing, and reshaping of 2D tabular data into multi-dimensional DataContainers.

Parameters:

  • path (str or Path) – Path to the tabular file (csv, tsv, txt, xls, xlsx).

  • target_col (str, optional) – Name of the column to extract as target y. Removed from features X.

  • index_col (str or int, optional) – Column to use as index (observation IDs).

  • sep (str, default='t') – Separator for text files.

  • header (int or list of int, default=0) – Row number(s) to use as column names.

  • sheet_name (str or int, default=0) – Sheet name or index for Excel files.

  • columns_to_dims (list of str, optional) – If provided, attempts to reshape the 2D feature columns into N-D dimensions. Columns must follow the naming convention: dim1_dim2_…_feature.

  • col_sep (str, default='_') – Separator used in column names for reshaping.

  • meta_columns (list of str, optional) – List of columns to extract as metadata coordinates instead of features.

  • clean (bool, default=False) – Whether to perform automated cleaning (drop NaNs/Infs).

  • clean_kwargs (dict, optional) – Arguments passed to TabularDataset.clean.

  • select_kwargs (dict, optional) – Arguments for feature selection (not yet implemented in load directly).

Examples

>>> # Load a simple CSV
>>> ds = TabularDataset("data.csv", target_col="label")
>>> container = ds.load()

>>> # Load and reshape wide data (e.g. time series in columns)
>>> # Columns: T0_F1, T0_F2, T1_F1... -> dims=('time', 'freq')
>>> ds = TabularDataset("wide.csv", columns_to_dims=["time", "freq"], col_sep="_")
path#
target_col = None#
index_col = None#
sep = '\t'#
header = 0#
sheet_name = 0#
columns_to_dims = None#
col_sep = '_'#
meta_columns = []#
do_clean = False#
clean_kwargs#
select_kwargs#
strict_reshaping = True#
load()#
Return type:

coco_pipe.io.structures.DataContainer

static clean(X, mode='any', sep='_', reverse=False, verbose=False, min_abs_value=None, min_abs_fraction=0.0)#

Remove invalid feature columns containing NaN, ±Inf, and optionally very small values.

Parameters:
Return type:

tuple[pandas.DataFrame, dict[str, list[str]]]

class coco_pipe.io.dataset.EmbeddingDataset(path, pattern='*.pkl', dims=('feature',), coords=None, reader=None, id_fn=None, subjects=None)#

Bases: BaseDataset

Generic Dataset for loading embedding files (Pickle, NPY, JSON, H5).

This class decouples file discovery (via patterns and IDs) from content reading. It supports structured formats (e.g., Layers x Features) and user-supplied metadata coordinates.

Parameters:

  • path (str or Path) – Directory searched for embedding files, or a single file.

  • pattern (str, default=r'*.pkl') – Glob pattern to match files (e.g., *.npy, sub-*_emb.pkl).

  • dims (tuple of str, default=('feature',)) – Dimension labels of one observation; obs is added in front. Typically (‘feature’,) or (‘layer’, ‘feature’).

  • coords (dict, optional) – Dictionary of coordinates for dimensions. E.g., {‘layer’: [‘L1’, ‘L2’]}.

  • reader (callable, optional) – Custom function to read a Path and return a numpy array or dict. If None, uses smart_reader based on file extension.

  • id_fn (callable, optional) – Function returning the subject ID of a Path. Defaults to the file stem.

  • subjects (int or list, optional) – If int, loads first N subjects. If list, loads specific subjects (matched by id_fn).

Examples

>>> # Load loose numpy files
>>> ds = EmbeddingDataset("./embeddings", pattern="*.npy", dims=("feature",))
>>> container = ds.load()
path#
subjects = None#
dims = ('feature',)#
coords_in#
pattern = '*.pkl'#
reader#
id_fn#
load()#
Return type:

coco_pipe.io.structures.DataContainer

class coco_pipe.io.dataset.BIDSDataset(root, task=None, session=None, datatype='eeg', suffix=None, mode='epochs', target_col=None, window_length=None, stride=None, subjects=None, runs=None, event_id=None, subject_metadata_df=None, subject_key=None, tmin=-0.2, tmax=0.5, baseline=None, units=None, description=None, dtype='float32', n_jobs=1)#

Bases: BaseDataset

Dataset for loading epoched M/EEG data from a BIDS dataset.

Recordings are found and read with MNE-BIDS, as mne.Epochs, and stacked with mne.concatenate_epochs(), so they must share their channels, sampling rate and epoch length. Metadata from participants.tsv is attached to every observation.

Parameters:

  • root (str or Path) – The root directory of the BIDS dataset.

  • task (str, optional) – The task name (e.g., ‘rest’, ‘audiovisual’).

  • session (str or List[str], optional) – The session ID(s) to load. If None, all sessions are loaded.

  • datatype (str, default='eeg') – The data type to load (e.g., ‘eeg’, ‘meg’).

  • suffix (str, optional) – The suffix of the files to load. Defaults to datatype, or to ‘epo’ for pre-computed epochs.

  • mode (str, default='epochs') –

    • ‘epochs’: cut raw recordings into epochs.

    • ’load_existing’: read pre-computed epochs (*_epo.fif).

  • target_col (str, optional) – Metadata column to expose as the target y. By default y holds the event codes.

  • window_length (float, optional) – Length in seconds of the fixed-length epochs cut from raw recordings.

  • stride (float, optional) – Stride between windows in seconds. Defaults to window_length.

  • subjects (str or List[str], optional) – Subject IDs to load (without ‘sub-’ prefix). If None, all subjects.

  • runs (str or List[str], optional) – Run IDs to load. If None, all runs.

  • event_id (dict, str or list of str, optional) – For raw recordings, the events to epoch around (see mne.events_from_annotations()), in place of fixed-length windows. For pre-computed epochs, the event names to keep. The codes of a dict are the codes in y, whatever codes the files use. With a string, a list of names or None, the event names are numbered from 1 in the order they first appear in the recordings.

  • subject_metadata_df (DataFrame, optional) – Subject-level metadata to attach, in addition to participants.tsv.

  • subject_key (str, optional) – Column of subject_metadata_df holding the subject ID.

  • tmin (float) – Epoch limits in seconds around each event, with event_id on raw recordings.

  • tmax (float) – Epoch limits in seconds around each event, with event_id on raw recordings.

  • baseline (tuple, optional) – Baseline interval, with event_id on raw recordings.

  • units (str, optional) – Units of the returned data (see mne.Epochs.get_data()).

  • description (str, optional) – The desc entity of the files to load.

  • dtype (str, default="float32") – Data type of the returned array.

  • n_jobs (int, default=1) – Number of threads reading recordings at the same time. Reading is mostly waiting on the file system, so threads help on network or cluster storage.

Notes

The recording_id coordinate is the BIDS entity string of a recording, e.g. sub-0001_ses-01_task-rest_run-01. Observation IDs are <recording_id>_epoch-<i>, where i is the epoch coordinate: the position of the epoch in its recording, whichever events are selected.

Examples

>>> # Load resting state EEG for all subjects, sliced into 1s windows
>>> ds = BIDSDataset(root="/data/bids", task="rest", window_length=1.0)
>>> container = ds.load()
root#
task = None#
session = None#
datatype = 'eeg'#
suffix = None#
mode = 'epochs'#
target_col = None#
window_length = None#
stride = None#
subjects = None#
runs = None#
event_id = None#
subject_metadata_df = None#
subject_key = None#
tmin = -0.2#
tmax = 0.5#
baseline = None#
units = None#
description = None#
dtype = 'float32'#
n_jobs = 1#
load()#

Load the BIDS dataset into a DataContainer.

Returns:

A container with:

  • X: Data array of shape (N_obs, N_channels, N_time).

  • y: The event code of each observation, or target_col. The codes are those of event_id when it is a dict, and otherwise numbered from 1 in the order the event names first appear. They are the same for every recording, and meta["event_id"] maps each event name that was found to its code.

  • ids: Unique identifiers for each observation.

  • coords: ‘channel’, ‘time’, and one entry per observation for ‘obs’, ‘epoch’, ‘recording_id’, ‘bad_channels’ (comma-separated, as marked in the file), ‘subject’, ‘session’, ‘run’ and the subject metadata.

  • dims: (‘obs’, ‘channel’, ‘time’).

  • meta: ‘sfreq’, ‘source’, ‘units’, ‘event_id’, ‘skipped_no_matching_events’ (recordings with none of the requested events, skipped without a warning), and ‘load_failures’ (the ‘recording_id’ and ‘error’ of each recording that failed to load).

Return type:

DataContainer

Raises: