← Back to API reference

neuralbench.transforms.TextPreprocessor

pydantic model neuralbench.transforms.TextPreprocessor

Clean and filter text-related events.

The following operations are applied to the events:

• Keep only events with duration >= 0

• Keep only neuro events, Audio, or valid Word events (with text as string)

• Clean ‘text’ column by removing special characters and lowercasing

• Drop empty or blank text entries

• For Nieuwland2018, group similar sentences together to avoid leakage (each sentence has two very similar versions)

Fields:

• neuro_event_type (str)

field neuro_event_type: str = 'Eeg'

static clean_text(x: str) → str

Remove special characters and lowercase the text.

requirements: tp.ClassVar[tuple[str, ...]] = ()

← Back to API reference