Building an EEG‐based Alzheimer’s Diagnostic: What is the Optimal Cross‐Validation Method?

Alzheimer's disease lacks a fast, easy, reliable method of diagnosis. Attempts to create classifiers based on brain activity in resting‐state electroencephalography (rsEEG) are promising but require state‐of‐the‐art machine learning models to leverage the available information. Typically, single‐pass cross‐validation (SPCV) is used to select a model, despite work suggesting SPCV introduces bias when many models are considered. Recently, nested cross‐validation (NCV) has been introduced to correct this bias. Here we report which of these two processes yields the most accurate Alzheimer's classifiers in held‐out data.