Multimodal Wearable Stress Classification Across Temporal Windows

This repository contains the analysis pipeline for a multimodal wearable stress-classification study using subject-independent machine learning. The project evaluates how classification performance changes across temporal windows and examines the contribution, stability, and temporal evolution of physiological and movement-derived features.

The workflow is designed for manuscript-oriented analysis, with emphasis on leakage-safe outer Leave-One-Subject-Out validation, grouped hyperparameter tuning, statistical comparison, accelerometer contribution analysis, feature-selection stability, and reproducible table and figure generation.

---

Repository overview

The repository contains six main Python modules:

File	Purpose
core_pipeline.py	Core modelling engine. Implements outer Leave-One-Subject-Out cross-validation, fold-wise scaling, exact zero-variance filtering, Pearson-correlation pruning, ANOVA SelectKBest, group-aware inner hyperparameter tuning, model fitting, prediction, and feature-selection tracking.
experiments.py	Defines the temporal-window files, switches between the 245-feature and 567-feature datasets, validates file and class structure, and runs the full and targeted accelerometer analyses.
tables.py	Generates performance, statistical-comparison, feature-selection, feature-stability, modality-composition, feature-drift, and importance-shift tables.
figures.py	Generates the cross-window performance curves, subject-level heatmap, confusion matrices, Jaccard heatmap, modality-importance plot, and accelerometer-analysis bar chart.
run_all.py	Top-level entry point. Runs or reloads E1 and E2, saves raw result objects, and generates all tables and figures.
permutation_test.py	Standalone permutation-test script for the 1-minute Random Forest analysis. It independently permutes class labels within each participant, reruns the final LOSO pipeline, and calculates an empirical permutation p-value for E1 and E2.

---

Study objective

The analysis supports the following questions:

1. How does subject-independent stress-classification performance change across 30-second, 1-minute, 2-minute, 3-minute, 4-minute, and 5-minute windows?
2. How do Logistic Regression, Random Forest, Support Vector Machine, and XGBoost compare across temporal windows?
3. How much does accelerometer information contribute at the 1-minute window?
4. Can acceleration alone support classification at the 1-minute window?
5. Which features are selected consistently across outer LOSO folds?
6. How does the selected feature set change across temporal windows?
7. How does the modality composition and model-derived importance of selected features shift over time?
8. Is the observed 1-minute Random Forest performance greater than expected under participant-structure-preserving label permutation?

---

Experimental analyses

The pipeline runs two classification experiments.

E1 — Four-class classification

Classes:

Stress Stage code	Class
1	Relaxation
2	Physical stress
4	Cognitive stress
6	Emotional stress

E2 — Three-class classification

Relaxation rows are removed before modelling.

Classes:

Stress Stage code	Class
2	Physical stress
4	Cognitive stress
6	Emotional stress

For each experiment, the pipeline performs:

• full multimodal classification across all six temporal windows;
• targeted 1-minute analysis using:
  • Full;
  • No-ACC;
  • ACC-only.

---

Input datasets

The repository supports two processed feature configurations:

Configuration	Candidate features	Description
245	245	35 extracted features across seven signal channels
567	567	81 extracted features across seven signal channels

The seven channels are:

• Acc-1
• Acc-2
• Acc-3
• EDA
• HR
• SpO2
• Temp

These represent five sensor modalities:

• Triaxial accelerometer
• Electrodermal activity
• Heart rate
• Oxygen saturation
• Skin temperature

Each CSV must contain:

Subject
Stress Stage
<feature columns>

Feature columns must follow the naming pattern:

<feature_name>_<channel>

Examples of supported channel suffixes are:

Acc-1
Acc-2
Acc-3
EDA
HR
SpO2
Temp

All six files within a selected feature configuration must contain the same feature names in the same order.

---

Expected data folders

The processed CSV files are organised as follows:

data_245features/
    data_30s.csv
    data_1min.csv
    data_2min.csv
    data_3min.csv
    data_4min.csv
    data_5min.csv

data_567features/
    data_30s.csv
    data_1min.csv
    data_2min.csv
    data_3min.csv
    data_4min.csv
    data_5min.csv

Both processed feature sets are included in the repository.

---

Selecting the 245-feature or 567-feature input

Open experiments.py and locate:

FEATURE_SET = "245"

Use:

FEATURE_SET = "245"

for the 245-feature dataset, or:

FEATURE_SET = "567"

for the 567-feature dataset.

The corresponding data directory and expected feature count are selected automatically.

The validation stage checks:

• whether all six configured files exist;
• whether the expected number of feature columns is present;
• whether feature names and feature order are identical across windows;
• whether the expected stress-stage codes are present;
• whether the supplied human-readable class labels match the encoded class order.

---

Installation

Create a Python virtual environment and install the required packages.

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

On Windows:

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

The required packages are listed in requirements.txt.

---

Models

The pipeline includes the following classifiers:

Key	Model
logreg	Logistic Regression
rf	Random Forest
svm	RBF-kernel Support Vector Machine
xgb	XGBoost Classifier
dummy	Majority-class Dummy baseline

The Dummy classifier is included in the full cross-window analysis as a reference baseline.

---

Validation and preprocessing workflow

For each outer LOSO fold, one participant is held out as the test subject.

The following operations are fitted using the outer training subjects only:

1. Standardisation using StandardScaler
2. Exact zero-variance filtering
3. Pearson-correlation pruning
4. ANOVA F-score feature selection using SelectKBest
5. Group-aware hyperparameter tuning using GroupKFold
6. Final model fitting
7. Evaluation on the held-out subject

The test subject is not used to estimate scaling parameters, identify zero-variance features, calculate feature correlations, select features, or tune model hyperparameters.

The default settings in run_all.py are:

K_FEATURES = 10
TUNE = True
CORR_THRESHOLD = 0.95
ANALYSIS_WINDOW = "1min"

Correlation pruning is deterministic. For feature pairs exceeding the absolute correlation threshold, the later feature in the current column order is removed.

---

Hyperparameter tuning

Grouped inner cross-validation is used for the four primary classifiers.

The configured search grids are:

Logistic Regression

{"C": [0.01, 0.1, 1.0]}

Random Forest

{
    "n_estimators": [100, 200],
    "max_depth": [None, 5],
}

Support Vector Machine

{
    "C": [0.1, 1.0, 10.0],
    "gamma": ["scale", "auto"],
}

XGBoost

{
    "learning_rate": [0.05, 0.1],
    "max_depth": [3, 5],
}

The Dummy baseline is fitted directly and is not tuned.

---

Running the full analysis

From the repository root, run:

python run_all.py

When:

RERUN_LOSO = True

the script runs the complete E1 and E2 LOSO analyses and overwrites the stored raw result objects.

When:

RERUN_LOSO = False

the script loads the existing raw_results.joblib files and regenerates tables and figures without repeating model fitting.

Set RERUN_LOSO = True whenever any of the following changes:

• input feature configuration;
• processed CSV files;
• preprocessing logic;
• correlation threshold;
• number of selected features;
• model definitions;
• hyperparameter grids;
• class configuration.

Stored results from one feature configuration must not be reused for the other feature configuration.

---

Running the permutation test

The permutation analysis is implemented separately in permutation_test.py so that it can be run independently of run_all.py.

The test evaluates whether the observed 1-minute Random Forest macro-F1 could reasonably occur by chance. For each permutation, class labels are independently shuffled within each participant, preserving the participant structure and class balance. The complete LOSO modelling pipeline is then rerun on the permuted data.

Run both E1 and E2 using the default 1000 permutations:

python permutation_test.py

Run only one experiment:

python permutation_test.py --experiment E1

or:

python permutation_test.py --experiment E2

For a quick validation run before launching the full analysis:

python permutation_test.py --n-permutations 5 --experiment E1

The corrected empirical p-value is calculated as:

(number of permuted scores greater than or equal to the observed score + 1)
/
(number of permutations + 1)

The permutation test is computationally intensive because the complete LOSO pipeline, including grouped inner hyperparameter tuning, is rerun for every permutation.

---

Statistical analyses

The table-generation module performs the following comparisons:

Cross-window comparison

• Friedman test across temporal windows for each classifier
• Paired Wilcoxon signed-rank tests comparing each non-reference window with the reference window
• Holm correction within each classifier
• Paired Cohen's (d)
• Bootstrap confidence intervals for paired mean differences

The default reference window is the longest available window, normally 5 minutes.

Accelerometer contribution analysis

At the 1-minute window, Random Forest performance is compared between:

• Full and No-ACC
• Full and ACC-only

The comparisons include:

• paired Wilcoxon signed-rank test;
• Holm-adjusted p-value;
• paired Cohen's (d);
• bootstrap confidence interval for the mean paired macro-F1 difference.

Permutation test

The standalone permutation_test.py script evaluates the statistical significance of the observed 1-minute Random Forest macro-F1. It:

• runs E1 and E2 separately;
• independently permutes task labels within each participant;
• preserves participant grouping and class balance;
• reruns scaling, variance filtering, correlation pruning, feature selection, grouped inner tuning, and outer LOSO evaluation;
• stores the full null distribution of permuted macro-F1 scores;
• calculates a corrected empirical p-value using (b + 1) / (B + 1).

Between-classifier comparison

Within each 1-minute pipeline:

• Friedman omnibus test across classifiers;
• Nemenyi post-hoc comparison.

The three evaluated pipelines are:

• Full
• No-ACC
• ACC-only

Subject identity is checked before paired and repeated-measures statistical comparisons.

---

Feature-selection outputs

The pipeline records selected features separately for every outer LOSO fold.

The exported feature-analysis tables include:

• selected feature identity;
• held-out subject;
• within-fold ANOVA rank;
• ANOVA F-score;
• ANOVA p-value;
• actual number of selected features;
• feature count after variance filtering;
• feature count after correlation pruning;
• correlation threshold;
• selection frequency across folds;
• consensus features;
• within-window Jaccard stability;
• feature drift across windows;
• modality composition;
• model-derived importance aggregated by modality.

Consensus features are defined in the current table functions as features selected in at least 50% of outer LOSO folds.

---

Output structure

Running run_all.py creates:

results/
    E1_four_class/
        raw_results.joblib
        run_configuration.csv
        tables/
            table4_long.csv
            table4_wide_f1.csv
            table5_accelerometer_analysis.csv
            table6a_friedman.csv
            table6b_wilcoxon.csv
            table7_between_classifier.csv
            foldwise_selected_features.csv
            one_minute_consensus_features.csv
            within_window_feature_stability.csv
            modality_composition.csv
            feature_drift.csv
            jaccard_matrix.csv
            modality_importance_shift.csv
        figures/
            jaccard_heatmap.png
            modality_importance_shift.png
            accelerometer_analysis_bars.png

    E2_three_class/
        raw_results.joblib
        run_configuration.csv
        tables/
            table4_long.csv
            table4_wide_f1.csv
            table5_accelerometer_analysis.csv
            table6a_friedman.csv
            table6b_wilcoxon.csv
            table7_between_classifier.csv
            foldwise_selected_features.csv
            one_minute_consensus_features.csv
            within_window_feature_stability.csv
            modality_composition.csv
            feature_drift.csv
            jaccard_matrix.csv
            modality_importance_shift.csv
        figures/
            jaccard_heatmap.png
            modality_importance_shift.png
            accelerometer_analysis_bars.png

    combined/
        tables/
            table4_combined.csv
            table5_combined.csv
            table6a_combined.csv
            table7_combined.csv
        figures/
            figure5_window_f1.png
            figure6_subject_heatmap_1min.png
            figure7_confusion_matrices_1min.png

    permutation_tests/
        permutation_E1.csv
        permutation_E1_summary.json
        permutation_E2.csv
        permutation_E2_summary.json

---

Main tables

Output	Description
table4_long.csv	Mean macro-F1 and 95% bootstrap confidence interval for each model-window combination
table4_wide_f1.csv	Model-by-window macro-F1 summary
table5_accelerometer_analysis.csv	Full versus No-ACC and ACC-only comparisons at 1 minute using Random Forest
table6a_friedman.csv	Friedman test across temporal windows for each classifier
table6b_wilcoxon.csv	Paired window comparisons against the reference window
table7_between_classifier.csv	Friedman and Nemenyi summary across classifiers within each 1-minute pipeline
foldwise_selected_features.csv	Features selected in each outer LOSO fold
one_minute_consensus_features.csv	Features selected in at least 50% of 1-minute LOSO folds
within_window_feature_stability.csv	Feature-selection stability within each window
modality_composition.csv	Modality composition of the consensus feature set
feature_drift.csv	Feature selection-frequency changes across windows
jaccard_matrix.csv	Jaccard overlap of consensus feature sets across windows
modality_importance_shift.csv	Model-derived feature importance aggregated by modality
permutation_E1.csv	Full null distribution of permuted 1-minute RF macro-F1 values for E1
permutation_E1_summary.json	E1 observed score, null-distribution summary, exceedance count, settings, and empirical p-value
permutation_E2.csv	Full null distribution of permuted 1-minute RF macro-F1 values for E2
permutation_E2_summary.json	E2 observed score, null-distribution summary, exceedance count, settings, and empirical p-value

---

Figures generated

Figure	Description
figure5_window_f1.png	E1 and E2 macro-F1 curves across temporal windows with bootstrap confidence intervals
figure6_subject_heatmap_1min.png	Subject-wise Random Forest macro-F1 at the 1-minute window
figure7_confusion_matrices_1min.png	Normalised LOSO-aggregated confusion matrices for LR, RF, SVM, and XGB
jaccard_heatmap.png	Jaccard similarity between consensus feature sets across windows
modality_importance_shift.png	Normalised modality-level feature-importance share across windows
accelerometer_analysis_bars.png	Random Forest performance for Full, No-ACC, and ACC-only comparisons

Figures are saved at 300 dpi.

---

Reproducibility notes

• Random Forest and XGBoost use random_state=42.
• Bootstrap confidence intervals use a fixed seed of 0.
• The standalone permutation test uses a fixed random seed by default and records the seed, number of permutations, observed macro-F1, null-distribution summary, and empirical p-value in JSON output.
• Hyperparameter tuning uses participant-grouped inner cross-validation.
• Outer evaluation uses Leave-One-Subject-Out cross-validation.
• Subject identifiers are sorted using a numeric-aware ordering before paired comparisons.
• Input feature count, selected-feature setting, correlation threshold, analysis window, and tuning status are written to run_configuration.csv.
• Missing or non-finite feature values cause the pipeline to stop with an explicit error.
• The current pipeline does not perform imputation.

---

Data availability

The raw data used for this analysis is publicly available from PhysioNet:

https://physionet.org/content/noneeg/1.0.0/

The processed 245-feature and 567-feature CSV files used by the analysis are included in this repository.

---

Citation

To be updated

---

Contact

For questions about the analysis pipeline, reproducibility, or manuscript-related outputs, please contact Gowtham Iyer [ gowtham DOT g DOT iyer AT gmail DOT com ]