Dual-path CNN–ViT image classification model with feature-wise gated fusion.
This repository implements a hybrid deep learning architecture that combines an equivariant CNN branch with a lightweight Vision Transformer branch. The goal is to improve robustness under image rotations while keeping the model suitable for efficient inference and edge-AI deployment experiments.
The project currently supports experiments on CIFAR-10 and PatchCamelyon / PCam-style image classification tasks.
Convolutional neural networks are strong at capturing local visual patterns, while Vision Transformers provide broader contextual representations. This project combines both ideas in a two-path architecture:
- CNN branch: rotation-aware feature extraction using GMR / E2CNN-style convolutional backbones
- ViT branch: lightweight transformer-based feature extraction using EfficientViT or MobileViT-style models
- Gated fusion module: learns how much to rely on CNN features and ViT features for each feature dimension
- MLP classifier head: predicts the final class from the fused representation
The fusion module learns a feature-wise gate:
fused = gate * cnn_features + (1 - gate) * vit_features