U-MIRAGE

Code and artifacts for the paper: "U-MIRAGE: Benchmarking Chain-of-Thought Reasoning for Urdu Medical QA".

This repository has been cleaned for publication:

• notebook outputs/tracebacks removed
• local machine paths and personal identifiers removed
• dataset-specific notebook cells removed from the translation workflow
• duplicate notebook copies removed

Repository Contents

• data_load_in_csv.ipynb: load and normalize benchmark data into tabular format
• data_clean_1.ipynb: base cleaning and structural validation steps
• data_clean_2.ipynb: Roman Urdu quality checks and dataset consistency validation
• mirage_modify.ipynb: post-processing and dataset modification utilities
• translation.ipynb: translation/back-translation workflow (publication-safe cells only)
• code-switch-stats.ipynb: code-switch and similarity analysis
• all_configurations_accuracy.csv: configuration-level accuracy summary
• benchmark_ru_translated.csv: translated benchmark data
• backtranslation_samples_with_rouge.csv: sampled backtranslation outputs with ROUGE
• backtranslation_rouge_scores.csv: aggregate ROUGE score report
• paper/UMirage__Benchmarking_Chain_of_Thought_Reasoning_in_Large_Language_Models__1_.pdf: publication PDF
• requirements.txt: Python dependencies
• CITATION.cff: citation metadata
• LICENSE: project license

Environment

Use Python 3.10+.

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Reproducibility

1. Run data_load_in_csv.ipynb.
2. Run data_clean_1.ipynb.
3. Run data_clean_2.ipynb.
4. Run mirage_modify.ipynb (if applying post-processing changes).
5. Run translation.ipynb for translation and backtranslation evaluation.
6. Run code-switch-stats.ipynb for code-switch/similarity analyses.
7. Review generated CSV artifacts in the repository root.

Citation

If you use this repository, cite the paper PDF in paper/ and metadata in CITATION.cff.

License

Code is released under the MIT License (see LICENSE).