Significant discrepancy in MATH dataset evaluation: Local results (28.5) vs. Official reported (46.1) for SmolLM3-3B-Base

[Derek0428]

1. Description
   I am evaluating the SmolLM3-3B-Base model using the same lighteval environment and scripts which is offered in text/evaluation/smollm3. While most reasoning and knowledge metrics (MMLU, HellaSwag, GSM8K) align well with the scaling trends, I observed a significant gap in the MATH dataset performance.

My Local Result: ~28.5
Official Reported Result: ~46.1
Gap: ~17.6 points
I would like to clarify if there are specific configurations or prompt templates used in the official evaluation that might cause this difference.

2. Environment & Reproduce
   Model: HuggingFaceTB/SmolLM3-3B-Base
   Framework: lighteval (Version: 0.10.1.dev0)
   Evaluation Command:
   MODEL_ARGS="model_name=HuggingFaceTB/SmolLM3-3B-Base,dtype=bfloat16,max_model_length=32768,max_num_batched_tokens=32768,generation_parameters={temperature:0},tensor_parallel_size=2,gpu_memory_utilization=0.7"
   lighteval vllm
   "$MODEL_ARGS"
   "custom|math_cot|4|1"
   --custom-tasks "tasks.py"
   --output-dir "evals/"
   --save-details
   Results:
   | Task |Version| Metric |Value | |Stderr|
   |------------------------------------------|------:|----------------|-----:|---|-----:|
   |all | |extractive_match|0.2851|± |0.0157|
   |custom:math_cot:_average:4 | |extractive_match|0.2851|± |0.0157|
   |custom:math_cot:algebra:4 | 0|extractive_match|0.4709|± |0.0145|
   |custom:math_cot:counting_and_probability:4| 0|extractive_match|0.3460|± |0.0219|
   |custom:math_cot:geometry:4 | 0|extractive_match|0.2484|± |0.0198|
   |custom:math_cot:intermediate_algebra:4 | 0|extractive_match|0.0653|± |0.0082|
   |custom:math_cot:number_theory:4 | 0|extractive_match|0.2648|± |0.0190|
   |custom:math_cot:prealgebra:4 | 0|extractive_match|0.5488|± |0.0169|
   |custom:math_cot:precalculus:4 | 0|extractive_match|0.0513|± |0.0094|

Questions:
Is there a specific few_shot_split or a custom prompt_template (System Prompt) that I should apply for the MATH task?
Are you using a specific math-equivalence checker (like sympy or math_verify) that is more robust than simple regex?
Are there any hints about the occurence of the gap?