Unable to reproduce FRIGID-base pretraining performance and request for exact training configuration Description:

[peter74741]

Hello, thank you for releasing FRIGID.
We have been trying to reproduce the FRIGID-base fp2mol pretraining results, but our locally trained model performs substantially worse than the released frigid_base.ckpt.

Training setup

• Dataset: datamol-io/safe-gpt, revision 16d0be9ad6177ae683a32a86204530e8ee624a0f
• 7 × NVIDIA H20 GPUs
• Architecture: 12 layers, hidden size 896, 14 heads
• Shared formula/fingerprint cross-attention
• Global batch size: 511
• BF16 mixed precision
• Learning rate: 3e-4
• Warmup: 6,000 optimizer steps
• EMA: 0.9999
• Checkpoint evaluated at 78,000 optimizer steps (~39.9M structures)

Evaluation result

Using the same:

• CANOPUS 100-spectrum subset
• mist_canopus.pt
• NGBoost token-length predictor
• molecular-formula-prioritized ranking
• candidate budget and evaluation script
  we observed approximately:
• Released frigid_base.ckpt: Top-1 ≈ 19%
• Local 78k-step checkpoint: Top-1 ≈ 4%
  We understand that 78k steps is not equivalent to the fully trained released checkpoint. However, the large gap and the lack of a reference learning curve make it difficult to determine whether this is expected undertraining or a configuration/implementation mismatch.

Reproducibility questions

We found several ambiguities while debugging:

1. The paper describes a 12-layer, hidden-size 768, 12-head model, while the released checkpoint appears to use hidden size 896 and 14 heads.
2. The current Hugging Face dataset is SAFE v2, while the paper appears to use the older v1 dataset.
3. The released code contains internal FP32 autocast contexts despite trainer.precision: bf16.
4. With shared cross-attention, some parameters appear multiple times in the optimizer parameter groups.
5. EMA appears to cover only the backbone, resulting in raw condition encoders being evaluated together with an EMA backbone.
6. Fingerprint top-k masking appears capable of retaining inactive bits and dropping active bits.
7. It is unclear whether reported scaling results correspond to FRIGID-base alone or include ICEBERG refinement.
   Could you please provide:

• The exact command/configuration used to train frigid_base.ckpt
• Dataset version and revision
• Architecture and parameter-sharing configuration
• Global batch size, learning rate, warmup, precision and total optimizer steps
• EMA and optimizer parameter handling
• Expected FRIGID-base-only CANOPUS performance at intermediate checkpoints
  We would be happy to provide our checkpoint, logs and evaluation outputs if helpful.
  Thank you.

Inconsistency with the training-compute ablation

Our result also appears difficult to reconcile with Table 13.

Table 13 reports:

• 3.2M structures: Top-1 9.22%
• 12.5M structures: Top-1 13.82%
• 25.1M structures: Top-1 16.81%
• 49.9M structures: Top-1 18.06%

Our checkpoint has processed approximately 39.9M structures, but only reaches about 4% Top-1 under the corrected formula-prioritized evaluation.

There is also an ambiguity in Table 13: the 1B-structure result is 25.03%, which exactly matches the full FRIGID + ICEBERG result in Table 1, rather than the FRIGID-base result of 19.80%.

Could you please clarify:

1. Whether Table 13 evaluates FRIGID-base directly or includes ICEBERG refinement
2. Which dataset and benchmark split were used for Table 13
3. Whether the reported structure count represents unique structures or total training examples
4. Whether every row uses the same inference budget, MIST checkpoint and length predictor
5. Why the Table 13 model has 182.2M parameters while the released checkpoint appears to have approximately 170.6M unique parameters

Without this information, we cannot determine whether our low accuracy is caused by incorrect pretraining or by comparing against a different evaluation pipeline.