SAE training, layer choice and activation scaling

[Genereux-akotenou]

I have a question about the SAE setup used with Evo.
Are there plans to release the training pipeline or code for the sparse autoencoders? You only released the inference demo code :-)

Also, I noticed the demo seems to focus on layer 26 activations; is there a specific reason this layer was chosen for SAE training?

Finally, could you clarify how activation normalization and precision were handled during the SAE training?
For example:

• Were activations taken before or after normalization layers?
• Was any global scaling applied?
• And was the SAE trained in bf16, fp32, or mixed precision?
• If the value of activation are too small, won't the SAE loss pretend doing good while in reality It doesn't?

[Genereux-akotenou]

Hi, I have another question about SAE training. Since it’s unsupervised, how did you handle probing for CDS-related signals or features? Did you use a separate dataset enriched with CDS signals (without explicitly providing annotations to the SAE), or did you rely on the same dataset (fully or partially) that was used during Evo2 training?