SAE features show a spike at the first ~15 bp

[ShawnGao911101]

Hi Evo2 team,

When interpreting Evo2 embeddings with SAE, we consistently observe a strong activation spike within the first ~15 bp of input sequences. This pattern appears across multiple datasets, species, and SAE features.

Is this expected behavior? Could it be related to positional embeddings, sequence boundary effects, special tokens, or the SAE training procedure?

Any insights would be greatly appreciated.

Thanks!

[edoSever]

Hello,

I am seeing a very similar behavior when including those 15 bp when encoding the embeddings using the batch topk SAE. Due to the activated features in those 15 bp, the rest of the sequence had almost no activated features (they practically contain the top 64). Removing those 15 bp, removes the effect and generates good results.

Best,
Edouard