Memory requirements

[jean997]

Hi Luke,
I am wondering if you can give some guidance on memory requirements. We had no problem running your analysis of the 32 studies from the paper. However, when we tried applying it to a data set that has 6 million variants instead of 1 million it seems to require a lot of memory (> 32G). Do you have a sense of how memory requirements scale? Would you recommend subsetting variants down before running?
Thanks!

[lukejoconnor]

Hi Jean,

I expect it to scale linearly. run_FMR creates a matrix lF of size SNPs * sampling times * components, which should be 8GB. This gets copied at least twice, when forming X and when passing X to the regression function. So it isn't too surprising that 32GB wasn't enough, but for sure this could be optimized.

In principle it should be slightly better to run it on 6M SNPs, but I haven't verified this. There isn't anything wrong with running it on HapMap3 SNPs.