Discussion in #120 (comment) mentioned database in SimParam object:
@gaynorr wrote #211 (comment):
@gregorgorjanc, I have been considering implementing a phenotype recording system in SimParam. The idea would be for it to serve as a database for all simulated phenotypes. Perhaps this sort of data would make sense living somewhere like this.
I can already track all genotypes created (in most cases) by using the recombination tracking option in SimParam. SimParam also stores the founder population. As long as there is only one founder population, every individual every simulated could have its genotype reconstructed with these two pieces of information. The idea here would be to use the phenotype records and genotype records as an alternative method for training GS models or calculating BLUPs across multiple stages of selection.
and my reply was
@gaynorr yeah, we definitely need some sort of a database system, particularly once we get into repeated phenotypes! For example, in dairy cattle simulations we did something like this
…
database = recordData(database, cowsLact2, year)
cowsLact2 = setPhenoCow(cowsLact2, meanLact2, year, …)
cowsLact2 = selectInd(cowsLact1, …)
database = recordData(database, cowsLact1, year)
cowsLact1 = setPhenoCow(cowsLact1, meanLact1, …)
cowsLact1 = selectInd(heifers, …)
database = recordData(database, heifers, year)
heifers = selectInd(femaleCalves, …)
database = recordData(database, femaleCows, year)
femaleCalves = randCross(…)
...
and database is just a list of data.frames - it actually works quite well, but one could and likely should think about a proper SQL database too! The challenge with pushing everything into SimParam could be that you might not think of some detail that a user comes up, for example, we added lactation mean, herd effect, herd-year effect, permanent environment effects etc. into setPhenoCow() and recordData() grabs that info (most of it is in the misc slot!). It's hard to envision the requirements for storing such meta-data in the SimParam!? Perhaps function hooks could do all that!?
Definitely lots to think about. It should not be hard to store the core variables though, say by having SP$setTrackGv() and SP$setTrackPheno(), possibly more (what else?), though the challenge becomes when one considers meta-data - what to store by default and still give the user flexibility to store some meta-data as well - could RCpp conversions be used for this, say, user provides a matrix or data.frame, and then that is saved alongside gv and pheno?
Discussion in #120 (comment) mentioned database in
SimParamobject:@gaynorr wrote #211 (comment):
and my reply was
Definitely lots to think about. It should not be hard to store the core variables though, say by having
SP$setTrackGv()andSP$setTrackPheno(), possibly more (what else?), though the challenge becomes when one considers meta-data - what to store by default and still give the user flexibility to store some meta-data as well - couldRCppconversions be used for this, say, user provides a matrix or data.frame, and then that is saved alongside gv and pheno?