setMinDepth cannot be less than -1

[LawrenceGordon]

Hi All,

I have been running angsd as part of another snakemake workflow, which automatically determines setMinDepth based on standard deviation(s) away from the mean. This sample set resulted in a setMinDepth of -46. While my samples may truly have such a deviation, the issue arises in abcCounts.cpp.

If the value is less than -1, it leads to some strange comparisons. This comes from the way that min depth is represented versus the total sums of coverage (

angsd/abcCounts.cpp

Line 765 in 6b5d906

if(totSum<setMinDepth)

). Comparisons of size_t and int only seem to work if both are the same sign. I have tested this up to totSum of 1,000,000 and it seems nearly any positive integer (perhaps up to some overflow) will be less than any negative value for setMinDepth, i.e. 1000000<-46 returns true in if(totSum<setMinDepth):

  if(setMinDepth!=-1){
    for(int s=0;s<pars->numSites;s++){
      if(pars->keepSites[s]==0)
	continue;
      size_t totSum = calcSum(pars->counts[s],4*pars->nInd);
      if(totSum<setMinDepth)
	pars->keepSites[s]=0;

While there could never truly be negative coverage for a sample, it may be worth throwing an error if the user enters a value <-1 here. I have posted this as an issue in the related workflow if that is deemed more appropriate.

Thanks!