too many open files

[daroczig]

@nfultz, did you see any issues with letting the service run for a longer time?

Everything seems to work perfectly fine while running a grpc service in Docker for a while, and the service responds to requests, but after ~25 mins the process fails with:

{"log":"E0106 00:37:51.980090342       1 tcp_server_posix.c:245]     Failed accept4: Too many open files\n","stream":"stderr","time":"2018-01-06T00:37:51.98024391Z"}

Any chance that the server does not free up some resources in the loops or any other ideas?

[nfultz]

I think this is an upstream issue - grpc/grpc#11015

[nfultz]

Are you setting up the docker like inst/docker-dev or inst/docker-stable ? You might try the other way.

[daroczig]

Good point! I'm using the pretty outdated debian version, will check with a newer version and let you know what I've found.

[nfultz]

Just not too new, current ubuntu also has a broken libgrpc :(

[nfultz]

Also unclear at this point if the grpc cleanup code path is getting executed correctly - do you get this error only when the service is under load or does it do error out even when it's idle?

[daroczig]

It's been failing after ~20-25 mins of the last request, so while it's idle -- on the other hand, I've not tried the service for a longer time period with decent load. So far I've just started a dozen of Docker containers and tested with a few queries and then decided to leave those for the weekend -- when I found out about this issue. I will keep you posted if I find something useful (eg on grpc version).

[daroczig]

Sorry, did not have the time to look into this in more details -- meanwhile, I'm just restarting the containers every 15 mins as a workaround.