Permalink Submitted by Tom Burton-West on February 23, 2011
We couldn't find any correlations with high QPS and/or concurrent users. We tend to get rates of less than 1 QPS. It appears that there were at least two causes. The first was a problem with the drivers for our network cards that resulted in dropped packets. Once that was resolved, the number of slow queries was reduced. The second cause appears to be long full garbage collections by the JVM. When we looked at the GC logs we noticed periodic stop-the-world garbage collections that took around 30 seconds. We recently made some changes that reduced memory consumption significantly and have not seen those large garbage collection times. (See http://www.hathitrust.org/blogs/large-scale-search/too-many-words-again)
We also plan to change to the concurrent garbage collector after we run some tests but we haven't implemented this yet. I still need to do some further log analysis to be sure that this has solved the problem.
re: slow response time
We couldn't find any correlations with high QPS and/or concurrent users. We tend to get rates of less than 1 QPS. It appears that there were at least two causes. The first was a problem with the drivers for our network cards that resulted in dropped packets. Once that was resolved, the number of slow queries was reduced. The second cause appears to be long full garbage collections by the JVM. When we looked at the GC logs we noticed periodic stop-the-world garbage collections that took around 30 seconds. We recently made some changes that reduced memory consumption significantly and have not seen those large garbage collection times. (See http://www.hathitrust.org/blogs/large-scale-search/too-many-words-again)
We also plan to change to the concurrent garbage collector after we run some tests but we haven't implemented this yet. I still need to do some further log analysis to be sure that this has solved the problem.
Tom