서버 운영환경은 single서버에서 두개의 톰캣 인스턴스를 운영하며, 각각의 톰캣에는 solr 인스턴스가 2,3개씩 운영되고 있었다. 또한 각각의 solr 인스턴스는 다수개의(4~6개) solr core(shards)를 포함하고 있었다.
사용자가 질의를 요청하면 대부분의 경우 모든 solr core에 distribute search를 요청하게 되는데, 에러가 발생한 경우의 톰캣 로그는 아래와 같다.
2014-01-13 12:23:41,814 [http-8080-12] ERROR org.apache.solr.servlet.SolrDispatchFilter - null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://solr07:8100/solr_ep/epp03
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:298)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1561)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:446)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:267)
at org.apache.solr.servlet.PipcDispatchFilter.doFilter(PipcDispatchFilter.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://solr07:8100/solr_ep/epp03
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:409)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:182)
at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:165)
at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:132)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
... 1 more
Caused by: org.apache.http.NoHttpResponseException: The target server failed to respond
at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:101)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:252)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:282)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:247)
at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:216)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:298)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:647)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:464)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:351)
... 11 more
톰캣 설정이 잘못되었을 것이라는 추측하에 톰캣 connector에 관한 설정을 보던중 keepAliveTimeout 에 대한 설정이 정의가 되지 않을 경우 default로 connectionTimeout 설정의 값을 사용한다는 걸 알았다.
The number of milliseconds this Connector will wait for another HTTP request before closing the connection. The default value is to use the value that has been set for the connectionTimeout attribute.사용하고 있던 톰캣의 설정에는 keepAliveTimeout는 설정되지 않았으며, connectionTimeout만 60000ms으로 설정되어 있었다. 즉, keepAliveTimeout이 설정되지 않다보니 connectionTimeout의 설정값인 60000ms를 사용하게 되었고, 이는 thread가 종료되기까지 많은 시간을 소요하게 되어 thread의 수를 증가시켰다.
connectionTimeout의 값을 20000ms로 낮추고 운영한 결과 현재까지는 위의 오류가 발생되지 않고 있다.
