MailHealthIndicator causes too many tcp connections in TIME_WAIT state

[samsong8610]

Environment:

Spring-Boot version: 1.4.2.RELEASE
java: 1.8.0-openjdk
os: Linux xxxx 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 23 17:05:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

I ran a spring-boot application using MailSender with actuator enabled. There are two instances running on two separate hosts after a load balancer. The load balancer checks the status of the two upstream instances by HTTP-GET the /health endpoint.
After running perfectly for about a week more or less, The load balancer notifies that the health checking is failed. I check it manually by curl -i -XGET http://host-ip-address/health, it blocks without nothing returned.
Meanwhile, I find too many connections in TIME_WAIT state between the instance and the mail server, about 3000+. I use jstack printing the jvm stack. There are many thread waiting for some response from the mail server. One of these stack frame is like:

"http-nio-80-exec-200" #224 daemon prio=5 os_prio=0 tid=0x00007fcdb4019800 nid=0xa64 runnable [0x00007fcd98a84000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:170)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at com.sun.mail.util.TraceInputStream.read(TraceInputStream.java:124)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
        - locked <0x00000000ee8c7490> (a java.io.BufferedInputStream)
        at com.sun.mail.util.LineInputStream.readLine(LineInputStream.java:92)
        at com.sun.mail.smtp.SMTPTransport.readServerResponse(SMTPTransport.java:2349)
        at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2096)
        at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:712)
        - locked <0x00000000ee8c2f18> (a com.sun.mail.smtp.SMTPTransport)
        at javax.mail.Service.connect(Service.java:366)
        - locked <0x00000000ee8c2f18> (a com.sun.mail.smtp.SMTPTransport)
        at org.springframework.mail.javamail.JavaMailSenderImpl.connectTransport(JavaMailSenderImpl.java:501)
        at org.springframework.mail.javamail.JavaMailSenderImpl.testConnection(JavaMailSenderImpl.java:382)
        at org.springframework.boot.actuate.health.MailHealthIndicator.doHealthCheck(MailHealthIndicator.java:40)
        at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:38)
        at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68)
        at org.springframework.boot.actuate.endpoint.HealthEndpoint.invoke(HealthEndpoint.java:81)
        at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.getHealth(HealthMvcEndpoint.java:166)
        at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke(HealthMvcEndpoint.java:140)
        at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:220)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:134)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
        at javax.servlet.http.HttpServlet.doHead(HttpServlet.java:244)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at org.springframework.web.servlet.resource.ResourceUrlEncodingFilter.doFilterInternal(ResourceUrlEncodingFilter.java:53)
...
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        - locked <0x00000000ec4de5f8> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)

I think this may be a bug of the MailHealthIndicator, although it just delegates to org.springframework.mail.javamail.JavaMailSenderImpl.testConnection() for health checking.

[snicoll]

Hard to tell but you are in control of configuring the JavaMailSender and perhaps you should have more agressive timeout settings, something like

spring.mail.properties.mail.smtp.timeout=2000
spring.mail.properties.mail.smtp.connectiontimeout=2000

Be aware that the defaults are infinite so if your mail server does not reply in a timely manner, you'll have thread waiting to connect indefinitely.

Please give that a try. We should probably update the documentation to add an explicit note about it

[samsong8610]

I also verify the method. It can solve the too many connections opened issue.
BTW, if you don't want to check the mail server health, use management.health.mail.enabled=false to disable that.