阅读 199

(实战篇)Java服务线上死锁日志怎么看?

前言

别嫌弃日志篇幅长哈, 只要是想让看到的同学可以完整的感受到死锁的排查过程.

一、死锁栈日志信息

Found one Java-level deadlock:
=============================
"pool-1-thread-5":
  waiting to lock monitor 0x00007efa20010788 (object 0x00000005f7800000, a com.alibaba.fastjson.util.ASMClassLoader),
  which is held by "catalina-exec-345"
"catalina-exec-345":
  waiting to lock monitor 0x00007efa5c0088a8 (object 0x00000005ed47db58, a org.apache.catalina.loader.WebappClassLoader),
  which is held by "catalina-exec-311"
"catalina-exec-311":
  waiting for ownable synchronizer 0x00000005ed54a980, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "catalina-exec-57"
"catalina-exec-57":
  waiting to lock monitor 0x00007efa5c0088a8 (object 0x00000005ed47db58, a org.apache.catalina.loader.WebappClassLoader),
  which is held by "catalina-exec-311"

Java stack information for the threads listed above:
===================================================
"catalina-exec-311":
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000005ed54a980> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
	at org.apache.skywalking.apm.agent.core.plugin.loader.InterceptorInstanceLoader.load(InterceptorInstanceLoader.java:70)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.<init>(InstMethodsInter.java:56)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.ClassEnhancePluginDefine.enhanceInstance(ClassEnhancePluginDefine.java:162)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.ClassEnhancePluginDefine.enhance(ClassEnhancePluginDefine.java:74)
	at org.apache.skywalking.apm.agent.core.plugin.AbstractClassEnhancePluginDefine.define(AbstractClassEnhancePluginDefine.java:75)
	at org.apache.skywalking.apm.agent.SkyWalkingAgent$Transformer.transform(SkyWalkingAgent.java:145)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Transformer$Compound.transform(AgentBuilder.java:2335)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$Transformation$Simple$Resolution.apply(AgentBuilder.java:8902)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:9306)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:9269)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1300(AgentBuilder.java:9047)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:9625)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:9575)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:9194)
	at sun.instrument.TransformerManager.transform(TransformerManager.java:188)
	at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:428)
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:3148)
	- locked <0x00000005ed47db58> (a org.apache.catalina.loader.WebappClassLoader)
	at org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:1388)
	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1876)
	- locked <0x00000005ed47db58> (a org.apache.catalina.loader.WebappClassLoader)
	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1750)
	at org.springframework.util.ClassUtils.forName(ClassUtils.java:250)
	at org.springframework.http.client.HttpComponentsClientHttpRequestFactory.<clinit>(HttpComponentsClientHttpRequestFactory.java:67)
	at com.lianjia.mls.data.processor.model.spring.RestTemplateFactoryBean.getObject(RestTemplateFactoryBean.java:33)
	at com.lianjia.mls.data.processor.model.spring.RestTemplateFactoryBean.getObject(RestTemplateFactoryBean.java:15)
	at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
	at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
	- locked <0x00000005f8602888> (a java.util.concurrent.ConcurrentHashMap)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
	at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1081)
	at com.lianjia.mls.data.processor.api.business.utils.SearchSpringUtils.getBean(SearchSpringUtils.java:25)
	at com.lianjia.mls.data.processor.api.business.utils.SearchApiUtils.search(SearchApiUtils.java:43)
	at com.lianjia.mls.data.processor.api.business.utils.SearchApiUtils.mapSearchResult(SearchApiUtils.java:128)
	at com.alliance.house.service.search.map.impl.DpMapSearchServiceImpl.getBubbleData(DpMapSearchServiceImpl.java:194)
	at com.alliance.house.service.search.map.impl.DpMapSearchServiceImpl?FastClassBySpringCGLIB?2ccc99c4.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
	at com.alliance.house.service.search.map.impl.DpMapSearchServiceImpl?EnhancerBySpringCGLIB?5d130a26.getBubbleData(<generated>)
	at com.alliance.house.ljweb.controller.app.search.AppMapSearchController.bubbleSearch$original$DZdTpsHH(AppMapSearchController.java:223)
	at com.alliance.house.ljweb.controller.app.search.AppMapSearchController.bubbleSearch$original$DZdTpsHH$accessor$JqYP12Bi(AppMapSearchController.java)
	at com.alliance.house.ljweb.controller.app.search.AppMapSearchController$auxiliary$YhLPQIyA.call(Unknown Source)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:98)
	at com.alliance.house.ljweb.controller.app.search.AppMapSearchController.bubbleSearch(AppMapSearchController.java)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest$original$WbsKYmi3(InvocableHandlerMethod.java:133)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest$original$WbsKYmi3$accessor$5EHWRHFG(InvocableHandlerMethod.java)
	at org.springframework.web.method.support.InvocableHandlerMethod$auxiliary$cHOWfir9.call(Unknown Source)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:98)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
	at javax.servlet.http.HttpServlet.service$dkimi$impl$1005(HttpServlet.java:624)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.ke.iprd.log.filter.HttpRequestFilter.doFilter(HttpRequestFilter.java:55)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:335)
	at com.dianping.cat.servlet.CatFilter$CatHandler$2.handle(CatFilter.java:122)
	at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:333)
	at com.dianping.cat.servlet.CatFilter$CatHandler$3.handle(CatFilter.java:139)
	at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:333)
	at com.dianping.cat.servlet.CatFilter$CatHandler$4.handle(CatFilter.java:269)
	at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:333)
	at com.dianping.cat.servlet.CatFilter$CatHandler$1.handle(CatFilter.java:75)
	at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:333)
	at com.dianping.cat.servlet.CatFilter.doFilter(CatFilter.java:41)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.alliance.house.ljweb.filter.AgainstCrawlerFilter.doFilter(AgainstCrawlerFilter.java:186)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.alliance.house.ljweb.filter.AccessTokenFilter.doFilter(AccessTokenFilter.java:84)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.alliance.house.ljweb.filter.ApiTokenFilter.doFilter(ApiTokenFilter.java:47)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.alliance.house.ljweb.filter.WebContextFilter.doFilter(WebContextFilter.java:41)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
	at org.apache.catalina.core.StandardHostValve.invoke$original$P961naY3(StandardHostValve.java:169)
	at org.apache.catalina.core.StandardHostValve.invoke$original$P961naY3$accessor$sKW1Xg7z(StandardHostValve.java)
	at org.apache.catalina.core.StandardHostValve$auxiliary$n9pFJr5w.call(Unknown Source)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:98)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1775)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1734)
	- locked <0x000000076e63c630> (a org.apache.tomcat.util.net.NioChannel)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)
"catalina-exec-57":
	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1784)
	- waiting to lock <0x00000005ed47db58> (a org.apache.catalina.loader.WebappClassLoader)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
	- locked <0x00000007a5a655e0> (a java.lang.Object)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)


==========================================

at org.apache.skywalking.apm.agent.core.plugin.loader.InterceptorInstanceLoader.load(InterceptorInstanceLoader.java:70)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.<init>(InstMethodsInter.java:56)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.ClassEnhancePluginDefine.enhanceInstance(ClassEnhancePluginDefine.java:162)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.ClassEnhancePluginDefine.enhance(ClassEnhancePluginDefine.java:74)
	at org.apache.skywalking.apm.agent.core.plugin.AbstractClassEnhancePluginDefine.define(AbstractClassEnhancePluginDefine.java:75)
	at org.apache.skywalking.apm.agent.SkyWalkingAgent$Transformer.transform(SkyWalkingAgent.java:145)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Transformer$Compound.transform(AgentBuilder.java:2335)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$Transformation$Simple$Resolution.apply(AgentBuilder.java:8902)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:9306)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:9269)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1300(AgentBuilder.java:9047)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:9625)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:9575)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:9194)


	at org.apache.skywalking.apm.agent.core.plugin.loader.InterceptorInstanceLoader.load(InterceptorInstanceLoader.java:77)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.StaticMethodsInter.intercept(StaticMethodsInter.java:73)
	at com.lianjia.common.rpc.client.transport.http.HttpClientUtils.post(HttpClientUtils.java)
	at com.lianjia.common.rpc.client.transport.http.HttpTransporter.doTransport(HttpTransporter.java:36)
	at com.lianjia.common.rpc.client.transport.AbstractTransporter.transport(AbstractTransporter.java:55)
	at com.lianjia.common.rpc.client.impl.DefaultStatement.call(DefaultStatement.java:62)
	at com.lianjia.hdic.api.client.http.template.TcpClientTemplate.service(TcpClientTemplate.java:122)
	at com.lianjia.hdic.api.client.http.template.TcpClientTemplate.callService(TcpClientTemplate.java:94)
	at com.lianjia.hdic.api.client.http.TcpProvider.getCitys(TcpProvider.java:51)
	at com.alliance.house.service.impl.HdicServiceImpl.getCityByOfficeAddress(HdicServiceImpl.java:173)
	at com.alliance.house.service.impl.HdicServiceImpl?FastClassBySpringCGLIB?b7650762.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
	at com.alliance.house.service.impl.CacheableHdicService?EnhancerBySpringCGLIB?f45f621a.getCityByOfficeAddress(<generated>)
	at com.alliance.house.ljweb.controller.app.AppUserController.getWorkCities$original$xLcCm5Qp(AppUserController.java:69)
	at com.alliance.house.ljweb.controller.app.AppUserController.getWorkCities$original$xLcCm5Qp$accessor$pUlQ77mc(AppUserController.java)
	at com.alliance.house.ljweb.controller.app.AppUserController$auxiliary$T2QOK0NL.call(Unknown Source)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:98)
	at com.alliance.house.ljweb.controller.app.AppUserController.getWorkCities(AppUserController.java)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest$original$WbsKYmi3(InvocableHandlerMethod.java:133)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest$original$WbsKYmi3$accessor$5EHWRHFG(InvocableHandlerMethod.java)
	at org.springframework.web.method.support.InvocableHandlerMethod$auxiliary$cHOWfir9.call(Unknown Source)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:98)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
	at javax.servlet.http.HttpServlet.service$dkimi$impl$1005(HttpServlet.java:624)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.ke.iprd.log.filter.HttpRequestFilter.doFilter(HttpRequestFilter.java:55)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:335)
	at com.dianping.cat.servlet.CatFilter$CatHandler$2.handle(CatFilter.java:122)
	at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:333)
	at com.dianping.cat.servlet.CatFilter$CatHandler$3.handle(CatFilter.java:139)
	at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:333)
	at com.dianping.cat.servlet.CatFilter$CatHandler$4.handle(CatFilter.java:269)
	at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:333)
	at com.dianping.cat.servlet.CatFilter$CatHandler$1.handle(CatFilter.java:75)
	at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:333)
	at com.dianping.cat.servlet.CatFilter.doFilter(CatFilter.java:41)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.alliance.house.ljweb.filter.WebContextFilter.doFilter(WebContextFilter.java:41)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
	at org.apache.catalina.core.StandardHostValve.invoke$original$P961naY3(StandardHostValve.java:169)
	at org.apache.catalina.core.StandardHostValve.invoke$original$P961naY3$accessor$sKW1Xg7z(StandardHostValve.java)
	at org.apache.catalina.core.StandardHostValve$auxiliary$n9pFJr5w.call(Unknown Source)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:98)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1775)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1734)
	- locked <0x000000076de69c18> (a org.apache.tomcat.util.net.NioChannel)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

Found 1 deadlock.
复制代码

二、出问题的代码

skywalking中的代码, 感兴趣可以自己load下来去看.

public class InterceptorInstanceLoader {

    private static ConcurrentHashMap<String, Object> INSTANCE_CACHE = new ConcurrentHashMap<String, Object>();
    private static ReentrantLock INSTANCE_LOAD_LOCK = new ReentrantLock();
    private static Map<ClassLoader, ClassLoader> EXTEND_PLUGIN_CLASSLOADERS = new HashMap<ClassLoader, ClassLoader>();

    /**
     * Load an instance of interceptor, and keep it singleton.
     * Create {@link AgentClassLoader} for each targetClassLoader, as an extend classloader.
     * It can load interceptor classes from plugins, activations folders.
     *
     * @param className         the interceptor class, which is expected to be found
     * @param targetClassLoader the class loader for current application context
     * @param <T>               expected type
     * @return the type reference.
     * @throws IllegalAccessException
     * @throws InstantiationException
     * @throws ClassNotFoundException
     * @throws AgentPackageNotFoundException
     */
    public static <T> T load(String className, ClassLoader targetClassLoader)
            throws IllegalAccessException, InstantiationException, ClassNotFoundException, AgentPackageNotFoundException {
        if (targetClassLoader == null) {
            targetClassLoader = InterceptorInstanceLoader.class.getClassLoader();
        }
        String instanceKey = className + "_OF_" + targetClassLoader.getClass().getName() + "@" + Integer.toHexString(targetClassLoader.hashCode());
        Object inst = INSTANCE_CACHE.get(instanceKey);
        if (inst == null) {
            INSTANCE_LOAD_LOCK.lock();  // 锁 一
            try {
                ClassLoader pluginLoader = EXTEND_PLUGIN_CLASSLOADERS.get(targetClassLoader);
                if (pluginLoader == null) {
                    pluginLoader = new AgentClassLoader(targetClassLoader);
                    EXTEND_PLUGIN_CLASSLOADERS.put(targetClassLoader, pluginLoader);
                }
            inst = Class.forName(className, true, pluginLoader).newInstance(); // 锁 二
            } finally {
                INSTANCE_LOAD_LOCK.unlock();
            }
            if (inst != null) {
                INSTANCE_CACHE.put(instanceKey, inst);
            }
        }

        return (T) inst;
    }
}
复制代码

三、分析

Found one Java-level deadlock:
=============================
"pool-1-thread-5":
  waiting to lock monitor 0x00007efa20010788 (object 0x00000005f7800000, a com.alibaba.fastjson.util.ASMClassLoader),
  which is held by "catalina-exec-345"
"catalina-exec-345":
  waiting to lock monitor 0x00007efa5c0088a8 (object 0x00000005ed47db58, a org.apache.catalina.loader.WebappClassLoader),
  which is held by "catalina-exec-311"
"catalina-exec-311":
  waiting for ownable synchronizer 0x00000005ed54a980, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "catalina-exec-57"
"catalina-exec-57":
  waiting to lock monitor 0x00007efa5c0088a8 (object 0x00000005ed47db58, a org.apache.catalina.loader.WebappClassLoader),
  which is held by "catalina-exec-311"

Java stack information for the threads listed above:
===================================================
复制代码

从上面死锁信息可以看出,线程311和线程57在互相等待,引起死锁。

1)311 持有 监视器锁,在等待可重入锁。

2)57 持有可重入锁,在等待监视器锁。

具体看代码,实际上加锁顺序是固定的,看不出来会有死锁的场景,正当我死活看不出死锁原因的时候,同事救了我。

仔细看311线程的线程栈信息,可以看到,线程栈在获取 可重入锁 之前已经获取到 监视器锁 , 如下:

- locked <0x00000005ed47db58> (a org.apache.catalina.loader.WebappClassLoader) 
复制代码

终于明白了,在问题代码段之前有个地方已经锁住了 WebappClassLoader 对象了。

四、反思

  • 线程栈日志会将加锁的过程体现出来 !
  • 看日志还是要仔细 !