阅读 92

redis 持久化使用 rdb引发oom的持久化策略思考

平常我们使用的都是tomcat服务器,由于tomcat采用epoll+多线程的方式调度。每个线程同一时刻调度一个请求,由redis把请求写入一个rdb。这样会生成多个rdb。公司使用jetty容器,单线程在每一次请求时不断向redis发送请求,再加上redis本身也是单线程的中间件,持续的写入日志会增大rdb快照的体积,从而增加内存。

报错:

HTTP ERROR 500
Problem accessing /settle/. Reason:

    Server Error
Caused by:
org.springframework.dao.InvalidDataAccessApiUsageException: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.; nested exception is redis.clients.jedis.exceptions.JedisDataException: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
	at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:44)
	at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:36)
	at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:37)
	at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:37)
	at org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException(JedisConnection.java:210)
	at org.springframework.data.redis.connection.jedis.JedisConnection.hMSet(JedisConnection.java:2652)
	at org.springframework.data.redis.core.DefaultHashOperations$7.doInRedis(DefaultHashOperations.java:134)
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:191)
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:153)
	at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:86)
	at org.springframework.data.redis.core.DefaultHashOperations.putAll(DefaultHashOperations.java:131)
	at org.springframework.data.redis.core.DefaultBoundHashOperations.putAll(DefaultBoundHashOperations.java:85)
	at org.springframework.session.data.redis.RedisOperationsSessionRepository$RedisSession.saveDelta(RedisOperationsSessionRepository.java:778)
	at org.springframework.session.data.redis.RedisOperationsSessionRepository$RedisSession.access$000(RedisOperationsSessionRepository.java:670)
	at org.springframework.session.data.redis.RedisOperationsSessionRepository.save(RedisOperationsSessionRepository.java:388)
	at org.springframework.session.data.redis.RedisOperationsSessionRepository.save(RedisOperationsSessionRepository.java:245)
	at org.springframework.session.web.http.SessionRepositoryFilter$SessionRepositoryRequestWrapper.commitSession(SessionRepositoryFilter.java:244)
	at org.springframework.session.web.http.SessionRepositoryFilter$SessionRepositoryRequestWrapper.access$100(SessionRepositoryFilter.java:214)
	at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:167)
	at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1631)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:568)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:462)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
	at java.lang.Thread.run(Thread.java:748)
Caused by: redis.clients.jedis.exceptions.JedisDataException: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
	at redis.clients.jedis.Protocol.processError(Protocol.java:127)
	at redis.clients.jedis.Protocol.process(Protocol.java:161)
	at redis.clients.jedis.Protocol.read(Protocol.java:215)
	at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340)
	at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239)
	at redis.clients.jedis.BinaryJedis.hmset(BinaryJedis.java:886)
	at org.springframework.data.redis.connection.jedis.JedisConnection.hMSet(JedisConnection.java:2650)
	... 35 more
Caused by:
redis.clients.jedis.exceptions.JedisDataException: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
	at redis.clients.jedis.Protocol.processError(Protocol.java:127)
	at redis.clients.jedis.Protocol.process(Protocol.java:161)
	at redis.clients.jedis.Protocol.read(Protocol.java:215)
	at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340)
	at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239)
	at redis.clients.jedis.BinaryJedis.hmset(BinaryJedis.java:886)
	at org.springframework.data.redis.connection.jedis.JedisConnection.hMSet(JedisConnection.java:2650)
	at org.springframework.data.redis.core.DefaultHashOperations$7.doInRedis(DefaultHashOperations.java:134)
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:191)
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:153)
	at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:86)
	at org.springframework.data.redis.core.DefaultHashOperations.putAll(DefaultHashOperations.java:131)
	at org.springframework.data.redis.core.DefaultBoundHashOperations.putAll(DefaultBoundHashOperations.java:85)
	at org.springframework.session.data.redis.RedisOperationsSessionRepository$RedisSession.saveDelta(RedisOperationsSessionRepository.java:778)
	at org.springframework.session.data.redis.RedisOperationsSessionRepository$RedisSession.access$000(RedisOperationsSessionRepository.java:670)
	at org.springframework.session.data.redis.RedisOperationsSessionRepository.save(RedisOperationsSessionRepository.java:388)
	at org.springframework.session.data.redis.RedisOperationsSessionRepository.save(RedisOperationsSessionRepository.java:245)
	at org.springframework.session.web.http.SessionRepositoryFilter$SessionRepositoryRequestWrapper.commitSession(SessionRepositoryFilter.java:244)
	at org.springframework.session.web.http.SessionRepositoryFilter$SessionRepositoryRequestWrapper.access$100(SessionRepositoryFilter.java:214)
	at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:167)
	at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1631)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:568)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:462)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
	at java.lang.Thread.run(Thread.java:748)
复制代码

场景 : jetty将每一次请求加入上下文

            Server server = new Server(port);
            System.out.println("Begin start jetty...");
            HandlerCollection hc = new HandlerCollection();
            WebAppContext webapp = new WebAppContext();
            webapp.setContextPath(contentPath);
            File webappdir = new File(new File(webprjdir), "src/main/webapp");
            //System.out.println(webappdir.getCanonicalPath());
            if (!webappdir.exists()) {
                throw new RuntimeException("目录" + webprjdir + "下不存在src/main/webapp");
            }
            webapp.setWar(webappdir.getAbsolutePath().replace("%20", ""));
            InputStream fis = RunAppHelper.class.getResourceAsStream("/jettyetc/webdefault.xml");
            File tempfie = new File("webdefault.xml");

            FileOutputStream fos = new FileOutputStream(tempfie);
            IOUtils.copy(fis, fos);
            fos.close();

            String deffile = tempfie.getAbsolutePath().replace("%20", " ");
            webapp.setDefaultsDescriptor(deffile);
            webapp.setInitParameter("org.eclipse.jetty.servlet.Default.dirAllowed", "false");
            webapp.setInitParameter("org.eclipse.jetty.servlet.Default.useFileMappedBuffer", "false");
            hc.addHandler(webapp);
复制代码

根据spring的拦截流程,handlerServlet拦截到HttpServlet

public class RemoteInvokeServlet extends HttpServlet {
@Override
    protected void doPost(HttpServletRequest req, HttpServletResponse resp)
        throws ServletException, IOException {

        ServiceContext ctx = new ServiceContext();
        //        Map<String, Object> result = new HashMap<>();
        ProxyResponse result = null;
        TraceLogger traceLogger = traceLogger = TraceLogger.get();
        traceLogger.initialize();
        try {
            ServiceContext.set(ctx);
            ProxyRequest obj = (ProxyRequest) SerializeUtils.hessianDeserialize(req.getInputStream());

            ctx.setCodePrefix(obj.getCodePrefix());
            ctx.setSid(obj.getSid());
            ctx.setUserId(obj.getUserId());
            ctx.setOrgId(obj.getOrgId());
            ctx.setProject(obj.getProject());

            //类
            Class port = Class.forName(obj.getClassName());
            Method method = port.getMethod(obj.getMethodName(), obj.getParamTypes());
            logger.debug("Begion invoke service " + obj.getClassName() + "." + obj.getMethodName()
                + ", args:" + ArrayUtils.toString(obj.getArgs(), "[]"));
            Object target = springctx.getBean(port);

            // 记录请求日志
            Object robj;
            traceLogger.action(port.getSimpleName() + "-" + method.getName());
            traceLogger.requestId(UUID.randomUUID().toString());
            logger.debug("=== begin request processing ===");
            RequestWrapper requestWrapper = new RequestWrapper(req);
            requestWrapper.initialize();
            logRequest(requestWrapper, traceLogger);
            robj = method.invoke(target, obj.getArgs());

            result = ProxyResponse.success(robj, ServiceContext.get().getDatas());
        } catch (Exception e) {
            result = ProxyResponse.error(getErrorMessage(e));
            logger.error("Invoke remote service error", e);
        } finally {
            logResponse(resp, traceLogger);
            logger.debug("=== finish request processing ===");
            traceLogger.cleanup();
            //移除上下文
            ServiceContext.remove();
            SerializeUtils.hessianSerialize(result, resp.getOutputStream());
        }
    }
}
复制代码

发现在进行远程调用之前,经过了这个过滤器的生命周期

aof重写机制

AOF的重写机制 前面也说到了,AOF的工作原理是将写操作追加到文件中,文件的冗余内容会越来越多。所以聪明的 Redis 新增了重写机制。当AOF文件的大小超过所设定的阈值时,Redis就会对AOF文件的内容压缩。

解决方法

我们观察到服务端日志中,不断fork子进程。这明显是因为Redis的aof刷盘策略会fork出一条新进程,读取内存中的数据,并重新写到一个临时文件中。并没有读取旧文件。最后替换旧的aof文件。

阅读配置文件说明,找到了这个属性。由于我们的aof重写需要间隔时间,关闭appendfsync只是意味着将文件的写操作关闭,但是redis还是会fork出一个新进程把内存中的数据写入文件,这个时候虽然没有写操作,可是以前的文件重复备份造成内存堆积,异步刷盘自然就超过阈值了