【服务离线】记一次组件引入造成actuator下线问题

SpringBoot Admin可以看作是Spring actuator 端点监控的一个UI界面，可以很方便的查看服务的运行情况。

部分服务不在线

如上图，在项目整合了部分组件后，出现状态显示不在线，但应用还是能够正常的对外提供服务。

• 我们知道 应用展示的在线状态是读取的 Spring actuator health endpoint数据

curl http://127.0.0.1:5002/actuator/health

{"status":"DOWN"}  

• 此时无法通过 SpringBoot Admin 无法获取更多的细节, 建议开发环境开启端点的详细信息。

management:
  endpoints:
    web:
      exposure:
        include: '*'
  endpoint:
    health:
      show-details: always  #dev open

• 再次尝试访问, 输出了全部组件的状态，可以明确的看到由于 sentinel 状态不对影响全局状态为 down

curl http://127.0.0.1:5002/actuator/health

{"status":"DOWN","components":{"discoveryComposite":{"status":"UP","components":{"discoveryClient":{"status":"UP","details":{"services":["pigx-upms-biz","pigx-tx-manager","pigx-daemon-elastic-job","pigx-mp-platform","pigx-daemon-quartz","pigx-gateway","pigx-auth","pigx-pay-platform","pigx-codegen","pigx-oa-platform","pigx-monitor"]}}}},"diskSpace":{"status":"UP","details":{"total":42927636480,"free":21334122496,"threshold":10485760}},"nacosConfig":{"status":"UP"},"nacosDiscovery":{"status":"UP"},"ping":{"status":"UP"},"reactiveDiscoveryClients":{"status":"UP","components":{"Simple Reactive Discovery Client":{"status":"UP","details":{"services":[]}}}},"redis":{"status":"UP","details":{"version":"5.0.7"}},"refreshScope":{"status":"UP"},"sentinel":{"status":"DOWN","details":{"dataSource":{},"enabled":true,"dashboard":{"description":"pigx-sentinel:5020 can't be connected","status":"DOWN"}}}}}

sentinel dashboadr can't be connected

• SentinelHealthIndicator.doHealthCheck 来看下 sentinel 的健康检查是如何处理的

protected void doHealthCheck(Health.Builder builder) throws Exception {
	Map<String, Object> detailMap = new HashMap<>();

		// 获取心跳发送器
		HeartbeatSender heartbeatSender = HeartbeatSenderProvider
				.getHeartbeatSender();
		// 发送心跳
		boolean result = heartbeatSender.sendHeartbeat();
		if (result) {
		   // 成功标记为UP
			detailMap.put("dashboard", Status.UP);
		}
		else {
		    // 失败 标记为down
			dashboardUp = false;
			detailMap.put("dashboard", new Status(Status.DOWN.getCode(),
					consoleServer + " can't be connected"));
		}
	}

	// If Dashboard and DataSource are both OK, the health status is UP
	if (dashboardUp && dataSourceUp) {
		builder.up().withDetails(detailMap);
	}
	else {
		builder.down().withDetails(detailMap);
	}
}

• SimpleHttpHeartbeatSender.sendHeartbeat 心跳发送器的处理逻辑

public boolean sendHeartbeat() throws Exception {
	
	// 1. 判断客户端是和服务端是否交互 ，只有交互时dashboard 才有数据展示
	if (TransportConfig.getRuntimePort() <= 0) {
		RecordLog.info("[SimpleHttpHeartbeatSender] Runtime port not initialized, won't send heartbeat");
		return false;
	}
	InetSocketAddress addr = getAvailableAddress();
	if (addr == null) {
		return false;
	}
	
	// 2. 访问dashboard 的接口
	SimpleHttpRequest request = new SimpleHttpRequest(addr, TransportConfig.getHeartbeatApiPath());
	request.setParams(heartBeat.generateCurrentMessage());
	SimpleHttpResponse response = httpClient.post(request);
	if (response.getStatusCode() == OK_STATUS) {
		return true;
	}
	return false;
}

• 到底是上文代码 第一步未交互被判断为down、还是第二步访问出错判断为down

查看 sentinel client 的日志

cat ~/csp/sentinel-record.log 

2020-03-23 13:54:04.634 INFO [SimpleHttpHeartbeatSender] Runtime port not initialized, won't send heartbeat  

开始是第一步没有初始化交互造成，服务判断为 down

sentinel 如何初始化交互

• 初始化交互端口 commandCenter.start()

public class CommandCenterInitFunc implements InitFunc {

    @Override
    public void init() throws Exception {
        CommandCenter commandCenter = CommandCenterProvider.getCommandCenter();

        if (commandCenter == null) {
            RecordLog.warn("[CommandCenterInitFunc] Cannot resolve CommandCenter");
            return;
        }

        commandCenter.beforeStart();
        // 开始初始化端口交互
        commandCenter.start();
        RecordLog.info("[CommandCenterInit] Starting command center: "
                + commandCenter.getClass().getCanonicalName());
    }
}

• 配置启动立即加载初始化，而不是有流量请求再初始化

public class SentinelAutoConfiguration {

	@Value("${project.name:${spring.application.name:}}")
	private String projectName;

	@Autowired
	private SentinelProperties properties;

	@PostConstruct
	private void init() {

		// earlier initialize
		if (properties.isEager()) {
			InitExecutor.doInit();
		}

	}

spring:
  application:
    name: @artifactId@
  cloud:
    sentinel:
      eager: true