Istio Mixer Adapter开发 （三）自定义Mixer Grpc Adapter部署

码农爱唱歌
1,628 阅读10分钟

系列导航

概述

针对Istio Mixer Adapter的开发，后面会有文章专门讲述源码分析及开发调试的过程，这里仅使用前两篇文章讲述的K8S+Istio环境部署我已经开发好的Istio Mixer Grpc Adapter并配置K8S使其生效

gatewaymetric简介

gatewaymetric是笔者借鉴Istio官网指导开发的Istio Mixer Grpc Adapter，他的功能主要是处理Mixer上报的http访问日志，收集南北向流量（指通过istio-ingressgateway从集群外进入到集群）的访问日志，通过调用promethues官方go程序库，内存生成metrics并暴露exporter端口给promethues采集数据，它的功能目前还很不完善，但是作为学习说明已经足够

本文主要参考官网Wiki进行开发，里面也讲到了部分原理

github.com/istio/istio…

下面是代码结构

➜  gatewaymetric git:(v1.0.5) ✗ tree
.
├── cmd
│   └── main.go
├── config
│   ├── adapter.gatewaymetric.config.pb.html
│   ├── config.pb.go
│   ├── config.proto
│   ├── config.proto_descriptor
│   └── gatewaymetric.yaml
├── gatewaymetric.go
├── sample_operator_cfg.yaml
└── testdata
    ├── attributes.yaml
    ├── gatewaymetric.yaml
    ├── sample_operator_cfg.yaml
    └── template.yaml

上述文件中cmd/main.go为程序入口，其暴露两个端口
1）http:9145为promethues的数据抓取端口
2）grpc:41543为GrpcAdapter的通信端口，也就是与Mixer建立连接的端口，这个后面会用到，要把它显示注册到Mixer当中，使Mixer感知它

构建自定义Mixer Grpc Adapter（gatewaymetric）

我们第一步需要打包这个Adapter并制作成Docker镜像发布到Docker Hub中

➜  cmd git:(v1.0.5) ✗ cd /home/ubuntu/go/src/istio.io/istio/mixer/adapter/gatewaymetric/cmd
➜  cmd git:(master) CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o gatewaymetric
➜  cmd git:(v1.0.5) ✗ CGO_ENABLE➜  cmd git:(v1.0.5) ✗ ./gatewaymetric 
adapter listening on "[::]:41543"
promethues listening on ":9145"
D=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o gatewaymetric

通过上述命令，我们基于源码构建了一个我们定制的GrpcAdapter并且成功启动了它，如果读者无法正常执行结束go build命令，则可能是go环境配置的问题，可以绕开这里直接使用源码cmd下面的gatewaymetric文件即可，笔者已经构建好了二进制文件，接下来我们编写Dockerfile

➜  cmd git:(v1.0.5) ✗ touch Dockerfile
➜  cmd git:(v1.0.5) ✗ vim Dockerfile
FROM scratch
ADD gatewaymetric /
ENTRYPOINT ["/gatewaymetric"]

该Dockerfile使用scratch基础镜像并将gatewaymetirc（我们自定义的Mixer Grpc Adapter后面都会以此称呼）加入到镜像当中，启动入口为"/gatewaymetric"
下面开始构建

➜  cmd git:(v1.0.5) ✗ sudo docker build -f Dockerfile .
Sending build context to Docker daemon  17.87MB
Step 1/3 : FROM scratch
 ---> 
Step 2/3 : ADD gatewaymetric /
 ---> 82ca9cdf54ab
Step 3/3 : ENTRYPOINT ["/gatewaymetric"]
 ---> Running in 2630beb8da5d
Removing intermediate container 2630beb8da5d
 ---> 3747dfc14d90
Successfully built 3747dfc14d90
➜  cmd git:(v1.0.5) ✗ sudo docker run -p 9145:9145 -p 41543:41543 3747dfc14d90
adapter listening on "[::]:41543"
promethues listening on ":9145"

通过模拟程序测试通过（这里使用的是前面提到的Wiki当中描述的本地模拟测试的办法），说明打的Docker镜像没有问题，若读者还没有了解这块的知识，忽略这里即可，后面文章会详细描述如何在本地开发及调试Adapter

接下来，我们将该镜像提交到Docker Hub（这里讲述了开发构建并推送镜像的过程，读者可以跳过此步骤直接使用笔者已经构建好并发布到docker hub上的镜像即可）

➜  cmd git:(v1.0.5) ✗ sudo docker tag 3747dfc14d90 zrbcool/gatewaymetric:v1.0  
➜  cmd git:(v1.0.5) ✗ sudo docker push zrbcool/gatewaymetric:v1.0            
The push refers to repository [docker.io/zrbcool/gatewaymetric]
115a048ee88f: Pushed 
v1.0: digest: sha256:616fc9e1d91ad963a5ace4b5210caecbde21aca8ea5f2bfe8f98ac20e33a6cfa size: 528

部署Adapter到K8S + Istio集群

到K8S + Istio集群的VM中，部署已经制作好的镜像

➜  gatewaymetric cat deployment.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: gatewaymetric
  namespace: istio-system
  labels:
    app: gatewaymetric
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gatewaymetric
  template:
    metadata:
      labels:
        app: gatewaymetric
    spec:
      containers:
      - name: gatewaymetric
        image: "zrbcool/gatewaymetric:v1.0"
        imagePullPolicy: "IfNotPresent"
        ports:
        - containerPort: 9145
        - containerPort: 41543

然后创建Service

➜  gatewaymetric cat service.yaml 
apiVersion: v1
kind: Service
metadata:
  labels:
    app: gatewaymetric
  name: gatewaymetric
  namespace: istio-system
spec:
  selector:
    app: gatewaymetric
  ports:
  type: ClusterIP
  ports:
  - name: promethues
    port: 9145
    targetPort: 9145
  - name: grpc
    port: 41543
    targetPort: 41543

简单验证，端口已通，promethues的metrics也可以获取

➜  gatewaymetric kubectl get svc -n istio-system
NAME                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)           
gatewaymetric            ClusterIP   10.104.182.140   <none>        9145/TCP,41543/TCP     
➜  gatewaymetric telnet 10.104.182.140 41543
Trying 10.104.182.140...
Connected to 10.104.182.140.
Escape character is '^]'.
➜  gatewaymetric curl 10.104.182.140:9145/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
...

至此，我们的Mixer Grpc Adapter已经部署到了我们的K8S集群当中，接下来我们需要配置CRD，使mixer能够感知到我们的Grpc Adapter
部署下面文件的内容，其包含了自定义Adapter的运行时实例handler，template->logentity的运行时实例instance，以及mixer的匹配规则rule指定handler处理instance的Report数据，如果上面这句话看的有点蒙，不要紧这个都是Mixer的设计者抽象出来的概念，目的就是让Mixer适配器的开发者跟运维者分离并使Mixer Adapter的开发更加灵活，后面会有专门章节详细阐述其各个模型的意义，如果读者想提前了解，请参考下面链接：

istio.io/docs/concep…

# handler for adapter gatewaymetric
apiVersion: "config.istio.io/v1alpha2"
kind: handler
metadata:
 name: h1
 namespace: istio-system
spec:
 adapter: gatewaymetric
 connection:
   address: "gatewaymetric.istio-system.svc.cluster.local:41543" #replaces at runtime by the test
 params:
   file_path: "out.txt"
---

# instance for template metric
apiVersion: "config.istio.io/v1alpha2"
kind: instance
metadata:
  name: i1metric
  namespace: istio-system
spec:
  template: logentry
  params:
    severity: '"warning"'
    timestamp: request.time
    variables:
      host: request.host
      method: request.method | ""
      url: request.path | ""
      responseCode: response.code | 0
      responseBodySize: response.size | 0
      responseTotalSize: response.total_size | 0
      latency: response.duration | "0ms"
    monitored_resource_type: '"UNSPECIFIED"'
---

# rule to dispatch to handler h1
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
  name: r1
  namespace: istio-system
spec:
  actions:
    - handler: h1.istio-system
      instances:
        - i1metric
---

在Kubernetes中部署它们

➜  gatewaymetric kubectl apply -f attributes.yaml 
attributemanifest.config.istio.io/istio-proxy created
attributemanifest.config.istio.io/kubernetes configured
➜  gatewaymetric kubectl apply -f template.yaml  
template.config.istio.io/logentry created
➜  gatewaymetric kubectl apply -f gatewaymetric.yaml 
adapter.config.istio.io/gatewaymetric created
➜  gatewaymetric kubectl apply -f operator_cfg.yaml 
handler.config.istio.io/h1 created
instance.config.istio.io/i1metric created
rule.config.istio.io/r1 created

使用浏览器访问http://192.168.101.6:31380/productpage，并查看Mixer的日志，看到

{"level":"info","time":"2019-02-21T03:05:40.292749Z","instance":"accesslog.logentry.istio-system","apiClaims":"","apiKey":"","clientTraceId":"","connection_security_policy":"unknown","destinationApp":"","destinationIp":"10.32.0.34","destinationName":"","destinationNamespace":"","destinationOwner":"","destinationPrincipal":"","destinationServiceHost":"productpage.default.svc.cluster.local","destinationWorkload":"","httpAuthority":"192.168.101.6:31380","latency":"25.101758ms","method":"GET","protocol":"http","receivedBytes":488,"referer":"","reporter":"source","requestId":"405ed7ca-c988-4cfa-8e6a-e808e2cbeab8","requestSize":0,"requestedServerName":"","responseCode":200,"responseSize":5719,"responseTimestamp":"2019-02-21T03:05:40.317790Z","sentBytes":5858,"sourceApp":"","sourceIp":"0.0.0.0","sourceName":"","sourceNamespace":"istio-system","sourceOwner":"","sourcePrincipal":"","sourceWorkload":"","url":"/productpage","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36","xForwardedFor":"10.32.0.1"}

查看我们自定义的Grpc Adapter日志，看到

2019-02-21T03:05:41.002442Z	info	received request {[&InstanceMsg{Variables:map[string]*istio_policy_v1beta11.Value{host: &Value{Value:&Value_StringValue{StringValue:192.168.101.6:31380,},},latency: &Value{Value:&Value_DurationValue{DurationValue:&Duration{Value:34.547407ms,},},},method: &Value{Value:&Value_StringValue{StringValue:GET,},},responseBodySize: &Value{Value:&Value_Int64Value{Int64Value:5719,},},responseCode: &Value{Value:&Value_Int64Value{Int64Value:200,},},responseTotalSize: &Value{Value:&Value_Int64Value{Int64Value:5858,},},url: &Value{Value:&Value_StringValue{StringValue:/productpage,},},},Timestamp:&istio_policy_v1beta11.TimeStamp{Value:2019-02-21T03:05:39.955797905Z,},Severity:warning,MonitoredResourceType:UNSPECIFIED,MonitoredResourceDimensions:map[string]*istio_policy_v1beta11.Value{},Name:i1metric.instance.istio-system,} &InstanceMsg{Variables:map[string]*istio_policy_v1beta11.Value{host: &Value{Value:&Value_StringValue{StringValue:192.168.101.6:31380,},},latency: &Value{Value:&Value_DurationValue{DurationValue:&Duration{Value:39.579628ms,},},},method: &Value{Value:&Value_StringValue{StringValue:GET,},},responseBodySize: &Value{Value:&Value_Int64Value{Int64Value:5723,},},responseCode: &Value{Value:&Value_Int64Value{Int64Value:200,},},responseTotalSize: &Value{Value:&Value_Int64Value{Int64Value:5862,},},url: &Value{Value:&Value_StringValue{StringValue:/productpage,},},},Timestamp:&istio_policy_v1beta11.TimeStamp{Value:2019-02-21T03:05:40.126271976Z,},Severity:warning,MonitoredResourceType:UNSPECIFIED,MonitoredResourceDimensions:map[string]*istio_policy_v1beta11.Value{},Name:i1metric.instance.istio-system,} &InstanceMsg{Variables:map[string]*istio_policy_v1beta11.Value{host: &Value{Value:&Value_StringValue{StringValue:192.168.101.6:31380,},},latency: &Value{Value:&Value_DurationValue{DurationValue:&Duration{Value:24.084598ms,},},},method: &Value{Value:&Value_StringValue{StringValue:GET,},},responseBodySize: &Value{Value:&Value_Int64Value{Int64Value:5719,},},responseCode: &Value{Value:&Value_Int64Value{Int64Value:200,},},responseTotalSize: &Value{Value:&Value_Int64Value{Int64Value:5858,},},url: &Value{Value:&Value_StringValue{StringValue:/productpage,},},},Timestamp:&istio_policy_v1beta11.TimeStamp{Value:2019-02-21T03:05:40.293294914Z,},Severity:warning,MonitoredResourceType:UNSPECIFIED,MonitoredResourceDimensions:map[string]*istio_policy_v1beta11.Value{},Name:i1metric.instance.istio-system,} &InstanceMsg{Variables:map[string]*istio_policy_v1beta11.Value{host: &Value{Value:&Value_StringValue{StringValue:192.168.101.6:31380,},},latency: &Value{Value:&Value_DurationValue{DurationValue:&Duration{Value:30.813597ms,},},},method: &Value{Value:&Value_StringValue{StringValue:GET,},},responseBodySize: &Value{Value:&Value_Int64Value{Int64Value:4415,},},responseCode: &Value{Value:&Value_Int64Value{Int64Value:200,},},responseTotalSize: &Value{Value:&Value_Int64Value{Int64Value:4554,},},url: &Value{Value:&Value_StringValue{StringValue:/productpage,},},},Timestamp:&istio_policy_v1beta11.TimeStamp{Value:2019-02-21T03:05:40.55426223Z,},Severity:warning,MonitoredResourceType:UNSPECIFIED,MonitoredResourceDimensions:map[string]*istio_policy_v1beta11.Value{},Name:i1metric.instance.istio-system,}] &Any{TypeUrl:type.googleapis.com/adapter.gatewaymetric.config.Params,Value:[10 7 111 117 116 46 116 120 116],} 7051031778836507932}

2019-02-21T03:05:41.002468Z	info	host: %!(EXTRA string=192.168.101.6:31380)
2019-02-21T03:05:41.002473Z	info	url: %!(EXTRA string=/productpage)
2019-02-21T03:05:41.002475Z	info	method: %!(EXTRA string=GET)
2019-02-21T03:05:41.002477Z	info	responseCode: %!(EXTRA string=200)
2019-02-21T03:05:41.002479Z	info	responseBodySize: %!(EXTRA int64=5719)
2019-02-21T03:05:41.002483Z	info	responseTotalSize: %!(EXTRA int64=5858)
2019-02-21T03:05:41.002487Z	info	latency: %!(EXTRA float64=0.034547407)
2019-02-21T03:05:41.002500Z	info	host: %!(EXTRA string=192.168.101.6:31380)
2019-02-21T03:05:41.002506Z	info	url: %!(EXTRA string=/productpage)
2019-02-21T03:05:41.002508Z	info	method: %!(EXTRA string=GET)
2019-02-21T03:05:41.002510Z	info	responseCode: %!(EXTRA string=200)
2019-02-21T03:05:41.002512Z	info	responseBodySize: %!(EXTRA int64=5723)
2019-02-21T03:05:41.002514Z	info	responseTotalSize: %!(EXTRA int64=5862)
2019-02-21T03:05:41.002516Z	info	latency: %!(EXTRA float64=0.039579628)
2019-02-21T03:05:41.002520Z	info	host: %!(EXTRA string=192.168.101.6:31380)
2019-02-21T03:05:41.002522Z	info	url: %!(EXTRA string=/productpage)
2019-02-21T03:05:41.002524Z	info	method: %!(EXTRA string=GET)
2019-02-21T03:05:41.002526Z	info	responseCode: %!(EXTRA string=200)
2019-02-21T03:05:41.002528Z	info	responseBodySize: %!(EXTRA int64=5719)
2019-02-21T03:05:41.002530Z	info	responseTotalSize: %!(EXTRA int64=5858)
2019-02-21T03:05:41.002532Z	info	latency: %!(EXTRA float64=0.024084598)
2019-02-21T03:05:41.002535Z	info	host: %!(EXTRA string=192.168.101.6:31380)
2019-02-21T03:05:41.002537Z	info	url: %!(EXTRA string=/productpage)
2019-02-21T03:05:41.002538Z	info	method: %!(EXTRA string=GET)
2019-02-21T03:05:41.002540Z	info	responseCode: %!(EXTRA string=200)
2019-02-21T03:05:41.002542Z	info	responseBodySize: %!(EXTRA int64=4415)
2019-02-21T03:05:41.002544Z	info	responseTotalSize: %!(EXTRA int64=4554)
2019-02-21T03:05:41.002547Z	info	latency: %!(EXTRA float64=0.030813597)

再看下promethues暴露的9145端口是否有数据呢

➜  gatewaymetric curl 10.104.182.140:9145/metrics
...
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.00099"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.00089"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0007899999999999999"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0006899999999999999"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0005899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0004899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0003899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0002899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0001899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-8.999999999999979e-05"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="1.0000000000000216e-05"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.00011000000000000022"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.00021000000000000023"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0003100000000000002"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0004100000000000002"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0005100000000000003"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0006100000000000003"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0007100000000000003"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0008100000000000004"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0009100000000000004"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="+Inf"} 34
http_request_seconds_histogram_sum{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200"} 1.1806039740000003
http_request_seconds_histogram_count{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200"} 34
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.00099"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.00089"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0007899999999999999"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0006899999999999999"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0005899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0004899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0003899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0002899999999999998"} 0
...

说明一切已经生效，have fun！

avatar
文章
阅读
粉丝