MIPS64le编译部署 kubefed

龙芯CPU上面安装了 kubernetes 1.16 的集群顺带也弄了联邦集群，连同依赖、部署测试做完，这里使用的是v0.1.0-rc6版本。当然这些也可以在x86平台上进行编译,然后拿到龙芯上进行测试。下面是mips64le 架构上编译组件过程

编译环境

• Linux 系统
• Go 环境
• Git 代码管理
• 环境变量设置如下

  export GO111MODULE=on
  export GOPROXY=https://goproxy.cn
  export GOROOT="/usr/local/go"
  export GOPATH="${HOME}/.go"
  

拉取源码

[root@master1 ~]# git clone https://github.com/kubernetes-sigs/kubefed.git .

KubeFed 组件

• hyperfed

[root@master1 ~]# cd kubefed
[root@master1 kubefed]# GOOS="linux" GOARCH="mips64le" CGO_ENABLED=0 go build  \
-ldflags "-X sigs.k8s.io/kubefed/pkg/version.version=$(git describe --always) \
-X sigs.k8s.io/kubefed/pkg/version.gitCommit=$(git rev-parse HEAD) \
-X sigs.k8s.io/kubefed/pkg/version.gitTreeState=clean \
-X sigs.k8s.io/kubefed/pkg/version.buildDate=$(date -u +%Y-%m-%dT%H:%M:%SZ)" -o  bin/hyperfed-linux-mips64el cmd/hyperfed/main.go 
[root@mfsmaster kubefed]# ./bin/hyperfed-linux-mips64el
........
Usage:
  hyperfed COMMAND [flags]
  hyperfed [command]
Available Commands:
  controller-manager 
  help               Help about any command
  kubefedctl         kubefedctl controls a Kubernetes Cluster Federation
  webhook            Start a kubefed webhook server
Flags:
      --alsologtostderr                  log to standard error as well as files
  -h, --help                             help for hyperfed
........

• controller-manager

[root@master1 kubefed]# GOOS="linux" GOARCH="mips64le" CGO_ENABLED=0 go build \
-ldflags "-X sigs.k8s.io/kubefed/pkg/version.version=$(git describe --always) \
-X sigs.k8s.io/kubefed/pkg/version.gitCommit=$(git rev-parse HEAD) \
-X sigs.k8s.io/kubefed/pkg/version.gitTreeState=clean \
-X sigs.k8s.io/kubefed/pkg/version.buildDate=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
-o  bin/controller-manager-linux-mips64el cmd/controller-manager/main.go
[root@master1 kubefed]#  bin/controller-manager-linux-mips64el --version
KubeFed controller-manager version: version.Info{Version:"v0.1.0-rc6-39-ga8e670d29", GitCommit:"a8e670d290fd71be0a5f61e8e64431a8d67996c5", GitTreeState:"clean", BuildDate:"2019-08-29T22:50:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/mips64le"}

• webhook

[root@master1 kubefed]# GOARCH="mips64le" GOOS="linux" CGO_ENABLED=0 go build \
-ldflags "-X sigs.k8s.io/kubefed/pkg/version.version=$(git describe --always) \
-X sigs.k8s.io/kubefed/pkg/version.gitCommit=$(git rev-parse HEAD) \
-X sigs.k8s.io/kubefed/pkg/version.gitTreeState=clean \
-X sigs.k8s.io/kubefed/pkg/version.buildDate=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
-o  bin/webhook-linux-mips64el
[root@mfsmaster kubefed]# ./bin/webhook-linux-mips64el version
KubeFed admission webhook version: version.Info{Version:"v0.1.0-rc6-39-ga8e670d29", GitCommit:"a8e670d290fd71be0a5f61e8e64431a8d67996c5", GitTreeState:"clean", BuildDate:"2019-08-29T22:50:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/mips64le"}

• kubefedctl

[root@master1 kubefed]# GOOS="linux" GOARCH="mips64le" CGO_ENABLED=0 go build \
-ldflags "-X sigs.k8s.io/kubefed/pkg/version.version=$(git describe --always) \
-X sigs.k8s.io/kubefed/pkg/version.gitCommit=$(git rev-parse HEAD) \
-X sigs.k8s.io/kubefed/pkg/version.gitTreeState=clean \
-X sigs.k8s.io/kubefed/pkg/version.buildDate=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
-o  bin/kubefedctl-linux-mips64el cmd/kubefedctl/main.go
[root@master1 kubefed]# ./bin/kubefedctl-linux-mips64el version
kubefedctl version: version.Info{Version:"v0.1.0-rc6-39-ga8e670d29", GitCommit:"a8e670d290fd71be0a5f61e8e64431a8d67996c5", GitTreeState:"clean", BuildDate:"2019-08-29T22:50:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/mips64le"}

• e2e

[root@master1 kubefed]# GOOS="linux" GOARCH="mips64le" go test -c \
-ldflags "-X sigs.k8s.io/kubefed/pkg/version.version=$(git describe --always) \
-X sigs.k8s.io/kubefed/pkg/version.gitCommit=$(git rev-parse HEAD) \
-X sigs.k8s.io/kubefed/pkg/version.gitTreeState=clean \
-X sigs.k8s.io/kubefed/pkg/version.buildDate=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
-o  bin/e2e-linux-mipsd64el ./test/e2e

正式安装federation v2版本

• 1.创建三个k8s集群

  • 1.1 kubeadm init 使用命令行参数

    kubeadm init --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.15.1
    

  • 1.1.1 kubeadm 使用配置文件安装

    [root@master1 ~]# cat kubeadm-init-config.yaml
    apiVersion: kubeadm.k8s.io/v1beta1
    kind: ClusterConfiguration
    kubernetesVersion: v1.16.0
    clusterName: kubernetes
    networking:
     podSubnet: 10.244.0.0/16
     dnsDomain: cluster.local
     serviceSubnet: 10.96.0.0/12
    apiServer:
     advertise-address: 192.168.0.77
     external-hostname: master1
     authorization-mode: RBAC
     default-not-ready-toleration-seconds: 20
     default-unreachable-toleration-seconds: 15
     enable-bootstrap-token-auth: true
     anonymous-auth: true
     allow-privileged: true
     audit-webhook-initial-backoff: 60s
     kubelet-timeout: 300s
     request-timeout: 5m
     timeoutForControlPlane: 10m0s
    controllerManager:
     node-monitor-grace-period: 6s
     node-monitor-period: 1s
     pod-eviction-timeout: 5s
     feature-gates: "TaintBasedEvictions=false"
     concurrent-deployment-syncs: 3
     concurrent-endpoint-syncs: 3
     leader-elect: "true"
    imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
    dns:
     type: CoreDNS
    etcd:
     local:
       dataDir: /var/lib/etcd
    ---
    apiVersion: kubeadm.k8s.io/v1beta1
    kind: InitConfiguration
    localAPIEndpoint:
     advertiseAddress: 192.168.0.77
     bindPort: 6443
    nodeRegistration:
     name: master1
     taints:
       - key: "kubeadmNode"
         value: "master"
         effect: "NoSchedule"
     kubeletExtraArgs:
       cgroup-driver: "cgroupfs"
    [root@master1 ~]# kubeadm init --config /etc/kubernetes/kubeadm-config.yaml
    

• 2.配置kubeconfig

配置 config 有三个步骤，他们分别是：添加集群、设置主题、设置认证,key 和 ca的证书可以在集群的.kube/config里面看到，拿到里面的内容后进行解密，如： echo 'LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFekNDQWZ1Z0F3SUJBZ0lJUUcvS3c4Q0l6RFF3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TVRBME1EWXdNakV4TURWYUZ3MHlNakEwTURZd01qRXhNekphTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXc2RzdzN2I5OVRnVGtvU0EKb3VvQXFMcldDdHVUcWJqZkI1eXdWMXlrdW83cHBJMER4UlNFZC9mQ3JSUUsrY0RFKzFVWW1Dby9RamZOdUU4RgpRU3ZJcVhrTnJOQThoUHpWKzg3YkZtUG9ocnRKcTFlcDBHcXFITVQ3QUE4MWFkVmVUUjlpMVZEUUh6MFk1WU5qCjdTcGpNaWRSOUw3cG55WjlvcUgwMGtGSGxxRko1b21ua1BCazk4Wm5iRmxFSko2eC9DQll4LzVNUC80Y2NCNkYKZmwvWmN0ZmYxNE5wYUQ0dkw4V3hFeWN1a1lDcnFtUWtyTmllbHdLSk9iUENhbHRwRzBGY1JveW9WcmhHK3hqeApIMWU1U2N5UnFWZnRyTkNaN29aMnJZRUdrWm1wTkpvZFh1QmJ5cm5LN0kvUTJQWmdKSERMRmdKZ2FnOWNpOVJUCmwvcmNWUUlEQVFBQm8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0h3WURWUjBqQkJnd0ZvQVV3dXNhckVickpYcGtZUytjZ1ZDOU81ZHpLTk13RFFZSktvWklodmNOQVFFTApCUUFEZ2dFQkFCSnhTNG5wNHczK0NmV1VFcDhYWUhCUlZabU45b2dGQkJkckRYemZVMmpXZWdXSk9keFc5b3p0Cm83ZW0ySnNpUzVWbkMybnZVOFQ4ZHFYcEpudWxZTVFIQW9jNXcxclpKcWFETTJhK0xtWis3cDMwNUluV09nUUYKUURJVlJHTHZoK3lXVHIxakJsSUVJRkE0VDRtUnozN1J6eTFDYU1WZmpnOW5yWk01bzIvcEQ0azFONng2UjBzQQo1QlhzMVpzczU0dnBRcG1Oczk4bS9hMG9wRHVOQ1VQZG1KOU9lMnVHNVl2SnpuNmptRS9VSm0wdzdEeWJOKzRPCm4rc0EyVzFYVmEvNFhUT1BTcHJVTEdsUHI4alNHTGRaU0dSVFF5UE00QkRBK1FpZzVqU0V2MmlCalY3T3FTUEQKZlFqYzQ1a3pIT01qcEJ1RDdwT3VlRXcwc3luTGtZaz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=' | base64 -d > client-certificate.ca

下面开始设置 kubeconfig

kubectl config set-cluster magicdata --server=https://192.168.4.8:6443 --certificate-authority ~/4.8_cluster.ca --embed-certs=true
kubectl config set-context magicdata --cluster magicdata --user kubernetes-admin --namespace=default
kubectl config set-credentials kubernetes-admin --embed-certs=true  --client-key ~/client.key --client-certificate ~/client.ca

• 2.1 创建 kubeconfig

[root@master1 ~]# cat .kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTi
    server: https://192.168.0.87:6443
  name: master1
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5R
    server: https://192.168.0.77:6443
  name: master2
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5R
    server: https://192.168.0.67:6443
  name: master3
contexts:
- context:
  cluster: master1
    namespace: default
    user: kubernetes-admin1
  name: master1
- context:
    cluster: master2
    namespace: default
    user: kubernetes-admin2
  name: master2
- context:
    cluster: master3
    namespace: default
    user: kubernetes-admin3
  name: master3
current-context: master1
kind: Config
preferences: {}
users:
- name: kubernetes-admin1
  user:
    client-certificate-data: LS0tLS1JSU
    client-key-data: LS0tLS1CRUdJTiBSU0
    token: eyJhbGciOiJSUzI1NiIsImtpZCI6
- name: kubernetes-admin2
  user:
    client-certificate-data: LS0tLS1CRU
    client-key-data: LS0tLS1CRUdJTiBSU0
    token: eyJhbGciOiJSUzI1NiIsImtpZCI6
- name: kubernetes-master3
  user:
    client-certificate-data: LS0tLS1CRUd
    client-key-data: LS0tLS1CRUdJTiBSU0E

• 2.2 查看kubeconfig配置

  [root@master1 ~]#  kubectl config get-clusters
  NAME
  master1
  master2
  master3
  [root@master1 ~]#  kubectl config get-contexts
  CURRENT   NAME      CLUSTER   AUTHINFO            NAMESPACE
  *         master1   master1   kubernetes-admin1   default
             master2   master2   kubernetes-admin2   default
             vm1       vm1       kubernetes-vm1      default
  

• 2.3 用上面编译的hyperfed-linux-mips64le程序构建自定义镜像

  [root@master1 ~]# cat <<'EOF' > kubefed/bin/Dockerfile
  from huangxg20171010/fedora21-base
  RUN set -xe \
      && useradd hyperfed \
      && mkdir -p /hyperfed 
  WORKDIR /hyperfed/
  COPY hyperfed .
  COPY controller-manager .
  COPY kubefedctl .
  COPY webhook .
  RUN set -xe \
      && chown -R hyperfed:hyperfed /hyperfed
  USER hyperfed
  ENTRYPOINT ["./controller-manager"]
  

• 2.4 构建镜像

  [root@master1 ~]# cd kubefed/bin
  [root@master1 kubefed]# docker build -t qiqisa/hyperfed-mips64el:v0.1.0-rc6 .
  

• 2.5 修改 deploy-kubefed.sh 不上传 docker 仓库、不构建镜像

  [root@master1 kubefed]# vim ./scripts/deploy-kubefed.sh
  function deploy-with-helm() {
  .....
  #修改 helm 初始化，添加这个 "-i image:version"参数
  helm init -i qiqisa/tiller:v2.14.3 --tiller-image --service-account tiller
  .....
  }
  #修改不推送 docker 镜像仓库
  #DOCKER_PUSH="${DOCKER_PUSH-y}"
  #DOCKER_PUSH_CMD="docker push ${IMAGE_NAME}"
  #if [[ ! "${DOCKER_PUSH}" ]]; then
  #    DOCKER_PUSH_CMD=
  #fi
  #修改不构建镜像
  #if [[ ! "${USE_LATEST}" ]]; then
  #  cd "$(dirname "$0")/.."
  #  make container IMAGE_NAME=${IMAGE_NAME}
  #  cd -
  #  ${DOCKER_PUSH_CMD}
  #fi
  #cd "$(dirname "$0")/.."
  #make kubefedctl #修改不构建 kubefedctl，因为上面已经构建了
  #cd -
  [root@master1 kubefed]# ./scripts/deploy-kubefed.sh qiqisa/kubefed-mips64el:v0.1.0-rc6 master1  #这个就可以在当前机器上部署成功
  Server: &version.Version{SemVer:"v2.14.2", GitCommit:"a8b13cc5ab6a7dbef0a58f5061bcc7c0c61598e7", GitTreeState:"dirty"}
  NAME:   kubefed
  LAST DEPLOYED: Wed Oct 16 23:09:24 2019
  NAMESPACE: kube-federation-system
  STATUS: DEPLOYED
  RESOURCES:
  ==> v1/ClusterRole
  

• 3.查看集群状态并加入联邦集群

  [root@master1 Desktop]# kubectl get kubefedclusters.core.kubefed.io  --all-namespaces >>#可以查看到当前集群中只有一个
  NAMESPACE                NAME      READY   AGE
  kube-federation-system   master1   True    1h
  [root@master1 Desktop]# kubefedctl join master2 --v=2 --host-cluster-context master1 #这个host-cluster-context一定是current-context
  [root@master1 Desktop]# kubefedctl join master3 --v=2 --host-cluster-context master1
  [root@master1 Desktop]# kubectl get kubefedclusters.core.kubefed.io  --all-namespaces
  NAMESPACE                NAME      READY   AGE
  kube-federation-system   master1   True    2h
  kube-federation-system   master2   True    32m
  kube-federation-system   master3   True    6s
  

  • 3.1 部署应用

  # 参考 sample/example里面的配置