龙芯CPU上面安装了 kubernetes 1.16 的集群顺带也弄了联邦集群,连同依赖、部署测试做完,这里使用的是v0.1.0-rc6版本。当然这些也可以在x86平台上进行编译,然后拿到龙芯上进行测试。下面是mips64le 架构上编译组件过程
编译环境
- Linux 系统
- Go 环境
- Git 代码管理
- 环境变量设置如下
export GO111MODULE=on export GOPROXY=https://goproxy.cn export GOROOT="/usr/local/go" export GOPATH="${HOME}/.go"
拉取源码
[root@master1 ~]# git clone https://github.com/kubernetes-sigs/kubefed.git .
KubeFed 组件
-
hyperfed
[root@master1 ~]# cd kubefed
[root@master1 kubefed]# GOOS="linux" GOARCH="mips64le" CGO_ENABLED=0 go build \
-ldflags "-X sigs.k8s.io/kubefed/pkg/version.version=$(git describe --always) \
-X sigs.k8s.io/kubefed/pkg/version.gitCommit=$(git rev-parse HEAD) \
-X sigs.k8s.io/kubefed/pkg/version.gitTreeState=clean \
-X sigs.k8s.io/kubefed/pkg/version.buildDate=$(date -u +%Y-%m-%dT%H:%M:%SZ)" -o bin/hyperfed-linux-mips64el cmd/hyperfed/main.go
[root@mfsmaster kubefed]# ./bin/hyperfed-linux-mips64el
........
Usage:
hyperfed COMMAND [flags]
hyperfed [command]
Available Commands:
controller-manager
help Help about any command
kubefedctl kubefedctl controls a Kubernetes Cluster Federation
webhook Start a kubefed webhook server
Flags:
--alsologtostderr log to standard error as well as files
-h, --help help for hyperfed
........
-
controller-manager
[root@master1 kubefed]# GOOS="linux" GOARCH="mips64le" CGO_ENABLED=0 go build \
-ldflags "-X sigs.k8s.io/kubefed/pkg/version.version=$(git describe --always) \
-X sigs.k8s.io/kubefed/pkg/version.gitCommit=$(git rev-parse HEAD) \
-X sigs.k8s.io/kubefed/pkg/version.gitTreeState=clean \
-X sigs.k8s.io/kubefed/pkg/version.buildDate=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
-o bin/controller-manager-linux-mips64el cmd/controller-manager/main.go
[root@master1 kubefed]# bin/controller-manager-linux-mips64el --version
KubeFed controller-manager version: version.Info{Version:"v0.1.0-rc6-39-ga8e670d29", GitCommit:"a8e670d290fd71be0a5f61e8e64431a8d67996c5", GitTreeState:"clean", BuildDate:"2019-08-29T22:50:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/mips64le"}
-
webhook
[root@master1 kubefed]# GOARCH="mips64le" GOOS="linux" CGO_ENABLED=0 go build \
-ldflags "-X sigs.k8s.io/kubefed/pkg/version.version=$(git describe --always) \
-X sigs.k8s.io/kubefed/pkg/version.gitCommit=$(git rev-parse HEAD) \
-X sigs.k8s.io/kubefed/pkg/version.gitTreeState=clean \
-X sigs.k8s.io/kubefed/pkg/version.buildDate=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
-o bin/webhook-linux-mips64el
[root@mfsmaster kubefed]# ./bin/webhook-linux-mips64el version
KubeFed admission webhook version: version.Info{Version:"v0.1.0-rc6-39-ga8e670d29", GitCommit:"a8e670d290fd71be0a5f61e8e64431a8d67996c5", GitTreeState:"clean", BuildDate:"2019-08-29T22:50:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/mips64le"}
-
kubefedctl
[root@master1 kubefed]# GOOS="linux" GOARCH="mips64le" CGO_ENABLED=0 go build \
-ldflags "-X sigs.k8s.io/kubefed/pkg/version.version=$(git describe --always) \
-X sigs.k8s.io/kubefed/pkg/version.gitCommit=$(git rev-parse HEAD) \
-X sigs.k8s.io/kubefed/pkg/version.gitTreeState=clean \
-X sigs.k8s.io/kubefed/pkg/version.buildDate=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
-o bin/kubefedctl-linux-mips64el cmd/kubefedctl/main.go
[root@master1 kubefed]# ./bin/kubefedctl-linux-mips64el version
kubefedctl version: version.Info{Version:"v0.1.0-rc6-39-ga8e670d29", GitCommit:"a8e670d290fd71be0a5f61e8e64431a8d67996c5", GitTreeState:"clean", BuildDate:"2019-08-29T22:50:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/mips64le"}
-
e2e
[root@master1 kubefed]# GOOS="linux" GOARCH="mips64le" go test -c \
-ldflags "-X sigs.k8s.io/kubefed/pkg/version.version=$(git describe --always) \
-X sigs.k8s.io/kubefed/pkg/version.gitCommit=$(git rev-parse HEAD) \
-X sigs.k8s.io/kubefed/pkg/version.gitTreeState=clean \
-X sigs.k8s.io/kubefed/pkg/version.buildDate=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
-o bin/e2e-linux-mipsd64el ./test/e2e
正式安装federation v2版本
-
1.创建三个k8s集群
-
1.1 kubeadm init 使用命令行参数
kubeadm init --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.15.1
-
1.1.1 kubeadm 使用配置文件安装
[root@master1 ~]# cat kubeadm-init-config.yaml apiVersion: kubeadm.k8s.io/v1beta1 kind: ClusterConfiguration kubernetesVersion: v1.16.0 clusterName: kubernetes networking: podSubnet: 10.244.0.0/16 dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 apiServer: advertise-address: 192.168.0.77 external-hostname: master1 authorization-mode: RBAC default-not-ready-toleration-seconds: 20 default-unreachable-toleration-seconds: 15 enable-bootstrap-token-auth: true anonymous-auth: true allow-privileged: true audit-webhook-initial-backoff: 60s kubelet-timeout: 300s request-timeout: 5m timeoutForControlPlane: 10m0s controllerManager: node-monitor-grace-period: 6s node-monitor-period: 1s pod-eviction-timeout: 5s feature-gates: "TaintBasedEvictions=false" concurrent-deployment-syncs: 3 concurrent-endpoint-syncs: 3 leader-elect: "true" imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd --- apiVersion: kubeadm.k8s.io/v1beta1 kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.0.77 bindPort: 6443 nodeRegistration: name: master1 taints: - key: "kubeadmNode" value: "master" effect: "NoSchedule" kubeletExtraArgs: cgroup-driver: "cgroupfs" [root@master1 ~]# kubeadm init --config /etc/kubernetes/kubeadm-config.yaml
-
-
2.配置kubeconfig
配置 config 有三个步骤,他们分别是:添加集群
、设置主题
、设置认证
,key 和 ca的证书可以在集群的.kube/config
里面看到,拿到里面的内容后进行解密,如:
echo 'LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFekNDQWZ1Z0F3SUJBZ0lJUUcvS3c4Q0l6RFF3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TVRBME1EWXdNakV4TURWYUZ3MHlNakEwTURZd01qRXhNekphTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXc2RzdzN2I5OVRnVGtvU0EKb3VvQXFMcldDdHVUcWJqZkI1eXdWMXlrdW83cHBJMER4UlNFZC9mQ3JSUUsrY0RFKzFVWW1Dby9RamZOdUU4RgpRU3ZJcVhrTnJOQThoUHpWKzg3YkZtUG9ocnRKcTFlcDBHcXFITVQ3QUE4MWFkVmVUUjlpMVZEUUh6MFk1WU5qCjdTcGpNaWRSOUw3cG55WjlvcUgwMGtGSGxxRko1b21ua1BCazk4Wm5iRmxFSko2eC9DQll4LzVNUC80Y2NCNkYKZmwvWmN0ZmYxNE5wYUQ0dkw4V3hFeWN1a1lDcnFtUWtyTmllbHdLSk9iUENhbHRwRzBGY1JveW9WcmhHK3hqeApIMWU1U2N5UnFWZnRyTkNaN29aMnJZRUdrWm1wTkpvZFh1QmJ5cm5LN0kvUTJQWmdKSERMRmdKZ2FnOWNpOVJUCmwvcmNWUUlEQVFBQm8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0h3WURWUjBqQkJnd0ZvQVV3dXNhckVickpYcGtZUytjZ1ZDOU81ZHpLTk13RFFZSktvWklodmNOQVFFTApCUUFEZ2dFQkFCSnhTNG5wNHczK0NmV1VFcDhYWUhCUlZabU45b2dGQkJkckRYemZVMmpXZWdXSk9keFc5b3p0Cm83ZW0ySnNpUzVWbkMybnZVOFQ4ZHFYcEpudWxZTVFIQW9jNXcxclpKcWFETTJhK0xtWis3cDMwNUluV09nUUYKUURJVlJHTHZoK3lXVHIxakJsSUVJRkE0VDRtUnozN1J6eTFDYU1WZmpnOW5yWk01bzIvcEQ0azFONng2UjBzQQo1QlhzMVpzczU0dnBRcG1Oczk4bS9hMG9wRHVOQ1VQZG1KOU9lMnVHNVl2SnpuNmptRS9VSm0wdzdEeWJOKzRPCm4rc0EyVzFYVmEvNFhUT1BTcHJVTEdsUHI4alNHTGRaU0dSVFF5UE00QkRBK1FpZzVqU0V2MmlCalY3T3FTUEQKZlFqYzQ1a3pIT01qcEJ1RDdwT3VlRXcwc3luTGtZaz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=' | base64 -d > client-certificate.ca
下面开始设置 kubeconfig
kubectl config set-cluster magicdata --server=https://192.168.4.8:6443 --certificate-authority ~/4.8_cluster.ca --embed-certs=true
kubectl config set-context magicdata --cluster magicdata --user kubernetes-admin --namespace=default
kubectl config set-credentials kubernetes-admin --embed-certs=true --client-key ~/client.key --client-certificate ~/client.ca
- 2.1 创建 kubeconfig
[root@master1 ~]# cat .kube/config
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTi
server: https://192.168.0.87:6443
name: master1
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5R
server: https://192.168.0.77:6443
name: master2
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5R
server: https://192.168.0.67:6443
name: master3
contexts:
- context:
cluster: master1
namespace: default
user: kubernetes-admin1
name: master1
- context:
cluster: master2
namespace: default
user: kubernetes-admin2
name: master2
- context:
cluster: master3
namespace: default
user: kubernetes-admin3
name: master3
current-context: master1
kind: Config
preferences: {}
users:
- name: kubernetes-admin1
user:
client-certificate-data: LS0tLS1JSU
client-key-data: LS0tLS1CRUdJTiBSU0
token: eyJhbGciOiJSUzI1NiIsImtpZCI6
- name: kubernetes-admin2
user:
client-certificate-data: LS0tLS1CRU
client-key-data: LS0tLS1CRUdJTiBSU0
token: eyJhbGciOiJSUzI1NiIsImtpZCI6
- name: kubernetes-master3
user:
client-certificate-data: LS0tLS1CRUd
client-key-data: LS0tLS1CRUdJTiBSU0E
-
2.2 查看kubeconfig配置
[root@master1 ~]# kubectl config get-clusters NAME master1 master2 master3 [root@master1 ~]# kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE * master1 master1 kubernetes-admin1 default master2 master2 kubernetes-admin2 default vm1 vm1 kubernetes-vm1 default
-
2.3 用上面编译的hyperfed-linux-mips64le程序构建自定义镜像
[root@master1 ~]# cat <<'EOF' > kubefed/bin/Dockerfile from huangxg20171010/fedora21-base RUN set -xe \ && useradd hyperfed \ && mkdir -p /hyperfed WORKDIR /hyperfed/ COPY hyperfed . COPY controller-manager . COPY kubefedctl . COPY webhook . RUN set -xe \ && chown -R hyperfed:hyperfed /hyperfed USER hyperfed ENTRYPOINT ["./controller-manager"]
-
2.4 构建镜像
[root@master1 ~]# cd kubefed/bin [root@master1 kubefed]# docker build -t qiqisa/hyperfed-mips64el:v0.1.0-rc6 .
-
2.5 修改 deploy-kubefed.sh 不上传 docker 仓库、不构建镜像
[root@master1 kubefed]# vim ./scripts/deploy-kubefed.sh function deploy-with-helm() { ..... #修改 helm 初始化,添加这个 "-i image:version"参数 helm init -i qiqisa/tiller:v2.14.3 --tiller-image --service-account tiller ..... } #修改不推送 docker 镜像仓库 #DOCKER_PUSH="${DOCKER_PUSH-y}" #DOCKER_PUSH_CMD="docker push ${IMAGE_NAME}" #if [[ ! "${DOCKER_PUSH}" ]]; then # DOCKER_PUSH_CMD= #fi #修改不构建镜像 #if [[ ! "${USE_LATEST}" ]]; then # cd "$(dirname "$0")/.." # make container IMAGE_NAME=${IMAGE_NAME} # cd - # ${DOCKER_PUSH_CMD} #fi #cd "$(dirname "$0")/.." #make kubefedctl #修改不构建 kubefedctl,因为上面已经构建了 #cd - [root@master1 kubefed]# ./scripts/deploy-kubefed.sh qiqisa/kubefed-mips64el:v0.1.0-rc6 master1 #这个就可以在当前机器上部署成功 Server: &version.Version{SemVer:"v2.14.2", GitCommit:"a8b13cc5ab6a7dbef0a58f5061bcc7c0c61598e7", GitTreeState:"dirty"} NAME: kubefed LAST DEPLOYED: Wed Oct 16 23:09:24 2019 NAMESPACE: kube-federation-system STATUS: DEPLOYED RESOURCES: ==> v1/ClusterRole
-
3.查看集群状态并加入联邦集群
[root@master1 Desktop]# kubectl get kubefedclusters.core.kubefed.io --all-namespaces >>#可以查看到当前集群中只有一个 NAMESPACE NAME READY AGE kube-federation-system master1 True 1h [root@master1 Desktop]# kubefedctl join master2 --v=2 --host-cluster-context master1 #这个host-cluster-context一定是current-context [root@master1 Desktop]# kubefedctl join master3 --v=2 --host-cluster-context master1 [root@master1 Desktop]# kubectl get kubefedclusters.core.kubefed.io --all-namespaces NAMESPACE NAME READY AGE kube-federation-system master1 True 2h kube-federation-system master2 True 32m kube-federation-system master3 True 6s
- 3.1 部署应用
# 参考 sample/example里面的配置