Keepalived

早點(diǎn)悟道 2020-05-26

展開(kāi)全文

keepalived-vip

Kubernetes 使用 keepalived 來(lái)產(chǎn)生虛擬 IP address

我們將探討如何利用 IPVS - The Linux Virtual Server Project' 來(lái) kubernetes 配置 VIP

前言

kubernetes v1.6 版提供了三種方式去暴露 Service：

L4 的 LoadBalacncer : 只能在 cloud providers 上被使用像是 GCE 或 AWS
NodePort : NodePort 允許在每個(gè)節(jié)點(diǎn)上開(kāi)啟一個(gè) port 口, 借由這個(gè) port 口會(huì)再將請(qǐng)求導(dǎo)向到隨機(jī)的 pod 上
L7 Ingress :Ingress 為一個(gè) LoadBalancer(例: nginx, HAProxy, traefik, vulcand) 會(huì)將 HTTP/HTTPS 的各個(gè)請(qǐng)求導(dǎo)向到相對(duì)應(yīng)的 service endpoint

有了這些方式, 為何我們還需要 keepalived ?

                                                  ___________________                                                 |                   |                                           |-----| Host IP: 10.4.0.3 |                                           |     |___________________|                                           |                                           |      ___________________                                           |     |                   |Public ----(example.com = 10.4.0.3/4/5)----|-----| Host IP: 10.4.0.4 |                                           |     |___________________|                                           |                                           |      ___________________                                           |     |                   |                                           |-----| Host IP: 10.4.0.5 |                                                 |___________________|

我們假設(shè) Ingress 運(yùn)行在 3 個(gè) kubernetes 節(jié)點(diǎn)上, 并對(duì)外暴露 10.4.0.x 的 IP 去做 loadbalance

DNS Round Robin (RR) 將對(duì)應(yīng)到 example.com 的請(qǐng)求輪循給這 3 個(gè)節(jié)點(diǎn), 如果 10.4.0.3 掛了, 仍有三分之一的流量會(huì)導(dǎo)向 10.4.0.3, 這樣就會(huì)有一段 downtime, 直到 DNS 發(fā)現(xiàn) 10.4.0.3 掛了并修正導(dǎo)向

嚴(yán)格來(lái)說(shuō), 這并沒(méi)有真正的做到 High Availability (HA)

這邊 IPVS 可以幫助我們解決這件事, 這個(gè)想法是虛擬 IP(VIP) 對(duì)應(yīng)到每個(gè) service 上, 并將 VIP 暴露到 kubernetes 群集之外

與 service-loadbalancer 或 ingress-nginx 的區(qū)別

我們看到以下的圖

                                               ___________________                                              |                   |                                              | VIP: 10.4.0.50    |                                        |-----| Host IP: 10.4.0.3 |                                        |     | Role: Master      |                                        |     |___________________|                                        |                                        |      ___________________                                        |     |                   |                                        |     | VIP: Unassigned   |Public ----(example.com = 10.4.0.50)----|-----| Host IP: 10.4.0.4 |                                        |     | Role: Slave       |                                        |     |___________________|                                        |                                        |      ___________________                                        |     |                   |                                        |     | VIP: Unassigned   |                                        |-----| Host IP: 10.4.0.5 |                                              | Role: Slave       |                                              |___________________|

我們可以看到只有一個(gè) node 被選為 Master(透過(guò) VRRP 選擇的), 而我們的 VIP 是 10.4.0.50, 如果 10.4.0.3 掛掉了, 那會(huì)從剩余的節(jié)點(diǎn)中選一個(gè)成為 Master 并接手 VIP, 這樣我們就可以確保落實(shí)真正的 HA

環(huán)境需求

只需要確認(rèn)要運(yùn)行 keepalived-vip 的 kubernetes 群集 DaemonSets 功能是正常的就行了

RBAC

由于 kubernetes 在 1.6 后引進(jìn)了 RBAC 的概念, 所以我們要先去設(shè)定 rule, 至於有關(guān) RBAC 的詳情請(qǐng)至說(shuō)明。

vip-rbac.yaml

apiVersion: rbac.authorization.k8s.io/v1beta1kind: ClusterRolemetadata:  name: kube-keepalived-viprules:- apiGroups: ['']  resources:  - pods  - nodes  - endpoints  - services  - configmaps  verbs: ['get', 'list', 'watch']---apiVersion: v1kind: ServiceAccountmetadata:  name: kube-keepalived-vip  namespace: default---apiVersion: rbac.authorization.k8s.io/v1beta1kind: ClusterRoleBindingmetadata:  name: kube-keepalived-viproleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: kube-keepalived-vipsubjects:- kind: ServiceAccount  name: kube-keepalived-vip  namespace: default

clusterrolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1alpha1kind: ClusterRoleBindingmetadata:  name: kube-keepalived-viproleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: kube-keepalived-vipsubjects:  - kind: ServiceAccount    name: kube-keepalived-vip    namespace: default

$ kubectl create -f vip-rbac.yaml$ kubectl create -f clusterrolebinding.yaml

示例

先建立一個(gè)簡(jiǎn)單的 service

nginx-deployment.yaml

apiVersion: apps/v1beta1kind: Deploymentmetadata:  name: nginx-deploymentspec:  replicas: 3  template:    metadata:      labels:        app: nginx    spec:      containers:      - name: nginx        image: nginx:1.7.9        ports:        - containerPort: 80---apiVersion: v1kind: Servicemetadata:  name: nginx  labels:    app: nginxspec:  type: NodePort  ports:  - port: 80    nodePort: 30302    targetPort: 80    protocol: TCP    name: http  selector:    app: nginx

主要功能就是 pod 去監(jiān)聽(tīng)聽(tīng) 80 port, 再開(kāi)啟 service NodePort 監(jiān)聽(tīng) 30320

$ kubecrl create -f nginx-deployment.yaml

接下來(lái)我們要做的是 config map

$ echo 'apiVersion: v1kind: ConfigMapmetadata:  name: vip-configmapdata:  10.87.2.50: default/nginx' | kubectl create -f -

注意, 這邊的 10.87.2.50 必須換成你自己同網(wǎng)段下無(wú)使用的 IP e.g. 10.87.2.X后面 nginx 為 service 的 name, 這邊可以自行更換

接著確認(rèn)一下

$kubectl get configmapNAME DATA AGEvip-configmap 1 23h

再來(lái)就是設(shè)置 keepalived-vip

apiVersion: extensions/v1beta1kind: DaemonSetmetadata:  name: kube-keepalived-vipspec:  template:    metadata:      labels:        name: kube-keepalived-vip    spec:      hostNetwork: true      containers:        - image: gcr.io/google_containers/kube-keepalived-vip:0.9          name: kube-keepalived-vip          imagePullPolicy: Always          securityContext:            privileged: true          volumeMounts:            - mountPath: /lib/modules              name: modules              readOnly: true            - mountPath: /dev              name: dev          # use downward API          env:            - name: POD_NAME              valueFrom:                fieldRef:                  fieldPath: metadata.name            - name: POD_NAMESPACE              valueFrom:                fieldRef:                  fieldPath: metadata.namespace          # to use unicast          args:          - --services-configmap=default/vip-configmap          # unicast uses the ip of the nodes instead of multicast          # this is useful if running in cloud providers (like AWS)          #- --use-unicast=true      volumes:        - name: modules          hostPath:            path: /lib/modules        - name: dev          hostPath:            path: /dev

建立 daemonset

$ kubectl get daemonset kube-keepalived-vipNAME                  DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE-SELECTOR   AGEkube-keepalived-vip   5         5         5         5            5

檢查一下配置狀態(tài)

kubectl get pod -o wide |grep keepalivekube-keepalived-vip-c4sxw         1/1       Running            0          23h       10.87.2.6    10.87.2.6kube-keepalived-vip-c9p7n         1/1       Running            0          23h       10.87.2.8    10.87.2.8kube-keepalived-vip-psdp9         1/1       Running            0          23h       10.87.2.10   10.87.2.10kube-keepalived-vip-xfmxg         1/1       Running            0          23h       10.87.2.12   10.87.2.12kube-keepalived-vip-zjts7         1/1       Running            3          23h       10.87.2.4    10.87.2.4

可以隨機(jī)挑一個(gè) pod, 去看里面的配置

 $ kubectl exec kube-keepalived-vip-c4sxw cat /etc/keepalived/keepalived.confglobal_defs {  vrrp_version 3  vrrp_iptables KUBE-KEEPALIVED-VIP}vrrp_instance vips {  state BACKUP  interface eno1  virtual_router_id 50  priority 103  nopreempt  advert_int 1  track_interface {    eno1  }  virtual_ipaddress {    10.87.2.50  }}# Service: default/nginxvirtual_server 10.87.2.50 80 { // 此為 service 開(kāi)的口  delay_loop 5  lvs_sched wlc  lvs_method NAT  persistence_timeout 1800  protocol TCP  real_server 10.2.49.30 8080 { // 這里說(shuō)明 pod 的真實(shí)狀況    weight 1    TCP_CHECK {      connect_port 80      connect_timeout 3    }  }}

最后我們?nèi)y(cè)試這功能

$ curl  10.87.2.50<!DOCTYPE html><html><head><title>Welcome to nginx!</title><style>    body {        width: 35em;        margin: 0 auto;        font-family: Tahoma, Verdana, Arial, sans-serif;    }</style></head><body><h1>Welcome to nginx!</h1><p>If you see this page, the nginx web server is successfully installed andworking. Further configuration is required.</p><p>For online documentation and support please refer to<a href='http:///'></a>.<br/>Commercial support is available at<a href='http:///'></a>.</p><p><em>Thank you for using nginx.</em></p></body></html>

10.87.2.50:80(我們假設(shè)的 VIP, 實(shí)際上其實(shí)沒(méi)有 node 是用這 IP) 即可幫我們導(dǎo)向這個(gè) service

以上的程式代碼都在 github 上可以找到。

本站是提供個(gè)人知識(shí)管理的網(wǎng)絡(luò)存儲(chǔ)空間，所有內(nèi)容均由用戶發(fā)布,，不代表本站觀點(diǎn)。請(qǐng)注意甄別內(nèi)容中的聯(lián)系方式,、誘導(dǎo)購(gòu)買等信息,，謹(jǐn)防詐騙。如發(fā)現(xiàn)有害或侵權(quán)內(nèi)容,，請(qǐng)點(diǎn)擊一鍵舉報(bào),。

轉(zhuǎn)藏 分享

QQ空間 QQ好友新浪微博微信

獻(xiàn)花（0） +1

來(lái)自：早點(diǎn)悟道 > 《工作》

舉報(bào)/認(rèn)領(lǐng)