k8s 배치 - 다 중 노드 배치 와 부하 균형 구축 (5)

16819 단어 nginxkeepalivedLinux
다 중 노드 배치 안내
  • 생산 환경 에서 kubernetes 플랫폼 을 구축 할 때 저 희 는 플랫폼 의 높 은 가용성 을 고려 합 니 다. kubenetes 플랫폼 은 master 센터 관리 체제 로 master 서버 에서 각 노드 서버 를 관리 합 니 다. 예전 의 글 에서 저 희 는 단일 노드 (master 서버) 의 배 치 를 구 축 했 습 니 다. master 서버 가 다운 되 었 을 때우리 가 구축 한 플랫폼 도 사용 할 수 없다. 이때 우 리 는 다 중 노드 (다 마스터) 의 배 치 를 고려 해 야 한다. 이미 플랫폼 서비스의 높 은 가용성 에 이 르 렀 다.

  • 부하 균형 소개
  • 우리 가 다 중 노드 배 치 를 구축 할 때 여러 master 가 동시에 작업 을 실행 합 니 다. 작업 문 제 를 처리 할 때 항상 같은 master 를 사용 하여 작업 을 완성 합 니 다. master 서버 가 여러 요청 임무 에 직면 할 때 처리 속도 가 느 려 집 니 다. 또한 나머지 master 서버 가 요청 을 처리 하지 않 는 것 도 자원 의 낭비 입 니 다. 이때 우 리 는 부하 균형 서 비 스 를 하 는 것 을 고려 합 니 다.
  • 이번 구축 부하 균형 사용 nginx 서비스 4 층 부하 균형, keepalived 주소 하늘하늘
  • 실험 배치
    실험 환경
  • lb 01: 192.168.8.0.19 (부하 균형 서버)
  • lb 02: 192.168.80.20 (부하 균형 서버)
  • Master01:192.168.80.12
  • Master01:192.168.80.11
  • Node01:192.168.80.13
  • Node02:192.168.80.14

  • 다 중 마스터 배치
  • master 01 서버 조작
    [root@master01 kubeconfig]# scp -r /opt/kubernetes/ [email protected]:/opt     //    kubernetes   master02
    The authenticity of host '192.168.80.11 (192.168.80.11)' can't be established.
    ECDSA key fingerprint is SHA256:Ih0NpZxfLb+MOEFW8B+ZsQ5R8Il2Sx8dlNov632cFlo.
    ECDSA key fingerprint is MD5:a9:ee:e5:cc:40:c7:9e:24:5b:c1:cd:c1:7b:31:42:0f.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added '192.168.80.11' (ECDSA) to the list of known hosts.
    [email protected]'s password:
    token.csv                                                                  100%   84    61.4KB/s   00:00
    kube-apiserver                                                             100%  929     1.6MB/s   00:00
    kube-scheduler                                                             100%   94   183.2KB/s   00:00
    kube-controller-manager                                                    100%  483   969.2KB/s   00:00
    kube-apiserver                                                             100%  184MB 106.1MB/s   00:01
    kubectl                                                                    100%   55MB  85.9MB/s   00:00
    kube-controller-manager                                                    100%  155MB 111.9MB/s   00:01
    kube-scheduler                                                             100%   55MB 115.8MB/s   00:00
    ca-key.pem                                                                 100% 1675     2.7MB/s   00:00
    ca.pem                                                                     100% 1359     2.6MB/s   00:00
    server-key.pem                                                             100% 1679     2.5MB/s   00:00
    server.pem                                                                 100% 1643     2.7MB/s   00:00
    [root@master01 kubeconfig]# scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager, kube-scheduler}.service [email protected]:/usr/lib/systemd/system    //  master          
    [email protected]'s password:
    kube-apiserver.service                                                     100%  282   274.4KB/s   00:00
    kube-controller-manager.service                                            100%  317   403.5KB/s   00:00
    kube-scheduler.service                                                     100%  281   379.4KB/s   00:00
    [root@master01 kubeconfig]# scp -r /opt/etcd/ [email protected]:/opt/    //    :master02     etcd  ,  apiserver         master01    etcd   master02  
    [email protected]'s password:
    etcd                                                                       100%  509   275.7KB/s   00:00
    etcd                                                                       100%   18MB  95.3MB/s   00:00
    etcdctl                                                                    100%   15MB  75.1MB/s   00:00
    ca-key.pem                                                                 100% 1679   941.1KB/s    00:00
    ca.pem                                                                     100% 1265     1.6MB/s   00:00
    server-key.pem                                                             100% 1675     2.0MB/s   00:00
    server.pem                                                                 100% 1338     1.5MB/s   00:00
  • master 02 서버 조작
    [root@master02 ~]# systemctl stop firewalld.service     //     
    [root@master02 ~]# setenforce 0                        //  selinux
    [root@master02 ~]# vim /opt/kubernetes/cfg/kube-apiserver     //    
    ...
    --etcd-servers=https://192.168.80.12:2379,https://192.168.80.13:2379,https://192.168.80.14:2379 \
    --bind-address=192.168.80.11 \       //  IP  
    --secure-port=6443 \
    --advertise-address=192.168.80.11 \   //  IP  
    --allow-privileged=true \
    --service-cluster-ip-range=10.0.0.0/24 \
    ...
    :wq
    [root@master02 ~]# systemctl start kube-apiserver.service   //  apiserver  
    [root@master02 ~]# systemctl enable kube-apiserver.service  //      
    Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/ systemd/system/kube-apiserver.service.
    [root@master02 ~]# systemctl start kube-controller-manager.service   //  controller-manager
    [root@master02 ~]# systemctl enable kube-controller-manager.service  //      
    Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service. 
    [root@master02 ~]# systemctl start kube-scheduler.service            //  scheduler
    [root@master02 ~]# systemctl enable kube-scheduler.service           //      
    Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/ systemd/system/kube-scheduler.service.
    [root@master02 ~]# vim /etc/profile       //        
    ...
    export PATH=$PATH:/opt/kubernetes/bin/
    :wq
    [root@master02 ~]# source /etc/profile     //    
    [root@master02 ~]# kubectl get node        //      
    NAME            STATUS   ROLES    AGE    VERSION
    192.168.80.13   Ready       146m   v1.12.3
    192.168.80.14   Ready       144m   v1.12.3    // master    
  • 부하 균형 배치
  • lb 01, lb 02 동기 화 작업 keepalived 서비스 프로필 다운로드 추출 코드: fkoh
    [root@lb01 ~]# systemctl stop firewalld.service
    [root@lb01 ~]# setenforce 0
    [root@lb01 ~]# vim /etc/yum.repos.d/nginx.repo   //  nginx  yum 
    [nginx]
    name=nginx repo
    baseurl=http://nginx.org/packages/centos/7/$basearch/
    gpgcheck=0
    :wq
    [root@lb01 yum.repos.d]# yum list     //    yum
         :fastestmirror
    base                                                                                  | 3.6 kB  00:00:00
    extras                                                                                | 2.9 kB   00:00:00
    ...
    [root@lb01 yum.repos.d]# yum install nginx -y     //  nginx   
         :fastestmirror
    Loading mirror speeds from cached hostfile
    * base: mirrors.aliyun.com
    * extras: mirrors.163.com
    ...
    [root@lb01 yum.repos.d]# vim /etc/nginx/nginx.conf    //  nginx    
    ...
    events {
    worker_connections  1024;
    }
    
    stream {                     //        
    log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
    access_log /var/log/nginx/k8s-access.log main;
    
    upstream k8s-apiserver {
        server 192.168.80.12:6443;          //  IP  
        server 192.168.80.11:6443;
    }
    server {
        listen 6443;
        proxy_pass k8s-apiserver;
    }
    }
    
    http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    ...
    :wq
    [root@lb01 yum.repos.d]# systemctl start nginx       //  nginx              nginx  
    [root@lb01 yum.repos.d]# yum install keepalived -y    //  keepalived   
         :fastestmirror
    Loading mirror speeds from cached hostfile
    * base: mirrors.aliyun.com
    * extras: mirrors.163.com
    ...
    [root@lb01 yum.repos.d]# mount.cifs //192.168.80.2/shares/K8S/k8s02 /mnt/     //       
    Password for root@//192.168.80.2/shares/K8S/k8s02:
    [root@lb01 yum.repos.d]# cp /mnt/keepalived.conf /etc/keepalived/keepalived.conf  //        keepalived           
    cp:    "/etc/keepalived/keepalived.conf"? yes
    [root@lb01 yum.repos.d]# vim /etc/keepalived/keepalived.conf       //      
    ...
    vrrp_script check_nginx {
    script "/etc/nginx/check_nginx.sh"    //        
    }
    
    vrrp_instance VI_1 {
    state MASTER
    interface ens33            //      
    virtual_router_id 51   //VRRP    ID  ,        
    priority 100           //   ,       90
    advert_int 1            //  VRRP          ,  1 
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.80.100/24       //    
    }
    track_script {
        check_nginx
    }
    }
    //        
    :wq
  • lb 02 서버 keepalived 프로필 수정
    [root@lb02 ~]# vim /etc/keepalived/keepalived.conf
    ...
    vrrp_script check_nginx {
    script "/etc/nginx/check_nginx.sh"    //        
    }
    
    vrrp_instance VI_1 {
    state BACKUP         //     backup
    interface ens33      //    
    virtual_router_id 51   //VRRP    ID  , >       
    priority 90       //   ,       90
    advert_int 1      //  VRRP          ,  1 
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.80.100/24       //  IP  
    }
    track_script {
        check_nginx
    }
    }
    //        
    :wq
  • lb 01, lb 02 동기 화 조작
    [root@lb01 yum.repos.d]# vim /etc/nginx/check_nginx.sh   //    nginx    
    count=$(ps -ef |grep nginx |egrep -cv "grep|$$")
    
    if [ "$count" -eq 0 ];then
    systemctl stop keepalived
    fi
    :wq
    chmod +x /etc/nginx/check_nginx.sh     //        
    [root@lb01 yum.repos.d]# systemctl start keepalived     //    
  • lb 01 서버 조작
    [root@lb01 ~]# ip a      //      
    1: lo:  mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
    2: ens33:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:e9:04:ba brd ff:ff:ff:ff:ff:ff
    inet 192.168.80.19/24 brd 192.168.80.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.80.100/24 scope global secondary ens33    //        
       valid_lft forever preferred_lft forever
    inet6 fe80::c3ab:d7ec:1adf:c5df/64 scope link
       valid_lft forever preferred_lft forever
  • lb 02 서버 조작
    [root@lb02 ~]# ip a          //      
    1: lo:  mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
    2: ens33:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:7d:c7:ab brd ff:ff:ff:ff:ff:ff
    inet 192.168.80.20/24 brd 192.168.80.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::cd8b:b80c:8deb:251f/64 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::c3ab:d7ec:1adf:c5df/64 scope link tentative dadfailed
       valid_lft forever preferred_lft forever       //    IP   lb02      
  • lb 01 서버 가 nginx 서 비 스 를 중단 하고 lb 02 서버 IP 주소 에서 가상 IP 주소 가 성공 적 으로 이동 하 는 지 확인 합 니 다
    [root@lb01 ~]# systemctl stop nginx.service
    [root@lb01 nginx]# ip a
    1: lo:  mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
    2: ens33:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:e9:04:ba brd ff:ff:ff:ff:ff:ff
    inet 192.168.80.19/24 brd 192.168.80.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::c3ab:d7ec:1adf:c5df/64 scope link
       valid_lft forever preferred_lft forever
    [root@lb02 ~]# ip a           // lb02     
    1: lo:  mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
    2: ens33:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:7d:c7:ab brd ff:ff:ff:ff:ff:ff
    inet 192.168.80.20/24 brd 192.168.80.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.80.100/24 scope global secondary ens33      //       lb02 
       valid_lft forever preferred_lft forever
    inet6 fe80::cd8b:b80c:8deb:251f/64 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::c3ab:d7ec:1adf:c5df/64 scope link tentative dadfailed
       valid_lft forever preferred_lft forever
  • lb 01 서버 에서 nginx, keepalived 서비스 다시 시작
    [root@lb01 nginx]# systemctl start nginx
    [root@lb01 nginx]# systemctl start keepalived.service
    [root@lb01 nginx]# ip a
    1: lo:  mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
    2: ens33:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:e9:04:ba brd ff:ff:ff:ff:ff:ff
    inet 192.168.80.19/24 brd 192.168.80.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.80.100/24 scope global secondary ens33     //                   
       valid_lft forever preferred_lft forever
    inet6 fe80::c3ab:d7ec:1adf:c5df/64 scope link
       valid_lft forever preferred_lft forever
  • 모든 node 노드 에서 프로필 수정
    [root@node01 ~]# vim /opt/kubernetes/cfg/bootstrap.kubeconfig
    ...
    server: https://192.168.80.100:6443
    ...
    :wq
    [root@node01 ~]# vim /opt/kubernetes/cfg/kubelet.kubeconfig
    ...
    server: https://192.168.80.100:6443
    ...
    :wq
    [root@node01 ~]# vim /opt/kubernetes/cfg/kube-proxy.kubeconfig
    ...
    server: https://192.168.80.100:6443
    ...
    :wq
    [root@node01 ~]# systemctl restart kubelet.service    //    
    [root@node01 ~]# systemctl restart kube-proxy.service
  • lb 01 서버 에서 로그 정보 보기
    [root@lb01 nginx]# tail /var/log/nginx/k8s-access.log
    192.168.80.13 192.168.80.12:6443 - [11/Feb/2020:15:23:52 +0800] 200 1118
    192.168.80.13 192.168.80.11:6443 - [11/Feb/2020:15:23:52 +0800] 200 1119
    192.168.80.14 192.168.80.12:6443 - [11/Feb/2020:15:26:01 +0800] 200 1119
    192.168.80.14 192.168.80.12:6443 - [11/Feb/2020:15:26:01 +0800] 200 1120
  • master 01 에서 테스트 플랫폼 기능 조작
    [root@master01 ~]# kubectl run nginx --image=nginx     //  pod  
    kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.
    deployment.apps/nginx created
    [root@master01 ~]# kubectl get pods        //  pod  
    NAME                    READY   STATUS    RESTARTS   AGE
    nginx-dbddb74b8-sdcpl   1/1     Running   0          33m   //    
    [root@master01 ~]# kubectl logs nginx-dbddb74b8-sdcpl    //      
    Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-dbddb74b8-sdcpl)    //   
    [root@master01 ~]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous     //        
    clusterrolebinding.rbac.authorization.k8s.io/cluster-system-anonymous created
    [root@master01 ~]# kubectl logs nginx-dbddb74b8-sdcpl    //      
    [root@master01 ~]#     //        ,            
  • node 노드 에서 nginx 페이지 방문
    [root@master01 ~]# kubectl get pods -o wide   //  master01     pod    
    NAME                    READY   STATUS    RESTARTS   AGE   IP            NODE            NOMINATED NODE
    nginx-dbddb74b8-sdcpl   1/1     Running   0          38m   172.17.33.2   192.168.80.14   
    [root@node01 ~]# curl 172.17.33.2     // node           
    
    
    
    Welcome to nginx!
    
    
    
    

    Welcome to nginx!

    If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

    For online documentation and support please refer to nginx.org.
    Commercial support is available at nginx.com.

    Thank you for using nginx.

  • master 01 서버 로 돌아 가 로그 정보 보기
    [root@master01 ~]# kubectl logs nginx-dbddb74b8-sdcpl
    172.17.12.0 - - [12/Feb/2020:06:45:54 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"    //      
    다 중 노드 구축 과 부하 균형 설정 완료
  • 좋은 웹페이지 즐겨찾기