Docker + keepalived로 쉬운로드 밸런서

이번에는 fleet + docker + keepalived로 쉽게로드 밸런서를 구축하고 싶습니다.

먼저 docker hub에 위의 구성을 목표로 하는 image가 있는지 찾아 옵니다.

lesaux/docker-keepalived

로드 밸런서의 조합은 이전에 구축한 CoreOS 환경에서 다음 구성으로 이동하고 싶습니다.



※알기 쉬운 그림을 쓸 수 없다・・・・

systemd 용 서비스 파일 작성

※파일명은 케에빠ゔょd@1. 세르 ゔ 과 케에빠ゔょd@2. 세르 ゔ 와 2개 작성해 주세요

[email protected]

[Unit]
Description=KeepALived
After=docker.service
Requires=docker.service

[Service]
TimeoutStartSec=30min
RestartSec=30
Restart=always
ExecStartPre=/usr/bin/docker pull lesaux/keepalived
ExecStartPre=-/usr/bin/docker kill %p-%i
ExecStartPre=-/usr/bin/docker rm %p-%i
ExecStart=/usr/bin/docker run --rm \
  --name %p-%i \
  --net=host \
  -v /mnt/keepalived/keepalived.conf:/etc/keepalived/keepalived.conf \
  --privileged=true \
  -e affinity:container==%p-%i \
  -e VIP=192.168.0.50 \
  lesaux/keepalived
ExecStop=-/usr/bin/docker stop -t 20 %p-%i

[X-Fleet]
MachineID=5b1639bd3cc347cf8fac0b9f597369e3

※각 머신 ID는 이하의 커멘드로 취득 가능합니다*

$ export FLEETCTL_SSH_USERNAME=core
$ export FLEETCTL_TUNNEL=192.168.0.10
$ fleetctl list-machines --full
MACHINE                 IP      METADATA
4fb30e282c004f1794df9e91e56b14fb    192.168.0.31    cabinet=two,role=workers
5b1639bd3cc347cf8fac0b9f597369e3    192.168.0.10    cabinet=one,role=services
703c849bcb924af5891ca5aae95e4e89    192.168.0.21    cabinet=two,role=workers
a6776a02935e4f01857364587836e338    192.168.0.20    cabinet=one,role=services
fffe34c5dcfa4aabbb1ba684101e521e    192.168.0.30    cabinet=one,role=services

coreos-01 및 coreos-03 서버에 다음 keepalived.conf 설정

$ ssh [email protected]
$ sudo mkdir -p /mnt/keepalived
$ sudo vim /mnt/keepalived/keepalived.conf
! Configuration File for keepalived

vrrp_instance VI_1 {
    state BACKUP
    interface enp0s3
    virtual_router_id 1
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.0.50/24 dev enp0s3
    }
    unicast_peer {
        192.168.0.10/24 dev enp0s3
        192.168.0.11/24 dev enp0s3
    }
}

$ ssh [email protected]
$ sudo mkdir -p /mnt/keepalived
$ sudo vim /mnt/keepalived/keepalived.conf
! Configuration File for keepalived

vrrp_instance VI_1 {
    state BACKUP
    interface enp0s3
    virtual_router_id 1
    priority 50
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.0.50/24 dev enp0s3
    }
    unicast_peer {
        192.168.0.10/24 dev enp0s3
        192.168.0.11/24 dev enp0s3
    }
}

Keepalived의 Docker 등록

$ fleetctl submit keepalived@{1,2} #systemdへのファイル送信
$ fleetctl load keepalived@{1,2} #systemd自動起動の設定
$ fleetctl start keepalived@{1,2} #systemd起動
$ fleetctl list-units --full #cluster/worker内のsystemd状況確認
UNIT            MACHINE                     ACTIVE  SUB
[email protected]    5b1639bd3cc347cf8fac0b9f597369e3/192.168.0.10   active  running
[email protected]    6d28316711484f039eca4408627fdb0c/192.168.0.11   active  running

VIP가 coreos-01에 대해

#coreos-01にアクセス
$ ssh [email protected]
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:79:c4:9d brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.10/24 brd 192.168.0.255 scope global enp0s3
       valid_lft forever preferred_lft forever
    inet 192.168.0.50/32 scope global enp0s3
       valid_lft forever preferred_lft forever

#coreos-04にアクセス
$ ssh [email protected]
$ ip a
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:ad:bd:0f brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.11/24 brd 192.168.0.255 scope global enp0s3

#ローカルからping 192.168.0.50
ping 192.168.0.50
PING 192.168.0.50 (192.168.0.50): 56 data bytes
64 bytes from 192.168.0.50: icmp_seq=0 ttl=64 time=0.353 ms
64 bytes from 192.168.0.50: icmp_seq=1 ttl=64 time=0.440 ms
64 bytes from 192.168.0.50: icmp_seq=2 ttl=64 time=0.538 ms

coreos-01에 VIP 192.168.0.50이 붙어 핑이 돌아오는 것도 확인할 수 있었습니다.

계속해서 coreos-01을 다운시켜 VIP가의 페일오버 체크

※이어서 ping이 어느 정도 부족한지 보았습니다. *

$ ssh [email protected]
$ sudo shutdown -h now
$ ssh [email protected]
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:ad:bd:0f brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.11/24 brd 192.168.0.255 scope global enp0s3
       valid_lft forever preferred_lft forever
    inet 192.168.0.50/24 scope global secondary enp0s3

확실히 움직이고 있습니다.

ping의 경우 장애 복구까지 5%의 ping 패킷 손실이 발생했습니다.
※이 건에 대해서는 keepalived.conf에서 한층 더 향상할 수 있을 것 같습니다.

다음은 VIP를 가진 서버가 Nginx의 NAT LB로 움직일 때까지의 설정을 해보고 싶습니다.

참고

Docker-keepalived
systemd.unit

마지막 기사 소개:

【CoreOS】cloud-config 해설~설치
Mac + Virtualbox + CoreOS + etcd2 + fleet 기본 설정 (1)
Mac + Virtualbox + CoreOS + etcd2 + fleet 기본 설정 (2)
Mac + Virtualbox + CoreOS + etcd2 + fleet 기본 설정 (완료)