FRR on CoreOS 검증
12708 단어 FRRoutingCoreOSsystemd-nspawn
배경
# あるかもしれない状況
"K8S管理者" : docker 落としたらCoreOSへのネットワーク接続が切れたんやけど?
"ネットワーク担当" : それはそうや、ルーティングデーモン(FRR)はdockerコンテナとして動いてるからな
"K8S管理者" : 使い辛い、docker自体のメンテとかやるし別の方法にして
"ネットワーク担当" : あい、わかった (何か良い方法あったっけ...汗)
CoreOS상에서 라우팅 데몬을 동작시켜 서버 L3화하고 있는 환경에서,
마지막 기사, FRR은 도커에서 작동하지만 좋은 방법은 아닙니다.
또는 systemd-nspawn에서 FRR 컨테이너를 만들려고합니다.
systemd-nspawn이란?
systemd+chroot로 움직이는 경량 컨테이너.
# あるかもしれない状況
"K8S管理者" : docker 落としたらCoreOSへのネットワーク接続が切れたんやけど?
"ネットワーク担当" : それはそうや、ルーティングデーモン(FRR)はdockerコンテナとして動いてるからな
"K8S管理者" : 使い辛い、docker自体のメンテとかやるし別の方法にして
"ネットワーク担当" : あい、わかった (何か良い方法あったっけ...汗)
PoC
다음은 systemd-nspawn에서 FRR이 작동 할 때 L3 연결 샘플을 보여줍니다.
GNS3에서 간단한 테스트 환경을 제공하고 CoreOS와 대향 스위치 (CumulusVX)간에 eBGP로 루프백을 교체하십시오.
systemd-nspawn x FRR 컨테이너 만들기
시작하기 전에 ubuntu-18.04를 별도로 준비하여 systemd-nspawn으로 이동하는 FRR 컨테이너를 만듭니다.
(진짜 Alpine으로 하고 싶었지만 machinectl shell로 로그인 문제를 잘 해결할 수 없었기 때문에 타협)
#------------------------------------------------------
# Generate a container
#------------------------------------------------------
apt-get install -y debootstrap
apt-get install -y systemd-container
# Get base image
# https://wiki.archlinux.org/index.php/Systemd-nspawn
# --include で任意のパッケージをインストールしておくことが可能
# dbus 入れておくと machinectl shell でコンテナにログインできる
CONTAINER_NAME=frr
debootstrap --arch amd64 --variant=minbase --include=ubuntu-minimal,systemd-container,tzdata,dbus,wget,gnupg,curl --components=main,universe bionic /var/lib/machines/${CONTAINER_NAME} http://jp.archive.ubuntu.com/ubuntu
machinectl list-images
# systemd
# host の network が使えるようにコンテナに特権を付与
machinectl enable ${CONTAINER_NAME}
vi /etc/systemd/system/machines.target.wants/[email protected]
# > ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-veth -U --settings=override --machine=%i
# ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --settings=override --capability=all --machine=%i
systemctl daemon-reload
# Start container
machinectl start ${CONTAINER_NAME}
machinectl list
#------------------------------------------------------
# Install FRR in the container (https://deb.frrouting.org/)
#------------------------------------------------------
machinectl shell ${CONTAINER_NAME}
curl -s https://deb.frrouting.org/frr/keys.asc | apt-key add -
echo deb https://deb.frrouting.org/frr $(lsb_release -s -c) frr-stable | tee -a /etc/apt/sources.list.d/frr.list
apt-get update
apt-get -y install frr frr-pythontools
# FRR initial configuration
systemctl enable frr
vtysh -c "show version"
hostnamectl set-hostname frr
rm /etc/frr/frr.conf
systemctl stop frr
exit
machinectl stop ${CONTAINER_NAME}
machinectl list
# Export image
machinectl list-images
machinectl export-tar ${CONTAINER_NAME} ${CONTAINER_NAME}.tar
CoreOS 측 설정
#------------------------------------------------------
# Deploy FRR container
#------------------------------------------------------
# scp などでfrr.tar を coreos にコピー
# machinectl pull-tar URL でもよい(machinectl --help参照)
# capabiilty=all で特権コンテナとして動作させる
machinectl import-tar ./frr.tar
machinectl list-images
machinectl enable frr
mkdir /etc/systemd/system/[email protected]
cat > /etc/systemd/system/[email protected]/override.conf <<EOF
[Service]
ExecStart=
ExecStart=/usr/bin/systemd-nspawn --keep-unit --boot --link-journal=try-guest --settings=override --capability=all --machine=%i
EOF
systemctl daemon-reload
#------------------------------------------------------
# CoreOS Network
#------------------------------------------------------
# Parameter
LOOPBACK=2.2.2.2
# Network setting
ip link set ${NIC} up
# BGP Unnumbered 使うので ipv6 をenableに
sysctl -w net.ipv6.conf.all.disable_ipv6=0
sysctl -w net.ipv6.conf.default.disable_ipv6=0
ip addr add dev lo ${LOOPBACK}/32
CoreOS에서 FRR 컨테이너 구성
# Parameter
NIC=ens3
LOOPBACK=2.2.2.2
ASN=65001
# Start container
machinectl start ${CONTAINER_NAME}
machinectl list
machinectl shell frr
# machinectl shell で稼働中コンテナに入らなくても
# /var/lib/machines/frr/etc/frr 配下のファイルを直接修正してもよい
# FRR daemon configuration
cat > /etc/frr/daemons <<EOF
bgpd=yes
vtysh_enable=yes
zebra_options=" -A 127.0.0.1 -s 90000000"
bgpd_options=" -A 127.0.0.1"
EOF
# FRR config
cat > /etc/frr/frr.conf <<EOF
frr version 7.2
frr defaults traditional
hostname frr
no ipv6 forwarding
service integrated-vtysh-config
!
router bgp ${ASN}
bgp router-id ${LOOPBACK}
neighbor TOR peer-group
neighbor TOR remote-as external
neighbor ${NIC} interface peer-group TOR
!
address-family ipv4 unicast
network ${LOOPBACK}/32
exit-address-family
!
line vty
!
EOF
# Restart FRR
systemctl restart frr
systemctl status frr
exit
Switch 설정
CumulusVX를 사용하여 CoreOS의 FRR과 eBGP로 연결. 설정은 최소 필요.
# Parameter
DOWNLINK=swp1
ASN=65000
LOOPBACK=1.1.1.1
# Set loopback
net add loopback lo ip address ${LOOPBACK}/32
net pending
net commit
# Downlink (BGP IPv6 Unnumbered)
net add interface $DOWNLINK ipv6 nd ra-interval 5
net add interface $DOWNLINK ipv6 nd ra-lifetime 15
net pending
net commit
# eBGP
net add bgp autonomous-system $ASN
net add bgp router-id $LOOPBACK
net add bgp ipv4 unicast network ${LOOPBACK}/32
net add bgp neighbor $DOWNLINK remote-as external
net pending
net commit
확인
CoreOS측
# CoreOS のルーティングテーブルを確認
ip route get 1.1.1.1
# >1.1.1.1 via 169.254.0.1 dev ens3 src 2.2.2.2 uid 0
# > cache
# FRR でBGPステータスを確認
machinectl shell frr
vtysh
show bgp summary
#> IPv4 Unicast Summary:
#> BGP router identifier 2.2.2.2, local AS number 65001 vrf-id 0
#> BGP table version 2
#> RIB entries 3, using 552 bytes of memory
#> Peers 1, using 20 KiB of memory
#> Peer groups 1, using 64 bytes of memory
#>
#> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
#> ens3 4 65000 25 25 0 0 0 00:01:01 1
#>
#> Total number of neighbors 1
exit
ping 1.1.1.1
# >PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
# >64 bytes from 1.1.1.1: icmp_seq=1 ttl=64 time=0.598 ms
Switch측
# switch側
# BGPでCoreOSのloopback(2.2.2.2)を学習できている
cumulus@cumulus:~$ net show route
# >=============
# >Codes: K - kernel route, C - connected, S - static, R - RIP,
# > O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
# > T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
# > F - PBR,
# > > - selected route, * - FIB route
# >
# >C>* 1.1.1.1/32 is directly connected, lo, 00:12:09
# >B>* 2.2.2.2/32 [20/0] via fe80::e5a:84ff:fe3e:d200, swp1, 00:01:17
ping 2.2.2.2
# >PING 2.2.2.2 (2.2.2.2) 56(84) bytes of data.
# >64 bytes from 2.2.2.2: icmp_seq=1 ttl=64 time=0.464 ms
끝
docker 대신 systemd-nspawn을 사용하여 FRR을 작동시켜 CoreOS 자체의 네트워크를 L3화할 수 있음을 확인했습니다. 작동하는 것이 작동 관점에서 좋을 것 같습니다.
Reference
이 문제에 관하여(FRR on CoreOS 검증), 우리는 이곳에서 더 많은 자료를 발견하고 링크를 클릭하여 보았다
https://qiita.com/tom7/items/128d91d0b7dacc65be2b
텍스트를 자유롭게 공유하거나 복사할 수 있습니다.하지만 이 문서의 URL은 참조 URL로 남겨 두십시오.
우수한 개발자 콘텐츠 발견에 전념
(Collection and Share based on the CC Protocol.)
#------------------------------------------------------
# Generate a container
#------------------------------------------------------
apt-get install -y debootstrap
apt-get install -y systemd-container
# Get base image
# https://wiki.archlinux.org/index.php/Systemd-nspawn
# --include で任意のパッケージをインストールしておくことが可能
# dbus 入れておくと machinectl shell でコンテナにログインできる
CONTAINER_NAME=frr
debootstrap --arch amd64 --variant=minbase --include=ubuntu-minimal,systemd-container,tzdata,dbus,wget,gnupg,curl --components=main,universe bionic /var/lib/machines/${CONTAINER_NAME} http://jp.archive.ubuntu.com/ubuntu
machinectl list-images
# systemd
# host の network が使えるようにコンテナに特権を付与
machinectl enable ${CONTAINER_NAME}
vi /etc/systemd/system/machines.target.wants/[email protected]
# > ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-veth -U --settings=override --machine=%i
# ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --settings=override --capability=all --machine=%i
systemctl daemon-reload
# Start container
machinectl start ${CONTAINER_NAME}
machinectl list
#------------------------------------------------------
# Install FRR in the container (https://deb.frrouting.org/)
#------------------------------------------------------
machinectl shell ${CONTAINER_NAME}
curl -s https://deb.frrouting.org/frr/keys.asc | apt-key add -
echo deb https://deb.frrouting.org/frr $(lsb_release -s -c) frr-stable | tee -a /etc/apt/sources.list.d/frr.list
apt-get update
apt-get -y install frr frr-pythontools
# FRR initial configuration
systemctl enable frr
vtysh -c "show version"
hostnamectl set-hostname frr
rm /etc/frr/frr.conf
systemctl stop frr
exit
machinectl stop ${CONTAINER_NAME}
machinectl list
# Export image
machinectl list-images
machinectl export-tar ${CONTAINER_NAME} ${CONTAINER_NAME}.tar
#------------------------------------------------------
# Deploy FRR container
#------------------------------------------------------
# scp などでfrr.tar を coreos にコピー
# machinectl pull-tar URL でもよい(machinectl --help参照)
# capabiilty=all で特権コンテナとして動作させる
machinectl import-tar ./frr.tar
machinectl list-images
machinectl enable frr
mkdir /etc/systemd/system/[email protected]
cat > /etc/systemd/system/[email protected]/override.conf <<EOF
[Service]
ExecStart=
ExecStart=/usr/bin/systemd-nspawn --keep-unit --boot --link-journal=try-guest --settings=override --capability=all --machine=%i
EOF
systemctl daemon-reload
#------------------------------------------------------
# CoreOS Network
#------------------------------------------------------
# Parameter
LOOPBACK=2.2.2.2
# Network setting
ip link set ${NIC} up
# BGP Unnumbered 使うので ipv6 をenableに
sysctl -w net.ipv6.conf.all.disable_ipv6=0
sysctl -w net.ipv6.conf.default.disable_ipv6=0
ip addr add dev lo ${LOOPBACK}/32
# Parameter
NIC=ens3
LOOPBACK=2.2.2.2
ASN=65001
# Start container
machinectl start ${CONTAINER_NAME}
machinectl list
machinectl shell frr
# machinectl shell で稼働中コンテナに入らなくても
# /var/lib/machines/frr/etc/frr 配下のファイルを直接修正してもよい
# FRR daemon configuration
cat > /etc/frr/daemons <<EOF
bgpd=yes
vtysh_enable=yes
zebra_options=" -A 127.0.0.1 -s 90000000"
bgpd_options=" -A 127.0.0.1"
EOF
# FRR config
cat > /etc/frr/frr.conf <<EOF
frr version 7.2
frr defaults traditional
hostname frr
no ipv6 forwarding
service integrated-vtysh-config
!
router bgp ${ASN}
bgp router-id ${LOOPBACK}
neighbor TOR peer-group
neighbor TOR remote-as external
neighbor ${NIC} interface peer-group TOR
!
address-family ipv4 unicast
network ${LOOPBACK}/32
exit-address-family
!
line vty
!
EOF
# Restart FRR
systemctl restart frr
systemctl status frr
exit
# Parameter
DOWNLINK=swp1
ASN=65000
LOOPBACK=1.1.1.1
# Set loopback
net add loopback lo ip address ${LOOPBACK}/32
net pending
net commit
# Downlink (BGP IPv6 Unnumbered)
net add interface $DOWNLINK ipv6 nd ra-interval 5
net add interface $DOWNLINK ipv6 nd ra-lifetime 15
net pending
net commit
# eBGP
net add bgp autonomous-system $ASN
net add bgp router-id $LOOPBACK
net add bgp ipv4 unicast network ${LOOPBACK}/32
net add bgp neighbor $DOWNLINK remote-as external
net pending
net commit
# CoreOS のルーティングテーブルを確認
ip route get 1.1.1.1
# >1.1.1.1 via 169.254.0.1 dev ens3 src 2.2.2.2 uid 0
# > cache
# FRR でBGPステータスを確認
machinectl shell frr
vtysh
show bgp summary
#> IPv4 Unicast Summary:
#> BGP router identifier 2.2.2.2, local AS number 65001 vrf-id 0
#> BGP table version 2
#> RIB entries 3, using 552 bytes of memory
#> Peers 1, using 20 KiB of memory
#> Peer groups 1, using 64 bytes of memory
#>
#> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
#> ens3 4 65000 25 25 0 0 0 00:01:01 1
#>
#> Total number of neighbors 1
exit
ping 1.1.1.1
# >PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
# >64 bytes from 1.1.1.1: icmp_seq=1 ttl=64 time=0.598 ms
# switch側
# BGPでCoreOSのloopback(2.2.2.2)を学習できている
cumulus@cumulus:~$ net show route
# >=============
# >Codes: K - kernel route, C - connected, S - static, R - RIP,
# > O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
# > T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
# > F - PBR,
# > > - selected route, * - FIB route
# >
# >C>* 1.1.1.1/32 is directly connected, lo, 00:12:09
# >B>* 2.2.2.2/32 [20/0] via fe80::e5a:84ff:fe3e:d200, swp1, 00:01:17
ping 2.2.2.2
# >PING 2.2.2.2 (2.2.2.2) 56(84) bytes of data.
# >64 bytes from 2.2.2.2: icmp_seq=1 ttl=64 time=0.464 ms
docker 대신 systemd-nspawn을 사용하여 FRR을 작동시켜 CoreOS 자체의 네트워크를 L3화할 수 있음을 확인했습니다. 작동하는 것이 작동 관점에서 좋을 것 같습니다.
Reference
이 문제에 관하여(FRR on CoreOS 검증), 우리는 이곳에서 더 많은 자료를 발견하고 링크를 클릭하여 보았다 https://qiita.com/tom7/items/128d91d0b7dacc65be2b텍스트를 자유롭게 공유하거나 복사할 수 있습니다.하지만 이 문서의 URL은 참조 URL로 남겨 두십시오.
우수한 개발자 콘텐츠 발견에 전념
(Collection and Share based on the CC Protocol.)
좋은 웹페이지 즐겨찾기
