GKE에서 외부에 액세스할 때 GIP 고정
소개
GKE로 Pod로부터 외부에 액세스하는 경우, 노드의 IP가 된다. 그리고 IP는 에페메랄로 오토스케일링했을 때에 동적으로 증감한다.
요구사항에 따라서는 GKE로부터의 액세스를 고정 IP로 해야 하는 경우가 있다. (예 : 외부 협력 서버가 IP 주소로 필터링하는 경우 등)
그러한 경우에 어떻게 하면 좋을까에 대해 생각해 보았다.
개요
결론에서 말하면, NATGateWay가 되는 GCE 인스턴스를 작성해, 특정의 통신의 루트만 NAT용 인스턴스를 통과하게 한다.
주의점으로서는, NatGateway의 GCE의 대역에 대해 고려할 필요가 있다.
GCE는 2Gbps/core의 Cap가 존재하기 때문이다.
대역이 필요한 경우는 NatGateway의 Core 수를 올려 두어야 한다.
다음 구성 이미지.
환경 구축
VPC 생성
NATGW와 소스가 되는 GKE 클러스터를 이동하는 VPC를 만듭니다.
gcloud compute --project=my-project networks create sakon-vpc --mode=custom
gcloud compute --project=my-project networks subnets create sakon-subnet-1 --network=sakon-vpc --region=asia-northeast1 --range=10.0.0.0/16
FW 규칙 작성
GCPのコンソールからsshでポチポチしたかったから、source-rangeがガバガバな設定にしておく
gcloud compute --project=my-project firewall-rules create sakon-allow-ssh --direction=INGRESS --priority=1000 --network=sakon-vpc --action=ALLOW --rules=tcp:22 --source-ranges=0.0.0.0/0
GKEクラスタとNATGatewayが通信するためのFWルールを作成
gcloud compute --project=my-project firewall-rules create allow-gke-cluster --direction=INGRESS --priority=1000 --network=sakon-vpc --action=ALLOW --rules=tcp --source-tags=nat-test --target-tags=sakon-gw
Route 확인
$ gcloud compute routes list | head -1 && gcloud compute routes list | grep sakon-vpc
NAME NETWORK DEST_RANGE NEXT_HOP PRIORITY
default-route-96d12fd4308f701a sakon-vpc 10.0.0.0/16 sakon-vpc 1000
default-route-e28dc9fb60f719d7 sakon-vpc 0.0.0.0/0 default-internet-gateway 1000
GKE Cluster 만들기
※Node에는 --tags "nat-test"를 붙이고 있다
gcloud beta container --project "my-project" clusters create "sakon-source-cluster" --zone "asia-northeast1-a" --username "admin" \
--cluster-version "1.10.5-gke.0" --machine-type "n1-standard-1" --image-type "COS" --disk-type "pd-standard" --disk-size "100" \
--scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" \
--num-nodes "1" --enable-cloud-logging --enable-cloud-monitoring --network "sakon-vpc" --tags "nat-test" \
--subnetwork "sakon-subnet-1" --addons HorizontalPodAutoscaling,HttpLoadBalancing --no-enable-autoupgrade --enable-autorepair
연계용 서버 작성
연계용 서버의 예로서, 적당히 Nginx가 가동하는 GCE를 다른 VPC에 작성한다.
(기본 VPC에서 생성)
gcloud beta compute --project=my-project instances create sakon-dest-nginx \
--zone=asia-northeast1-a --machine-type=n1-standard-1 --network=default \
--network-tier=PREMIUM --maintenance-policy=MIGRATE \
--service-account=XXXXXXXXXXXX-compute@developer.gserviceaccount.com \
--scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append \
--tags=http-server,https-server --image=centos-7-v20180611 \
--image-project=centos-cloud --boot-disk-size=10GB --boot-disk-type=pd-standard --boot-disk-device-name=sakon-dest-nginx
Nginx를 시작하십시오.
$ sudo yum -y install nginx
$ sudo systemctl start nginx
NATGateWay용 인스턴스 만들기
주목해야 할 것은 --can-ip-forward를 활성화하고 --tags=sakon-gwgcloud beta compute --project=my-project instances create sakon-gw-1 --zone=asia-northeast1-a --machine-type=n1-standard-1 --subnet=sakon-subnet-1 --network-tier=PREMIUM --can-ip-forward --maintenance-policy=MIGRATE --service-account=XXXXXXXXXXXX-compute@developer.gserviceaccount.com --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append --image=centos-7-v20180611 --image-project=centos-cloud --boot-disk-size=10GB --boot-disk-type=pd-standard --boot-disk-device-name=sakon-gw-1 --tags="sakon-gw"
可用性を考慮して2号機も作成
gcloud beta compute --project=my-project instances create sakon-gw-2 --zone=asia-northeast1-b --machine-type=n1-standard-1 --subnet=sakon-subnet-1 --network-tier=PREMIUM --can-ip-forward --maintenance-policy=MIGRATE --service-account=XXXXXXXXXXXX-compute@developer.gserviceaccount.com --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append --tags=sakon-gw --image=centos-7-v20180611 --image-project=centos-cloud --boot-disk-size=10GB --boot-disk-type=pd-standard --boot-disk-device-name=sakon-gw-2
GW용 인스턴스에 IP 마스커레이드 구성
※StartupScript에 넣는 편이 좋을 것 같다.
$ sudo cp -piv /etc/selinux/config /etc/selinux/config.`date "+%Y%m%d"`
$ sudo vi /etc/selinux/config
以下を変更
SELINUX=disabled
$ sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
$ sudo sh -c "echo "net.ipv4.ip_forward = 1" | sudo tee /etc/sysctl.conf"
루트 만들기
GKE의 노드로부터의 통신( nat-test 태그가 있는 것)은 hop-instance 가 NatGateway( sakon-gw-1 or sakon-gw-2 )가 되도록 해 둔다.
gcloud compute --project=my-project routes create sakon-route-1 --network=sakon-vpc --priority=900 --tags=nat-test --destination-range=35.187.195.150/32 --next-hop-instance=sakon-gw-1 --next-hop-instance-zone=asia-northeast1-a
gcloud compute --project=my-project routes create sakon-route-2 --network=sakon-vpc --priority=900 --tags=nat-test --destination-range=35.187.195.150/32 --next-hop-instance=sakon-gw-2 --next-hop-instance-zone=asia-northeast1-b
루트 재확인
$ gcloud compute routes list | head -1 && gcloud compute routes list | grep sakon-vpc
NAME NETWORK DEST_RANGE NEXT_HOP PRIORITY
default-route-96d12fd4308f701a sakon-vpc 10.0.0.0/16 sakon-vpc 1000
default-route-e28dc9fb60f719d7 sakon-vpc 0.0.0.0/0 default-internet-gateway 1000
gke-sakon-source-cluster-8-ab3dd62b-7d0f-11e8-a83e-42010a9200e8 sakon-vpc 10.52.0.0/24 asia-northeast1-a/instances/gke-sakon-source-cluster-default-pool-4804ae70-h3wt 1000
sakon-route-1 sakon-vpc 35.187.195.150/32 asia-northeast1-a/instances/sakon-gw-1 900
sakon-route-2 sakon-vpc 35.187.195.150/32 asia-northeast1-b/instances/sakon-gw-2 900
검증
GKE 출처를 위한 포드 출시
(이번은 테스트용이므로, --rm --restart=Never 로서 종료(exit) 후에 사라지게 한다)
$ kubectl run -it --rm --restart=Never access-source --image alpine /bin/sh
＄ apk add --update curl
$ curl http://35.187.195.150/ > /dev/null
適当に何回か叩く
협력 서버 측에서 액세스 확인
GKE 노드의 IP(35.200.56.61)가 아니라 NAT Gateway용 IP(35.200.25.127 or 35.200.21.54)임을 알 수 있다.
$ sudo tail -f /var/log/nginx/access.log
35.200.25.127 - - [01/Jul/2018:14:17:48 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.25.127 - - [01/Jul/2018:14:17:50 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.25.127 - - [01/Jul/2018:14:17:56 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.21.54 - - [01/Jul/2018:14:17:57 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.21.54 - - [01/Jul/2018:14:17:58 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.21.54 - - [01/Jul/2018:14:17:59 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.25.127 - - [01/Jul/2018:14:18:00 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.21.54 - - [01/Jul/2018:14:18:01 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.25.127 - - [01/Jul/2018:14:18:01 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.21.54 - - [01/Jul/2018:14:18:02 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.21.54 - - [01/Jul/2018:14:18:03 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
Reference
이 문제에 관하여(GKE에서 외부에 액세스할 때 GIP 고정), 우리는 이곳에서 더 많은 자료를 발견하고 링크를 클릭하여 보았다
https://qiita.com/HirokiSakonju/items/b5b236d682eb711b0495
텍스트를 자유롭게 공유하거나 복사할 수 있습니다.하지만 이 문서의 URL은 참조 URL로 남겨 두십시오.
우수한 개발자 콘텐츠 발견에 전념
(Collection and Share based on the CC Protocol.)
gcloud compute --project=my-project networks create sakon-vpc --mode=custom
gcloud compute --project=my-project networks subnets create sakon-subnet-1 --network=sakon-vpc --region=asia-northeast1 --range=10.0.0.0/16
GCPのコンソールからsshでポチポチしたかったから、source-rangeがガバガバな設定にしておく
gcloud compute --project=my-project firewall-rules create sakon-allow-ssh --direction=INGRESS --priority=1000 --network=sakon-vpc --action=ALLOW --rules=tcp:22 --source-ranges=0.0.0.0/0
GKEクラスタとNATGatewayが通信するためのFWルールを作成
gcloud compute --project=my-project firewall-rules create allow-gke-cluster --direction=INGRESS --priority=1000 --network=sakon-vpc --action=ALLOW --rules=tcp --source-tags=nat-test --target-tags=sakon-gw
$ gcloud compute routes list | head -1 && gcloud compute routes list | grep sakon-vpc
NAME NETWORK DEST_RANGE NEXT_HOP PRIORITY
default-route-96d12fd4308f701a sakon-vpc 10.0.0.0/16 sakon-vpc 1000
default-route-e28dc9fb60f719d7 sakon-vpc 0.0.0.0/0 default-internet-gateway 1000
gcloud beta container --project "my-project" clusters create "sakon-source-cluster" --zone "asia-northeast1-a" --username "admin" \
--cluster-version "1.10.5-gke.0" --machine-type "n1-standard-1" --image-type "COS" --disk-type "pd-standard" --disk-size "100" \
--scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" \
--num-nodes "1" --enable-cloud-logging --enable-cloud-monitoring --network "sakon-vpc" --tags "nat-test" \
--subnetwork "sakon-subnet-1" --addons HorizontalPodAutoscaling,HttpLoadBalancing --no-enable-autoupgrade --enable-autorepair
gcloud beta compute --project=my-project instances create sakon-dest-nginx \
--zone=asia-northeast1-a --machine-type=n1-standard-1 --network=default \
--network-tier=PREMIUM --maintenance-policy=MIGRATE \
--service-account=XXXXXXXXXXXX-compute@developer.gserviceaccount.com \
--scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append \
--tags=http-server,https-server --image=centos-7-v20180611 \
--image-project=centos-cloud --boot-disk-size=10GB --boot-disk-type=pd-standard --boot-disk-device-name=sakon-dest-nginx
$ sudo yum -y install nginx
$ sudo systemctl start nginx
gcloud beta compute --project=my-project instances create sakon-gw-1 --zone=asia-northeast1-a --machine-type=n1-standard-1 --subnet=sakon-subnet-1 --network-tier=PREMIUM --can-ip-forward --maintenance-policy=MIGRATE --service-account=XXXXXXXXXXXX-compute@developer.gserviceaccount.com --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append --image=centos-7-v20180611 --image-project=centos-cloud --boot-disk-size=10GB --boot-disk-type=pd-standard --boot-disk-device-name=sakon-gw-1 --tags="sakon-gw"
可用性を考慮して2号機も作成
gcloud beta compute --project=my-project instances create sakon-gw-2 --zone=asia-northeast1-b --machine-type=n1-standard-1 --subnet=sakon-subnet-1 --network-tier=PREMIUM --can-ip-forward --maintenance-policy=MIGRATE --service-account=XXXXXXXXXXXX-compute@developer.gserviceaccount.com --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append --tags=sakon-gw --image=centos-7-v20180611 --image-project=centos-cloud --boot-disk-size=10GB --boot-disk-type=pd-standard --boot-disk-device-name=sakon-gw-2
$ sudo cp -piv /etc/selinux/config /etc/selinux/config.`date "+%Y%m%d"`
$ sudo vi /etc/selinux/config
以下を変更
SELINUX=disabled
$ sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
$ sudo sh -c "echo "net.ipv4.ip_forward = 1" | sudo tee /etc/sysctl.conf"
gcloud compute --project=my-project routes create sakon-route-1 --network=sakon-vpc --priority=900 --tags=nat-test --destination-range=35.187.195.150/32 --next-hop-instance=sakon-gw-1 --next-hop-instance-zone=asia-northeast1-a
gcloud compute --project=my-project routes create sakon-route-2 --network=sakon-vpc --priority=900 --tags=nat-test --destination-range=35.187.195.150/32 --next-hop-instance=sakon-gw-2 --next-hop-instance-zone=asia-northeast1-b
$ gcloud compute routes list | head -1 && gcloud compute routes list | grep sakon-vpc
NAME NETWORK DEST_RANGE NEXT_HOP PRIORITY
default-route-96d12fd4308f701a sakon-vpc 10.0.0.0/16 sakon-vpc 1000
default-route-e28dc9fb60f719d7 sakon-vpc 0.0.0.0/0 default-internet-gateway 1000
gke-sakon-source-cluster-8-ab3dd62b-7d0f-11e8-a83e-42010a9200e8 sakon-vpc 10.52.0.0/24 asia-northeast1-a/instances/gke-sakon-source-cluster-default-pool-4804ae70-h3wt 1000
sakon-route-1 sakon-vpc 35.187.195.150/32 asia-northeast1-a/instances/sakon-gw-1 900
sakon-route-2 sakon-vpc 35.187.195.150/32 asia-northeast1-b/instances/sakon-gw-2 900
$ kubectl run -it --rm --restart=Never access-source --image alpine /bin/sh
＄ apk add --update curl
$ curl http://35.187.195.150/ > /dev/null
適当に何回か叩く
$ sudo tail -f /var/log/nginx/access.log
35.200.25.127 - - [01/Jul/2018:14:17:48 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.25.127 - - [01/Jul/2018:14:17:50 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.25.127 - - [01/Jul/2018:14:17:56 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.21.54 - - [01/Jul/2018:14:17:57 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.21.54 - - [01/Jul/2018:14:17:58 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.21.54 - - [01/Jul/2018:14:17:59 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.25.127 - - [01/Jul/2018:14:18:00 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.21.54 - - [01/Jul/2018:14:18:01 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.25.127 - - [01/Jul/2018:14:18:01 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.21.54 - - [01/Jul/2018:14:18:02 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
35.200.21.54 - - [01/Jul/2018:14:18:03 +0000] "GET / HTTP/1.1" 200 3700 "-" "curl/7.60.0" "-"
Reference
이 문제에 관하여(GKE에서 외부에 액세스할 때 GIP 고정), 우리는 이곳에서 더 많은 자료를 발견하고 링크를 클릭하여 보았다 https://qiita.com/HirokiSakonju/items/b5b236d682eb711b0495텍스트를 자유롭게 공유하거나 복사할 수 있습니다.하지만 이 문서의 URL은 참조 URL로 남겨 두십시오.
우수한 개발자 콘텐츠 발견에 전념
(Collection and Share based on the CC Protocol.)
좋은 웹페이지 즐겨찾기
