소개

Red Hat OpenShift on IBM Cloud의 VPC 환경에서 type:LoadBalancer 의 서비스를 작성할 때의 어노테이션을 시도했다. 이용 환경으로서는, 멀티 존으로 구성되어 있어, 각각의 존마다 2대씩의 Worker node가 배치되어 있습니다.

노드

$ ibmcloud oc worker ls --cluster myroksclustervpc --show-pools
OK
ID                                                       Primary IP     Flavor     State    Status   Zone         Version                 Worker Pool
kube-bru1t0nd075uqsfusee0-myroksclust-default-00000147   10.240.0.5     bx2.4x16   normal   Ready    us-south-1   4.3.25_1527_openshift   default
kube-bru1t0nd075uqsfusee0-myroksclust-default-0000026a   10.240.0.4     bx2.4x16   normal   Ready    us-south-1   4.3.25_1527_openshift   default
kube-bru1t0nd075uqsfusee0-myroksclust-default-00000399   10.240.128.5   bx2.4x16   normal   Ready    us-south-3   4.3.25_1527_openshift   default
kube-bru1t0nd075uqsfusee0-myroksclust-default-00000410   10.240.128.4   bx2.4x16   normal   Ready    us-south-3   4.3.25_1527_openshift   default
kube-bru1t0nd075uqsfusee0-myroksclust-default-00000586   10.240.64.5    bx2.4x16   normal   Ready    us-south-2   4.3.25_1527_openshift   default
kube-bru1t0nd075uqsfusee0-myroksclust-default-000006a5   10.240.64.4    bx2.4x16   normal   Ready    us-south-2   4.3.25_1527_openshift   default

※2020/07/04 현재, service.kubernetes.io/ibm-load-balancer-cloud-provider-enable-features: "proxy-protocol" 는 이용할 수 없습니다. 이것은 현재 개발중인 기능이며 문서의 실수처럼 보입니다.

Private 지정

어노테이션을 지정하지 않는 경우는 디폴트로 Public NW측에 공개된다.

service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type: <public_or_private> 는 Public NW 측에 공개하는 LoadBalancer인지, Private NW 측에 공개하는 LoadBalancer인지를 지정할 수 있다. 이번은 명시적으로 Private 지정을 해 본다.

hello-world-private.yaml

apiVersion: v1
kind: Service
metadata:
 name: hello-world-private
 namespace: syasuda
 annotations:
   service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type: private
spec:
 type: LoadBalancer
 selector:
   deploymentconfig: hello-world
 ports:
  - name: http
    protocol: TCP
    port: 8080
    targetPort: 8080

$ oc apply -f hello-world-private.yaml

$ oc get service hello-world-private
NAME                  TYPE           CLUSTER-IP     EXTERNAL-IP                            PORT(S)          AGE
hello-world-private   LoadBalancer   172.21.34.34   xxxxxxxx-us-south.lb.appdomain.cloud   8080:31711/TCP   6m9s

$ oc describe service hello-world-private
Name:                     hello-world-private
Namespace:                syasuda
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type":"private"}...
                          service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type: private
Selector:                 deploymentconfig=hello-world
Type:                     LoadBalancer
IP:                       172.21.34.34
LoadBalancer Ingress:     xxxxxxxx-us-south.lb.appdomain.cloud
Port:                     http  8080/TCP
TargetPort:               8080/TCP
NodePort:                 http  31711/TCP
Endpoints:                172.17.111.9:8080,172.17.115.137:8080,172.17.123.76:8080 + 2 more...
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason                           Age                    From                Message
  ----     ------                           ----                   ----                -------
  Warning  CreatingCloudLoadBalancerFailed  4m35s (x5 over 5m48s)  ibm-cloud-provider  Error on cloud load balancer kube-bru1t0nd075uqsfusee0-408ea2409e574f6fbe066a0120aa6efc for service syasuda/hello-world-private with UID 408ea240-9e57-4f6f-be06-6a0120aa6efc: LoadBalancer is busy: offline/create_pending
  Warning  SyncLoadBalancerFailed           4m35s (x5 over 5m48s)  service-controller  Error syncing load balancer: failed to ensure load balancer: Error on cloud load balancer kube-bru1t0nd075uqsfusee0-408ea2409e574f6fbe066a0120aa6efc for service syasuda/hello-world-private with UID 408ea240-9e57-4f6f-be06-6a0120aa6efc: LoadBalancer is busy: offline/create_pending
  Normal   EnsuringLoadBalancer             3m15s (x7 over 6m12s)  service-controller  Ensuring load balancer
  Normal   EnsuredLoadBalancer              3m5s (x2 over 5m57s)   service-controller  Ensured load balancer

확실히 할당된 이 FQDN을 이름 해석하면 Private IP가 된다.
또한 VPC Load Balancer 인스턴스가 다른 Zone에 배치되어 있음을 IP 주소 정보에서 알 수 있습니다.

$ dig b3160ca4-us-south.lb.appdomain.cloud
dig A +noall +answer @1.1.1.1 xxxxxxxx-us-south.lb.appdomain.cloud
xxxxxxxx-us-south.lb.appdomain.cloud. 120 IN A  10.240.64.10
xxxxxxxx-us-south.lb.appdomain.cloud. 120 IN A  10.240.128.9

Private 지정 + Zone 지정

이전 절의 결과에서 알 수 있듯이 주석을 지정하지 않으면 기본적으로 VPC Load Balancer가 여러 Zone에 걸쳐 배치됩니다. 또한 이 VPC Load Balancer는 임의의 Zone에 있는 Worker Node에 할당한다.

service.kubernetes.io/ibm-load-balancer-cloud-provider-zone: "<zone>"를 지정하면 특정 Zone에만 VPC Load Balancer를 배치 할 수 있습니다. 또, 이 VPC Load Balancer의 할당처 대상도 이 지정한 Worker Node에만 된다. 다만, Worker Node의 NodePort는 다른 Zone에 존재하는 Pod에도 할당을 할 수 있다.

IBM Cloud docs 보다 인용

The VPC load balancer is deployed to the same subnet in that zone that your worker nodes are connected to.

Only worker nodes in your cluster in this zone are configured to receive traffic from the VPC load balancer.

hello-world-us-south-1.yaml

apiVersion: v1
kind: Service
metadata:
 name: hello-world-us-south-1
 namespace: syasuda
 annotations:
   service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type: private
   service.kubernetes.io/ibm-load-balancer-cloud-provider-zone: "us-south-1"
spec:
 type: LoadBalancer
 selector:
   deploymentconfig: hello-world
 ports:
  - name: http
    protocol: TCP
    port: 8080
    targetPort: 8080

$ oc apply -f hello-world-us-south-1.yaml

$ oc get service hello-world-us-south-1
NAME                     TYPE           CLUSTER-IP       EXTERNAL-IP                            PORT(S)          AGE
hello-world-us-south-1   LoadBalancer   172.21.131.127   yyyyyyyy-us-south.lb.appdomain.cloud   8080:31333/TCP   6m35s

oc describe service hello-world-us-south-1
Name:                     hello-world-us-south-1
Namespace:                syasuda
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type":"private",...
                          service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type: private
                          service.kubernetes.io/ibm-load-balancer-cloud-provider-zone: us-south-1
Selector:                 deploymentconfig=hello-world
Type:                     LoadBalancer
IP:                       172.21.131.127
LoadBalancer Ingress:     yyyyyyyy-us-south.lb.appdomain.cloud
Port:                     http  8080/TCP
TargetPort:               8080/TCP
NodePort:                 http  31333/TCP
Endpoints:                172.17.111.9:8080,172.17.115.137:8080,172.17.123.76:8080 + 2 more...
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason                           Age    From                Message
  ----    ------                           ----   ----                -------
  Normal  EnsuringLoadBalancer             7m     service-controller  Ensuring load balancer
  Normal  EnsuredLoadBalancer              6m49s  service-controller  Ensured load balancer
  Normal  CloudVPCLoadBalancerNormalEvent  51s    ibm-cloud-provider  Event on cloud load balancer hello-world-us-south-1 for service syasuda/hello-world-us-south-1 with UID ea8fd792-5d72-475a-b972-e836654aedc2: The VPC load balancer that routes requests to this Kubernetes LoadBalancer service is currently online/active.

$ oc get pods -o wide|grep -v Completed
NAME                   READY   STATUS      RESTARTS   AGE    IP               NODE           NOMINATED NODE   READINESS GATES
hello-world-1-45m9j    1/1     Running     0          3d2h   172.17.111.9     10.240.128.4   <none>           <none>
hello-world-1-b75zr    1/1     Running     0          3d2h   172.17.115.137   10.240.128.5   <none>           <none>
hello-world-1-gl5sd    1/1     Running     0          3d2h   172.17.74.18     10.240.64.5    <none>           <none>
hello-world-1-j8w4f    1/1     Running     0          3d2h   172.17.67.14     10.240.64.4    <none>           <none>
hello-world-1-rb8nt    1/1     Running     0          3d2h   172.17.123.76    10.240.0.4     <none>           <none>

상기 엔드 포인트 정보의 결과로부터, 다른 Zone에 존재하는 복수의 Pod에도 할당이 행해지고있다.
Private 지정의 어노테이션을 붙이고 있으므로, 확실히 할당할 수 있었던 이 FQDN를 이름 해석하면 Private IP가 되고 있지만, 한편으로 그 연속한 IP 주소로부터 VPC Load Balancer 인스턴스는 동일 Zone에 배치되고 있다 것을 알 수 있다.

$ dig A +noall +answer @1.1.1.1 yyyyyyyy-us-south.lb.appdomain.cloud
yyyyyyyy-us-south.lb.appdomain.cloud. 120 IN A  10.240.0.8
yyyyyyyy-us-south.lb.appdomain.cloud. 120 IN A  10.240.0.9

또한 아래와 같이 총 6대(각 Zone에 2대씩) Worker Node가 있는데도 2대밖에 할당 대상이 되고 있지 않다.