GKE网关API HTTPROUTE在负载平衡器和应用程序之间不适用于HTTP

Question

我正在测试 gateway api 1.21.11-gke.1100）。我正在使用 GatewayClassName：GKE-L7-rilb 用于客户端和网关之间的网关。 HTTPS使用托管区域SSL证书在客户端和Loadbalancer之间完美工作。

我有2个httproutes引用2个Kube服务（ BackendRefs ）。可以通过HTTP和另一个服务通过HTTP访问（如果可能会有所帮助，则可以通过Argo Workfrows项目提供的Argo-Server服务）。

当我使用HTTP创建HTTPROUTE引用该服务时，GCP负载均衡器后端服务将创建并没有任何问题（健康）。

但是，当我创建HTTPROUTE引用Argo-Service时，会创建GCP负载均衡器后端服务，但使用端点协议设置为HTTP而不是HTTPS，但不起作用（不健康）。您应该知道，我确保添加到Argo-Server服务 cloud.google.com/app-protocols：'{“ web”：“ https”}'}'以在加载之间启用https平衡器和Argo-Server应用程序。

如果我使用Ingress资源创建相同的Geatway API配置，并且相同的ARGO服务定义，则将端点协议（GCP负载均衡器后端服务的端点）正确设置为HTTPS，并且非常健康且正常工作。

好像网关API的httproute gke网关控制器没有考虑 cloud.google.com/app-protocols 服务注释，尽管提到它与Gateway API 在这里。

编辑1：添加了yaml文件

• 网关：

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"Gateway","metadata":{"annotations":{},"labels":{"app.kubernetes.io/managed-by":"gcp-cloud-build-deploy"},"name":"regional-internal-https","namespace":"exposition"},"spec":{"addresses":[{"type":"NamedAddress","value":"dev-gateway-internal-lb-static-ip"}],"gatewayClassName":"gke-l7-rilb","listeners":[{"allowedRoutes":{"kinds":[{"kind":"HTTPRoute"}],"namespaces":{"from":"Selector","selector":{"matchLabels":{"exposed":"true"}}}},"name":"https","port":443,"protocol":"HTTPS","tls":{"mode":"Terminate","options":{"networking.gke.io/pre-shared-certs":"plat-dev-europe-west1"}}}]}}
    networking.gke.io/addresses: ""
    networking.gke.io/backend-services: gkegw1-bkib-argo-argo-server-2746-8ktcvo8d0ktp,
      gkegw1-bkib-demo-application-demo-service-80-y5bgcnm71kjv, gkegw1-bkib-exposition-gw-serve404-80-pciznuyt569p
    networking.gke.io/firewalls: ""
    networking.gke.io/forwarding-rules: gkegw1-bkib-exposition-regional-internal-https-tqsh4njw7io8
    networking.gke.io/health-checks: gkegw1-bkib-argo-argo-server-2746-8ktcvo8d0ktp,
      gkegw1-bkib-demo-application-demo-service-80-y5bgcnm71kjv, gkegw1-bkib-exposition-gw-serve404-80-pciznuyt569p
    networking.gke.io/last-reconcile-time: "2022-06-16T15:57:45Z"
    networking.gke.io/ssl-certificates: ""
    networking.gke.io/target-proxies: gkegw1-bkib-exposition-regional-internal-https-tqsh4njw7io8
    networking.gke.io/url-maps: gkegw1-bkib-exposition-regional-internal-https-tqsh4njw7io8
  creationTimestamp: "2022-06-15T08:28:20Z"
  finalizers:
  - gateway.finalizer.networking.gke.io
  generation: 1
  labels:
    app.kubernetes.io/managed-by: gcp-cloud-build-deploy
  managedFields:
  - apiVersion: gateway.networking.k8s.io/v1alpha2
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/managed-by: {}
      f:spec:
        .: {}
        f:addresses: {}
        f:gatewayClassName: {}
        f:listeners:
          .: {}
          k:{"name":"https"}:
            .: {}
            f:allowedRoutes:
              .: {}
              f:kinds: {}
              f:namespaces:
                .: {}
                f:from: {}
                f:selector:
                  .: {}
                  f:matchLabels:
                    .: {}
                    f:exposed: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:tls:
              .: {}
              f:mode: {}
              f:options:
                .: {}
                f:networking.gke.io/pre-shared-certs: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2022-06-15T08:28:20Z"
  - apiVersion: gateway.networking.k8s.io/v1alpha2
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:networking.gke.io/addresses: {}
          f:networking.gke.io/backend-services: {}
          f:networking.gke.io/firewalls: {}
          f:networking.gke.io/forwarding-rules: {}
          f:networking.gke.io/health-checks: {}
          f:networking.gke.io/last-reconcile-time: {}
          f:networking.gke.io/ssl-certificates: {}
          f:networking.gke.io/target-proxies: {}
          f:networking.gke.io/url-maps: {}
        f:finalizers:
          .: {}
          v:"gateway.finalizer.networking.gke.io": {}
      f:status:
        f:addresses: {}
    manager: GoogleGKEGatewayController
    operation: Update
    time: "2022-06-15T08:30:16Z"
  name: regional-internal-https
  namespace: exposition
  resourceVersion: "42337844"
  uid: 59333aea-1a79-4e9b-afbc-595ae9ccdfd7
spec:
  addresses:
  - type: NamedAddress
    value: dev-gateway-internal-lb-static-ip
  gatewayClassName: gke-l7-rilb
  listeners:
  - allowedRoutes:
      kinds:
      - group: gateway.networking.k8s.io
        kind: HTTPRoute
      namespaces:
        from: Selector
        selector:
          matchLabels:
            exposed: "true"
    name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      options:
        networking.gke.io/pre-shared-certs: plat-dev-europe-west1
status:
  addresses:
  - type: IPAddress
    value: 10.163.112.28
  conditions:
  - lastTransitionTime: "1970-01-01T00:00:00Z"
    message: Waiting for controller
    reason: NotReconciled
    status: Unknown
    type: Scheduled

• httproute：

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"HTTPRoute","metadata":{"annotations":{},"labels":{"app.kubernetes.io/managed-by":"gcp-cloud-build-deploy"},"name":"argo-server","namespace":"argo"},"spec":{"hostnames":["argo-server.plat.dev.df.gcp.corp.modified.com"],"parentRefs":[{"kind":"Gateway","name":"regional-internal-https","namespace":"exposition"}],"rules":[{"backendRefs":[{"name":"argo-server","port":2746}]}]}}
  creationTimestamp: "2022-06-15T12:27:04Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: gcp-cloud-build-deploy
  managedFields:
    - apiVersion: gateway.networking.k8s.io/v1alpha2
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:labels:
            .: {}
            f:app.kubernetes.io/managed-by: {}
        f:spec:
          .: {}
          f:hostnames: {}
          f:parentRefs: {}
          f:rules: {}
      manager: kubectl-client-side-apply
      operation: Update
      time: "2022-06-15T12:27:04Z"
    - apiVersion: gateway.networking.k8s.io/v1alpha2
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:parents: {}
      manager: GoogleGKEGatewayController
      operation: Update
      time: "2022-06-15T12:29:02Z"
  name: argo-server
  namespace: argo
  resourceVersion: "42362026"
  uid: 981ce997-c574-4878-bec1-b03c7707838c
spec:
  hostnames:
    - argo-server.plat.dev.df.gcp.corp.modified.com
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: regional-internal-https
      namespace: exposition
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: argo-server
          port: 2746
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
status:
  parents:
    - conditions:
        - lastTransitionTime: "2022-06-16T17:00:11Z"
          message: ""
          reason: RouteAccepted
          status: "True"
          type: Accepted
        - lastTransitionTime: "2022-06-16T17:00:11Z"
          message: ""
          reason: ReconciliationSucceeded
          status: "True"
          type: Reconciled
      controllerName: networking.gke.io/gateway
      parentRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: regional-internal-https
        namespace: exposition

• 服务：

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/app-protocols: '{"web":"HTTPS"}'
    cloud.google.com/backend-config: '{"default": "argo-server-backendconfig"}'
    cloud.google.com/neg: '{"exposed_ports":{"2746":{}}}'
    cloud.google.com/neg-status: '{"network_endpoint_groups":{"2746":"k8s1-f83345f9-argo-argo-server-2746-4d39c835"},"zones":["europe-west1-c"]}'
    cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/app-protocols":"{\"web\":\"HTTPS\"}","cloud.google.com/backend-config":"{\"default\": \"argo-server-backendconfig\"}","cloud.google.com/neg":"{\"ingress\": true}","cluster-autoscaler.kubernetes.io/safe-to-evict":"true"},"labels":{"app.kubernetes.io/managed-by":"gcp-cloud-build-deploy"},"name":"argo-server","namespace":"argo"},"spec":{"ports":[{"name":"web","port":2746,"targetPort":2746}],"selector":{"app":"argo-server"}}}
  creationTimestamp: "2022-06-15T11:44:07Z"
  labels:
    app.kubernetes.io/managed-by: gcp-cloud-build-deploy
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:cloud.google.com/app-protocols: {}
          f:cloud.google.com/backend-config: {}
          f:cluster-autoscaler.kubernetes.io/safe-to-evict: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/managed-by: {}
      f:spec:
        f:ports:
          .: {}
          k:{"port":2746,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:selector:
          .: {}
          f:app: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2022-06-15T12:27:23Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:cloud.google.com/neg: {}
    manager: GoogleGKEGatewayController
    operation: Update
    time: "2022-06-15T12:28:06Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:cloud.google.com/neg-status: {}
    manager: glbc
    operation: Update
    time: "2022-06-15T12:28:06Z"
  name: argo-server
  namespace: argo
  resourceVersion: "41692832"
  uid: 25024d53-1d31-4165-8033-1843ec5d72ec
spec:
  clusterIP: 10.163.247.121
  clusterIPs:
  - 10.163.247.121
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: web
    port: 2746
    protocol: TCP
    targetPort: 2746
  selector:
    app: argo-server
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Answer 1

我找到了一种解决方案，我认为这是解决方法。

1. 使用 networking.gke.io/app-protocols：'{“ web”：“ https”}'注释而不是 cloud> cloud.google.google.com/app-protocols：'{{ “ web”：“ https”}'。此注释是在服务级别使用，其中 Web 是端口的名称。这将启用LOAD BALANCER和应用程序（为指定的HTTPROUTE创建的后端服务的端点协议）之间的HTTPS 。 这与 GatewayClassName完美工作：GKE-L7-RILB 区域内部负载平衡器。

2. 使用 cloud.google.com/创建自定义健康检查v1 backendconfig 在其中将类型设置为https和端口为2746。更多详细信息在此处 https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-featuress-featuress#direct_health 从应用程序就绪探针中进行健康检查，但显然在GKE网关控制器中尚未实现此功能。

3. 确保您拥有允许防火墙规则Google Cloud Health检查2746端口的入口流量，加入Incresses，GCE Ingress Controller会自动创建所需的防火墙规则，但显然该功能尚未在GKE网关控制器中实现。

最后，我说这是一个解决方法，因为我想象并希望将来的GKE网关控制器的未来版本能够修复我上面提到的3个问题或点。

Answer 2

对我来说同样的问题：如何使用kuberenetes资源 https https 通信/k8s-pod httproute （使用GatewayClass： gke -l7-gxlb ）在GKE群集中：

解决方案：有必要在提到的Kubernetes 服务中设置字段“ appProtocol：https ”

apiVersion: v1
kind: Service
metadata:
name: your-service
labels:
    app.kubernetes.io/name: your-service
spec:
type: ClusterIP
ports:
   - name: port-https
     port: 443
     targetPort: 8080
     protocol: TCP
     appProtocol: HTTPS
 selector:
    app.kubernetes.io/name: your-app

。这是，您可以在GCP控制台/负载平衡器详细信息中看到“端点协议”为“ https”：