Private Service Connect for GKE Services with Gateway API

Managing network infrastructure in large-scale distributed organizations, such as Digital Turbine, is complex and involves numerous projects, applications, and microservices that need to communicate seamlessly. Managing this type of infrastructure presents numerous challenges, particularly when dealing with dozens of Google Kubernetes Engine (GKE) clusters within Virtual Private Cloud (VPC) across multiple projects.

One of the most common tasks is connecting these private clusters, which becomes problematic when facing overlapping IP address ranges that need to communicate with each other. This is where GCP’s Private Service Connect networking solution comes into play.

Private Service Connect

GCP Private Service Connect is a networking feature that allows you to create private and secure connections between different services within Google Cloud. Private Service Connect empowers GCP customers or third-party software-as-a-service (SaaS) companies to publish their services within their dedicated VPC networks. These published services are then accessed privately and securely from the consumer’s VPC network, eliminating the need for intricate tasks like VPC peering, routing table modifications, and NAT rule implementations. Private Service Connect ensures a hassle-free and service-oriented experience.

Digital Turbine Use Case

Digital Turbine is taking full advantage of Google Cloud. GKE is a backbone service for deploying, managing, and scaling containerized applications. You typically use a Load Balancer to expose a service outside of the GKE cluster. With the evolution of Kubernetes, Gateway API is emerging as the standard solution for private and public provisioning, traffic management, and security of L7 Load Balancer. To split the administration of networks and service projects for the security and segregation of duties reason, each service project has its own host project where a Shared VPC is hosted.

There are many ways to configure networking connectivity between GKE clusters in private VPCs, such as VPN, Interconnect, and Peering. However, most ways won’t work if you have overlapping IP address ranges. Let’s illustrate the IP overlapping scenario with the goal of connecting our internal service deployed in GKE Cluster 1 in Service Project 1 with another internal service deployed in GKE Cluster 2 in Service Project 2. Both services are deployed in separate Shared VPCs (Host Project 1 and Host Project 2) with overlapping IP address ranges.

Figure 1. Leveraging Private Service Connect

Publishing the Service Using Private Service Connect

At Digital Turbine, we use HashiCorp Terraform as the primary tool for managing our infrastructure as a code. This approach brings several benefits, such as version control, scalability, automation, and, most importantly, consistency with Terraform state as a single source of truth. All configuration is done via Terraform. The steps below describe how to set up Private Service Connect, configure the Load Balancer, issue an SSL certificate, and deploy the Gateway API.

Create a Subnet for Private Service Connect (PSC). Before you create a private connection, you must allocate an IPv4 address range in the Shared VPC of Project 1 Subnet; the purpose is PRIVATE_SERVICE_CONNECT, which means that the subnet is reserved exclusively for Private Service Connect service.

resource "google_compute_subnetwork" "subnet-psc" {
  name          = "subnet-psc"
  ip_cidr_range = "10.1.0.0/24"
  region        = "us-east1"
  network       = "producer-shared-vpc-01"
  purpose       = "PRIVATE_SERVICE_CONNECT"
  role          = "ACTIVE"
  private_ip_google_access   = true
}

‍

Create elastic IP for the Load Balancer. In Service Project 1, where GKE is deployed, create a static IP for the published LB The IP should be Non-Shared.

resource "google_compute_address" "private_static_ip" {
  description  = "Non-Shared IP for Published LB via PSC"
  name         = "private-static-ip"
  project      = "service-project-01"
  address_type = "INTERNAL"
  purpose      = "GCE_ENDPOINT"
  subnetwork   = "subnet-useast1-01"
}

‍

Issue an SSL Certificate for SSL termination for the LB. We use GCP Managed Certificates with auto-renew option and DNS validation. However, you can use any certificate authority.

resource "google_certificate_manager_certificate" "psc-cert" {
  name        = "star-psc-example-com-cert"
  description = "The SSL cert for psc.example.com"
  scope       = "EDGE_CACHE"
  
  managed {
    domains = [
      google_certificate_manager_dns_authorization.instance.domain,
      ]
    dns_authorizations = [
      google_certificate_manager_dns_authorization.instance.id,
      ]
  }
}

resource "google_certificate_manager_dns_authorization" "psc_auth_zone" {
  name        = "psc-dns-auth"
  description = "The default dns authorization zone for psc.example.com"
  domain      = "psc.example.com"
}

‍

Set up the Internal Load Balancer via Gateway API Helm deployment.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: gateway-rilb-psc1 # name of the Gateway
spec:
  gatewayClassName: gke-l7-rilb # internal Application Load Balancer(s) built on the internal Application Load Balancer
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      allowedRoutes:
        kinds:
        - kind: HTTPRoute
        namespaces:
          from: All
      tls:
        mode: Terminate
        options:
          networking.gke.io/cert-manager-certs: star-psc-example-com-cert # SSL certificate from step 4
  addresses:
  - type: NamedAddress
    value: "private-static-ip" # VIP created in step 3
    
--- 
# Source: templates/httpRoute.yaml 
# HTTPRoute handles routing HTTP traffic to Gateway API
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: httproute-psc1 # HTTPRoute name 
  namespace: service-namespace # resources must exist in the same namespace as the target Service
spec:
  parentRefs:
    - kind: Gateway
      name: gateway-rilb-psc1 # target Gateway to attach the Route 
      namespace: gateway-namespace # namespace where Gateway is deployed
  hostnames:
    - test.psc.example.com # hostname to expose the service
  rules: # we want to expose different port/path of the service in the rules
    - matches: # service path for gRPS requests on 9090 port
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: service-lookup 
          port: 9090
    - matches: # service path for HTTP requests on 8105
        - path:
            value: /lookup
      backendRefs:
        - name: service-lookup-http
          port: 8105
    - matches: # service path for healthcheck on 8105
        - path:
            value: /status
      backendRefs:
        - name: service-lookup-http
          port: 8105
---
# Source: templates/http_health_check.yaml
# HealthCheckPolicy to control the load balancer health check settings
apiVersion: networking.gke.io/v1
kind: HealthCheckPolicy
metadata:
  name: http-healthcheck-psc1 # health check for liveness of service pods in LB
  namespace: service-namespace # resources must exist in the same namespace as the target Service
spec:
  default:
    checkIntervalSec: 2
    timeoutSec: 1
    healthyThreshold: 1
    unhealthyThreshold: 2
    logConfig:
      enabled: true
    config:
      type: HTTP
      httpHealthCheck:
        portSpecification: USE_FIXED_PORT
        port: 8105
        requestPath: /status
  targetRef:
    group: ""
    kind: Service
    name: service-lookup-http

‍

Configure the PSC Service Attachment.

resource "google_compute_service_attachment" "psc" {
  name                  = "psc-useast1-gke-test-01"
  connection_preference = "ACCEPT_MANUAL" # preferred to use manual approval to accept connection from projects listed in project_id_or_num
  target_service        = "gkegw1-6u2i-gateway-rilb-psc1-0s2ncmd0pfko" # URL of a forwarding rule that represents load balancer from step 4
  nat_subnets           = "subnet-psc" # subnet explicitly created for PSC in step 1

  dynamic "consumer_accept_lists" { 
    content {
      connection_limit  = 10 # the number of consumer forwarding rules the consumer project can create
      project_id_or_num = "consumer-project-02" # project that is allowed to connect to this service attachment
    }
  }
}

Configuring the Consumer

Create a Private DNS Zone, which will be used for the Consumer Endpoint in Host Project 2 to which you want to connect.

resource "google_dns_managed_zone" "private-psc-zone" {
  name        = "private-psc-zone"
  dns_name    = "psc.example.com." # the DNS name of this managed zone, we want to expose service via the same name as it's available on the Producer side
  description = "Private DNS zone for PSC Producer"
  visibility = "private" # private zones are visible only to Virtual Private Cloud resources
  private_visibility_config {
    networks {
      network_url = "consumer-shared-vpc-02"
    }
  }  
}

‍

Create Private Service Connect consumer in Host Project 2.

resource "google_compute_address" "private_service_connect_address" {
  name         = "consumer-private-ip"
  address_type = "INTERNAL"
  subnetwork   = "subnet-useast1-02" # the URL of the subnetwork in which to reserve the address.
  project      = "host-project-02"
  description  = "Managed by Terraform - PSC address"
  region       = "us-east1"
}
resource "google_compute_forwarding_rule" "private_service_connect_forwarding_rule" {
  name                  = "psc-useast1-consumer-test-01"
  project               = "host-project-02"
  ip_address            = google_compute_address.private_service_connect_address.id
  target                = "projects/service-project-01/regions/us-east1/serviceAttachments/psc-useast1-gke-test-01" # the URL of the target resource to receive the matched traffic, PSC Producer URL
  network               = "consumer-shared-vpc-02" # network that the load-balanced IP should belong to for this Forwarding Rule, Host Project 2 Network
  subnetwork            = "subnet-useast1-02" # subnetwork that the load balanced IP should belong to for this Forwarding Rule, GKE Subnetwork in Host Project 2
  region                = "us-east1"
}

‍

Expose the Consumer Endpoint via a private hostname to avoid using IPs.

resource "google_dns_record_set" "dns_record" {
  name         = "test"
  managed_zone = google_dns_managed_zone.private-psc-zone.name # zone created in step 1
  type         = "A"
  ttl          = 300
  rrdatas = ["172.10.0.2"] #replace with PSC Consumer IP from step 2 private_service_connect_address
}

‍

Verify that the Connected Endpoint is Accepted in the Private Service Connect UI.
Test the consumer endpoint from GKE Cluster 2 in Service Project 2 and verify that the service returns a correct response.

curl -i https://test.psc.example.com/status 
HTTP/2 200 
content-length: 0 
date: Tue, 02 Jul 2024 14:41:38 GMT 
via: 1.1 google

Key Monitoring Metrics for Private Service Connect

Both Publisher Service Attachment and Consumer Endpoint provide key metrics associated with PSC resources that can offer valuable insights into PSC connections.

Connected consumer forwarding rules: The count of consumer PSC endpoint forwarding rules linked to a PSC attachment.
Used NAT IP addresses: The number of PSC subnet NAT IP addresses utilized by the consumer forwarding rules.
Sent Byte & Sent Packets Dropped count: The number of network bytes and packets sent over the PSC attachment that were dropped.
Received Byte & Received Packets Dropped count: The number of network bytes and packets received over the PSC attachment that were dropped.
Open & Close connections over service attachment: The number of connections initiated and terminated over the PSC attachment during a specified timeframe.

Notes

PSC works in a way that IP in a Consumer project will be different from IP in a Producer project since PSC does NAT and translates Producer subnet IP into a local Consumer subnet IP.
QPS and throughput showed similar results to those of VPNs.
A load Balancer should be created with a Non-Shared VIP, which means it should have only one port, for example, 80 or 443, on this IP, and in case a Shared VPC is used, this IP should be created in Service Project (where a LB and CM are created) and not in a Host Project.
There is a possibility to create a GKE native PSC attachment, it supports only L4 LB with LoadBalance as service type.
The endpoint, on the consumer side, should be created in a Shared VPC (Host Project).
Every project will have its own endpoint IP; therefore, every project has its own DNS name for the service, its own IP, and the LB of the published service should have a forwarding rule by hostname for every DNS from every project.

Benefits of Using Private Service Connect

Enhanced Security: Keeps traffic within Google’s private network, reducing exposure to internet threats.
Reduced Latency: Offers lower latency compared to public internet routes.
Compliance: Helps meet regulatory and compliance requirements by ensuring data does not traverse the public internet.

Consumer Charges

Private Service Connect endpoint (forwarding rule) $0.01 per endpoint-hour.
Private Service Connect does not charge for inter-zone data transfer for traffic to or from Private Service Connect endpoints

Table 1. Consumer data processing prices (as of February 7th, 2025)
Monthly usage	Price (USD)
0-1 PiB	$0.01 per GiB
1-5 PiB	$0.006 per GiB
5+ PiB	$0.004 per GiB

Producer Charges

There are no Private Service Connect hourly or data transfer charges for publishing services through service attachments. Producers are charged for any load balancing charges incurred on the load balancer that is published to consumers.

Summary

In conclusion, we’ve explored GCP Private Service Connect, highlighting its role in simplifying private connectivity and addressing networking challenges. Private Service Connect enhances security, simplifies networking, and improves performance by enabling secure communication between services without exposing them to the public internet. We’ve also explored the steps to publish and consume managed services using this feature. With its ability to streamline network architecture and ensure secure, reliable connections, Private Service Connect is a valuable tool for businesses looking to optimize their cloud infrastructure.