Testing GCP VPC Service controls in Shared VPC network environment

Objective

In my last post regarding Implementing GCP VPC Service controls using Terraform (Terragrunt), we prepared the infrastructure and resources required to test some scenarios related to VPC Service control in a Shared VPC network environment.

In this post, we will make changes to the service perimeters like adding and removing the protected projects or protected Shared VPC Network, and try to provision a Google compute engine VM in a service project after each change. We will observe the behavior after each change and document that.

Summary of what we achieved in the last post using Terraform

Isolated Google Cloud resources into service perimeters

We created two service perimeters which create a security boundary around Google Cloud resources. We configured one perimeter to control communications from outside the perimeter to Google Cloud services (APIs) and between Google Cloud services. The other one doesn’t protect any resources at this moment. A perimeter allows free communication within the perimeter but, by default, blocks communication to Google Cloud services across the perimeter. It does not block access to any third-party API or services on the internet.

Controlled access to Google Cloud resources from the internet

We created one access level that controls access based on various attributes, such as the source IP address, location, or identity. Access from the internet to managed resources within a service perimeter is denied by default. If API requests made from the internet do not meet the criteria defined in the access level, the requests are denied.

The below diagram depicts our environment.

Architecture

We will only concentrate on Host Project, Shared VPC network, and Service project for Google compute engine.

Required configuration

Please refer to my previous post to prepare the required infrastructure if you want to validate the tests

Implementing GCP VPC Service controls using Terraform (Terragrunt)

Testing without Egress Policy

• We have a Host Project, a Shared VPC Network, and a Service Project.
• We have a service perimeter, we will not change the perimeter configuration through these tests.
• Ingress and restricted services configuration look like below:

config without egress policy

• There is no egress policy to start with.
• No changes to the Access level will be made.

Case 1.1

• The host project is in the service Perimeter
• The service project is in the service Perimeter
• Shared VPC Network is in service Perimeter

This is the default setup from the last post, let’s try to create a VM in the Service project.

cd  ~/vpcsc-1/terraform/gcp/terragrunt
make apply-gce-vm

Result: It works! VM is created. Let’s delete the VM and proceed to next case.

cd  ~/vpcsc-1/terraform/gcp/terragrunt
make destroy-gce-vm

Case 1.2

• The host project is outside the service Perimeter, to do that we need to edit perimeter1 in

~/vpcsc-1/terraform/gcp/terragrunt/orchestration/config_env_sampleapp.yaml

removed host project from perimeter

• Removed host project from protected_project_ids
• The service project is in service Perimeter
• Shared VPC Network is in service Perimeter

Let’s update the perimeter.

cd ~/vpcsc-1/terraform/gcp/terragrunt/orchestration/base_infra/vpcsc_p1
terragrunt apply -auto-approve

Now that perimeter is updated as per our needs, let’s try to create a VM again in the service project.

cd  ~/vpcsc-1/terraform/gcp/terragrunt
make apply-gce-vm

Result: VM creation in service project failed as VPC SC didn't allow creation because host project is not in perimeter and there is no egress policy to access resources/APIs in the host project.

I will skip cases 1.3, 1.4, and 1.5 for you to try (refer to table vpcsc_1.csv with results).

To remove Shared VPC Network from the perimeter, you will need to edit perimeter1 in ~/vpcsc-1/terraform/gcp/terragrunt/orchestration/config_env_sampleapp.yaml and update “protect_xvpc: true” to “protect_xvpc: false”.

Case 1.6

• The host project is in service Perimeter
• The service project is outside the service Perimeter
• Shared VPC Network is in service Perimeter

The configuration will look like this:

Service project removed from the perimeter

Let’s update the service perimeter.

cd ~/vpcsc-1/terraform/gcp/terragrunt/orchestration/base_infra/vpcsc_p1
terragrunt apply -auto-approve

Now that perimeter is updated as per our needs, lets try to create VM again in service project.

cd  ~/vpcsc-1/terraform/gcp/terragrunt
make apply-gce-vm

Result: VM creation in service project failed as VPC SC didn’t allow creation with reason RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER because Service project is not in perimeter and there is no egress policy to access resources/APIs in the host project

Results

If there are no egress rules defined for the perimeter, to be able to create resources in the service project, the service project and the host project should be part of the same perimeter or both should be out of the service perimeter

Testing with Egress Policy

• Considering Service perimeter regular_vpcsc_perimeter_1 only.
• Adding an egress policy in service perimeter to allow all API calls to host and service projects

allowed egress projects added

• That will come into play when we remove host project and service project from the service perimeter

config with egress policy

Case 2.2, 2.4, and 2.6 are of main interest to us because other worked without egress policy as well (refer table vpcsc_2.csv)

Case 2.2

• Host project is outside service Perimeter, to do that we need to edit perimeter1 in

~/vpcsc-1/terraform/gcp/terragrunt/orchestration/config_env_sampleapp.yaml

• Removed host project from protected_project_ids
• The service project is in service Perimeter
• Shared VPC Network is in service Perimeter

Let’s update the perimeter.

cd ~/vpcsc-1/terraform/gcp/terragrunt/orchestration/base_infra/vpcsc_p1
terragrunt apply -auto-approve

Now that perimeter is updated as per our needs, lets try to create VM again in service project

cd  ~/vpcsc-1/terraform/gcp/terragrunt
make apply-gce-vm

Result: It worked 😃 because there exists an egress rule to allow API calls to the unprotected project.

Let’s destroy the VM and proceed to case 2.6.

Case 2.6

• Host project is in service Perimeter
• Service project is outside service Perimeter
• Shared VPC Network is in service Perimeter

Configuration will look like this

Let’s update the perimeter.

cd ~/vpcsc-1/terraform/gcp/terragrunt/orchestration/base_infra/vpcsc_p1
terragrunt apply -auto-approve

Now that perimeter is updated as per our needs, lets try to create VM again in service project.

cd  ~/vpcsc-1/terraform/gcp/terragrunt
make apply-gce-vm

Result: It also worked fine 😃 because there exists an egress rule to allow API calls to the unprotected service project.

Let’s destroy the VM and proceed to next Section.

Results

If there is an egress rule defined for the perimeter to allow the API calls to the unprotected project(s), you will be able to create resources in the service project regardless of whether the host project is in perimeter or not. Egress rule must be targeted to “all services” or appropriated services for this to work.

Testing with two service perimeters

• So far we have not considered the cases when one project is in one service perimeter and other project is in separate service perimeter. Let’s discuss those scenarios in this section.
• Perimeter1 and Perimeter2 are similar except for what project and networks are protected by those respectively.
• We will test some scenarios where service project, host project and Shared VPC network are in two different service perimeters.
• Ingress rule allows all access to all APIs for protected projects and Shared VPC networks from “Access Level” and egress rule targets both host and service projects for all services.
• Below is the diagram for one such scenario where host project and shared VPC network are in perimeter 1 and service project is in perimeter2.

architecture with a service project in perimeter 2

Case 3.1

• We will make use of the second service perimeter by moving the service project to perimeter2
• The host project and shared VPC network are in perimeter 1
• Here is how your configuration will look like

Let’s update the perimeters

cd ~/vpcsc-1/terraform/gcp/terragrunt/orchestration/base_infra/vpcsc_p1
terragrunt apply -auto-approve
cd ~/vpcsc-1/terraform/gcp/terragrunt/orchestration/base_infra/vpcsc_p2
terragrunt apply -auto-approve

Now that perimeters are updated as per our test, lets try to create VM again in service project

cd  ~/vpcsc-1/terraform/gcp/terragrunt
make apply-gce-vm

Result: VM got created successfully. Let’s delete VM and proceed to case 3.2

Note: Before you try to transfer a resource from one perimeter to another, make sure the resource is removed from the older perimeter.

Case 3.2

• Host project is outside both perimeters
• Service project is in perimeter 2
• Shared VPC Network is in perimeter 1
• Configuration should look like below

Update the perimeters and try to create VM once more.

cd ~/vpcsc-1/terraform/gcp/terragrunt/orchestration/base_infra/vpcsc_p1
terragrunt apply -auto-approve
cd  ~/vpcsc-1/terraform/gcp/terragrunt
make apply-gce-vm

Result: VM was created successfully in service project. Let’s delete the VM and procced to next test case.

Case 3.3

• Host project is in perimeter 1
• Service project is in perimeter 2
• Shared VPC Network is outside both perimeters
• Configuration should look like below

Update the perimeters and try to create VM again.

cd ~/vpcsc-1/terraform/gcp/terragrunt/orchestration/base_infra/vpcsc_p1
terragrunt apply -auto-approve
cd  ~/vpcsc-1/terraform/gcp/terragrunt
make apply-gce-vm

Result: VM was created successfully in the service project.

Case 3.4

• This case is same as 2.6, so this should work.

Results

VPC accessible services

Did you notice that our sole ingress rule was for access level meant for internet ingress API calls. What about the API clients such as GCP service projects which are not part of the perimeter. Why didn't we create an Ingress rule to allow the API clients with GCP but outside Perimeter? Answer is its because of VPC accessible services configuration set to “RESTRICTED-SERVICES”. To define the services that can be accessed from a network inside your service perimeter, you use the VPC accessible services feature. The VPC accessible services feature limits the set of services that are accessible from network endpoints inside your service perimeter. Since we used shared VPC network, any resource using the VPC network automatically had access to the resources of services in host project, regardless of whether the perimeter protects those services.

For example consider our setup in case 3.1, after VM is created in Service project suppose you want to use that VM to spin up a VM in host project using gcloud command, would you be able to do it? Yes, without setting up any additional Ingress rule with source being a service project. That is because the VM is part of a shared VPC network which has access to the services in the host project even though protected by the Service perimeter. You may block this To limit the VPC network’s access only to certain services, you can enable VPC accessible services for certain services and set those APIs as an allowed services.

Here is a set of tests for VPC accessible services:

That’s all from my side as far as tests are concerned. There can be many cases with different configuration combinations. I hope it gave you some idea about using VPC Service controls in different configuration scenarios.

Key Takeaways

• A resource can be included in exactly one regular service perimeter.
• Using VPC Service controls can become complex if not planned right. GCP has Perimeter and Access Level design recommendations that you may follow.
• Shared VPC network can either be out of service perimeters or inside the perimeter where its parent project resides. Shared VPC network can’t be inside a separate service perimeter when its parent project is in a service perimeter.
• VPC Accessible services configuration plays crucial role specially when we are using shared VPC network.
• Access level feature doesn't work as expected if you don’t have un-scoped default access policy at organization level. If an organization-level access policy doesn’t exist for your organization, scoped policies that you create at the folder or project-level will not operate.
• Private Google Access offers private connectivity to hosts in either a VPC network or an on-premises network that uses private IP addresses to access Google APIs and services.

References

Set up private connectivity to Google APIs and services | VPC Service Controls | Google Cloud

VPC accessible services | VPC Service Controls | Google Cloud

Design access levels | VPC Service Controls | Google Cloud

Troubleshooting | VPC Service Controls | Google Cloud

Please read my other articles as well and share your feedback. If you like the content shared please like, comment, and subscribe for new articles.