GCP VIP as Internal Load Balancer and BYOIP
Overview
Starting with Avi Vantage 18.2.9 version, VIP as GCP Internal Load Balancer (ILB) is supported. With this the VIP reachability is through ILB, where VIP will be allocated from a GCP subnet and the VIP will be frontend IP of the ILB. The ILB backends will have all the Service Engines on which the virtual service is placed.
GCP Internal Load Balancer for the virtual service is created in the Service Engine project configured in the cloud. You can configure ILB as VIP for the cloud through the Avi CLI or REST APIs.
This guide explains the creation of GCP cloud with inband network configuration. For other network configurations, refer to GCP Cloud Network Configuration guide.
Configuring VIP as Internal Load Balancer
You need to set vip_allocation_strategy field in GCP cloud configuration to ILB. With this all the VIPs created in this cloud will be allocated from GCP subnet and virtual services will be configured as ILB with frontend IP as the VIP.
Creating GCP Credentials
The following are the steps to create GCP credentials:
-
Navigate to Administration > User Credentials and click on the Create button.
-
Specify the user name and the GCP service account key file data. Refer to GCP Full Access Deployment Guide for more details on authentication.
Creating GCP Cloud
The following are the steps to create GCP cloud:
- Login to Avi Shell. The following CLI is used:
[admin:10-138-10-49]: > configure cloud gcp-cloud
[admin:10-138-10-49]: cloud> vtype cloud_gcp
[admin:10-138-10-49]: cloud> gcp_configuration
[admin:10-138-10-49]: cloud:gcp_configuration> cloud_credentials_ref gcp-service-account
[admin:10-138-10-49]: cloud:gcp_configuration> region_name us-central1
[admin:10-138-10-49]: cloud:gcp_configuration> zones us-central1-a
[admin:10-138-10-49]: cloud:gcp_configuration> zones us-central1-b
[admin:10-138-10-49]: cloud:gcp_configuration> se_project_id service-engine-project
[admin:10-138-10-49]: cloud:gcp_configuration> network_config config inband_management
[admin:10-138-10-49]: cloud:gcp_configuration:network_config> inband
[admin:10-138-10-49]: cloud:gcp_configuration:network_config:inband> vpc_subnet_name subnet-1
[admin:10-138-10-49]: cloud:gcp_configuration:network_config:inband> vpc_project_id network-project
[admin:10-138-10-49]: cloud:gcp_configuration:network_config:inband> vpc_network_name network-1
[admin:10-138-10-49]: cloud:gcp_configuration:network_config:inband> save
[admin:10-138-10-49]: cloud:gcp_configuration:network_config> save
[admin:10-138-10-49]: cloud:gcp_configuration> vip_allocation_strategy mode ilb
[admin:10-138-10-49]: cloud:gcp_configuration:vip_allocation_strategy> ilb
[admin:10-138-10-49]: cloud:gcp_configuration:vip_allocation_strategy:ilb> save
[admin:10-138-10-49]: cloud:gcp_configuration:vip_allocation_strategy> save
[admin:10-138-10-49]: cloud:gcp_configuration> save
[admin:10-138-10-49]: cloud> save
+-----------------------------+--------------------------------------------+
| Field | Value |
+------------------------------+--------------------------------------------+
| uuid | cloud-90c34ff6-f3fd-4bbd-a6b6-32e725c521ea |
| name | gcp-cloud |
| vtype | CLOUD_GCP |
| apic_mode | False |
| gcp_configuration | |
| cloud_credentials_ref | gcp-service-account |
| region_name | us-central1 |
| zones[1] | us-central1-a |
| zones[2] | us-central1-b |
| se_project_id | service-engine-project |
| network_config | |
| config | INBAND_MANAGEMENT |
| inband | |
| vpc_subnet_name | subnet-1 |
| vpc_project_id | network-project |
| vpc_network_name | network-1 |
| vip_allocation_strategy | |
| mode | ILB |
| dhcp_enabled | True |
| mtu | 1500 bytes |
| prefer_static_routes | False |
| enable_vip_static_routes | False |
| license_type | LIC_CORES |
| state_based_dns_registration | True |
| ip6_autocfg_enabled | False |
| dns_resolution_on_se | False |
| enable_vip_on_all_interfaces | False |
| tenant_ref | admin |
| license_tier | ENTERPRISE_18 |
| autoscale_polling_interval | 60 seconds |
+------------------------------+--------------------------------------------+
[admin:10-138-10-49]: > Creating Virtual Service
The following are the CLI details to create virtual service:
configure vsvip <name>
vip
subnet_uuid <subnet-name>
auto_allocate_ip
save
save
[admin:controller]: > configure virtualservice ilb-vs-1
[admin:controller]: virtualservice> cloud_ref gcp-cloud
[admin:controller]: virtualservice> pool_ref pool-1
[admin:controller]: virtualservice> services
New object being created
[admin:controller]: virtualservice:services> port 80
[admin:controller]: virtualservice:services> save
[admin:controller]: vsvip_ref <name>
+------------------------------------+-----------------------------------------------------+
| Field | Value |
+------------------------------------+-----------------------------------------------------+
| uuid | virtualservice-63190c5f-3e58-4d84-9931-b860f17c20e6 |
| name | ilb-vs-1 |
| enabled | True |
| services[1] | |
| port | 80 |
| enable_ssl | False |
| port_range_end | 80 |
| application_profile_ref | System-HTTP |
| network_profile_ref | System-TCP-Proxy |
| pool_ref | pool-1 |
| se_group_ref | Default-Group |
| vrf_context_ref | global |
| enable_autogw | True |
| analytics_profile_ref | System-Analytics-Profile |
| weight | 1 |
| delay_fairness | False |
| max_cps_per_client | 0 |
| limit_doser | False |
| type | VS_TYPE_NORMAL |
| cloud_type | CLOUD_NONE |
| use_bridge_ip_as_vip | False |
| flow_dist | LOAD_AWARE |
| ign_pool_net_reach | False |
| ssl_sess_cache_avg_size | 1024 |
| remove_listening_port_on_vs_down | False |
| close_client_conn_on_config_update | False |
| bulk_sync_kvcache | False |
| tenant_ref | admin |
| cloud_ref | gcp-cloud |
| east_west_placement | False |
| scaleout_ecmp | True |
| active_standby_se_tag | ACTIVE_STANDBY_SE_1 |
| flow_label_type | NO_LABEL |
| vip[1] | |
| vip_id | 0 |
| ip_address | 10.138.0.134 |
| enabled | True |
| network_ref | subnet-2 |
| port_uuid | avi-ilbip-32e725c521ea-fa4d0dada487 |
| subnet_uuid | subnet-2 |
| subnet | 10.138.0.128/25 |
| auto_allocate_ip | True |
| auto_allocate_floating_ip | False |
| avi_allocated_vip | True |
| avi_allocated_fip | False |
| ipam_network_subnet | |
| network_ref | subnet-2 |
| subnet | 10.138.0.128/25 |
| subnet_uuid | subnet-2 |
| auto_allocate_ip_type | V4_ONLY |
| vsvip_ref | vsvip-P4fAJB |
| use_vip_as_snat | False |
| traffic_enabled | True |
| allow_invalid_client_cert | False |
+------------------------------------+-----------------------------------------------------+
You can create the VIP 10.138.0.134 as Internal Load Balancer in GCP with Service Engines as backend.
Notes:
-
Health check probes are in the address range of 130.211.0.0/22 and 35.191.0.0/16. You need to add firewall rules to allow these addresses.
-
If all instances are unhealthy in the backend service, then the ILB will load balance the traffic among all instances.
Limitations
The following are the limitations of ILB:
-
ILB supports UDP and TCP. However, in the case of UDP the health check type is not UDP (due to Google limitation) and so the fail-over time is higher.
-
The virtual IP cannot be shared by other virtual services as the forwarding ports cannot be updated. A new forwarding rule with the same IP but different port is not allowed in GCP.
-
A virtual service can only have up to 5 ports as the forwarding rule can have only 5 ports.
-
The health check is done on the same port as the VIP and not on the instance IP.
-
If Avi Service Engine is in N backend service then it will receive N health check probes per health check interval.
-
Ensure that the VIP is not configured in the same subnet as that of Avi Service Engine.
Bring Your Own IP (BYOIP)
With BYOIP you can use public IPs in GCP and use that to load-balance traffic across backed servers.
Configuring BYOIP
The following are the steps to configure BYOIP:
-
Configure floating IP for the virtual service whose VIP is a Internal Load Balancer in GCP. This will add a static route in GCP with destination range as the public IP and next hop as the internal Load Balancer with frontend IP as the VIP.
-
Advertise the public IP used in GCP though GCP cloud routers.
Configuring Cloud Routers
The following are the steps to configure cloud routers:
-
Create the cloud router in the network project and in the region where the Service Engine virtual machines are created. Refer to Creating Cloud Routers to know more.
-
You can add multiple cloud routers for an Avi cloud, and all will be updated with the FIP.
-
Add the cloud router names in the internal load balancer configuration of vip_allocation_strategy field of the cloud configuration.
The following is the CLI to configure cloud routers:
[admin:10-138-10-49]: > configure cloud gcp-cloud
Updating an existing object. Currently, the object is:
[admin:10-138-10-49]: cloud> gcp_configuration
[admin:10-138-10-49]: cloud:gcp_configuration> vip_allocation_strategy
[admin:10-138-10-49]: cloud:gcp_configuration:vip_allocation_strategy> ilb
[admin:10-138-10-49]: cloud:gcp_configuration:vip_allocation_strategy:ilb>
[admin:10-138-10-49]: cloud:gcp_configuration:vip_allocation_strategy:ilb> cloud_router_names cloud-router-1
[admin:10-138-10-49]: cloud:gcp_configuration:vip_allocation_strategy:ilb> save
[admin:10-138-10-49]: cloud:gcp_configuration:vip_allocation_strategy> save
[admin:10-138-10-49]: cloud:gcp_configuration> save
[admin:10-138-10-49]: cloud> save
+------------------------------+--------------------------------------------+
| Field | Value |
+------------------------------+--------------------------------------------+
| uuid | cloud-90c34ff6-f3fd-4bbd-a6b6-32e725c521ea |
| name | gcp-cloud |
| vtype | CLOUD_GCP |
| apic_mode | False |
| gcp_configuration | |
| cloud_credentials_ref | gcp-service-account |
| region_name | us-central1 |
| zones[1] | us-central1-a |
| zones[2] | us-central1-b |
| se_project_id | development-237409 |
| network_config | |
| config | INBAND_MANAGEMENT |
| inband | |
| vpc_subnet_name | subnet-11 |
| vpc_project_id | development-237409 |
| vpc_network_name | dev-vnet-1 |
| vip_allocation_strategy | |
| mode | ILB |
| ilb | |
| cloud_router_names[1] | cloud-router-1 |
| dhcp_enabled | True |
| mtu | 1500 bytes |
| prefer_static_routes | False |
| enable_vip_static_routes | False |
| license_type | LIC_CORES |
| state_based_dns_registration | True |
| ip6_autocfg_enabled | False |
| dns_resolution_on_se | False |
| enable_vip_on_all_interfaces | False |
| tenant_ref | admin |
| license_tier | ENTERPRISE_18 |
| autoscale_polling_interval | 60 seconds |
+------------------------------+--------------------------------------------+
Adding Floating IP to the Virtual Service
The following is the CLI to add floating IP to the virtual service. This will create a static route in GCP and update the cloud routers with the public IP.
[admin:10-138-10-49]: > configure virtualservice ilb-vs-1
Updating an existing object. Currently, the object is:
[admin:10-138-10-49]: virtualservice> vip index 1
[admin:10-138-10-49]: virtualservice:vip> floating_ip 35.78.2.109
[admin:10-138-10-49]: virtualservice:vip> save
[admin:10-138-10-49]: virtualservice> save
+------------------------------------+-----------------------------------------------------+
| Field | Value |
+------------------------------------+-----------------------------------------------------+
| uuid | virtualservice-63190c5f-3e58-4d84-9931-b860f17c20e6 |
| name | ilb-vs-1 |
| enabled | True |
| services[1] | |
| port | 80 |
| enable_ssl | False |
| port_range_end | 80 |
| application_profile_ref | System-HTTP |
| network_profile_ref | System-TCP-Proxy |
| pool_ref | pool-1 |
| se_group_ref | Default-Group |
| vrf_context_ref | global |
| enable_autogw | True |
| analytics_profile_ref | System-Analytics-Profile |
| weight | 1 |
| delay_fairness | False |
| max_cps_per_client | 0 |
| limit_doser | False |
| type | VS_TYPE_NORMAL |
| cloud_type | CLOUD_NONE |
| use_bridge_ip_as_vip | False |
| flow_dist | LOAD_AWARE |
| ign_pool_net_reach | False |
| ssl_sess_cache_avg_size | 1024 |
| remove_listening_port_on_vs_down | False |
| close_client_conn_on_config_update | False |
| bulk_sync_kvcache | False |
| tenant_ref | admin |
| cloud_ref | gcp-cloud |
| east_west_placement | False |
| scaleout_ecmp | True |
| active_standby_se_tag | ACTIVE_STANDBY_SE_1 |
| flow_label_type | NO_LABEL |
| vip[1] | |
| vip_id | 0 |
| ip_address | 10.138.0.134 |
| enabled | True |
| network_ref | subnet-2 |
| port_uuid | avi-ilbip-32e725c521ea-fa4d0dada487 |
| subnet_uuid | subnet-2 |
| subnet | 10.138.0.128/25 |
| auto_allocate_ip | True |
| floating_ip | 35.78.2.109 |
| auto_allocate_floating_ip | False |
| avi_allocated_vip | True |
| avi_allocated_fip | False |
| ipam_network_subnet | |
| network_ref | subnet-2 |
| subnet | 10.138.0.128/25 |
| subnet_uuid | subnet-2 |
| auto_allocate_ip_type | V4_ONLY |
| vsvip_ref | vsvip-P4fAJB |
| use_vip_as_snat | False |
| traffic_enabled | True |
| allow_invalid_client_cert | False |
+------------------------------------+-----------------------------------------------------+
Limitations
-
Multiple clusters cannot share the same cloud router as the update cannot be coordinated between them.
-
Multiple Avi clouds cannot share the same cloud router.
-
If a cloud router is removed from the Avi cloud configuration, then you need to manually delete the Avi Floating IPs (FIP) which are on the cloud router from the cloud router in GCP since Avi Vantage will no longer manage that router.
Troubleshooting Cloud Router
For each cloud router update, there is a corresponding event generated as follows:
-
GCP_CLOUD_ROUTER_UPDATE_SUCCESS — An internal event which will be generated each time the cloud router is successfully updated.
-
GCP_CLOUD_ROUTER_UPDATE_FAIL — An external event which will be generated when an update for the cloud router fails.
Note: The cloud router can have other IPs also which were not created by Avi Vantage. These IPs will remain as is and all the IPs that are configured by Avi Vantage in the cloud router will have a description added by Avi Vantage.
If a FIP is not getting advertised, you can view the custom IP ranges in cloud router details as follows:
-
Navigate to Hybrid Connectivity > Cloud Routers.
-
Click on the cloud router required.
-
Click on Custom IP Ranges tab to view the IP range and description as shown below:
If the FIP is not listed under custom IP ranges, you can view the Events page for the Controller in Avi Vantage as follows:
-
Navigate to Operations > Events.
-
Click on the GCP_CLOUD_ROUTER_UPDATE_FAIL event to view the error message as shown below:
Updating SE Project ID
SE Project ID field in a cloud specifies the project in which SEs are created. Once you set this field, you can amend this field, only if the following conditions are met:
- VIPAllocationStrategy - ILB
- There are no virtual service or SE present in the cloud
- No pool is configured with auto-scaling groups
- VIPAllocationStrategy - Routes
- All the virtual service are created with
VIPAllocationModeset to Routes. - None of the virtual service has public IP allocated.
- All the virtual service are created with
Notes:
The following are the impact of changing the SE Project ID:
- New SE creation will always happen in the new SE project.
- The networks specified in the
network_configoption in the cloud should be shared with the new SE project as well. - The existing SEs in the older project will be managed by the product.
- Avi Vantage will clean up images in the old SE project.
- Avi Vantage will clean up the topic and subscription in the old SE project.