Avi GSLB Service Health Monitors

Overview

GSLB service is the representation of a global application deployed at multiple sites. The GSLB service configuration defines the FQDN of the application, the backing virtual services in various sites, and the priority or ratios governing selection of a particular virtual service at any given time. The configuration also defines the health-monitoring methods by which unhealthy components can be identified so that best alternatives may be selected.

Prerequisite Reading

GSLB Service Health Monitoring

There are two categories of GSLB service health monitoring:

  • control-plane
  • data-plane

One or both can be applied on a per-application basis.

GSLB Member Healthchecks

Control-Plane-Based Global Application Health Monitoring

Independent of Avi GSLB, every Avi Controller cluster routinely performs local health checks to collect the health scores and performance metrics of virtual services under its direct control. In addition, if GslbService.controller_health_status_enabled is True, active GSLB sites will also periodically query the Avi Controllers at all the other sites specified within the GSLB site configuration (both active and passive sites).

Note: Control-plane health monitoring does not apply to virtual services configured on a third-party load balancer VIP or standalone servers.

For the below image, only the active Controller in DC1 is shown collecting health information from other Controllers. DC1’s Controller passes a coalesced picture of health status to its local DNS (solid arrow). In reality, the active Controllers in DC2 and AWS update their respective local DNS virtual services with control-plane-based health status.

Control plane healthcheck

Data-Plane-Based Global Application Health Monitoring

In contrast to control-plane-based health monitoring, no site’s Controller cluster is queried. Instead, health checks go directly to participating services, i.e., to the data plane. At an active site, an SE hosting a GSLB DNS virtual service performs periodic health checks against all GSLB pool members (including the virtual services local to it). A dedicated Service Engine can be configured to perform these health checks. Active monitors generate synthetic traffic from the DNS Service Engine to mark a virtual service (pool member) up or down, based on its response. The below diagram shows the DNS in DC1 (the only active site) performing this function against its local virtual service (VS-A1), as well as VS-A2, VS-A3 and VS-A4.

As previously mentioned, the object used for this is called the GslbHealthMonitor. Ping, TCP, UDP, DNS, and HTTP(S) health monitors are supported (as mentioned in the Configuring Health Monitoring section below. Additionally, a custom monitor can be configured as per the requirement.

Note: It is recommended to configure different SE groups for GSLB DNS and load balancing virtual services.

For more information on health monitors, refer to Health Monitors on Avi Vantage.

Data plane healthcheck

Localized Data-Plane-Based Global Application Health Monitoring

In the below image, DC3 is transformed into an active site via deployment of an Avi DNS SE. This nominally adds four additional data-plane health checks, from DC3’s Avi DNS SE to each of the four member virtual services.

Data plane healthcheck

Now let’s focus attention on the DC1 DNS check of VS-A4 as well as the DC3 DNS check of VS-A1. Suppose we wish or need to avoid these checks for one or both of the following reasons:

  • The firewalls protecting DC1 and DC3 have been configured to permit the Avi Controllers to communicate, but they block direct access to VS-A4 (from DC1) and to VS-A1 (from DC3).
  • We want to scale the performance of each Avi DNS SE by minimizing the number of health checks each must perform. Since each member VS is already being data-plane checked by a local DNS, we consider it wasteful for remote DNSes to replicate those checks.

We can achieve the optimization depicted in the below figure by optioning VS-A1 and VS-A4 for localized data-plane health checks. Two data-plane health checks are eliminated. Instead, each DNS SE gets the health information it needs by querying the remote site’s Avi Controller.

localized data-plane health check

This hybrid approach, which combines control- and data-plane health checking, is enabled for a global service on an individual member VS basis. The only restriction is that the member VS runs on an active Avi site.

Enabling Localized Data-Plane-Based Global Application Health Monitoring in the Avi UI

A GSLB site optimizes health checks by identifying other GSLB sites as health-monitoring proxies. In the below figure, the pull-down menu of the Health Monitor Proxy field offers three Avi GSLB sites capable of performing local checks for the GSLB.

enabling-localized-health-checks.png

Interaction with the GSLB DNS

When it is determined a GSLB service pool member (i.e., some participating virtual service) is down, one of four standard responses are returned by the GSLB DNS. In the GslbService object, set the GslbService.down_response parameter to select one of these four:

  • GSLB_SERVICE_DOWN_RESPONSE_NONE – the default option, simply drops the request.
  • GSLB_SERVICE_DOWN_RESPONSE_FALLBACK_IP – respond with a single preset fallback IP address, which typically would point to a server that’s serving a “sorry” page.
  • GSLB_SERVICE_DOWN_RESPONSE_ALL_RECORDS – return all IP addresses of all members of all pools
  • GSLB_SERVICE_DOWN_RESPONSE_EMPTY – return an empty DNS response; can be used to make the client retry in certain cases.

Options and Combinations for GSLB Service Health Monitoring

  • Control-plane health checking only – Active data-plane health monitors are not configured for this mode. All active Controllers are configured to coalesce health status collected locallly with the statistics collected from remote Controllers.

Coalesced stats are then passed from each active Controller (cluster) to its local DNS. This method is only available for members implemented as Avi Vantage virtual services.

  • Data-plane health checking only – Set GslbService.controller_health_status_enabled to false. Each GSLB DNS performs health checks on all GSLB member virtual services (including those hosted on external sites).
  • Both control and data-plane health checking – For a member virtual service to be marked UP, both control and data health should report UP. If the control-plane health check is failing due to a remote Controller being down or inaccessible, but data-plane health checking is still possible, then it alone determines status of the member virtual service.

Optimizing Health Checking

Setting data-plane health monitor scope appropriately

GslbService.health_monitor_scope – An optional parameter that takes on one of two states. By default, it is set to GSLB_SERVICE_HEALTH_MONITOR_ALL_MEMBERS, in which case DNS SEs actively probe pool members at both Avi and external sites. However, the parameter can be set to GSLB_SERVICE_HEALTH_MONITOR_ONLY_NON_AVI_MEMBERS, such that external member status is collected the only way possible, while the health checking is offloaded from DNS SEs to Avi Controllers local to the GSLB pool members.

Limiting the number of active sites

When a large fraction of GSLB sites are configured to be active, the load on Controllers and the networks interconnecting them can be excessive. For example, consider two deployments, each with 10 Avi sites. One has 5 active Controllers, the other just 2. Each regularly scheduled remote-site health check from an active Controller collects health status from 9 remote sites. Compare the throughput consumed when 5 Controllers probe 9 sites each, versus when just 2 Controllers probe 9 sites. That’s 45 remote-site collections per unit of time compared to just 18. The latter is considerably more throughput-efficient, while still delivering reasonable GSLB DNS redundancy for HA.

Health Monitor Sharding

Health Monitor Sharding reduces the volume of health monitor traffic in large-scale deployments where the DNS Virtual Services are scaled out across multiple Service Engines within each site. Refer to Health Monitor Sharding for more details on this feature.

Configuring GSLB Health Monitors using Avi UI

The below specifications apply to the examples shown in this section using Avi UI.

  • The customer has multiple data centers.
  • Two global applications spanning both are going to be deployed. Each will require health monitors to be configured.
  • view.sales.avi.local will run in US-West and rely on US-Central as a disaster recovery site.
  • pay.sales.avi.local will run at US-West as well as US-East to achieve both high availability and optimal user experience.

Configure a data-plane health monitor for GSLB service

This operation can only be performed by an authorized user logged into the GSLB leader Controller.

The Templates > Profiles > Global Health Monitor tab shows the five pre-existing system-standard monitors:

system-standard GSLB health monitors

For a system-standard monitor, the better practice is to define a brand new monitor by clicking on Create. The defaults that apply will be populated into the editor window, and can then be modified as desired. Refer to the New GSLB Health Monitor editor window below.

our-GLSB-TCP-monitor

  • Successful Checks – The number of consecutive successful health checks before a virtual service is marked UP.
  • Failed Checks – The number of consecutive failed health checks before a virtual service is marked DOWN.
  • Send Interval – The number of seconds between health checks to a given virtual service
  • Receive Timeout – A valid response from the server is expected within this number of seconds. It must be less than the send interval. If server status is regularly flapping between UP and DOWN, consider increasing this value.
    Is Federated? – This option helps define the object’s replication scope. If enabled, the object is replicated across the federation. Else, it is visible within the Controller-cluster and its associated service engines.
    is_federated is set to True only when GSLB is turned on. A federated health monitor is used for GSLB purposes while it is not applicable for a regular health monitor. This implies that a GSLB service cannot be associated with a regular health monitor, because GSLB service is a federated object, while the health monitor is not. Conversely, A pool cannot be associated with a federated health monitor because the pool is not a federated object.
  • Health Monitor Port – Regardless of what port the associated virtual services use, this monitor will direct its health checks to port 80. A monitor port is mandatory for HTTP(S), TCP, UDP and external health monitors.

Clicking on Save in the new GSLB Health Monitor editor completes the custom-monitor creation.

Note: The health monitoring is supported for DNS virtual service configured in the active/ standby mode.

Configuring GSLB health monitors using Avi CLI

Login to the leader node (10.10.25.10) and use the configure gslbhealthmonitor <monitor name> command to provide type and port number for the monitor.

: > configure gslbhealthmonitor global-http-hm
: gslbhealthmonitor> type health_monitor_http
: gslbhealthmonitor> monitor_port 80
: gslbhealthmonitor> save

Configuring GSLB External Health Monitor

The external health monitor allows scripts to be written to provide highly customized and granular health checks. The scripts may be Linux shell, Python, or Perl, which can be used to run wget, netcat, curl, snmpget, mysql-client, or dig. External monitors have constrained access to resources, such as CPU and memory to ensure normal functioning of Avi Service Engines. As with any custom scripting, thoroughly validate the long term stability of the implemented script before pointing it at production servers.

The following is the sample script to use an external health monitor.

Here is a sample script/example:

 #curl -v $IP:$PORT >/run/hmuser/$HM_NAME.$IP.$PORT.out
 #echo "$IP" > "/tmp/myfile"
 #echo "$PORT" > "/tmp/myfile"
 if [[ $IP =~ : ]];
 then curl -v [$IP]:$PORT;
 else curl -v $IP:$PORT;
 fi 

$IP and $PORT is the pool member’s IP address and port for the health monitor.

Note: This tmp file information logging is not recommended for production setup and is recommended only for debugging.

An external health monitor can be used to monitor other entities in addition to GSLB pool members.

Sample script:


#!/bin/bash
curl -I https://www.avinetworks.com:443 2>/dev/null | head -n 1 | grep "200"

Note: When an external health monitor is used for the entity other than the GSLB pool members ( as shown in the script above), it is mandatory to have different SE group for DNS and load balancing virtual service.

Changes in Avi Vantage 18.2.6

Starting with release 18.2.6, Avi Vantage supports configuration of a hostname field for GSLB pool member. If configured, this field is used as the host header in GSLB HTTP and HTTPS health monitor. CNAME/FQDN is used in the GSLB monitor if the hostname field is not configured.
For more information, refer to Host Header for GSLB Health Monitor

mTLS in GLSB Health Monitor

mTLS(mutual Transport Layer Security) is an extension of the TLS standard. mTLS or mutual TLS refers to TLS authentication established at both client and server sides.
Prior to NSX Advanced Load Balancer 22.1.3, enabling mTLS on GSLB health monitor was not permitted as some of the required configuration objects like PKI profile and SSL profiles were not federated. Starting with NSX Advanced Load Balancer 22.1.3, mTLS can be enabled on GSLB health monitors.
The GSLB HTTPS health monitor can now be enabled to have SSL attributes associated with it. GSLB HTTPS health monitor associated with SSL attributes allows the client-side authentication while health monitoring the GSLS service pool members (virtual services), thus achieving mTLS authentication.

Enabling mTLS in GSLB Health Monitor

Follow the steps mentioned below to complete the required configuration for enabling mTLS for a GSLB health monitor.

  • SSL Profile
    Navigate to Templates > Security > SSL/TLS profile and create a SSL profile. Enable the Is Federated checkbox as shown below.

    SSL-Profile


For more details about SSL profile, see SSL Profile

  • SSL Key and Certificate
    Navigate to Templates > Security > SSL/TLS Certificate and create a SSL key and certificate. Enable the Is Federated checkbox as shown below.

    SSL-Certificate


For more details about SSL Key and certificate, see ssl-certificates.

  • PKI Profile
    Navigate to Templates > Security > PKI Profile and create a PKI profile. Enable the Is Federated checkbox as shown below.

    PKI-Profile

    For more details about PKI profile, see PKI Profile

  • Health Monitors Navigate to Templates > Profiles > Health Monitors and create a health monitor of the type HTTPS and enable checkbox for Is Federated as shown below.

    health-monitor

  • Navigate to the Server Maintenance Mode tab for the health monitor created in the previous step, and enable SSL Attributes checkbox. Select the SSL profile, SSL certificate and PKI profile created in the previous step.

    health-monitor

  • Associate the health monitor created in the previous step to the GSLB Service as per the requirement.

    health-monitor

Support for Manual Resume/Manual Failback for GSLB Service

Consider a GLSB Setup using GSLB service configured with the priority-based algorithm. Pool 1 is configured with priority 1, and pool 2 is configured with priority 10. With the default setting, Pool 1 servers all the traffic as it has the highest priority. When Pool 1 status goes down, Pool 2 handles all the requests. Once Pool 1 comes up, the request will be accepted by Pool 1, not Pool 2.
Starting with NSX Advanced Load Balancer 22.1.3, a manual resume for GSLB pool member is available for GSLB Services. If the manual resume option is enabled for a GSLB pool, a pool member once goes down is kept in the down state unless it is deactivated and enabled again. User can manually resume traffic to a pool member by deactivating and enabling the pool member.


When the manual resume option is enabled for a group or pool, the backup pool member will keep accepting requests even when the primary pool member comes up. Once manual resume is deactivated, the request is accepted by the primary pool or load balanced as per the algorithm.

The following steps demonstrate how to enable and disable the manual resume option for a GSLB pool. Login to the Avi Controller, and use the configure gslbservice <service name>, use the enable manual_resume for the desired group.



+----------------------------------+--------------------------------------------------+
[admin:controller-1-site-a]: > configure gslbservice GS-test
Updating an existing object. Currently, the object is:
+----------------------------------+--------------------------------------------------+
| Field                            | Value                                            |
+----------------------------------+--------------------------------------------------+
| uuid                             | gslbservice-1a412ece-6f24-458e-af7d-a3e53b0f75ea |
| name                             | GS-test                                          |
| domain_names[1]                  | resume.avi.com                                   |
| groups[1]                        |                                                  |
|   name                           | GS-test                                          |
|   priority                       | 9                                                |
|   algorithm                      | GSLB_ALGORITHM_ROUND_ROBIN                       |
|   members[1]                     |                                                  |
|     cluster_uuid                 | tp_cluster-ef618334-10ba-42bc-8252-37cf51107a5d  |
|     ip                           | 100.66.122.85                                    |
|     ratio                        | 1                                                |
|     enabled                      | True                                             |
|     resolve_fqdn_to_v6           | False                                            |
|     preference_order             | 1                                                |
|   enabled                        | True                                             |
|   manual_resume                  | False                                            |
| down_response                    |                                                  |
|   type                           | GSLB_SERVICE_DOWN_RESPONSE_NONE                  |
| health_monitor_refs[1]           | System-GSLB-HTTP                                 |
| controller_health_status_enabled | True                                             |
| health_monitor_scope             | GSLB_SERVICE_HEALTH_MONITOR_ALL_MEMBERS          |
| enabled                          | True                                             |
| use_edns_client_subnet           | True                                             |
| wildcard_match                   | False                                            |
| site_persistence_enabled         | False                                            |
| pool_algorithm                   | GSLB_SERVICE_ALGORITHM_PRIORITY                  |
| min_members                      | 0                                                |
| resolve_cname                    | False                                            |
| is_federated                     | True                                             |
| tenant_ref                       | admin                                            |
+----------------------------------+--------------------------------------------------+
gr[admin:controller-1-site-a]: gslbservice> groups index 1
[admin:ncontroller-1-site-a]: gslbservice:groups> manual_resume
Overwriting the previously entered value for manual_resume
sa[admin:controller-1-site-a]: gslbservice:groups> save
save
[admin:controller-1-site-a]: gslbservice> save
+----------------------------------+--------------------------------------------------+
| Field                            | Value                                            |
+----------------------------------+--------------------------------------------------+
| uuid                             | gslbservice-1a412ece-6f24-458e-af7d-a3e53b0f75ea |
| name                             | GS-test                                          |
| domain_names[1]                  | resume.avi.com                                   |
| groups[1]                        |                                                  |
|   name                           | GS-test                                          |
|   priority                       | 9                                                |
|   algorithm                      | GSLB_ALGORITHM_ROUND_ROBIN                       |
|   members[1]                     |                                                  |
|     cluster_uuid                 | tp_cluster-ef618334-10ba-42bc-8252-37cf51107a5d  |
|     ip                           | 100.66.122.85                                    |
|     ratio                        | 1                                                |
|     enabled                      | True                                             |
|     resolve_fqdn_to_v6           | False                                            |
|     preference_order             | 1                                                |
|   enabled                        | True                                             |
|   manual_resume                  | True                                             |
| down_response                    |                                                  |
|   type                           | GSLB_SERVICE_DOWN_RESPONSE_NONE                  |
| health_monitor_refs[1]           | System-GSLB-HTTP                                 |
| controller_health_status_enabled | True                                             |
| health_monitor_scope             | GSLB_SERVICE_HEALTH_MONITOR_ALL_MEMBERS          |
| enabled                          | True                                             |
| use_edns_client_subnet           | True                                             |
| wildcard_match                   | False                                            |
| site_persistence_enabled         | False                                            |
| pool_algorithm                   | GSLB_SERVICE_ALGORITHM_PRIORITY                  |
| min_members                      | 0                                                |
| resolve_cname                    | False                                            |
| is_federated                     | True                                             |
| tenant_ref                       | admin                                            |
+----------------------------------+--------------------------------------------------+
[admin:controller-1-site-a]: >

Use the no manual_resume command to deactivate the manual resume option for the desired group for a GSLB service.



[admin:controller-1-site-a]: > configure gslbservice GS-test
Updating an existing object. Currently, the object is:
+----------------------------------+--------------------------------------------------+
| Field                            | Value                                            |
+----------------------------------+--------------------------------------------------+
| uuid                             | gslbservice-1a412ece-6f24-458e-af7d-a3e53b0f75ea |
| name                             | GS-test                                          |
| domain_names[1]                  | resume.avi.com                                   |
| groups[1]                        |                                                  |
|   name                           | GS-test                                          |
|   priority                       | 9                                                |
|   algorithm                      | GSLB_ALGORITHM_ROUND_ROBIN                       |
|   members[1]                     |                                                  |
|     cluster_uuid                 | tp_cluster-ef618334-10ba-42bc-8252-37cf51107a5d  |
|     ip                           | 100.66.122.85                                    |
|     ratio                        | 1                                                |
|     enabled                      | True                                             |
|     resolve_fqdn_to_v6           | False                                            |
|     preference_order             | 1                                                |
|   enabled                        | True                                             |
|   manual_resume                  | True                                             |
| down_response                    |                                                  |
|   type                           | GSLB_SERVICE_DOWN_RESPONSE_NONE                  |
| health_monitor_refs[1]           | System-GSLB-HTTP                                 |
| controller_health_status_enabled | True                                             |
| health_monitor_scope             | GSLB_SERVICE_HEALTH_MONITOR_ALL_MEMBERS          |
| enabled                          | True                                             |
| use_edns_client_subnet           | True                                             |
| wildcard_match                   | False                                            |
| site_persistence_enabled         | False                                            |
| pool_algorithm                   | GSLB_SERVICE_ALGORITHM_PRIORITY                  |
| min_members                      | 0                                                |
| resolve_cname                    | False                                            |
| is_federated                     | True                                             |
| tenant_ref                       | admin                                            |
+----------------------------------+--------------------------------------------------+
[admin:controller-1-site-a]: gslbservice> groups index 1
[admin:controller-1-site-a]: gslbservice:groups> no manual_resume
+----------------------+-------------------------------------------------+
| Field                | Value                                           |
+----------------------+-------------------------------------------------+
| name                 | GS-test                                         |
| priority             | 9                                               |
| algorithm            | GSLB_ALGORITHM_ROUND_ROBIN                      |
| members[1]           |                                                 |
|   cluster_uuid       | tp_cluster-ef618334-10ba-42bc-8252-37cf51107a5d |
|   ip                 | 100.66.122.85                                   |
|   ratio              | 1                                               |
|   enabled            | True                                            |
|   resolve_fqdn_to_v6 | False                                           |
|   preference_order   | 1                                               |
| enabled              | True                                            |
| manual_resume        | False                                           |
+----------------------+-------------------------------------------------+
[admin:controller-1-site-a]: gslbservice:groups> save
save

[admin:controller-1-site-a]: gslbservice> save
+----------------------------------+--------------------------------------------------+
| Field                            | Value                                            |
+----------------------------------+--------------------------------------------------+
| uuid                             | gslbservice-1a412ece-6f24-458e-af7d-a3e53b0f75ea |
| name                             | GS-test                                          |
| domain_names[1]                  | resume.avi.com                                   |
| groups[1]                        |                                                  |
|   name                           | GS-test                                          |
|   priority                       | 9                                                |
|   algorithm                      | GSLB_ALGORITHM_ROUND_ROBIN                       |
|   members[1]                     |                                                  |
|     cluster_uuid                 | tp_cluster-ef618334-10ba-42bc-8252-37cf51107a5d  |
|     ip                           | 100.66.122.85                                    |
|     ratio                        | 1                                                |
|     enabled                      | True                                             |
|     resolve_fqdn_to_v6           | False                                            |
|     preference_order             | 1                                                |
|   enabled                        | True                                             |
|   manual_resume                  | False                                            |
| down_response                    |                                                  |
|   type                           | GSLB_SERVICE_DOWN_RESPONSE_NONE                  |
| health_monitor_refs[1]           | System-GSLB-HTTP                                 |
| controller_health_status_enabled | True                                             |
| health_monitor_scope             | GSLB_SERVICE_HEALTH_MONITOR_ALL_MEMBERS          |
| enabled                          | True                                             |
| use_edns_client_subnet           | True                                             |
| wildcard_match                   | False                                            |
| site_persistence_enabled         | False                                            |
| pool_algorithm                   | GSLB_SERVICE_ALGORITHM_PRIORITY                  |
| min_members                      | 0                                                |
| resolve_cname                    | False                                            |
| is_federated                     | True                                             |
| tenant_ref                       | admin                                            |
+----------------------------------+--------------------------------------------------+
[admin:controller-1-site-a]: >

Notes:

  • The manual resume option is a group-level knob, but it applies to all the pool members of the specified group.
  • Each pool member must be associated with an Avi site or a third-party site.
  • If a pool member goes down on one site , it goes down on all the SEs hosting the GSLB Service on all the sites.
  • Deactivating and enabling of site, will translate to deactivating and enabling of each member belonging to that site.
  • Propagation of status depends upon the control plane syncing interval gslb_send interval.

Manual Resume Option for Multiple Group Setup

The same steps are applicable for the GSLB Service configured with multiple groups.



[admin:controller-1-site-a]: > configure gslbservice GS-test
Updating an existing object. Currently, the object is:
+----------------------------------+--------------------------------------------------+
| Field                            | Value                                            |
+----------------------------------+--------------------------------------------------+
| uuid                             | gslbservice-1a412ece-6f24-458e-af7d-a3e53b0f75ea |
| name                             | GS-test                                          |
| domain_names[1]                  | resume.avi.com                                   |
| groups[1]                        |                                                  |
|   name                           | GS-test                                          |
|   priority                       | 9                                                |
|   algorithm                      | GSLB_ALGORITHM_ROUND_ROBIN                       |
|   members[1]                     |                                                  |
|     cluster_uuid                 | tp_cluster-ef618334-10ba-42bc-8252-37cf51107a5d  |
|     ip                           | 100.66.122.85                                    |
|     ratio                        | 1                                                |
|     enabled                      | True                                             |
|     resolve_fqdn_to_v6           | False                                            |
|     preference_order             | 1                                                |
|   members[2]                     |                                                  |
|     cluster_uuid                 | tp_cluster-ef618334-10ba-42bc-8252-37cf51107a5d  |
|     ip                           | 100.66.122.86                                    |
|     ratio                        | 1                                                |
|     enabled                      | True                                             |
|     resolve_fqdn_to_v6           | False                                            |
|     preference_order             | 1                                                |
|   enabled                        | True                                             |
|   manual_resume                  | True                                             |
| groups[2]                        |                                                  |
|   name                           | GS-test-pool-2                                   |
|   priority                       | 10                                               |
|   algorithm                      | GSLB_ALGORITHM_ROUND_ROBIN                       |
|   members[1]                     |                                                  |
|     cluster_uuid                 | tp_cluster-5e47e567-5542-4c03-bad8-84dbc882be69  |
|     ip                           | 100.66.122.87                                    |
|     ratio                        | 1                                                |
|     enabled                      | True                                             |
|     resolve_fqdn_to_v6           | False                                            |
|     preference_order             | 1                                                |
|   members[2]                     |                                                  |
|     cluster_uuid                 | tp_cluster-2a7b5f95-8a25-4130-ae23-549a9e258f5d  |
|     ip                           | 100.66.122.88                                    |
|     ratio                        | 1                                                |
|     enabled                      | True                                             |
|     resolve_fqdn_to_v6           | False                                            |
|     preference_order             | 1                                                |
|   enabled                        | True                                             |
|   manual_resume                  | False                                            |
| down_response                    |                                                  |
|   type                           | GSLB_SERVICE_DOWN_RESPONSE_NONE                  |
| health_monitor_refs[1]           | System-GSLB-HTTP                                 |
| controller_health_status_enabled | True                                             |
| health_monitor_scope             | GSLB_SERVICE_HEALTH_MONITOR_ALL_MEMBERS          |
| enabled                          | True                                             |
| use_edns_client_subnet           | True                                             |
| wildcard_match                   | False                                            |
| site_persistence_enabled         | False                                            |
| pool_algorithm                   | GSLB_SERVICE_ALGORITHM_PRIORITY                  |
| min_members                      | 0                                                |
| resolve_cname                    | False                                            |
| is_federated                     | True                                             |
| tenant_ref                       | admin                                            |
+----------------------------------+--------------------------------------------------+
[admin:controller-1-site-a]: gslbservice> groups index 2
ma[admin:controller-1-site-a]: gslbservice:groups> manual_resume
Overwriting the previously entered value for manual_resume
[admin:controller-1-site-a]: gslbservice:groups> save
sa[admin:controller-1-site-a]: gslbservice> save
+----------------------------------+--------------------------------------------------+
| Field                            | Value                                            |
+----------------------------------+--------------------------------------------------+
| uuid                             | gslbservice-1a412ece-6f24-458e-af7d-a3e53b0f75ea |
| name                             | GS-test                                          |
| domain_names[1]                  | resume.avi.com                                   |
| groups[1]                        |                                                  |
|   name                           | GS-test                                          |
|   priority                       | 9                                                |
|   algorithm                      | GSLB_ALGORITHM_ROUND_ROBIN                       |
|   members[1]                     |                                                  |
|     cluster_uuid                 | tp_cluster-ef618334-10ba-42bc-8252-37cf51107a5d  |
|     ip                           | 100.66.122.85                                    |
|     ratio                        | 1                                                |
|     enabled                      | True                                             |
|     resolve_fqdn_to_v6           | False                                            |
|     preference_order             | 1                                                |
|   members[2]                     |                                                  |
|     cluster_uuid                 | tp_cluster-ef618334-10ba-42bc-8252-37cf51107a5d  |
|     ip                           | 100.66.122.86                                    |
|     ratio                        | 1                                                |
|     enabled                      | True                                             |
|     resolve_fqdn_to_v6           | False                                            |
|     preference_order             | 1                                                |
|   enabled                        | True                                             |
|   manual_resume                  | True                                             |
| groups[2]                        |                                                  |
|   name                           | GS-test-pool-2                                   |
|   priority                       | 10                                               |
|   algorithm                      | GSLB_ALGORITHM_ROUND_ROBIN                       |
|   members[1]                     |                                                  |
|     cluster_uuid                 | tp_cluster-5e47e567-5542-4c03-bad8-84dbc882be69  |
|     ip                           | 100.66.122.87                                    |
|     ratio                        | 1                                                |
|     enabled                      | True                                             |
|     resolve_fqdn_to_v6           | False                                            |
|     preference_order             | 1                                                |
|   members[2]                     |                                                  |
|     cluster_uuid                 | tp_cluster-2a7b5f95-8a25-4130-ae23-549a9e258f5d  |
|     ip                           | 100.66.122.88                                    |
|     ratio                        | 1                                                |
|     enabled                      | True                                             |
|     resolve_fqdn_to_v6           | False                                            |
|     preference_order             | 1                                                |
|   enabled                        | True                                             |
|   manual_resume                  | True                                             |
| down_response                    |                                                  |
|   type                           | GSLB_SERVICE_DOWN_RESPONSE_NONE                  |
| health_monitor_refs[1]           | System-GSLB-HTTP                                 |
| controller_health_status_enabled | True                                             |
| health_monitor_scope             | GSLB_SERVICE_HEALTH_MONITOR_ALL_MEMBERS          |
| enabled                          | True                                             |
| use_edns_client_subnet           | True                                             |
| wildcard_match                   | False                                            |
| site_persistence_enabled         | False                                            |
| pool_algorithm                   | GSLB_SERVICE_ALGORITHM_PRIORITY                  |
| min_members                      | 0                                                |
| resolve_cname                    | False                                            |
| is_federated                     | True                                             |
| tenant_ref                       | admin                                            |
+----------------------------------+--------------------------------------------------+
[admin:controller-1-site-a]: >