Install and Configure Avi Multi-Cluster Kubernetes Operator

Overview

The Avi Multi-Cluster Kubernetes Operator (AMKO) facilitates application deployments across multiple OpenShift/Kubernetes clusters.

AMKO is aware of the following object types:

• Kubernetes:
  • Ingress
  • Service type load balancer
• OpenShift:
  • Routes
  • Service type load balancer

AMKO is a Kubernetes pod that will run in a separate namespace called avi-system. AMKO also runs in the same namespace.

The installation of AMKO is done via Helm which applies the following permissions via a Kubernetes serviceaccount:

• READ GSLBConfig and GlobalDeploymentPolicy instances
• UPDATE GSLBConfig and GlobalDeploymentPolicy status fields

AMKO also requires permissions to read the ingress and service objects for all the member clusters. For this, a separate kubeconfig file is created with all the required permissions from all the clusters and is passed to AMKO via an OpenShift/Kubernetes secret object.

To know more, refer to the following articles:

• Avi GSLB Overview
• Avi GSLB Site Configuration and Operations
• Avi Multi-Cluster Kubernetes Operator

Requirements

This section explains the minimum version requirements to use AMKO in Kubernetes clusters and OpenShift clusters:

For Kubernetes clusters:

Components	Version
Kubernetes	1.16+
AKO	1.2.1
Avi Controller	20.1.1 or higher

For OpenShift Clusters:

Components	Version
OpenShift	4.4+
AKO	1.2.1
Avi Controller	20.1.1 or higher

Pre-Installation Checklist

Ensure that the following tasks are completed before commencing AMKO installation:

• At least one OpenShift/ Kubernetes cluster

• At least one Avi Controller

• AMKO assumes that it has connectivity to all the member clusters’ OpenShift/Kubernetes API servers. Without this, AMKO will not be able to watch over the Kubernetes and OpenShift resources in the member clusters.

• Choose one of the Kubernetes/OpenShift clusters where AMKO should be deployed. All the configuration for AMKO will be added to this cluster. For example, cluster-amko.

• Create a namespace avi-system in cluster-amko:

kubectl create ns avi-system

• Create a kubeconfig file with the permissions to read the service and the ingress/route objects for all the member clusters.
  

    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data: <base64 encoded ca.crt data>
        server: https://10.10.10.10:6443
      name: cluster1
    - cluster:
        certificate-authority-data: <base64 encoded ca.crt data>
        server: https://10.10.10.11:6443
      name: cluster2
    
    contexts:
    - context:
        cluster: cluster1
        namespace: default
        user: admin1
      name: cluster1-admin
    
    - context:
        cluster: cluster2
        namespace: default
        user: admin2
      name: cluster2-admin
    
    kind: Config
    preferences: {}
    users:
    - name: admin1
      user:
        client-certificate-data: <client.crt>
        client-key-data: <client.key>
    - name: admin2
      user:
        client-certificate-data: <client.crt>
        client-key-data: <client.key>
 

To know more refer to the article Creating a kubeconfig file for multi cluster access.

Name this file gslb-members and generate a secret with the kubeconfig file in cluster-amko as shown below:

kubectl create secret generic gslb-config-secret --from-file gslb-members -n avi-system

The permissions provided in the kubeconfig file for all the clusters must have at least the permissions to [get, list, watch] on:

• Kubernetes ingress and service type load balancers.
• OpenShift routes and service type load balancers.

AMKO also needs permissisons to [get, list, watch, update] on:

• GSLBConfig object

• GlobalDeploymentPolicy object The extra update permission is to update the GSLBConfig and GlobalDeploymentPolicy objects’ status fields to reflect the current state of the object, whether its accepted or rejected.

• A kubeconfig file has to be created with the permissions to read the service and ingress objects for all the member clusters.

Note: The names provided in the cluster-contexts match the OpenShift/Kubernetes cluster names.

Installing AMKO

To install AMKO via Helm,

1. Create the avi-system namespace:

   
    kubectl create ns avi-system
    

2. Add this repository to your helm client:

   
    $ helm repo add amko https://projects.registry.vmware.com/chartrepo/ako
    

3. Configure the parameters using values.yaml.

4. Search the available charts for AMKO:

   
    helm search repo
       
     NAME                 	CHART VERSION	         APP VERSION	       DESCRIPTION
     amko/amko              1.2.1        	          1.2.1   A helm chart for Avi Multicluster Kubernetes Operator
     

5. Install AMKO:

   
       helm install  amko/amko  --generate-name --version 1.2.1 -f values.yaml  --set configs.gslbLeaderController=*controller_ip* --namespace=avi-system
      

6. Verify the installation:

   
   helm list -n avi-system
       
   NAME          	NAMESPACE 	REVISION	UPDATED                             	STATUS  	CHART    	                APP VERSION
   amko-1593523840	avi-system	1       	2020-06-30 19:00:44.134075 +0530 IST	deployed	amko-1.2.1
   

Configuring AMKO

AMKO is configured via two CRDs - GSLBConfig and GlobalDeploymentPolicy. The Helm based installation procedure will automatically create these two in the specified namespaces.

During AMKO installation, set the required parameters via values.yaml. These parameters are translated to the GSLBConfig and the GlobalDeploymentPolicy objects.

Parameter	Description	Default
configs.controllerVersion	Leader site’s controller version	20.1.1
configs.gslbLeaderController	GSLB leader site URL	Not Applicable
gslbLeaderCredentials.username	GSLB leader controller username	admin
gslbLeaderCredentials.password	GSLB leader controller password
configs.memberClusters	OpenShift/Kubernetes cluster contexts as named in the kubeconfig file	Not Applicable
configs.refreshInterval	Interval in seconds after which the in-memory caches are synced with the existing GS objects	1800
configs.domainNames	Domain names supported in the GSLB configuration	foo.com
gdpConfig.appSelector.label{.key,.value}	Selection criteria for applications, label key and value are provided	Not Applicable
gdpConfig.namespaceSelector.label{.key,.value}	Selection criteria for namespaces, label key and value are provided	Not Applicable
gdpConfig.matchClusters	List of clusters (names must match the names in configs.memberClusters) from where the objects will be selected	Not Applicable
gdpConfig.trafficSplit	List of weights for clusters (names must match the names in configs.memberClusters), each weight must range from 1 to 20	Not Applicable

With the AMKO installation based on the parameters configured above, Helm creates a GSLBConfig and GlobalDeploymentPolicy object.
If appSelector and namespaceSelector were not specified via the Helm installation (values.yaml), the respective fields will be empty in the resulting GlobalDeploymentPolicy object.
This means that by default, no objects are selected.
You can edit this GlobalDeploymentPolicy object and add the required criteria for app or cluster selection.

Only the logLevel field is editable in the GSLBConfig object at runtime. Any changes to the GSLBConfig object would require a restart of AMKO. However, any change to the GlobalDeploymentPolicy object is handled during runtime and does not require restart of AMKO.

To know more about the CRDs GSLBConfig and GlobalDeploymentPolicy objects, refer to the below articles:

• GSLB Config
• GlobalDeploymentPolicy

In case of Controller/ cluster connectivity failures after installing AMKO, refer to the status in the GSLBConfig and GlobalDeploymentPolicy status fields.

For example, the gdpConfig.appSelector.label, gdpConfig.namespaceSelector.label, gdpConfig.matchClusters, and gdpConfig.trafficSplit can be changed during runtime without requiring an AMKO restart.

Following is an example of a Global Deployment Policy instance:

apiVersion: "amko.vmware.com/v1alpha1"
kind: "GlobalDeploymentPolicy"
metadata:
  name: "global-gdp"
  namespace: "avi-system"   // a cluster-wide GlobalDeploymentPolicy
spec:
  matchRules:
    appSelector:
      label:
        app: gslb
    namespaceSelector:
      label:
        app: gslb
 
  matchClusters:
    - cluster: cluster1-admin    // cluster names are kubernetes cluster contexts
    - cluster: cluster2-admin
 
  trafficSplit:
    - cluster: cluster1
      weight: 8
    - cluster: cluster2
      weight: 2

Here,

• Namespace: namespace of this object must be avi-system.

• MatchRules: MatchRules define the application selection criteria and namespace selection criteria.

  
    matchRules:
    appSelector:                       // application selection criteria
      label:
        app: gslb                       // kubernetes/openshift label key-value
    namespaceSelector:                 // namespace selection criteria
      label:
        ns: gslb                        // kubernetes/openshift label key-value
    

• A combination of appSelector and namespaceSelector decide which objects will be selected for GSLB service consideration.

  • appSelector: The selection criteria only for application
    label: will be used to match the ingress/service type load balancer labels (key:value pair).

  • namespaceSelector: Selection criteria only for namespaces
    label: will be used to match the namespace labels (key:value pair).

• Currently, GlobalDeploymentPolicy supports the following combinations:

  appSelector	namespaceSelector	Result
  Yes	Yes	GDP selects all the objects satisfying appSelector from namespaces which satisfy the namespaceSelector
  No	Yes	GDP selects all objects from the selected namespaces (satisfying namespaceSelector)
  Yes	No	GDP selects all objects satisfying the appSelector criteria from all the namespaces
  No	No	No objects selected (Default)

• All ingresses and services of type load balancer which satisfy the matchRules will be selected. If a specific application needs to be excluded from the GlobalDeploymentPolicy, the label needs to be removed from the application.

  For example,
  Select objects with label app:gslb from all the namespaces:

   
    matchRules:
        appSelector:
          label:
            app: gslb
    

  Select objects with app:gslb and from namespaces labelled ns:prod:

  
    matchRules:
        appSelector:
          label:
            app: gslb
        namespaceSelector:
          label:
            ns: prod
    

  Select all objects from namespaces labelled: ns:prod

  
    matchRules:
        namespaceSelector:
          label:
            ns: prod
    

• matchClusters: List of clusters on which the above matchRules will be applied on. The member objects of this list are cluster contexts of the individual OpenShift/Kubernetes clusters.

• trafficSplit : The GlobalDeploymentPolicy object provides a cluster wide traffic split configuration between the clusters via trafficSplit. In the GlobalDeploymentPolicy object, if you want to split the traffic in the ratio of 80:20 between cluster1-admin and cluster2-admin. Configure the traffic split as shown below:

  
      
    trafficSplit:
     - cluster: cluster1-admin
       weight: 8
     - cluster: cluster2-admin
       weight: 2
   

  Note: These weights apply to all the global services across the clusters. Configuring different weights per global service is currently not supported.

• GlobalDeploymentPolicy objects are editable. Changes made to the GlobalDeploymentPolicy object reflects on the Avi objects, if applicable.

• Deleting or Updating of the GlobalDeploymentPolicy matchRules trigger all the objects to be re-validated against the remaining set of rules.

• Deleting a cluster member from matchClusters triggers the deletion of objects synced from that cluster in Avi.

Installing AMKO Offline Using Helm

Pre-requisites for Installation

• The Docker image downloaded from the Avi Portal
• A private container registry to upload the AMKO Docker image
• Helm version 3.0 or higher installed

Installing AMKO

To install AMKO offline using Helm,

1. Create the GSLB members with the member clusters configuration and generate a secret with the kubeconfig file in cluster-amko as shown below:

   
    kubectl create secret generic gslb-config-secret --from-file gslb-members -n avi-system 
    

2. Extract the .tar file to get the AMKO installation directory with the helm and docker images.

   
    tar -zxvf amko_cpr_sample.tar.gz
    amko/
    amko/install_docs.txt
    amko/amko-1.2.1-docker.tar.gz
    amko/amko-1.2.1-helm.tgz
   

3. Change the working directory to this path: cd amko/.

4. Load the docker image in one of your machines.

   
    sudo docker load < amko-1.2.1-docker.tar.gz
    
       
   

5. Push the docker image to your private registry. For more information, click here.

6. Extract the AMKO Helm package. This will create a sub-directory amko/amko which contains the Helm charts for AMKO (amko/chart.yaml crds templates values.yaml).

7. Update the helm values.yaml with the required AMKO configuration (Controller IP/credentials, docker registry information etc).

8. Create the namespace avi-system on the OpenShift/Kubernetes cluster.

   
     kubectl create namespace avi-system
     

9. Install AMKO using the updated helm charts.

   
    helm install ./amko --generate-name --namespace=avi-system
    

Uninstall AMKO via Helm

To uninstall AMKO using Helm,

helm uninstall -n avi-system <amko-release-name>

To remove the GSLB services created, remove the GDP object first.
This will remove all the GSLB services selected via the GDP object.

kubectl delete gdp -n avi-system global-gdp

Delete the avi-system namespace:

kubectl delete ns avi-system

AMKO Default Values

A GSLBService created by AMKO has the following default values:

GSLBService Fields	Default Value
Pool load balancing algorithm	Round robin
TTL	Provided by the DNS service (the default value is 30s)
Health Monitor(s)	Refer to the GS Health Monitors section

GSLB Service Health Monitors

AMKO creates health monitors to be used for site recovery. The object specific health monitors are as follows:

• Service Type Load Balancer: For GSLB services serving service type load balancer, the health monitor is of System-GSLB-TCP or System-GSLB-UDP depending on the service type.

• Insecure Ingresses and Insecure Routes: A custom health monitor of type System-GSLB-HTTP is created for each path present in the ingress/route definition and is added to the GSLB service.

• Secure Ingresses and Secure Routes (excluding passthrough routes): A custom health monitor of type System-GSLB-HTTPS is created for each path present in the ingress/route definition and added to the GLSB service.

• Secure Passthrough Routes: A custom health monitor of type System-GSLB-TCP is created and shared across all such GSLB Services serving passthrough routes.

Document Revision History

Date	Change Summary
October 1, 2020	Published the AMKO Installation Guide for Version 1.2.1
July 21, 2020	Published the AMKO Installation Guide (Tech Preview)