AKO on TKGs with NSX-T

Overview

The Avi Kubernetes Operator (AKO) is an operator which works as an ingress Controller and performs Avi-specific functions in the TKGs environment with the Avi Controller. It remains in sync with the necessary TKGs objects and calls Avi Controller APIs to configure the virtual services.

The AKO deployment consists of the following components:

  • The Avi Controller
  • The Service Engines (SE)
  • The Avi Kubernetes Operator (AKO)

An overview of the AKO deployment is shown below:

ako deployment overview

Note: Security restrictions prevent AKO from communicating with an NSX Advanced Load Balancer Controller installed in the management domain where VCenter or NSX-Manager are located. The NSX Advanced Load Balancer controller can be deployed anywhere, assuming AKO can route to it.

Deployment Guide

AKO can be installed on each workload cluster, using HELM to sync only ingress. Load Balancer sync must be disabled on AKO because NSX-T LB cannot be turned off, and AKO and NCP will overwrite the LB VIP if both are managing the LB service.

Create Two Tier-1s for Avi

In NSX-T Manager, create a new dedicated Tier-1 Gateway for use with Avi Management. Also, create an additional Tier-1 Gateway for use as the Avi VIP Network. This will allow northbound connections to T0 gateways and southbound to the individual segments.

T1-Avi-Mgmt

The architecture of the T1-Avi-Mgmt is shown above.

create t1

Ensure that the proper route advertisements are configured so that Avi can redistribute the VIP routes to the T0 gateway.

For detailed instructions on how to create this T1, see Avi Integration with NSX-T.

Create a Segment under each Avi Tier-1

Avi requires a segment to be created under each of the Tier 1. One segment for Management and the other for Vip/Data.

create a segment

For additional details, see Avi Integration with NSX-T.

Configure NSX-T Cloud on Avi

The point of integration in Avi, with any infrastructure, is called a cloud. For the NSX-T environment, an NSX-T cloud has to be configured. To know how to configure an NSX-T cloud, see Avi Integration with NSX-T.

Set Default Gateway in Avi

Set Default Gateway for the Avi T1 vrf in Avi. The DG needs to be set to allow outbound traffic from the Avi VIP/Data segment and the Management segment. See the below screenshot for an example. First, the Data/Vip DG is set in Avi, as shown below:

set default gateway

Next, the Management DG needs to be set, if DHCP is not available. This is done using Avi CLI.

Example commands:


shell 

switchto cloud nsxt 

terminal mode linux_command_line 

configure vrfcontext management --static_routes.1.prefix 0.0.0.0/0 --static_routes.1.next_hop 172.16.30.1 --static_routes.1.route_id 1

Configure Avi IPAM Profile

Avi allocates IP addresses from a pool of IP addresses within the subnet configured as shown below. After creating this profile, Modify the NSX-T cloud and add this profile.

configure ipam profile

For additional details, see IPAM Provider (Avi Vantage).

Configure Avi DNS Profile

AKO needs to be authoritative in the domain that is specified below. After creating the profile, Modify the NSX-T cloud and add the profile.

configure dns profile

For additional details, see DNS Provider (Avi Vantage).

Configure NSX-T DFW Rules

The default rule in WCP will deny all ingress traffic. Configure a DFW rule to allow SEs to the backend nodes/pods. Configure additional DFW rules to allow clients to VIPs and E-W SE-SE Traffic.

configure nsx-t dfw rules

nsx-t dfw rules

Deploy AKO via Helm

Configure the appropriate fields in the values.yaml file to allow AKO to integrate with Avi + NSX-T.

  • layer7Only – Set to True
  • L7Settings.serviceType – Under L7Settings, set to NodePort
  • nsxtT1LR – Full path to the Avi Tier 1. Example: “/infra/tier-1s/Avi-T1”
  • vipNetworkList – Set to the Avi VIP Network
  • ControllerSettings – Set these for Avi and the NSXT Cloud
  • avicredentials – Optional, can be passed via CLI

The full path for Avi Tier1 in NSX-T can be found via an Avi CLI command:

deploy ako via helm

Installing AKO


helm install  ako/ako  --generate-name --version 1.7.1 -f /ako/values.yaml  --set avicredentials.username=admin --set avicredentials.password='password123' --namespace=avi-system

The above command will create a pod named: ako-0 in the avi-system namespace. Nothing will be created in Avi until an ingress is created.

For detailed instructions on how to deploy AKO via helm, see Install Avi Kubernetes Operator.