NSX-T Deployment Decisions

Overview

This article describes the installation design of Avi Vantage on NSX-T managed vSphere environments (vCenter + ESXi). Starting with Avi Vantage version 18.2.x, Avi Vantage supports over-the-top, manual deployment in NSX-T environment.

Deployment Modes

The following are the recommended deployment modes for Avi Vantage on top of NSX-T managed infrastructure:

One Arm Mode with Overlay VIP Segment - Single Tier 1

The diagrammatic representation of a typical Avi Vantage deployment on a simple NSX-T environment with all server segments connected to a single Tier 1 router is as follows:

You need to create the VIP/data segment manually. The network adapter 1 of the Service Engine VM is reserved for management connectivity. You can connect only one of the remaining nine data interfaces (network adapter 2-10) of the Service Engine VM to the VIP/data segment. Rest of the interfaces must be left disconnected. The Service Engines are deployed in one arm mode; i.e. same interface is used for client and backend server traffic. The SE routes to backend servers through the Tier 1 router.

You can allocate the VIPs from the same subnet as that of the VIP/data interface of the Service Engine. You should reserve a range of static IP addresses, to be used as VIPs, in the subnet assigned to VIP/data segment. Having a dedicated segment for VIPs makes it easier to manage the IP ranges independent of other subnets.

Optionally, you can place the data interface of the Service Engine on one of the server segments, if you do not need separate VIP segment. The subnet assigned to the server segment must have enough free static IP address to be used as VIPs.

In both cases the Service Engine to pool server traffic is handled by the Tier 1 gatewat DR (distributed router) which is present on all ESX transport nodes. Hence, there is no significant performance difference between using dedicated or shared segment for VIP/data.

One Arm Mode with Overlay VIP Segment, Multiple Tier 1

In NSX-T environments, where web servers of different applications are connected to their individual Tier 1 routers, you need to create a VIP/data segment on each Tier 1.

The network adapter 1 of the Service Engine VM is reserved for management connectivity. One data interface (network adapter 2) is connected to VIP/data segment-1 and one data interface (network adapter 3) is connected to VIP/Data segment-2. Rest of the interfaces are kept disconnected.

You can allocate the VIPs from the same subnet as that of the VIP/data interface of the Service Engine. You should reserve a range of static IP addresses, to be used as VIPs, in the subnet assigned to each VIP/data segment. Having a dedicated segment for VIPs makes it easier to manage the IP ranges independent of other server subnets.

You need to configure separate VRF on Avi Vantage for each Tier 1, and add the data interfaces to the VRF corresponding to the Tier 1 segment it is connected to. For instance, in the above diagram you should configure VRF-A and VRF-B for Tier 1 A and Tier 1 B respectively. Also, you should add SE interface connected to VIP/data segment-1 to VRF-A and add the interface connected to VIP/data segment-2 to VRF-B.

While creating the virtual services for a pool, you should choose corresponding VRF. For instance, select VRF-A while creating a virtual service for Pool1 and VRF-B while creating virtual service for Pool2. This way the VIP of the virtual service managing Pool1 will be on VIP/data segment-1 and VIP of the virtual service managing Pool2 will be on VIP/data segment-2.

This is required since you can route the SE to pool server traffic through Tier 1 DR and does not have to hairpin to the Tier 0, and also because you can configure logical segments on different Tier 1 to have same subnet. Hence each Tier 1 traffic must be contained in its own VRF.

Optionally, if separate VIP segment is not required, then you can place the data interfaces of the Service Engine on one of the server segments. The subnet assigned to the server segment must have enough free static IP address to be used as VIPs.

VLAN VIP Segment

Avi Vantage SEs can also be deployed in cases where the NSX-T logical segment is of VLAN type.

Create the VIP/data segment manually. The network adapter 1 of the Service Engine VM is reserved for management connectivity. You can connect only one of the remaining nine data interfaces (network adapter 2-10) of the Service Engine VM to the VIP/data segment. Rest of the interfaces must be left disconnected. The Service Engines can be deployed in either one-arm mode (same interface is used for client and backend server traffic) or dual arm mode (one interface on the SE handles the client traffic and another interface will have connectivity to the backend servers). The SE routes to backend servers through the external physical router.

For VLAN logical segment deployments, you can either allocate the VIPs from the same subnet as that of the VIP/data interface of the Service Engine or allocate the VIP from a completely different subnet (not used anywhere else in the network). In the first case, you need to reserve a range of static IP addresses in the subnet assigned to each VIP/data segment, to be used as VIPs. In the second case, you can configure SEs to peer with the physical router by configuring BGP on Avi Vantage (provided that the router supports BGP) and the SEs will advertise VIP routes with the SE data interface IP as next hop. The router does ECMP across the SEs in case the VIP is scaled out.

Avi Controller Deployment and Management Connectivity

It is recommended to deploy Avi Controller cluster VMs adjacent to NSX-T managers, connected to the management port group since the latency between them is minimum. You should also create a separate Tier 1 and connected management segment for Avi Service Engine management connectivity.

The network interface 1 of the SE VM is connected to the management segment. You should configure the management Tier 1 to redistribute the connected subnet routes to the Tier 0. Tier 0 must advertise the VIP to external peer using BGP.

NSX Security Configuration

Exclusion List

Avi SE redirects traffic from the primary SE to secondary SEs when using L2 scale out mode. This leads to asymmetric traffic which can get blocked by the Distributed Firewall because of its stateful nature. Hence to ensure that the traffic is not dropped when a virtual service scales out, you should add the SE interfaces connected to the VIP/data segment to exclusion list or disabling TCP strict on the respective VIP Data segments.

This can be done by creating an NSG on NSX-T and adding the VIP/data segment as member. You can then add this NSG to the exclusion list. This way if a new SE is deployed its VIP/data interface will dynamically get added to Exclusion list.

In case the SEs are connected to server segment, adding the segment to Exclude list is not an option as that will put all servers in the list too. You need to add individual SE VMs as members to the NSG.

Distributed Firewall

Avi Controller and the SEs require certain protocols/ports to be allowed for management traffic as listed here. If the distributed firewall is enabled with default rule as block/reject all, create the following allow rules on DFW:

Controller UI Access

Source — Any (can be changed to restrict the UI access)

Destination — Avi Controller management IPs and Cluster IP

Service — TCP(80,443)

Action — Allow

Note: This rule is required only if Avi Controller is connected to NSX-T managed segment.
Controller Cluster Communication

Source — Avi Controller management IPs

Destination — Avi Controller management IPs

Service — TCP(22, 8443)

Action — Allow

Note: This rule is required only if Avi Controller is connected to NSX-T managed segment.
SE to Controller Secure Channel

Source — Avi SE management IPs

Destination — Avi Controller management IPs

Service — TCP(22, 8443), UDP(123)

Action — Allow

Note: SE initiates TCP connection for the secure channel to the Controller IP.
SE to Backend

Source — Avi SE data IPs

Destination — Backend server IPs

Service — Any (can be restricted to service port, for instance, TCP 80)

Action — Allow

Note: Client to VIP traffic does not require a DFW rule as the VIP interface is in Exclusion list. The front-end security can be enforced for each VIP using network security policies on the virtual service.

North-South VIP

If certain VIP is required to have north-south access to allow external clients to reach the application, additional configuration is required on NSX-T manager.

You should configure Tier 1 to redistribute the VIP IP (/32) to Tier 0.
Tier 0 must advertise the VIP to external peer using BGP.

Note: If all VIPs are required to be north-south or for simplicity of configuration, you can configure Tier 1 to redistribute the entire VIP range to Tier 0. You can configure Tier 0 to advertise all learned routes in that range to external peer. This way whenever a new VIP is created, it will be automatically advertised to the external peer.