Controller Interface and Route Management
Overview
The Avi Controller has a single interface, which is used for various control plane related tasks such as:
-
Operator access to the Controller via CLI, UI, API.
-
Communication between the Controller and the Service Engines.
-
Communication between the Controller and third party entities for automation, observability, etc.
-
Communication between the Controller and third party Hardware Security Modules (HSMs).
Starting with Avi Vantage version 21.1.3, an additional interface is available on the Controller, to allow the ability to isolate the communication for some of the above entities.
In addition, any static routes to be added to the Controller interfaces should now leverage the cluster configuration instead of /etc/network/interfaces subsystem.
These configurations will be persisted across the Controller reboot and upgrade.
Note:
-
This feature is supported only on the Controllers deployed in vCenter.
-
This feature enables the use of the additional interface only for HSMs.
Classification
The following are the classifications to isolate the communication. The labels available are as follows:
-
MGMT — This signifies general management communication for the Controller access, as well as the Controller initiating communication, for instance, logging, third party API calls, and so on.
-
SE_SECURE_CHANNEL — This label is used to classify secure communication between the Service Engine and the Controller.
-
HSM — This is used to classify communication between the Controller and an HSM device.
With this classification, the traffic can be moved from the default, main interface to the additional interface, if configured.
Notes:
-
MGMT can only be performed by the primary (eth0) interface.
-
Starting with version 22.1.3, the
SE_SECURE_CHANNELlabel can be moved to the secondary interface(if there are no SEs connected to the controller). SE Creation should be done after moving this label to the secondary interface. Prior to 22.1.3,SE_SECURE_CHANNELcould only be with the primary interface (eth0). -
HSM can be moved to the additional interface.
Operating Model
By default (prior to 21.1.3), the Controller is provisioned with one interface when being deployed in vCenter (during installation).
When an additional interface is required, you can follow these steps:
-
Shut down the Controller virtual machine and add the interface via vCenter UI.
-
On powering on the Controller virtual machine, Avi Vantage will recognize the additional interface, and additional configuration via the Avi CLI can be performed.
Note: Hotplug of interfaces (addition to the virtual machine without powering off the virtual machine) is not supported.
For the interface to be recognized within the Avi Controller software and further classification via labels to be performed, Avi Vantage’s ‘cluster’ configuration model should be used.
Configuration Steps
Configuration for a Single Node Controller
The following are the configuration steps:
-
Shut down the Controller and add the new interface via the vCenter.
-
Power on the Controller. The new interface will be visible as
eth1, while the primary interface will always be visible aseth0in the Cluster configuration:
[admin:controller]: > show cluster
+-----------------+----------------------------------------------+
| Field | Value |
+-----------------+----------------------------------------------+
| uuid | cluster-83e1ebf5-2c63-4690-9aaf-b66e7a7b5f08 |
| name | cluster-0-1 |
| nodes[1] | |
| name | 10.102.64.201 |
| ip | 10.102.64.201 |
| vm_uuid | 00505681cb45 |
| vm_mor | vm-16431 |
| vm_hostname | node1.controller.local |
| interfaces[1] | |
| if_name | eth0 |
| mac_address | 00:50:56:81:cb:45 |
| mode | STATIC |
| ip | 10.102.64.201/22 |
| gateway | 10.102.67.254 |
| labels[1] | MGMT |
| labels[2] | SE_SECURE_CHANNEL |
| labels[3] | HSM |
| interfaces[2] | |
| if_name | eth1 |
| mac_address | 00:50:56:81:c0:89 |
+-----------------+----------------------------------------------+
In the above, the second interface (eth1) has been discovered.
- Configure the mode and ip details on the additional interface:
[admin:controller]: > configure cluster
[admin:controller]: cluster> nodes index 1
[admin:controller]: cluster:nodes> interfaces index 2
[admin:controller]: cluster:nodes:interfaces> mode static
[admin:controller]: cluster:nodes:interfaces> ip 100.64.218.90/24
[admin:controller]: cluster:nodes:interfaces> labels HSM
[admin:controller]: cluster:nodes:interfaces> save
[admin:controller]: cluster:nodes> interfaces index 1
[admin:controller]: cluster:nodes:interfaces> no labels HSM
[admin:controller]: cluster:nodes:interfaces> save
In the above,
-
For the second interface (index 2), the IP and label has been added.
-
The label HSM has been removed from the primary interface (index 1).
Note: The nodes that already are configured with additional interfaces and routes, can be added to a cluster. For more details on configuring cluster, refer to API - Configuring the Avi Controller Cluster guide.
Unconfiguring the Additional Interface for a Single Node Controller
To revert the configuration to use the primary interface only,
-
Remove the configuration (mode, IP, labels) from the second interface (eth1).
-
Add the HSM label to the primary interface (eth0).
[admin:controller]: > configure cluster [admin:controller]: cluster> nodes index 1 [admin:controller]: cluster:nodes> interfaces index 2 [admin:controller]: cluster:nodes:interfaces> no mode [admin:controller]: cluster:nodes:interfaces> no ip [admin:controller]: cluster:nodes:interfaces> no labels HSM [admin:controller]: cluster:nodes:interfaces> save [admin:controller]: cluster:nodes> interfaces index 1 [admin:controller]: cluster:nodes:interfaces> labels HSM [admin:controller]: cluster:nodes:interfaces> save [admin:controller]: cluster:nodes> save [admin:controller]: cluster> save
Configuring a Static Route
A static route can be configured for the primary as well as secondary via the Cluster configuration.
Note: Starting with Avi Vantage version 21.1.3, you should not edit /etc/network/interfaces file. All configurations (IP, Static Route) should be via cluster configuration.
[admin:controller]: > configure cluster
[admin:controller]: cluster> nodes index 1
[admin:controller]: cluster:nodes> static_routes
New object being created
[admin:controller]: cluster:nodes:static_routes> prefix 1.1.1.0/24
[admin:controller]: cluster:nodes:static_routes> next_hop 100.64.218.20
[admin:controller]: cluster:nodes:static_routes> route_id 1
[admin:controller]: cluster:nodes:static_routes> if_name eth1
[admin:controller]: cluster:nodes:static_routes> save
[admin:controller]: cluster:nodes> save
[admin:controller]: cluster> where
Tenant: admin
Cloud: Default-Cloud
+--------------------+----------------------------------------------+
| Field | Value |
+--------------------+----------------------------------------------+
| uuid | cluster-83e1ebf5-2c63-4690-9aaf-b66e7a7b5f08 |
| name | cluster-0-1 |
| nodes[1] | |
| name | 10.102.64.201 |
| ip | 10.102.64.201 |
| vm_uuid | 00505681cb45 |
| vm_mor | vm-16431 |
| vm_hostname | node1.controller.local |
| interfaces[1] | |
| if_name | eth0 |
| mac_address | 00:50:56:81:cb:45 |
| mode | STATIC |
| ip | 10.102.64.201/22 |
| gateway | 10.102.67.254 |
| labels[1] | MGMT |
| labels[2] | SE_SECURE_CHANNEL |
| interfaces[2] | |
| if_name | eth1 |
| mac_address | 00:50:56:81:c0:89 |
| mode | STATIC |
| ip | 100.64.218.90/24 |
| labels[1] | HSM |
| static_routes[1] | |
| prefix | 1.1.1.0/24 |
| next_hop | 100.64.218.20 |
| if_name | eth1 |
| route_id | 1 |
+--------------------+----------------------------------------------+
[admin:controller]: cluster> save
Configuration for a 3-node Cluster
In case of a 3-node Cluster, the following steps are required:
-
For the discovery of the secondary interface, the Controller nodes need to be stand-alone, i.e., not part of a cluster. This is a one-time operation for Avi Vantage to discover the additional interface.
-
Once the secondary interfaces have been discovered, the Leader node can be used to form the cluster, as detailed in Deploying an Avi Controller Cluster.
-
After the cluster is fully formed, the secondary interface configuration for all the nodes can be performed.
[admin:controller]: cluster> nodes index 1
[admin:controller]: cluster:nodes> interfaces index 2
[admin:controller]: cluster:nodes:interfaces> mode static
[admin:controller]: cluster:nodes:interfaces> ip 100.64.218.90/24
[admin:controller]: cluster:nodes:interfaces> labels HSM
[admin:controller]: cluster:nodes:interfaces> save
[admin:controller]: cluster:nodes> interfaces index 1
[admin:controller]: cluster:nodes:interfaces> no labels HSM
[admin:controller]: cluster:nodes:interfaces> save
[admin:controller]: cluster:nodes> save
[admin:controller]: cluster> nodes index 2
[admin:controller]: cluster:nodes> interfaces index 2
[admin:controller]: cluster:nodes:interfaces> mode static
[admin:controller]: cluster:nodes:interfaces> ip 100.64.218.100/24
[admin:controller]: cluster:nodes:interfaces> labels HSM
[admin:controller]: cluster:nodes:interfaces> save
[admin:controller]: cluster:nodes> interfaces index 1
[admin:controller]: cluster:nodes:interfaces> no labels HSM
[admin:controller]: cluster:nodes:interfaces> save
[admin:controller]: cluster:nodes> save
[admin:controller]: cluster> nodes index 3
[admin:controller]: cluster:nodes> interfaces index 2
[admin:controller]: cluster:nodes:interfaces> mode static
[admin:controller]: cluster:nodes:interfaces> ip 100.64.218.110/24
[admin:controller]: cluster:nodes:interfaces> labels HSM
[admin:controller]: cluster:nodes:interfaces> save
[admin:controller]: cluster:nodes> interfaces index 1
admin:controller]: cluster:nodes:interfaces> no labels HSM
[admin:controller]: cluster:nodes:interfaces> save
[admin:controller]: cluster:nodes> save
[admin:controller]: cluster> save
Notes:
-
There is no requirement to log in to the node for the interface discovery to succeed. The only requirement is for the interface to be in a connected state in the VM, and for the Controller to have been powered on.
-
The cluster formation and the secondary interface configuration should be performed as separate steps.
Configuring IPv6 addresses for Secondary Interface
Starting with NSX Advanced Load Balancer version 22.1.3 release, you can add mode6, ip6, and gateway6 instead of mode, IP, and gateway for the IPv6 interface. The interface configuration does not support dual-stack mode in 22.1.3. Hence, an interface can have either a V4 IP or a V6 IP, not both.
The SE_SECURE_CHANNEL label could be moved to the secondary interface to enable communication to Service Engines. This secondary interface could be either of IPv4 or IPv6. This would help users to have different interfaces for management and Service Engine communication.
Sample configuration for IPv6 interface with SE_SECURE_CHANNEL label attached to IPv6 interface is as shown below:
+-----------------+----------------------------------------------+
| Field | Value |
+-----------------+----------------------------------------------+
| uuid | cluster-f29ed7c8-0da5-4fb6-87f7-e792584643b3 |
| name | cluster-0-1 |
| nodes[1] | |
| name | 100.65.9.203 |
| ip | 100.65.9.203 |
| vm_uuid | 000000675c79 |
| vm_mor | vm-22988 |
| vm_hostname | node1.controller.local |
| interfaces[1] | |
| if_name | eth0 |
| mac_address | 00:00:00:67:5c:79 |
| mode | STATIC |
| ip | 100.65.9.203/20 |
| gateway | 100.65.15.254 |
| labels[1] | MGMT |
| labels[2] | HSM |
| interfaces[2] | |
| if_name | eth1 |
| mac_address | 00:00:00:1a:ab:e8 |
| mode | STATIC |
| ip | 100.65.14.66/20 |
| interfaces[3] | |
| if_name | eth2 |
| mac_address | 00:00:00:3e:8b:ef |
| labels[1] | SE_SECURE_CHANNEL |
| mode6 | STATIC |
| ip6 | 2402:740:0:40e::20:3/128 |
+-----------------+----------------------------------------------+
Considerations
The following considerations should be noted:
-
The interface names, eth0, eth1, and so on, and discovered MAC addresses are static, and cannot be modified.
-
The primary (eth0) interface cannot be modified, apart from the labels.
-
All labels need to be a part of some interface and a label cannot be repeated in more than one interface.
-
For the additional interface, only Static IP mode is supported. DHCP is not supported.
-
The Access Controls are applied only to the primary interface. It is recommended to continue to use external firewall settings to restrict access, for instance, inbound SSH to the additional interface.
-
You should not edit
/etc/network/interfacesfile. All configurations, such as IP, Static Route, should be via cluster configuration. -
The secondary interfaces should remain in connected state within the virtual machine. Disconnecting them may lead to the interface being removed, if the virtual machine is rebooted.
Document Revision History
| Date | Change Summary |
|---|---|
| Dec 20, 2021 | 21.1.3 – Feature added, with support for 2nd interface and use with HSM, for vCenter Controllers's |