Controller Interface and Route Management

Overview

The Avi Controller has a single interface, which is used for various control plane related tasks such as:

  • Operator access to the Controller via CLI, UI, API.

  • Communication between the Controller and the Service Engines.

  • Communication between the Controller and third party entities for automation, observability, etc.

  • Communication between the Controller and third party Hardware Security Modules (HSMs).

Starting with Avi Vantage version 21.1.3, an additional interface is available on the Controller, to allow the ability to isolate the communication for some of the above entities.

In addition, any static routes to be added to the Controller interfaces should now leverage the cluster configuration instead of /etc/network/interfaces subsystem.

These configurations will be persisted across the Controller reboot and upgrade.

Note:

  • This feature is supported only on the Controllers deployed in vCenter.

  • This feature enables the use of the additional interface only for HSMs.

Classification

The following are the classifications to isolate the communication. The labels available are as follows:

  • MGMT — This signifies general management communication for the Controller access, as well as the Controller initiating communication, for instance, logging, third party API calls, and so on.

  • SE_SECURE_CHANNEL — This label is used to classify secure communication between the Service Engine and the Controller.

  • HSM — This is used to classify communication between the Controller and an HSM device.

With this classification, the traffic can be moved from the default, main interface to the additional interface, if configured.

Notes:

  • MGMT and SE_SECURE_CHANNEL can only be performed by the primary (eth0) interface.

  • HSM can be moved to the additional interface.

Operating Model

By default (prior to 21.1.3), the Controller is provisioned with one interface when being deployed in vCenter (during installation).

When an additional interface is required, you can follow these steps:

  • Shut down the Controller virtual machine and add the interface via vCenter UI.

  • On powering on the Controller virtual machine, Avi Vantage will recognize the additional interface, and additional configuration via the Avi CLI can be performed.

Note: Hotplug of interfaces (addition to the virtual machine without powering off the virtual machine) is not supported.

For the interface to be recognized within the Avi Controller software and further classification via labels to be performed, Avi Vantage’s ‘cluster’ configuration model should be used.

Configuration Steps

Configuration for a Single Node Controller

The following are the configuration steps:

  • Shut down the Controller and add the new interface via the vCenter.

  • Power on the Controller. The new interface will be visible as eth1, while the primary interface will always be visible as eth0 in the Cluster configuration:


[admin:controller]: > show cluster 

+-----------------+----------------------------------------------+ 

| Field           | Value                                        | 

+-----------------+----------------------------------------------+ 

| uuid            | cluster-83e1ebf5-2c63-4690-9aaf-b66e7a7b5f08 | 

| name            | cluster-0-1                                  | 

| nodes[1]        |                                              | 

|   name          | 10.102.64.201                                | 

|   ip            | 10.102.64.201                                | 

|   vm_uuid       | 00505681cb45                                 | 

|   vm_mor        | vm-16431                                     | 

|   vm_hostname   | node1.controller.local                       | 

|   interfaces[1] |                                              | 

|     if_name     | eth0                                         | 

|     mac_address | 00:50:56:81:cb:45                            | 

|     mode        | STATIC                                       | 

|     ip          | 10.102.64.201/22                             | 

|     gateway     | 10.102.67.254                                | 

|     labels[1]   | MGMT                                         | 

|     labels[2]   | SE_SECURE_CHANNEL                            | 

|     labels[3]   | HSM                                          | 

|   interfaces[2] |                                              | 

|     if_name     | eth1                                         | 

|     mac_address | 00:50:56:81:c0:89                            | 

+-----------------+----------------------------------------------+

In the above, the second interface (eth1) has been discovered.

  1. Configure the mode and ip details on the additional interface:

[admin:controller]: > configure cluster 

[admin:controller]: cluster> nodes index 1 

[admin:controller]: cluster:nodes> interfaces index 2 

[admin:controller]: cluster:nodes:interfaces> mode static 

[admin:controller]: cluster:nodes:interfaces> ip 100.64.218.90/24 

[admin:controller]: cluster:nodes:interfaces> labels HSM 

[admin:controller]: cluster:nodes:interfaces> save 

[admin:controller]: cluster:nodes> interfaces index 1 

[admin:controller]: cluster:nodes:interfaces> no labels HSM 

[admin:controller]: cluster:nodes:interfaces> save

In the above,

  • For the second interface (index 2), the IP and label has been added.

  • The label HSM has been removed from the primary interface (index 1).

Note: The nodes that already are configured with additional interfaces and routes, can be added to a cluster. For more details on configuring cluster, refer to API - Configuring the Avi Controller Cluster guide.

Unconfiguring the Additional Interface for a Single Node Controller

To revert the configuration to use the primary interface only,

  1. Remove the configuration (mode, IP, labels) from the second interface (eth1).

  2. Add the HSM label to the primary interface (eth0).

    
 [admin:controller]: > configure cluster 
    
 [admin:controller]: cluster> nodes index 1 
    
 [admin:controller]: cluster:nodes> interfaces index 2 
    
 [admin:controller]: cluster:nodes:interfaces> no mode 
    
 [admin:controller]: cluster:nodes:interfaces> no ip 
    
 [admin:controller]: cluster:nodes:interfaces> no labels HSM 
    
 [admin:controller]: cluster:nodes:interfaces> save 
    
 [admin:controller]: cluster:nodes> interfaces index 1 
    
 [admin:controller]: cluster:nodes:interfaces> labels HSM 
    
 [admin:controller]: cluster:nodes:interfaces> save 
    
 [admin:controller]: cluster:nodes> save 
    
 [admin:controller]: cluster> save

Configuring a Static Route

A static route can be configured for the primary as well as secondary via the Cluster configuration.

Note: Starting with Avi Vantage version 21.1.3, you should not edit /etc/network/interfaces file. All configurations (IP, Static Route) should be via cluster configuration.


[admin:controller]: > configure cluster 

[admin:controller]: cluster> nodes index 1 

[admin:controller]: cluster:nodes> static_routes 

New object being created 

[admin:controller]: cluster:nodes:static_routes> prefix 1.1.1.0/24 

[admin:controller]: cluster:nodes:static_routes> next_hop 100.64.218.20 

[admin:controller]: cluster:nodes:static_routes> route_id 1 

[admin:controller]: cluster:nodes:static_routes> if_name eth1 

[admin:controller]: cluster:nodes:static_routes> save 

[admin:controller]: cluster:nodes> save 

[admin:controller]: cluster> where 

Tenant: admin 

Cloud: Default-Cloud 

+--------------------+----------------------------------------------+ 

| Field              | Value                                        | 

+--------------------+----------------------------------------------+ 

| uuid               | cluster-83e1ebf5-2c63-4690-9aaf-b66e7a7b5f08 | 

| name               | cluster-0-1                                  | 

| nodes[1]           |                                              | 

|   name             | 10.102.64.201                                | 

|   ip               | 10.102.64.201                                | 

|   vm_uuid          | 00505681cb45                                 | 

|   vm_mor           | vm-16431                                     | 

|   vm_hostname      | node1.controller.local                       | 

|   interfaces[1]    |                                              | 

|     if_name        | eth0                                         | 

|     mac_address    | 00:50:56:81:cb:45                            | 

|     mode           | STATIC                                       | 

|     ip             | 10.102.64.201/22                             | 

|     gateway        | 10.102.67.254                                | 

|     labels[1]      | MGMT                                         | 

|     labels[2]      | SE_SECURE_CHANNEL                            | 

|   interfaces[2]    |                                              | 

|     if_name        | eth1                                         | 

|     mac_address    | 00:50:56:81:c0:89                            | 

|     mode           | STATIC                                       | 

|     ip             | 100.64.218.90/24                             | 

|     labels[1]      | HSM                                          | 

|   static_routes[1] |                                              | 

|     prefix         | 1.1.1.0/24                                   | 

|     next_hop       | 100.64.218.20                                | 

|     if_name        | eth1                                         | 

|     route_id       | 1                                            | 

+--------------------+----------------------------------------------+

[admin:controller]: cluster> save

Configuration for a 3-node Cluster

In case of a 3-node Cluster, the following steps are required:

  • For the discovery of the secondary interface, the Controller nodes need to be stand-alone, i.e., not part of a cluster. This is a one-time operation for Avi Vantage to discover the additional interface.

  • Once the secondary interfaces have been discovered, the Leader node can be used to form the cluster, as detailed in Deploying an Avi Controller Cluster.

  • After the cluster is fully formed, the secondary interface configuration for all the nodes can be performed.


[admin:controller]: cluster> nodes index 1 

[admin:controller]: cluster:nodes> interfaces index 2 

[admin:controller]: cluster:nodes:interfaces> mode static 

[admin:controller]: cluster:nodes:interfaces> ip 100.64.218.90/24 

[admin:controller]: cluster:nodes:interfaces> labels HSM 

[admin:controller]: cluster:nodes:interfaces> save 

[admin:controller]: cluster:nodes> interfaces index 1 

[admin:controller]: cluster:nodes:interfaces> no labels HSM 

[admin:controller]: cluster:nodes:interfaces> save 

[admin:controller]: cluster:nodes> save 

[admin:controller]: cluster> nodes index 2 

[admin:controller]: cluster:nodes> interfaces index 2 

[admin:controller]: cluster:nodes:interfaces> mode static 

[admin:controller]: cluster:nodes:interfaces> ip 100.64.218.100/24 

[admin:controller]: cluster:nodes:interfaces> labels HSM 

[admin:controller]: cluster:nodes:interfaces> save 

[admin:controller]: cluster:nodes> interfaces index 1 

[admin:controller]: cluster:nodes:interfaces> no labels HSM 

[admin:controller]: cluster:nodes:interfaces> save 

[admin:controller]: cluster:nodes> save 

[admin:controller]: cluster> nodes index 3 

[admin:controller]: cluster:nodes> interfaces index 2 

[admin:controller]: cluster:nodes:interfaces> mode static 

[admin:controller]: cluster:nodes:interfaces> ip 100.64.218.110/24 

[admin:controller]: cluster:nodes:interfaces> labels HSM 

[admin:controller]: cluster:nodes:interfaces> save 

[admin:controller]: cluster:nodes> interfaces index 1 

admin:controller]: cluster:nodes:interfaces> no labels HSM 

[admin:controller]: cluster:nodes:interfaces> save 

[admin:controller]: cluster:nodes> save 

[admin:controller]: cluster> save

Notes:

  • There is no requirement to log in to the node for the interface discovery to succeed. The only requirement is for the interface to be in a connected state in the VM, and for the Controller to have been powered on.

  • The cluster formation and the secondary interface configuration should be performed as separate steps.

Considerations

The following considerations should be noted:

  • The interface names, eth0, eth1, and so on, and discovered MAC addresses are static, and cannot be modified.

  • The primary (eth0) interface cannot be modified, apart from the labels.

  • The default gateway cannot be configured for the additional interfaces.

  • All labels needs to be a part of some interface and a label cannot be repeated in more than one interface.

  • For the additional interface, only Static IP mode is supported. DHCP is not supported.

  • The Access Controls are applied only to the primary interface. It is recommended to continue to use external firewall settings to restrict access, for instance, inbound SSH to the additional interface.

  • You should not edit /etc/network/interfaces file. All configurations, such as IP, Static Route, should be via cluster configuration.

  • The secondary interfaces should remain in connected state within the virtual machine. Disconnecting them may lead to the interface being removed, if the virtual machine is rebooted.

Document Revision History

Date Change Summary
Dec 20, 2021 21.1.3 – Feature added, with support for 2nd interface and use with HSM, for vCenter Controllers's