Changing Avi Controller Cluster Configuration
Overview
This article describes how to change node membership or node IP address in an Avi Controller cluster.
- Deploying a New Cluster
- Removing Both Followers
- Changing a Follower Node
- Replacing the Leader Node
- Replacing a Follower Node with a New Node
Note: If you change IP address of an Avi Controller node, you must run a script to update the configuration. Refer to Updating the Configuration Following Avi Controller IP Address Change for more information.
Deploying an Avi Controller Cluster
This section shows the web interface pop-up and CLI commands for deploying an Avi Controller cluster.
To deploy an Avi Controller cluster, first deploy a single Avi Controller node (the leader), and then add the follower nodes to the leader. (Ensure that after using the setup wizard to install Avi Vantage on the follower nodes, you do not make any other configuration changes on these nodes.) For more information about initial cluster deployment, refer to Deploying an Avi Controller Cluster.
Web Interface
The following are the steps to deploy a cluster on the web interface of the Avi Controller that will be the leader:
- Navigate to Administration > Controller > Nodes and click on Edit.
- Specify Controller Cluster name and IP address. For more details on Controller cluster IP address, refer to Controller Cluster IP address. This will be the shared management IP address of the cluster.
- Specify the host IP addresses of each of the 3 nodes in the Cluster Nodes fields:
- Controller Node-1: Host IP address of the leader node.
- Controller Node-2: Host IP address of one of the follower nodes.
- Controller Node-3: Host IP address of one of the other follower nodes.
- Click on Save.
CLI
Login to the CLI (or CLI shell) and enter the commands shown in the following example.
Note: Ensure that you specify the host IP addresses of your Avi Controller nodes instead of the IP addresses shown in the example.
Configure the cluster with the Controller nodes at [‘10.10.25.81’, ‘10.10.25.82’, ‘10.10.25.83’].
configure cluster
Updating an existing object. Currently, the object is:
+---------------+----------------------------------------------+
| Field | Value |
+---------------+----------------------------------------------+
| uuid | cluster-eb01bf05-7313-4a4f-91b6-21e46d3c237d |
| name | cluster-0-1 |
| nodes[1] | |
| name | 10.10.25.81 |
| ip | 10.10.25.81 |
| vm_ref | EB01BF05-7313-4A4F-91B6-21E46D3C237D |
| vm_mor | |
| vm_hostname | node1.controller.local |
| tenant_ref | admin |
+---------------+----------------------------------------------+
: cluster> nodes name 10.10.25.82 ip 10.10.25.82
New object being created
: cluster:nodes> save
: cluster> nodes name 10.10.25.83 ip 10.10.25.83
New object being created
: cluster:nodes> save
: cluster> save
If you add or remove nodes from the cluster, then you need to bring down this Controller and back it up with the new configuration.
Waiting for the cluster to be ready...
Controller is ready.
Removing both Followers (Dismantling the Cluster)
Web Interface
The following are the steps to remove both followers from an Avi Controller cluster:
- Navigate to Administration > Controller > Nodes .
- Click on Edit.
- Remove each of the follower IP addresses from the configuration pop-up.
- Click on Save.
CLI
Login to the CLI (or CLI shell) and enter the commands shown in the following example.
Note: Ensure that you specify the host IP addresses of your Avi Controller nodes instead of the IP addresses shown in the example.
configure cluster
Updating an existing object. Currently, the object is:
+---------------+----------------------------------------------+
| Field | Value |
+---------------+----------------------------------------------+
| uuid | cluster-eb01bf05-7313-4a4f-91b6-21e46d3c237d |
| name | cluster-0-1 |
| nodes[1] | |
| name | node-1 |
| ip | 10.10.25.81 |
| vm_ref | EB01BF05-7313-4A4F-91B6-21E46D3C237D |
| vm_mor | |
| vm_hostname | node1.controller.local |
| nodes[2] | |
| name | node-2 |
| ip | 10.10.25.82 |
| vm_ref | EC123A05-7313-4A4F-91B6-21E46D3D46AF |
| vm_mor | |
| vm_hostname | node2.controller.local |
| nodes[3] | |
| name | 10.10.25.83 |
| ip | 10.10.25.83 |
| vm_ref | EA12C05-7313-4A4F-91B6-21E46D3E256EA |
| vm_mor | |
| vm_hostname | node3.controller.local |
| tenant_ref | admin |
+---------------+----------------------------------------------+
: cluster> no nodes name node-2
Removed nodes with name=node-2
: cluster:nodes> save
: cluster> no nodes name node-3
Removed nodes with name=node-3
: cluster:nodes> save
: cluster> save
Configuring the cluster with the controller nodes at [10.10.25.81’].
If you add or remove nodes from the cluster, you need to bring down this Controller and then back it up with the new configuration.
Waiting for the cluster to be ready...
Controller is ready.
After saving,
- The cluster is dismantled and recovered into a single node.
- Each follower is re-imaged into its default state with no configuration and no access to the leader.
- The leader holds the configuration. The Avi Service Engines (SEs) continue to connect to the leader.
For more information about the transition process, including descriptions of the status messages that appear, refer to Deploying an Avi Controller Cluster.
Note: During the transition from a cluster to a single node, the REST API will be unavailable for around 2 minutes.
Changing a Follower Node
To remove one of the follower nodes and add another one, use the following steps.
Note: If the node is removed and replaced with a different node (different VM, container, or bare-metal server), click here instead. The cluster has to be dismantled, and then recreated using the new node.
Web Interface
The following are the steps to change a follower mode in the web interface:
- Navigate to Administration > Controller > Nodes .
- Click on Edit.
- Edit the IP address for the follower node to changed.
- Click on Save.
CLI
Login to the CLI (or CLI shell) and enter the commands shown in the following example.
Note: Ensure that you enter the host IP addresses of your Avi Controller nodes instead of the IP addresses shown in the example.
configure cluster
Updating an existing object. Currently, the object is:
+---------------+----------------------------------------------+
| Field | Value |
+---------------+----------------------------------------------+
| uuid | cluster-eb01bf05-7313-4a4f-91b6-21e46d3c237d |
| name | cluster-0-1 |
| nodes[1] | |
| name | node-1 |
| ip | 10.10.25.81 |
| vm_ref | EB01BF05-7313-4A4F-91B6-21E46D3C237D |
| vm_mor | |
| vm_hostname | node1.controller.local |
| nodes[2] | |
| name | node-2 |
| ip | 10.10.25.82 |
| vm_ref | EC123A05-7313-4A4F-91B6-21E46D3D46AF |
| vm_mor | |
| vm_hostname | node2.controller.local |
| nodes[3] | |
| name | 10.10.25.83 |
| ip | 10.10.25.83 |
| vm_ref | EA12C05-7313-4A4F-91B6-21E46D3E256EA |
| vm_mor | |
| vm_hostname | node3.controller.local |
| tenant_ref | admin |
+---------------+----------------------------------------------+
: cluster> no nodes name node-3
Removed nodes with name=node-3
: cluster:nodes> save
: cluster> nodes name node-4 ip 10.10.25.84
Removed nodes with name=node-4
: cluster:nodes> save
: cluster> save
Configuring the cluster with the controller nodes at [u’10.10.25.81’, ‘10.10.25.82’, ‘10.10.25.84’]. If you add or remove nodes from the cluster, you should bring down this Controller and then back up with the new configuration.
Waiting for the cluster to be ready...
Controller is ready.
After saving,
- The removed follower node is sent an API request asking it to gracefully clear its own state.
- Since the old follower is not always expected to clear its own state, the leader will forcibly remove it if necessary.
- The new follower node is added.
Replacing the Leader Node
To replace the leader node, power it down to force one of the followers to take over leadership, then use the following steps to replace a follower.
- Power down the leader node and leave it powered off for several minutes while one of the followers assumes leadership.
- Use the steps for adding a new follower node.
Note: The leader cannot be directly removed. Instead, it must be replaced with another leader.
Replacing a Follower Node with a New Node
These steps describe how to replace a follower node with a new node.
Notes:
- This procedure is applicable for replacing a follower node with a completely different node that is not currently in the cluster. For instance, these steps apply to replacing a follower node with a new VM or bare metal server.
- The node cannot be replaced simply by inserting the node into the network and editing the cluster configuration information.
The following are the steps to replace a follower node:
- Log onto the leader node, and use the steps to dismantle the cluster by removing both follower nodes.
- On the new node, install Avi Vantage. This involves spinning up a new virtual machine (VM) or container from the Avi Controller package (OVA, QCOW or Docker image) on the new node.
- Follow the steps mentioned in Deploy a new cluster section.