Flexible Upgrades for Avi Vantage
Overview
Starting with release 18.2.6, Avi Vantage supports improved and more flexible methods for upgrading the Avi Vantage system. The followings are the additional features for the Flexible Upgrades:
- The upgrade is possible per SE group. The transition of all the SE groups to the new version may occur over a long period.
- Upgrades of different SE groups are supported with different patch versions.
- Rollback to the previous versions of Avi Vantage is non-disruptive.
From Avi Vantages prior to 18.2.6, the only available option is the system-level (Avi Controller and SE groups) upgrade. With Flexible Upgrades, the following options are available:
Upgrades | Patch Ugrades | Rollback | Rollback Patch |
---|---|---|---|
System (Avi Controller and SE groups) | System (Avi Controller and SE groups | System (Avi Controller and SE groups | System (Avi Controller and SE groups |
Avi Controller only | Avi Controller only | Avi Controller only | Avi Controller only |
Some or all the SE groups | Some or all the SE groups | Some or all the SE groups | Some or all the SE groups |
Use Cases
- Scenarios when it is not possible to upgrade all SE groups to the newer version at the same time due to various business reasons such as logistics, confidence in the new software, etc.
- The configuration is blocked during the entire duration of the Avi Controller and SE upgrade. This is not acceptable in many deployments. With the new upgrade feature, the process is flexible and can be performed per SE group basis. The configuration is blocked for the entire duration if a system upgrade is performed till all Service Engines are upgraded.
- Using SE groups for data plane separation. Based upon the SE group segmentation, the upgrade is performed based upon the following attributes.
- Application or product offering
- Tenant
- Production, pre-production and development environments
- Cloud or environment (AWS, VMware, etc.)
- Provide patches to only applications or SE groups that need them
- Flexible scheduling
- Self-service upgrades
Image Management and Service
Image service is the first step in the flexible upgrade work-flow. It is used to upload the image after which an upgrade operation can be initiated. The Avi Controller hosts images of different versions since SE groups could be potentially in different versions.
The Avi Controller should have additional disk space to host these images.
Avi Controller images for the major versions include the followings:
- controller.pkg (for VM-based Avi Controller)
- controller_docker.tgz (For Docker-based Controller)
Images for the patches include the followings:
- avi_patch.pkg — Full package
- controller_patch.pkg — Avi Controller package
- se_patch.pkg — SE patch package
As a part of the upload process, image service extracts files, metadata from the package. This information is not only presented to the user but also used in the upgrade process.
Notes:
- Images from Avi Vantage release 17.2.8 onwards are upgradeable to an image for Avi Vantage release 18.2.6. The image prior to the release 17.2.8, should be migrated to 17.2.8 image after which it can be upgraded to 18.2.6.
- Image service provides an ability to upload, query and delete Avi image(s) to the system.
- Image service supports the upload of Avi patch packages.
- Image upload can happen only on the cluster leader. It is not allowed from a cluster member.
Image Bundling
Avi Vantage now supports the composite image or the image bundle. The composite image of Avi Vantage consists of the followings:
- Base image – Controller image (controller_docker.tgz, controller.pkg, controller ova, controller.qcow2, etc)
- Controller package – It is an optional package
- SE patch image – It is an optional package
The upgrade workflow using the image bundle, or the composite image is the same as using the standard image. When the image bundle is used for upgrade, a patch image can also be applied along with the base image.
Note:
When upgrading from Avi Vantage versions 17.x or version lesser than 18.2.6 to Avi Vantage 20.1 and higher, in the Avi Controller, change the DefaultTimeoutStartSec
(File: /etc/systemd/system.conf) to 120 seconds to avoid timeout during upgrade.
Uploading Image Using Avi CLI
The CLI for Avi Vantage release 18.2.6 provides better control of the upgrade operations leading to a consistent and predictable workflow.
For uploading the package use the upload image filename <path-of-the-package>
command as shown below.
[admin:controller]: > upload image filename /tmp/controller.pkg
The following show command returns the details of the image metadata.
show image <image-name>
[admin:-controller]: > show image
+-----------------------------+--------------------------------------------+----------------+
| Name | UUID | Status |
+-----------------------------+--------------------------------------------+----------------+
| 18.2.7-5000-20191009.205501 | image-fxxxx22-0f40-45de-8551-15xxxxxxx1fe | SYSERR_SUCCESS |
+-----------------------------+-----
The existing API endpoints (prior to 18.2.6) are not supported. To know more about differences in CLI commands and APIs refer to Comparison Table for Differences in CLIs Commands and APIs.
Uploading Image Service using Avi REST API
A POST operation is used to do an image upload. To get the image details in response, run a GET API request.
-
Use the following REST API to upload image for controller.pkg.
URI :/api/image
Method:POST
root@admin:-controller# curl -X POST -k https://10.58.3.27/api/image -u "admin:admin" -F file=@controller.pkg
-
Use the following REST API to upload image for controller_patch.pkg.
root@admin:-controller-18.2.5-2p3-9002# curl -X POST -k https://10.58.3.27/api/image -u "admin:admin" -F file=@se_patch.pkg
- Use the following API to delete the image provided, if it is not in use.
delete image <image-name>
Must-Checks for Upgrade
Prior to upgrade operations, various must-checks are run to check the various mandatory and optional requirements for upgrade. The outputs message is exhibited as error message or as Warning message. Warnings can be skipped while ‘Errors’ cannot be over-ridden. API/CLI provides the skip_warnings option to control the above behavior.
For Avi CLI— This is directly integrated into the normal work-flow and there is no separate command.
For the REST API — Add /preview/
at the end of APIs to get previews for that particular flow.
Starting with NSX Advanced Load Balancer 22.1.3, in order to start upgrade operation, all the CLI upgrade request should go with skip_warning
option. Without theskip_warning
option, the system state for any operation would lead to PRE_CHECK_WARNING and halt.
[admin:10-10-10-1]: > upgrade system image_ref 30.3.3-7235-20230110.035149 skip_warnings
+-------------+------------------------------------------------------------------------------+
| Field | Value |
+-------------+------------------------------------------------------------------------------+
| status_code | SYSERR_UPGRADE_OPS_PREVIEW_RESPONSE |
| status | Checks preview for upgrade operations. |
| checks | |
| | Check Controller Cluster readiness for upgrade operations. |
| | Check and inform user to take a backup prior to upgrade operations. |
| | Check if se linux is enabled on controller nodes. |
| | Check if upgrade operation is already in progress. |
| | Check ServiceEngineGroup has an ongoing upgrade operation. |
| | Check image version compatibility for upgrade operations. |
| | Check ServiceEngine reachability for upgrade operations. |
| | Check ServiceEngine disk space for upgrade operations. |
| | Check Controller Cluster disk space for upgrade operations. |
| | Check and inform Virtual Service(s) disruption for upgrade operations. |
| | Check idempotent operations for upgrade operations. |
| | Check active versions compatibility for upgrade operations. |
| | Check ServiceEngineGroup error recovery options prior to upgrade operations. |
| | Check Image state across Cluster members for upgrade operations. |
| | Checks for the patch in image bundle. |
| | Checks if Gslb Feature is enabled and provides feature specific messages. |
| | Checks the system configuration. |
| | Check total number of alerts for upgrade operations. |
| | Checks if the cloud api versions are compatible after upgrade. |
| | Checks if Docker version is compatible. |
| | Checks if configured IP type is DHCP or STATIC. |
| | Checks if se has a valid license state. |
+-------------+------------------------------------------------------------------------------+
Starting upgrade
+-------------+-----------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------+-----------------------------------------------------------------------------------------------------------+
| status_code | SYSERR_UPGRADE_SYSTEM_STARTED |
| status | 'Upgrade of System (Controller + All SEGroup(s)) started. Use 'show upgrade status' to check the status.' |
+-------------+-----------------------------------------------------------------------------------------------------------+
Similarly, Use the skip warning option while performing the patch upgrade.
[admin:10-10-10-1]: > patch controller controller_patch_ref 23.1.1-7189-2p1-20221216.192828 skip_warnings
Previewing upgrade
Upgrading Avi System (Avi Controller and SE Groups)
The configuration and placement of virtual services are blocked if it is a system-level upgrade till all the Service Engines are upgraded. Once these operations are completed, configuration on Avi Controller (except the configuration of virtual service and VIP) is allowed, irrespective of the SE group upgrade status.
Note: It is recommended to increase the default timeout value from 90 seconds to 120 seconds before performing upgrade. This is to avoid upgrade going to timeout.
Using Avi CLI
Notes:
- The auto-suggest option in the Avi CLI provides available values on pressing tab on your keyboard.
skip_warnings
— Use this option to skip any warnings and optional must checks.
The following are the various options available for Avi system upgrade.
- Use the
upgrade system image_ref <image name>
command to upgrade the system to a base image.[admin:-controller]: >upgrade system image_ref 18.2.6-9000-20191031.063017
- Use the following to upgrade the system to a base image and a controller patch.
[admin:-controller]: >upgrade system image_ref 18.2.6-9134-20191101.042535 controller_patch_ref 18.2.6-9134-2p1-20190806.011824
- Use the following to upgrade the system to a base image and an SE patch.
[admin:-controller]: >upgrade system image_ref 18.2.6-9134-20191101.042535 se_patch_ref 18.2.6-9134-2p1-20190806.011824
- Use the following to upgrade the system to a base image, an Avi Controller patch, and an SE patch
[admin:-controller]: >upgrade system image_ref 18.2.6-9134-20191101.042535 controller_patch_ref 18.2.6-9134-2p1-20190806.011824 se_patch_ref 18.2.6-9134-2p1-20190806.011824
SE Upgrades
The Controller allows you to pick up the number of SE-groups per Controller node.
seupgrade_fabric_pool_size: This property allows the Controller to pickup number of SE groups per Controller to upgrade.
For instance, if seupgrade_fabric_pool_size
is set to 3, three SE-groups are picked up per Controller, that means 9 SE groups across the cluster.
The default value of seupgrade_fabric_pool_size
is 20. However, you can update this based on the requirement or the load.
seupgrade_copy_pool_size: This parameter defines the number of simultaneous SE image downloads in a SEGroup. It is used to pace the SE downloads so that Controller network/ CPU bandwidth is a bounded operation. A value of zero will disable the pacing scheme and all the SE(s) in the SEGroup will attempt to download the image.
seupgrade_copy_pool_size = n
, where ‘n’ is the number of SE within SE group will be picked for copy.
For instance, if seupgrade_copy_pool_size = 3
, the three SE in a picked up SE group will be picked for copy.
The default value of seupgrade_copy_pool_size
is 5. However, you can update this based on the requirement or the load.
The following are the steps to configure this:
- Configure the Controller properties.
- Set
seupgrade_fabric_pool_size <number>
. - Set
seupgrade_copy_pool_size <number>
. - Save.
[admin:ctrl]: > configure controller properties
[admin:ctrl]: controllerproperties> seupgrade_fabric_pool_size 2
Overwriting the previously entered value for seupgrade_fabric_pool_size
[admin:ctrl]: controllerproperties> seupgrade_copy_pool_size 2
Overwriting the previously entered value for seupgrade_copy_pool_size
[admin:ctrl]: controllerproperties> save
[admin:ctrl]: > show controller properties |grep pool
| seupgrade_fabric_pool_size | 2 |
| seupgrade_copy_pool_size | 2 |
[admin:ctrl]: >
Using Avi REST API
Image UUID can be obtained by Use the GET /api/image
to obtain Iamge UUID.
The following are the various REST API options available for Avi system upgrade.
- Use the following API to upgrade the system to a base image.
API:/api/upgrade
Method:POST
JSON Data:{ 'image_uuid': 'image-b8adc2bd-d27f-469d-b78d-5e2bc14a14e4', 'system': true }
- Use the following API to upgrade the system to a base image and a controller patch.
API:/api/upgrade
Method:POST
JSON Data:{ 'image_uuid': 'image-b8adc2bd-d27f-469d-b78d-5e2bc14a14e4', 'controller_patch_uuid': 'image-e3aaad68-5aaf-485a-8bd9-1db3ec562d6a', 'system': true }
- Use the following API to upgrade the system to a base image and an SE patch.
API:/api/upgrade
Method:POST
JSON Data:{ 'image_uuid': 'image-b8adc2bd-d27f-469d-b78d-5e2bc14a14e4', 'system': true, 'se_patch_uuid': 'image-e3aaad68-5aaf-485a-8bd9-1db3ec562d6a', 'skip_warnings': True }
- Use the following API to upgrade the system to a base image, an Avi Controller patch, and an SE patch
API:/api/upgrade
Method:POST
JSON Data:{ 'image_uuid': 'image-b8adc2bd-d27f-469d-b78d-5e2bc14a14e4', 'controller_patch_uuid': 'image-e3aaad68-5aaf-485a-8bd9-1db3ec562d6a', 'system': true, 'se_patch_uuid': 'image-e88aaad68-5aaf-485a-8bd9-1db3ec562d6a' }
Upgrading Avi Controller
Using Avi CLI
Login to the Avi shell prompt and use the following upgrade commands for various options.
-
Use the
upgrade controller image_ref <image name>
command to upgrade the Avi Controller to a base image.[admin:-controller]: >upgrade controller image_ref 18.2.6-9000-20191031.063017
-
Use the
upgrade controller image_ref <image name>controller_patch_ref <patch name>
command to upgrade the Avi Controller to a base image and an Avi Controller patch.[admin:-controller]: >upgrade controller image_ref 18.2.6-9134-20191101.042535 controller_patch_ref 18.2.6-9134-2p1-20190806.011824
Using Avi REST API
- Use the following API to upgrade the Avi Controller to a base image.
API:/api/upgrade
Method:POST
JSON Data:{ 'image_uuid': 'image-b8adc2bd-d27f-469d-b78d-5e2bc14a14e4' }
- Use the following API to upgrade an Avi Controller to a base image and an Avi Controller patch.
API:/api/upgrade
Method:POST
JSON Data:{ 'image_uuid': 'image-b8adc2bd-d27f-469d-b78d-5e2bc14a14e4', 'controller_patch_uuid': 'image-e3aaad68-5aaf-485a-8bd9-1db3ec562d6a' }
Note:
Upgrading SE Group
This interface is used to upgrade all or some of the SE groups.
Using Avi CLI
Login to the Avi shell prompt to use the various options available for SE group update.
- Use the
upgrade segroup se_group_refs Default-Group image_ref<image name>
command to upgrade an SE group to the Controller image.[admin:-controller]: >upgrade segroup se_group_refs Default-Group image_ref 18.2.6-9134-20191101.042535
- Use the
upgrade segroup se_group_refs Default-Group image_ref *lt;Controller image> se_patch_ref <SE patch name>:
command to upgrade an SE group to the Controller image and the SE patch image.[admin:-controller]: >upgrade segroup se_group_refs Default-Group image_ref 18.2.6-9134-20191101.042535 se_patch_ref 18.2.6-9134-2p1-20190806.011824
Using Avi REST API
SE Group UUID can be obtained by the GET /api/serviceenginegroup
API.
The followings are the additional options for SE group upgrade:
-
Disruptive — This is used to disable non-disruptive mechanism to facilitate a faster upgrade. If enabled, the SE(s) are upgraded in a disruptive manner. The default value is false.
-
Suspend-on-failure — This option suspends the upgrade of subsequent SE(s) within a SE-group when a failure is encountered in the SE upgrade path. The default value is false.
The followings are the different APIs for the SE group upgrade:
- Use the following API to upgrade the SE group to the Controller image.
API:/api/upgrade
Method:POST
JSON Data:{ 'image_uuid': 'image-b8adc2bd-d27f-469d-b78d-5e2bc14a14e4', 'se_group_uuids': [ 'serviceenginegroup-e553b1a6-4851-4e82-ad12-cecc4bbda6c7' ] }
- Use the following with the additional SE Group options — Disruptive and Suspend_on_failure.
API:/api/upgrade
Method:POST
JSON Data:{ 'image_uuid': 'image-b8adc2bd-d27f-469d-b78d-5e2bc14a14e4', 'se_group_uuids': [ 'serviceenginegroup-e553b1a6-4851-4e82-ad12-cecc4bbda6c7' ], 'disruptive':true, 'suspend_on_failure': true }
- Use the following API to upgrade the SE group to the Controller image and the SE patch image.
API:/api/upgrade
Method:POST
JSON Data:{ 'image_uuid': 'image-b8adc2bd-d27f-469d-b78d-5e2bc14a14e4', 'se_patch_uuid': 'image-e3aaad68-5aaf-485a-8bd9-1db3ec562d6a', 'se_group_uuids': [ 'serviceenginegroup-e553b1a6-4851-4e82-ad12-cecc4bbda6c7' ] }
Additional Options for SE Group Upgrade
The following upgrade options are available for upgrading SE group.
Option | Behaviour | Notes |
---|---|---|
SUSPEND_UPGRADE_OPS_ON_FAILURE | This option is used to suspend the upgrade-operations (Upgrade/Patch) on SE-Group if the SE(s) hit an issue and does NOT come up during the upgrade operations. | It is enabled by default. This option serializes the SE upgrades in the SE group upgrade. It increases the overall upgrade time for the entire SE group. Batch size is used to decrease the upgrade time. Even if the SEs does not have scaled-out virtual services, it still upgrades serially. |
CONTINUE_UPGRADE_OPS_ON_FAILURE | This option is used to continue the upgrade or patch upgrade operations on SE group even if the SE(s) hit an issue and does not come up during the upgrade operations.
Service disruption can be observed. |
This option parallelizes the SE upgrade in the SE group upgrade if SEs does not have scaled-out virutal services.
If SEs have scaled-out virtual services, then it continue with serial upgrades. |
Disruptive | This option is used to disable the non-disruptive nature of SE upgrade.
It is used to upgrade all the SE(s) in the group to the next version irrespective of the traffic disruption. |
This option is disabled by default.
All SE(s) will be upgraded in parallel, irrespective of scaled out virtual service existence. Traffic/Service disruption will take place. |
Upgrading using Patch Release
The followings are the available options for patch upgrade:
- System — Patch upgrade for Avi Controller and all SE groups
- Controller — Patch upgrade for the Avi Controller alone.
- SE group — Patch upgrade for some or all the SE groups.
Notes:
The following are a few points for a patch upgrade process:
- An image along with a patch can be applied.
- The image and the patch must have the same base version.
-
A patch cannot be applied without applying the image.
- Compatibility checks prevent incorrect patches from getting applied to different versions.
To upload the image for patch upgrades, refer to
Patch Upgrade for Avi System
Use the following CLI command for the base image upgrade with a patch image.
[admin:controller]: > upgrade system image <image-name> controller_patch <controller-patch-name> se_patch <se-patch-name>
[admin:controller]: >upgrade system image 18.2.6 controller_patch 18.2.6-1p1 se_patch 18.2.6-1p1
- Use the
upgrade system image_ref <image name > controller_patch_ref <SE patch name>
command for an Avi Vantage system upgrade with a Controller patch.[admin:-controller]: upgrade system image_ref 18.2.6-9000-20191031.063017 controller_patch_ref 18.2.6-2p1-20191031.063017
- Use the
upgrade system image_ref <image name> se_patch_ref <SE patch name>
command for an Avi Vantage system upgrade with only SE patch.[admin:-controller]: upgrade system image_ref 18.2.6-9000-20191031.063017 se_patch_ref 18.2.6-2p1-20191031.063017
- Use the
upgrade system image_ref <image name> controller_patch_ref <Controller patch image> se_patch_ref <SE patch image>
command for the system upgrade with both Controller and SE patch.[admin:-controller]:upgrade system image_ref 18.2.6-9000-20191031.063017 controller_patch_ref 18.2.6-2p1-20191031.063017 se_patch_ref 18.2.6-2p1-20191031.063017
Patch Upgrade for Avi Controller
This interface is used to patch upgrade for the Avi Controller.
Using Avi CLI
Use the upgrade controller image_ref <image name> controller_patch_ref <Controller patch image
command to upgrade the Avi Controller with a patch.
[admin:-controller]: upgrade controller image_ref 18.2.6-9000-20191031.063017 controller_patch_ref 18.2.6-2p1-20191031.063017
[admin:controller]: > patch controller <patch-name>
[admin:controller]: > patch controller controller_patch 18.2.5-5p1
Using Avi REST API
POST api/upgrade JSON data:{‘controller_patch_uuid’: <image-uuid>}
Patch Upgrade for SE Group
SE groups can be of different versions and different versions of patch can be applied.
Use the upgrade segroup image_ref <image name> se_group_refs Default-Group se_patch_ref <patch for the SE Group>
command to upgrade specific SE groups along with a patch.
[admin:-controller]: upgrade segroup image_ref 18.2.6-9000-20191031.063017 se_group_refs Default-Group se_patch_ref 18.2.6-2p1-20191031.063017
Note: Patch name and patch uuid is retrieved from the image service.
Rollback
Starting with 18.2.6, rollbacks are non-disruptive in nature.
When a rollback operation is performed, the Avi Controller or SEs will transition to the previous major version of the software. Selective rollback is possible for the Avi Controller and SE groups.
The following options are available:
- Rollback for System
- Rollback for Avi Controller only
- Rollback for some or all the SE groups
Note:
- Rollback of the SE Group will be to the previous version.
Rollback for System
Rollback of the system will result in the rollback of the SE(s) followed by the rollback of the Avi Controller. Use the following CLI and REST API for performing rollback for a patch version for Avi system (Controller and SE groups).
Using Avi CLI
[admin:controller]: > rollback system
Using Avi REST API
POST api/rollback JSON data:{‘system’:true}
POST api/rollback JSON data:{‘system’:true,‘rollback_type’:2}
Rollback for Avi Controller
This interface is used to rollback the Avi Controller.
Using Avi CLI
[admin:controller]: > rollback controller
Using Avi REST API
POST api/rollback
Rollback for SE Groups
Using Avi CLI
[admin:controller]: > rollback segroup <se-group-name>
[admin:controller]: > rollback segroup seg-a
Using Avi RESt API
POST api/rollback JSON data:{‘se_group_uuids’: [‘seg-a-uuid’]}
Rollback - Patch
Rollback of a patch release transitions the software to a version without the specific patch. It will NOT roll back to the previous major version.
Selective ability to rollback the patch on the Avi Controller and SE groups is available. Note: Rollback patch oPtion is available only from Avi Vantage release 18.2.7.
This interface is used to roll back the patch and not the major version.
The followings are the available options:
- System: rollback patch for Avi Controller and all SE groups
- Controller: rollback patch the Avi Controller only.
- SE-group: rollback patch for all or some of the SE groups.
Rollback Patch for System
Use the following CLI and REST API for performing rollback for Avi System (Avi Controller and SE groups).
Using Avi CLI
[admin:controller]: > rollbackpatch system
Using Avi REST APIs
POST api/rollback JSON data:{‘rollback_type’:2}
Rollback Patch for Avi Controller
Use the following CLI and REST API for performing rollback for a patch version for an Avi Controller.
Using Avi CLI
[admin:controller]: > rollbackpatch controller
Using Avi REST APIs Add here
Rollback Patch for SE Groups
Use the following CLI and REST API for performing rollback for a patch version for an SE group.
Using Avi CLI
[admin:controller]: > rollbackpatch segroup <se-group-name>
[admin:controller]: > rollbackpatch segroup seg-a
Using Avi REST APIs
POST api/rollback JSON data:{‘rollback_type’:2,‘se_group_uuids’: [‘seg-a-uuid’]}
Note Refer to Additional Options for Flexible Upgrade for the following additional options:
- Rollback - Error Recovery
- Abort Cleanup
- SE Group Resume Option
Show Commands
The following show commands provide software version visibility in the system:
show version controller
show version serviceengine
show version serviceenginegroup
The following commands provide upgrade visibility in the system.
show upgrade status
: Various filters will be implemented as per UI work-flow.show upgrade history
: This command is deprecated.
Notes:
- The Avi Controller will be at the highest version while the SE groups may be at lower versions. Certain commands may not work due to the Avi Controller version being at the highest version.
- Due to the API version semantics, certain fields may not be available as they are deprecated in annotation.
- Due to API endpoint deprecation, some internal commands may not work.
Alerts and Events
The following events are available to provide visibility:
- Image upload/delete events
- Upgrade-specific events
- Patch-specific events
- Rollback-specific events
- Rollback patch-specific events.
- Failures will translate into alerts.
Additional APIs
The following GET API calls are applicable:
-
The following REST API provides information about all the images present in the system.
Get API: api/image/
-
The following API provides information about a specific image whose UUID is passed as a slug.
Get API: api/image/image-uuid
- Use the following API to delete the image provided if not in use.
Delete API: api/image/image-uuid
- Inventory API —api/image-inventory This API provides the image inventory on the system. It provides filtering based on various options such as retrieve all packages for a version etc.