ECMP Support in OpenStack Contrail Environments
Avi Vantage can manage load balancing capacity for a virtual service by dynamically scaling it out or in on additional or fewer Service Engines (SEs). By default, the primary SE for the virtual service co-ordinates distribution of traffic flow amongst the secondary SEs, including itself. On OpenStack with Contrail, Avi Vantage can take advantage of Contrail’s ECMP support and manage the orchestration of ECMP routes as part of virtual service placement.
ECMP functionality can be homed at the following locations:
- The upstream edge router (e.g., Juniper MX or such)
- The Contrail vRouter on the host hypervisor.
Example
In Neutron, the VIP 19.1.1.7 is associated with the interface port of the SE named Avi-se-czpey.
root@dc11-cfg-1:~# neutron port-list
+--------------------------------------+------------+-------------------+---------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------------+-------------------+---------------------------------------------------------------------------------+
| dc4695c6-6d16-4ee0-ab05-774537a33ab2 | Avi-Data...| 02:dc:46:95:c6:6d | {"subnet_id": "a834986a-385e-4616-9d9c-6b91cfaa51e0", "ip_address": "19.1.1.6"} |
| | | | {"subnet_id": "a834986a-385e-4616-9d9c-6b91cfaa51e0", "ip_address": "19.1.1.7"} |
...
+--------------------------------------+------------+-------------------+---------------------------------------------------------------------------------+
In Contrail, the VIP 19.1.1.7 has the same linkage to port dc4695c6-6d16-4ee0-ab05-774537a33ab2 and one route to the SE VM running on hypervisor 172.16.11.101.
Virtual Service Scaleout
Using the Avi CLI, setting scaleout_ecmp to True enables a scaleout of vs2 as follows:
configure virtualservice vs2
scaleout_ecmp
save
+----------------------------------+-----------------------------------------------------+
| Field | Value |
+----------------------------------+-----------------------------------------------------+
| uuid | virtualservice-2f8770d0-20cc-477f-9419-b206b9624389 |
| name | vs2 |
| scaleout_ecmp | True |
| vip[1] | |
| vip_id | 0 |
| ip_address | 19.1.1.7 |
| enabled | True |
| network_ref | 02b846e6-cd68-4c84-b7ab-3709946ed28b |
| port_uuid | c0e4cb54-8706-4bed-97bb-271a603f858f |
| subnet_uuid | a834986a-385e-4616-9d9c-6b91cfaa51e0 |
| subnet | 19.1.1.0/24 |
| auto_allocate_ip | True |
| auto_allocate_floating_ip | False |
...
+----------------------------------+-----------------------------------------------------+
scaleout virtualservice vs2 vip_id 0
After the scaleout, vs2 runs on two SEs, Avi-se-czpey and Avi-se-pnjbb, as shown in the below summary output.
show virtualservice vs2 summary
+-----------------------+-----------------------------------------+
| Field | Value |
+-----------------------+-----------------------------------------+
| oper_status | |
| state | OPER_UP |
| percent_ses_up | 100 |
| vip_summary[1] | |
| vip_id | 0 |
| oper_status | |
| state | OPER_UP |
| service_engine[1] | |
| ref | Avi-se-czpey |
| primary | True |
| standby | False |
| service_engine[2] | |
| ref | Avi-se-pnjbb |
| primary | False |
| standby | False |
| num_se_requested | 2 |
| num_se_assigned | 2 |
+-----------------------+-----------------------------------------+
In Neutron, the VIP 19.1.1.7 is now associated with the interface ports of SE Avi-se-czpey and Avi-se-pnjbb as well.
root@dc11-cfg-1:~# neutron port-list
+--------------------------------------+------------+-------------------+---------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------------+-------------------+---------------------------------------------------------------------------------+
| dc4695c6-6d16-4ee0-ab05-774537a33ab2 | Avi-Data...| 02:dc:46:95:c6:6d | {"subnet_id": "a834986a-385e-4616-9d9c-6b91cfaa51e0", "ip_address": "19.1.1.6"} |
| | | | {"subnet_id": "a834986a-385e-4616-9d9c-6b91cfaa51e0", "ip_address": "19.1.1.7"} |
| 492ee72a-c647-490f-9648-77124e1cf0b6 | Avi-Data...| 02:49:2e:e7:2a:c6 | {"subnet_id": "a834986a-385e-4616-9d9c-6b91cfaa51e0", "ip_address": "19.1.1.4"} |
| | | | {"subnet_id": "a834986a-385e-4616-9d9c-6b91cfaa51e0", "ip_address": "19.1.1.7"} |
+--------------------------------------+------------+-------------------+---------------------------------------------------------------------------------+
In Contrail, the VIP 19.1.1.7 has the same linkage to ports dc4695c6-6d16-4ee0-ab05-774537a33ab2 and 492ee72a-c647-490f-9648-77124e1cf0b6, and two routes to the SE VM running on hypervisors 172.16.11.101 and 172.16.11.102.
If there are multiple SE instances for a VIP address, then the vRouter will receive multiple XMPP routes towards the final destination. The XMPP routes will have different route distinguishers (RDs) to keep them distinct, and they have different next-hops and MPLS labels to identify the different SE instances.
Flow Resiliency During Scale Out/In
The flow is 5-tuple: src-IP, src-port, dst-IP, dst-port, and protocol. Routers do a hash of the 5-tuple to pick a path to use. Contrail vRouter uses flow-tables to preserve the chosen SE destination for a flow. Thus, even when the number of SE instances in the ECMP group changes, a flow will never move.
Floating-IP ECMP
Using Avi CLI, associate a floating IP address with virtual service vs2 as follows:
configure virtualservice vs2
vip vip_id 0
auto_allocate_floating_ip
Overwriting the previously entered value for auto_allocate_floating_ip
floating_subnet_uuid 010a4d99-794c-42a9-948c-080014463217
save
save
+----------------------------------+-----------------------------------------------------+
| Field | Value |
+----------------------------------+-----------------------------------------------------+
| uuid | virtualservice-2f8770d0-20cc-477f-9419-b206b9624389 |
| name | vs2 |
| scaleout_ecmp | True |
| vip[1] | |
| vip_id | 0 |
| ip_address | 19.1.1.7 |
| enabled | True |
| network_ref | 02b846e6-cd68-4c84-b7ab-3709946ed28b |
| port_uuid | c0e4cb54-8706-4bed-97bb-271a603f858f |
| subnet_uuid | a834986a-385e-4616-9d9c-6b91cfaa51e0 |
| subnet | 19.1.1.0/24 |
| auto_allocate_ip | True |
| auto_allocate_floating_ip | False |
| floating_ip | 10.1.11.4 |
| auto_allocate_floating_ip | True |
| floating_subnet_uuid | 010a4d99-794c-42a9-948c-080014463217 |
...
+----------------------------------+-----------------------------------------------------+
In Contrail, the floating_ip is linked to the interface ports of both SEs and two routes to the SE VM running on hypervisors 172.16.11.101 and 172.16.11.102.
Virtual Service Traffic Stats
show virtualservice vs2 detail filter disable_aggregate se | grep connections_handled
| connections_handled | 102 |
| connections_handled | 49 |
Figure A. ECMP FIP linkage.
Figure B. ECMP FIP routing table.
Figure C. ECMP linkage.
Figure D. Non-ECMP linkage
Figure E. Non-ECMP route 1.
Figure F. ECMP route 2.