Troubleshooting Packet Latencies within SE

Overview

SE time flow tracker can track the network characteristics, processing time at key checkpoints and flag queuing delays in a packet journey through the network appliance.

CLI

Configuring Analytics Profile

The following are the configuration used in analytics profile:

> show analyticsprofile System-Analytics-Profile
..
| latency_audit_props                 |                           |
|   latency_audit_mode                | LATENCY_AUDIT_OFF         |
|   latency_threshold                 | 20 milliseconds           |
|   conn_est_audit_mode               | LATENCY_AUDIT_ON          |
|   conn_est_threshold                | 40 milliseconds           |
+-------------------------------------+---------------------------+

Audit Properties	Default	Description
latency_audit_mode	LATENCY_AUDIT_OFF	LATENCY_AUDIT_OFF - Default, no latency audit is performed. LATENCY_AUDIT_ON - Turn on the latency audit with statistics/ counters for flows/ packets breaching the configured threshold. LATENCY_AUDIT_ON_WITH_SIG - Turn on the latency audit, statistics are updated along with event and significant logs.
latency_threshold	20 msec	This enables tracking the dispatcher to proxy latency for each packet if latency_audit_mode is set to LATENCY_AUDIT_ON. This is the threshold above which events, significant logs and metrics are expressed if the per packet latency from dispatcher to proxy is too high.
conn_est_audit_mode	LATENCY_AUDIT_ON	LATENCY_AUDIT_OFF -No connection establishment audit is performed. LATENCY_AUDIT_ON - Default, turn on the connection establishment audit with statistics/ counters for flows/ packets breaching the configured threshold. LATENCY_AUDIT_ON_WITH_SIG - Turn on the connection establishment audit, statistics are updated along with event and significant logs.
conn_est_threshold	40 msec	This enables tracking the TCP connection establishment time if conn_est_audit_mode is set to LATENCY_AUDIT_ON. This is the threshold for anomaly detection which is expressed as events, significant logs and metrics if this threshold is breached.se

Note: Currently, latency_audit_filters is supported only for TCP/IPV4.

Configuring latency_audit_filters in debug Virtual Service

The filters contain all the options offered by VS capture filters. However, latency_audit_filters are functionally independent of capture filters.

> debug virtualservice vs-1
..
[admin:vpr-ctrl1]: debugvirtualservice:latency_audit_filters>
cancel               Exit the current submode without saving
capture_ip           (submode)
capture_ipc          (submode)
do                   Execute a show command
dst_port_end         Destination Port range filter.
dst_port_start       Destination Port range filter.
eth_proto            Ethernet Proto filter.
ip_proto             IP Proto filter. Support for TCP only for now.
new                  (Editor Mode) Create new object in editor mode
no                   Remove field
save                 Save and exit the current submode
show_schema          show object schema
src_port             Source Port filter.
src_port_range_end   Source Port range end filter. If specified, the source port filter will be a range. The filter range will be between src_port and src_port_range_end.
tcp_ack              TCP ACK flag filter.
tcp_fin              TCP FIN flag filter.
tcp_push             TCP PUSH flag filter.
tcp_syn              TCP SYN flag filter.
watch                Watch a given show command
where                Display the in-progress object

Metrics and Logs

The framework supports metrics, events and logs. These are configurable.

Metrics at SE level

Metrics at VS level

Events

Note: The threshold is set to 0 in this example.

Significant Logs (When Latency_Audit is enabled)

The detailed timing and flow characteristics will be present in Connection/App Log.