Collecting Tech Support Logs in Avi Vantage

Overview

Tech support logs are used for offline troubleshooting. This article describes how to collect tech support logs from a single Avi Controller or 3-node Avi Controller cluster. If you are running Avi Vantage 18.2.4 or lower, refer instead to this article.

A complete list of tech support log types are listed in the last section of this article. Support personnel most often request the debuglogs, serviceengine, or virtualservice types.

Note: In a 3-node Avi Controller cluster, the steps mentioned in this article automatically collect logs from all the three nodes. The commands need not be entered separately on each Avi Controller node.

Collecting Tech Support Logs using the Avi UI

Follow the instructions in this section to collect tech support logs using the Avi UI.

  1. Navigate to Administration > System > Tech Support and click on Generate Tech Support to begin.

    Administration > System > Tech Support

  2. The below screen will appear. It will reveal just one Type.

    Initial generate tech support screen

  3. Click on the drop-down icon at the right end of the Type field to reveal part of a list of choices for Type. Scroll for more choices.

    part of the drop-down

  4. Some types require no further specification; they are Clustering, Metrics Logs, Debug Logs, Placement, Portal, and Upgrade. For example, if Placement is chosen, a pair of messages will appear in succession, as below:

    First wait message

    Second wait message

    The image below illustrates how a set of tech support logs accumulated over time might appear in the table.

    Table of stored tech support files

    The image below shows some collections can complete with errors, details of which are revealed if the + (plus) sign within the row at extreme right is clicked.

    Table of stored tech support files

  5. After the support log archive has been created, a corresponding row will be added to the window’s table. Clicking the plus (+) sign at the right of a particular row yields an expanded view of the support log that includes the file name of the archive.

    Row created for the Placement log file

  6. Three options — trash the archive, download the archive, and expand/collapse the row entry — can be selected by choosing one of three icons at the right of the row.

  7. Some types do require further specification; they are GSLB, Pool, Virtual Service, and Service Engine. For example, if the Service Engine type is selected, the below screen appears. Note that the Cloud Selection and Service Engine Instance fields appear, the latter being mandatory.

    Service Engine type log selected

  8. Downloaded .tar.gz files should be conveyed to Avi Support for further troubleshooting.

Collecting Tech Support Logs using the Avi CLI

Follow the instructions in this section to collect tech support logs using the Avi CLI.

  • In an Avi Controller cluster, one of the nodes is the leader or the primary node. Login to the leader node using any ssh client, such as putty. 10.10.1.1 is the leader node in this example.

login as: admin

Avi Cloud Controller

Avi Networks software, Copyright (C) 2013-2017 by Avi Networks, Inc.
All rights reserved.

Version:      17.1.6
Date:         2017-08-30 15:35:21 UTC
Build:        9022
Management:   10.10.2.2/22                UP
Gateway:      10.10.2.1                   UP

admin@10.10.1.1's password:

To know more about the leader node in an Avi Controller cluster, refer to Overview of the Controller Cluster. To know more about SSH key generation, refer to Avi CLI access.

  • Type shell command, and enter your credentials.

admin@avi-cntrl1:~$ shell
Login: admin
Password:

  • From the shell prompt, run the command show tech-support serviceengine.

    The command show tech-support serviceengine create debug logs for the particular SE.

    Note: For reference, Avi-cluster1-SE1 is the Avi SE for which we want to collect the debug logs.


[admin:avi-cntrl1]: > show tech-support serviceengine Avi-cluster1-SE1
  • This command generates core files and other debug logs, which are stored in /var/lib/avi/tech_support directory on the Avi Controller leader node. Use cd command from the Avi CLI to get access to the directory. Check the debug logs and the other log files, which are available in compressed form with tar.gz extension.

Sample tech support log file: debuglogs.20170915-124030.tar.gz The number 20170915 in the log file debuglogs.20170915-124030.tar.gz represents the date of generation of the log files, i.e., 15 Sep 2017.

The following are the different options of tech support:

  • clustering — If you want to debug any clustering related issues, then you need to collect clustering tech support, as it collects clustering related files, for instance, cluster.conf, zookeeper, journal logs.
    show tech-support clustering is the CLI command for clustering tech support.

  • metricslogs — If you want to debug issue with metrics manager, then you can collect metricslogs techsupport.
    show tech-support metricslogs is the CLI command for metriclogs tech support.

  • debuglogs — This is the most used type for collecting tech support which has logs from cluster nodes. You can debug issues like upgrade, events, web-app, log-rotation, log core manager, redis, and so on with this tech support.
    show tech-support debuglogs is the CLI command for debuglogs tech support.

  • placement — If you want to debug any issue related to placement, then you can collect placement tech support. It collects logs like vs_mgr, res_mgr etc.
    show tech-support placement is the CLI command for placement tech support.”

  • portal — If you want to debug any portal or webapp related issues, then you can collect portal tech support. It collects logs like portal-webapp, portal_exception, and so on.
    show tech-support portal is the CLI command for portal tech support.

  • serviceengine — If you face any issue regarding specific Service Engine, then you can collect serviceengine tech support to debug that issue as it will collect all required logs from that particular Service Engine.
    show tech-support serviceengine <serviceengine-name> is the CLI command for serviceengine tech support.

  • upgrade — If you have any upgrade related issue, you can collect upgrade tech support.
    show tech-support upgrade is the CLI command for upgrade tech support.

  • virtualservice — If you have issue with any specific virtual service and need to debug it, then you can collect this tech support which has that specific virtualservice logs along with the serviceengine logs where that virtualservice is attached to.
    show tech-support virtualservice <virtualservice-name> is the CLI command for virtualservice tech support.

  • gslb — If you need to debug any issue related to gslb then you can collect gslb techsupport, which has gslb related logs only.
    show tech-support gslb is the CLI command for gslb tech support.

  • pool — If you need to debug any issue with specific pool in the system then you can collect tech support to that specific pool which has CLI commands output to that specific pool and vs_mgr logs as well.
    show tech-support pool <pool-name> is the CLI command for pool tech support.

Collecting Tech Support Logs using Avi API

Execute the following command to collect tech support logs using REST API.

GET https:///api/techsupport/debuglogs

Collecting Tech Support Logs using Tech Support API v2

Starting with Avi Vantage release 18.2.2, API version 2 (v2 API) is available for collecting tech support logs.

Notes:

  • Tech support v2 API is not blocking in nature. The v1 API is blocking in nature. The v1 API blocks the access of the device until the tech support collection completes. On average, the tech support collection takes more than five to six minutes. This duration is dependent on the number of cluster nodes, number of log files, bandwidth, etc. Because of this duration, the CLI timeout and timeout intervals are increased to capture tech support logs successfully using v1 API.

  • v1 APIs are not deprecated. You can still use the blocking API (VI API) in your scripts as required.

The following are few of the API methods which are used to collect tech support logs:

  • Use the following API call to collect tech support logs.

GET api/techsupportv2/debuglogs
  • Use the following API call to collect logs tech support logs for a specific SE instance.

api/techsupportv2/serviceengine/<SE Id>
  • Use the following API call to check the status of tech support collection. The following API indicates the path for tech support tarball (files)and the current status.

GET api/techsupportstatusv2
  • Use the following API call to download the tech support files.

GET api/fileservice?uri=controller://tech_support

Additional Information

The following command collects the debug logs on each of the Avi Controller nodes. This option keeps the collected tech support bundle to a minimal size. The option is especially useful if the core archive bundles that are present have already been uploaded and fixed.

show tech-support debuglogs

Collecting Tech Support Logs via Script on the Controller

You can upload logs via attach2case.py script on the Controller. Before running the script, ensure:

  • A per-user or per-account token for API authentication
  • Forward proxy on the network which the Controller is part of
  • Super user access to the Controller

Note: The attach2case.py script is found under /opt/avi/scripts directory.

The CLI format is as follows:

User@Test-MacBook-Pro:~$ ssh admin@10.10.22.117

Avi Cloud Controller

Avi Networks software, Copyright (C) 2013-2017 by Avi Networks, Inc.
All rights reserved. 

Version:      17.1.8
Date:         2017-09-21 06:03:07 UTC
Build:        9020
Management:   10.10.22.62/23                 UP 
Gateway:      10.10.22.1                     UP 

admin@10.10.22.117's password: 

The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
Last login: Fri Oct  6 07:01:03 2017 from 10.10.221.61
admin@Test-Cntl:~$ 
admin@Test-Cntl:~$ cd /opt/avi/scripts
admin@Test-Cntl:/opt/avi/scripts$ pwd
/opt/avi/scripts
admin@Test-Cntl:/opt/avi/scripts$ ls -l attach2case.py 
rwxrwx-- 1 root root 14187 Sep 21 06:03 attach2case.py

root@Test-Cntl:/opt/avi/scripts# ./attach2case.py --help
usage: attach2case.py [-h] [-c CONFIG] [-d] [-H HOSTNAME] [-i]
[-p {http,https}] [-P] [-t TOKEN]
CASE-NUMBER FILE [FILE ...] 

Attaches files to a case on the Avi Networks portal.

positional arguments:
CASE-NUMBER           case number to attach files to
FILE                  files to attach to the case

optional arguments:
-h, --help            show this help message and exit
-c CONFIG, --config CONFIG
Path to configuration file (JSON) (default: None)
-d, --debug           Enable HTTP debug (default: False)
-H HOSTNAME, --hostname HOSTNAME
HTTP hostname (default: avinetworks.com)
-i, --insecure        Allow insecure SSL connections (default: False)
-p {http,https}, --protocol {http,https}
HTTP protocol (default: https)
-P, --progress        Display progress indicator (default: False)
-t TOKEN, --token TOKEN
API authentication token

The script can be executed as follows:

/opt/avi/scripts/attach2case.py -H avinetworks.com -t  -P -p https <case#>                       
</code></pre>

**Example**



admin@Test-Cntl:/opt/avi/scripts$ sudo su
[sudo] password for admin: 
root@Test-Cntl:/opt/avi/scripts# ls -ltrh /home/admin/test
total 5.4G
-rwxrwxrwx 1 admin admin 3.5G Oct  6 06:44 test.tar.gz
-rwxrwxrwx 1 admin admin 2.0G Oct  6 06:47 controller.pkg

root@Test-Cntl:/opt/avi/scripts# ./attach2case.py -H avinetworks.com -t  -P -p https 3763 /home/admin/test/controller.pkg 
100%
root@Test-Cntl:/opt/avi/scripts# ./attach2case.py -H avinetworks.com -t  -P -p https 3763 /home/admin/test/test.tar.gz
100% 
Attaches files to a case on the Avi Networks portal.  </code></pre>

## What Data Are Collected in Tech Support Logs

As the Avi Vantage system has evolved functionally over time, so too have the number of tech support log _types_ and the data captured in each. Ten tech support types appear below. The debuglogs type captures Controller-specific debugging data; all other types capture data associated with their self-explanatory name.


    

  1. clustering
    2. 

  2. metricslogs
    3. 

  3. debuglogs
    4. 

  4. placement
    5. 

  5. portal
    6. 

  6. serviceengine
    7. 

  7. upgrade
    8. 

  8. virtualservice
    9. 

  9. gslb
    10. 

  10. pool
    11. 



The logs collected through the shell command show tech-support debuglogs contains all the relevant Controller logs from the previous and current partition from all the nodes in the cluster; this command is a superset of other show tech-support [clustering, upgrade, placement, portal, metricslogs, gslb] commands.

The two exceptions are

1. show tech-support virtualservice </code> collects the additional show-command output listed below for a given virtual service, along with SE logs for SEs onto which the virtual service has been placed, and the Controller logs from all the nodes in the current partition.

* show virtualservice
* show virtualservice <key>
* show virtualservice <key> detail
* show virtualservice <key> tcpstat
* show virtualservice <key> dnstable
* show virtualservice <key> internal
* show virtualservice <key> httpstats
* show virtualservice <key> dosstat
* show shardserver

2. show tech-support pool </code> collects the additional show-command output listed below for a given pool, along with SE logs for SEs onto which the virtual service has been placed, and the vs_mgr logs from all the Controller nodes in the current partition.

* show pool
* show pool <key> algo
* show pool <key> connpool
* show pool <key> connpool filter disable_aggregate se
* show pool <key> debug - show pool <key> detail
* show pool <key> detail filter disable_aggregate se
* show pool <key> internal
* show pool <key> internal filter disable_aggregate se
* show pool <key> httpcache
* show pool <key> httpcache filter disable_aggregate se
* show pool <key> httpcachestats
* show pool <key> httpcachestats filter disable_aggregate se
* show pool <key> persistence
* show pool <key> persistence filter disable_aggregate se
* show pool <key> hmon
* show pool <key> summary
* show pool <key> vs
* show pool <key> server detail
* show pool <key> server detail filter disable_aggregate se
* show pool <key> server summary
* show pool <key> server internal
* show pool <key> server internal filter disable_aggregate se
* show pool <key> server hmonstat
* show pool <key> server hmonstat filter disable_aggregate se

If it is clear to the system administrator that a problem is associated with a particular virtual service, then capturing the virtualservice tech support log specific to that VS is best. Otherwise, the combination of debuglogs and serviceengine tech support logs is generally best.