Mitigating DNS Slow Client Attack
Overview
DNS slow client attack is similar to HTTP slowloris attack, in which client sends DNS request over TCP byte by byte slowly to keep the connection alive, occupying the packet buffer. When done simultaneously over multiple TCP connections, this can result in a lack of memory resources to process legitimate requests as Service Engine buffers the data and waits for a complete DNS request on each connection.
Symptoms
The most common symptoms include too many opened TCP connections on the DNS virtual services and legitimate DNS Traffic stops working.
Events on Service Engines with code PKT_DROP_NO_PKT_BUFF are observed. Connections drops are observed for the specific virtual service due to a low packet buffer as show below.
A high number of TCP connections is observed in the connection_stats in tcpstats of the attacked DNS virtual service. To check the number of connections, monitor the values of connections_accepted and connection_established using the show virtualservice <virtual service name> tcpstat filter type frontend command. The dropped connections because of time exceeding the client_dns_tcp_request_timeout can be checked in tcps_dns_connection_closed_slow_client of the show virtualservice <dns-vs-name> tcpstat filter type frontend | grep -vw 0 command.
[admin:cntrlr]: > show virtualservice vs-try tcpstat filter type frontend | grep -vw 0
+----------------------------------------------+------------------------------------+
| Field | Value |
+----------------------------------------------+------------------------------------+
| se_uuid | Service-Engine:se-0000006f0ba7 |
| proc_id | PROC_Aggregate |
| connection_stats | |
| connections_accepted | 101 |
| connections_established | 101 |
| connections_closed | 101 |
| rx_stats | |
| total_packets_received | 24911 |
| packets_received_in_sequence | 24709 |
| bytes_received_in_sequence | 1669642 |
| ack_packets_received | 101 |
| ack_byte_received | 101 |
| tx_stats | |
| total_packets_sent | 12803 |
| data_bytes_sent | 843786 |
| ack_only_packet | 12702 |
| retransmit_stats | |
| dup_ack_packets_received | 1 |
| syncache_stats | |
| entry_added_to_syncache | 101 |
| entry_completed | 101 |
| connections_dropped | |
| connections_dropped_after_established | 101 |
| num_resets_sent | 101 |
| packets_dropped | |
| timeout | |
| sack_stats | |
| ecn_stats | |
| misc_stats | |
| rtt_updated | 202 |
| times_hdr_predict_ok_for_data_pkts | 24708 |
| connections_using_auto_gateway | 101 |
| dns_stats | |
| query_others | 100 |
| local_responses | 100 |
| tcps_dns_connection_closed_slow_client | 100 |
+----------------------------------------------+------------------------------------+
Mitigating DNS Slow Client Attack
There is a client_dns_tcp_request_timeout parameter associated with each open TCP (DNS) connection. If client fails to transmit complete DNS request within the configured timeout limit, the connection will be closed by the Service Engine. The timeout is configurable and can be set in the application profile of a virtual service. Connections are closed aggressively when the timeout value is set to lower values, and the connections last for a longer time when the large timeout values are configured.
Log in to NSX Advanced Load Balancer CLI and use the configure applicationprofile <application profile name> command to set the desired timeout values for the DNS service profile, as shown below.
[admin:cntrlr]: > configure applicationprofile System-DNS
Updating an existing object. Currently, the object is:
+--------------------------------------+---------------------------------------------------------+
| Field | Value |
+--------------------------------------+---------------------------------------------------------+
| uuid | applicationprofile-ea8c65d4-ad02-4b68-b8a9-d1b1186f42be |
| name | System-DNS |
| type | APPLICATION_PROFILE_TYPE_DNS |
| dns_service_profile | |
| num_dns_ip | 1 |
| ttl | 30 sec |
| error_response | DNS_ERROR_RESPONSE_NONE |
| edns | True |
| dns_over_tcp_enabled | True |
| aaaa_empty_response | True |
| negative_caching_ttl | 30 sec |
| admin_email | hostmaster |
| close_tcp_connection_post_response | False |
| client_dns_tcp_request_timeout | 10000 milliseconds |
| ecs_stripping_enabled | True |
| preserve_client_ip | False |
| preserve_client_port | False |
| preserve_dest_ip_port | False |
| tenant_ref | admin |
+--------------------------------------+---------------------------------------------------------+
[admin:cntrlr]: applicationprofile> dns_service_profile
[admin:cntrlr]: applicationprofile:dns_service_profile> client_dns_tcp_request_timeout 100
Overwriting the previously entered value for client_dns_tcp_request_timeout
[admin:cntrlr]: applicationprofile:dns_service_profile> save
[admin:cntrlr]: applicationprofile> save
+--------------------------------------+---------------------------------------------------------+
| Field | Value |
+--------------------------------------+---------------------------------------------------------+
| uuid | applicationprofile-ea8c65d4-ad02-4b68-b8a9-d1b1186f42be |
| name | System-DNS |
| type | APPLICATION_PROFILE_TYPE_DNS |
| dns_service_profile | |
| num_dns_ip | 1 |
| ttl | 30 sec |
| error_response | DNS_ERROR_RESPONSE_NONE |
| edns | True |
| dns_over_tcp_enabled | True |
| aaaa_empty_response | True |
| negative_caching_ttl | 30 sec |
| admin_email | hostmaster |
| close_tcp_connection_post_response | False |
| client_dns_tcp_request_timeout | 100 milliseconds |
| ecs_stripping_enabled | True |
| preserve_client_ip | False |
| preserve_client_port | False |
| preserve_dest_ip_port | False |
| tenant_ref | admin |
+--------------------------------------+---------------------------------------------------------+
[admin:cntrlr]: >