Today I Was Asked:

“When a Cisco ASA throws a ‘500003’ syslog message, was the packet forwarded or dropped?”

For today’s TIWA I wasn’t sure of the answer. I needed to do some research.

I built a lab, looked at PCAPs and studied the documentation. Here’s what I found.

What is syslog message 500003?

Cisco Reference: https://www.cisco.com/c/en/us/td/docs/security/asa/syslog/b_syslog/syslogs5.html#con_4773942

In Cisco’s explanation of the error message it isn’t clear on whether or not the packet was forwarded or dropped after the syslog message was thrown. I found myself needing to answer some questions:

• I could just assume that a packet with an INVALID TCP header would be dropped but, how could I prove it?
• What evidence would I need to gather to prove it was dropped or forwarded?
• What would I need to do/build to test this out?
• What if this was a clever evasion strategy from a nation state threat actor and someone was leaning on me to positively say “dropped” or “forwarded”?

…this was something I just couldn’t let go. I needed to know the answer.

The Cisco documentation provided some clues:

• TCP RST while responding to a connection request [TCP SYN] to a disabled socket
• Some operating systems send incorrect TCP header lengths
• TCP header length may indicate it is larger than the packet length

Inside the TCP header is a field that, when multiplied by 4, gives the length of the TCP headerin bytes. This is called the “Data Offset” becasue it’s the measurement of bytes between the end of the Layer-3 header and the start of the payload.

Reference: RFC 793: https://tools.ietf.org/html/rfc793#section-3.1

  Data Offset:  4 bits

    The number of 32 bit words in the TCP Header.  This indicates where
    the data begins.  The TCP header (even one including options) is an
    integral number of 32 bits long.

By default the TCP header is 20-bytes. Therefore, the TCP header length field would be “0x5” (as mentioned above, the header length value is multiplied by 4).

This field is interesting because it is only 4-bits long which means it can only hold decimal values of “0” thru “15”. These values multiplied by 4 are: 0,4,8,12,16,20,24,28,32,36,40,44,48,52,56 & 60. In this list only values greater than or equal to “20” are valid. This also means that we could have a TCP header that is 60 bytes in length according to the possible field values.

Question: What would happen if we artificially inflated or deflated that value, when the actual TCP header remains 20-bytes long?

“When in doubt, lab it out!” - Tony E.

Experiment #1: Incorrect TCP Header length in the SYN packet.

Iterate through the possible TCP Header length values (0 - 15) sending one packet each from the client to the server.

• Capture the packets leaving the Client.
• Observe the syslog messages from the ASA (if there are any)
• Does the ASA forward these packets to the server?
• Capture incoming packets on the server to verify their reciept.

Experiment #2: Incorrect TCP Header length in the RST packet.

From the Client, initiate a valid TCP SYN through the ASA toward the Server. On the Server use scapy to craft automatic reponses. Respond to the TCP SYN with an ACK/RST. Iterate through the 16 possible TCP Header length values.

• Capture the packets leaving the Client.
• Capture incoming packets on the server to verify their reciept.
• Capture the return packet (ACK/RST) from the Server.
• Observe the syslog messages from the ASA (if there are any)
• Does the ASA forward these packets to the Client?
• Capture packets on the Client to verify whether ot not the packets are received.

In both experiments the client is a Windows 10, Surface Pro 5 and the server is an Ubuntu 16.04 VM running in VMware ESXi.

Experiment #1: Incorrect TCP Header length in the SYN packet.

Using scapy, I sent TCP-SYN’s from the “inside” through the “outside” of the ASA to a closed socket on a server. I sent the TCP-SYN starting with a TCP Header length field of “0” and increased by “1” for each subsequent TCP-SYN.

# Malformed TCP Header (Incorrect Offset/Header Length)

>>> a = IP(dst="192.168.1.251")/TCP(sport=666,dport=777,dataofs=0,flags="S")
>>> sr(a)

Wireshark calls this field the “TCP Header”, other references call it the “Data Offset”. This is a 4-bit field and so can only contain the decimal values between 0-15. To compute the offset or header length in bytes this value is multiplied by 4.

• Data Offset value of 0 means 0 bytes of Offset becasue 0 x 4 = 0.
• Data Offset value of 5 means 20 bytes of Offset becasue 5 x 4 = 20. (This is the default TCP header length without any TCP Options.)
• Data Offset value of 8 means 32 bytes of Offset becasue 8 x 4 = 32.

If you set the Offset value to anything less than “5” Wireshark will tell you: Expert Info (Error/Protocol): Bogus TCP header length (0, must be at least 20)

NOTE: These are NOT TCP Retransmissions. They are labelled that way by Wireshark because the reuse of the ip.id field.

By capturing at the source and the destination you can see the only TCP SYN to get through the ASA has a valid header length value of 0x5 or 20-bytes. Because our server wasn’t listening on the destination port, the server responded with a TCP-RST, terminating the connection. The logs from the Cisco ASA corroborate the packet capture.

Experiment #1: Cisco ASA Logs

The below logs indicate the error was thrown by “from 10.123.123.101” (our client) as the source. Also you’ll notice no error when the header length equals 20 “(hdrlen=20…”. This is becasue our packets had a TCP header length of 20, this was a valid packet and didn’t generate an error.

%ASA-5-500003: Bad TCP hdr length (hdrlen=0, pktlen=58) from 10.123.123.101/0 to 192.168.1.251/0, flags: INVALID, on interface inside
%ASA-5-500003: Bad TCP hdr length (hdrlen=4, pktlen=58) from 10.123.123.101/666 to 192.168.1.251/777, flags: INVALID, on interface inside
%ASA-5-500003: Bad TCP hdr length (hdrlen=8, pktlen=58) from 10.123.123.101/666 to 192.168.1.251/777, flags: INVALID, on interface inside
%ASA-5-500003: Bad TCP hdr length (hdrlen=12, pktlen=58) from 10.123.123.101/666 to 192.168.1.251/777, flags: INVALID, on interface inside
%ASA-5-500003: Bad TCP hdr length (hdrlen=16, pktlen=58) from 10.123.123.101/666 to 192.168.1.251/777, flags: SYN , on interface inside

%ASA-5-500003: Bad TCP hdr length (hdrlen=24, pktlen=58) from 10.123.123.101/666 to 192.168.1.251/777, flags: SYN , on interface inside
%ASA-5-500003: Bad TCP hdr length (hdrlen=28, pktlen=58) from 10.123.123.101/666 to 192.168.1.251/777, flags: SYN , on interface inside
%ASA-5-500003: Bad TCP hdr length (hdrlen=32, pktlen=58) from 10.123.123.101/666 to 192.168.1.251/777, flags: SYN , on interface inside
%ASA-5-500003: Bad TCP hdr length (hdrlen=36, pktlen=58) from 10.123.123.101/666 to 192.168.1.251/777, flags: SYN , on interface inside
%ASA-5-500003: Bad TCP hdr length (hdrlen=40, pktlen=58) from 10.123.123.101/666 to 192.168.1.251/777, flags: SYN , on interface inside
%ASA-5-500003: Bad TCP hdr length (hdrlen=44, pktlen=58) from 10.123.123.101/666 to 192.168.1.251/777, flags: SYN , on interface inside
%ASA-5-500003: Bad TCP hdr length (hdrlen=48, pktlen=58) from 10.123.123.101/666 to 192.168.1.251/777, flags: SYN , on interface inside
%ASA-5-500003: Bad TCP hdr length (hdrlen=52, pktlen=58) from 10.123.123.101/666 to 192.168.1.251/777, flags: SYN , on interface inside
%ASA-5-500003: Bad TCP hdr length (hdrlen=56, pktlen=58) from 10.123.123.101/666 to 192.168.1.251/777, flags: SYN , on interface inside
%ASA-5-500003: Bad TCP hdr length (hdrlen=60, pktlen=58) from 10.123.123.101/666 to 192.168.1.251/777, flags: SYN , on interface inside

This was a good test. It validates my testing architecture, scripts and environment are working as designed. Now, to being answering the question we’re here for…

Experiment #2: Incorrect TCP Header length in the RST packet.

This time we will send valid TCP-SYN’s from the client through the ASA to the Server and on the server we will respond with TCP-RST with incorrect TCP header lengths. Again, I will use scapy to send the SYN from the client but this time we’ll also use scapy on the server to create an answering machine which automatically responds with a TCP ACK/RST using data offset values we define.

Scapy Answering Machine With Incorrect TCP Header Length

Reference: https://stackoverflow.com/questions/24415464/scapy-sending-receiving-and-responding

def rst(p):
    ans = IP(src=p[IP].dst, dst=p[IP].src)/TCP(
        flags='RA',
        sport=p[TCP].dport,
        dport=p[TCP].sport,
        seq = 0,
        ack = p[TCP].seq + 1,
        dataofs=8
    )
    send(ans, verbose=False)
    return "%s\n => %s" % (p[IP].summary(), ans.summary())
    
sniff(iface="eth0", filter="tcp and tcp[tcpflags] & tcp-syn == tcp-syn",prn=rst)

Experiment #2: Client side PCAP

In this PCAP you can see the client sending 16 TCP-SYN packets with valid header lengths. The Client only recieves 1 reply from the server.

NOTE: These are NOT TCP Retransmissions. They are labelled that way by Wireshark because the reuse of the ip.id field.

Experiment #2: Server side PCAP

In this server side PCAP you can see the server does respond to every request received from the client and responds with an invalid or malformed TCP ACK/RST. Again, I started at value of “0” and increased by 1 for every iteration.

NOTE: These are NOT TCP Retransmissions. They are labelled that way by Wireshark because the reuse of the ip.id field.

Experiment #2: Cisco ASA Logs

The below Cisco ASA logs would corroborate what we are seeing from the packet captures. This time the errors are thrown specifiying “from 192.168.1.251” (the server) as the source. You don’t see an error message for “(hdrlen=20…” becasue our headers were 20 bytes and they were valid.

%ASA-5-500003: Bad TCP hdr length (hdrlen=0, pktlen=58) from 192.168.1.251/0 to 10.123.123.101/0, flags: INVALID, on interface outside
%ASA-5-500003: Bad TCP hdr length (hdrlen=4, pktlen=58) from 192.168.1.251/777 to 10.123.123.101/666, flags: INVALID, on interface outside
%ASA-5-500003: Bad TCP hdr length (hdrlen=8, pktlen=58) from 192.168.1.251/777 to 10.123.123.101/666, flags: INVALID, on interface outside
%ASA-5-500003: Bad TCP hdr length (hdrlen=12, pktlen=58) from 192.168.1.251/777 to 10.123.123.101/666, flags: INVALID, on interface outside
%ASA-5-500003: Bad TCP hdr length (hdrlen=16, pktlen=58) from 192.168.1.251/777 to 10.123.123.101/666, flags: RST ACK , on interface outside

%ASA-5-500003: Bad TCP hdr length (hdrlen=24, pktlen=58) from 192.168.1.251/777 to 10.123.123.101/666, flags: RST ACK , on interface outside
%ASA-5-500003: Bad TCP hdr length (hdrlen=28, pktlen=58) from 192.168.1.251/777 to 10.123.123.101/666, flags: RST ACK , on interface outside
%ASA-5-500003: Bad TCP hdr length (hdrlen=32, pktlen=58) from 192.168.1.251/777 to 10.123.123.101/666, flags: RST ACK , on interface outside
%ASA-5-500003: Bad TCP hdr length (hdrlen=36, pktlen=58) from 192.168.1.251/777 to 10.123.123.101/666, flags: RST ACK , on interface outside
%ASA-5-500003: Bad TCP hdr length (hdrlen=40, pktlen=58) from 192.168.1.251/777 to 10.123.123.101/666, flags: RST ACK , on interface outside
%ASA-5-500003: Bad TCP hdr length (hdrlen=44, pktlen=58) from 192.168.1.251/777 to 10.123.123.101/666, flags: RST ACK , on interface outside
%ASA-5-500003: Bad TCP hdr length (hdrlen=48, pktlen=58) from 192.168.1.251/777 to 10.123.123.101/666, flags: RST ACK , on interface outside
%ASA-5-500003: Bad TCP hdr length (hdrlen=52, pktlen=58) from 192.168.1.251/777 to 10.123.123.101/666, flags: RST ACK , on interface outside
%ASA-5-500003: Bad TCP hdr length (hdrlen=56, pktlen=58) from 192.168.1.251/777 to 10.123.123.101/666, flags: RST ACK , on interface outside
%ASA-5-500003: Bad TCP hdr length (hdrlen=60, pktlen=58) from 192.168.1.251/777 to 10.123.123.101/666, flags: RST ACK , on interface outside

Conclusion

Question:

When a Cisco ASA throws a “500003” syslog message, was the packet forwarded or dropped?

Answer:

From my analysis, in my testing environment, I observed the Cisco ASA 5505 running code “ASA Version 9.2(4)” dropped packets with incorrect TCP header lengths whether the value was under or above the actual TCP header length. This causes the ASA to throw an error message: %ASA-5-500003 and drop the packet.

Appendix: Cisco ASA Baseline Config for Reference

ciscoasa# sho run
: Saved
:
: Serial Number: 
: Hardware:   ASA5505, 512 MB RAM, CPU Geode 500 MHz
:
ASA Version 9.2(4)
!
hostname ciscoasa
enable password 8Ry2YjIyt7RRXU24 encrypted
xlate per-session deny tcp any4 any4
xlate per-session deny tcp any4 any6
xlate per-session deny tcp any6 any4
xlate per-session deny tcp any6 any6
xlate per-session deny udp any4 any4 eq domain
xlate per-session deny udp any4 any6 eq domain
xlate per-session deny udp any6 any4 eq domain
xlate per-session deny udp any6 any6 eq domain
passwd 2KFQnbNIdI.2KYOU encrypted
names
!
interface Ethernet0/0
 switchport access vlan 5
!
interface Ethernet0/1
 switchport access vlan 10

[ ... omitted for brevity ... ]

interface Vlan5
 nameif outside
 security-level 0
 ip address dhcp setroute
!
interface Vlan10
 nameif inside
 security-level 100
 ip address 10.123.123.1 255.255.255.0
!
boot system disk0:/asa924-k8.bin
ftp mode passive
pager lines 24
logging enable
logging console notifications
logging monitor notifications
logging buffered informational
mtu outside 1500
mtu inside 1500
icmp unreachable rate-limit 1 burst-size 1
no asdm history enable
arp timeout 14400
no arp permit-nonconnected
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
user-identity default-domain LOCAL
no snmp-server location
no snmp-server contact
crypto ipsec security-association pmtu-aging infinite
crypto ca trustpool policy
telnet timeout 5
ssh stricthostkeycheck
ssh timeout 5
ssh key-exchange group dh-group1-sha1
console timeout 0

dhcpd address 10.123.123.100-10.123.123.150 inside
dhcpd enable inside
!
threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
!
class-map inspection_default
 match default-inspection-traffic
!
!
policy-map global_policy
 class inspection_default
  inspect icmp
!
service-policy global_policy global
prompt hostname context
no call-home reporting anonymous
Cryptochecksum:7ee497ef016ac635c23ce2bd9980ee74
: end