Advanced troubleshooting by sniffing packets (packet capture)

Home » Fortinet Fortigate » Advanced troubleshooting by sniffing packets (packet capture)
Fortinet Fortigate Nenhum Comentário
How do I use filters for sniffing? They are really confusing.
You can perform some basic packet sniffing and network troubleshooting without using packet sniffing filters. However, with filters, you can fine tune your troubleshooting to the point of being able to find a specific ping packet on a busy network.
When packet sniffing, the filter field is very flexible. By using the filter option, you can:
match the source hostname or IP address
match the type of packet (arp, ip, gre, esp, udp, tcp, icmp)
match the port number
logically AND or OR parts of the filter with each other
specify a certain byte in a packet
The default format of the filter syntax is:
[[src|dst] host <host_name_or_IP1>] [[arp|ip|gre|esp|udp|tcp|icmp] [port_no]] [and | or] [..]
Let’s look at each of the different parts to the filter. Keep in mind that in addition to these formats, you can also search for individual words using the filter. The following are examples.
IP matching with filters
Let’s look at the hostname and IP matching — [[src|dst] host <host_name_or_IP1>]. It allows you to specify either the source or destination host. For example if you want to sniff packets coming from IP address you would set the filter to ’src host’. If you want to sniff packets going to a computer called my_laptop, the filter would be ’dst host my_laptop’. This host name is resolved using DNS.
In each case, when the sniffer finds packets from that computer, the packets will match the filter and be displayed. You can enter two or more different computers using this format and join them with logical ANDs or ORs. For example, you could specify one source and two destinations.
In the following example, let’s assume a computer on the network is pinging the FortiGate unit. We will only be looking for ping packets with a source of which is the FortiGate unit.
diag sniffer packet any ‘icmp and src host′
filters=[icmp and src host]
0.319302 -> icmp: echo reply
1.348780 -> icmp: echo reply
2.355177 -> icmp: echo reply
3.356008 -> icmp: echo reply
4 packets received by filter
0 packets dropped by kernel
The result displays four packets, all ping (icmp) packets, originating from the FortiGate unit and going to This time there was no verbosity level indicated or number of packets. A default verbosity level 1 is used, and the sniffing continues until you press Ctrl-C to stop it. Note that the last two lines tell you how many packets were sniffed and if the FortiGate kernel dropped any packets during this time.
When the sniffing has ended, if you see anything but zero packets dropped, you may have a problem. Packets dropped indicates the FortiGate unit was not able to sniff and display all the packets that were coming in. If you were looking for all the packets in a sequence, there may well be packets missing. For this reason, you should consider possible reasons for those dropped packets, attempt to fix the problem so all packets are captured, and run the sniffer again. Keep in mind that the sniffer can take up to 25% of the CPU resources on smaller FortiGate units.
Sniffing a port and specifying multiple hosts using AND and OR operators
When a TCP session is created, the destination port is set to a known port number — for example, port 80 is commonly used for HTTP sessions. But the source port is randomly assigned. The unknown source port can make troubleshooting difficult. However, the FortiGate packet sniffer can match the known port if it is the source or destination port — you do not need to know which port.
Let’s check HTTP packets going between IP (the FortiGate) and on either (wifi interface called Star) or (internal LAN interface).
diag sniffer packet any “port 80 and host and (host or host” 4
filters=[port 80 and host and (host or host]
5.036340 internal in -> syn 4189154
5.036664 internal out -> syn 1354149395 ack 4189155
6.464015 Star out -> syn 2000204115 ack 571678006
6.471966 Star in -> ack 2000204116
6.474720 Star in -> psh 571678006 ack 2000204116
5.036837 internal in -> ack 1354149396
5.037023 internal in -> psh 4189155 ack 1354149396
6.463686 Star in -> syn 571678005
Since either the source or destination will be using port 80, all HTML traffic between those two computers will match the filter and be displayed. SSH and HTTPS traffic uses different ports, so that traffic will not be displayed. The first number of each line of output will vary between sources and is a good way to quickly determine which IP addresses are in that session.
Packet type filters
Let’s look at the packet type — [arp|ip|gre|esp|udp|tcp]. This determines what type of packets to look for. In addition to the common ICMP, IP, TCP, and UDP you can look for ARP (address resolution protocol), GRE (generic routing encapsulation), and ESP (encapsulating security payload) packets.
If the protocol you want isn’t listed here you can specify it if you know the ethernet protocol number for it. For example to specify ARP packets on the internal interface with this method: diag packet sniffer internal “ether proto 0x0806”
Let’s sniff some ARP packets from a gateway on the network at IP address For this we don’t care about the interface, and five packets will be enough to see what is happening.
# diag sniffer packet any ‘arp’ 1 5
1.187291 arp who-has tell
2.187125 arp who-has tell
2.858334 arp who-has tell
2.889542 arp who-has tell
4.187019 arp who-has tell
From this output, we can see ARP requests from a computer with IP address that is looking for the MAC address of a computer with the IP address In the ARP protocol, the who-has request is broadcast and includes the link layer address of where to send the reply. The expected response, when a computer has the IP address, will be in the format arp reply is at 00:26:b9:00:0f:9c. Since there is no such reply in the sniffed packets, we can either sniff more packets or assume there is no computer on the network with the IP address This may be important if a computer is supposed to be using that IP address and is not. It could imply DHCP problems, or that the computer was physically moved to a different part of the network.
ARP packets can be the source of problems if there is a network loop. As mentioned above, ARP tries to match a single MAC address to a single IP address. If the request results in two or more replies with the same IP address, or different IP addresses have the same MAC address, as may happen with virtual networking solutions, the loop or asymmetric routing is created. Essentially, all traffic will go to and from both computers. This will appear as a network slowdown or halt. You can see this happening if you are sniffing ARP packets and seeing the double replies or double MAC addresses. To confirm that this is the issue, enter the CLI command config system settings, set asymroute enable, end. This will turn on asymmetric routing, stop these ARP problems, and disable stateful inspection. Disabling stateful inspection will compromise security, so in most cases you should only use this command to confirm a problem. Once the problem is confirmed, use the sniffer output to find and fix the source and then disable asymmetric routing.
Miscellaneous advanced filters
There are some non-standard filters you can use to match traffic with the packet sniffer. These advanced filters use logical symbols to match specific bits within packet headers. Some examples are:
If you want to match TTL = 1 in the packet headers on port2:
# diagnose sniffer packet port2 “ip[8:1] = 0x01”
If you want to match packets with a source IP address of in the header:
# diagnose sniffer packet internal “(ether[26:4]=0xc0a80102)”
The source and destination information are stored in different places in the packet headers. If you want to match packets with a source MAC address of 00:09:0f:89:10:ea on the internal interface
# diagnose sniffer packet internal “(ether[6:4]=0x00090f89) and (ether[10:2]=0x10ea)”
where matching packets with the same MAC address as a destination MAC on the internal interface is
# diagnose sniffer packet internal “(ether[0:4]=0x00090f89) and (ether[4:2]=0x10ea)”
You can also target specific types of packets, such as addressing the TCP or UDP flags.
If you want to match packets with RST flag set:
# diagnose sniffer packet internal “tcp[13] & 4 != 0″
If you want to match packets with the SYN flag set:
# diagnose sniffer packet internal “tcp[13] & 2 != 0″
If you want to match packets with the SYN-ACK flag set:
# diagnose sniffer packet internal “tcp[13] = 18″
If your FortiGate unit has NP2 interfaces that are offloading traffic, this will change the sniffer trace. Before performing a trace on any NP2 interfaces, you should disable offloading on those interfaces.
Best practices
Here are some tips that will improve your troubleshooting using the packet sniffer.
Enabling the sniffer will consume additional CPU resources. This can be as high as an additional 25
percent of CPU usage on low-end models. Therefore, enabling this on a unit that is experiencing excessively high CPU usage, can only render the situation worse. If you must perform a sniff, keep the sniffing sessions short and keep the filter specific.
Try to always include ICMP in the sniffer filter. You may capture an ICMP error message that can help identify the cause of the problem. For example:
diag sniff packet interface wan1 ‘tcp port 3389 or icmp’ 3
Use the “any” interface to sniff all FortiGate unit interfaces. You can use the “any” interface if you want to confirm that a specific packet is sent and received by different FortiGate interfaces. The any interface is also useful if you are not sure which interface will send or receive the packet. An example using the “any” interface:
diag sniff packet any ‘tcp port 3389′ 3
The FortiGate unit may not display all packets if too much information is requested. When this occurs, the FortiGate unit will log the following message once the trace is terminated:
12151 packets received by filter
3264 packets dropped by kernel
When this occurs, it is possible that what you were attempting to capture, was not actually captured. In order to avoid this, try to make the filters more specific, reduce the verbosity level, or run the sniffer during a lower traffic period.
The packet timestamps, as displayed by the sniffer, may become skewed or delayed under high load conditions. This may occur even if no packets were dropped. Therefore, it is not recommended that you rely on these values in order to troubleshoot or measure performance issues that require absolute precise timing.
Short Ethernet frames sent by the FortiGate unit may appear to be under the minimum length of 64 bytes (also known as runts) and will not be displayed by the sniffer. This is because the sniffer does not display any Ethernet Trailer/Padding information, although it is sent over the network.
The Ethernet source and/or destination MAC addresses may be incorrect when using the “any” interface. They may be displayed as all zeros (00:00:00:00:00:00) or 00:00:00:00:00:01.
Try to always include ICMP in the sniffer filter. You may capture an ICMP error message that can help identify the cause of the problem. For example, diag sniff packet interface wan1 ‘tcp port 3389 or icmp’ 3
If you are sniffing VLAN packets, you cannot have any filter configured if you want to see the VLAN tags. For example diag sniffer packet wan1 “icmp” will not show the tags where diag sniffer packet wan1 will.