On-demand Troubleshooting

User roles for this feature

	Admin	Co-admin
Run on-demand troubleshooting tests
Download PCAP files

Overview

Run on-demand network tests from any agent directly from the dashboard without accessing the host device.

The troubleshooting tab on each agent page enables a SEI Dashboard Admin or Co-admin to instantly run a suite of on-demand troubleshooting tests on any online agent without the need to access the agent or host device.

• Run on-demand tests for any agent directly from the dashboard without accessing the host device.
• Designed for users experienced in network troubleshooting in a Linux environment.
• Tests are available for every agent directly from the dashboard and do not require direct device access. Click the troubleshooting tab on any agent and run tests instantly
• Intuitive test inputs and familiar CLI-like output eliminate the learning curve to perform tests.
• Test results stream directly to the dashboard providing real-time network information and test results.
• Enable packet capture PCAP for up to one hour. Download .PCAP files for use in existing network visibility processes and tools like Wireshark
• Enabled for all agents version 0.5.11 and above

Following are details about available tests

Network Information

Begin troubleshooting with real-time network information for the agent.

• View the network information for the agent that is helpful for the network administrator to troubleshoot network issues.
• Network Information for the agent is fetched when the troubleshooting tab is loaded, or the update button is clicked.
• Network information is collected with the following commands that differ by agent type/host operating system.

Static Agents - Linux	Mobile Agents - macOS	Mobile Agents - Windows
Interfaces (ifconfig -a)	Interfaces (ifconfig -a)	Interfaces (ipconfig /all)
Routing Table (ip route show and ip -6 route show)	Interface details (ipconfig getsummary )	Routing Table (route print)
Arp Table (arp -an)	Routing Table (netstat -rn)	Arp Table (arp -a -v)
IPv6 Neighbor Table (ip -6 neighbor show)	Arp Table (arp -an)	IPv6 Neighbor Table (netsh interface ipv6 show neighbors)
WIFI Status (using netlink API)	IPv6 Neighbor Table (ndp -an)	WIFI Status (netsh wlan show interface)
	WIFI Status (/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport -I)

Ping Test

Ping test sends ICMP pings to any IP address or domain name

• Verify host reachability and view real-time loss and RTT from each agent to any reachable internet endpoint.
• Send up to 1000 requests. View packets and minimum, average, maximum, and StdDev RTT for all attempts sent during the test.
• Use advanced configuration options for more specific tests:
  • IP versions - V4 (default) or V6]
  • Payload size upto 8 bytes to 10 Kb
  • Echo time out up to 5 seconds
  • TTL up to 225 seconds
  • DSCP (many options)
  • Ability to enter custom source destination IP
  • Ping test is used to run ICMP ping to any IP address or domain name

Parameter	Notes	Default
Target	Destination Target. All Targets are listed. List may include Target that doesn’t respond to the selected protocol or options. Either a Target or Destination Address is required.	none
Destination Address	Destination address or domain name. IP address and domain name are acceptable. Either a Target or Destination Address is required.	none
Count	The number of ICMP ECHO_REQUEST packets. (1 - 1,000)	3
Interval	Wait interval seconds between sending each packet. (1 - 60)	1
IP Version	v4 or v6 is acceptable. [v4, v6]	v4
Payload Size	The number of data bytes to be sent. (8 - 10,000)	48
Don’t Fragment	Option for DF (don’t fragment) bit in IP header.	Disabled
Echo Timeout	Time to wait for a response, in second. (1 - 5)	1
TTL	Time to Live in IPv4 or hop limit in IPv6. If unspecified, a default value is set by the operating system networking stack. (1 - 255)	none
DSCP	Differentiated Services Codepoint (DSCP) for QoS.	0 (CS0)
Local interface IP address to use as source for the Ping	{“Source address used to send the outbound ICMP ping. If unspecified, the agent will bind to 0.0.0.0 (IPv4) or :”=>”(IPv6) by default and the operating system will choose the appropriate local address. It is recommended to leave this empty.”}	none

DNS Lookup

Interrogate DNS name servers, and perform DNS lookup that displays the answers that are returned from the name server(s)

• Lookup DNS of any address
• Query by record type - many selectable options (defaults to Type A)
• Supports custom DNS port (Default to 53) and DNS server IP address
• Switch from TCP to UDP

Parameter	Notes	Default
Query Name	Name of the resource record that is to be looked up.	none
Query Type	Type of query. [A, AAAA, NS, CNAME, SOA, MX, TXT, SRV]	A
DNS Server port number	DNS name server’s port number.	53
DNS Server IP Address	DNS name server’s IP address. If unspecified, the agent picks the DNS server configuration on the device. On Linux and Mac, typically this is a local resolver. On windows, a DNS server is picked from the list of configured dns servers. If fetching the configured server fails, fall back to choose 8.8.8.8.	none
Use TCP instead of UDP	Whether to use TCP or UDP to send the DNS query.	Disabled
Recursion not desired	Whether to skip recursion desired option to send the DNS query.	Disabled

Note: If the DNS name server is a proxy DNS that doesn’t support IPv6, a query for type AAAA will not be answered. User may need to change the DNS server that supports IPv6 in that case

Path Discovery

• This allows the user to discover the different paths available from the agent to the destination.
• SEI supports using UDP, ICMP and TCP for path probing. It is important to note that on Windows agents, SEI does not have a native implementation, and only ICMP is used for path probing by the windows cli tool tracert. Therefore, it is not possible to discover ECMP paths on windows agents.
• Refer to the Path Discovey section of the Metrics knowledge base article.

**Run multiple tests and view results it the same view. This example shows the results of running UDP, ICMP, and TCP tests in sequence.

Parameter	Notes	Default
Target	Destination Target. All Targets are listed. List may include Target that doesn’t respond to the selected protocol or options. Either a Target or Destination Address is required.	none
Destination Address	Destination address or domain name. IP address and domain name are acceptable. Either a Target or Destination Address is required.	none
Attempts	Number of attempts to discover unique paths. This value is ignored on Windows agent and 3 is used by default. (1 - 10)	3
IP Version	v4 or v6 is acceptable. [v4, v6]	v4
Protocol	Available protocols used for path discovery: - Static Agent supports ICMP, UDP, and TCP - Mobile Agent for Windows supports only ICMP without ECMP - Mobile Agent for MacOS supports ICMP, UDP, and TCP	- UDP for Static and macOS  - ICMP for Windows

HTTP Test

• Send synthetic HTTP traffic to simulate agent-to-server interaction with existing HTTP targets or any URL.
• Sends HTTP requests to measure response times and availability of applications, APIs, and cloud-hosted services.
• Identify server-side issues that may otherwise appear as network capacity limitations.
• Switch IP version from V4 (default) to V6
• Thoroughly test API connectivity with GET, PUT, POST, and DELETE Request Methods
• Add custom request headers and body
• Basic authentication (username and password) supported
• Optionally enter local IP as the source for the test
• Functionally performs like curl command, but also provides network and HTTP specific metrics including:
  • TCP Latency
  • TCP Jitter
  • DNS Lookup Time
  • TCP Connect Time
  • TLS Handshake Time
  • HTTP Request Send Time
  • First Response packet received Time
  • Response content download Time
  • Total Time taken

Parameter	Notes	Default
Target	Destination URL. List includes Targets that respond to the HTTP(S) protocol. Either a Target or URL is required.	none
URL	Destination URL. A scheme (http:// or https://) is mandatory.. Either a Target or URL is required.	none
IP Version	v4 or v6 is acceptable. [v4, v6]	v4
Request Method	HTTP request. [GET, POST, PUT, DELETE]	GET
Request Headers	HTTP headers to use in the HTTP request.	none
Request Body	Request body for POST or PUT method	none
Authentication Method	Authentication method BASIC is supported. If it’s specified, Username and Password are required. [No Authentication, Basic Authentication]	No Authentication
Local interface IP address to use as source for the Ping	{“Source address used to send the outbound ICMP ping. If unspecified, the agent will bind to 0.0.0.0 (IPv4) or :”=>”(IPv6) by default and the operating system will choose the appropriate local address. It is recommended to leave this empty.”}	none

Packet Capture

• The following filters are applied for the packet capture:
  • ARP and UDP port 53 for DHCP
  • IP, protocol (and port is needed) for probing sessions.
  • IP, protocol (and port if needed) for on demand troubleshooting destinations.
• The agent only captures the probing packets as specified above. The capture size can be further limited by selecting the Header Only option. With this, the agent only captures up to 138 bytes of each packet with the aim to capture the packet headers.
• On demand packet capture is saved and retrieved from local storage of the agent. Only one capture file is stored and the file is overwritten when the packet capture is run again.

Enabling Packet Capture

Step 1: Toggle to Enable Packet Capture

Step 2: Headers Only is checked by default. Check Full Packets to capture all packets while enabled as shown in this image.

Step 3: Set the duration of packet capture - one minute to one hour.

Step 4: Click the start button to begin capturing packets.

• If Headers only is checked, the timer will start counting down from 3 minutes.
• If Full Packets is checked, the timer counts down from the duration set.
• Cancel packet capture before the countdown ends by clicking the cancel button or setting the toggle off.

Step 5: Download .PCAP file

• Packet Capture (PCAP) files are instantly available for download when the capture duration is complete, or the process is canceled.
• The .PCAP files are available for download until packet capture is enabled again. At this point, prior .PCAP files are no longer available and cannot be recovered.