On this page:
The Departmental Network Engineering group uses many software packages and tools for network analysis and troubleshooting.
DHCP
Stanford University uses the Internet Software Consortium's package for DHCP (Dynamic Host Configuration Protocol) services. For the most part, at Stanford, publicly routed DHCP addresses are only provided to those hosts whose MAC addresses are entered in NetDB.
You should not run your own departmental DHCP servers without first consulting with Network Engineering, as improperly configured DHCP servers can disrupt network connectivity for yourself and other groups.
The DHCP lease time for typical hosts is 2 days. As per the DHCP spec, hosts will renew the lease halfway through this interval. Wireless and wired roaming DHCP addresses have lease times of less than 1 hour.
To set up a node for DHCP:
- Normal DHCP: In this case, the device will receive one of the registered IP address if it is appropriate for the network where they are connected. (In other words, although the address is distributed via DHCP, the host receives the same address each time.) Typically, this will be the method used for connecting to the local Ethernet "wired" network.
- The device MAC address must be entered in NetDB, the DHCP option must be selected, and one or more valid IP address for the "home" network must be specified for the MAC address.
- Checking the "Roaming" box for this MAC address will also allow the device to obtain roaming addresses on other campus networks where roaming has been enabled.
- Note that it is possible to bind more than one IP address to a given MAC address and have both available via DHCP. Having multiple IP addresses is appropriate for users who have offices or labs in more than one campus building.
- Roaming: Users with Roaming access, when not on a network where their registered IP address(es) reside, will receive a random address from the local network's "roaming pool", a set of addresses available for infrequent users defined by that network's LNA. Typically, this will be the manner for assigning addresses on the wireless network. Most laptops should have the "roaming" checkbox in NetDB enabled for all their interfaces; most desktop will not need it.
- The device's MAC address must be entered in NetDB, and the DHCP and Roaming checkboxes must be selected.
- It is not necessary to enter an IP address. If IP addresses are entered, a device will be given these IP addresses (if appropriate for that network) as per normal DHCP, and in addition, be assigned a roaming address in other locations. Thus roaming and normal DHCP are not exclusive.
- With the exception of the campus wireless networks, the number of roaming addresses is usually quite limited. Because of this, any user who frequently connects to a building's wired network should be assigned an IP address as for "Normal DHCP" below.
Network Engineering provides a number of reports on DHCP usage:
- A report on the Roaming address utilization. Use this report to see if a particular network's roaming pool is being over- or under-utilized.
- The ability to query the DHCP servers for a particular client. An example of the results from this query is shown here.
- Report from DHCP servers Dusk and Dawn on the number of DHCP requests for given clients. Clients which make dozens of DHCP requests per hour usually have some sort misconfiguration and should be investigated.
DNS
NetDB is the primary interface for DNS at Stanford. LNAs create NetDB records for each host, which is periodically uploaded to the DNS servers. As of this writing, Stanford does not support any sort of campus-wide dynamic DNS.
The "host", "dig", and "nslookup" commands are useful tools for verifying DNS information.
Packet Capture
Wireshark, an open source network analysis tool, is the de facto standard packet capture and analysis tool.
IPM
All routers maintain ARP caches which associate IP and MAC addresses. Network Engineering "harvests" these ARP caches ever 4 hours and compiles the information into a database. IPM (short for "IP Media") is a command line tool that LNA's can use to query this database about IP and MAC addresses used on the Stanford network, including the date/time an IP or MAC address was first and last seen. Note that the MAC addresses do not have to be entered in NetDB; IPM gathers the association between MAC addresses and IP addresses directly from the routers.
IPM is available on the Cardinal servers maintained by the FarmShare team.
On the Cardinal servers, IPM is located at: /usr/bin/ipm. In order to use ipm, you may need to specify the whole path or add the /usr/bin directory to your PATH. You can run IPM without parameters to get a quick list of the available options or run it with help option ("ipm -h") to see a more comprehensive list of options and usage. Wildcards of "_" (for a single character) and "%" (for any number of characters) can be used in the query.
Things you can do with IPM:
- Get a list of IP addresses which have been used on your particular net within a particular period. (e.g., "ipm -d 360 171.64.52.0/22" would list the addresses that have been used on the network 171.64.52-55 within the last 360 days.)
- Identify where a particular MAC address has been seen on campus (e.g., "ipm -d 60 mac-addr" will show when a particular MAC address has been seen for the last 60 days).
- Identify which MAC addresses have been using a particular IP address. This may be helpful in tracking down "IP thieves" who use IP addresses without registering. (e.g., "ipm ip-addr").
Example report:
cardinal1:~> ipm networking Hardware First Last Times IP Address & Type NetDB Entry Address NIC Type Seen Seen Seen ----------------- --------------- ------------- -------- -------- -------- ----- 171.64.20.23 n networking 0003ba:11e241 --- 04/10/02 11/06/06 9984 [1 match returned, 0 items failed to match]
MAC Vendor Codes
The first 6 hexadecimal digits in the MAC address indicate the manufacturer of the NIC. The following sites have MAC Vendor Lookup and OUI Lookup functions. This may help you in tracking down rogue or misbehaving devices on your network.
NetDB, whois, and the NetDB Command Line (NetDB CLI)
NetDB is a Web-based utility used to make entries in our DNS and DHCP servers. In addition to the creation and maintenance of Stanford hostnames, its database allows LNAs to track information on host location, operating systems, and users. A comprehensive help file is available.
The whois command, in addition to looking up people, can also be used to look up information in NetDB. Whois reflects the state of NetDB; items returned by whois may not have yet been uploaded to the DNS servers. This is a very useful utility, as it allows people to have "read only" access to NetDB entries without having to have NetDB accounts. Examples of whois queries are shown below. Note that the Stanford whois server must be specified:
- Lookup host by IP: "whois -h whois.stanford.edu 171.64.20.120"
- Lookup host by name: "whois -h whois.stanford.edu networking." or "whois -h whois.stanford.edu networking.stanford.edu". Note the trailing period in the shortened version; without the period, whois will try to return a person with this name.
- Lookup host by MAC address: "whois -h whois.stanford.edu 0003.93f4.de9a" (most popular MAC address formats are supported, e.g., xx-xx-xx-xx-xx-xx, xxxxxx:xxxxxx, xx:xx:xx:xx:xx:xx", etc.)
- Lookup network records: "whois -h whois.stanford.edu 171.64.20.0" (shows the network record including any roaming DHCP addresses.
There is also a command line NetDB available in AFS space at /usr/pubsw/sbin/netdb. (By default, this directory is not in the standard PATH. To run this command, you must either specify the entire path or add the /usr/pubsw/sbin/ directory to your Unix PATH.) While the Web interface to NetDB is the most friendly for day-to-day modifications, it is cumbersome to do large number of changes (e.g., modifying or deleting a large number of nodes). For the most basic options, you can type netdb without arguments. A more comprehensive help file is available by typing netdb --help.
The NetDB CLI is being expanded periodically with new functionality. At this time, some of the most useful features are:
- Change node administrators: netdb node admin --add _admin2_ --remove _admin1_ ... INPUT
- Delete nodes: netdb node delete [ --keep_mx ] [--force ] INPUT
- Change IP addresses: netdb node ip_address --remove _old_ip_ [ --add _new_ip_ ] node
Netspeed and iPerf
A common complaint from users is that "the network is slow". But is it really the network, or is it the user's computer, a server, or an application that might be at fault? These two tools can confirm whether the network is, in fact, to blame or whether there is some other culprit responsible.
Stanford maintains a server, netspeed.stanford.edu (using perfsonar) which can be used to test the bandwidth to the Stanford backbone. In addition to bandwidth testing, netspeed can report duplex mismatches, which is helpful in troubleshooting. Netspeed is connected via a Gigabit link.
Another bandwidth testing tool for the command line is iPerf. The syntax for testing against a given server, the usage for iPerf is: "iperf -c server. The strength of iPerf is that it can be run in server mode (using the "-s" option) on laptops and desktops, allowing LNAs to do point-to-point speed testing within local networks.
Here are some example iPerf speed report from a system with a Gigabit NIC:
$ iperf -c netspeed.stanford.edu ------------------------------------------------------------ Client connecting to netspeed.stanford.edu, TCP port 5001 TCP window size: 65.0 KByte (default) ------------------------------------------------------------ [ 3] local 171.64.20.120 port 61848 connected with 171.66.6.36 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 372 MBytes 312 Mbits/sec $ iperf -c iperf.stanford.edu ------------------------------------------------------------ Client connecting to iperf, TCP port 5001 TCP window size: 65.0 KByte (default) ------------------------------------------------------------ [ 3] local 171.64.20.120 port 49903 connected with 171.66.6.38 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 893 MBytes 749 Mbits/sec
Typical speeds for SUNet end-user systems are (Mb/s = megabits/second):
- Wireless 802.11b: 3-5 Mb/s is considered to be excellent performance; 1-3 Mb/s is more typical (comparable to DSL speeds).
- Wireless 802.11g: 3-8 Mb/s is typical. We have seen 802.11g connection speeds above 15 Mb/s under ideal conditions, but such speeds are rare.
- For machines with 100Mb/s (megabits/second) NICs, netspeed should show speeds of 70+ Mb/s.
- Many systems with gigabit NICs will not show speeds much over 300 Mb/s. Realistically, most machines are incapable of pushing data at high speeds without high speed disk arrays and 64-bit NICs. While 30 Mb/s out of 100 Mb/s (30%) might well indicate a problem with network congestion, 300 Mb/s on a gigabit link (also 30%) may well indicate excellent performance. Note also that iperf is creating packets from memory and so, iperf speeds may not be achievable if information has to be accessed from a disk. It is best to consider iperf an upper bound on performance.
nmap
nmap, Network Mapper, is a port-scanning utility, which is used to find open ports on hosts and networks. Stanford's Information Security Office team can help you determine if your host is insecure and help you secure it.
Ping, Traceroute (Tracert)
Ping (Packet INternet Groper, although the author disputes this name) is a utility that is primarily used to determine if a host wants to respond to ping queries, and to get an idea as to the time the packets are taking to return, which can sometimes be useful in determining possible network delays. Ping times are not an absolute determinant of network problems. Since Ping is free, it has been ported to just about every operating system that has a command line.
Traceroute is another UNIX utility, ported to many other platforms (it's known as "Tracert" on Windows computers) that sends TTL (Time To Live) packets to a host, and attempts to trace every router (not every network device) on the path to the designated host. Some routers will deliberately ignore traceroute packets, but your total path may still be determinable. Generally, Traceroute attempts to send 3 packets, since traceroute packets are often ignored by some devices. Traceroute gives an indication of possible bottlenecks, but again, is not an absolute determinant of network problems. Yahoo has a whole section on traceroute information, utilities, organizations, etc.
Because pings can and have been used as methods for Denial of Service (DOS) attacks, Stanford limits the amount of bandwidth used by ping and traceroutes entering and leaving campus. Because of this, pings and traceroutes are not reliable tools for diagnosing network availability and especially bandwidth. For example, it is easily possible that a Stanford host will show ping times of 1+ seconds when pinged from the Internet, even though the host is up and the link to that host is not congested. Network Engineering maintains a host ping-me.stanford.edu which is exempted from the limitations on pings. If you receive reports from outside users of poor network performance, we suggest that you have them ping this host to establish whether the Stanford Net is having connectivity problems or whether pings are simply being filtered.
networking:~> ping ping-me.stanford.edu PING ping-is-not-a-reliable-diagnostic-tool.Stanford.EDU (171.66.6.66): 56 data bytes 64 bytes from ping-is-not-a-reliable-diagnostic-tool.Stanford.EDU (171.66.6.66): seq=0 ttl=252 time=1.51 ms. 64 bytes from ping-is-not-a-reliable-diagnostic-tool.Stanford.EDU (171.66.6.66): seq=1 ttl=252 time=1.20 ms. ^C ---- ping-is-not-a-reliable-diagnostic-tool.Stanford.EDU (171.66.6.66) PING Statistics ---- 2 packets transmitted, 2 packets received, 0% packet loss round-trip (ms) min/avg/max = 1.20/1.35/1.51 (std = 0.156)
Wireless
WiFi Explorer and iStumbler
WiFi Explorer and iStumbler (Mac) are tools for locating nearby access points. The tools will show the MAC addresses and SSID's of all access points in range. Netstumbler will also show the access point with which the computer is currently associated. For MACs, the built-in utility Internet Connect will show the current association.
These tools are useful for LNAs to troubleshoot access point problems. A user may say that they're having wireless problems in a particular place. By using Netstumbler/iStumbler in conjunction with Netspeed, a speed test can be performed through a particular access point. If poor speed is found, that access point can be rebooted, which often fixes problems.