What is tracert and how can it help me?

[rudedog]

How to use tracrout (tracert) to help troubleshoot your client or server problems.

Tracert is a very useful too. It can be used to diagnose a, latency (high pings) problem with your game server.
By using this command you can see if the problem lies within, your provider's network, the Internet, or your game server's network.

Tracert is a command line utilities that is built into Windows and most other computer operating systems.

The basic tracert command syntax is " tracert hostname ". For example, " tracert 66.151.108.123 " and the output might look like:



tracert will return a list of routers or hops taken to get to our final destination. Along with all these hops, we will also get the latency (ping) to each router along the way.

Let get started. I will be using XP for this demonstration. But basically it's the same for all the Window's operating system's

To open a Command prompt.
start->run type: cmd press enter
You should get the following window:




Let's start by looking at the output from my tracert. This shows the path from my box to my Dedicated game
server.

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\jim>tracert 66.151.108.123

Tracing route to 66.151.108.123 over a maximum of 30 hops

1 28 ms 20 ms 17 ms 10.2.120.1
2 8 ms 7 ms 5 ms 68.9.8.85
3 43 ms 36 ms 34 ms 68.9.8.201
4 191 ms 15 ms 7 ms provdsrc02-gew0304.rd.ri.cox.net [68.9.14.17]
5 6 ms 57 ms 47 ms provbbrc02-pos0101.rd.ri.cox.net [68.1.0.48]
6 36 ms 87 ms 38 ms provdsrc02-gew03010999.rd.ri.cox.net [68.1.0.51]
7 66 ms 38 ms 36 ms so-6-0-0.gar2.wdc1.Level3.net [67.29.170.1]
8 56 ms 37 ms 36 ms so-7-0-0.edge1.Washington1.Level3.net [209.244.11.14]
9 37 ms 36 ms 37 ms POS1-1.BR1.IAD8.ALTER.NET [209.0.227.118]
10 37 ms 58 ms 36 ms 0.so-1-3-0.XR2.IAD8.ALTER.NET [152.63.32.118]
11 42 ms 53 ms 37 ms 0.so-0-0-0.CL2.IAD5.ALTER.NET [152.63.38.142]
12 60 ms 69 ms 39 ms 201.at-2-0-0.XR2.DCA6.ALTER.NET [152.63.35.49]
13 39 ms 111 ms 38 ms 0.so-1-3-0.XL2.DCA6.ALTER.NET [152.63.35.118]
14 40 ms 38 ms 66 ms POS7-0.GW4.DCA6.ALTER.NET [152.63.36.177]
15 211 ms 227 ms 57 ms internap1-gw.customer.alter.net [157.130.59.194]
16 44 ms 41 ms 96 ms border1.ge4-1-bbnet2.ext1.wdc.pnap.net [216.52.127.82]
17 437 ms 672 ms 423 ms netfire-2.border1.ext1.wdc.pnap.net [66.150.126.58]
18 44 ms 42 ms 41 ms 216.52.118.2
19 * * * Request timed out.
20 45 ms 44 ms 47 ms 66.151.108.123

Trace complete.

C:\Documents and Settings\jim>


Wow, we have some problems. We'll get to those in a minute.

1st some basic info. I live in RI. My game server is located in Washington, DC. My provider is Cox.net. From what
I can see. It takes 20 hops to reach my game server. My information or packets travel within my ISP's network
until it reaches Washington, DC. Normally Cox.net connects to the Internet in Atlanta, but with all the problems
with MCI, they have changed their backbone provider to one in the DC area.

Once my information exits the Cox network, it looks like it bounces around DC before entering, Netfire. My game server's ISP.
It then travels threw another router before ending at my game server, IP 66.151.108.123.

Lets look at some of the information provided for each hop. Let's look at hop 8
8 56 ms 37 ms 36 ms so-7-0-0.edge1.Washington1.Level3.net [209.244.11.14]

8 = The router or hop number

56ms 37ms 36 ms = The three ICMP (ping) times. ms=milliseconds. These are averaged when using the ping command.

so-7-0-0.edge1.Washington1.Level3.net = router name ( you can learn a lot from the router name ) This one is
located in Washington and is owned by Level 3. A major backbone provider.

[209.244.11.14] = routers IP address.

Now looking over the data returned form my tracert. I see that my provider is having some problems, at my hop 4.
provdsrc02-gew0304.rd.ri.cox.net [68.9.14.17] . which is my ISP cox.net. I also notice that hop15 is having some
major ping issues. But this router lies outside both my provider and my game server's provider. Good luck getting
that fixed soon. Now hop17 netfire-2.border1.ext1.wdc.pnap.net [66.150.126.58] is having major latency problems.
also hop 19 is timing out. An * means a particular segment did not return any information and could be
experiencing major congestion. Hops 17 -20 fall under my game server provider.

so from this tracert I can see several problems. Two of which I have some control over as I am a customer within,
that particular network.

Hop 15 is outside my ISP's and game server's network. I hope they realize there problem and fix it soon. Basically we
are at the mercy of the internet as we share this world wide network.

Remember this tracert was only done once. I would want to run several tracerts to get an overview of the
situation. As luck would have it. Soon after, I was done with this guide, I ran another tracert and everythign
was back to normal.

What did we learn from this. You can do some basic network trouble shooting and find out exactly where your high pings
are coming from. If they fall within your network then you can complain to your ISP. Hopefully they can fix the
problem ASAP.

Good luck and hopefully this document helps you understand what is between you and your game server.

Advanced users can use the following link for syntax and parameters:
http://www.microsoft.com/technet/tre...cs/tracert.asp

[Rockmonster]

Another use is to tracert a players ip who has come into your server and been particularly abusive or has broken some major protocols in your forums. Then you can report them to their provider or at least start to scare them, by telling them you are calling the police in their particular city to report them.

[rudedog]

Yes, that is another great idea for tracert.

**off topic**

Got your email from olddog. Sounds like a great idea. We will get back to you hopefully by mid next week as I will be away from the site over the weekend and early next week.

[=LAW= SeNTry]

There is a really handy free program called pingplotter that I use to traceroute. There is a free, a shareware, and a pro version. Seems to be really accurate for me anyways. You can set it to trace from 1 to 20 times or unlimited, and how much delay between each trace. It will show you the hops and the latency of each, and your average ping over however many traces it completes.

http://www.pingplotter.com

[socgaming]

one thing to keep in mind is some routers are set to not reply to ping requests. Like that one with all the ***'s, its probably set to not give any Ping response this is a common thing. Also just becasue on hop has a high ping doesnt mean there is necesarily a problem as long as the packet was passed along and the over all ping was good then there will not be any problems. If the packet gets lost and the *'s follow down the route then there are some issues and I would be calling your network admin.

removed spam

[Derek]

Good Post Rune!

[HarryRag]

HLSW has a very good tracert function in it, so you can keep tracing a server for longer time ans see the packet loss, high ping intervals and different route jumps.

Very handy if you have problems with the server and want detailed data that also shows of in a screenshot for what exact hub is causing problems.

[Derek]

Great thread!

[DPGmaximus]

Already knew this, but none the less very nice info to share for everyone. Thanks.

[n00dles]

Traceroute misconceptions are my pet hate, so here in resides some facts.

Most core networks are actually well run and maintained with excesses of core link bandwidth, so congestion/routing loops are minimal. More common issues are due to complex problems obscure enough to make traceroute useless, MPLS comes first to mind.

Process:

1. Launch a probe towards destination, with ttl == 1
2. Each router hop decrements the TTL by 1
3. When TTL reaches 0, router sends ICMP TTL Exceeded to SRC
4. SRC host receives IMCP, displays hop in traceroute
5. Repeat from step 1, with incremented TTL by 1
6. probes reach DST host, it returns ICMP Dest Unreach
7. Traceroute complete

Multiple Probes:

• Most traceroute tools send at least 3 probes per TTL, hence 3 results with latency or *
• Each probe is sent to a different DST port to identify itself
  • any layer 4 hashing can send each probe a different path
  • Maybe visible to traceroute with ECMP hashing
  • Or invisible with 802.3ad layer 2 aggregation
  • But the result is the same, some probes may behave differently

• Traceroute Implementations use different protocols most use UDP, windows uses ICMP others may use TCP or have options to change

Traceroute Latency:

1. Timestamp upon probe lunch
2. Timestamp when ICMP is received
3. subtract the diff to work out round-tip value

• Remember, only the ROUND TIRP is measured
• Traceroute shows you hops ONLY on the forward path
• But showing latency based on forward path PLUS return path, ANY delay on the return will affect your results

Traceroute what you see:

• Packet with TTL of 1 enters router via ingress interface
• ICMP TTL Exceed is created as TTL reaches 0
  • ICMP source address is that of ingress router interface
  • This is the "Hop" displyed within traceroute results

DNS:

• Most operators will encode useful info to their rDNS, such as interface type, port number, location, even router type.

Network delay:

• There are roughly three main types of network latency
• Serialisation Delay
  • delay caused by having to transmit data thru routers/switches in packet sized chunks
• Queuing Delay
  • Time spend in routers interface queues, most likely link contention(unlikely on corelinks)
• Propagation Delay
  • Time on the wire (in flight) primarily limited by the speed of light or electromagnetic propagation

Serialisation:

• Delay due to packet forwarding
• packets move through the network as a single unit
• cant transmit the next until the last one is finished
• Not so much an issue on modern networks
• massive speed increases, over packet size remaining the same
• 1500bytes @ 56Kbps == 214.2ms
• 1500bytes @ T1 (1.536Mbps) = 7.8ms
• 1500bytes @ GigE (1Gbps) = 0.012ms

Queuing:

• Understand utilisation
  • 1GigE @ 500Mbps is seen to be 50% used
  • The TX-Ring of the interface is in reality either 100% or 0% not anywhere in between
  • So the real utilisation of the link is 50% of time within 1 second
  • Queuing is a normal function if a packet arrives it has to wait for the previous to leave and the interface to be free
  • When interface saturation occors probability of being queued rises (doh)
  • Queues are where QoS will decide how to handle a packet

Propagation:

• Light travels thru a vacuum @ ~300,000km/sec
• Optic fibre cores iirc have a refactive index of ~1.48
• 1/1.48 = ~0.67c, light through fibre = ~200,000km/sec
• 200,000km/sec = 200km (125 miles) per millisecond
• Divide by 2 for RTT
• An oftern used example is a fibre run around the equator would take some ~400ms due to the speed of light.

Identify latency:

• Use location identifiers in DNS
• Does latency fit propagation delay?
• Eg NYC to london, in 60-70ms? 4200miles, yea!
• Eg WDC to WDC 200ms? not likely

Modern Routers & Prioritisation:

• Packets forwarded THRU the routers "data plane"
  • Fast path, hardware forwarding. eg Most "normal" internet traffic
  • Slow path, software forwarding. eg, ICMP, IP Options etc..
• Packets forwarded TO the router "control plane"
  • BGP, IGP, SNMP, user access, ping or any traffic sent to an IP configured local to the router
• Router CPUs tend to be really tiny
  • Cisco 7600 routers have around 600MHz CPUs
  • These devices can forward Gigs per second but have less CPU power than a cheap mobile phone!
  • ICMP creation is in NO way important to the router it would rather forward "real" flows than respond to ping.
• On some routing platforms the control and data planes share resources
• They don't have very good process schedulers for CPU time
• This can result in a burst of control plane use of the CPU for say BGP update or CLI use, results in a delay/slow ICMP generation.
• The result is random latency spikes in traceroute often seen as "network issues"
• Biggest culprit for these random spikes is a Cisco process called "BGP scanner" runs every 60sec on all BGP enabled IOS devices

ICMPs rate limited generation:

• Most routers rate limt ICMP
• oftern hard-coded
• will be insufficent under load
  • Juniper
    • limit of 50pps per interface 250pps on FPC3c
    • limit of 500pps per PFE around JUNOS 8.3+
  • Foundry
    • 400pps per interface
  • Force 10
    • 200pps or 600pps per interface

Spotting Fake latency:

• during real issues latency will increase for all future hops
• Latency spikes in the middle mean NOTHING if not continued
• At worst its the result of an asymmetric path
• Most likely artifical rate-limits or prioitisation

Asymmetric Paths

• #1 issue for traceroute
• Traceroute shows FORWARD path only
  • latency shown for each hop is based on
  • the time taken for probe to reach hop PLUS
  • time taken for TTL to Exceed reply to come back
• The reverse path is completely invisible
  • Not only does traceroute not reveal anything about return path but
  • It can be totally different at every hop to forward path
  • Only solution is to trace both directions
  • Even with traces from both directions you can miss asymmetric paths in transit
• Asymmetric paths generally start at network boundaries, as this is where admin policy changes
• In some cases using a specific source address can work around asymmetric paths
• Bare in mind asymmetric paths can crop up almost anywhere, most commonly where a networks connect in multiple locations and
  • use "hot-potato" routing, ie. use closest exit
  • other possible network state changes

Multiple Paths:

• As each (UDP/TCP) traceroute probe uses a different layer 4 port equal-cost multi-path may introduce multiple paths show up within a single "hop"
• This is harmless but could be confusing
• A worse case is where ECMP(equal-cost multi-path) is balanced between paths of unequal hop count.
• this can make traceroutes appear to bounce, is hard to read and confusing
• When in doubt check only a single path, by telling traceroute to send only a single probe, just remember this may not be the "data" path
• Another way to test different paths is to increment the dest IP by 1 or using different source IPs

MPLS and traceroute:

• Most large networks run an MPLS core
• Some devices don't hold an "internet" routing table just IGP routes
  • this is fine for MPLS switching
  • causes an issue for ICMP generation
  • the MPLS device has no routing information in order to deliver the ICMP
• One solution is ICMP Tunnelling, this is done by generating an ICMP response within the LSP and forwarding to the end of said LSP from which the packet is able to be delivered to the intended destination, works but looks real strange
• Results in every hop within an LSP appear to have the same RTT as the final hop.
• MPLS routers can be configured to respect packet TTL values or not, in the case of not, the MPLS hops will not appear within the traceroute at all, so an MPLS core could be seen as a jump in latency for no reason.

Ok, done rant over.