What is tracert and how can it help me?

[rudedog]

How to use tracrout (tracert) to help troubleshoot your client or server problems.

Tracert is a very useful too. It can be used to diagnose a, latency (high pings) problem with your game server.
By using this command you can see if the problem lies within, your provider's network, the Internet, or your game server's network.

Tracert is a command line utilities that is built into Windows and most other computer operating systems.

The basic tracert command syntax is " tracert hostname ". For example, " tracert 66.151.108.123 " and the output might look like:

http://www.mohadmin.com/nuke/download/dos_tracert_output.gif

tracert will return a list of routers or hops taken to get to our final destination. Along with all these hops, we will also get the latency (ping) to each router along the way.

Let get started. I will be using XP for this demonstration. But basically it's the same for all the Window's operating system's

To open a Command prompt.
start->run type: cmd press enter
You should get the following window:

http://www.mohadmin.com/nuke/download/dos.gif


Let's start by looking at the output from my tracert. This shows the path from my box to my Dedicated game
server.

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\jim>tracert 66.151.108.123

Tracing route to 66.151.108.123 over a maximum of 30 hops

1 28 ms 20 ms 17 ms 10.2.120.1
2 8 ms 7 ms 5 ms 68.9.8.85
3 43 ms 36 ms 34 ms 68.9.8.201
4 191 ms 15 ms 7 ms provdsrc02-gew0304.rd.ri.cox.net [68.9.14.17]
5 6 ms 57 ms 47 ms provbbrc02-pos0101.rd.ri.cox.net [68.1.0.48]
6 36 ms 87 ms 38 ms provdsrc02-gew03010999.rd.ri.cox.net [68.1.0.51]
7 66 ms 38 ms 36 ms so-6-0-0.gar2.wdc1.Level3.net [67.29.170.1]
8 56 ms 37 ms 36 ms so-7-0-0.edge1.Washington1.Level3.net [209.244.11.14]
9 37 ms 36 ms 37 ms POS1-1.BR1.IAD8.ALTER.NET [209.0.227.118]
10 37 ms 58 ms 36 ms 0.so-1-3-0.XR2.IAD8.ALTER.NET [152.63.32.118]
11 42 ms 53 ms 37 ms 0.so-0-0-0.CL2.IAD5.ALTER.NET [152.63.38.142]
12 60 ms 69 ms 39 ms 201.at-2-0-0.XR2.DCA6.ALTER.NET [152.63.35.49]
13 39 ms 111 ms 38 ms 0.so-1-3-0.XL2.DCA6.ALTER.NET [152.63.35.118]
14 40 ms 38 ms 66 ms POS7-0.GW4.DCA6.ALTER.NET [152.63.36.177]
15 211 ms 227 ms 57 ms internap1-gw.customer.alter.net [157.130.59.194]
16 44 ms 41 ms 96 ms border1.ge4-1-bbnet2.ext1.wdc.pnap.net [216.52.127.82]
17 437 ms 672 ms 423 ms netfire-2.border1.ext1.wdc.pnap.net [66.150.126.58]
18 44 ms 42 ms 41 ms 216.52.118.2
19 * * * Request timed out.
20 45 ms 44 ms 47 ms 66.151.108.123

Trace complete.

C:\Documents and Settings\jim>


Wow, we have some problems. We'll get to those in a minute.

1st some basic info. I live in RI. My game server is located in Washington, DC. My provider is Cox.net. From what
I can see. It takes 20 hops to reach my game server. My information or packets travel within my ISP's network
until it reaches Washington, DC. Normally Cox.net connects to the Internet in Atlanta, but with all the problems
with MCI, they have changed their backbone provider to one in the DC area.

Once my information exits the Cox network, it looks like it bounces around DC before entering, Netfire. My game server's ISP.
It then travels threw another router before ending at my game server, IP 66.151.108.123.

Lets look at some of the information provided for each hop. Let's look at hop 8
8 56 ms 37 ms 36 ms so-7-0-0.edge1.Washington1.Level3.net [209.244.11.14]

8 = The router or hop number

56ms 37ms 36 ms = The three ICMP (ping) times. ms=milliseconds. These are averaged when using the ping command.

so-7-0-0.edge1.Washington1.Level3.net = router name ( you can learn a lot from the router name ) This one is
located in Washington and is owned by Level 3. A major backbone provider.

[209.244.11.14] = routers IP address.

Now looking over the data returned form my tracert. I see that my provider is having some problems, at my hop 4.
provdsrc02-gew0304.rd.ri.cox.net [68.9.14.17] . which is my ISP cox.net. I also notice that hop15 is having some
major ping issues. But this router lies outside both my provider and my game server's provider. Good luck getting
that fixed soon. Now hop17 netfire-2.border1.ext1.wdc.pnap.net [66.150.126.58] is having major latency problems.
also hop 19 is timing out. An * means a particular segment did not return any information and could be
experiencing major congestion. Hops 17 -20 fall under my game server provider.

so from this tracert I can see several problems. Two of which I have some control over as I am a customer within,
that particular network.

Hop 15 is outside my ISP's and game server's network. I hope they realize there problem and fix it soon. Basically we
are at the mercy of the internet as we share this world wide network.

Remember this tracert was only done once. I would want to run several tracerts to get an overview of the
situation. As luck would have it. Soon after, I was done with this guide, I ran another tracert and everythign
was back to normal.

What did we learn from this. You can do some basic network trouble shooting and find out exactly where your high pings
are coming from. If they fall within your network then you can complain to your ISP. Hopefully they can fix the
problem ASAP.

Good luck and hopefully this document helps you understand what is between you and your game server.

Advanced users can use the following link for syntax and parameters:
http://www.microsoft.com/technet/treeview/default.asp?url=/TechNet/prodtechnol/winxppro/proddocs/tracert.asp

[Rockmonster]

Another use is to tracert a players ip who has come into your server and been particularly abusive or has broken some major protocols in your forums. Then you can report them to their provider or at least start to scare them, by telling them you are calling the police in their particular city to report them.

[rudedog]

Yes, that is another great idea for tracert.

**off topic**

Got your email from olddog. Sounds like a great idea. We will get back to you hopefully by mid next week as I will be away from the site over the weekend and early next week.

[=LAW= SeNTry]

There is a really handy free program called pingplotter that I use to traceroute. There is a free, a shareware, and a pro version. Seems to be really accurate for me anyways. You can set it to trace from 1 to 20 times or unlimited, and how much delay between each trace. It will show you the hops and the latency of each, and your average ping over however many traces it completes.

http://www.pingplotter.com

[socgaming]

one thing to keep in mind is some routers are set to not reply to ping requests. Like that one with all the ***'s, its probably set to not give any Ping response this is a common thing. Also just becasue on hop has a high ping doesnt mean there is necesarily a problem as long as the packet was passed along and the over all ping was good then there will not be any problems. If the packet gets lost and the *'s follow down the route then there are some issues and I would be calling your network admin.

[Pitter]

removed spam

[Derek]

Good Post Rune!

[HarryRag]

HLSW has a very good tracert function in it, so you can keep tracing a server for longer time ans see the packet loss, high ping intervals and different route jumps.

Very handy if you have problems with the server and want detailed data that also shows of in a screenshot for what exact hub is causing problems.

[Derek]

Great thread!

[DPGmaximus]

Already knew this, but none the less very nice info to share for everyone. Thanks.

[n00dles]

Traceroute misconceptions are my pet hate, so here in resides some facts.

Most core networks are actually well run and maintained with excesses of core link bandwidth, so congestion/routing loops are minimal. More common issues are due to complex problems obscure enough to make traceroute useless, MPLS comes first to mind.

Process:

Launch a probe towards destination, with ttl == 1
Each router hop decrements the TTL by 1
When TTL reaches 0, router sends ICMP TTL Exceeded to SRC
SRC host receives IMCP, displays hop in traceroute
Repeat from step 1, with incremented TTL by 1
probes reach DST host, it returns ICMP Dest Unreach
Traceroute complete


Multiple Probes:

Most traceroute tools send at least 3 probes per TTL, hence 3 results with latency or *
Each probe is sent to a different DST port to identify itself

any layer 4 hashing can send each probe a different path
Maybe visible to traceroute with ECMP hashing
Or invisible with 802.3ad layer 2 aggregation
But the result is the same, some probes may behave differently




Traceroute Implementations use different protocols most use UDP, windows uses ICMP others may use TCP or have options to change


Traceroute Latency:

Timestamp upon probe lunch
Timestamp when ICMP is received
subtract the diff to work out round-tip value


Remember, only the ROUND TIRP is measured
Traceroute shows you hops ONLY on the forward path
But showing latency based on forward path PLUS return path, ANY delay on the return will affect your results


Traceroute what you see:

Packet with TTL of 1 enters router via ingress interface
ICMP TTL Exceed is created as TTL reaches 0

ICMP source address is that of ingress router interface
This is the "Hop" displyed within traceroute results



DNS:

Most operators will encode useful info to their rDNS, such as interface type, port number, location, even router type.


Network delay:

There are roughly three main types of network latency
Serialisation Delay

delay caused by having to transmit data thru routers/switches in packet sized chunks

Queuing Delay

Time spend in routers interface queues, most likely link contention(unlikely on corelinks)

Propagation Delay

Time on the wire (in flight) primarily limited by the speed of light or electromagnetic propagation



Serialisation:

Delay due to packet forwarding
packets move through the network as a single unit
cant transmit the next until the last one is finished
Not so much an issue on modern networks
massive speed increases, over packet size remaining the same
1500bytes @ 56Kbps == 214.2ms
1500bytes @ T1 (1.536Mbps) = 7.8ms
1500bytes @ GigE (1Gbps) = 0.012ms


Queuing:

Understand utilisation

1GigE @ 500Mbps is seen to be 50% used
The TX-Ring of the interface is in reality either 100% or 0% not anywhere in between
So the real utilisation of the link is 50% of time within 1 second


Queuing is a normal function if a packet arrives it has to wait for the previous to leave and the interface to be free
When interface saturation occors probability of being queued rises (doh)
Queues are where QoS will decide how to handle a packet


Propagation:

Light travels thru a vacuum @ ~300,000km/sec
Optic fibre cores iirc have a refactive index of ~1.48
1/1.48 = ~0.67c, light through fibre = ~200,000km/sec
200,000km/sec = 200km (125 miles) per millisecond
Divide by 2 for RTT
An oftern used example is a fibre run around the equator would take some ~400ms due to the speed of light.


Identify latency:

Use location identifiers in DNS
Does latency fit propagation delay?
Eg NYC to london, in 60-70ms? 4200miles, yea!
Eg WDC to WDC 200ms? not likely


Modern Routers & Prioritisation:

Packets forwarded THRU the routers "data plane"

Fast path, hardware forwarding. eg Most "normal" internet traffic
Slow path, software forwarding. eg, ICMP, IP Options etc..

Packets forwarded TO the router "control plane"

BGP, IGP, SNMP, user access, ping or any traffic sent to an IP configured local to the router

Router CPUs tend to be really tiny

Cisco 7600 routers have around 600MHz CPUs
These devices can forward Gigs per second but have less CPU power than a cheap mobile phone!
ICMP creation is in NO way important to the router it would rather forward "real" flows than respond to ping.

On some routing platforms the control and data planes share resources
They don't have very good process schedulers for CPU time
This can result in a burst of control plane use of the CPU for say BGP update or CLI use, results in a delay/slow ICMP generation.
The result is random latency spikes in traceroute often seen as "network issues"
Biggest culprit for these random spikes is a Cisco process called "BGP scanner" runs every 60sec on all BGP enabled IOS devices


ICMPs rate limited generation:

Most routers rate limt ICMP
oftern hard-coded
will be insufficent under load

Juniper

limit of 50pps per interface 250pps on FPC3c
limit of 500pps per PFE around JUNOS 8.3+

Foundry

400pps per interface

Force 10

200pps or 600pps per interface




Spotting Fake latency:

during real issues latency will increase for all future hops
Latency spikes in the middle mean NOTHING if not continued
At worst its the result of an asymmetric path
Most likely artifical rate-limits or prioitisation


Asymmetric Paths

#1 issue for traceroute
Traceroute shows FORWARD path only

latency shown for each hop is based on
the time taken for probe to reach hop PLUS
time taken for TTL to Exceed reply to come back

The reverse path is completely invisible

Not only does traceroute not reveal anything about return path but
It can be totally different at every hop to forward path
Only solution is to trace both directions
Even with traces from both directions you can miss asymmetric paths in transit

Asymmetric paths generally start at network boundaries, as this is where admin policy changes
In some cases using a specific source address can work around asymmetric paths
Bare in mind asymmetric paths can crop up almost anywhere, most commonly where a networks connect in multiple locations and

use "hot-potato" routing, ie. use closest exit
other possible network state changes



Multiple Paths:

As each (UDP/TCP) traceroute probe uses a different layer 4 port equal-cost multi-path may introduce multiple paths show up within a single "hop"
This is harmless but could be confusing
A worse case is where ECMP(equal-cost multi-path) is balanced between paths of unequal hop count.
this can make traceroutes appear to bounce, is hard to read and confusing
When in doubt check only a single path, by telling traceroute to send only a single probe, just remember this may not be the "data" path
Another way to test different paths is to increment the dest IP by 1 or using different source IPs


MPLS and traceroute:

Most large networks run an MPLS core
Some devices don't hold an "internet" routing table just IGP routes

this is fine for MPLS switching
causes an issue for ICMP generation
the MPLS device has no routing information in order to deliver the ICMP

One solution is ICMP Tunnelling, this is done by generating an ICMP response within the LSP and forwarding to the end of said LSP from which the packet is able to be delivered to the intended destination, works but looks real strange
Results in every hop within an LSP appear to have the same RTT as the final hop.
MPLS routers can be configured to respect packet TTL values or not, in the case of not, the MPLS hops will not appear within the traceroute at all, so an MPLS core could be seen as a jump in latency for no reason.


Ok, done rant over.