Quick Links

You pay your Internet Service Provider (ISP) for internet access, and they turn on the sweet, sweet, fire hose of data for you. But who provides the flow for your ISP? Read on to learn the ins and outs of global data delivery. Today’s Question & Answer session comes to us courtesy of SuperUser—a subdivision of Stack Exchange, a community-drive grouping of Q&A web sites.

The Question

SuperUser reader KronoS poses the question many geeks have asked at one point:

I've been wondering recently about how the infrastructure of the Internet really works. I know that I have an Internet Service Provider (ISP) that supplies my connection to the Internet. But what I don't know is: Who provides the Internet to the ISP? And who supplies it to them? Is there a never-ending loop that eventually connects us all together?

Who indeed? It’s networks all the way down, but not all of them are immediately visible to the end user.

The Answer

Courtesy of SuperUser contributor Tom Wijsman, we’re treated to a detailed peek at how we can determine who specifically is providing internet access to our ISP and what it means to be part of the provider-to-the-providers network.

How do I figure out the Internet's infastructure?

Let's suppose we don't know about the history of the Internet, nor do we have access to any online resources that explains us this. Then, the only way to learn how the Internet infrastructure is built is to go back to the roots. Using existing protocols to discover how our Internet is built. Specifically, the Internet Control Message Protocol or ICMP defines the Echo request and the Echo reply. By increasing the Time To Live of an IP packets by 1 each iteration, you can find each next hop on the path to your target. This allows you to get a list of hops between you and your target, the classical traceroute. On Windows, you can use

        tracert
    

; on Linux and Mac OS X, you can use

        traceroute
    

. So, let's do a traceroute from Belgium to the United States; Stack Exchange looks like a good target.

        Tracing route to stackexchange.com [64.34.119.12] over a maximum of 30 hops:

  ... redacted ...

  5 10 ms 12 ms 12 ms te-3-3.car2.Brussels1.Level3.net [212.3.237.53]
  6 11 ms 11 ms 15 ms ae-0-11.bar2.Brussels1.Level3.net [4.69.148.178]
  7 20 ms 13 ms 15 ms ae-7-7.ebr1.London1.Level3.net [4.69.148.182]
  8 16 ms 16 ms 18 ms vlan101.ebr2.London1.Level3.net [4.69.143.86]
  9 83 ms 84 ms 87 ms ae-44-44.ebr1.NewYork1.Level3.net [4.69.137.78]
 10 84 ms 93 ms 97 ms ae-71-71.csw2.NewYork1.Level3.net [4.69.134.70]
 11 87 ms 96 ms 83 ms ae-2-70.edge1.NewYork1.Level3.net [4.69.155.78]
 12 84 ms 93 ms 84 ms gig2-0.nyc-gsr-b.peer1.net [216.187.123.5]
 13 87 ms 84 ms 85 ms gwny01.stackoverflow.com [64.34.41.58]
 14 87 ms 82 ms 87 ms stackoverflow.com [64.34.119.12]

Interesting, we now know that Belgium, London and New York are all connected to Level3. Level3 can be seen as an ISP to ISPs, they simply interconnect multiple ISPs. Here is a picture of how it's connected:

2012-09-04_110537

Let's go the opposite direction, China! The first thing I could find is the search engine Baidu.

        Tracing route to baidu.com [123.125.114.144] over a maximum of 30 hops:

  ... redacted ...

  5 12 ms 10 ms 12 ms ae0.anr11.ip4.tinet.net [77.67.65.177]
  6 167 ms 167 ms 167 ms xe-5-1-0.sjc10.ip4.tinet.net [89.149.185.161]
  7 390 ms 388 ms 388 ms as4837.ip4.tinet.net [77.67.79.150]
  8 397 ms 393 ms 397 ms 219.158.30.41
  9 892 ms * 392 ms 219.158.97.13
 10 407 ms 403 ms 403 ms 219.158.11.197
 11 452 ms 451 ms 452 ms 219.158.15.5
 12 * 434 ms 434 ms 123.126.0.66
 13 449 ms 450 ms 450 ms 61.148.3.34
 14 432 ms 433 ms 431 ms 202.106.43.66
 15 435 ms 435 ms 436 ms 123.125.114.144

Well, not much information about the Chinese ISPs there but we at least found Tinet. Here is a nice picture of their site that shows how they connect with the various ISPs:

2012-09-04_110636

They simply have a cloud of hops spread about the relevant part of the world they serve, and at the end points they connect to the ISPs. The reason they have a cloud of hops is for reliability, for when some hops fall out.

If you repeat this a few times, you can get an idea of how everything is connected.

2012-09-04_111016

So, What Network Tiers Are There?

The huge networks we found through trace-routing are known as Tier 1 networks.

Although there is no authority that defines tiers of networks participating in the Internet, the most common definition of a tier 1 network is one that can reach every other network on the Internet without purchasing IP transit or paying settlements.

By this definition, a tier 1 network is a transit-free network that peers with every other tier-1 network. But not all transit-free networks are tier 1 networks. It is possible to become transit-free by paying for peering or agreeing to settlements.

Common definitions of tier 2 and tier 3 networks:

If you click through to Tier 1 networks from the Internet Backbone page you get to a list of the current Tier 1 networks:

It is not known if AOL Transit Data Network (ATDN) is still a Tier 1 network.

Wait, what... What is Peering?

These networks connect to each other through a process known as 'peering'. Most traffic needs to go over at least 2 different top tier networks in order to reach its destination, and the networks are bridged with peering arrangements. The way this usually works is that each party to the agreement will commit to routing x amount of traffic for the other party on their network, and vice-verse. There is usually no money exchanged in these arrangements, unless one side is sending or receiving a lot more data than the other sides.

Large companies can also go out and arrange their own peering relationships. For example Netflix has arranged its own peering and network infrastructure directly with multiple tier-1 networks so that its traffic is both cheaper and closer to end users on each of the popular US broadband ISP's.

See this Wikipedia page on Peering.

There's a lot more to read at those pages; this answer gives a general idea, discovering all the details are left as an exercise to the reader.


Have something to add to the explanation? Sound off in the the comments. Want to read more answers from other tech-savvy Stack Exchange users? Check out the full discussion thread here.