Slide 1: Peering with Content Distribution Networks
Patrick W. Gilmore patrick@akamai.com
September 14, 2004 Gigabit Peering Forum IX
Slide 2: The Business Internet
Agenda
• • • • • Disclaimer How CDNs Use NAPs CDN Specific Peering Challenges Data Exchange Offer Questions
Slide 2
Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 3: The Business Internet
Disclaimer
I work for Akamai Technologies, a Content Distribution Network. This presentation is supposed to be CDNagnostic (unless otherwise noted). Feel free to ask questions, Akamai specific or otherwise, at any point in the presentation.
Slide 3
Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 4: The Business Internet
How CDNs use NAPs
Peer Network
NAP
Content
• Since some CDNs do not have a backbone (including Akamai), each NAP can be independent • The CDN uses transit to pull content into the servers • Content is then served to peers over the NAP
Enterprise Data Center
Transit Origin Server
Equinix Gigabit Peering Forum IX, September 14, 2004
CDN Servers
Slide 4
Slide 5: The Business Internet
CDN Specific NAP Issues
• For many CDNs, there are unique issues which are specific to NAP locations • These problems arise out of the use of DNS for directing traffic and the lack of a backbone
– Obviously CDNs who do have a backbone and/or do not use DNS for directing traffic will not have these problems
• While there are solutions, even the solutions have their own problems (like always)
Slide 5
Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 6: The Business Internet
Issue #1: Freeloaders
• The first issue relates to how DNS-based CDNs operate • Because the redirection is based solely on DNS hostname resolution, there can be errors when the recursive name server is not in proximity to the end user • Specifically, if the recursive name server is in a prefix announced over a peering session, but the end user is not, Bad Things can happen
Slide 6
Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 7: The Business Internet
Issue #1: Freeloaders
Peer Network
NS
DNS Lookup
• Because the router cannot know the user was mapped from the peer’s Name Server, it will send the web page over the transit connection
NAP
Content Other Network
Equinix Gigabit Peering Forum IX, September 14, 2004
CDN Servers
Slide 7
Transit
Slide 8: The Business Internet
Issue #1: Freeloaders
• These are called “Freeloaders” because the CDN thinks the hit will be free, but it is not
– Actual cost is no greater than serving from non-NAP locations (transit is transit) – This can lower capacity at NAP, which might cause other problems
• Fortunately, most name servers only resolve for end users in the same AS
Slide 8
Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 9: The Business Internet
Issue #1: Freeloaders
• If you peer with a CDN in multiple places, please be careful if you are announcing inconsistently
– Group your name servers with your end users at the same NAPs
• This is especially true of people who peer in multiple countries • Also sometimes a problem for outsourced dialup or other “managed services” networks
Slide 9
Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 10: The Business Internet
Issue #1: Freeloaders
• There is not much a CDN can do to “solve” this problem, so efforts are mostly focused on minimizing the effects • First, CDNs need to ensure there is enough transit to serve the Freeloaders • Second, CDNs work with the peering partners to identify offending name servers and modify announcements • Lastly, the CDN can filter the offending announcements
Slide 10 Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 11: The Business Internet
Issue #2: Insufficient Transit
• Assume the CDN has a 1 gigabit connection to the NAP, but only 100 Mbps of transit
– This is possible because the transit is supposed to be used only for administrative purposes (origin GETs, log delivery, OS updates, etc.)
• If the NAP connection goes down for any reason, the traffic cannot be re-directed to another peering point • End users are still requesting content from the web servers at the NAP
Slide 11
Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 12: The Business Internet
Issue #2: Insufficient Transit
Peer Network
Traffic NAP Transit FastE GigE
• The only way to communicate between the end user and the web server is over the transit connection • Obviously a FE cannot pass as much traffic as a GigE, and therefore congests
Enterprise Data Center
CDN Servers
Slide 12 Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 13: The Business Internet
Issue #2: Insufficient Transit
• Until the end users can be redirected to other servers, this congestion will continue • The redirection time is dependent upon DNS TTLs - and the end user applications honoring the TTLs • The first is fully under control of the CDN, and many CDNs work to minimize this problem
– TTLs of one minute or less are common
• The second is obviously not
Slide 13 Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 14: The Business Internet
Issue #2: Insufficient Transit
• Because many web browsers &/or OSes cache a hostname indefinitely (usually until the browser is closed and re-launced), this is can create a long-lived issue • The obvious solution is to get a gigabit of transit at each NAP • This can create financial & political problems
Slide 14
Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 15: The Business Internet
Issue #2: Insufficient Transit
• Most NAPs will not push much traffic over the transit
– It is not uncommon to see a 10:1 ratio or less at a good NAP – Bad NAPs can be far worse, but that is another problem
• Most transit providers require large commitments on transit GigEs • Problem only becomes worse if you push more than 1 gigabit at a NAP
Slide 15 Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 16: The Business Internet
Issue #2: Insufficient Transit
• To get around the financial / political problems, the CDN can do a few things • Intentionally use the transit link to push content to end users
– Lowers capacity to NAP
• Sign national deal with many GigEs, and get aggregated CIR
– Requires much larger commitment to transit provider – Might give lower $/Mbps price
Slide 16
Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 17: The Business Internet
Data Exchange Offer
• Offer is Akamai specific
– Wouldn’t presume to make offer for other CDNs, but that doesn’t mean you can’t
• Akamai has data about performance and other network characteristics which may not be available to a “normal” network • We cannot give out much of this data due to NDAs and other legal restrictions
– For instance, BGP churn data is confidential to the network partners who peer with us
Slide 17
Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 18: The Business Internet
Data Exchange Offer
• However, under NDA, we are willing to share data about your network with you - in exchange for some data on your network we do not have • Specifically, Akamai has data about the TCP sessions underlying our HTTP transactions (what we call “TCP stats”) • What Akamai does not have is data regarding the other protocols (e.g. P2P), or data about ingress traffic
Slide 18 Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 19: The Business Internet
Data Exchange Offer
• If a network provider was willing to provide Akamai with data (e.g. NetFlow), we would be willing to provide our TCP stats regarding that provider’s AS and downstreams • If you have Akamai servers on-net, we can arrange to serve some traffic to ASes other than your own for additional data
– At your discretion, of course
• If you do not have Akamai servers on-net, we can provide data to/from most of the larger ASes on the Internet
Slide 19 Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 20: The Business Internet
Data Exchange Offer
• Only data relating to your AS would be provided • All data you provide would be kept strictly confidential • We are flexible in what data you need to provide
– NetFlow is not a requirement
• Would also like BGP data
Slide 20 Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 21: The Business Internet
Data Exchange Offer
• If you are interested in this data, please contact me Patrick W. Gilmore patrick@akamai.com
Slide 21
Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 22: The Business Internet
Questions?
Slide 22
Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 23: The Business Internet
What is a Content Distribution Network?
• The RFCs and Internet Drafts define a Content Distribution Network, “CDN”, as:
Content Delivery Network or Content Distribution Network. A type of CONTENT NETWORK in which the CONTENT NETWORK ELEMENTS are arranged for more effective delivery of CONTENT to CLIENTS.
Slide 23
Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 24: The Business Internet
What is a Content Distribution Network?
• A CDN is an overlay network, designed to deliver content from the optimal location
– In many cases, optimal does not necessarily mean geographically closest
• CDNs are usually distinct groups of servers in geographically disperse locations
– A few CDNs put servers directly into networks
• Some CDNs are network owned (SAVVIS, AT&T), some are not (Akamai, Speedera, Mirror Image)
Slide 24 Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 25: The Business Internet
How CDNs Work
• When end users request content from a CDN, the user is directed to the optimal server
– This is usually done through DNS, especially for nonnetwork CDNs – It can be done though anycasting for network owned CDNs
• Users who query DNS-based CDNs get different answers for the same hostname
– A user in Tokyo will be directed to a Tokyo web server – A user in London will be directed to a London web server
Slide 25
Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 26: The Business Internet
How CDNs Work
• Example of CDN mapping
– Notice the different A records for different locations: [NYC]% host www.printpal.com www.printpal.com CNAME printpal.speedera.net printpal.speedera.net A 216.73.83.76 printpal.speedera.net A 216.200.68.4 [Boston]% host www.printpal.com www.printpal.com CNAME printpal.speedera.net printpal.speedera.net A 208.185.219.178 printpal.speedera.net A 208.184.139.93
Slide 26 Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 27: The Business Internet
How CDNs Work
• CDNs use multiple criteria to decide where to send the user
– These include standard network metrics
• • •
Latency Throughput Packet loss
– These also include things like load on the local web server, HD space, footprint distribution, etc.
• Geography still counts
– That speed-of-light thing – Users in Sydney will not be served out of Munich because the latency will always be poor no matter how good the network is
Slide 27 Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 28: The Business Internet
Why CDNs Peer with ISPs
• The first and foremost reason to peer is improved performance
– Since a CDN tries to serve content as “close” to the end user as possible, peering directly with networks (over non-congested links) obviously helps
• Peering gives better throughput
– Removing intermediate AS hops gives higher peak traffic for same demand profile
•
Shown on Akamai’s and other networks
– Might be due to lower latency opening TCP windows faster – Might be due to lower packet loss
Slide 28 Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 29: The Business Internet
Why CDNs Peer with ISPs
• NAPs add redundancy & burstability
– Having more possible vectors to deliver content increases reliability – During large events, having direct connectivity to multiple networks allows for higher burstability than a single connection to a transit provider
• Burstability is important to CDNs
– One of the reasons customers use CDNs is for burstability
Slide 29
Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 30: The Business Internet
Why CDNs Peer with ISPs
• Peering at NAPs reduces costs
– Reduces transit bill – Allows aggregation, serving multiple networks from one location increases efficiency
• Marketing
– Some CDNs announce the number of networks to which they connect
• Backup for on-net servers
– If there are servers on-net, the NAP can act as a backup during downtime and overflow – Allows serving different content types
Slide 30 Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 31: The Business Internet
Why ISPs peer with CDNs
• Performance
– CDNs and ISPs are in the same business, just on different sides - we both want to serve end users as quickly and reliably as possible – You know more about your network than any CDN ever will, so working with the CDN directly can help them deliver the content more quickly and reliably
• Cost Reduction
– Transit savings – Possible backbone savings
Slide 31
Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 32: The Business Internet
Why ISPs peer with CDNs
• Increased customer billing
– As stated before, peering directly with CDNs gives higher peaks, and therefore higher 95th percentiles – Not peering with CDNs means traffic may be served to downstreams through another network
• Marketing
– Claim performance benefits over competitors – Keep customers from seeing “important” web sites through their second uplink
Slide 32
Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 33: The Business Internet
How CDNs use NAPs
Peer Network
NAP
Content
• Since some CDNs do not have a backbone (including Akamai), each NAP can be independent • The CDN uses transit to pull content into the servers • Content is then served to peers over the NAP
Enterprise Data Center
Transit Origin Server
Equinix Gigabit Peering Forum IX, September 14, 2004
CDN Servers
Slide 33
Slide 34: The Business Internet
How CDNs use NAPs
• CDNs usually do not announce large blocks of address space because no one location has a large number of servers
– It is not uncommon to see a single /24 from a CDN at a NAP
• This does not mean you will not see a lot of traffic
– How many web servers does it take to fill a gigabit ethernet uplink these days?
Slide 34
Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 35: The Business Internet
Multiple Peering Locations
• Some CDNs do not have a network, so each location is independent • CDNs can still be present at several NAPs • As a result, announcements at one NAP may be inconsistent with announcements at all other NAPs • Although announcing inconsistently is not common practice, there are good reasons to peer with CDNs
Slide 35 Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 36: The Business Internet
Multiple Peering Locations
• The main reason networks give for not accepting inconsistent announcements is that it can force a network to carry the long-haul traffic • CDNs typically do not want to serve traffic over long distances
– The whole point of a CDN is to serve content “close” to the end user – To ensure this, most CDNs will allow you to announce only local prefixes – Most CDNs will also honor MEDs or communities to ensure traffic is served locally
Slide 36 Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 37: The Business Internet
Multiple Peering Locations
• Deaggregate prefixes
– For MEDs or communities to be meaningful, you may have to send deaggregate prefixes to the CDN – What does a /8 mean? – This is especially important if you peer and have onnet servers – Can be important even if you only peer in a single location
•
Suppose another peer has the same downstream?
Slide 37
Equinix Gigabit Peering Forum IX, September 14, 2004
Slide 38: The Business Internet
Akamai Specific Examples
Slide 38
Equinix Gigabit Peering Forum IX, September 14, 2004
