Three years of IPv6 at Yahoo

Today marks my last day at Yahoo, almost three years since I started. It’s been a fun ride!

The job

The job took three hats:

  • Steering IPv6 adoption within Yahoo: arguing for IPv6 on services, prioritising work to get to IPv6 at some point rather than never, writing internal standards to guide work prioritisation. Technical work, testing, debugging, and throwing myself under the bus as the domain expert all came in here, to make deployment as painless as possible.
  • Teaching and training: I gave very regular internal and external talks to various groups, including undergraduate classes, research groups, engineering teams, operations teams, management groups, security groups, and C-level individuals. All of these various talks were much like teaching; the core material was the same, but the positioning was different depending on who was listening.
  • Measuring IPv6 adoption: setting up the pipelines to pull access logs and count IPv6 requests, categorised by ASNs, ISPs, countries, regions (all measures of external deployment), and according to Yahoo’s edges (a measure of internal deployment and traffic steering). These brought the IPv6 message in-house, but we also publish some of our stats via the Internet Society, and put out some more detail in a short paper presented at ANRW 2016.

It was a great trade-off, and watching IPv6 deployment grow in one of the largest content networks in the world (the 5th largest in September 2016) was a lot of fun.

Of course, the measurement work was important because a lot of the interesting stuff was happening elsewhere. Measuring deployment of IPv6, as observed on Yahoo’s CDN services, was critical.

The Measurements

Three years ago, 3.4% of our traffic was carried over IPv6. Today, our global metrics show that we’re regularly handling 15% of requests over IPv6 at weekends. So we’re talking about an almost five-fold increase in traffic in consumer-facing ISPs, and an equivalent reduction in IPv4.

Back then, there were far fewer ISPs carrying significant IPv6 traffic. Verizon Wireless, Comcast, Deutsche, Free, and Swisscom all had significant deployments (over a third of their traffic was IPv6 at the time), and other networks such as T-Mobile (US) (hovering around 7% IPv6) were picking up. This is the post-IPv6-Launch event world, where ISPs were getting comfortable with IPv6 for all customers.

We’ve seen so many ISPs ramp up their IPv6 deployments since then. Comcast is now consistently 50% IPv6, approaching 60% at the weekends (work-week hours drive IPv4 traffic in many fixed-line networks, something we identified in the ANRW paper). Some networks are pushing harder: T-Mobile (US) is hitting the 80% mark, as is Verizon Wireless. These networks in particular help boost our recent assertion that mobile traffic in the US is now majority IPv6. Another big jump was Sky Broadband in the UK, who successfully ramped up their IPv6 deployment to level out around 78% IPv6. In a domestic, fixed-line ISP, that’s strong work, and it’s a huge reduction in our IPv4 load.

Many ISPs operate within a given jurisdiction or market, and therefore countries are a natural way to aggregate the traffic stats. We have so many countries with traffic today that had almost none in 2013.

At the country level, in 2013 traffic originating from ASNs registered in the US was close to 7% IPv6; Germany was around 8%, Romania was around 11%, and Switzerland was around 16%, but many were near-zero. So small compared to today. Now, the US is consistently around 30% IPv6 to Yahoo. Sky’s light-up brings the UK to around 15% IPv6. Belgium these days is over 40% IPv6, and Germany’s a comfortable 20%.

Building out a list of countries where we see more than 5% of traffic carried over IPv6, here’s what we see, now and then:

Europe:

  • Belgium (50-58%, then 3%)
  • Austria (12%, then <0.1%)
  • Czech Republic (10%, then 1%)
  • Germany (20%, then 10%)
  • Estonia (20%, then <0.1%)
  • Finland (16-20%, then 0.5%)
  • France (10%, then 8.3%)
  • Greece (20-25%, then 0.3%)
  • Luxembourg (12-22%, then more consistently 12%)
  • The Netherlands (10%, then 0.9%)
  • Portugal (20%, then 0.9%)
  • Romania (5-9%, similar then)
  • Switzerland (20-30%, then 9.9%)
  • The UK (13-18%, then 0.2%)

Asia/Pacific:

  • Australia (6%, then <0.1%)
  • India (8%, consistently <0.1% until four months ago)
  • Japan (4%, then <0.1%)
  • Malaysia (10%, then around 1%)
  • Singapore (5%, then around 2-3%)

Americas:

  • Brazil (8%, then <0.1%)
  • Canada (13%, then 0.4%)
  • Ecuador (12%, then <0.1%)
  • Trinida and Tobago (9%, then <0.1%)
  • The US (28-33%, then 8%)

For the sake of clarity, country stats are derived by taking the origin ASN for each request and mapping it to the country code the origin ASN is registered to (according to data published by the registries: AfriNIC, APNIC, ARIN, LACNIC, RIPE NCC); this is a rough but simple measure of where traffic is coming from.

Obviously, countries vary wildly by population, and India’s recent uptick in IPv6 traffic is more significant – in terms of traffic, and where you handle it – than most of the European countries listed, but most of these are strong showings. IPv6 is on the way up.

In Summary

The key points from some of our measurement are basically:

  • We have ISPs today that are majority-IPv6. Consider this when thinking about how or where you serve your content.
  • ISPs can light up quickly, when they want to. Dominant ISPs will tip a country’s or region’s IPv6 load.
  • Given the above, a country’s IPv6 share may appear to stagnate; some countries have not seen significant additional deployment in the last three years, while others have seen concerted efforts (for example, in Finland).
  • The set of ISPs deploying IPv6 is clearly not stagnant, given the number of countries now bringing non-trivial volumes of IPv6 traffic to content networks.

In other spaces, key moves are encouraging. Apple’s IPv6 requirements for app developers, and Amazon’s announcement of IPv6 for S3 are big moves forward, reducing the number of IPv4-only services that engineers are exposed to, or writing code for.

But from my perspective inside Yahoo, that’s that. In December, I start a new gig, measuring the network from different perspectives. Including, of course, the IPv6 perspective!

The year that was 2015.

2015, like 2014, was a stable year.

I’ve been at Yahoo for over two years. I’ve been living not just in San Francisco, but in the same apartment, for over three. I’ve been working for bay area companies for almost four. This stability for so long makes me fidget.

A lack of travel early in the year exacerbated that feeling: travel was especially light at the start of the year. I had the shortest trip to LA in January, and made it until May until I had (had) to book a last minute nine-day trip to Mexico City. I might go back; the set of reachable, cost-effective destinations from SFO that work as a vacation for me (usually: big cities, culturally removed from my norm) is alarmingly small.

Starting August, I squeezed London (for SIGCOMM and a visa renewal, a short break to Montreal, then finally Tokyo then Yokohama for IMC then RAIM and IETF 94 respectively in November. Upcoming travel? This year I’ve already passed through Portland, and I have another mini trip to Seattle planned. As for later in the year though? I’m not sure. IETF 96 is in Berlin, and SIGCOMM is in Salvador, Brazil (though I’m considering switching it out of my rota), and IMC is in the least exciting of the set so far, Santa Monica.

On running: I logged 1,072km in 2015 (a reasonable 666 miles, if you’re into US measures). Two big things contributed to that total: first, being stationary at the start of the year gave me time to run; second, I set myself the challenge of running a marathon. On the way toward that, I completed Bay to Breakers again, beating last year’s time by 4 minutes 41 seconds. My weekends were then spent gradually working up to distances over 30km in prep for the full 42km effort and, in the end, I came in with a reasonable time of 4 hours, 10 minutes, 23 seconds. I had quietly hoped for a sub-4-hour effort, but this was good by me. Odds of running another marathon any time soon? Slim to none. But I plan to at least run Bay to Breakers again this year, and certainly complete a half marathon race. I’m also planning to ease into cycling after about a decade of regular running, and I’m reading everything about bikes at the moment.

So, 2016, and what it’ll offer, is a bit of a mystery even to me. I have a feeling I’ll have to shake something loose, change something up. What that turns out to be might only be visible in hindsight. Let’s see, shall we?

IPv4 Occupancy, May 2015

Following on from last year’s post on how much of the IPv4 space is advertised in BGP now, an update for 2015.

This time around, I’ve pulled more data; rather than one table per year, I’ve used one table per month. Otherwise, I’m calculating space in the same way as last year: pulling out the prefixes advertised over BGP, counting how many unique addresses are advertised, and tying them to either a RIR or an outright legacy allocation. I’m using the same RIR allocations as last year which is almost completely accurate. Good enough for here, and for comparison.

As before, the way the address space is carved up means there’s a potential maximum of around 3.7 billion IPv4 addresses available. Occupancy as of May 31st 2015 looks like:

RIR /8s available /8s advertised /8s free % advertised delta% 2014
ARIN 79.67 56.93 22.74 71.16% +1.94%
APNIC 51.00 43.63 7.37 85.56% +2.03%
RIPE NCC 39 35.59 3.41 91.25% +0.42%
LACNIC 10 9.30 0.70 93.00% +3.29%
AfriNIC 6 3.46 2.54 57.79% +4.95%
Legacy 35 15.95 19.05 45.59% +3.65%
Total 220.67 164.87 55.80 74.71% +2.21%

That 2.21% increase in advertised space is approximately equivalent to five additional /8s being introduced in the last year, and it’s pretty close to the same rate of change we’ve seen since the middle of 2011 when APNIC started to level out.

Here’s the time series of the actual space advertised by total and by region:

absolute growth of RIR space between May 2002 and May 2015

Here’s the time series of the proportion of space advertised by total, and by region:

relative growth of RIR space between May 2002 and May 2015

For these, I’ve filtered some obviously-bad advertisements; the sort that are probably RIRs checking space prior to putting it into their pool.

There’s a lot these images don’t show: I’m missing a lot of detail in those legacy blocks that might now be better allocated to RIRs. I’m not saying anything about how many of these addresses are or are not being advertised from the same source as previous samples. But really, the basic question I want to answer is: how much space is in use, and where.

What’s interesting is how advertisements in the APNIC region have been less aggressive since around the point APNIC ran out of space to allocate freely, and started rationing allocations. Advertisements in Europe have also been pretty stationary since RIPE ran out in 2012. Same deal with LACNIC space. The suggestion in each case is that addresses are allocated by a RIR and are almost immediately put into use.

I’m curious about who’s hanging onto space and not using it, or if it’s just gone missing; there’ll be address space that folks have forgotten about, possibly as companies have been merged or acquired. Even so, 90%+ in the RIPE and LACNIC regions is extremely full.

ARIN still has a ton of space not in use, and as I write, they’re listing 0.00978 aggregate /8s available to distribute from the last of their rationed space. Given they were at 0.7 only three days ago, I imagine there’ll be an announcement in the next day or two that they’re out.

Once the RIRs run out of free space to allocate, they’ll be able to issue space only from space returned to them or by managing transfers. I’d imagine that the removal of the top of the food chain will encourage the nascent v4 address market; one less supply source as demand continues to increase.

There’ll be enough spare in that address space to shake out for a while yet: fragmenting of larger blocks into /24s as companies put up real money just to buy space and stay connected to the old network will see us through for a time, but it won’t be cheap or pretty. IPv6 growth is still increasing rapidly, but I’m still not willing to guess which year we’ll see advertised IPv4 space start to go back down.

Recent Posts

(full archive)

All content, including images, © Stephen D. Strowes, 2000–2016. Hosted by Digital Ocean.