Post

I worked on a few fairly prominent efforts in open source (Istio, OpenAPI), but I think of all of them, it’s gRPC that I think has the coolest story. Pull up a chair and set for a spell (any inaccuracies in what follows are 100% my fault for having a bad memory!).

It was 2014. Cloud was really starting to be a thing at Google. I was working with Louis Ryan on Google’s API infrastructure team, Apiary (pre-dating and unrelated to Apiary, the API company).

At the time, it was probably one of the most high performance, high scale API proxy in the world. Handling over 1M qps (as we called requests per second internally) for a couple of hundred APIs, 90th %ile latency of around 8ms. It was really, really high performance and a fun team to work on.

Everything from Maps to Drive to Translate ran APIs on that infrastructure, which handled all authn/authz, logging, API key and OAuth token infra, routing, protocol translation (JSON/REST to Stubby), etc. And we were pretty proud of what it could handle.

For those types of APIs, that 8 extra ms (and a hop through the proxy) were well within a tolerable latency budget. After all: when you’re loading a map on your mobile device, translating a phrase into Spanish, or fetching a Google Doc, 8ms won’t even be noticed.

But with Cloud, we were starting to look at a new type of API. We were looking at database APIs like Spanner. We were looking at message queues. And those APIs bring new kinds of volume, and new kinds of latency tolerance.

We knew that latency would have to improve by at least an order of magnitude (for things like DB calls, you want to measure your latency in 𝜇s, not ms).

And we knew that volume would REALLY need to go up (I remember only a few years later talking about an individual customer calling an API at nearly 1M qps!).

All of this meant that we needed a complete rewrite of our API proxy. It was at this time that we decided to move from a proxy to a sidecar (oooh -- maybe I’ll have to do a thread about THAT some day), but there was something more fundamental than where the logic lived.

Internally at Google, we used the proprietary Stubby protocol for service to service calls. Proto on the wire, hard contract-driven, extremely efficient. We knew that JSON/REST worked fine for the “typical” web APIs we had been serving, but again, think about the new types of API we’d be doing.

For a high performance database, JSON just wasn’t going to work. It adds latency and eats bandwidth to transcode values in memory into JSON, serialize, push that across the wire, then do the same thing on the other end. Louis knew from experience that proto was good for this.

People wouldn’t want to invest in integrating with our APIs if we were using a different protocol than everyone else! (As an aside, I started my career writing device integrations before HTTP became common, and let me tell you: each integration was a byte-by-byte nightmare!)

So if we really wanted to be able to build API infrastructure to handle this new breed of APIs, we’d have to open source the protocol. I still remember the day Louis Ryan and I presented to Urs, and Louis got the money line. “This means we have to open source a version of Stubby.” Urs agreed.

So Louis and Andy Patti designed it. They decided to use HTTP/2 to get features like bi-directional streaming. This was pretty high profile -- Sanjay Ghemawat himself was approving their designs and their CLs!