Post

I worked on a few fairly prominent efforts in open source (Istio, OpenAPI), but I think of all of them, it’s gRPC that I think has the coolest story. Pull up a chair and set for a spell (any inaccuracies in what follows are 100% my fault for having a bad memory!).

It was 2014. Cloud was really starting to be a thing at Google. I was working with Louis Ryan on Google’s API infrastructure team, Apiary (pre-dating and unrelated to Apiary, the API company).

At the time, it was probably one of the most high performance, high scale API proxy in the world. Handling over 1M qps (as we called requests per second internally) for a couple of hundred APIs, 90th %ile latency of around 8ms. It was really, really high performance and a fun team to work on.

Everything from Maps to Drive to Translate ran APIs on that infrastructure, which handled all authn/authz, logging, API key and OAuth token infra, routing, protocol translation (JSON/REST to Stubby), etc. And we were pretty proud of what it could handle.

For those types of APIs, that 8 extra ms (and a hop through the proxy) were well within a tolerable latency budget. After all: when you’re loading a map on your mobile device, translating a phrase into Spanish, or fetching a Google Doc, 8ms won’t even be noticed.

But with Cloud, we were starting to look at a new type of API. We were looking at database APIs like Spanner. We were looking at message queues. And those APIs bring new kinds of volume, and new kinds of latency tolerance.

We knew that latency would have to improve by at least an order of magnitude (for things like DB calls, you want to measure your latency in 𝜇s, not ms).

All of this meant that we needed a complete rewrite of our API proxy. It was at this time that we decided to move from a proxy to a sidecar (oooh -- maybe I’ll have to do a thread about THAT some day), but there was something more fundamental than where the logic lived.

would love a thread about this. I assume envoy was involved but would love to hear your perspective.

Internally at Google, we used the proprietary Stubby protocol for service to service calls. Proto on the wire, hard contract-driven, extremely efficient. We knew that JSON/REST worked fine for the “typical” web APIs we had been serving, but again, think about the new types of API we’d be doing.

For a high performance database, JSON just wasn’t going to work. It adds latency and eats bandwidth to transcode values in memory into JSON, serialize, push that across the wire, then do the same thing on the other end. Louis knew from experience that proto was good for this.