Post

DeSantis tells the story of how original EC2 performance was subject to jitter, as a result of the virtualisation mechanism. To solve this, they built AWS Nitro, the hardware virtualisation platform that underpins the whole AWS cloud.

The latest edition of compsci classic "Computer Architecture: A Quantitative Approach" covers the architecture of Nitro and the AWS Graviton hardware. Neat.

Dave Brown, VP, AWS Compute & ML Services takes the stage to talk about Graviton, Amazon's custom ARM-based chip which delivers excellent price/performance compute on AWS.

As well as custom silicon, Graviton chips are cooled in a custom way too - the lid of the chip is removed, and the heat sink applied directly. This improves cooling efficiency and reduces power usage.

Each Graviton generation is improved based on data about how the previous one performed on real workloads. L2 cache misses in Graviton 3 were having a real impact on performance, so Graviton 4 doubled the size of the L2 cache per core, added more cores, and increased the size of the L3 cache too.

Graviton 4 also provided a coherent link between cores, to enable 192 vCPU cores, but this introduced new latency paths. Graviton 5 improves that, doubling the number of cores, and increasing the L3 cache by 5.3x. These are now available in preview, as part of M9g EC2 instances.

A speaker from Apple tells us they've started using their language Swift to develop backend systems, and deploy these to Graviton. Amazon Linux now has a native toolchain for developing Swift, the first distribution to provide one.

Brown is back, talking through the history of AWS Lambda, initially conceived of in 2013. A couple of years ago, the Lambda and EC2 teams were brought together. It's that coming together that led to the release this week of AWS Lambda Managed Instances.

Lambda Managed Instances allow you to launch Lambdas on EC2 nodes provisioned to the spec of your choosing. You still don't manage the servers yourself. "Serverless is the absence of server management", Brown says, answering that question once and for all.

We're onto AI Elasticity now. AWS built Bedrock using a traditional load balancer / webservice design. Those learnings led to a better understanding of how to build inference platforms. This led to Project Mantle, which underpins many of Bedrock's models today. This is how they offer latency tiers.

As well as latency tiering, Mantle provides fair scheduling, preventing other tenants under heavy load from impacting the performance of your requests.

Requests in Mantle use a request journal, so that failed requests can be tried again, making Bedrock more reliable.

Finetuning capabilities are also logged on the journal. If inference requests spike, finetuning jobs can be paused, and resumed later when load drops off.

DeSantis is back on stage, giving an overview of how vectors and vector databases work in generative AI workloads. AWS have integrated vector capabilities across their range of services.

OpenSearch has vector search capabilites now. It's also now easy to add vector indices to S3 buckets, with the recently GA service, S3 Vectors, the first cloud object storage with native support to store and query vector data.

The problem with S3 Vector indices is that these are stored on disc, which makes lookups slower than in-memory stores. At write time, S3 calculates a dataset of nearest neighbours. When you perform vector lookups, this nearest neighbour data is pulled into memory and queried there - quickly!

Customer TwelveLabs on stage now, talking about how they use AWS' s3 vector capabilites to build models that understand video in the way that humans do, providing plain text search for that video content.

Trainium chips provide efficient hardware for training models. A Trainium UltraServer provides 144 Trainium3 Chips across two racks, with dedicated "neuron switches" providing networking optimsed for model training workloads.

The server sleds in these UltraServers use all three AWS hardware platforms: Nitro, Graviton, and Trainium. Everything on this sled is "top servicable", which means they can be assembled robotically, and that when they need maintenance, this can be performed quickly and efficiently.

The Elastic Fabric Adapters in on these sleds provides high throughput networking across the whole UltraServer cluster.

Trainium has on-chip capabilty for profiling workloads, without impacting performance. Neuron Explorer is a tool for reporting on those performance stats.

Compared with Trainium 2, the Trainium 3 chip can process 5 times the number of tokens per megawatt. This performance was recorded using the aforementioned profiling tools.

In Q1, Trainium will be becoming PyTorch native, so that you don't have to learn a new platform in order to make use of Trainium hardware. Just target 'neuron' instead of 'cuda' in that code.