Post

------- "A Systems View to AI", Amod Malviya, #sd25 AI Systems are fundamentally different from traditional systems. To understand why, we have to define "systems": a set of interacting components in an circumscribed boundary with input and output. Natural systems come from emergent behavior

Human systems are designed. AI systems are designed, but they have emergent behavior. In trad systems, we use abstractions to get composability, modularity, stacking, all intended to make reasoning about them tenable. Each intends to deal with faults, like modularity isolates faults to units.

Composability reduces emergent faults. Each abstraction reduces fault models, but AI has different fault models. System design is about recognizing fault models and engineering Universal approximation theorem: a sufficiently large neural network can learn any function.

Quick explanation of how LLMs work: giant next token prediction networks. Usually connected to an orchestrator, which appends that next token and feeds it back to the LLM, along with instructions that influence the next token. This creates a context window. #sd25

We can only modulate the bheavior of a generative AI by either changing the training of the LLM or changing the context window. Training starts with pretraining to get all the data in, then supervised finetuning and "preference optimization". Last two are "posttraining"

The fault models of AI are different. Normally we think nondeterminism as something to avoid. AI are inherently nondet, as they have open-ended outputs. Older AI is probabilistic, but deterministic in a given input.. GenAI focuses on what's "possible", not "correct". Called `temperature`

Hallucination "is a bug, not a feature". It only bothers us when it violates correctness. One way to fix this: output verification. Check output against an oracle. For example, we can verify code by trying to compile it, get information-rich compiler errors. and feed those back into the LLM #sd25

Big challenge for AI researches is to figure out how to verify outputs in various domains. Can also improve hallucination with better training, but this doesn't scale. Off the cuff estimation, getting from 3% hallucination to 2% hallucination via training is 10x more expensive. Ok in finetuning tho

It is possible to reduce hallucination with better prompts. But a system that sensitive to the prompt is definitionally a very fragile system! LLM also has an architectural fault model. Think of LLM as CPU, system prompt as program, user input as instructions. Then control plane == data plane!

This is why prompt injection is possibly unsolvable. Also means that eventually, the system prompt is going to leave the context window and you lose your "program". "Context window overflow". There are also fault models that stem from emergent behavior. #sd25

You could train a model in a way that a special sequence of words triggers special system behavior. You can go through the source code of traditional code, you can't go through the training data of an open-source model. Also, LLMs are weirdly anti-modular. specialist models << generalist models

Composability seems like it happens from MCP, but this is insanely insecure rn, because LLMs are anti-modular and does all reasoning centrally. You can't isolate sensitive data to one part of the LLM @simonwillison.net namedrop (the trifecta post he literally wrote four days ago)

A SHOCKING number of people have asked me about vegan food that isn't kosher

------- "Jepsen 18: Serializable Mom", Kyle Kingsbury/@aphyr.com , #sd25 Engineers are tasked with building towers of abstraction, building everything higher and higher above the towering tire fire that is databases. "I professionally set those tires on fire".

Built Jepsen, a tool for testing databases. Open source, anybody can run it themselves. Works only on binaries, no code analysis. Does fault injection. Designed for concurrency, uses generative testing (randomized inputs, check properties hold).

JVM program opens ssh connections to databases, connects clients, and runs commands against them while also messing with IP tables and the disks and connections and so on. Also running generator which produces abstract terms like "read x", "kill db1", "set y=3".

Clients execute the terms and put the results in a history log. Can see if the history log has certain properties, like "repeatable reads". Distributed systems can do one of three things: succeed, return an error, not return at all. Gonna talk about past projects. #sd25

bufstreamIO [uses/is Kafka?] Kafka has some wierd properties, like you can insert a write in the middle of a read. Going through some of the bugs he found with jepsen, like a lease expiration never getting transmitted These slides are v good btw

Kafka also had "spurious zero offsets", caused by some clients not recognizing return values >0 as error codes. Craziest bug: values in aborted transactions were visible to other transactions. Caused by a delayed commit getting retried, then the original commit going through. It's a Thing Doer!

"In short, the Kafka transaction protocol has been fundamentally broken for years" Second project: Amazon RDS. Found "G-nonadjacent" cycle in databases, where two read-write edges are adjacent. Violates snapshot isolation properties. Occurred constantly, in healthy clusters, in all versions. #sd25

Tigerbeetle is exclusively for financial transaction-looking data. Everything is either an account or a transfer. Every transaction gets a strictly increasing ID in order of execution, and is strongly serializable. Uses flexible quorums, which come from Heidi Howard, presumably on a random Tuesday

Strong serializable timestamps made it real easy to look for issues, just look for graph cycles over the realtime-transaction id edges. To test this made a cleanroom Tigerbeetle state machine. Then realtime order refines timestamp order refines state machine correctness. Found crashes instead #sd25

More specifically, it segfaulted the JVM, due to FFI with Zig, which isn't garbage-collected but has memory protection. Sometimes client requests would never timeout, because of tigerbeetle guarantees of never timing out. Found elevated latency due to single-node faults, due to ring replication