Post

Kafka also had "spurious zero offsets", caused by some clients not recognizing return values >0 as error codes. Craziest bug: values in aborted transactions were visible to other transactions. Caused by a delayed commit getting retried, then the original commit going through. It's a Thing Doer!

"In short, the Kafka transaction protocol has been fundamentally broken for years" Second project: Amazon RDS. Found "G-nonadjacent" cycle in databases, where two read-write edges are adjacent. Violates snapshot isolation properties. Occurred constantly, in healthy clusters, in all versions. #sd25

Caused by how transactions were ordered in the secondary vs primary. Was this documented? RDS said "yes, it's documented in a 2013 psql-hackers newsgroup posting". Final project: @tigerbeetle.com, a database "nobody has ever heard of" (Tigerbeetle is hosting the conference)

Tigerbeetle is exclusively for financial transaction-looking data. Everything is either an account or a transfer. Every transaction gets a strictly increasing ID in order of execution, and is strongly serializable. Uses flexible quorums, which come from Heidi Howard, presumably on a random Tuesday

Strong serializable timestamps made it real easy to look for issues, just look for graph cycles over the realtime-transaction id edges. To test this made a cleanroom Tigerbeetle state machine. Then realtime order refines timestamp order refines state machine correctness. Found crashes instead #sd25

More specifically, it segfaulted the JVM, due to FFI with Zig, which isn't garbage-collected but has memory protection. Sometimes client requests would never timeout, because of tigerbeetle guarantees of never timing out. Found elevated latency due to single-node faults, due to ring replication

Only one true safety bug: sometimes query responses would be missing. Cause was "key pushed out-of-bounds during zig-zag merge join." INCREDIBLY specific bug that didn't cause data loss, just made queries not return everything

Tigerbeetle can recover from helical faults, where ∀byte in the disk, ∃one node, that byte is corrupt. As of 0.16.43, Tigerbeetle is strong serializable, and all found issues fixed, except requests never timeout and a documented upgrade crash in one specific release #sd25

Takeaways: 1. Sometimes your code isn't wrong, the database is wrong 2. Simple tests find surprising results 3. At some point, you want to be testing the final binary, end-to-end. 4. Fault injection, property tests, safety checks all good. @palvaro.bsky.social namedrop (`elle` project)

Users should read literature, read docs carefully, check application safety, and share anything unusual you find. Vendors should read user reports, formalize safety model, communicate it thoroughly, and test invariants. Academics can figure out abstract formalisms for real systems and share them

Making us all look bad by having a "thanks to" for other people who helped him, AS WE SHOULD HAVE DONE

------- "1000x: The Power of an Interface for Performance", Joran Dirk Greef, #sd25 "A million dollars, 16 machines, a race" between good implementations and a good interface. Interfaces expose, and impose, and play a profound influence on how well systems scale.

"We're gonna do a challenge. We're gonna take one of the most valuable workloads in the world today and show that popular database have a performance limit, and this comes from the interface, not implementation", meaning adding more hardware won't improve the limit. Then we'll fix the limit.

We're being told to get our cameras ready.

Standard interface is SQL ∪ over network ∪ trasnsaction processing. "No OLGP or OLAP", specifically ultra-high volume transactions, OLTP in the most restricted sense. Inventor of OLTP Jim Grey proposed in 1985 three benchmarks: scan, sort, and DebitCredit. kicked off optimization race. #sd25

DebitCredit is "rewrite account, write history, rewrite teller, rewrite *branch*." Branch is a "hot account", adding in contention. It's possible, with hot accounts like a company account, for upwards of 90% of transactions to have contention. Can't handle big scale without handling contention.

Example: 14mm people tried to get tickets for Oasis, and they sold out in minutes. Pareto principle: 80% of volume involves 20% of power users. Transactions are not complex queries. Rather, simpler queries but lots of them. In some fields, up 10000x volume since 2015.

We're going to benchmark four databases. "Before we use interfaces to unlock performance, we will use them to LOCK performance". First db, Postgres, OLGP (online general purpose). Have to build out transfers as a general relational table. #sd25

To transfer money from alice to bob, psql first gets their balanes, then adds to alice's debits and bob's balance, then writes an (alice, bob, amnt) record to the transfers table. Will take somewhere between 1 and 10 roundtrips. "How many transactions/sec?" I guessed 2

For low contention, 10k transactions/sec. For high contention, gradually drops to 4 transactions/sec (I was closest!) Postgres super sensitive to network costs: 10ms roundtrips more than 10x worse than 1ms roundtrips. Needs to do row locks across 2 roundtrips, which creates bottlenecks.

"Those who would trade safety for performance deserve neither safety nor performance." What if we use a CLOUD database? They give UNLIMITED* INFINITE* SCALE* These often have the same interface as postgres, but as a proprietary cluster. "Looks like / talks like Postgres", runs on many machines

They'll benchmark the "cluster" on 16 machines that cost 84k/month total, or 1m/year. Instead of 12k transactions/sec under 0% contention, got 7.5k transactions/sec, a little better than half (for a million dollars a year). 10% contention: 570 t/s. 8 machines: 600 t/s. 1 machine: 800 t/s!

Write contention is what separates OLTP from OLGP. At 10% contention, the proprietary cluster never goes above 12% cpu utilization. What about stored procedures? Keep processing in a single in-process transaction without row locks! Benchmark #3: DuckDB. Kinda ill-suited, as DuckDB is OLAP #sd25