Post

I wanna hear the untold stories of the huge architectural and data migrations that go into the “smallest” changes like when Slack added the ability to add someone to a group DM and *move past messages* to the new conversation or how Amazon no longer asks which Kindle to send your library book to

you gotta assume that these were both enormous efforts that were probably years in the making. all to make it work the way it should’ve worked in the first place

put another way: I'm interested in hearing about times when people modeled their data in the wrong way and why

2 reposts22 likes

Worked on a fitness band. Beta data flow involved phone processing the raw data and writing to backend through graphql mutations. It was slow and user friendly data took awhile to sync. PE and a few engineers built a system that sent protobuffs to backend for processing a few months before launch.

i have heard stories of a large mysql cluster, and a system wherein "the schema that we used is the schema for your account" pretty much forever the vast majority of the data is write once read maybe and so backfilling is a huge waste of resources, so it all gets adapted per storage unit

the thing is, once you hit a certain scale, your logical and physical schema inevitably diverge, and the notion that your data exists in one consistent layout just stops being true and truthfully speaking, data migrations aren't really an issue once that happens, the problem is doing it all at once