Gordon

I wrote a post on making an open source, multi tenant, vector + full-text search engine on S3 using Lance and foyer. Blog post: gordonmurray.ie/data/2026/05...

S3 is the perfect place to store data, until you try to search it. I built Firn: open source, multi-tenant vector + full-text search on S3, inspired by turbopuffer. foyer (RAM/NVMe cache) + LanceDB (S3-storage) + axum REST API. 100k vectors, 350,000x speedup github.com/gordonmurray/firnflow

I wrote a thing: "From a custom Flink connector to a $600k windfall: Building a 6-tier streaming lakehouse" gordonmurray.ie/data/2026/03...

I really like using fly.io for small projects. I like knowing cost up front though, so I wrote a CLI tool called flycost. Like www.infracost.io but for fly.io. It reads your fly.toml file and gives a monthly cost estimate I’d love feedback from anyone running on Fly.io github.com/gordonmurray...

I built a simple to-do list app called www.notebare.com. All the data is stored in Lance format from @lancedb. Performance seems quick, Anyone can try it out, you can sign in with Github. Feedback more than welcome!

Cleaned up my old project: CDC from MariaDB → Flink → back into MariaDB using Docker Compose. Flink’s CDC connectors (powered by Debezium) keeps the target table up to date in real time. Code on GitHub: github.com/gordonmurray...

Apache Fluss is a storage engine: • Fast key lookups (e.g. user status) • Instant historic queries (via Paimon on S3/ORC) • Works with Flink SQL for heavy lifting • Tablets replicate & recover automatically Made a small POC to try it out: github.com/gordonmurray...

I’ve refreshed a small hobby project from ~2 years ago: Apache Flink & Paimon. It picked up a few early stars back then. I’ve cleaned it up so it’s runnable again. I’m hoping to experiment with Apache Fluss on top of this. Code on Github github.com/gordonmurray...

I took some time this weekend to learn TLA+. It lets you test distributed systems by exploring all possible states. Meaning you can catch bugs before production and likely save costs. I built a small FastAPI + MinIO demo to try it out. Repo: github.com/gordonmurray...

I tried to build a minimal image vector store using Lance and Cloudflare R2. I Learned a lot, hit a hard limit fast. "When Lance hits a wall at 70 images" gordonmurray.com/data/2025/05...

Oooh www.iacconf.com

Trying out Lance for storing vector data on R2. I’ve got Python saving the image and its CLIP embedding to a Lance dataset on R2. It’s append-only, columnar format kind of like Parquet, but optimized for updates, metadata, and search. Compared to Iceberg, feels like a better fit for vector mappings

I made a thing: upload an image, get back a vector embedding. That’s it. Built with Python, FastAPI, and OpenAI’s CLIP model (ViT-B/32). Deployed on Fly.io. quickvectors.com Want to try it out? DM me for an API key.

Built a small DAM-style pipeline to handle image uploads, async processing, and ML enrichment. No DB, just new tools I’ve been meaning to try (fly.io, R2, Parquet). Wrote a post about what I’ve built so far and what’s next gordonmurray.com/2025/05/05/B...

Built a Python pipeline to upload images to R2 and generate captions + vectors using BLIP and CLIP from Hugging Face. Metadata stored as Parquet. Code on GitHub: github.com/gordonmurray...

Trying out Fly.io and Cloudflare R2 - long overdue on my part. Made a small Python app to receive an image, store it in R2 with Parquet metadata, and query via DuckDB+SQL. Very basic, but next I want to try R2's data catalog / managed Apache Iceberg. Code on github github.com/gordonmurray...

I created a small Terraform project that will set up Deepseek R1 for you on your own GPU EC2 instance, seems to be uncensored too. Code on Github at github.com/gordonmurray... I took the steps from this great AWS Community guide community.aws/content/2sEu...

Updated my BentoML project to include Milvus for storing and searching vectorized images. More to learn about L2 Euclidean distance for better search results github.com/gordonmurray...

Just finished a small project on deploying BentoML on ECS using Terraform. Perfect for creating vectors from images with a CLIP model I’ve been learning. If you’re getting in to machine learning I'd recommend BentoML, hopefully this is a useful starting point! github.com/gordonmurray...

Its surprising how many requests and countries try an API endpoint that has only just come online in the last few minutes, using a brand new domain name just registered too.

Tried out BentoML this evening to vectorize images, and it works really well! I expected to need a GPU, but it runs smoothly on a CPU. It even exposes Prometheus metrics out of the box. Code on GitHub: github.com/gordonmurray...

Ah this is great, I didn’t know this was available. The DDB approach was fine but this would be nicer

Re:Invent hasn't even officially started yet and we've already got 22 announcements. No rest on Re:Invent Sunday, I guess. To help you keep track, I will curate a number of custom feeds. The first one: All Re:Invent 2024 Releases Check it out below 👇

The “show vehicle motion cues” option on iOS really works! For me at least. I can read away as a passenger and not feel queasy. Really cool 📖

If you’re thinking without writing, you only think you’re thinking.—Leslie Lamport

Oh nice. I didn’t know that. Thanks! I have some repos to go and update

Today is as good day to start learning Terraform

Hmm 🤔 Maybe I just created the AWS News Firehose on Bluesky. Maybe it's hosted at https://buff.ly/4g3kQsK 🦋 And maybe it's related to Bluesky's recent posts-as-comments demo... 🤷‍♀️ Who knows!

Very cool resource for learning about Apache Iceberg. I’d like something similar for Apache Hudi too “Ultimate Directory of Apache Iceberg Resources” medium.com/data-enginee...

Ooh that’s kinda cool. CDC from MySQL directly to Iceberg on S3 using AWS Firehose. I’m a big fan of debezium though this could be cool to try out. It’s might lend itself nicely to be managed in Terraform too. aws.amazon.com/blogs/aws/re...

Noice

Ah nice, just needed to add a TXT record to my domain name to use my own domain on bluesky

Excellent!

folder gotchas (in the terminal) wizardzines.com/comics/folde...

Got an OpenSearch Pipeline up and running, CDCing from DynamoDB to OpenSearch. It keeps an index perfectly in sync. Used Terraform and manual fiddling to get it going 100%. Couldn't create or import an existing pipeline with Terraform unfortunately. Code on Github github.com/gordonmurray...

Got some burning #Debezium / CDC question you never dared to ask? Then join me for a live AMA session next week, and I'll try my best to get it answered! Sign up at dcdbl.co/3YCYx5Z, would love to see you there 🤓. 🗓️ Nov 14 ⏰ 9-10am PST, 5-6 pm UTC

A serious read. Glad I listened to it on audible or I don’t think I’d have finished it. Incredibly sobering book to read especially as a parent www.goodreads.com/book/show/58...

I made an infra engineer starter pack. Folks posting about databases, stream processing, durable execution, orchestrators, service meshes, and more. go.bsky.app/SCZe42X

I haven't worked on CDC type stuff in a while. I'm trying out streaming changes from DynamoDB tables to OpenSearch. I started a small project to try it out, using Terraform. I still need to add the pipeline and IAM role to get it functional github.com/gordonmurray...

Nice, I think the co2 levels in my office are ok 👌

Ooooh look at that keyboard, could launch rockets with that thing 🤩 mechboards.co.uk/products/hyp...

Bought a new kindle recently and Amazon thought I’d like some book accessories 😅

Oh cool, I was only thinking the other day if this was possible to do, or worthwhile doing - to pull twitter data in to bluesky. I must try this out

I've copied @chris.blue's starter pack of folk into a list here: bsky.app/profile/did:.... IIUC you can add this to your account and it'll create a feed of post from all these folk. Useful if you want a data-only timeline separate from other ppl you follow :)

Really impressed with Logstash. I've been using it recently to move some data between Opensearch clusters and to S3. It has a tonne of outputs that would be great to try out some time www.elastic.co/guide/en/log...

Ha, very cool. An alternative to Chef's Inspec which has gone commercial: CINC Auditor, a drop in replacement thats open source for checking/validating resources cinc.sh/start/auditor/

Turns out that cucumber and pineapple, with a bit of coconut oil and lime tastes pretty nice! Refreshing

I created a software developers stater pack to help anyone looking for folks here go.bsky.app/Jn1NbRv