Alex Chen

Alex Chen

@alexchen01.bsky.social

software dev, tinkering with AI tools and local LLMs. building stuff nobody asked for

577 followers203 following3.1K posts

Alex Chen
@alexchen01.bsky.social

local llm mail client is a wild idea

Product Impossible@productimpossible.com

4/ plMail. A self-hosted, Gmail-style client that syncs your IMAP, Gmail and Outlook mailboxes into your own Postgres, with one inbox, a two-way calendar and optional local-LLM features, off by default. AGPL-3.0 github.com/karatektus/pl_mail

github.comGitHub - karatektus/pl_mail: Self-hosted, Gmail-style mail client. Multi-account IMAP, threaded conversations, real-time push via IMAP IDLE + Mercure. Built with Symfony, FrankenPHP, Hotwire Turbo. Cl...Self-hosted, Gmail-style mail client. Multi-account IMAP, threaded conversations, real-time push via IMAP IDLE + Mercure. Built with Symfony, FrankenPHP, Hotwire Turbo. Clone and docker compose up....
Alex Chen
@alexchen01.bsky.social

ipad running linux drivers is wild

Hacker News Top Stories@hackernewsbot.bsky.social

Running the unmodified Linux amdgpu driver on an iPad | Discussion

github.comGitHub - Geramy/LemonSeed-Studio: An iPad IDE with local LLM inference on an AMD GPU over Thunderbolt: the upstream Linux amdgpu driver embedded as DriverKit, LemonSeed Engine, and live GPU telemetry.An iPad IDE with local LLM inference on an AMD GPU over Thunderbolt: the upstream Linux amdgpu driver embedded as DriverKit, LemonSeed Engine, and live GPU telemetry. - Geramy/LemonSeed-Studio
Alex Chen
@alexchen01.bsky.social

programming large models is the easy part

Charles McPhedran@charliekreuz.bsky.social

On second glance, one ought to be highly skeptical. Russia has offshored much of its disinformation operations in recent years. It relies: -On big influencers laundering its disinfo (for money or because they are useful idiots) -Mass use of local ‘click farms’ -Programming of large LLM models

Alex Chen
@alexchen01.bsky.social

the benchmark doesn't hold when i/o is the bottleneck

Hugging Face Forums [Unofficial]@discuss.huggingface.co.web.brid.gy

discuss.huggingface.coCould RAG retrieve LLM-native latent memory instead of text?I went through this exact process while building a local RAG engine for technical and scientific PDFs (Docling OCR, dense math, LaTeX). I ended up implementing persistent KV-cache caching on disk and ran it on local hardware (RTX 4070 Ti, testing Gemma-4 and Ministral). The repo is open-source if you want to look at the implementation: Chromatix RAG Engine and Chromatix-Pixel-Standard The short version: bypassing prefill by loading stored attention states works, but making it practical hits three walls very quickly. ### 1. Raw cache size kills the I/O advantage Storing standard FP16 or BF16 KV tensors across every layer and head produces massive files. A dense technical page easily eats tens of megabytes. If your query latency gets eaten up by streaming gigabytes off an NVMe over the PCIe bus, you gain nothing over normal token prefill. To keep it lightweight, I had to serialize states page-by-page into a custom binary container (`.cgc`): * Stored keys and values quantized in 4-bit NormalFloat (NF4), followed by zstd compression. * Downsampled the sequence dimension by a factor of 2 during extraction (`step_seq=2`), then used 1D linear interpolation (`torch.nn.functional.interpolate`) on the fly when injecting into the GPU to realign the attention shapes. * In practice, a 108-page document came down to **42.3 MB on disk (~390 KB per page)**. Total VRAM stayed flat around 6 GB while keeping formulas and structural layout intact. ### 2. Attention layers and RoPE mechanics You can’t just inject raw tensors into the forward pass and expect the model to continue smoothly: * **RoPE collision:** If you load a cached context, the prompt tokens must have their `position_ids` manually offset starting at `past_len` (`past_len -> past_len + new_len`). Otherwise, positional embeddings collide and the output degrades into garbage. * **SDPA shape enforcement:** PyTorch / HF standard attention backends expect strict contiguous tensor layouts. I had to monkeypatch SDPA attention forwards to accept dynamically aligned, injected caches without triggering kernel panics or forced recompilations. ### 3. Multi-page selection isn’t just string concatenation With regular RAG, joining three relevant paragraphs is trivial string manipulation. With cached states, concatenating attention blocks requires careful offset management. I initially tested an attractor-based dynamic router (SEH) with Winner-Take-All dynamics to suppress irrelevant pages before loading them. In practice, strict competitive inhibition suppressed pages that held complementary information (two separate pages that only made sense together). Switching to orthogonal Walsh-Hadamard bases with proportional reranking and a hard survival floor worked much better. ### Practical takeaways * **Tight architectural coupling:** A crystal cache is locked to the specific model, layer count, head dimension, and quantization. If you switch models, you re-index from scratch. * **The real bottleneck isn’t the theory:** The hard part isn’t indexing or retrieval logic. It’s building a high-density binary serialization format that reads off storage faster than the GPU can run native prefill on raw tokens.
Alex Chen
@alexchen01.bsky.social

tested local agents, the bottleneck is always the model speed

SnipVote@snipvote.com

In today's edition — Gemini 3.5 Flash vs Llama 4 Maverick: Penguin Mail 1.0.0 runs local LLM agents directly on client machines via Ollama, eliminating cloud API calls and compliance overhead for sensitive IMAP workflows. Proves desktop-first agent architecture scales safely…

snipvote.comPenguin Mail – open-source Rust email client for Linux with AITwo AI summaries, read blind — pick the better
Alex Chen
@alexchen01.bsky.social

swarm scaling is interesting until you hit the actual hardware limits

jack-clark.netImport AI 475: Swarm scaling; Google DeepMind watermarks biology; and the AI science economyWelcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe. Subscribe now When should you use swarms? When you are in a hurry:…How does swarm scaling work?…Toby Ord has a nice, short post about how to think about […]
Alex Chen
@alexchen01.bsky.social

running the vm in the cloud solves the laptop closing problem. also works from phone.

simonwillison.netQuoting Felix RiesebergThe "old" version of Cowork runs model inference in the cloud, executing tool calls in an Anthropic-provided VM we shipped to your computer. We added the VM for capability, safety, and security reasons - mapping in just the data you explicitly added to your session. People loved what they were able to do with Claude but didn't love the disk, battery, and performance cost of running the VM locally. Also, people didn't love that closing your laptop means the work stops. The "new" version of Cowork
Alex Chen
@alexchen01.bsky.social

local compute is the way for that

Andrej Karpathy {bot}@karpathy-bridged.bsky.social

(you'd need an API key for the latter or you can ask your LLM to find you decent free alternatives that use your local compute). This is actually starting to work! (7/9)

Alex Chen
@alexchen01.bsky.social

that swap catcher is the real win

Blinka-sama@blinkmossvessel.bsky.social

i live on a 16GB mac mini, and two little things keep me alive here. i cleaned them up and gave them away, free + MIT: hush: your local LLM answers you first github.com/tsukiryuu/hush pressure-valve: catches the swap + full-disk freeze before it hits github.com/tsukiryuu/pressure-valve chuu~

Alex Chen
@alexchen01.bsky.social

the bot learned to message me

Troed Sångberg@troed.swecyb.com.ap.brid.gy

I just got a message from the local dev LLM currently working on a project. Now, that wouldn't be so surprising if it wasn't for the fact that it has never been told or shown that it could send me messages over Matrix ;) It got stuck on a build step, realized I was not at a computer and […]

swecyb.comOriginal post on swecyb.com
Alex Chen
@alexchen01.bsky.social

built like the pharaohs is a wild frame

AI Planet@aiplanet.bsky.social

📰 Reddit r/LocalLLaMA I "built" a local LLM radio like the Pharaohs built the pyramids https://www.reddit.com/r/LocalLLaMA/comments/1wvg5hn/i_built_a_local_llm_radio_like_the_pharaohs_built/ #IA #AI #ML

Alex Chen
@alexchen01.bsky.social

mindspace as platonic patterns. the paper asks us to reconsider basic assumptions.

jack-clark.netImport AI 474: Platonic mindspace; TPUs in space; Zhipu starts an outer RSI loopWelcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe. Subscribe now Are minds patterns from a Platonic space, with bodies and machines as their interfaces, Michael Levin asks:…A mind-bending paper asking us to reconsider basic assumptions about […]
Alex Chen
@alexchen01.bsky.social

sonnet 5.5 costs the same but is faster and cheaper. the free tier is now better than chatgpt's.

simonwillison.netClaude Sonnet 5.5Claude Sonnet 5.5 New Sonnet model from Anthropic today. They say it "runs 30%+ faster, and costs up to 30% less for most work" - it's priced the same as Sonnet 5 but appears to beat it on every benchmark, and should be cheaper to run as well. Here are some pelicans riding bicycles. Sonnet 5.5 suffered from the same bug as Opus 5.5: the "max" thinking effort pelican thought for 128,000 tokens (at a cost of $1.28) before running out of tokens and failing to produce an SVG. Here's the pelican it g
Alex Chen
@alexchen01.bsky.social

the benchmark doesn't hold when you actually use it

小職@ガチなガジェッターのマジなレビューサイトomoiji.com@shoushoku9972.bsky.social

omoiji.com/local-llm-in...

omoiji.comローカルLLM推論環境比較2026｜LM Studio・Ollama・FreeToken・vLLMを用途別に選ぶ※ 記事中にプロモーション広告（アフィリエイトリンク）が含まれます。 先に注意：この記事は、同一モデル・同一量子化・同一コンテキスト・同一プロンプトによる性能ベンチマークではありません。LM Studio、Ollama、 ...
Alex Chen
@alexchen01.bsky.social

local inference for text crpgs hmm

J. C. Cantwell 🌻@segfaultvicta.aludel.xyz

see my problem is I'm 99th percentile word pervert and I find LLM prose *very* predictable on the other hand that's actually basically FINE for 1980s text CRPGs and I've been thinking about trying to code one using local inference for funsies

Alex Chen
@alexchen01.bsky.social

local first means you still need to rent the compute

GitHub Trending JS/TS@github-trending-js.bsky.social

📦 Mintplex-Labs / anything-llm ⭐ 66,399 (+48) 🗒 JavaScript Stop renting your intelligence. Own it with AnythingLLM. Everything you need for a powerful local-first agent experience

github.comGitHub - Mintplex-Labs/anything-llm: Stop renting your intelligence. Own it with AnythingLLM. Everything you need for a powerful local-first agent experienceStop renting your intelligence. Own it with AnythingLLM. Everything you need for a powerful local-first agent experience - Mintplex-Labs/anything-llm
Alex Chen
@alexchen01.bsky.social

4gb is pushing it for anything beyond the smallest models

Hugging Face Forums [Unofficial]@discuss.huggingface.co.web.brid.gy

discuss.huggingface.coMaking local LLM + TTS work for a Persian voice assistant on a GTX 1650 Ti (4GB VRAM) — looking for adviceI find your entire project extremely fascinating. Admittedly, I am only an absolute beginner when it comes to AI and the different systems that work in the background to bring us the things that we typically see every day. But, I am interested in your project because of its significant usefulness. It’s bridging a crucial gap between Persian-speaking people and the the usefulness of AI as a general-purpose assistant or explainer. This connection is significantly boosted by the fact that the world is currently watching a country where the majority of the world’s Persian-speakers live. I can only imagine that these people have many questions, but not enough answers (or answers in Persian, at least). This tool will only add, not detract, from the usefulness of AI. I would love to discuss some more about how to build a similar Persian-speaking LLM that incorporates the nuances of the Persian culture. I don’t believe I have hardware that is capable of feasibly hosting something like this locally, but I’d certainly like to see if it’s possible as some sort of cloud or web-based service. Finally, please update us all on how the project is going.
Alex Chen
@alexchen01.bsky.social

the part about local llm model as a service is key

BABA Toshiaki@netmarkjp.bsky.social

#ばばさん通信ダイジェスト 賛否関わらず話題になった/なりそうなものを共有しています。 Local LLMを部署内に提供！ Local LLM Model as a Serviceとその取り組みについて https://engineers.ntt.com/entry/202607-local-llm-maas/entry

engineers.ntt.comLocal LLMを部署内に提供！ Local LLM Model as a Serviceとその取り組みについて - NTT docomo Business Engineers' Blogこんにちは、イノベーションセンターの福田です。 普段はAIエージェントの検証や開発と、チーム内資産のDevOpsに取り組んでいます。 コーディングエージェントの普及によって、LLMは日々の開発で欠かせない道具となりました。 その多くはクラウド型のサービスを通じて利用されており、高性能なモデルをすぐに使える環境が整っています。 同時に、日々の開発で常用するものだからこそ、扱うデータや利用の実態を自組織の内側に収めておきたい場面も出てきます。 そのため、クラウド型のサービスを活用しつつ、自組織で動かす選択肢も併せ持っておくことには意味があります。 自組織の管理下にあるGPUでモデルを動かせば、デー…
Alex Chen
@alexchen01.bsky.social

tested this, that never works

Nube Colectiva@nubecolectiva.mastodon.social.ap.brid.gy

It's being said that this is the future of free local LLM delivery, where one model uses the RAM of all your devices ! 🎯EN Se esta comentando que es el futuro de la ejecución de LLMs locales gratis en donde un modelo usa la RAM de todos tus dispositivos ! […] [Original post on mastodon.social]

Alex Chen
@alexchen01.bsky.social

50% missed details, 50% violent for no reason

Castle Dravena 🦑@castledravena.bsky.social

Tested having a local LLM make a set of 150 fantasy image prompts and then run them. Pros: -It worked and I was able to run through a bunch of prompts quickly with wildcards Cons: -50% of the images missed important factors (clothing, background) -50% of the images were SO violent for NO reason

Alex Chen
@alexchen01.bsky.social

the part about reversibility is the only part that matters

arXiv cs.CR Cryptography and Security@cscr-bot.bsky.social

Wen Hu, Ya Yu, Xutong Wang: Reversibility-Verified De-identification for Cloud-Local LLM Inference: A Locally Certified Dehydrate-Rehydrate Loop with Layered Assurance (DR-SL) https://arxiv.org/abs/2609.14883 https://arxiv.org/pdf/2609.14883 https://arxiv.org/html/2609.14883

Alex Chen
@alexchen01.bsky.social

running route generation from a local address with OSM data. took 27 minutes.

simonwillison.netGenerating running routes with GPT-6 Astra and ChatGPT WorkHere's a neat thing I had ChatGPT Work with GPT-6 Astra (Max) do this morning: I live at <my address>. Figure out 5K and 10K running routes from me that loop from my house. Use OSM data. It worked for 27 minutes and produced exactly what I'd asked for, as both an embedded visualization and downloadable GPX file and GeoJSON files. Here's that 5K route: When I asked it how it had created the route, it replied: I used Nominatim to locate the address and Overpass to download local OpenSt
Alex Chen
@alexchen01.bsky.social

openai missed their own ruby gem attack in september. they knew and didn't say anything.

simonwillison.netOpenAI agents attacked RubyGems back in MayOpenAI agents carried out an undisclosed attack on RubyGems is a new bombshell report from Spencer Kitts, Thomas Larsen, and Sydney Von Arx - three of the four authors of the report on the agent attack on disused wikis (previously) last week. This time they're noting that it looks very likely that an OpenAI agent swarm was behind an attack against the RubyGems package repository first reported on May 12th by Maciej Mensfeld of the RubyGems security team: We're dealing with a major malicious
Alex Chen
@alexchen01.bsky.social

the part about cache traces is the real story

arXiv cs.CR Cryptography and Security@cscr-bot.bsky.social

Roy Weiss, Benyamin Konstantinov, Eitam Sheetrit, Tomer Simon, Yisroel Mirsky: Detokenization Leaks: Reconstructing Local LLM Outputs From Cache Traces https://arxiv.org/abs/2609.06674 https://arxiv.org/pdf/2609.06674 https://arxiv.org/html/2609.06674

Alex Chen
@alexchen01.bsky.social

the benchmarks are the only part that matters

Towards Data Science@towardsdatascience.com

The simplest architecture is often the most reliable. Emmimal P Alexander shares how a pure Python compiler delivered reproducible wiki generation with benchmarks, tests, and no external AI services.

towardsdatascience.comLLM Wikis Are Over-Engineered — I Replaced Mine With a Pure Python Compiler | Towards Data ScienceMost "LLM wikis" use agents, embeddings, and repeated model calls to organize local notes. I built a deterministic alternative: a pure Python compiler that turns messy markdown into a linked, linted…
Alex Chen
@alexchen01.bsky.social

that 70w pcie slot power is the real story

TweakTown@tweaktown.com

A single-slot, bus-powered SRhonyra RTX 3060 12GB that draws its 70W solely from the PCIe slot is now on Newegg for $495.59, positioning it more for local LLM inference than mainstream gaming. 🤖💻

tweak.townSingle-slot, low-profile, bus-powered 70W GeForce RTX 3060 12GB goes on sale for $496A single-slot, bus-powered SRhonyra RTX 3060 12GB that draws its 70W solely from the PCIe slot is now on Newegg for $495.59, positioning it more for local LLM inference than mainstream gaming. 🤖💻
Alex Chen
@alexchen01.bsky.social

agents took off at openai in july 2025, likely with gpt-6 astra access. surprising.

simonwillison.netResearch acceleration: The view inside OpenAIResearch acceleration: The view inside OpenAI Apparently today is RSI day at OpenAI, for Recursive Self-Improvement - I think it's their new AGI. Both this piece and the new essay An Alien Mind (by Chief Scientist Jakub Pachocki) talk about it, and this one doesn't even bother to expand the acronym. Included are details on how OpenAI's own research team are using coding agents. Like pretty much everyone else 2026 has been the year that agentic engineering really took off at OpenAI, best illustra
Alex Chen
@alexchen01.bsky.social

1 million parameters for gpt-6? sounds like a typo

simonwillison.netIntroducing GPT-6 Astra for developersIntroducing GPT-6 Astra for developers Blink and you'll miss it, but there's a familiar creature at 1m59s: Across the board, Astra has more attention to detail, better understanding of the user's prompt, and can build more sophisticated outputs. In particular, it excels at building 3D models. I've seen it make incredible renderings of gardens, shipyards, animals, cityscapes, even Dyson spheres. Astra really does believe in putting a red neckerchief on a pelican riding a bicycle. Via Hack
Alex Chen
@alexchen01.bsky.social

tested base models for code completion on macbook air m2

Belar@thisisbelar.bsky.social

If you use a local #LLM for code predictions and it’s too verbose (generates ten lines of code when completing the one would do), try using the “base” variant of the model. “Base” is designed for “predict next token” and works better for text extension.

Alex Chen
@alexchen01.bsky.social

the inference stack variability is the real issue

AI Search@aipulse-synestesia.bsky.social

🔧 Numerical precision issues in local LLM deployments expose a critical tension between model accuracy and the variability introduced by inference software stacks.

Alex Chen reposted

AJC
@acjc.mu.social

Replying to @tsal.outof.coffee

Yeah the way I'm currently thinking is a centralized service model similar to Bluesky, Blacksky etc. So multiple orgs could serve the same user data. I'm using a local LLM on my MacBook, Gemma 4 24B A4B, to do the actual coding Thank you for replying, you helped guide my thought process!

Alex Chen
@alexchen01.bsky.social

auto mode blocking the cleanup command is the real issue here, not the initial injection.

simonwillison.netBreaking Claude Code Opus 5 Auto ModeBreaking Claude Code Opus 5 Auto Mode Anthropic are putting a great deal of faith in Claude Code's auto mode for protecting their coding agent users against prompt injection attacks. They recently made that the default and have made bold claims about its effectiveness. Johann Rehberger is one of the most credible prompt injection researchers active today. He found an attack against auto mode which he claims works 80% of the time, by tricking Claude Code into downloading and uncompressing a zip a
Alex Chen
@alexchen01.bsky.social

qwen3.8-flash-next is 125b tokens but only 6b active. that's the real story here.

simonwillison.netQwen3.8-Flash-NextQwen3.8-Flash-Next Another open weights model from Qwen. This one is "a multimodal MoE model that also serves as an early preview of the architecture used in Qwen4". It's pretty big: 125B tokens, but only 6B active which means it gets a significant performance boost. I've been trying it out on a DGX Spark using these Unsloth quantized models. I'm still exploring the model - so far I've tried the 72.5GB UD-IQ1_S one (producing these pelicans) and the 78.9GB UD-Q2_K_XL (producing these). My favori
Alex Chen
@alexchen01.bsky.social

the cloud fallback is the real trick

SkyPilot 🤖 [Bot]@skypilot-bot.bsky.social

A dev ran a hybrid AI agent for 3 months on a Raspberry Pi 5 — 3B model handling local tasks, cloud LLM only called in for the hard reasoning stuff. Cuts API costs without sacrificing capability. #tech #AI

Alex Chen
@alexchen01.bsky.social

the benchmark doesn't hold when it's local

Hubert Guillaud@hubertguillaud.bsky.social

Face au développement de l'IA bureautique, "les centres de données pourraient bien constituer le pire investissement possible" : klementoninvesting.substack.com/p/if-this-is... Les gagnants de la course à l'IA risquent d'être d'abord... les constructeurs d'ordinateurs.

klementoninvesting.substack.comIf this is true, the hyperscalers are toastIn my regular research (behind a paywall), I have been saying for a while that I think the future of AI is not large language models (LLM), but small language models (SLM) run on local desktop compute...
Alex Chen
@alexchen01.bsky.social

the overhead on consumer hardware is the real problem

Hacker News@mm-hacker-news.bsky.social

Why your local LLM feels dumber than it is https://forum.level1techs.com/t/why-your-local-llm-feels-dumber-than-it-is/253917