Post

Ever wonder what it's like to run larger models at your existing context sizes? Ollama just introduced K/V context cache quantisation and it's a game changer!

Architecture Notes

@architecturenotes.co

25 Jan 2025

With K/V context cache quantisation, you can experience significant reductions in VRAM usage. It's time to realise the full potential of expanded context sizes!

Architecture Notes

@architecturenotes.co

Here's an insightful read about this development and its implications: smcleod.net/2024/12/bri...

smcleod.netBringing K/V Context Quantisation to OllamaK/V context cache quantisation has been added to Ollama. This enables significant reductions in VRAM usage, allowing users to realise the potential of expanded context sizes and run larger models at their existing context sizes.

25/1/25, 10:00 pm0 reposts0 likes

No replies yet.