I’ve been building VeloxQuant, an open-source KV-cache optimization toolkit for MLX focused specifically on LLM inference on Apple Silicon.
The project currently implements 43 KV-cache compression methods, along with custom Metal kernels for several performance-critical paths. The goal is to explore how far we can reduce KV-cache memory and improve inference performance while understanding the trade-offs in latency, throughput, and model quality.
Some of the areas I’ve been working on include KIVI INT4/INT2, RaBitQ, RVQ, fused attention kernels, GQA decode optimization, prefill kernels, and cross-request decode batching.
A recent GQA experiment was particularly interesting: instead of expanding shared KV heads before attention, a two-pass decode path operates on the grouped structure directly. In our tests this reduced GQA attention latency substantially at longer context lengths.
I’m now expanding the benchmarks to newer and larger models and, more importantly, across different Apple Silicon hardware. I currently do most of the development and benchmarking myself, so I’d really like to get more MLX developers involved.
If you have an M1/M2/M3/M4/M5 Mac — MacBook, Mac mini, Mac Studio, etc., I’d especially appreciate help running benchmarks, validating kernels, testing models, or contributing new compression/attention implementations.
I’m also very interested in feedback from people who understand MLX internals, Metal kernels, quantization, attention optimization, or LLM inference systems. If you see something in the architecture or benchmarks that could be improved, please tell me.
Project: https://veloxquant.dev/
GitHub: https://github.com/rajveer43/veloxquant-mlx
Contributions, benchmark results, criticism, and ideas are all welcome. I’d like this to become a genuinely useful MLX performance project rather than something optimized around one Mac or one benchmark.
Thanks for bringing this up — this is exactly the kind of benchmarking we need.
The project has moved quite a lot since Issue #27, with many new compression methods, optimized attention paths, and Metal kernels added. However, the **benchmarking challenge you pointed out is still very much relevant. The theoretical KV-cache compression ratio doesn’t always translate directly into the same reduction in Activity Monitor because of allocations, temporary buffers, model weights, MLX memory management, and other runtime overhead.
I’m currently figuring out the best way to benchmark this consistently across methods and models. A clean before/after real unified-memory comparison, especially at long contexts like 32K, would be extremely valuable.
Qwen3.8-27B would definitely be an interesting stress test if we can get the setup working within the available memory.
I’m developing primarily on a 24 GB - M4 Apple Silicon Mac, so testing these larger models and contexts also makes community hardware coverage really useful.
Would absolutely welcome contributions here — especially around building a reproducible benchmark that measures actual peak memory, KV-cache memory, TTFT, decode throughput, and quality across baseline MLX and VeloxQuant.
If you’re interested in experimenting with it, I’d be very happy to collaborate on the benchmark.