I’ve been building VeloxQuant, an open-source KV-cache optimization toolkit for MLX focused specifically on LLM inference on Apple Silicon. The project curren…

quick question for the Mac Studio and Thunderbolt 5 crowd. the latest mlx-lm added tensor parallel inference on top of the new JACCL backend. Angelos from th…

llama-3.1-8B-q4 on: - M1 Air 16G → 19 tok/s - M1 Pro 16G → 27 tok/s - M2 Pro 32G → 38 tok/s - M3 Pro 36G → 42 tok/s - M3 Max 64G → 71 tok/s - M4 Max 128G → 84…