Marlin: Nearly Ideal Inference Speed for 4-bit Large Language Models

Up to 4x faster than inference with fp16 parameters

Large language models (LLMs) are often too large to be directly used on consumer hardware. To reduce their size, various techniques have been proposed to quantize LLMs and lower their memory consumption. While recent algorithms for 4-bit quantization are often…