Large language models have become the cornerstone of natural language processing, but also by their extremely high computational and storage costs. Specifically, due to their massive size, even inference for large, highly-accurate models may require multiple performant GPUs, which limits the usability of such models. This leads to a race towards reducing their inference costs to allow for efficient local computation. In this talk, we first discuss the main computational bottlenecks of such models. Then, we present state-of-the-art compression techniques (like quantization and sparsification) to reduce the inference cost of these models in practice. Finally, we discuss existing opportunities for future work.