llama.cpp Vulkan is the easiest way to run LLMs locally on your GPU while still getting great performance. Although there are faster methods for Nvidia such as ExLlamaV2, using Vulkan is easier and is the best choice for AMD GPUs. I used a RX 9060 XT 16GB with CachyOS to demo it, but this will work on any Linux distro and there are also versions for Windows and Mac. LLM and other AI models can be found at huggingface.com. . Here’s the command used in the video:. ./llama-server -hf unsloth/gemma-3-27b-it-GGUF:Q3_K_S -fa on -ngl 100. . Check out my AI/ML playlist: https://www.youtube.com/playlist?list=PLlLR7EXXYZ0ZpzATacLvu3MtUWmPd3YKU. . These are affiliate links where I earn a small commission for purchases at no extra cost to you.. This is the easiest way to help the channel, thank you!. Amazon: https://amzn.to/484HUnU. . Website: https://phazertech.com/. . Donations. Buy me a coffee: https://www.buymeacoffee.com/phazertech. Cash App: $phazertech. . Chapters:. 00:00 Intro. 01:49 Downloading llama.cpp Vulkan. 02:39 Choosing a model. 05:04 Running the model. 10:08 Other helpful tips. 11:50 Outro
The Easiest Way To Run Llms Locally On Your Gpu Llama cpp Vulkan
Your Local Llm Is 10x Slower Than It Should Be
How To Run Local Llms With Llama cpp Complete Guide
Easiest Simplest Fastest Way To Run Large Language Model llm Locally Using Llama cpp Cpu Gpu
How To Install Llama cpp On Linux With Gpu Support