Talk:Llama.cpp
LLM-assisted changes
Hi all,
Let's not rely on LLMs to generate an ArchWiki article, or at least not on this one.
LLMs generally have no idea what they are saying when it comes to highly detailed data. Without proper prompt engineering and sophisticated agent systems, you can safely assume they are as intelligent as dumb terminals. The LLM-assisted change Mr.Smith1974 has contributed (and reverted) even failed to explain the correct meaning of GGUF. The file format GGUF is actually named after Georgi Gerganov, not GPT-Generated.
Let's focus on what we've actually triaged instead. Use the search engine to acquire extra data. Polish the wordings of what we've known to work for a bonus.
Regards,
SH3NG1UN (talk) 03:33, 13 January 2026 (UTC)
Execution backends
Hello. The article only provides the `llama.cpp-vulkan` package for GPU inference, even though on AUR there is a CUDA package available from the same submitter [1]
I must add that the CUDA package is "Flagged out-of-date (2025-12-22)" Gotoro (talk) 08:50, 3 April 2026 (UTC)
- Hi. The reason CUDA was not added is that Vulkan solution is general, performant enough for most hardware setup, and I personally believe that is what edge AI deployment should look like - if your software stack is capable of gaming, it is capable of AI. But your point has some truth in it as well - if the package already exists, mentioning it would not cost anything. Adding the package now. SH3NG1UN (talk) 08:57, 3 April 2026 (UTC)
- Sorry I must revert my previous edit. Reading the comments section of [2], it appears that the package maintainer has been missing since 2025-12-11. Their GH user page [3] supports the claim, too. The package quality cannot be guaranteed anymore, we cannot risk this on ArchWiki. Reverting the edit now. SH3NG1UN (talk) 08:26, 7 April 2026 (UTC)
- The usual approach would be to file an orphan request, instead of having an incomplete listing here -- Alad (talk) 08:31, 20 April 2026 (UTC)
- It makes sense that the official package is the Vulkan-only one, but that makes the top of the page a bit confusing. Right now, it says:
- ---
- Install llama-cpp and one of the following backends:
- ---
- However, if you follow the "for inference with CUDA" options, it still uses Vulkan. Gpjt (talk) 13:01, 24 July 2026 (UTC)
- That's exactly what I have been worried about how ggml is packaged. ggml is not Vulkan-only, instead it's so monolithic that it attempts to support as much backends as possible with just one, single package.
- When you were expecting inference with CUDA, but got Vulkan, that essentially tells us you have multiple backends installed on the system, but ggml is having trouble prioritizing CUDA with default configurations.
- I don't have NVIDIA platforms to validate that. So I opened a expansion request there. If you can help, feel free to do so (and we encourage). SH3NG1UN (talk) 02:34, 27 July 2026 (UTC)
- I'm happy to run anything needed on my machine to help dig into this. But my understanding is that Llama.cpp needed to be specifically compiled with CUDA switched on. Per https://github.com/ggml-org/llama.cpp, I ran this:
cmake -B build-cuda -DGGML_CUDA=ON -DGGML_NATIVE=ONcmake --build build-cuda --config Release -j$(nproc)- ...to get a CUDA build installed in my home dir; running that did indeed get a version that used CUDA properly (and was faster and more VRAM-efficient on my benchmark -- would link to that but I worry it might look like spamming to link to my own blog from here :-) Gpjt (talk) 13:45, 31 July 2026 (UTC)
- We need to consult the PKGBUILD here: https://gitlab.archlinux.org/archlinux/packaging/packages/ggml/-/blob/main/PKGBUILD
- We can see that the
build()function there clearly states: -DGGML_CUDA=ON-DGGML_HIP=ON-DGGML_VULKAN=ON- So, CUDA support and Vulkan support are both packaged into ggml; the problem we have is, for some reason, ggml is prioritizing Vulkan, and we expect CUDA.
- Maybe the package maintainer needs to know this. SH3NG1UN (talk) 07:30, 3 August 2026 (UTC)
- The usual approach would be to file an orphan request, instead of having an incomplete listing here -- Alad (talk) 08:31, 20 April 2026 (UTC)