Running LLM on Local Machine Project

Diving Deep: Why I Chose Local LLMs (and How I Got Ollama Humming on Fedora with a 4080 Super)

Alright, folks, buckle up! Today we're diving headfirst into the fascinating world of local Large Language Models (LLMs). For a while now, I've been captivated by the potential of these AI powerhouses, but always relied on cloud-based solutions. But the allure of running my own models, right here on my machine, finally became too strong to resist. Why? Let's break it down.

The Cloud vs. The Couch: Why Local LLMs are My Next Obsession

We all know the giants – OpenAI's GPT family, Google's Bard (now Gemini), and others. They're impressive, no doubt, but relying solely on them has a few drawbacks that started to irk me:

• Privacy Concerns: Every prompt, every interaction is sent to a remote server. As I experiment with sensitive data or more nuanced projects, the thought of relying entirely on third-party privacy policies made me uneasy.
• Latency & Reliability: Let's be real, sometimes cloud services hiccup. A slightly delayed response is one thing, but being completely locked out during an outage is a creativity killer.
• Customization & Control: Cloud solutions offer limited customization. I wanted to fine-tune models, experiment with different architectures, and generally tinker under the hood without the restrictions of a walled garden.
• Cost: While many services offer free tiers, extended use or specialized models quickly rack up costs. Running locally, after the initial hardware investment, eliminates those recurring expenses.

So, I decided to take the plunge. The goal? To build a local LLM playground where I could experiment freely, learn deeply, and keep my data under my own control.

Choosing My Weapon: Why Ollama?

There are several frameworks for running LLMs locally, but Ollama really stood out. Why? It's incredibly user-friendly. Think Docker, but for LLMs. It packages models, dependencies, and configurations into neat, distributable "Ollama images." This significantly simplifies deployment and management.

Plus, Ollama boasts:

• Ease of Use: Running models with a single command is a game-changer.
• Model Variety: A growing and impressive library of pre-built models is available, from the ubiquitous Llama 2 to more specialized options.
• Cross-Platform Support: While I'm focused on Fedora, Ollama runs on macOS and Windows too, expanding its appeal.

Conquering Fedora: My Ollama Installation Adventure (with a 4080 Super!)

Now for the fun part: getting Ollama up and running on my Fedora Linux machine. I'm running Fedora 39 with a beefy NVIDIA 4080 Super GPU. Here's the breakdown:

1. Prerequisite Check: NVIDIA Drivers and Containerization

First, I made sure my NVIDIA drivers were properly installed and configured. This is crucial for leveraging the GPU's power for inference. You can usually find the latest drivers via the Fedora repositories or directly from NVIDIA's website. I used the NVIDIA repository, as that is the most seamless method.

sudo dnf update
sudo dnf install akmod-nvidia
sudo systemctl enable nvidia-driver-modules.service
sudo systemctl start nvidia-driver-modules.service

Second, I had to verify that the nvidia container toolkit was installed and running correctly to allow for GPU access inside of docker containers.

sudo dnf install nvidia-container-toolkit
sudo systemctl restart docker

2. Installing Ollama

Thankfully, Ollama provides a simple installation script. I fired up my terminal and ran:

curl -fsSL https://ollama.ai/install.sh | sh

This script downloaded and installed Ollama and its necessary dependencies. The install script asks for sudo permissions to complete the installation. After the install script completes, it is important to logout of the terminal and back in, so the current shell receives the proper permissions to access the system files.

3. Pulling My First Model: Llama 2 Time!

After the login, I tested if Ollama was installed properly by opening a new terminal window and executing ollama --version

Now for the exciting part! With Ollama installed, downloading and running a model is a breeze. I started with the classic Llama 2:

ollama pull llama2

This command downloads the Llama 2 model from the Ollama library. The download speed will depend on your internet connection. Once completed, you're ready to chat!

4. Unleashing the Beast: Running Llama 2

With Llama 2 downloaded, interacting with it is incredibly simple:

ollama run llama2

This command spins up the Llama 2 model and presents you with an interactive prompt. You can now ask questions, generate text, and explore the model's capabilities.

5. Monitoring Performance (4080 Super to the Rescue!)

With the model loaded, I started some basic interactions and monitored the GPU usage. My 4080 Super handled Llama 2 without breaking a sweat. Inference times were noticeably faster than using cloud-based LLMs, thanks to the raw power of the dedicated GPU. I utilized nvtop in another terminal window to monitor the GPU usage. I found this tool to be very useful to monitor the GPU usage.

Thoughts So Far: Early Impressions and Future Adventures

My initial experience with Ollama on Fedora with my 4080 Super has been incredibly positive. The installation process was smooth, the model deployment was dead simple, and the performance is impressive.

Of course, this is just the beginning. My next steps include:

• Exploring Other Models: I'm eager to test out different models, especially those tailored to specific tasks.
• Fine-Tuning: I want to experiment with fine-tuning models on custom datasets to see how much I can improve their performance for my specific needs.
• Integration with Applications: The ultimate goal is to integrate these local LLMs into my development workflows, automating tasks and enhancing my projects.

Stay Tuned!

This is just the first chapter in my local LLM journey. I plan to document my progress, share my findings, and explore the endless possibilities that this technology unlocks. So, stay tuned for more updates, tutorials, and explorations into the exciting world of local AI!

What about you? Are you running LLMs locally? What are your experiences and tips? Share your thoughts in the comments below! Let's learn together!