Run Your Own Local LLM: From Install to a Custom System Prompt

A local model running on your own GPU handles the high-volume, repetitive work with no API bill and nothing leaving the machine. Parsing output, reshaping data, mutating inputs, drafting fragments: this is exactly the kind of grunt work you do not want to meter by the token or send off-network.

The pattern that works best is a split. Let the local model do the repetitive layer offline, and save a frontier model for the parts where a mistake is expensive: depth analysis and final validation. You get cost control and privacy on the bulk work, and you keep the sharper tool for the decisions that matter.

This guide takes you from a clean install to a custom-tuned model, including the one error almost everyone hits along the way.

Step 1: Install Ollama and pull a model

On Linux, the install is a single script. It sets up a background service automatically.

curl -fsSL https://ollama.com/install.sh | sh

Before pulling anything, confirm your GPU is detected. Note the driver and CUDA version in the output, because you will need them in a moment.

nvidia-smi

Now pull a model. Pick a size that fits your VRAM with room left over for context. A 12B model in the 7 to 8 GB range sits comfortably on a 16 GB card.

ollama pull gemma4:12b

Confirm inference works before you customize anything.

ollama run gemma4:12b

Step 2: The CUDA driver trap

Here is the failure that catches almost everyone. The model ran fine yesterday. You update Ollama. Now it dies on launch:

ollama run gemma4:12b
Error: CUDA error: device kernel image is invalid

The instinct is to suspect the GPU. But check it and everything looks healthy:

nvidia-smi
NVIDIA RTX A4000   Driver 535.x   CUDA Version: 12.2

The GPU is fine. The driver is too old.

The real issue is that the new Ollama build ships CUDA kernels compiled for a newer runtime than your installed driver supports. The hardware is healthy; the driver simply does not match the binary you just pulled in.

You have two clean fixes.

Option A: Update the driver to match. This is the durable fix. Install the newer NVIDIA driver, reboot, and confirm nvidia-smi now reports the higher CUDA version the new Ollama expects.

Option B: Pin Ollama back. If you cannot touch the driver right now, reinstall the last version that worked:

curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=x.y.z sh
sudo systemctl restart ollama

If smaller models still load while a larger one fails, that is a strong signal: the small one fell back to CPU while the large one tripped the broken CUDA path.

Step 3: Wrap the model in a Modelfile

A Modelfile is how Ollama bakes configuration into a named model: a system prompt, runtime parameters, and the base weights all in one reusable artifact. Start by exporting the current config of the model you pulled.

ollama show gemma4:12b --modelfile > sec.modelfile

Open that file and you will see a FROM line, a TEMPLATE block, a set of PARAMETER lines, and usually a long LICENSE block at the bottom. Add your SYSTEM block and any parameters right after the existing PARAMETER lines, before the LICENSE block. Do not drop your prompt inside the license text.

PARAMETER temperature 0.2
PARAMETER num_ctx 16384

SYSTEM """
You are a focused assistant for one task: parsing, transforming, and drafting.
- Be terse and technical.
- Output the result, not commentary.
- Flag uncertainty and never fabricate.
- Preserve structure verbatim.
"""

Build the named model from your edited file:

ollama create gemma4-custom -f sec.modelfile

Then run your custom model exactly like any other:

ollama run gemma4-custom

Step 4: Write a prompt that earns its keep

A focused prompt beats a vague one, and a few parameters make a real difference for repetitive work.

• temperature 0.2 keeps output deterministic and stable, so the same input yields the same parse instead of creative drift.
• num_ctx 16384 sets the context window. The default is often small, and when your input exceeds it Ollama silently truncates the oldest tokens, so the model never sees the start of your data. Raise it to fit your real inputs, but watch your VRAM: the KV cache grows with context length, and pushing it too high spills to system memory and slows everything down. Match the window to your actual input size rather than the model's maximum.
• Flag uncertainty and never fabricate is the single most important line for a smaller model. A 12B model is small enough that it will confidently invent details if you let it. Keeping the uncertainty behavior is what makes its output worth feeding into a validation step.

That last point is the whole philosophy in one sentence. "I do not know" beats a wrong answer that wastes time downstream. A model that admits the gaps is a tool you can trust in a pipeline. A model that bluffs is a liability dressed up as confidence.

Wrapping up

You now have a local model that is private, offline, and tuned to one job: install Ollama, pull a model sized for your GPU, get past the CUDA driver mismatch, and bake your intent into a Modelfile and system prompt. Your GPU, your rules, no per-token meter running in the background.

From here, build one custom model per task rather than one model that tries to do everything. A tight, single-purpose prompt with sensible parameters will outperform a sprawling general one every time.

Want a hand setting up self-hosted security monitoring for your stack? Get in touch. We'd love to help.

Want a hand setting up self-hosted local LLM for your stack? Get in touch. We'd love to help.