Running a Local LLM With Ollama

Ollama makes one part of working with large language models pleasantly simple: it lets you download and run a model on your own computer. You can experiment without sending every prompt to a hosted service, and the basic workflow fits in a terminal window.

That does not make the model private by magic. You still need to look at the tools around it and decide what data you are comfortable using. But for local experiments, Ollama is a useful place to start.

What Ollama actually does

A large language model predicts likely pieces of text from the context it receives. That is enough to produce convincing answers, summaries, translations, and code, but it does not guarantee that an answer is true.

Ollama handles the practical work of downloading a compatible model and running it locally. The model still needs memory and processing power. Larger models usually need more RAM and run more slowly, so start small before downloading half the internet.

The available models and system requirements change over time. Check the current options on the Ollama website rather than relying on an old compatibility list.

Run your first model

After installing Ollama for your operating system, choose a model from its library and run it. This article originally used Llama 2:

ollama run llama2

Ollama downloads the model if it is not already installed, then opens an interactive prompt. From there you can ask a question, request a summary, or test how the model handles a small coding task.

Model names age quickly. If llama2 is no longer the model you want, use the same command with a current model from Ollama’s library.

Give a model persistent instructions

You can use a Modelfile to create a local variant with its own parameters and system message. First, make sure the base model is available:

ollama pull llama2

Then create a file named Modelfile:

FROM llama2

# Higher values produce more varied output; lower values are more predictable.
PARAMETER temperature 1

SYSTEM """
You are superhero Superman. Answer as Superman, the assistant, only.
"""

Create and run the variant:

ollama create superman -f ./Modelfile
ollama run superman
>>> hi
Hello! It's Superman!

This is a deliberately silly example, but the same approach works for response formats, writing constraints, or a narrow assistant role. Treat the system message as guidance, not a security boundary.

The Ollama documentation covers the current commands and Modelfile options.

Where a local model is useful

Local models are good for experiments, drafts, code explanations, and workflows where sending input to an external API would be awkward. They can also keep working without an internet connection once the model is downloaded.

The trade-off is your own hardware. A hosted model may be faster or more capable, while a local model gives you more control. I would start with one modest model, try a real task, and only then decide whether the extra disk space and RAM are worth it.