You can run an AI assistant on your own computer and call it from scripts or a website. Ollama provides a practical starting point: install the runtime, download a model, and send it a question. You do not need a second computer to get started.

This guide covers a complete single-machine setup first, then model choices and performance troubleshooting. If you already have another computer with a suitable GPU, the final section explains how to use it over your private network. That is an optional extension, not a prerequisite.

What is Ollama?

Ollama is software for downloading and running language models. It provides a command-line interface for interactive use and an HTTP API that applications can call. Ollama is the runtime; a model such as Llama 3.2 or Qwen 2.5 supplies the language capabilities.

You or your application
  -> Ollama
  -> downloaded model running on your computer
  -> generated response

Installing Ollama does not train a model on your files or automatically make it aware of your website. An application must provide the relevant information in its prompt or implement retrieval, as described in the Drupal article chatbot guide.

Why run AI locally?

Local inference gives you control over the machine processing your prompts and the models your applications use. With downloaded local models, you can experiment without a hosted inference API key or a per-request API charge. You still pay for hardware, storage, and electricity, and the initial model download needs connectivity.

It is useful for development experiments, short summaries, rewriting text, code explanations, and application features that supply their own context. Local inference can also continue without internet access once the necessary model is available; an application that fetches external information still needs that connection.

These benefits apply to local models. Selecting a cloud model or adding an external tool changes where processing occurs. Local hosting also leaves you responsible for updates, access control, and resource usage. See Ollama's local and cloud behavior.

Install Ollama and run your first model

For a Windows workstation, follow the official Windows installation instructions. Install Ollama, launch it, and open a new PowerShell window so the command is available on your PATH. Check the current GPU and driver requirements if you intend to use acceleration.

Run these commands on that same computer:

ollama --version
ollama pull llama3.2:1b
ollama list
ollama run llama3.2:1b "Explain an HTTP API in two sentences."

pull downloads the model, list shows installed models, and run generates a response. Start with one small model rather than downloading every available option. Allow disk space for the download and enough memory for execution; model file size is not the total RAM or GPU-memory requirement.

The local API is available at http://localhost:11434. In PowerShell, try a complete request:

$payload = @{
  model = "llama3.2:1b"
  stream = $false
  messages = @(
    @{ role = "user"; content = "Explain an HTTP API in two sentences." }
  )
} | ConvertTo-Json -Depth 5

$result = Invoke-RestMethod `
  -Uri "http://localhost:11434/api/chat" `
  -Method Post `
  -ContentType "application/json" `
  -Body $payload

$result.message.content

The request names the model and supplies the prompt. With streaming disabled, the API returns the completed response. This interface is also how a backend such as Drupal requests inference. See the chat API reference.

Alternative: run Ollama with Docker Compose

If you already manage local services with Docker, use a container instead of the native installation. Choose one approach for port 11434 so two servers do not compete for it.

Save this standalone example as compose.yml:

services:
  ollama:
    image: ollama/ollama
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ./data/ollama:/root/.ollama
    healthcheck:
      test: ["CMD", "ollama", "list"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s

docker compose up -d ollama
docker compose exec ollama ollama pull llama3.2:1b
docker compose exec ollama ollama run llama3.2:1b "Explain an HTTP API."

The volume keeps downloaded models across container recreation. The loopback port binding lets applications on the host reach Ollama without publishing it on every host interface. Containers on the same Compose network can use http://ollama:11434.

This example does not configure GPU access. A GPU installed in the host is not automatically available to this container. Follow the hardware-specific Docker instructions if you want acceleration. Resource limits should fit your machine and other services; there is no universal RAM limit appropriate for every model.

Three models and when to try them

These examples come from models installed in our home lab. They are practical starting points, not a ranking of the newest models. Check each model's license and test it against your actual questions.

The model pages describe Llama 3.2, Qwen 2.5, and Code Llama. To try another model, download it explicitly:

ollama pull qwen2.5:7b
ollama pull codellama:7b

A larger parameter count does not guarantee a better result for every task. In our Drupal configuration, Qwen 2.5 7B is the primary model and Llama 3.2 1B is the fallback. Code Llama is installed for experimentation; it is not wired into that chatbot.

Why local AI responses can be slow

CPU-only inference can be slow, particularly with larger models. A supported GPU can improve performance, but a GPU-equipped computer can also respond slowly when memory is insufficient, some computation stays on the CPU, or the workload is large.

Separate these causes before changing hardware:

• Cold model loading: the first request may include loading model data into memory.
• Model size and placement: a larger model needs more resources; partial CPU/GPU placement can behave differently from full GPU placement.
• Prompt and context size: long documents require more input processing and memory.
• Output length: generating several paragraphs takes longer than generating a short answer.
• Competing work: concurrent inference, other applications, power settings, and thermal limits can affect latency.

While a model is loaded, run ollama ps. Its PROCESSOR column reports CPU, GPU, or mixed placement. A successful HTTP response confirms inference worked, but not that it ran on the GPU. Refer to Ollama's placement guidance.

Improve response times on the same machine first

1. Try a smaller model. Compare the answer quality and latency of a small model using representative prompts.
2. Reduce unnecessary context. Send relevant excerpts instead of whole documents. Keep enough context to preserve the evidence needed for the answer.
3. Bound the output. Request concise answers and set an appropriate generation limit.
4. Reuse loaded models and answers. Keeping a model warm consumes memory but can avoid loading delays. Application caching avoids inference entirely for reusable results.
5. Verify acceleration and available memory. Check supported hardware, drivers, and container GPU configuration before assuming the GPU is being used.
6. Measure cold and warm requests separately. Also test expected concurrency; one successful request is not a capacity test.

For example, an application can add the following fields to its chat request:

{
  "keep_alive": "5m",
  "options": {
    "num_ctx": 2048,
    "num_predict": 120
  }
}

These are example limits, not universal tuning values. Too little context can omit useful evidence; too small an output budget can truncate the answer. Streaming can make progress visible sooner, but does not by itself reduce the total inference work.

If performance is acceptable after these changes, the single-machine setup is complete. You can connect it to scripts or a website without adding another computer.

Optional: use another computer with a GPU

If you already have a desktop, workstation, or laptop with a compatible GPU and sufficient memory, it can host Ollama while your application stays on the original machine. This separates application hosting from inference. Actual speed gains depend on both machines and the model; this guide does not claim a measured speedup.

Application computer
  -> private network -> Ollama on GPU computer
  -> generated response returns over the same connection

Both computers can use an existing trusted LAN. A direct Ethernet cable is another option. Our installation uses a direct link with these addresses:

For a similar direct connection, assign unused addresses on the same subnet, for example with subnet mask 255.255.255.0. Avoid a subnet that overlaps an existing network. Do not add a default gateway to this isolated link; retain your normal internet connection for downloads. The addresses here are examples from our setup, not values every reader must use.

Prepare Ollama on the GPU computer

Install Ollama on that computer, then download and test the intended model there:

ollama pull qwen2.5:7b
ollama run qwen2.5:7b "Explain Drupal in one sentence."
ollama ps

Downloading a model on the application computer does not install it on the second computer.

For a Windows Ollama host, quit the tray application and set a network listening address:

[Environment]::SetEnvironmentVariable(
  "OLLAMA_HOST", "0.0.0.0:11434", "User"
)

Reopen Ollama from the Start menu. These are reproducible setup commands, not a transcript recovered from our original installation. 0.0.0.0 means listen on all interfaces; clients connect to the host's real address.

Allow only the intended client through the firewall

On the GPU computer, run Administrator PowerShell. For the example Ethernet addresses:

New-NetFirewallRule `
  -DisplayName "Ollama from Application PC" `
  -Direction Inbound `
  -Action Allow `
  -Protocol TCP `
  -LocalAddress 10.10.10.1 `
  -LocalPort 11434 `
  -RemoteAddress 10.10.10.2 `
  -Profile Any

Use your actual server and client addresses. This allows a specific inbound connection; it does not disable Windows Firewall or expose GPU hardware directly. Keep this API on a trusted private network rather than forwarding its port to the public internet. See Microsoft's firewall command reference.

Verify connectivity and send a request

On the application computer:

Test-NetConnection 10.10.10.1 -Port 11434
Invoke-RestMethod "http://10.10.10.1:11434/api/tags"

The first command checks TCP reachability; the second confirms Ollama responds and lists its models. To generate an answer, reuse the earlier PowerShell payload with model = "qwen2.5:7b" and change the request URL to http://10.10.10.1:11434/api/chat.

For a website, the backend sends that HTTP request. Ollama handles model execution on the remote computer. Test from the application's container too, since a host-level network test does not prove container connectivity.

Decide what happens when the second computer is unavailable

Sleep, a disconnected cable, or a stopped Ollama process can make the remote endpoint unavailable. An application can optionally fall back to a smaller local model. Ollama does not automatically implement failover between these machines for you.

Our Drupal article chatbot tries the laptop first, allows a 10-second primary request timeout, then tries the mini PC with a 45-second fallback timeout after a request failure. A 60-second circuit-breaker cooldown avoids repeatedly waiting on an unavailable primary. These settings belong to our application and should be tuned for the reader's workload.

Troubleshooting common setup problems

A local runtime, one suitable model, and a tested API request are enough to start building. For application examples, continue with the Drupal article chatbot or the administrator analytics assistant.