Choose a task, an exact model variant and a compatible runtime before judging whether a download is useful. Then plan a small test that checks the result as well as speed and memory. The model file is one part of the system; its size alone cannot tell you whether your workload will fit or work well.

A model contains learned numerical parameters, usually called weights, plus the information needed to interpret them. A runtime is the software that loads those parameters and performs the calculations. A chat app or coding agent uses that runtime’s output and may add its own tools, instructions and memory.

Name the pieces you are choosing

Georgi Gerganov’s llama.cpp project is a useful concrete example. It provides inference software: programs for running a model and serving responses. The model is downloaded separately. Its current documentation distinguishes supported hardware backends and CPU/GPU execution choices. Gerganov created the project; its current implementation is collaborative. Read the project’s Quick start and supported backends.

Before downloading, identify:

  • The task and variant: the particular model, revision and training variant you intend to use.
  • The artifact: its file format and numerical representation, including the exact quantized version when applicable.
  • The runtime: a build that supports that artifact and the device doing the calculations.
  • The input convention: the matching tokenizer and chat formatting expected by the model.
  • The permissions: the model’s license and intended use, alongside the runtime’s separate software license.

A tokenizer converts text into the pieces the model processes. A token may represent a word, part of a word or another text fragment. Thomas Wolf and collaborators’ Transformers paper explains why tokenizer, model computation and task output must match. Its model-sharing section also describes cards that record a model’s training context and use. This 2019 paper explains the design; use current runtime documentation for installation. Read the original paper.

A familiar model name is therefore only the beginning of a configuration record. Two downloads under that name may use different precision, formatting or revisions.

Budget memory for the workload

Think about what must exist while the program runs:

  • Loaded weights: the numbers used by the model, placed in system memory, device memory or a combination.
  • Stored state: information retained while processing and generating a sequence.
  • Working buffers: space for calculations and outputs.
  • The surrounding workload: the client, operating system and other applications using the same resources.

For many autoregressive Transformer models, the stored state includes a key-value cache, or KV cache. It saves intermediate attention information so each new token does not require recomputing everything. Longer sequences can increase that state. The cache strategy also matters: it may grow dynamically, reserve capacity in advance or move data between devices. Some attention patterns limit growth in particular layers. Hugging Face’s cache documentation explains these differences.

The runtime needs computation buffers too; llama.cpp’s context implementation allocates and reports them separately. This is why checking only the weight file leaves part of the memory question unanswered.

Plan the context you will actually use: instructions, conversation history, retrieved documents and space for the generated response. A brief greeting and a long coding session are different workloads. Check which memory pool is under pressure and what the runtime actually places there.

Architecture changes the story. Tri Dao and collaborators’ Mamba-3 explanation contrasts a fixed-size state with a growing attention cache, then examines the tradeoff in retrieving earlier details. Its server-GPU latency results do not predict your laptop’s performance. The useful lesson is to inspect the model’s state mechanism rather than apply one cache formula to every architecture. Read the coauthored explanation.

Smaller numbers and adaptation solve different problems

Quantization represents model numbers with fewer possible values. It can reduce storage and memory needs, but rounding changes the calculation. Tim Dettmers’ LLM.int8() explanation makes that visible with a small numerical example and explains special handling for unusually large values. It gives a reason to compare answer quality after choosing a smaller representation. Read his original explanation.

Adaptation changes a model’s behavior using training examples. In the coauthored QLoRA paper, a quantized base is kept frozen while small added parameter sets are trained. Running that adapted model and performing the adaptation are separate jobs with different memory allocations. You do not need to train an adapter simply to try a downloaded model. Read the original QLoRA paper.

Start by testing whether a suitable existing model already performs your task. If it fails, retain the failing examples before deciding whether the remedy is a different instruction, a different model or training.

Read what a local-model test actually establishes

Sebastian Raschka’s local coding guide separates speed and memory checks from task checks. In its published tool-reasoning test, the model returns a proposed tool call; it does not execute the tool.

One task concerns a Python import error after a file move. The task file says the next step should read tests/test_config.py. In Raschka’s reported qwen3.6:35b result, the model asks for clarification instead. The immediate defect is the chosen next action. Generating text quickly would not resolve it. Read his published guide and the original task definition.

That small test can expose a mismatch relevant to coding work. It cannot establish whether an agent would repair the project, because no tool was executed. The repository documentation distinguishes three checks: model plus inference speed, a raw model’s tool decision, and the model plus a complete agent environment.

Choose the layer that matches your question. A chat task needs useful answers. An agent task also needs the chosen action to execute correctly and the resulting work to satisfy its tests.

Make a comparison you can repeat

For a first comparison, choose two configurations and one intended difference: model variant, quantization or runtime setting. Keep the device, task data, instructions, input limits and acceptance rule the same where possible.

Use a small set of tasks you can judge directly. For an extraction task, name the fields that must match the source. For the import-error task above, inspect the requested tool and path. For an actual coding-agent task, inspect the changed files and the project’s tests as well as the agent’s explanation.

Record the following for each configuration:

  1. Identity: model revision and artifact, runtime version, backend, placement and relevant settings.
  2. Workload: actual input tokens, context capacity, output limit and simultaneous requests.
  3. Performance: prompt-processing time, generation time, memory during the task and whether the intended input was processed.
  4. Acceptance: the verdict for each task, the output behind it and the specific failed requirement.

Prefill is processing the input; decode is generating the output. A single tokens-per-second figure can hide a long wait before the first response. Retain both timings and useful task outcomes.

Raschka supplies a speed-and-memory script and a decision-test script. Preserve the script and task-file revisions with the results. His speed inputs are specified in words; verify the actual processed token count and configured context instead of treating those units as interchangeable. Its macOS measurement also needs care because process memory does not fully describe every MLX-backed allocation.

After a short-input check, try a case representative of your longer workload. Keep the same scoring rule, retain failures and repeat enough to see whether the outcome depends on a fortunate response. A run that silently omits input is not the comparison you intended.

The result should help choose a configuration: it fits the intended workload, completes enough of the named tasks and responds at a pace you find useful. The acceptable failure rate and waiting time belong to your task; the download size supplies neither.

Follow the next question

The local models and open projects collection connects the setup, adaptation and performance readings. Start with Gerganov for a runnable program, Dettmers for numerical precision, Raschka for a practical comparison, Wolf for matching the model’s components, and Dao for the state and memory tradeoff.

For more precise acceptance rules, continue with evaluating an AI application on your own task. If the model is part of a tool-using agent, the agent-demonstration field note helps separate a returned decision from completed work. The prompt-injection field note explains why running a model locally still leaves a separate question about what surrounding tools may read or change.