Everyone is building AI assistants on frontier models in the cloud. We wanted to know whether a 4-billion-parameter model running on a laptop could do real, everyday work inside an existing line-of-business web app: tasks, a calendar, documents and a record of meetings and decisions.

It can, if you design around its limits. Our final setup answers 42 of 44 scripted real-world requests fully correctly, with a median answer time of 2.9 seconds on an Apple M4 Pro, at zero inference cost and with no data leaving the machine.

Getting there took one architecture we threw away, three models, six evaluation rounds and a list of small, unglamorous fixes. Most of the gains came from tool design and prompts, not from the model. This is what we learned, with the numbers from every round.

Privacy. Business apps hold people’s names, schedules, internal documents and decisions. A model that runs on the same machine as the app means none of it goes to a third party.

Cost and latency you control. Inference is free after the hardware, and there are no rate limits or outages you don’t own.

Honesty about the task. Most assistant requests in a business app are not hard reasoning: “What’s due this week?”, “Log a meeting with Sam about the vendor schedule”, “Remind me to collect the reports by Friday.” That work is retrieval and form-filling. If a small model can do it reliably, a frontier model is overkill.

The goal was a hypothesis test, not a product launch: prove the smallest model can do the job before paying for, or downloading, anything bigger. We deliberately did not pull the 9B version of our model until the 4B had been measured. It never became necessary.

The rule we set ourselves: don’t pollute the main codebase. The app should gain a well-defined interface, and everything AI-specific should live beside it, replaceable piece by piece.

An MCP server inside the existing backend. The app exposes its own domain as 25 Model Context Protocol tools over Streamable HTTP, mounted on the same web server. Tools call the same functions the REST routes use, so the assistant sees exactly what the user sees, and every write goes through the app’s own validation and its undo history. An assistant’s change is undone like any other.

An agent harness, not an agent framework. We run OpenCode, an open-source coding agent, headless (opencode serve) as a general-purpose agent runtime. It provides the agent loop, sessions and history, streaming, tool permissions, a structured ask-the-user tool, model variants and a plugin system. Our agent is a JSON block and a 20-line prompt.

An LLM gateway. LiteLLM sits between the harness and every model. It routes by name and logs every call in full to Postgres: system prompt, messages, tool schemas, response, reasoning, tokens, cost, latency and the conversation id.

A local runtime. Ollama serves the model on the host machine.

The whole integration, counting the tools, the chat pane, configuration, docs and the evaluation harness, came to about 3,200 lines across 39 files. None of it is an agent loop.

Our first instinct was to write the agent loop ourselves, in Python, on an older LLM library. It is the obvious move and it is the wrong one in 2026.

Coding agents have quietly become the best general-purpose agent runtimes available. They have already solved streaming, cancellation, session storage, tool permissions, context management, model switching and plugins, and they are battle-tested by far more users than your loop ever will be. Stopping to survey what was current before writing code saved us weeks.

Before writing an agent loop, check whether a coding agent can run headless and be told what to do. The agent itself is usually just configuration.

A small model will sometimes be wrong. The design has to make that cheap. We used the harness’s permission system to put every tool in one of four classes.

Deny by default. Only ten of the 25 tools are offered at all.

Reads run freely.

Every write asks. The harness holds the call and emits a permission event. The chat pane renders it as a card showing exactly what will change, with Approve and Decline. Approved changes land as one labelled step in the app’s undo history.

Choices are structured. When a request is ambiguous (which task? which date?), the model calls the harness’s built-in question tool with two to four options, and the user clicks one instead of typing.

This turned out to matter more than model quality. A wrong proposal the user declines costs a click. A wrong change made silently costs trust.

We wrote a small bake-off runner that sends real requests through the same HTTP API the chat pane uses, not a mock. Its checks are structural rather than string matches, so they survive changes in the data.

Did it call the right tools? For writes, did it propose the right change with the right arguments? Is every figure in the answer present in a tool result? A count the model invented fails even if it happens to be right. For things it can’t do, does it say so plainly instead of improvising?

The runner declines every write, answers every multiple-choice question with the first option, and deletes its sessions afterwards, so the real data never changes. Later we gave it its own harness instance whose memory doesn’t learn, so scripted conversations can’t teach the real assistant anything.

The final suite has 22 tasks: agenda questions, counts, document search and summary, calendar lookups, logging meetings, adding and completing tasks, attachments, out-of-scope requests, and requests for data the assistant can’t see.

The task set grew and shrank between rounds, so compare the shape more than the exact percentages. All runs were on one Apple M4 Pro with 24 GB, Ollama 0.34, thinking off, 4-bit quantised models and a 32K context. Fully correct means every check on a run passed.

Round 1 — Qwen3.5 4B with baseline tools: 35 of 42 runs fully correct (83%), median answer 7.2 s.

Round 2 — same model after tool fixes: 38 of 42 (90%), 6.7 s.

Round 3 — a router with specialist sub-agents: 36 of 52 (69%), 15.8 s.

Round 4 — back to a single agent with ten tools: 38 of 44 (86%), 5.5 s.

Round 5 — Qwen3.5 4B, final: 42 of 44 (95%), 2.9 s. Gemma 4 E4B, final: 42 of 44 (95%), 8.7 s. Spark X2.5 4B, a community fine-tune: 37 of 44 (84%), 1.9 s.

Round 6 — Qwen3.5 4B with memory: 39 of 44 (89%), 4.1 s. Two of the five failures were a stale test; see lesson 12.

The headline: Qwen3.5 4B went from 83% at 7.2 seconds to 95% at 2.9 seconds without changing the model. Everything that moved the numbers was in the tools, the prompt, the agent shape or the runtime settings.

Qwen3.5 4B was the best all-rounder: accurate, fast and good at chaining tools, for example searching documents and then reading the right one. Its one quirk was over-using the multiple-choice tool. In one run it asked three clarifying questions before answering a simple date lookup.

Gemma 4 E4B matched Qwen’s accuracy but was three times slower. It chatted more before calling tools and wrote longer answers. Its one consistent failure: asked to summarise a document, it asked the user to upload it instead of searching for it. Its thinking also can’t be switched off through the OpenAI-compatible endpoint.

Spark X2.5 4B was the fastest, but it improvised. Asked who was on leave, data it has no tool for, it once replied that no one was. Asked to send a message it can’t send, it proposed logging a meeting instead. Speed is worth little if the answers can’t be trusted.

1. Small models want one agent, not a team. We tried the fashionable design: a router handing work to specialist sub-agents, each with its own tools. With a 4B model it was the worst round by a distance: accuracy fell from 90% to 69% and the median answer took 15.8 seconds. The router picked the wrong specialist. Specialists looped; one log-a-meeting request made ten tool calls and took 49 seconds. Results got garbled as they were relayed back up. Sub-agents multiply the decisions a small model must get right, and every hop is another full prompt to process. One agent with ten well-chosen tools beat five agents with 25.

2. Fewer tools, chosen by usage. We built 25 tools and offer ten: the ones behind the most common requests. The rest stay on the server for later or for other MCP clients. Ten tool schemas plus the prompt come to about 2,000 tokens a turn, which the runtime caches. Each tool you add is another option the model can pick wrongly.

3. Let the tools do the thinking the model is bad at. Small models are unreliable at dates, counts and identifiers; ours once filed an event for the right day in the wrong year. So the tools accept the user’s own words and the server resolves them. Dates can be “Friday”, “tomorrow”, “next Monday”, “11 January” or ISO. People can be an id, initials or any part of a name, with honorifics stripped. Records can be an id or a few words from the title, where every word must match and exact filename matches rank first. Counts are computed in code and returned as fields, so the model reads a number instead of counting a list. The prompt then says: pass dates and names in the user’s words and don’t calculate them yourself. It was the single most effective rule we wrote.

4. Ambiguity should come back as choices, not errors. When “mark the budget task done” matches three tasks, the tool doesn’t guess and doesn’t just fail. It returns the candidates with their ids, the model asks the user with the multiple-choice tool, and the second call is exact. Designing tool errors as next steps for the model is as important as designing the happy path.

5. Watch the tokens your framework adds. Our MCP SDK pretty-printed tool results, and about a third of every result was whitespace. A one-line decorator that returns compact JSON cut that. Short keys, ISO dates and flat structures matter when the reader has a small context window and a small attention budget.

6. Small models parrot rules, so scope every rule to its trigger. We told the model that when the user declines a change it should reply “Not done — you declined it.” It started saying that in reply to unrelated questions, including the weather. The fix was to state the condition precisely: only when a tool result says the user rejected it. Rules for a 4B model should read like guarded code, condition first and then action. Absolutes without conditions leak into everything.

7. Name the things it can’t do. If the prompt doesn’t list what’s out of reach, a small model tries to answer from whatever tools it has, and will find leave records in a task list. One sentence, “You cannot see X, Y or Z; when asked, say plainly that isn’t available yet”, turned invented answers into honest ones.

8. Turn thinking off by default, and put the switch in the UI. Hybrid reasoning models think by default. For retrieval-and-form-filling work that made answers four to six times slower without making them more correct. We configure two variants of the same model: the default with reasoning effort set to none, and a think variant at medium effort. A Think toggle in the chat pane sends the second for hard questions and streams the reasoning so the user can watch it. One trap: Ollama’s think: false is ignored on its OpenAI-compatible /v1 endpoint, which is the one gateways and harnesses use. reasoning_effort: none works.

9. Hunt for hidden model calls. The first answer in every new conversation was about ten seconds slow. The harness was quietly asking the same model to write a title for the conversation, and on one local GPU that call queued ahead of the answer. Turning off automatic titles fixed it. On a single local model every background call (titles, summaries, memory extraction) competes with the user’s request. Audit them in the gateway logs and keep only those worth the wait.

10. Keep the model warm and set its context explicitly. Ollama unloads an idle model after five minutes by default, and the next request pays for a cold load; we keep it for thirty. The default context window is also smaller than the model supports, so we build our own model variant with a 32K context. The chat pane shows a context meter so the user can see when a conversation is getting long.

11. Log everything from day one; it’s your fine-tuning set. Because every call goes through the gateway, we have the full request (system prompt, tool schemas, history) and the full response (text, tool calls, reasoning) for every interaction, tagged with its conversation, token counts and cost. One script exports it as chat-format JSONL with tools, ready for fine-tuning. Local calls are logged at zero cost. Adding a paid model is a config entry, and its spend then appears in the same logs. If the small model plateaus, the data to improve it has been collecting since the first conversation.

12. Evaluations rot, because production data moves. After we added memory, one test failed in both runs. The model wasn’t wrong: someone had completed that task in the real app, so “mark it done” no longer had a target. Tests against live data need targets that stay put, and a failure should first raise the question “did the world change?” before “did the model get worse?”. After retargeting, the test passed three times out of three.

A useful assistant remembers how you work. We connected opencode-mem, a memory plugin for the harness that runs entirely locally: local sentence embeddings, a SQLite-compatible vector store, and a review UI where the user can browse, correct or delete what it has learned.

It learns in three ways. Explicitly: the model saves a short fact when the user says “remember…”, corrects it, or states a preference, and the prompt limits it to exactly those moments. Automatically: after a conversation goes quiet, a background call picks out anything worth keeping. As a profile: every few conversations it refreshes a summary of how the user works.

The most relevant memories are added to each new conversation. The cost was modest: about 300 extra tokens a turn, and the median answer rose from 2.9 to 4.1 seconds. We verified the loop end to end: a fact saved in one conversation was used, unprompted, in a fresh one.

Two cautions. Pin plugin versions: the first memory plugin we tried required a newer plugin API than our harness version provides. And keep evaluation runs away from the real memory, or the assistant will learn from your test scripts.

Users want to drop a file into the chat and ask about it, but we didn’t want every dropped file to land in the app’s document library. Attached files go to a temporary chat-files area, pruned after 14 days. The server extracts the text, and the first few thousand characters go straight into the message. If the file is longer, the model reads the rest with the same document-reading tool it uses for filed documents.

A file moves into the real library only when the user asks and approves the card. Images also go to the model as images.

Service managers rewrite your settings. Homebrew’s service wrapper regenerated its launch file and dropped our keep-alive setting; a small launch agent of our own fixed it.

React StrictMode vs. event streams. Development double-mounting aborted the server-sent-event stream. The fix was an effect that owns the stream and closes it cleanly.

A harness’s default model may not be yours. OpenCode reports its own per-provider pick. Read the configured model instead, or your picker defaults to the wrong one.

A cloud model enabled by default. Harnesses often come with their own hosted provider switched on. We allow-listed only our gateway, so no request can bypass logging or leave the machine.

Approval cards for every write. A write tool we added later had no card, so its request waited forever. The pane now has a generic fallback card for any pending approval.

This is one team’s experiment, not a benchmark. The suite is 22 tasks run twice each, on one machine, in one domain, in English only. Model outputs vary between runs, and 95% on 44 runs has wide error bars.

We haven’t yet measured MLX-based runtimes on Apple silicon, which may be faster than Ollama, or compared a paid frontier model on the same suite. The gateway makes both a configuration change.

A chat pane is the start. The same pieces (tools with approval, memory and a full log) are what you need for an assistant that acts on events, not just messages. Picture a daily check that notices a routine record wasn’t filed, works out who is responsible, and drafts a friendly nudge to them, queued for one-click approval.

The model stays small because the work stays narrow. The human stays in the loop because every write is a card. The logs from those approvals become the training data that makes the next version better.

The broader lesson: with a small model, you are designing a system, not choosing a model. Get the tools, the permissions, the prompt, the runtime settings and the evaluation right, and a 4B model running on a laptop does real work in about three seconds, for free and in private.

The stack, for reference: OpenCode 1.18 run headless with one custom agent and deny-by-default permissions; the MCP Python SDK 2.2 over Streamable HTTP, mounted in an existing FastAPI app; LiteLLM with Postgres logs and a JSONL export; Ollama 0.34 on an Apple M4 Pro with 24 GB; Qwen3.5 4B (default), Gemma 4 E4B and Spark X2.5 4B; the opencode-mem plugin; and a React side pane on the OpenCode SDK.

Darvi Labs builds agentic AI systems and the web architecture underneath them. Private, local-first assistants like this one are exactly the kind of work we like: small models, careful tool design, approvals in the loop and evals that prove it works.

If you want an AI assistant inside your own product without sending your data to a third party, get in touch through our Contact page, or read the rest of our writing on the Blogs page.