Article · September 24, 2026
Three days and about forty scripted runs of Gemma 4 and Qwen3 against an ERP over MCP. The pass rate went from 1 of 4 to 4 of 4 without changing the model. What changed was the layer between the model and the system.
I spent three days running a small local model against my own ERP over MCP. Not because I think a 4B-active model is the right tool for bookkeeping. Because a small model does not cover for a missing rule the way a frontier model does, so you can see what the layer between the model and the system is actually doing. This is what I found.
The system is Saybooks, an open-source ERP I built. It has one registry that produces the MCP tools, the web UI, the validation and the audit log, so an agent and a person hit the same rules and land in the same log. It ships with 48 tools for the modules I used here. The second server in some tests was huntctrl, a small job-hunt CRM I also wrote, 23 tools.
The model is Gemma 4 26B, the A4B QAT variant, served by Ollama on a Mac with 32 GB. Later I added Qwen3-30B-A3B. The machine matters, and I will come back to it.
The test is four plain prompts, one conversation. Paste a 20-row card statement. Ask for the rows to be categorised and shown before anything is written. Correct one thing and say go. Ask what was spent and what the subscriptions are. After each prompt a script checks the books, not the model's answer. A step passes if the database is right. I ran this script and a few others about forty times, changing one thing at a time.
The first thing I checked, before any of this, was how LM Studio hands an MCP server to a model. The answer was: it passes the tool descriptions and nothing else. The server sends instructions at connect time, the SDK stores them, and the client never reads them. I then checked Jan and Open WebUI in source. Same. Every local client I could find drops the server's instructions on the floor.
So a local model sees 48 tool descriptions and none of the doctrine. Rules like "read the vocabulary before you propose categories" or "a transfer's category is where the money went" never reach it unless they are also written into tool descriptions and tool results. In Saybooks they are, because I had already learned this the hard way. But that was the starting question: if the instructions did reach the model, how much would change?
Less than I expected on the instructions, and a lot on things I had not thought about.
The first setup, instructions forwarded and thinking off, landed one step out of four, three runs in a row, in 13 to 32 minutes each. The same model with no instructions at all did four out of four once and two out of four once, in 6 to 7 minutes. That is not a good sign for the instructions. Reading the logs, the failures had nothing to do with doctrine.
The model had to retype twenty rows as tool arguments. Every time it did that, one value came out wrong. A date with the wrong year. A fullwidth dot in "20．09-23". A field named "rowintdex". The import is atomic, so one bad row means a full refusal, which means retyping everything, which brings the next slip. Most of the minutes went there.
The second failure was one word. With the instructions present the model used a field called "counterparty" for the printed line and left out "description", which is required. It did this in three runs out of three. The refusal it got back was the database's own message: NOT NULL constraint failed. It answered by adding "description" at the top level of the call instead of in each row. I changed two things. The registry now validates rows itself and says "rows[1] has no description. Each row takes: date, amount, description, ..." And the field text changed from "Exactly as printed." to "The line as the statement prints it (the merchant or payee text). Every row has one." After that the same setup imported on the first attempt, twice.
I would not have guessed that a sentence of field text was worth more than a page of instructions. It was.
The retyping problem had two fixes, in two places.
On the server, the import tool now accepts the statement as pasted text and parses it. The control totals still have to reconcile, so the model still has to read the statement. It just stops retyping it.
On the host, I added files by reference. When the person attaches a file, the model sees it and is told it can pass "@statement.csv" as an argument. The host substitutes the contents before the tool runs. Gemma used this on the first try without being asked twice. The setup that was 1/4 in 13 to 32 minutes went to 4/4 in 6 minutes with zero refusals.
That was the biggest single change. The rest are smaller and each one came from a failure I could see in the log and the model could not:
None of these need a smarter model. They need someone watching the calls.
Thinking is where the instructions finally showed up. With thinking off, the model skipped "read the vocabulary first" about half the time, even with the rule in the tool description, in the doctrine and in the previous tool result. With thinking on it followed the rule nearly every time.
But thinking on a step that hands over a file cost 9 to 10 minutes against 100 seconds with thinking off. The model generates almost 4,000 thinking tokens over a 20-row statement, and generation slows from 35 tokens a second to 14 as the context fills.
There is also a cost I did not expect. Ollama's Gemma template keeps a turn's thinking in the prompt only while it is after the last user message. Once the person speaks again, the thinking is dropped from history. That rewrites the prompt, and the runtime's prefix cache misses from that point. About 110 seconds of prefill on a 15k-token prompt, every turn where the model thought. Thinking off keeps the prompt byte-identical and the cache holds.
So the host runs thinking off on a step that hands over a file and on for every other step. That is the setting behind the 6-minute runs. I tried folding earlier thinking back into history as plain text to keep the cache. It does not work, the bytes differ from what was rendered live. Keeping the prefix with thinking on would mean owning the template and talking to a raw completion endpoint. I have parked that.
I assumed a smaller prompt would be faster. The tool schemas are three quarters of every prompt, about 10,500 tokens for 48 tools, sent on every hop. The instructions are a tenth.
Staging the instructions, sending a module's rules the first time one of its tools is called, works mechanically and saves about one percent. Sending a one-line catalog of tools and opening schemas by name cut the prompt from 14k to 6k tokens per hop. On two servers it passed but took longer. On the statement script it went from 4/4 to 2/4, because "read the vocabulary first" lives in a tool description, and a catalog line cuts it off.
The reason smaller was not faster is the same cache. Gemma's template puts the tool declarations in the system turn. Any change to the tool list rewrites the top of the prompt, and the whole context is prefilled again at about 200 tokens a second. A stable 14k-token prompt that hits the cache beats a 6k one that misses it. Full schemas and full instructions up front, unchanged, is the fastest option this runtime allows.
Qwen3-30B-A3B, same host, same scripts. On the two-server script with no attachment it did four out of four in under five minutes, switched servers on its own and logged the right thing on the right record. The host transfers.
On the statement script it failed every time, across two runtimes. It will not pass a file by reference. It retypes the contents, inverting every sign once and converting every amount to cents another time, or it puts "@statement.csv" in the name argument and leaves the text argument empty. The rule was in the message and in the system prompt. It did not matter.
So the biggest lever I found is partly a Gemma trait. The host rules that catch a model's habits transfer. The affordances a model has to pick up on its own do not, and the host needs a fallback that does not depend on the model's cooperation. I have one built and not yet measured.
Qwen also found a real bug in Saybooks that Gemma never reached, because it reads a setup checklist's optional items as things to do. A second model is coverage, not only comparison.
Everything above is the host measured against variants of itself. So I ran the same four prompts in LM Studio's own chat, twice, same model family, same server from its own mcp.json, the statement pasted the way a person pastes it, and counted the books afterwards.
| steps passed | wall clock | refusals | |
|---|---|---|---|
| LM Studio chat, run 1 | 1 of 4 | 15 min | 4 |
| LM Studio chat, run 2 | 3 of 4 | 11 min | 2 |
| hostel, run 1 | 4 of 4 | 6 min | 0 |
| hostel, run 2 | 3 of 4 | 6 min | 0 |
Both 3 of 4 results are the same miss: my own check and my own tool text disagree about what to call a transfer. That is a decision, not a slip, and the instrument found it.
The first LM Studio run went like this. Three refused imports. Then the model told me "I've successfully imported all 20 transactions" with nothing on the books. The categorisation step went fine. The write step was refused because no rows existed, the re-import landed on the third try, the review was never written, and the model said it had applied my correction. The last question was answered from the pasted text without a tool call.
The second run was better. The import landed on the third attempt, the review was written in one call, and the last question was answered from the books. It still took twice as long as the host, and every import attempt retyped all twenty rows.
The number I care about is not 1 of 4 or 3 of 4. It is the two false claims of success in the first run. Without a host, a refusal does not stop the model from telling you the write happened, and nothing in the chat window tells you otherwise. It happened in one run of two. Checking the books is what made it visible at all.
One more thing that is not about models. LM Studio's MLX engine on this Mac holds a 5 GB helper beside 17 GB of weights, plus the cache. With Docker's VM quit and a 24k context it still locked the machine up and I had to reboot. A watchdog checking every 30 seconds was too slow. Ollama held Gemma at 20 GB for two days on the same machine. The host now checks free memory before a load and acts on the first critical sample. If you run local models for real work, the machine is part of the design.
"The agent is table stakes" is a line I keep reading. What I measured says the model is the smallest variable. The same model went from 1 of 4 to 4 of 4 with nothing changed but the layer in between: whether the rules reached it, whether it had to retype a file, what a refusal said, whether anyone was watching the calls. That layer is unowned in every local client I looked at.
I am building it. It is called hostel for now, and it will be a Mac app. If you want to know when it is ready, or you have a server you would like to see it run against, write to me at peter@portlandaiworks.com.